We have prepared a set of images for training the network. I got images of size 449x449 pixels.

Prepare a set of images for testing. Run the script in the "Test" mode.

A set of sequential hourly screenshots will be saved to ...\MQL5\Files. They are 12558 now. Do not separate or group them, as the neural network should perform this grouping by itself. To be more precise, the network should show the probability of that an image corresponds to the conditions under which the network was trained. Upward move and turn down. Downward move and turn up. 

Cut these files and past to ...CNN\Test\Resp.

We have prepared a set of images for testing the responses and optimizing the strategy. The files with date and time EURUSDDate, remaining under ...\MQL5\Files, should be moved to the CNN folder.

I would like to note one specific feature of MetaTrader 5. It would be more convenient and reliable to prepare a set of images using an Expert Advisor in the strategy tester. However, MetaTrader 5 does not provide the ability to create screenshots in the strategy tester. Anyway, this specific feature will not affect trading robot creation.

Let us consider every block of the program code.

First load the necessary modules of neural network libraries.

from tensorflow.python.keras.preprocessing.image import ImageDataGenerator
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.layers import Conv2D, MaxPooling2D
from tensorflow.python.keras.layers import Activation, Dropout, Flatten, Dense
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras.models import load_model

Then set the required hyperparameters.

# Directory with data for training
train_dir = 'train'
# Directory with data for validation
val_dir = 'val'
# Image dimensions
img_width, img_height = 449, 449
# Image-based tensor dimension for input to the neural network
# backend Tensorflow, channels_last
input_shape = (img_width, img_height, 3)
# Number of epochs
epochs = 20
# Mini-sample size
batch_size = 7
# Number of images for training
nb_train_samples = 4289
# Number of images for validation
nb_validation_samples = 1836
# Number of images for testing
#nb_test_samples = 3736

Let us create the network architecture.

• Set a sequential convolutional neural network architecture
• The input image is of size 449x449 pixels, three-channel (red, green and blue). Here we use a color image. You can also try a monochrome image
• Create the first convolution layer for working with two-dimensional data: 96 feature maps each having it sown 3x3 convolution kernel. Each neuron of the convolutional layer is connected to a 3x3 square section of the image 
• Activation layer using the "relu" function, which is less computationally intensive
• To reduce the dimension, add a subsampling layer with a 2x2 kernel with the selection of the maximum value from this square
• Next, add two more convolution layers with 32 and 16 cores, two activation layers and two subsamples
• Converting two-dimensional data to one-dimensional format
• Pass the converted data to the fully connected layer having 64 neurons
• Using the Dropout(0.5) regularization layer function try to avoid overfitting
• Add a fully connected output layer with one neuron. Two image classes are used, so network response will be received in a binary response. It is also possible to try several classes. For example, two for trend and one for flat. In this case, the output layer would have three neurons. Also, we would need to divide images into three classes.
• The "sigmoid" activation function. It is good for classification and it has shown the best performance during my experiments

This example of architecture can be easily modernized - we can increase the number of layers and their sizes, change their position depending on the sequence, change the dimension of the convolution kernel, modify activation functions and work with fully connected layers. However, a dilemma arises here: when using a GPU, it is necessary to increase the RAM of a video card in case we want to increase the architecture of a neural network. Optionally, the image size should be reduced. Otherwise, we can spend a lot of time using CPU.

model = Sequential()
model.add(Conv2D(96, (3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(16, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))

Compile the neural network. Use the error function: binary-crossentropy. In this case, we will use a response consisting of two classes which should take the values 0 or 1, ideally. However, actual values will be distributed from 0 to 1. Select the Gradient Descent optimizer. It seems to be the most suitable for neural network training. Choose the accuracy metric, i.e. the percentage of correct answers. 

model.compile(loss='binary_crossentropy',
              optimizer='sgd',
              metrics=['accuracy'])

Normalize the data on the intensity of image pixels.

datagen = ImageDataGenerator(rescale=1. / 255)

Use Keras generators to read data from the disk and to create the training and validation image arrays for the neural network. Again, class_mode is 'binary'. Set Shuffle to 'False'. So, image shuffling is disabled.

train_generator = datagen.flow_from_directory(
    train_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='binary',
    shuffle=False)
val_generator = datagen.flow_from_directory(
    val_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='binary',
    shuffle=False)

The callbacks function saves the trained neural network after each epoch. This allows selecting the most suitable network in terms of error values and hit percentage.

callbacks = [ModelCheckpoint('cnn_Open{epoch:1d}.hdf5')]

Now, proceed to network training.

model.fit(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=val_generator,
    validation_steps=nb_validation_samples // batch_size,
    callbacks=callbacks)

12. Run the program as shown in the figure.

If the previous steps have been performed correctly, the neural network will start learning.

After the end of the training process, 20 trained neural networks will appear under the CNN folder.

Let us view training results and select a neural network for further use.

At first glance, at epoch 18 the neural network learned with an error rate of 30% and with 85% correct results. However, when running the neural network on the validation set, we see that the error grows, and the percentage of correct answers falls. SO, we should select a network trained at epoch 11. It has the most suitable results at the validation set: val_loss = 0.6607 and val_accuracy = 0.6129. Ideally, the error value should tend to 0 (or at least no more than 35-40%), while the accuracy should be close to 1 (at least no less than 55-60%). In this case optimization could be omitted or could be performed by minimum parameters to improve trading quality. Even with these training results it is possible to create a profitable trading system.

Interpreting the neural network response

Let us now check if all the above work has any practical meaning.

Run Jupyter Notebook without GPU support and open Test.jpynb from the CNN directory.

Let us consider the code blocks.

First load the necessary modules of neural network libraries.

from tensorflow.python.keras.preprocessing.image import ImageDataGenerator
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.layers import Conv2D, MaxPooling2D
from tensorflow.python.keras.layers import Activation, Dropout, Flatten, Dense
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras.models import load_model
import pandas as pd

Set required parameters.

predict_dir = 'Test'
img_width, img_height = 449, 449
nb_predict_samples = 12558

Read data and time of tested images from the file.

Date=pd.read_csv('EURUSDDate.csv', delimiter=';',header=None)

Load the neural network saved after the 11th epoch.

model=load_model('cnn_Open11.hdf5')

Normalize the images.

datagen = ImageDataGenerator(rescale=1. / 255)

Read data from the disk using the generator.

predict_generator = datagen.flow_from_directory(
    predict_dir,
    target_size=(img_width, img_height),
    shuffle=False)

Get responses from the neural network.

indicator=model.predict(predict_generator, nb_predict_samples )

Show the result. Feedback obtaining process takes much time, so below is a notification of process completion.

print(indicator)

Save the obtained result to a file.

Date=pd.DataFrame(Date)
Date['0'] =indicator
Date.to_csv('Indicator.csv',index=False, header=False,sep=';')

Run the program. Once it is complete, the Indicator.csv file will be created under the CNN directory.

Move it to C:\Users\...\AppData\Roaming\MetaQuotes\Terminal\Common\Files.

Run indicator NWI on the EURUSD H1 chart.

//+------------------------------------------------------------------+
//|                                                          NWI.mq5 |
//|                                 Copyright © 2019, Andrey Dibrov. |
//+------------------------------------------------------------------+
#property copyright "Copyright © 2019, Andrey Dibrov."
#property link      "https://www.mql5.com/en/users/tomcat66"
#property version   "1.00"
#property indicator_separate_window
#property indicator_buffers 2
#property indicator_plots   2
#property indicator_type1   DRAW_LINE
#property indicator_type2   DRAW_LINE
#property indicator_color1  Red
#property indicator_color2  DodgerBlue


int Handle;
int i;
int h;
input int Period=5;
double    ExtBuffer[];
double    SignBuffer[];
datetime Date1;
datetime Date0;
string File_Name="Indicator.csv";

//+------------------------------------------------------------------+
//| Custom indicator initialization function                         |
//+------------------------------------------------------------------+
void OnInit()
  {
   SetIndexBuffer(0,ExtBuffer,INDICATOR_DATA);
   SetIndexBuffer(1,SignBuffer,INDICATOR_DATA);
   IndicatorSetInteger(INDICATOR_DIGITS,5);
   Handle=FileOpen(File_Name,FILE_CSV|FILE_SHARE_READ|FILE_ANSI|FILE_COMMON,";");
   //FileClose(Handle);
  }
//+------------------------------------------------------------------+
//| Relative Strength Index                                          |
//+------------------------------------------------------------------+
int OnCalculate(const int rates_total,
                const int prev_calculated,
                const int begin,
                const double &price[])
  {
   MqlDateTime stm;
   Date0=StringToTime(FileReadString(Handle));
   i=iBarShift(NULL,PERIOD_H1,Date0,false);
   Handle=FileOpen(File_Name,FILE_CSV|FILE_SHARE_READ|FILE_ANSI|FILE_COMMON,";");
   ArraySetAsSeries(ExtBuffer,true);
   ArraySetAsSeries(SignBuffer,true);
   while(!FileIsEnding(Handle) && !IsStopped())
     {
      Date1=StringToTime(FileReadString(Handle));
      ExtBuffer[i]=StringToDouble(FileReadString(Handle));
      h=Period-1;
      if(i>=0)
        {
         while(h>=0)
           {
            SignBuffer[i]=SignBuffer[i]+ExtBuffer[i+h];
            h--;
           }
        }
      SignBuffer[i]=SignBuffer[i]/Period;
      TimeToStruct(Date1,stm);
      i--;
     }
   FileClose(Handle);
   return(rates_total);
  }
//+------------------------------------------------------------------+

For simplicity, we will interpret the network responses using the intersection of the main indicator line with a simple average line. Use the TestCNN Expert Advisor.

//+------------------------------------------------------------------+
//|                                                      TestCNN.mq5 |
//|                                 Copyright © 2019, Andrey Dibrov. |
//+------------------------------------------------------------------+
#property copyright " Copyright © 2019, Andrey Dibrov."
#property link      "https://www.mql5.com/en/users/tomcat66"
#property version   "1.00"
#property strict

#include<Trade\Trade.mqh>

CTrade  trade;

input int Period=5;
input int H1;
input int H2;
input int H3;
input int H4;
input int LossBuy;
input int ProfitBuy;
input int LossSell;
input int ProfitSell;

ulong TicketBuy1;
ulong TicketSell0;

datetime Count;

double Per;
double Buf_0[];
double Buf_1[];
bool send1;
bool send0;

int h=4;
int k;
int K;
int bars;
int Handle;
//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
int OnInit()
  {
//---

   Handle=FileOpen("Indicator.csv",FILE_CSV|FILE_SHARE_READ|FILE_ANSI|FILE_COMMON,";");

   while(!FileIsEnding(Handle)&& !IsStopped())
     {
      StringToTime(FileReadString(Handle));
      bars++;
     }
   FileClose(Handle);
   ArrayResize(Buf_0,bars);
   ArrayResize(Buf_1,bars);
   Handle=FileOpen("Indicator.csv",FILE_CSV|FILE_SHARE_READ|FILE_ANSI|FILE_COMMON,";");

   while(!FileIsEnding(Handle)&& !IsStopped())
     {
      Count=StringToTime(FileReadString(Handle));
      Buf_0[k]=StringToDouble(FileReadString(Handle));
      h=Period-1;
      if(k>=h)
        {
         while(h>=0)
           {
            Buf_1[k]=Buf_1[k]+Buf_0[k-h];
            h--;
           }
         Buf_1[k]=Buf_1[k]/Period;
        }
      k++;
     }
   FileClose(Handle);

   int deviation=10;
   trade.SetDeviationInPoints(deviation);
   trade.SetTypeFilling(ORDER_FILLING_RETURN);
   trade.SetAsyncMode(true);

//---
   return(INIT_SUCCEEDED);
  }

//+------------------------------------------------------------------+
//| Expert deinitialization function                                 |
//+------------------------------------------------------------------+
void OnDeinit(const int reason)
  {
//---

  }
//+------------------------------------------------------------------+
//| Expert tick function                                             |
//+------------------------------------------------------------------+
void OnTick()
  {
//---
   MqlDateTime stm;
   TimeToStruct(TimeCurrent(),stm);

   int    digits=(int)SymbolInfoInteger(_Symbol,SYMBOL_DIGITS);
   double point=SymbolInfoDouble(_Symbol,SYMBOL_POINT);
   double PriceAsk=SymbolInfoDouble(_Symbol,SYMBOL_ASK);
   double PriceBid=SymbolInfoDouble(_Symbol,SYMBOL_BID);

   double SL1=NormalizeDouble(PriceBid-LossBuy*point,digits);
   double TP1=NormalizeDouble(PriceAsk+ProfitBuy*point,digits);
   double SL0=NormalizeDouble(PriceAsk+LossSell*point,digits);
   double TP0=NormalizeDouble(PriceBid-ProfitSell*point,digits);

   if(LossBuy==0)
      SL1=0;

   if(ProfitBuy==0)
      TP1=0;

   if(LossSell==0)
      SL0=0;

   if(ProfitSell==0)
      TP0=0;

//---------Buy1
   if(send1==false && K>0 && Buf_0[K-1]>Buf_1[K-1] && Buf_0[K]<Buf_1[K] && iLow(NULL,PERIOD_H1,1)<iLow(NULL,PERIOD_H1,2) && stm.hour>H1 && stm.hour<H2 && H1<H2)
     {
      send1=trade.PositionOpen(_Symbol,ORDER_TYPE_BUY,1,PriceAsk,SL1,TP1);
      TicketBuy1 = trade.ResultDeal();
     }

   if(send1==true && K>0 && Buf_0[K-1]<Buf_1[K-1] && Buf_0[K]>Buf_1[K] && iHigh(NULL,PERIOD_H1,1)>iHigh(NULL,PERIOD_H1,2))
     {
      trade.PositionClose(TicketBuy1);
      send1=false;
     }

//---------Sell0

   if(send0==false && K>0 && Buf_0[K-1]<Buf_1[K-1] && Buf_0[K]>Buf_1[K] && iHigh(NULL,PERIOD_H1,1)>iHigh(NULL,PERIOD_H1,2) && stm.hour>H3 && stm.hour<H4 && H3<H4)
     {
      send0=trade.PositionOpen(_Symbol,ORDER_TYPE_SELL,1,PriceBid,SL0,TP0);
      TicketSell0 = trade.ResultDeal();
     }

   if(send0==true && K>0 && Buf_0[K-1]>Buf_1[K-1] && Buf_0[K]<Buf_1[K] && iLow(NULL,PERIOD_H1,1)<iLow(NULL,PERIOD_H1,2))
     {
      trade.PositionClose(TicketSell0);
      send0=false;
     }
   K++;
  }
//+------------------------------------------------------------------+

Let us optimize simultaneously by the signal line period, time and stop orders for deals in both directions.

The neural network was trained on a time period preceding the chart line, and optimization was performed up to the vertical red line. Then test was performed for optimized neural network responses. The above charts show two random positive optimization results, which the optimizer has shown as the highest priority result.