Backtesting a Neural Network
Yes you can access that data, but it defies the purpose.
Train history, then replay and act on the same data?
Unless you test a part that lies outside of the training data of course.
a simple while(bar<bars) loop may just do the trick.
if you have problems in OnInit() you can also run the code upon the first incoming tick, then set boolean flag runonce to true and skip it on the next ticks.
Thank for your quick response and my apologies for a tardy one of mine.
Forum on trading, automated trading systems and testing trading strategies
Marco vd Heijden, 2016.08.12 18:58
Yes you can access that data, but it defies the purpose.
Train history, then replay and act on the same data?
Unless you test a part that lies outside of the training data of course.
a simple while(bar<bars) loop may just do the trick.
if you have problems in OnInit() you can also run the code upon the first incoming tick, then set boolean flag runonce to true and skip it on the next ticks.
Training data must be from a different interval of time then test data therefore it is not the same. Hence, I am testing a part that lies outside of the training data. This partitioning is required to avoid overfitting the model (neural network) to the training data otherwise the model might perform exceptionally well on backtesting however, not so well on data it has never seen before hence, on live trading. Testing must be done on data the network did not see in training otherwise that would be like going to an exam where all of the exercises in the exam paper are exactly the same (even the numbers) that you had solved in the past. That would be a good test on your memory rather than ability to solve problems.
Please the the code below, hopefully it would add some clarity of what I'm trying to achieve. data[] array would need to contain all of the prices data up to real date (not backtesting date) before calling getWeights function. Then OnTick() would have an if statement (if (testingTime < currentTime)) to check whether to start testing based on time defined while running OnInit():
int iTestTime = vSize*(testingPercentage); //indicates at which bar from now testing starts testingTime = Time[iTestTime]; //time at which testing starts
In my case, when I start backtesting OnInit() has only 100 bars of 1H data. At very end of backtesting OnTick() has around 10000 bars of 1H data which I would need to have while running OnInit() in order to train the network propely.
This is what I thought would work initially however...
#import "LMBRDLL.dll" double getWeights(double &data[], int &topology[], int topSize, double &TVT[], int vSize, int timeSteps, int nVabs, double &weights[]); double testWeights(double &weights[], double ¤tData[], int &topology[], int topSize, int timeSteps, int nVabs); #import #include <stdlib.mqh> string symbol = Symbol(); ENUM_TIMEFRAMES timeframe = Period(); double weights[]; //array contains weight values of the network int nVabs = 0; //number of variables used (close, open, high, low) initialized to zero int topology[]; //array contains number of layers (topology[] size) and neurons in specific layer (element value) int topSize = 0; //topology size will be the size of topology[] initialized to zero double TVT[3]; //array contains percentages of data that will be used training, validation and testing int vSize; //vector size: a number of bars of available data datetime testingTime; //point of time where testing (placing orders) starts input double Lots = 0.01; input double TakeProfit = 500; input double StopLoss = 300; input int timeSteps = 3; //number of bars that will be used to predict next closing price input int hiddenLayers = 2; //number of hidden layers of the network input int hiddenLayerNeurons = 4; //number of neurons in each hidden layer input double trainingPercentage = 0.85; //percentage of historical data that will be used for training input double validationPercentage = 0.00; //percentage of historical data that will be used for validation input double testingPercentage = 0.15; //percentage of historical data that will be used for testing input bool open = true; //open prices to be used in the network input bool high = false; //high prices to be used in the network input bool low = false; //low prices to be used in the network //+------------------------------------------------------------------+ //| Expert initialization function | //+------------------------------------------------------------------+ int OnInit() { //--- TVT[0] = trainingPercentage; TVT[1] = validationPercentage; TVT[2] = testingPercentage; int i = 0; vSize = Bars(symbol, timeframe) - 1; //minus one because current bar cannot be used due to absents of final close, high and low prices int iTestTime = vSize*(testingPercentage); //indicates at which bar from now testing starts testingTime = Time[iTestTime]; //time at which testing starts int vCount = 1; //variable count initialized to one because closing prices are compulsory open == true ? vCount++ : vCount; high == true ? vCount++ : vCount; low == true ? vCount++ : vCount; nVabs = vCount; topSize = hiddenLayers + 2; //all hidden layers + input and output layers ArrayResize(topology, topSize); for (i = 0; i < topSize; i++) { if (i == 0) { topology[i] = timeSteps*nVabs; //number of neurons in input layer } else if (i == hiddenLayers + 1) { topology[i] = 1; //number of neurons in output layer which is always one, predicted closing price } else { topology[i] = hiddenLayerNeurons; //number of neurons in hidden layer } } double data[]; ArrayResize(data, nVabs*vSize); //data[] size: number or variables used times number of bars available int indexdata = 0; //point in data[] where data of next variable is added int indexadd = 0; //incremented after each bar of each variable is added and assigned to indexdata after each variable is fully added //adds close prices to data[] double C[]; ArrayResize(C, vSize); for (i = 0; i<vSize; i++) { C[i] = Close[(vSize)-i]; } for (i = indexdata; i<vSize; i++) { data[i] = C[i]; indexadd++; } indexdata = indexadd; //adds open prices to data[] if (open == true) { double O[]; ArrayResize(O, vSize); for (i = 0; i<vSize; i++) { O[i] = Open[(vSize)-i]; } for (i = 0; i<vSize; i++) { data[i + indexdata] = O[i]; indexadd++; } } indexdata = indexadd; //adds high prices to data[] if (high == true) { double H[]; ArrayResize(H, vSize); for (i = 0; i<vSize; i++) { H[i] = High[(vSize)-i]; } for (i = 0; i<vSize; i++) { data[i + indexdata] = H[i]; indexadd++; } } indexdata = indexadd; //adds low prices to data[] if (low == true) { double L[]; ArrayResize(L, vSize); for (i = 0; i<vSize; i++) { L[i] = Low[(vSize)-i]; } for (i = 0; i<vSize; i++) { data[i + indexdata] = L[i]; indexadd++; } } indexdata = indexadd; //computes number of weights used in the network and risizes weights[] double weightsSize = ((timeSteps*nVabs)*hiddenLayerNeurons + (hiddenLayers - 1)*(hiddenLayerNeurons*hiddenLayerNeurons) + hiddenLayerNeurons * 1); ArrayResize(weights, weightsSize); //calls getWeights from the DLL. The function divides data[] into training, validation and testing data //then trains, validates the network and fills weights[] which is then used running OnTick() getWeights(data, topology, topSize, TVT, vSize, timeSteps, nVabs, weights); //--- return(INIT_SUCCEEDED); } //+------------------------------------------------------------------+ //| Expert tick function | //+------------------------------------------------------------------+ void OnTick() { datetime currentTime = TimeCurrent(); if (testingTime < currentTime) { //checks whether it is testing time or not if ((iVolume(symbol, timeframe, 0) == 1)) { int i = 0; double currentData[]; //contains data out of which prediction of next closing price is being made ArrayResize(currentData, nVabs*(timeSteps + 1)); //one is added because prices are being converted into percentage changes int indexdata = 0; int indexadd = 0; //adds close prices to currentData[] double C[]; ArrayResize(C, (timeSteps + 1)); for (i = 0; i<(timeSteps + 1); i++) { C[i] = Close[(timeSteps + 1) - i]; } for (i = indexdata; i<(timeSteps + 1); i++) { currentData[i] = C[i]; indexadd++; } indexdata = indexadd; //adds open prices to currentData[] if (open == true) { double O[]; ArrayResize(O, (timeSteps + 1)); for (i = 0; i<(timeSteps + 1); i++) { O[i] = Open[(timeSteps + 1) - i]; } for (i = 0; i<(timeSteps + 1); i++) { currentData[i + indexdata] = O[i]; indexadd++; } } indexdata = indexadd; //adds high prices to currentData[] if (high == true) { double H[]; ArrayResize(H, (timeSteps + 1)); for (i = 0; i<(timeSteps + 1); i++) { H[i] = High[(timeSteps + 1) - i]; } for (i = 0; i<(timeSteps + 1); i++) { currentData[i + indexdata] = H[i]; indexadd++; } } indexdata = indexadd; //adds low prices to currentData[] if (low == true) { double L[]; ArrayResize(L, (timeSteps + 1)); for (i = 0; i<(timeSteps + 1); i++) { L[i] = Low[(timeSteps + 1) - i]; } for (i = 0; i<(timeSteps + 1); i++) { currentData[i + indexdata] = L[i]; indexadd++; } } indexdata = indexadd; double output = testWeights(weights, currentData, topology, topSize, timeSteps, nVabs); //DLL function returns predicted percentage change double predictedPrice = output*Close[1] + Close[1]; //converts predicted percentage change into predicted price if (output > 0) { //places BUY order if predicted percentage change is above zero OrderSend(Symbol(), OP_BUY, Lots, Ask, 3, Ask - StopLoss*Point, Ask + TakeProfit*Point, "NeuralNet", 777, 0, Green); } if (output < 0) { //places SELL order if predicted percentage change is below zero OrderSend(Symbol(), OP_SELL, Lots, Bid, 3, Bid + StopLoss*Point, Bid - TakeProfit*Point, "NeuralNet", 777, 0, Red); } } } }
Do not use OnInit for lengthy operations requiring access for whole history. Move all your logic into OnTick. Start training/backtesting only after sufficient data is loaded.
If you need to have many bars at very beginning, you should move the start time of the tester back in the past, then skip a required number of bars in OnTick until you get the time initially planned as start, and only then switch to the training/backtesting.
Do not use OnInit for lengthy operations requiring access for whole history. Move all your logic into OnTick. Start training/backtesting only after sufficient data is loaded.
If you need to have many bars at very beginning, you should move the start time of the tester back in the past, then skip a required number of bars in OnTick until you get the time initially planned as start, and only then switch to the training/backtesting.
Thanks Stanislav,
I get what you are saying. I was just naively hoping there is a way to access the historical data without extra fiddliness.
Only dividing data into percentages I will be able to work with
different time frames efficiently and make the algorithm user-friendly. Getting whole number of bars and data OnInit() at the very beginning would have been easy to work with solution.
Now, unless there is some function in MQL4 to get actual time (as it is on my PC) I will have to call another DLL function to get the actual time, calculate time difference between TimeCurrent() and value returned by DLL function and then divide the difference percentage wise in order to know when to start training/backtesting.
Adrijus
Why not to start the tester from very deep history (if not at very beginning of available data) and process bars according to their numbers (I suppose you know number of patterns you want to feed into NN)? For example if you need 10000 bars in total (and 10% of them for validation and 10% of them for forward test) skip bars with numbers larger 10000, use 10000-2000 for training, 2000-1000 for validation, and the rest for testing.
Use of multiple timeframes is not clear so far. Also I don't get why you need real time in the tester - it looks like a flaw.
int barsToWait=5000; //+------------------------------------------------------------------+ //| Expert tick function | //+------------------------------------------------------------------+ void OnTick() { //--- static int timeToStart=(int)Time[0]+_Period*60*barsToWait; if((int)Time[0]<timeToStart) return; //--- }
Why not to start the tester from very deep history (if not at very beginning of available data) and process bars according to their numbers (I suppose you know number of patterns you want to feed into NN)? For example if you need 10000 bars in total (and 10% of them for validation and 10% of them for forward test) skip bars with numbers larger 10000, use 10000-2000 for training, 2000-1000 for validation, and the rest for testing.
Use of multiple timeframes is not clear so far. Also I don't get why you need real time in the tester - it looks like a flaw.
Indeed I am starting at the very beginning of available data by setting the tester to test from earliest time point available.
Not knowing how many patterns will be in use is the problem I am trying to solve right now. The more patterns, the better NN will be trained and by setting them to some rigid number I would probably miss some of the data available which would be really inefficient. Hence, to use all of the data the algorithm needs to know how much of it is available so it could divide it before starting anything else. I could, for example, use all bars for training up to some certain number (allOfBars -> trainingData <- 2000 -> testingData <- 0), that would not miss any of the data available however, partitioning would be very disproportional and most of the data could be used for training (which is not so bad) or for testing (which would not train NN as well as it could be trained).
By working with different time frames (M1, M5, M15, etc.) I could not set a valid number of patterns to feed into NN. For instance, if I set 10000 bars to be used with hourly data that might be OK however, if I set up same number of bars for minute data then a lot of available data, that could be used for training, would be missed because probably there are more than 10000 bars available. Vice versa, if there are less then 10000 bars available, lets say for daily data, then setting a number like this would cause an error.
By knowing real time in the tester algorithm could divide whole interval of time (from TimeCurrent() to real time) into intervals of training/validation data and testing data according to percentages. Then it would train the NN at a specific time (with the bars loaded) and start placing orders after NN is trained.
My apologies if that does not look sophisticated way of testing NN but I cannot think of any better way.
Adrijus
Indeed I am starting at the very beginning of available data by setting the tester to test from earliest time point available.
Not knowing how many patterns will be in use is the problem I am trying to solve right now. The more patterns, the better NN will be trained and by setting them to some rigid number I would probably miss some of the data available which would be really inefficient. Hence, to use all of the data the algorithm needs to know how much of it is available so it could divide it before starting anything else. I could, for example, use all bars for training up to some certain number (allOfBars -> trainingData <- 2000 -> testingData <- 0), that would not miss any of the data available however, partitioning would be very disproportional and most of the data could be used for training (which is not so bad) or for testing (which would not train NN as well as it could be trained).
By working with different time frames (M1, M5, M15, etc.) I could not set a valid number of patterns to feed into NN. For instance, if I set 10000 bars to be used with hourly data that might be OK however, if I set up same number of bars for minute data then a lot of available data, that could be used for training, would be missed because probably there are more than 10000 bars available. Vice versa, if there are less then 10000 bars available, lets say for daily data, then setting a number like this would cause an error.
By knowing real time in the tester algorithm could divide whole interval of time (from TimeCurrent() to real time) into intervals of training/validation data and testing data according to percentages. Then it would train the NN at a specific time (with the bars loaded) and start placing orders after NN is trained.
I see your point, but I think you're wrong in your base statement, which I marked in bold. I'm not sure which specific type of NN you use, but most of modern NNs are known to have a "memory size", that is a number of patterns which they can successfuly learn (with good generalization and without overfitting) at a given NN size. I assume you have a known structure of NN, so you should calculate the number of patterns to learn, and this "rigid number" will only provide an optimum operation of NN, and eliminate the need to know real time. Moreover, markets change over time, so H1 based systems should be probably limited to 1-2 years of history, and lesser timeframes - to even lesser period. Otherwise you'll train NN on outdated irrelevant data, IMHO.
Yes, I do understand the the sentence you marked in bold is really
debatable. I have not heard about a term "memory size" before however, I
fully understand what you mean by that as it makes perfect sense.
Somehow I made an impression that the more training data, the better and
never thought about this concept of memory size.
Trying not to deviate from the topic too much, I am using a Feed-forward Neural Network with it's inputs as Tapped delay line hence, overall known as Time delay neural network. It uses Levenberg-Marquardt algorithm for quicker convergence and Bayesian Regularization to reduce overfitting. Also it stops training when validation error stops decreasing. As I understand this TDNN is really susceptible to memory size. The way it constructed right now is that data[] is being converted into a matrix of numberOfVariables*timeDelays by numberOfPatterns. Then it is being divided into training and validation matrices and testing part is disregarded and training part is fed through NN. That was all under assumption that all of the data is available at the very beginning (how naive).
All that being said I will have to have take your advise for the sake of better generalization without increasing NN size where it would make the training unbearably slow and even then it might not generalize well enough. In terms of data being outdated I though about it myself however, I was going to backtest with all data before limiting it to see what kind of difference it would make (maybe just my curiosity). I am also concerned about the amount of data my broker provides on lets say EUR/USD 1H which is only about 1.5 years so if choose longer period of time for more patterns that might cause some issues.
Regards,
Adrijus
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
Hello,
Can a backtesting be done where all of historical data (up to real date) is needed before algorithm starts placing trades in backtesting mode? The logic behind it is that a neural network has to be trained with a historical data. However, for backtesting results not to be biased all of the data is divided into three parts: training, validation and testing. The last one is used in backtesting and the other two are needed in training process. For instance, if we have a period of 100 days, the data of the first 70 days could be used for training of the algorithm. Days 71-85 would contain validation data and then algorithm would only start placing trades in the day 86.
At first I thought that the network could be trained and validated during run of OnInit() and the rest could be done OnTick(), however, during backtesting OnInit() does not contain all of the historical data (all 100 days) that is up to date and could only be used in live trading. Please, feel free to correct me if I’m wrong but that is the impression I got while debugging.
I hope this makes sense and if not so much I might put the code later on.
Regards,
Adrijus