help creating a deep learning EA
hi
I don't know much about mql5, but need help on how to approach this.
I whant to make a deep learning EA, that has;
1º The first inputs ohlc
2º Theese inputs are used as inputs in all the oscilators, indicatos
3º The outputs of 2º) are used as the input layer of the deep learnig
... and so on as explained here (with probably two hidden layers or more) Programming a Deep Neural Network from Scratch using MQL Language - Neural Networks - 26 January 2019 - Traders' Blogs (mql5.com)
So, here is my q:
How can I do from step 1 to step 2?
Is step 2º an EA or a header?
having trouble to do steps 1 to 2, so any help here will be appreciated
cheers!
You can use the indicators as inputs to the network but you are correct to assume that giving the raw data to the network it may "build" its own indicator inside the hidden layers.
Fastest route is exporting the data and using pytorch or other libraries to process it deploy the networkYou can use the indicators as inputs to the network but you are correct to assume that giving the raw data to the network it may "build" its own indicator inside the hidden layers.
Fastest route is exporting the data and using pytorch or other libraries to process it deploy the networkWouldn't I just have to normalize the imputs, before passing it to de dl in ontick for example?
Please explain more why I should pass the data to python instead of making the dl from scratch and solving with the tester? (the whole point of doing this is to use the tester ... but you might be correct coz python is very fast)
I see another inconvenient ... I have not been able to create libreries ... so I just can use the oscilators and indicators I have in mt5 ... that data differs from python's talib or other. and also: how can I extract the data from the inputs to python?
The prob I see ... I to have the same oscilators and indicators in both mt5 and python (thats too much work! when you have a simple command in ta that uses all together ... ) (if you whant it ... here's a link (donBotto https://raw.githubusercontent.com/jsgaston001/tradingbots/main/donBotto.py ) and also how what I've said of the data ... )
Yes, it might be easier to do it in python (at least) ... but, can you test it? are you using the dl with the parameters that are given from the py? does it work correctly? ... What inputs are you using in the mql (inputs and weights are gotten from python? how does mt5 know its like the python one? does it trade ontick?
Wouldn't I just have to normalize the imputs, before passing it to de dl in ontick for example?
Please explain more why I should pass the data to python instead of making the dl from scratch and solving with the tester? (the whole point of doing this is to use the tester ... but you might be correct coz python is very fast)
I see another inconvenient ... I have not been able to create libreries ... so I just can use the oscilators and indicators I have in mt5 ... that data differs from python's talib or other. and also: how can I extract the data from the inputs to python?
The prob I see ... I to have the same oscilators and indicators in both mt5 and python (thats too much work! when you have a simple command in ta that uses all together ... ) (if you whant it ... here's a link (donBotto https://raw.githubusercontent.com/jsgaston001/tradingbots/main/donBotto.py ) and also how what I've said of the data ... )
Yes, it might be easier to do it in python (at least) ... but, can you test it? are you using the dl with the parameters that are given from the py? does it work correctly? ... What inputs are you using in the mql (inputs and weights are gotten from python? how does mt5 know its like the python one? does it trade ontick?
How would you do it in the tester ?
You'd train the model and test the trading at the same time ?
You can separate the two into the signal and its management .
Python will take over training the signal .
Once you arrive at a configuration for the network , because unfortunately you have to do that to , you will train it per a time period .
For instance , let's say you want to trade every day . Since it is important for the past and the "live" to be symmetrical in all ways possible , you would split your data into daily segments .
Each day represents an advance in the training of the network .
You maintain a buffer of data and you decide on "when" in your data , the test starts .
The part from the start of your data till the "start of the test" is the first "crunch" your network must do.
Once (and if) that reaches a decent loss level you save it and move on to the next day.(the first crunch will take the longest)
You will store the optimal state of your network for each day , not one state but many states.
Then you can extract a measurement that tells you how well the signal performs on "yesterday's" (each yesterday's) network state.
To simplify , if you had a neural net to trade with today it could have been trained up to yesterday's data, so your measurement is giving you an estimate
of how good the signals ballpark . Then you take these states , load them on the tester and let the genetic algorithm work on trade management of those signals .
If you give the neural network indicator outputs as its inputs for the samples then you are asking the network to become the best signal for these indicators with these indicator parameters that produced the indicator values , you are not finding optimal indicator parameters .
How would you do it in the tester ?
You'd train the model and test the trading at the same time ?
You can separate the two into the signal and its management .
Python will take over training the signal .
Once you arrive at a configuration for the network , because unfortunately you have to do that to , you will train it per a time period .
For instance , let's say you want to trade every day . Since it is important for the past and the "live" to be symmetrical in all ways possible , you would split your data into daily segments .
Each day represents an advance in the training of the network .
You maintain a buffer of data and you decide on "when" in your data , the test starts .
The part from the start of your data till the "start of the test" is the first "crunch" your network must do.
Once (and if) that reaches a decent loss level you save it and move on to the next day.(the first crunch will take the longest)
You will store the optimal state of your network for each day , not one state but many states.
Then you can extract a measurement that tells you how well the signal performs on "yesterday's" (each yesterday's) network state.
To simplify , if you had a neural net to trade with today it could have been trained up to yesterday's data, so your measurement is giving you an estimate
of how good the signals ballpark . Then you take these states , load them on the tester and let the genetic algorithm work on trade management of those signals .
If you give the neural network indicator outputs as its inputs for the samples then you are asking the network to become the best signal for these indicators with these indicator parameters that produced the indicator values , you are not finding optimal indicator parameters .
thanks for all and for all your responses.
I didn't thought of that (last part), you are right ... but ... Ive used a dl and didn't give good results, so ...
are you getting good results? if so, could you please help me build a better dl? I used this one ( Programming a Deep Neural Network from Scratch using MQL Language - Neural Networks - 26 January 2019 - Traders' Blogs (mql5.com) ) and hopping to get better results, modified it by adding more hidden layers but results where worst ... could you give me more hits of what to do to make it more accurate? or whats your opinion of that neural netw?
Also ... from my clases of the uni .. I learned that predictive models are good for 3 time periods (more give much less accuracy) .... I used 1 time period with ml, to be as accurate as possible, but still ... didn't reach good results ... If your bot is looking for the hh or ll over the future in a big sample ... it might not be so good (I learned this with climate forecast).... How are you taking into account this?
I know ... it should give good results ... but in practice it doesn't (at least that has been my case with ml and dl) ... so thats why I whant to make it auto, to keep and test it with real data (before spending more money) ... Could there be some way to make it get the optimal parameters while trading ? don't mind how much cpu is using (for that we have servers) . what does python use to approach the best parameters? I suppose it can be reproduceble in mql5, it's just an output that we would use as input ... and ... lets have a nice day out :)
- 2019.01.27
- www.mql5.com
thanks for all and for all your responses.
I didn't thought of that (last part), you are right ... but ... Ive used a dl and didn't give good results, so ...
are you getting good results? if so, could you please help me build a better dl? I used this one ( Programming a Deep Neural Network from Scratch using MQL Language - Neural Networks - 26 January 2019 - Traders' Blogs (mql5.com) ) and hopping to get better results, modified it by adding more hidden layers but results where worst ... could you give me more hits of what to do to make it more accurate? or whats your opinion of that neural netw?
Also ... from my clases of the uni .. I learned that predictive models are good for 3 time periods (more give much less accuracy) .... I used 1 time period with ml, to be as accurate as possible, but still ... didn't reach good results ... If your bot is looking for the hh or ll over the future in a big sample ... it might not be so good (I learned this with climate forecast).... How are you taking into account this?
I know ... it should give good results ... but in practice it doesn't (at least that has been my case with ml and dl) ... so thats why I whant to make it auto, to keep and test it with real data (before spending more money) ... Could there be some way to make it get the optimal parameters while trading ? don't mind how much cpu is using (for that we have servers) . what does python use to approach the best parameters? I suppose it can be reproduceble in mql5, it's just an output that we would use as input ... and ... lets have a nice day out :)
I've found this ... to make a library for caffe ... How to install Caffe in windows in 5 min - YouTube I hope I can make it and this gives a new scenary
- 2016.06.07
- www.youtube.com
thanks for all and for all your responses.
I didn't thought of that (last part), you are right ... but ... Ive used a dl and didn't give good results, so ...
are you getting good results? if so, could you please help me build a better dl? I used this one ( Programming a Deep Neural Network from Scratch using MQL Language - Neural Networks - 26 January 2019 - Traders' Blogs (mql5.com) ) and hopping to get better results, modified it by adding more hidden layers but results where worst ... could you give me more hits of what to do to make it more accurate? or whats your opinion of that neural netw?
Also ... from my clases of the uni .. I learned that predictive models are good for 3 time periods (more give much less accuracy) .... I used 1 time period with ml, to be as accurate as possible, but still ... didn't reach good results ... If your bot is looking for the hh or ll over the future in a big sample ... it might not be so good (I learned this with climate forecast).... How are you taking into account this?
I know ... it should give good results ... but in practice it doesn't (at least that has been my case with ml and dl) ... so thats why I whant to make it auto, to keep and test it with real data (before spending more money) ... Could there be some way to make it get the optimal parameters while trading ? don't mind how much cpu is using (for that we have servers) . what does python use to approach the best parameters? I suppose it can be reproduceble in mql5, it's just an output that we would use as input ... and ... lets have a nice day out :)
It doesn't have to be one solution in one place .
You can export the data in mt5 (its free for now) and vet it :
- Choose the region of data that has no gaps .
- Then you'd need to download news events from the calendar . Still free for now.
- Then you "draw" regions around news and you cut them off , but they exist in the data .
- Your system just knows that if it hits an "off" region it closes all open trades or it does not produce signals
- For the "python" part you are interested in the "does not produce signals"
Then you have your database
But you have a problem , how will you evaluate what the network produces ? Meaning , if you feed it let's say 20 ohlc bars as features for one sample then what is the correct output ?
You have to decide that , the simplest being guessing the direction of the bar after the sequence.
"The python" could find the optimal parameters (for the indicators) with a genetic algorithm library . This is peanut butter with coke for genetic algorithms . But mt5 has a decent built in genetic algorithm too in the tester . For python i don't know.
If you wanted to find the "indicator parameters" you could go the "matrix" indicator route but you'd have to add another dimension.
Let's say you could collect 1000 RSIs and 1000 moving average to price deltas and feed these in as the sample features . The strongest weights could point to the best parameters but you'd most likely get a cocktail of indicators which would be very inefficient in mt5 testing (would take too long).
edit :
Also mind your planning of the approach , don't try to find everything before you are certain its what you think it is . A change in one step in this field may mean starting over again .
Your first step is to choose what type of strategy you want to develop.
Then pick a platform to do the networks in , if you want to learn how to do it on your own its also a good idea because then you will know what is going on with each library because you know how it is supposed to operate internally .
Then build your network and try to hit the goal . How you'll use it is a question for later .
That is your first checkpoint , get it to converge .
It doesn't have to be one solution in one place .
You can export the data in mt5 (its free for now) and vet it :
- Choose the region of data that has no gaps .
- Then you'd need to download news events from the calendar . Still free for now.
- Then you "draw" regions around news and you cut them off , but they exist in the data .
- Your system just knows that if it hits an "off" region it closes all open trades or it does not produce signals
- For the "python" part you are interested in the "does not produce signals"
Then you have your database
But you have a problem , how will you evaluate what the network produces ? Meaning , if you feed it let's say 20 ohlc bars as features for one sample then what is the correct output ?
You have to decide that , the simplest being guessing the direction of the bar after the sequence.
"The python" could find the optimal parameters (for the indicators) with a genetic algorithm library . This is peanut butter with coke for genetic algorithms . But mt5 has a decent built in genetic algorithm too in the tester . For python i don't know.
If you wanted to find the "indicator parameters" you could go the "matrix" indicator route but you'd have to add another dimension.
Let's say you could collect 1000 RSIs and 1000 moving average to price deltas and feed these in as the sample features . The strongest weights could point to the best parameters but you'd most likely get a cocktail of indicators which would be very inefficient in mt5 testing (would take too long).
edit :
Also mind your planning of the approach , don't try to find everything before you are certain its what you think it is . A change in one step in this field may mean starting over again .
Your first step is to choose what type of strategy you want to develop.
Then pick a platform to do the networks in , if you want to learn how to do it on your own its also a good idea because then you will know what is going on with each library because you know how it is supposed to operate internally .
Then build your network and try to hit the goal . How you'll use it is a question for later .
That is your first checkpoint , get it to converge .
Thanks, your are enlightened.
One of the reasons I whant to learn mql5 is to make a good algobot to not spend time with the laptop and spend my time doing things I prefer. This is why I'm traying to make this deep learning bot in mql5.
For what you say, I see two problems.
The first one is the news, need to tell to not make signal while 5 min before and after the news, I think this can be made possible, but I might be mistaken. How can I extract the data and take out the time lapses? How do you do that manually? can't it be automatized?
and the second problem is to converge. Wouldn't it be possible to make a dll to connect with for example caffe or other installed in the computer, and let it work, and when its done feed the ea? If this is possible, for hard it is, I think its worth while. What do you think of this? what problems do you see here?
ps: are you using two hidden layers? From here I've read this:
- A network with three layers (numLayers=3: one input, one hidden and one output) is enough for a vast majority of cases. According to the Cybenko Theorem (1989), a network with one hidden layer is capable of approximating any continuous, multivariate function to any desired degree of accuracy; a network with two hidden layers is capable of approximating any discontinuous, multivariate function:
ps2: what is the best fit activation function? & sorry, you where right, sigmoid is only between 0 & 1.
pd3: I think caffe is for images ... here is pytorch for c++
qn: Linear layers? do you use normalization layers?
qnn: you said hours ... how many epochs?
qnnn: Do you know why theese results are so despair? what could I change to make them better 
- www.mql5.com
Not 5 minutes , not about avoiding trading around news when you are live . Its about not learning from spikes that are irrelevant to what the network can see.
You are giving your network ohlc , the Non Farm Payrolls spike in price ,for example, has nothing to do with the ohlc that came before it , the gmt time however , or , the minute inside a session could be relevant . But you must be careful with features that describe the sample space rather than the sample as the network can memorize things.
That is step 10 . You are on step 1 . Design it ,feed data the manual way at first, get it to converge , start training/testing and when you are happy with the signal start asking these questions on how to get it to mt5 , or mt4.You don't need to panic , you are a coder you can do anything .
Depends on what you want to do . Even an indicator can be an activation function in theory if you can differentiate it. I like reLu in the hidden layers as its like programming with the below zero cutoff and sigmoid or tanh on the output layer.
Sigmoid is not problematic , you have to decide how to turn it back to what you want , or you can use more nodes too.
PyTorch is nice , if you have an Nvidia GPU too it will save you time , but learn OpenCL (not now) its fun. (and short AMD #nfa)
i have not used it yet in PyTorch . The data goes in normalized from the mt5 export , both features and outcomes.
Hours to export data not train the model .
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
hi
I don't know much about mql5, but need help on how to approach this.
I whant to make a deep learning EA, that has;
1º The first inputs ohlc
2º Theese inputs are used as inputs in all the oscilators, indicatos
3º The outputs of 2º) are used as the input layer of the deep learnig
... and so on as explained here (with probably two hidden layers or more) Programming a Deep Neural Network from Scratch using MQL Language - Neural Networks - 26 January 2019 - Traders' Blogs (mql5.com)
So, here is my q:
How can I do from step 1 to step 2?
Is step 2º an EA or a header?
having trouble to do steps 1 to 2, so any help here will be appreciated
cheers!