Discussion of article "Third Generation Neural Networks: Deep Networks" - page 12

New comment
 
kimkarus:

Vladimir Perervenko

Add to the article additional information on working with R Studio

  • To make everything run correctly you need to make sure that there are no other references to "localhost" in the hosts file.

Good afternoon.

I don't understand about the hosts file. Can you give more details ?

Good luck

 
kimkarus:

Vladimir Perervenko

Add to the article additional information on working with R Studio

  • To run everything correctly you need to make sure that there are no other references to "localhost" in the hosts file.
  • Be sure to install all packages and run them with the command (in the R workspace):

install.packages("R.matlab")
install.packages("deepnet")
install.packages("caret")
install.packages("h2o")
install.packages("TTR")
install.packages("rminer")
install.packages("foreach")
install.packages("doParallel")
install.packages("svSocket")
install.packages("lattice")
install.packages("ggplot2")
install.packages("statmod")
install.packages("kknn")
install.packages("iterators")
install.packages("parallel")

library("R.matlab")
library("deepnet")
library("caret")
library("h2o")
library("TTR")
library("rminer")
library("foreach")
library("doParallel")
library("svSocket")
library("lattice")
library("ggplot2")
library("statmod")
library("kknn")
library("iterators")
library("parallel")

I use another form of record to check installed packages:

packets <- Hmisc::Cs(R.matlab, deepnet, caret, h2o, TTR, rminer, foreach, doParallel, 
                        svSocket, lattice, ggplot2, statmod, kknn, iterators, parallel)
for(i in 1:length(packets)) {
  if (! (packets[i] %in% rownames(installed.packages()))) { 
    install.packages(packets[i]) }
}

You should load libraries in the description of functions that use them. Although you can do it this way when initialising the Expert Advisor.

So why do you need to run the Expert Advisor in the tester?

Good luck

 

While debugging R scripts for a long time, identified a bug that is hard to catch if the incoming data has NA. It simply will not trigger the signal. In the "e_SAE_init.r" file, it is recommended to add a NA cleanup term to the Test(dt,x) function before new.data <- predict(prepr, tail(x, 50)): x <- na.omit(x);

This seems like a "crutch", but I haven't thought of anything better yet.

Without it, a hidden error will occur:

Error in if (sqrt(denom) > .Machine$double.eps) x/sqrt(denom) else x * : missing value where TRUE/FALSE needed

 
kimkarus:

While debugging R scripts for a long time, identified a bug that is hard to catch if the incoming data has NA. It simply will not trigger the signal. In the "e_SAE_init.r" file, it is recommended to add a NA cleanup term to the Test(dt,x) function before new.data <- predict(prepr, tail(x, 50)): x <- na.omit(x);

This seems like a "crutch", but I haven't thought of anything better yet.

Without it, a hidden error will occur:

Error in if (sqrt(denom) > .Machine$double.eps) x/sqrt(denom) else x * : missing value where TRUE/FALSE needed

kimkarus:

While debugging R scripts for a long time, I found a bug that is hard to catch if the incoming data has NA. It simply will not trigger the signal. In the "e_SAE_init.r" file, it is recommended to add a NA cleanup term to the Test(dt,x) function before new.data <- predict(prepr, tail(x, 50)): x <- na.omit(x);

This seems like a "crutch", but I haven't thought of anything better yet.

Without it, a hidden error will occur:

Error in if (sqrt(denom) > .Machine$double.eps) x/sqrt(denom) else x * : missing value where TRUE/FALSE needed

This statement is incorrect.

In the Test(dt, x) function, x is the input data calculated by the In() function. Let's look at it in the "i_SAE_fun.r" script

In <- function(p = 16){
        require(TTR)
        adx <- ADX(price, n = p)
        ar <- aroon(price[ ,c('High', 'Low')], n = p)[ ,'oscillator']
        cci <- CCI(price[ ,2:4], n = p)
        chv <- chaikinVolatility(price[ ,2:4], n = p)
        cmo <- CMO(price[ ,'Med'], n = p)
        macd <- MACD(price[ ,'Med'], 12, 26, 9)[ ,'macd']
        osma <- macd - MACD(price[ ,'Med'],12, 26, 9)[ ,'signal']
        rsi <- RSI(price[ ,'Med'], n = p)
        stoh <- stoch(price[ ,2:4], 14, 3, 3)
        smi <- SMI(price[ ,2:4],n = p, nFast = 2, nSlow = 25, nSig = 9)
        vol <- volatility(price[ ,1:4], n = p, calc="yang.zhang", N=96)
        In <- cbind(adx, ar, cci, chv, cmo, macd, osma, rsi, stoh, smi, vol)
        return(In)
}

This is a number of indicators. Let's calculate them on the price[] history with the length of 2000 bars.

> x <- In()Loading required package: TTR Let's see what data we got
> summary(x)
      DIp             DIn                DX          
 Min.   :20.24   Min.   :  9.546   Min.   : 0.04605  
 1st Qu.:43.61   1st Qu.: 31.022   1st Qu.:10.28793  
 Median :50.46   Median : 39.297   Median :19.31075  
 Mean   :49.01   Mean   : 41.833   Mean   :21.82957  
 3rd Qu.:55.78   3rd Qu.: 49.441   3rd Qu.:31.18539  
 Max.   :74.50   Max.   :116.050   Max.   :71.84495  
 NA's   :16      NA's   :16        NA's   :16        
      ADX               ar                cci           
 Min.   : 7.038   Min.   :-100.000   Min.   :-364.2786  
 1st Qu.:15.559   1st Qu.: -56.250   1st Qu.: -86.9604  
 Median :20.450   Median : -12.500   Median :  -6.4301  
 Mean   :21.878   Mean   :  -1.147   Mean   :  -0.3145  
 3rd Qu.:27.330   3rd Qu.:  56.250   3rd Qu.:  86.1551  
 Max.   :47.191   Max.   : 100.000   Max.   : 331.4449  
 NA's   :31       NA's   :16         NA's   :15         
      chv                cmo                macd          
 Min.   :-0.63538   Min.   :-88.8628   Min.   :-0.219469  
 1st Qu.:-0.28769   1st Qu.:-29.6125   1st Qu.:-0.030508  
 Median :-0.01415   Median : -0.8713   Median : 0.001847  
 Mean   : 0.12162   Mean   : -1.6646   Mean   : 0.009282  
 3rd Qu.: 0.35276   3rd Qu.: 27.7824   3rd Qu.: 0.037112  
 Max.   : 7.37405   Max.   : 86.9767   Max.   : 0.703234  
 NA's   :31         NA's   :16         NA's   :25         
      osma               rsi             fastK       
 Min.   :-0.10903   Min.   : 8.614   Min.   :0.0000  
 1st Qu.:-0.01063   1st Qu.:41.108   1st Qu.:0.2246  
 Median :-0.00016   Median :50.547   Median :0.4553  
 Mean   :-0.00006   Mean   :49.953   Mean   :0.4793  
 3rd Qu.: 0.01021   3rd Qu.:58.706   3rd Qu.:0.7425  
 Max.   : 0.28849   Max.   :84.854   Max.   :1.0000  
 NA's   :33         NA's   :16       NA's   :13      
     fastD             slowD              SMI         
 Min.   :0.01645   Min.   :0.03279   Min.   :-77.616  
 1st Qu.:0.23056   1st Qu.:0.23618   1st Qu.:-27.662  
 Median :0.45989   Median :0.46420   Median : -2.998  
 Mean   :0.47916   Mean   :0.47922   Mean   : -3.468  
 3rd Qu.:0.72776   3rd Qu.:0.71850   3rd Qu.: 21.330  
 Max.   :0.98610   Max.   :0.96254   Max.   : 73.964  
 NA's   :15        NA's   :17        NA's   :25       
     signal             vol          
 Min.   :-74.526   Min.   :0.001235  
 1st Qu.:-24.781   1st Qu.:0.003168  
 Median : -2.446   Median :0.004686  
 Mean   : -3.358   Mean   :0.005457  
 3rd Qu.: 19.029   3rd Qu.:0.006484  
 Max.   : 71.664   Max.   :0.047742  
 NA's   :33        NA's   :16        
 All variables have NA. But they are located at the beginning!!! As is normal for all indicators. 
Therefore, when we write in the script

 new.data <- predict(prepr, tail(x, 500));

We cut off the undefined data. Condition : nrow(x) > 500 + max(NA). I.e. at least in our case 533. To be safe, put nrow(x) = 600-700.

I don't see how you got an uncertain NA in x.

Good luck

 

HI Vladimir, 

 

Here is from Brazil!!!

 

I read your instructions about the Neural Network using R but I have a dumb question (sorry, I am newbie in this!)

 

In the tutorial that you wrote ( https://www.mql5.com/en/articles/1103#ch_3), in the "Section 3.3.1 - Source Data" you described a function, called pr.OHLC that I understood very well. 

 

Bu  you show some results that it was not clear to me which are the parameters necessary to the results below

 

> head(price)
        Open    High     Low   Close      Med     CO
[1,] 1.33848 1.33851 1.33824 1.33844 1.338375 -4e-05
[2,] 1.33843 1.33868 1.33842 1.33851 1.338550  8e-05
[3,] 1.33849 1.33862 1.33846 1.33859 1.338540  1e-04
[4,] 1.33858 1.33861 1.33856 1.33859 1.338585  1e-05
[5,] 1.33862 1.33868 1.33855 1.33855 1.338615 -7e-05 

Could you, please, help me with this?

 

Best,

 

Fábio 

Third Generation Neural Networks: Deep Networks
Third Generation Neural Networks: Deep Networks
  • 2015.02.05
  • Vladimir Perervenko
  • www.mql5.com
This article is dedicated to a new and perspective direction in machine learning - deep learning or, to be precise, deep neural networks. This is a brief review of second generation neural networks, the architecture of their connections and main types, methods and rules of learning and their main disadvantages followed by the history of the third generation neural network development, their main types, peculiarities and training methods. Conducted are practical experiments on building and training a deep neural network initiated by the weights of a stacked autoencoder with real data. All the stages from selecting input data to metric derivation are discussed in detail. The last part of the article contains a software implementation of a deep neural network in an Expert Advisor with a built-in indicator based on MQL4/R.
 
fabiocarvalho:

HI Vladimir, 

 

Here is from Brazil!!!

 

I read your instructions about the Neural Network using R but I have a dumb question (sorry, I am newbie in this!)

 

In the tutorial that you wrote ( https://www.mql5.com/en/articles/1103#ch_3), in the "Section 3.3.1 - Source Data" you described a function, called pr.OHLC that I understood very well. 

 

Bu  you show some results that it was not clear to me which are the parameters necessary to the results below

 

Could you, please, help me with this?

 

Best,

 

Fábio 

Hi Fabio,

What's not clear?

pr.OHLC <- function (o, h, l, c) 
{
  #Unite quote vectors into a matrix having previously expanded them
  #Indexing of time series of vectors in R starts with 1. 
  #Direction of indexing is from old to new ones.   
  price <- cbind(Open = rev(o), High = rev(h), Low = rev(l), Close = rev(c))
  Med <- (price[, 2] + price[, 3])/2 #We calculate average price (HIgh + Low)/2
  CO <- price[, 4] - price[, 1] # We calculate body candles (Close - Open)
  #add Med and CO to the matrix
  price <- cbind(price, Med, CO)#We are putting it all in a matrix
}
 

Hi Vladimir,

 

Is there a possibility to have the files for MT5 ?

Regards

 

Fabio lima 

 
fabioflimaster:

Hi Vladimir,

 

Is there a possibility to have the files for MT5 ?

Regards

 

Fabio lima 

Hi Fabio,

I'm sorry.

I do not write on MKL5.

Best regards

Vladimir


 

One question . I am not clear on the order of the price vector .

You do a reversal here : price <- cbind(Open = rev(o), High = rev(h), Low = rev(l), Close = rev(c))

What is the original order of o,h,l,c ?  

 
jake89:

One question . I am not clear on the order of the price vector .

You do a reversal here : price <- cbind(Open = rev(o), High = rev(h), Low = rev(l), Close = rev(c))

What is the original order of o,h,l,c ?  

Hi,

The MT4 numbering bars from the most recent to oldest. The R on the contrary, from the old to the new, new bar last.

1...567891011121314151617
New comment