How to import historical data into csv to symbol custom using CustomRatesUpdate?

[Fabio Fontes]

* Sorry English is not my native language

I want to create multiple custom symbols and import historical data from a csv

I have managed to do this however the .hcc file is large and the writing speed is slow.

I do not know so much programming in my mind he is reading line by line and then writing and this should be one of the reasons for the slowness

Can someone help me

This was my attempt:

   int file = FileOpen(symbol+"_1 Min_Bid_2008.01.01_2018.08.10.csv",FILE_READ|FILE_CSV|FILE_ANSI ,';');
   
   while (!FileIsEnding(file)) {
   
      datetime time = StringToTime(FileReadString(file));       
      double open = FileReadNumber(file);
      double high = FileReadNumber(file);
      double low = FileReadNumber(file);
      double close = FileReadNumber(file);
      long volume = StringToInteger(FileReadString(file));
      
      MqlRates rates[1];
      rates[0].time = time;
      rates[0].open = open;
      rates[0].high = high;
      rates[0].low = low;
      rates[0].close = close;
      rates[0].tick_volume = volume;
      rates[0].spread = 0;    
      //rates[0].real_volume = volume;  
      Print("Time: ", rates[0].time," Open: ", rates[0].open," High: ", rates[0].high," Low: ", 
        rates[0].low, " Close: ", rates[0].close, " Volume: ", rates[0].tick_volume);
      CustomRatesUpdate(symbol+suffixOut,rates);
   
   }
   
   FileClose(file); 
   Print("Finished creating the symbol " + symbol+suffixOut);

Grateful already

[fxsaber]

https://www.mql5.com/en/docs/customsymbols/customratesreplace

[Fabio Fontes]

fxsaber:
https://www.mql5.com/en/docs/customsymbols/customratesreplace

Thanks for the answer

I have tested more still very slow

Until then I know the above code does:

1. Open the csv file

2. Read a line

3. Write a line

4. Loop

It would be faster to read the entire csv file, saving all in one array, and then use CustomRatesReplace?

I'm lost because I'm an inexperienced programmer.

[fxsaber]

Fabio:

It would be faster to read the entire csv file, saving all in one array, and then use CustomRatesReplace?

Yes.

[Fabio Fontes]

Reading the entire csv file is not possible because of the memory limits and save all in the array also had problems with memory.

Some days breaking the head because I'm an inexperienced programmer and for some reason the metaeditor stopped and made me lose what I had already programmed (always backup!!!).

This works for me:

string delimitador =",";
string CustomSymbol = "XAUUSD";
string SuffixInp = "_1 Min_Bid_2008.01.01_2018.08.10.csv";
string SuffixSym = "_my";
int LimitMemory = 144000;

void OnStart() {
   customHistory();     
}

// file csv 6 columns (Time (UTC),Open,High,Low,Close,Volume)

void customHistory() {
   MqlRates rates[];
   int limit = LimitMemory;
   ArrayResize(rates,limit+1);
   int openFile = FileOpen(CustomSymbol+SuffixInp,FILE_READ|FILE_CSV|FILE_ANSI ,delimitador);
   FileSeek(openFile, 40, SEEK_SET);
   for(int i = 0; !FileIsEnding(openFile);i++) {
      if(i >= limit-1) {
         CustomRatesReplace(CustomSymbol+SuffixSym,rates[0].time,rates[limit-2].time, rates);
         Print("Import from: " +(string)rates[0].time + " to " + (string)rates[limit-2].time);
         ZeroMemory(rates); 
         ZeroMemory(i);
      }
      rates[i].time = (datetime)FileReadString(openFile);
      rates[i].open = (double)FileReadString(openFile);
      rates[i].high = (double)FileReadString(openFile);
      rates[i].low = (double)FileReadString(openFile);
      rates[i].close = (double)FileReadString(openFile);
      rates[i].tick_volume = (long)FileReadString(openFile);
      rates[i].spread = 0;
      rates[i].real_volume = 0;
      if(FileIsEnding(openFile)) break;
   }
   FileClose(openFile);
   CustomRatesReplace(CustomSymbol+SuffixSym, rates[0].time, TimeLocal(), rates);
   Print("Import from: " + (string)rates[0].time + " to " + "end");
}

It is probably possible to program to use the maximum memory automatically, if someone wants to collaborate (I do not know if it's worth the trouble too).

[fxsaber]

Show the contents of the CSV-file.

[Fabio Fontes]

fxsaber:
Show the contents of the CSV-file.

I do not know if you say to send csv file

If yes here is the sample, if not, sorry my English is not good.

Time (UTC),Open,High,Low,Close,Volume 

[fxsaber]

bool StringToRates( const string &Str, MqlRates &Rates, const ushort Delimeter = ',' )
{
  string StrArray[];
  
  const bool Res = (StringSplit(Str, Delimeter, StrArray) >= 6);
  
  if (Res)
  {
    Rates.time = (datetime)StrArray[0];
    Rates.open = (double)StrArray[1];
    Rates.high = (double)StrArray[2];
    Rates.low = (double)StrArray[3];
    Rates.close = (double)StrArray[4];
    Rates.tick_volume = (long)StrArray[5];
    Rates.spread = 0;
    Rates.real_volume = 0;
  }
  
  return(Res);
}

int FileToString( const string FileName, string &Str[] )
{
  uchar Bytes[];
  
  return(StringSplit(CharArrayToString(Bytes, 0, (int)FileLoad(FileName, Bytes)), '\n', Str));
}

int FileToRates( const string FileName, MqlRates &Rates[], const int Shift = 1 )
{
  string Str[];  
  const int Size = ArrayResize(Rates, FileToString(FileName, Str) - Shift);
  
  for (int i = 0; i < Size; i++)
    StringToRates(Str[i + Shift], Rates[i]);
  
  return(Size);
}

void OnStart()
{
  MqlRates Rates[];
  
  FileToRates("XAUUSD_1 Min_Bid_2008.01.01_2018.08.10.csv", Rates);
  
  ArrayPrint(Rates);
}

Result

                 [time]    [open]    [high]     [low]   [close] [tick_volume] [spread] [real_volume]
[0] 2008.01.01 00:00:00 833.02200 833.12000 833.02200 833.09700             0        0             0
[1] 2008.01.01 00:01:00 833.09700 833.09700 833.09700 833.09700             0        0             0
[2] 2008.01.01 00:02:00 833.04500 833.09200 833.04300 833.05600             0        0             0
[3] 2008.01.01 00:03:00 833.08200 833.16300 833.00100 833.16300             0        0             0
[4] 2008.01.01 00:04:00 833.01300 833.06000 832.97000 833.01300             0        0             0
[5] 2008.01.01 00:05:00 832.97200 833.10500 832.96000 832.98800             0        0             0
[6] 2008.01.01 00:06:00 832.98800 832.98800 832.98800 832.98800             0        0             0
[7] 2008.01.01 00:07:00 832.98800 832.98800 832.98800 832.98800             0        0             0
[8] 2008.01.01 00:08:00 832.98800 832.98800 832.98800 832.98800             0        0             0

[fxsaber]

Fabio:

It is probably possible to program to use the maximum memory automatically, if someone wants to collaborate (I do not know if it's worth the trouble too).

// Sets the maximum size of an array.
template <typename T>
int ArrayResize( T &Array[] )
{
  int MinSize = ArraySize(Array);
  int MaxSize = INT_MAX;
  int AvgSize;
  
  while ((MinSize < MaxSize - 1) && !IsStopped())
  {
    AvgSize = (int)((MinSize + (long)MaxSize) >> 1);        
    
//    ArrayFree(Array);
    
    if (ArrayResize(Array, (int)AvgSize) == AvgSize)
      MinSize = AvgSize;
    else
      MaxSize = AvgSize;
  }
  
  return(ArrayResize(Array, MinSize));
}

void OnStart()
{
  MqlRates Array[];
  
  Print(ArrayResize(Array));
}

[Fabio Fontes]

thanks fxsaber I'll look at the code you posted

[mhfhaji]

fxsaber #:

Result

Hello,thanks for your code,i want to creat custom symbol and your code help me to convert my csv data to mqlrate array,now i want to import mqlrates data for my customsymbol to plot it's chart,can you help me?