Developing a Replay System (Part 63): Playing the service (IV)

Introduction

In the previous article, Developing a Replay System (Part 62): Playing the Service (III), we addressed how ticks would be treated as if they were real. This excess is not our primary concern; however, it can complicate the application's ability to achieve proper timing, ensuring that the one-minute bar is fully constructed within the designated one-minute window.

Despite the progress made in the previous article, I mentioned at the end that certain errors had emerged due to the implementation of simulation within real data. I want to emphasize once again that these errors do not exist in a pure simulation. However, when the simulation is combined with real data, such errors inevitably occur. This is not necessarily a problem, as it allows us to conduct tests and determine whether the system being developed is viable for maintenance and improvement. Often, we develop something simply to test its feasibility. If the underlying concept proves unworkable, we can discard it, minimizing the time spent on fixes and other adjustments.

In that same article, I identified these errors and their causes. However, given the extensive information already presented (information that you, dear reader, must thoroughly understand before any further progress is introduced) I concluded the article at a certain point. Here, however, we will address and correct these errors, not due to code changes, but because we are implementing something that would not be necessary if the system's timing mechanism were not overloaded. This overload is not yet apparent because we are still in the early stages of implementing everything that will be required.

Since these issues are interrelated, we could start by addressing any of them. However, being meticulous about certain details, I will begin by resolving the issue of the minimum number of ticks that need to be simulated when using real data.

What is the Minimum Number of Ticks That Must Be Generated?

Answering this question is not as straightforward as it may seem. This is because, when modifying this application that enables replay or simulation, one must always remember a key point. In replay mode, this issue can be addressed by forcing the simulation class to use a minimum number of ticks. Remember, when using the Replay feature, we are dealing with real data. However, when simulating ticks to control timing within a one-minute window, the approach is different. This applies when the simulation is based on obtained rate data. The problem arises because, at present, users cannot adjust the maximum number of ticks that can be utilized.

I plan to introduce this option in the configuration file, allowing users to control this value. The reason is simple: if a user's workstation can generate more ticks than the application allows, they can adjust the value in the configuration file to achieve a replay experience that more accurately reflects reality. Conversely, if a user's workstation cannot handle a specific number of ticks, they can reduce the quantity to a lower value, ensuring smoother and less stressful operation.

Another factor complicating this question is that it is not simply a matter of specifying a number. This is because it is possible to simulate data externally, save it, and use it as if it were a real database. This is the worst-case scenario for the replay system. However, for now, we will not address this aspect. Our immediate focus is on correcting misconfigurations made by the application user.

Returning to the main point: What is the minimum number of ticks required? The answer depends. To understand this, consider the following. If the opening, closing, high, and low prices are all the same, a single tick is sufficient. However, this is the simplest case. To account for other scenarios, we must define some conditions. The first is that opening and closing must necessarily be performed within the maximum and minimum. Thus, we get the following scenarios:

• If the opening price equals one of the limits and the closing price equals the other, two ticks are required.
• If either the opening or closing price equals one of the limits while the opposite limit is beyond these values, three ticks are necessary.
• If all four OHLC (Open, High, Low, Close) values are different, or if the opening and closing prices are identical but do not coincide with the bar's high or low, four ticks are needed.

This provides the foundation for determining the minimum number of ticks required for simulation. The simulation has been running smoothly so far. However, I want to allow users to configure a limit, ensuring that their workstation can run the replay/simulator application optimally.

Perhaps this explanation seems complex in textual form, but it becomes much clearer in code. Therefore, instead of reproducing the entire C_Simulation.mqh file, I will include only the relevant code fragment below. This is the part that was modified to resolve the existing issue.

128. //+------------------------------------------------------------------+
129. inline int Simulation(const MqlRates &rate, MqlTick &tick[], const int MaxTickVolume)
130.          {
131.             int    i0, i1, i2, dm = 0;
132.             bool   b0;
133.             
134.             m_Marks.iMax   = (MaxTickVolume <= 0 ? 1 : (MaxTickVolume >= def_MaxTicksVolume ? def_MaxTicksVolume : MaxTickVolume));
135.             dm = (dm == 0 ? ((rate.open == rate.high == rate.low == rate.close) ? 1 : dm) : dm);
136.             dm = (dm == 0 ? (((rate.open == rate.high) && (rate.low == rate.close)) || ((rate.open == rate.low) && (rate.close == rate.high)) ? 2 : dm) : dm);
137.             if ((dm == 0 ? ((rate.open == rate.close == rate.high) || (rate.open == rate.close == rate.low) ? 3 : 4) : dm) == 0) return -1;            
138.             m_Marks.iMax   = (MaxTickVolume <= dm ? dm : MaxTickVolume);
139.             m_Marks.iMax   = (((int)rate.tick_volume > m_Marks.iMax) || ((int)rate.tick_volume < dm) ? m_Marks.iMax : (int)rate.tick_volume - 1);
140.             m_Marks.bHigh    = (rate.open == rate.high) || (rate.close == rate.high);
141.             m_Marks.bLow    = (rate.open == rate.low) || (rate.close == rate.low);
142.             Simulation_Time(rate, tick);
143.             MountPrice(0, rate.open, rate.spread, tick);
144.             if (m_Marks.iMax > 10)
145.             {
146.                i0 = (int)(MathMin(m_Marks.iMax / 3.0, m_Marks.iMax * 0.2));
147.                i1 = m_Marks.iMax - i0;
148.                i2 = (int)(((rate.high - rate.low) / m_TickSize) / i0);
149.                i2 = (i2 == 0 ? 1 : i2);
150.                b0 = (m_Marks.iMax >= 1000 ? ((rand() & 1) == 1) : (rate.high - rate.open) < (rate.open - rate.low));
151.                i0 = RandomWalk(1, i0, rate.open, (b0 ? rate.high : rate.low), rate.high, rate.low, rate.spread, tick, 0, i2);
152.                RandomWalk(i0, i1, (m_IsPriceBID ? tick[i0].bid : tick[i0].last), (b0 ? rate.low : rate.high), rate.high, rate.low, rate.spread, tick, 1, i2);
153.                RandomWalk(i1, m_Marks.iMax, (m_IsPriceBID ? tick[i1].bid : tick[i1].last), rate.close, rate.high, rate.low, rate.spread, tick, 2, i2);
154.                m_Marks.bHigh = m_Marks.bLow = true;
155. 
156.             }else Random_Price(rate, tick);
157.             if (!m_IsPriceBID) DistributeVolumeReal(rate, tick);
158.             if (!m_Marks.bLow) MountPrice(Unique(rate.high, tick), rate.low, rate.spread, tick);
159.             if (!m_Marks.bHigh) MountPrice(Unique(rate.low, tick), rate.high, rate.spread, tick);
160.             MountPrice(m_Marks.iMax, rate.close, rate.spread, tick);
161.             CorretTime(tick);
162. 
163.             return m_Marks.iMax;
164.          }
165. //+------------------------------------------------------------------+

Fragment of the file C_Simulation.mqh

The line numbering in this fragment indicates exactly where you need to modify the code found in full in the previous article. Notice that on line 131, I added a new variable and initialized it immediately upon declaration. Also, note that the original line 134 must be removed and replaced with three new lines. These three lines implement the points I recently mentioned, ensuring that the minimum required number of ticks is created.

However, there is a crucial detail you must pay close attention to. On line 138, during the ternary operator test, the value of MaxTickVolume is compared with the recently adjusted value to determine the minimum number of ticks. If MaxTickVolume is smaller than this adjusted value, the adjusted value will be used, regardless of any other data. Furthermore, on line 139, we verify this same condition again. If the value in rate.tick_volume is also lower than the adjusted value, the adjusted value will take precedence.

Now, remember to test the return value of this simulation function, especially if you intend to use it for other purposes. This is important because if an error occurs while adjusting the minimum number of ticks, the function will return immediately on line 137. Therefore, do not attempt to use the values in the returned array without first verifying the function's return, as the array values could be invalid.

A minor modification has been made here compared to the code presented in the previous article. However, toward the end of this article, I will revisit this same fragment to ensure that the explanation of this change makes complete sense.

With that, we have resolved our first issue. Now, let's move on to the next topic and address the second problem.

Fixing the Tick Adjustment Issue

This second issue requires more effort to resolve. However, the fact that it is more labor-intensive does not necessarily mean it is more difficult - just that it will take a bit more work. So here's what we'll do: let's review the code fragment from the previous article responsible for handling movements and adjusting ticks to ensure they can be used for chart plotting. The fragment is provided below.

01. //+------------------------------------------------------------------+
02.       datetime LoadTicks(const string szFileNameCSV, const bool ToReplay, const int MaxTickVolume)
03.          {
04.             int      MemNRates,
05.                      MemNTicks,
06.                      nDigits,
07.                      nShift;
08.             datetime dtRet = TimeCurrent();
09.             MqlRates RatesLocal[],
10.                      rate;
11.             MqlTick  TicksLocal[];
12.             bool     bNew;
13.             
14.             MemNRates = (m_Ticks.nRate < 0 ? 0 : m_Ticks.nRate);
15.             nShift = MemNTicks = m_Ticks.nTicks;
16.             if (!Open(szFileNameCSV)) return 0;
17.             if (!ReadAllsTicks()) return 0;         
18.             rate.time = 0;
19.             nDigits = SetSymbolInfos(); 
20.             ArrayResize(TicksLocal, def_MaxSizeArray);
21.             m_Ticks.bTickReal = true;
22.             for (int c0 = MemNTicks, c1, MemShift = nShift; c0 < m_Ticks.nTicks; c0++, nShift++)
23.             {
24.                if (!BuildBar1Min(c0, rate, bNew)) continue;
25.                if (bNew)
26.                {
27.                   if ((m_Ticks.nRate >= 0) && (ToReplay)) if (m_Ticks.Rate[m_Ticks.nRate].tick_volume > MaxTickVolume)
28.                   {
29.                      nShift = MemShift;
30.                      C_Simulation *pSimulator = new C_Simulation(nDigits);
31.                      if ((c1 = (*pSimulator).Simulation(m_Ticks.Rate[m_Ticks.nRate], TicksLocal, MaxTickVolume)) < 0) return 0;
32.                      ArrayCopy(m_Ticks.Info, TicksLocal, nShift, 0, c1);
33.                      nShift += c1;
34.                      delete pSimulator;
35.                   }
36.                   MemShift = nShift;
37.                   ArrayResize(m_Ticks.Rate, (m_Ticks.nRate > 0 ? m_Ticks.nRate + 2 : def_BarsDiary), def_BarsDiary);
38.                };
39.                m_Ticks.Rate[(m_Ticks.nRate += (bNew ? 1 : 0))] = rate;
40.             }
41.             ArrayFree(TicksLocal);
42.             if (!ToReplay)
43.             {
44.                ArrayResize(RatesLocal, (m_Ticks.nRate - MemNRates));
45.                ArrayCopy(RatesLocal, m_Ticks.Rate, 0, 0);
46.                CustomRatesUpdate(def_SymbolReplay, RatesLocal, (m_Ticks.nRate - MemNRates));
47.                dtRet = m_Ticks.Rate[m_Ticks.nRate].time;
48.                m_Ticks.nRate = (MemNRates == 0 ? -1 : MemNRates);
49.                m_Ticks.nTicks = MemNTicks;
50.                ArrayFree(RatesLocal);
51.             }else m_Ticks.nTicks = nShift;
52.                            
53.             return dtRet;
54.          };
55. //+------------------------------------------------------------------+

Fragment of the C_FileTicks.mqh file

Please note that this code contains errors that we will need to fix in this article. Now pay close attention to the next fragment and compare it with the previous one.

01. //+------------------------------------------------------------------+
02.       datetime LoadTicks(const string szFileNameCSV, const bool ToReplay, const int MaxTickVolume)
03.          {
04.             int      MemNRates,
05.                      MemNTicks,
06.                      nDigits,
07.                      nShift;
08.             datetime dtRet = TimeCurrent();
09.             MqlRates RatesLocal[],
10.                      rate;
11.             MqlTick  TicksLocal[];
12.             bool     bNew;
13.             
14.             MemNRates = (m_Ticks.nRate < 0 ? 0 : m_Ticks.nRate);
15.             nShift = MemNTicks = m_Ticks.nTicks;
16.             if (!Open(szFileNameCSV)) return 0;
17.             if (!ReadAllsTicks()) return 0;         
18.             rate.time = 0;
19.             nDigits = SetSymbolInfos(); 
20.             m_Ticks.bTickReal = true;
21.             for (int c0 = MemNTicks, c1, MemShift = nShift; c0 < m_Ticks.nTicks; c0++, nShift++)
22.             {
23.                if (nShift != c0) m_Ticks.Info[nShift] = m_Ticks.Info[c0];
24.                if (!BuildBar1Min(c0, rate, bNew)) continue;
25.                if (bNew)
26.                {
27.                   if ((m_Ticks.nRate >= 0) && (ToReplay)) if (m_Ticks.Rate[m_Ticks.nRate].tick_volume > MaxTickVolume)
28.                   {
29.                      nShift = MemShift;
30.                      ArrayResize(TicksLocal, def_MaxSizeArray);
31.                      C_Simulation *pSimulator = new C_Simulation(nDigits);
32.                      if ((c1 = (*pSimulator).Simulation(m_Ticks.Rate[m_Ticks.nRate], TicksLocal, MaxTickVolume)) > 0)
33.                         nShift += ArrayCopy(m_Ticks.Info, TicksLocal, nShift, 0, c1);
34.                      delete pSimulator;
35.                      ArrayFree(TicksLocal);
36.                      if (c1 < 0) return 0;
37.                   }
38.                   MemShift = nShift;
39.                   ArrayResize(m_Ticks.Rate, (m_Ticks.nRate > 0 ? m_Ticks.nRate + 2 : def_BarsDiary), def_BarsDiary);
40.                };
41.                m_Ticks.Rate[(m_Ticks.nRate += (bNew ? 1 : 0))] = rate;
42.             }
43.             if (!ToReplay)
44.             {
45.                ArrayResize(RatesLocal, (m_Ticks.nRate - MemNRates));
46.                ArrayCopy(RatesLocal, m_Ticks.Rate, 0, 0);
47.                CustomRatesUpdate(def_SymbolReplay, RatesLocal, (m_Ticks.nRate - MemNRates));
48.                dtRet = m_Ticks.Rate[m_Ticks.nRate].time;
49.                m_Ticks.nRate = (MemNRates == 0 ? -1 : MemNRates);
50.                m_Ticks.nTicks = MemNTicks;
51.                ArrayFree(RatesLocal);
52.             }else m_Ticks.nTicks = nShift;
53.                            
54.             return dtRet;
55.          };
56. //+------------------------------------------------------------------+

Fragment of the C_FileTicks.mqh file (final)

Do you see the differences? They're not particularly drastic, but they are there. You can notice that I made a few changes to the execution order of certain operations. The most obvious change is in the allocation and deallocation of memory for the ticks created during the simulation. Since the LoadTicks function is a primary function, meaning it's executed before the system fully initializes and begins demanding performance, we can afford to lose a bit of time during memory allocation and deallocation calls.

If you feel that such a time loss is unacceptable, feel free to adjust the execution order as needed. Nevertheless, in the corrected fragment shown below, we must not overlook the importance of calling the destructor of the simulation class in the event of a failure. When comparing the code fragments, you'll notice that in the corrected version, the function only returns in the event of a failure at line 36. But before that, on lines 34 and 35, we explicitly call the destructor and release the allocated memory. In that specific order.

If the simulation succeeds and the data is ready to be moved, we perform this step on line 33, where we also use the return value from a library function to update the new offset value.

With this, we resolve the issue that previously occurred when the simulation returned as a failure. However, there's still one more issue here that needs to be explained. If you look at the corrected fragment, you'll notice something odd. It may not make much sense at first glance, this can be seen on line 23. Why is line 23 comparing the tick counter with the offset value? What's the purpose of this check?

Truthfully, it doesn’t seem to make much sense. That much is true. However, if the simulator is executed, the offset value will differ from the tick counter. When this happens, some real ticks will end up indexed incorrectly. If you don't correct this, then when line 52 is executed, a number of real ticks may simply vanish. Not to mention the issue that arises between simulated and non-simulated bars, where some strange ticks may appear between them due to the incorrect indexing.

Now I believe you truly understand the root of the issue. So, by adding a check at line 23 to verify and reposition the real ticks, everything will execute properly, and we won' see any anomalies on the chart. It's a simple fix, but it completely resolves the problem. The final version of the code, found in the C_FileTicks.mqh file, is as follows:

001. //+------------------------------------------------------------------+
002. #property copyright "Daniel Jose"
003. //+------------------------------------------------------------------+
004. #include "C_FileBars.mqh"
005. #include "C_Simulation.mqh"
006. //+------------------------------------------------------------------+
007. #define macroRemoveSec(A)   (A - (A % 60))
008. #define def_MaxSizeArray    16777216 // 16 Mbytes
009. //+------------------------------------------------------------------+
010. class C_FileTicks
011. {
012.    protected:
013.       enum ePlotType {PRICE_EXCHANGE, PRICE_FOREX};
014.       struct stInfoTicks
015.       {
016.          MqlTick     Info[];
017.          MqlRates    Rate[];
018.          int         nTicks,
019.                      nRate;
020.          bool        bTickReal;
021.          ePlotType   ModePlot;
022.       };
023. //+------------------------------------------------------------------+
024. inline bool BuildBar1Min(const int iArg, MqlRates &rate, bool &bNew)
025.          {
026.             double    dClose = 0;
027.             
028.             switch (m_Ticks.ModePlot)
029.             {
030.                case PRICE_EXCHANGE:
031.                   if (m_Ticks.Info[iArg].last == 0.0) return false;
032.                   dClose = m_Ticks.Info[iArg].last;
033.                   break;
034.                case PRICE_FOREX:
035.                   dClose = (m_Ticks.Info[iArg].bid > 0.0 ? m_Ticks.Info[iArg].bid : dClose);
036.                   if ((dClose == 0.0) || (m_Ticks.Info[iArg].bid == 0.0)) return false;
037.                   break;
038.             }
039.             if (bNew = (rate.time != macroRemoveSec(m_Ticks.Info[iArg].time)))
040.             {
041.                rate.time = macroRemoveSec(m_Ticks.Info[iArg].time);
042.                rate.real_volume = 0;
043.                rate.tick_volume = (m_Ticks.ModePlot == PRICE_FOREX ? 1 : 0);
044.                rate.open = rate.low = rate.high = rate.close = dClose;
045.             }else
046.             {
047.                rate.close = dClose;
048.                rate.high = (rate.close > rate.high ? rate.close : rate.high);
049.                rate.low = (rate.close < rate.low ? rate.close : rate.low);
050.                rate.real_volume += (long) m_Ticks.Info[iArg].volume_real;
051.                rate.tick_volume += (m_Ticks.bTickReal ? 1 : (int)m_Ticks.Info[iArg].volume);
052.             }
053.             
054.             return true;         
055.          }
056. //+------------------------------------------------------------------+
057.    private   :
058.       int         m_File;
059.       stInfoTicks m_Ticks;
060. //+------------------------------------------------------------------+
061. inline bool Open(const string szFileNameCSV)
062.          {
063.             string szInfo = "";
064.             
065.             if ((m_File = FileOpen("Market Replay\\Ticks\\" + szFileNameCSV + ".csv", FILE_CSV | FILE_READ | FILE_ANSI)) != INVALID_HANDLE)
066.             {
067.                for (int c0 = 0; c0 < 7; c0++) szInfo += FileReadString(m_File);
068.                if (szInfo == "<DATE><TIME><BID><ASK><LAST><VOLUME><FLAGS>") return true;
069.                Print("File ", szFileNameCSV, ".csv not a traded tick file.");
070.             }else
071.                Print("Tick file ", szFileNameCSV,".csv not found...");
072.                
073.             return false;
074.          }
075. //+------------------------------------------------------------------+
076. inline bool ReadAllsTicks(void)
077.          {
078.             string    szInfo;
079.                         
080.             Print("Loading replay ticks. Please wait...");
081.             ArrayResize(m_Ticks.Info, def_MaxSizeArray, def_MaxSizeArray);
082.             m_Ticks.ModePlot = PRICE_FOREX;
083.             while ((!FileIsEnding(m_File)) && (m_Ticks.nTicks < (INT_MAX - 2)) && (!_StopFlag))
084.             {
085.                ArrayResize(m_Ticks.Info, m_Ticks.nTicks + 1, def_MaxSizeArray);
086.                szInfo = FileReadString(m_File) + " " + FileReadString(m_File);
087.                m_Ticks.Info[m_Ticks.nTicks].time = StringToTime(StringSubstr(szInfo, 0, 19));
088.                m_Ticks.Info[m_Ticks.nTicks].time_msc = (m_Ticks.Info[m_Ticks.nTicks].time * 1000) + (int)StringToInteger(StringSubstr(szInfo, 20, 3));
089.                m_Ticks.Info[m_Ticks.nTicks].bid = StringToDouble(FileReadString(m_File));
090.                m_Ticks.Info[m_Ticks.nTicks].ask = StringToDouble(FileReadString(m_File));
091.                m_Ticks.Info[m_Ticks.nTicks].last = StringToDouble(FileReadString(m_File));
092.                m_Ticks.Info[m_Ticks.nTicks].volume_real = StringToDouble(FileReadString(m_File));
093.                m_Ticks.Info[m_Ticks.nTicks].flags = (uchar)StringToInteger(FileReadString(m_File));
094.                m_Ticks.ModePlot = (m_Ticks.Info[m_Ticks.nTicks].volume_real > 0.0 ? PRICE_EXCHANGE : m_Ticks.ModePlot);
095.                m_Ticks.nTicks++;
096.             }
097.             FileClose(m_File);
098.             if (m_Ticks.nTicks == (INT_MAX - 2))
099.             {
100.                Print("Too much data in tick file.\nIt is not possible to continue...");
101.                return false;
102.             }
103.             return (!_StopFlag);
104.          }
105. //+------------------------------------------------------------------+
106.       int SetSymbolInfos(void)
107.          {
108.             int iRet;
109.             
110.             CustomSymbolSetInteger(def_SymbolReplay, SYMBOL_DIGITS, iRet = (m_Ticks.ModePlot == PRICE_EXCHANGE ? 4 : 5));
111.             CustomSymbolSetInteger(def_SymbolReplay, SYMBOL_TRADE_CALC_MODE, m_Ticks.ModePlot == PRICE_EXCHANGE ? SYMBOL_CALC_MODE_EXCH_STOCKS : SYMBOL_CALC_MODE_FOREX);
112.             CustomSymbolSetInteger(def_SymbolReplay, SYMBOL_CHART_MODE, m_Ticks.ModePlot == PRICE_EXCHANGE ? SYMBOL_CHART_MODE_LAST : SYMBOL_CHART_MODE_BID);
113.             
114.             return iRet;
115.          }
116. //+------------------------------------------------------------------+
117.    public   :
118. //+------------------------------------------------------------------+
119.       C_FileTicks()
120.          {
121.             ArrayResize(m_Ticks.Rate, def_BarsDiary);
122.             m_Ticks.nRate = -1;
123.             m_Ticks.nTicks = 0;
124.             m_Ticks.Rate[0].time = 0;
125.          }
126. //+------------------------------------------------------------------+
127.       bool BarsToTicks(const string szFileNameCSV, int MaxTickVolume)
128.          {
129.             C_FileBars    *pFileBars;
130.             C_Simulation  *pSimulator = NULL;
131.             int           iMem = m_Ticks.nTicks,
132.                           iRet = -1;
133.             MqlRates      rate[1];
134.             MqlTick       local[];
135.             bool          bInit = false;
136.             
137.             pFileBars = new C_FileBars(szFileNameCSV);
138.             ArrayResize(local, def_MaxSizeArray);
139.             Print("Converting bars to ticks. Please wait...");
140.             while ((*pFileBars).ReadBar(rate) && (!_StopFlag))
141.             {
142.                if (!bInit)
143.                {
144.                   m_Ticks.ModePlot = (rate[0].real_volume > 0 ? PRICE_EXCHANGE : PRICE_FOREX);
145.                   pSimulator = new C_Simulation(SetSymbolInfos());
146.                   bInit = true;
147.                }
148.                ArrayResize(m_Ticks.Rate, (m_Ticks.nRate > 0 ? m_Ticks.nRate + 3 : def_BarsDiary), def_BarsDiary);
149.                m_Ticks.Rate[++m_Ticks.nRate] = rate[0];
150.                if (pSimulator == NULL) iRet = -1; else iRet = (*pSimulator).Simulation(rate[0], local, MaxTickVolume);
151.                if (iRet < 0) break;
152.                for (int c0 = 0; c0 <= iRet; c0++)
153.                {
154.                   ArrayResize(m_Ticks.Info, (m_Ticks.nTicks + 1), def_MaxSizeArray);
155.                   m_Ticks.Info[m_Ticks.nTicks++] = local[c0];
156.                }
157.             }
158.             ArrayFree(local);
159.             delete pFileBars;
160.             delete pSimulator;
161.             m_Ticks.bTickReal = false;
162.             
163.             return ((!_StopFlag) && (iMem != m_Ticks.nTicks) && (iRet > 0));
164.          }
165. //+------------------------------------------------------------------+
166.       datetime LoadTicks(const string szFileNameCSV, const bool ToReplay, const int MaxTickVolume)
167.          {
168.             int      MemNRates,
169.                      MemNTicks,
170.                      nDigits,
171.                      nShift;
172.             datetime dtRet = TimeCurrent();
173.             MqlRates RatesLocal[],
174.                      rate;
175.             MqlTick  TicksLocal[];
176.             bool     bNew;
177.             
178.             MemNRates = (m_Ticks.nRate < 0 ? 0 : m_Ticks.nRate);
179.             nShift = MemNTicks = m_Ticks.nTicks;
180.             if (!Open(szFileNameCSV)) return 0;
181.             if (!ReadAllsTicks()) return 0;         
182.             rate.time = 0;
183.             nDigits = SetSymbolInfos(); 
184.             m_Ticks.bTickReal = true;
185.             for (int c0 = MemNTicks, c1, MemShift = nShift; c0 < m_Ticks.nTicks; c0++, nShift++)
186.             {
187.                if (nShift != c0) m_Ticks.Info[nShift] = m_Ticks.Info[c0];
188.                if (!BuildBar1Min(c0, rate, bNew)) continue;
189.                if (bNew)
190.                {
191.                   if ((m_Ticks.nRate >= 0) && (ToReplay)) if (m_Ticks.Rate[m_Ticks.nRate].tick_volume > MaxTickVolume)
192.                   {
193.                      nShift = MemShift;
194.                      ArrayResize(TicksLocal, def_MaxSizeArray);
195.                      C_Simulation *pSimulator = new C_Simulation(nDigits);
196.                      if ((c1 = (*pSimulator).Simulation(m_Ticks.Rate[m_Ticks.nRate], TicksLocal, MaxTickVolume)) > 0)
197.                         nShift += ArrayCopy(m_Ticks.Info, TicksLocal, nShift, 0, c1);
198.                      delete pSimulator;
199.                      ArrayFree(TicksLocal);
200.                      if (c1 < 0) return 0;
201.                   }
202.                   MemShift = nShift;
203.                   ArrayResize(m_Ticks.Rate, (m_Ticks.nRate > 0 ? m_Ticks.nRate + 2 : def_BarsDiary), def_BarsDiary);
204.                };
205.                m_Ticks.Rate[(m_Ticks.nRate += (bNew ? 1 : 0))] = rate;
206.             }
207.             if (!ToReplay)
208.             {
209.                ArrayResize(RatesLocal, (m_Ticks.nRate - MemNRates));
210.                ArrayCopy(RatesLocal, m_Ticks.Rate, 0, 0);
211.                CustomRatesUpdate(def_SymbolReplay, RatesLocal, (m_Ticks.nRate - MemNRates));
212.                dtRet = m_Ticks.Rate[m_Ticks.nRate].time;
213.                m_Ticks.nRate = (MemNRates == 0 ? -1 : MemNRates);
214.                m_Ticks.nTicks = MemNTicks;
215.                ArrayFree(RatesLocal);
216.             }else m_Ticks.nTicks = nShift;
217.                            
218.             return dtRet;
219.          };
220. //+------------------------------------------------------------------+
221. inline stInfoTicks GetInfoTicks(void) const
222.          {
223.             return m_Ticks;
224.          }
225. //+------------------------------------------------------------------+
226. };
227. //+------------------------------------------------------------------+
228. #undef def_MaxSizeArray
229. //+------------------------------------------------------------------+

Header file C_FileTicks.mqh

Now that we've resolved those issues, we can move on to the next step. This involves allowing the user to define a value that will be used as the maximum number of ticks allowed within a one-minute bar. To keep things organized, we'll address this in a new section.

Allowing the User to Make Adjustments

This part is undoubtedly the easiest, simplest, and most enjoyable to implement. That's because the only real task you, as the developer, will have is to define the name of the key that will be used to set the value we need to adjust.

This key, in essence, is the value the user must enter in the configuration file for the asset. I've shown how to do this in previous articles in this series. However, since this is a very simple implementation, I won’t break the code down into fragments to explain each addition. Instead, before looking at the finalized code, let's first see an example of how this new functionality can be used. Let's see a sample configuration file for the replay/simulator application. The example is as follows.

01. [Config]
02. Path = WDO
03. PointsPerTick = 0.5
04. ValuePerPoints = 5.0
05. VolumeMinimal = 1.0
06. Account = NETTING
07. MaxTicksPerBar = 2800
08. 
09. [Bars]
10. WDON22_M1_202206140900_202206141759
11. 
12. [ Ticks -> Bars]
13. 
14. [ Bars -> Ticks ]
15. 
16. [Ticks]
17. WDON22_202206150900_202206151759

Configuration file example

Notice line seven, where a new configuration setting has been introduced. If you try using this configuration file with an earlier version of the replay/simulation application for MetaTrader 5, you'll receive an error message indicating that line seven contains an unrecognized parameter. However, from the version I'm presenting here, the application is able to interpret what line seven means.

One important note: neither you nor the end user is required to specify this new setting on line seven. If provided, the configuration will override the default value embedded within the compiled application. If omitted, the replay/simulator service will fall back to using the internally defined default set during compilation.

I'm pointing this out before showing the code because I want to emphasize that this configuration is entirely optional. However, when used, it will take precedence over the precompiled value. One final reminder: each configuration file is unique. This means you can define a different maximum tick count for each one. So feel free to experiment with different setups until you find the configuration that maintains good performance in MetaTrader 5 and ensures the chart is rendered smoothly.

Now, let's take a look at the updated header file responsible for interpreting and applying the value from the configuration file. I'm talking about C_ConfigService.mqh. Its updated code is shown in full below.

001. //+------------------------------------------------------------------+
002. #property copyright "Daniel Jose"
003. //+------------------------------------------------------------------+
004. #include "Support\C_FileBars.mqh"
005. #include "Support\C_FileTicks.mqh"
006. #include "Support\C_Array.mqh"
007. //+------------------------------------------------------------------+
008. class C_ConfigService : protected C_FileTicks
009. {
010.    protected:
011. //+------------------------------------------------------------------+
012.    private   :
013.       enum eWhatExec {eTickReplay, eBarToTick, eTickToBar, eBarPrev};
014.       enum eTranscriptionDefine {Transcription_INFO, Transcription_DEFINE};
015.       struct st001
016.       {
017.          C_Array *pTicksToReplay, *pBarsToTicks, *pTicksToBars, *pBarsToPrev;
018.          int      Line,
019.                   MaxTickVolume;
020.          bool     AccountHedging;
021.          char     ModelLoading;
022.          string   szPath;
023.       }m_GlPrivate;
024. //+------------------------------------------------------------------+
025. inline void FirstBarNULL(void)
026.          {
027.             MqlRates rate[1];
028.             int c0 = 0;
029.             
030.             for(; (GetInfoTicks().ModePlot == PRICE_EXCHANGE) && (GetInfoTicks().Info[c0].volume_real == 0); c0++);
031.             rate[0].close = (GetInfoTicks().ModePlot == PRICE_EXCHANGE ? GetInfoTicks().Info[c0].last : GetInfoTicks().Info[c0].bid);
032.             rate[0].open = rate[0].high = rate[0].low = rate[0].close;
033.             rate[0].tick_volume = 0;
034.             rate[0].real_volume = 0;
035.             rate[0].time = macroRemoveSec(GetInfoTicks().Info[c0].time) - 86400;
036.             CustomRatesUpdate(def_SymbolReplay, rate);
037.          }
038. //+------------------------------------------------------------------+
039. inline eTranscriptionDefine GetDefinition(const string &In, string &Out)
040.          {
041.             string szInfo;
042.             
043.             szInfo = In;
044.             Out = "";
045.             StringToUpper(szInfo);
046.             StringTrimLeft(szInfo);
047.             StringTrimRight(szInfo);
048.             if (StringSubstr(szInfo, 0, 1) == "#") return Transcription_INFO;
049.             if (StringSubstr(szInfo, 0, 1) != "[")
050.             {
051.                Out = szInfo;
052.                return Transcription_INFO;
053.             }
054.             for (int c0 = 0; c0 < StringLen(szInfo); c0++)
055.                if (StringGetCharacter(szInfo, c0) > ' ')
056.                   StringAdd(Out, StringSubstr(szInfo, c0, 1));               
057.             
058.             return Transcription_DEFINE;
059.          }
060. //+------------------------------------------------------------------+
061. inline bool Configs(const string szInfo)
062.          {
063.             const string szList[] = {
064.                      "PATH",
065.                      "POINTSPERTICK",
066.                      "VALUEPERPOINTS",
067.                      "VOLUMEMINIMAL",
068.                      "LOADMODEL",
069.                      "ACCOUNT",
070.                      "MAXTICKSPERBAR"
071.                                     };
072.             string    szRet[];
073.             char      cWho;
074.             
075.             if (StringSplit(szInfo, '=', szRet) == 2)
076.             {
077.                StringTrimRight(szRet[0]);
078.                StringTrimLeft(szRet[1]);
079.                for (cWho = 0; cWho < ArraySize(szList); cWho++) if (szList[cWho] == szRet[0]) break;
080.                switch (cWho)
081.                {
082.                   case 0:
083.                      m_GlPrivate.szPath = szRet[1];
084.                      return true;
085.                   case 1:
086.                      CustomSymbolSetDouble(def_SymbolReplay, SYMBOL_TRADE_TICK_SIZE, StringToDouble(szRet[1]));
087.                      return true;
088.                   case 2:
089.                      CustomSymbolSetDouble(def_SymbolReplay, SYMBOL_TRADE_TICK_VALUE, StringToDouble(szRet[1]));
090.                      return true;
091.                   case 3:
092.                      CustomSymbolSetDouble(def_SymbolReplay, SYMBOL_VOLUME_STEP, StringToDouble(szRet[1]));
093.                      return true;
094.                   case 4:
095.                      m_GlPrivate.ModelLoading = StringInit(szRet[1]);
096.                      m_GlPrivate.ModelLoading = ((m_GlPrivate.ModelLoading < 1) && (m_GlPrivate.ModelLoading > 4) ? 1 : m_GlPrivate.ModelLoading);
097.                      return true;
098.                   case 5:
099.                      if (szRet[1] == "HEDGING") m_GlPrivate.AccountHedging = true;
100.                      else if (szRet[1] == "NETTING") m_GlPrivate.AccountHedging = false;
101.                      else
102.                      {
103.                         Print("Entered account type is not invalid.");                        
104.                         return false;
105.                      }
106.                      return true;
107.                   case 6:
108.                      m_GlPrivate.MaxTickVolume = (int) MathAbs(StringToInteger(szRet[1]));
109.                      return true;         
110.                }
111.                Print("Variable >>", szRet[0], "<< not defined.");
112.             }else
113.                Print("Configuration definition >>", szInfo, "<< invalidates.");
114.                
115.             return false;
116.          }
117. //+------------------------------------------------------------------+
118. inline bool WhatDefine(const string szArg, char &cStage)
119.          {
120.             const string szList[] = {
121.                      "[BARS]",
122.                      "[TICKS]",
123.                      "[TICKS->BARS]",
124.                      "[BARS->TICKS]",
125.                      "[CONFIG]"
126.                                     };
127.                                     
128.             cStage = 1;
129.             for (char c0 = 0; c0 < ArraySize(szList); c0++, cStage++)
130.                if (szList[c0] == szArg) return true;
131.                
132.             return false;
133.          }
134. //+------------------------------------------------------------------+
135. inline bool CMD_Array(char &cError, eWhatExec e1)
136.          {
137.             bool       bBarsPrev = false;
138.             string     szInfo;
139.             C_FileBars *pFileBars;
140.             C_Array    *ptr = NULL;
141.             
142.             switch (e1)
143.             {
144.                case eTickReplay : ptr = m_GlPrivate.pTicksToReplay; break;
145.                case eTickToBar  : ptr = m_GlPrivate.pTicksToBars;   break;
146.                case eBarToTick  : ptr = m_GlPrivate.pBarsToTicks;   break;
147.                case eBarPrev    : ptr = m_GlPrivate.pBarsToPrev;    break;
148.             }            
149.             if (ptr != NULL)
150.             {
151.                for (int c0 = 0; (c0 < INT_MAX) && (cError == 0); c0++)
152.                {
153.                   if ((szInfo = ptr.At(c0, m_GlPrivate.Line)) == "") break;
154.                   switch (e1)
155.                   {
156.                      case eTickReplay:
157.                         if (LoadTicks(szInfo, true, m_GlPrivate.MaxTickVolume) == 0) cError = 4;
158.                         break;
159.                      case eTickToBar :
160.                         if (LoadTicks(szInfo, false, m_GlPrivate.MaxTickVolume) == 0) cError = 5; else bBarsPrev = true;
161.                         break;
162.                      case eBarToTick :
163.                         if (!BarsToTicks(szInfo, m_GlPrivate.MaxTickVolume)) cError = 6;
164.                         break;
165.                      case eBarPrev   :
166.                         pFileBars = new C_FileBars(szInfo);
167.                         if ((*pFileBars).LoadPreView() == 0) cError = 3; else bBarsPrev = true;
168.                         delete pFileBars;
169.                         break;
170.                   }
171.                }
172.                delete ptr;
173.             }
174.             
175.             return bBarsPrev;
176.          }
177. //+------------------------------------------------------------------+
178.    public   :
179. //+------------------------------------------------------------------+
180.       C_ConfigService()
181.          :C_FileTicks()
182.          {
183.             m_GlPrivate.AccountHedging = false;
184.             m_GlPrivate.ModelLoading = 1;
185.             m_GlPrivate.MaxTickVolume = 2000;
186.          }
187. //+------------------------------------------------------------------+
188. inline const bool TypeAccountIsHedging(void) const
189.          {
190.             return m_GlPrivate.AccountHedging;
191.          }
192. //+------------------------------------------------------------------+
193.       bool SetSymbolReplay(const string szFileConfig)
194.          {
195. #define macroFileName ((m_GlPrivate.szPath != NULL ? m_GlPrivate.szPath + "\\" : "") + szInfo)
196.             int      file;
197.             char     cError,
198.                      cStage;
199.             string   szInfo;
200.             bool     bBarsPrev;
201.                         
202.             if ((file = FileOpen("Market Replay\\" + szFileConfig, FILE_CSV | FILE_READ | FILE_ANSI)) == INVALID_HANDLE)
203.             {
204.                Print("Failed to open configuration file [", szFileConfig, "]. Service being terminated...");
205.                return false;
206.             }
207.             Print("Loading data for playback. Please wait....");
208.             cError = cStage = 0;
209.             bBarsPrev = false;
210.             m_GlPrivate.Line = 1;
211.             m_GlPrivate.pTicksToReplay = m_GlPrivate.pTicksToBars = m_GlPrivate.pBarsToTicks = m_GlPrivate.pBarsToPrev = NULL;
212.             while ((!FileIsEnding(file)) && (!_StopFlag) && (cError == 0))
213.             {
214.                switch (GetDefinition(FileReadString(file), szInfo))
215.                {
216.                   case Transcription_DEFINE:
217.                      cError = (WhatDefine(szInfo, cStage) ? 0 : 1);
218.                      break;
219.                   case Transcription_INFO:
220.                      if (szInfo != "") switch (cStage)
221.                      {
222.                         case 0:
223.                            cError = 2;
224.                            break;
225.                         case 1:
226.                            if (m_GlPrivate.pBarsToPrev == NULL) m_GlPrivate.pBarsToPrev = new C_Array();
227.                            (*m_GlPrivate.pBarsToPrev).Add(macroFileName, m_GlPrivate.Line);
228.                            break;
229.                         case 2:
230.                            if (m_GlPrivate.pTicksToReplay == NULL) m_GlPrivate.pTicksToReplay = new C_Array();
231.                            (*m_GlPrivate.pTicksToReplay).Add(macroFileName, m_GlPrivate.Line);
232.                            break;
233.                         case 3:
234.                            if (m_GlPrivate.pTicksToBars == NULL) m_GlPrivate.pTicksToBars = new C_Array();
235.                            (*m_GlPrivate.pTicksToBars).Add(macroFileName, m_GlPrivate.Line);
236.                            break;
237.                         case 4:
238.                            if (m_GlPrivate.pBarsToTicks == NULL) m_GlPrivate.pBarsToTicks = new C_Array();
239.                            (*m_GlPrivate.pBarsToTicks).Add(macroFileName, m_GlPrivate.Line);
240.                            break;
241.                         case 5:
242.                            if (!Configs(szInfo)) cError = 7;
243.                            break;
244.                      }
245.                      break;
246.                };
247.                m_GlPrivate.Line += (cError > 0 ? 0 : 1);
248.             }
249.             FileClose(file);
250.             CMD_Array(cError, (m_GlPrivate.ModelLoading <= 2 ? eTickReplay : eBarToTick));
251.             CMD_Array(cError, (m_GlPrivate.ModelLoading <= 2 ? eBarToTick : eTickReplay));
252.             bBarsPrev = (CMD_Array(cError, ((m_GlPrivate.ModelLoading & 1) == 1 ? eTickToBar : eBarPrev)) ? true : bBarsPrev);
253.             bBarsPrev = (CMD_Array(cError, ((m_GlPrivate.ModelLoading & 1) == 1 ? eBarPrev : eTickToBar)) ? true : bBarsPrev);
254.             switch(cError)
255.             {
256.                case 0:
257.                   if (GetInfoTicks().nTicks <= 0)
258.                   {
259.                      Print("There are no ticks to use. Service is being terminated...");
260.                      cError = -1;
261.                   }else if (!bBarsPrev) FirstBarNULL();
262.                   break;
263.                case 1  : Print("The command on the line ", m_GlPrivate.Line, " not recognized by the system..."); break;
264.                case 2  : Print("The system did not expect the contents of the line: ", m_GlPrivate.Line);         break;
265.                default : Print("Error accessing the file indicated in the line: ", m_GlPrivate.Line);
266.             }
267.                      
268.             return (cError == 0 ? !_StopFlag : false);
269. #undef macroFileName
270.          }
271. //+------------------------------------------------------------------+
272. };
273. //+------------------------------------------------------------------+

Source code of the header file C_ConfigService.mqh

The code of the C_ConfigService class is truly a pleasure to work with. That's because it's the kind of code that requires minimal changes and lets us accomplish a lot with very little effort. This is a rare occurrence. But let's get to the point. This is where all the underlying layers do the heavy lifting to allow us to load the data needed for replay or simulation. Everything is orchestrated according to the contents of the configuration file, like the one we showed earlier.

So, the first thing we did in this code was to define a new variable. This can be seen on line 19. Its name is quite intuitive and clearly indicates what we intend to do in the following steps. This same variable is initialized on line 185, within the class constructor. Because of this initialization, if the configuration file does not specify a different value, this default one will be used.

Now, you might be wondering: "Wait a minute, why aren't you using the def_MaxTicksVolume definition from the Defines.mqh file?" The reason is that definition no longer exists. It was removed because it's no longer necessary. Since we started working with the configuration file, we no longer depend on certain hardcoded values. That's why def_MaxTicksVolume was removed from Defines.mqh. If you'd like to keep it, that's perfectly fine. But don't be surprised if, in future articles where we revisit Defines.mqh, that definition is no longer there.

Let's now understand why that definition became unnecessary. During the development of the feature that allows the replay system to simulate ticks when there's an overflow within a one-minute bar, I initially hadn't defined where this information would come from. To avoid cluttering the code with too many values, I created a general-purpose definition. Thus def_MaxTicksVolume was born and placed in Defines.mqh. This type of design decision is common when you're building something expected to evolve over time. However, once we reached the C_ConfigService class, I had the idea to allow users to customize this value without needing to recompile the entire project.

In any case, the classes responsible for the hard work were already implemented. So, all we had to do was pass this maximum tick count value down to them. With just a minor code edit, we were able to ensure the appropriate classes could receive and use a value defined for the entire application. The corresponding method calls appear on lines 157, 160, and 163.

Notice that we didn't really add any major new logic but just made small adjustments to enable the necessary support for this configuration. However, this gives us control over the maximum number of ticks that can be simulated or exist within a single one-minute bar.

Now, here's the real reason everything was funneled into this class. This is to allow the user to define the maximum number of ticks allowed within a one-minute bar. If you're jumping into this code out of context, you might assume this task is complicated and extensive. But it's not. It's actually simple, easy, and fast to implement the ability for a user to specify the maximum number of ticks.

First, we add the configuration key. This is done by adding another string to the list of keys used for configuration parameters. That array is defined in line 63. Now pay attention. I've explained this before, but it's worth repeating. When adding a new key, always define it in uppercase. It doesn't matter what the key is. Just make sure it's in uppercase. Our new key is defined in line 70. Now, there's another trick: in line 75, we use an MQL5 library function to split the key-value pairs. The equal sign is used as the separator. So, whatever is before the equal sign is considered the key, and whatever comes after is the value assigned to that key.

The next key point is in line 79, where we search for the key in the array to find its index. Because of this, if you change the order of keys in the array, you must also update the corresponding index used later. This isn't difficult, it just requires careful attention.

In our case, the index for the new setting is 6. So, in line 107, we define how this setting should be applied. Since we expect an integer value, we use another MQL5 library function to perform the conversion. And that's how the user can set a value to be used as the maximum tick count for a one-minute bar.

Important note: In line 108, where we convert the value from the configuration file into a usable integer, we do not perform any validation checks to ensure the value falls within acceptable parameters. The only safeguard in place is to ensure the value is positive. If the value is otherwise inconsistent or invalid, the simulation process might fail.

Final conclusions

Before concluding this article, I'd like to remind you of something mentioned in the previous one. I'm talking about the video that was included there. Although the main focus of this article has been on implementing a tick limit system for the replay functionality, I want to emphasize that the bugs shown in that video weren't forgotten.

While those bugs aren't particularly critical, as they don't seem to compromise application stability or crash the platform, they do exist. In particular, some graphical objects are not removed properly when the chart is closed. Although this only occurs under very specific circumstances, we are already working on a fix. Once that's resolved, I will publish a follow-up article explaining how that issue was addressed.

To conclude this article, I invite you to watch the short video embedded below. It demonstrates how the system currently operates, especially how it behaves depending on whether or not the tick limit is defined in the configuration file.

The video is brief but effectively highlights the noticeable difference between using simulated data and real tick data. You'll see that when we use Random Walk simulation to fill in the ticks, the resulting movement looks quite different from the actual tick data generated by real trades.

One last point I want to bring up concerns another issue. This one is minor and more annoying than harmful. You can notice it in the video. Occasionally, the replay/simulation service will randomly pause itself, requiring you to hit play again.

While this issue is relatively harmless, it does need to be resolved quickly. So, in the next article, we'll address this bug and ensure that the replay/simulator doesn’t unexpectedly enter pause mode. We will also begin re-enabling some features that are still temporarily disabled in the service.

Demo video