Record Indicator Buffer and Playback in Backtest

2020.05.18 Indicators Experts

Specification

Hi!

I want to create a tool that has two parts. The idea is to record indicator buffers from indicators that cannot be used in backtesting (Multiple Time Frame, Multicurrentcy, Feeds from Internet html sites and telegram etc.) and play them back in a backtest so that optimizations can be done on the recorded data.

This would be a much more effective way to walk forward optimise than have 100+ instances of MT open each with different settings (and I'm sure my broker would prefer that anyway)

An EA for recording and an Indicator for playback will need to be made.

What I am looking to do is record 20 individual buffer values (ideally X number, but at least 20) along with live tick data (Bid and Ask and volumes if available) to separate CSV files.

The tick data can then be converted into hst and fxt files using Birt's script (or other tools) for backtesting. The indicator buffers will then be read from the recorded buffers CSV file and played back during the backtest.

The first CSV file will record the raw price tick data as received by the terminal. It must be in the format:

EURUSD.recordedPriceTickData.csv
-----------------------------------------------------
Timestamp,Bid price,Ask price,Bid volume,Ask volume
20200515 10:53:24:108,1.08096,1.08097,1.1,1.2
20200515 10:53:24:210,1.08094,1.08097,1.5,1.3
etc.
-----------------------------------------------------

If separate Bid or Ask volumes are unavailable set both to the same volume, also have an extern volume if that information is not available from the terminal.

The EA will have extern inputs for 20 different  values of indicator name, buffer number, and shift.

For example

-----------
....
extern string Indicator1 = "Ichimoku";
extern int Indicator1Buffer = 2;
extern int Indicator1BufferShift = 0;
extern string Indicator2 = "FFC"
extern int Indicator2Buffer = 0;
extern int Indicator2BufferShift = 1;
extern string Indicator3 = "Market\\Universal Heiken Ashi"
extern int Indicator3Buffer = 1;
extern int Indicator3BufferShift = 44;
...
-----------

We need to record each buffer value for each indicator for each tick. This is necessary to handle "repainting" indicators. That is, the raw buffer values need to be recorded each tick irrespective of previous or future ticks.

EURUSD.recordedIndicatorBufferData.csv
-----------------------------------------------------
Timestamp,CandleStart,CandleTickNumber,buffer1,buffer2,buffer3,buffer4,buffer5,buffer6 etc...
....
20200515 10:53:24:108,20200515 10:53:0:0,33,1.24233,0,1.2337,4.0,5.0,6.0, ...
20200515 10:53:24:210,20200515 10:53:0:0,34,1.24234,1,1.2344,4.0,5.0,6.0, ...
etc.
-----------------------------------------------------

The indicator must exist as a separate window and will have 20 output buffers that only populate the current tick/bar - we can record multiple shifts of the same indicator to the buffer and read previous shifts that way (so that we can record repainting indicators up to 20 bars back if we use all the recording space for that).

The above I am able to do with my knowledge of mql4. However, the next part is where it gets difficult:

Each tick of price must line up with the recorded buffers that is played back during backtesting to the tick. Exactly to the tick.

This must work on normal timeframe charts and have the option of working on Renko charts too.
---I.e. if we make renko charts from the recorded tick data, the recorded indicator values must work with the generated renko charts too.  <--- This will probably be difficult

The Recording EA must have an option of pausing recording with a button on the chart. When recording is paused, the CSV files must be closed so that I can copy/paste/delete without corrupting data.


The CSV files must be concatinateable. I.e. it must continue recording to the same file when unpaused, or when MT4 is restarted.

EG:
--------------------------

handle = FileOpen("TickData_"+Symbol()+".csv", FILE_CSV|FILE_WRITE|FILE_READ|FILE_SHARE_READ);
  if (handle > 0) FileSeek(handle, 0, SEEK_END);

etc...

--------------------------

The playback indicator must be able to handle this when backtesting. When this happens, it must still playback data exactly to the tick.

I want to have ex4 files to test during development of the EA and indicator. I will test with recording say MA buffer values and checking that they line up - both with normal charts and Renko charts.

I want to own exclusive the rights to this idea.

That is all :)

Suggestions are welcome, and the layout of the buffers CSV file is just a suggestion. The layout of the tick data CSV must be as the example so that it works with normal CSV to FXT scripts.

Job has been deleted

Project information

Budget
50+ USD
For the developer
45 USD
Deadlines
from 7 to 30 day(s)