problem with ibarshift getting values from higher timeframes in strategy tester - page 2

New comment
 
He did answer your question and provided the solution. Test and debug your indicator in the indicator sandbox. Then test your EA in the EA sandbox (it can use iCustom just fine.) Do not attach the indicator to the chart.
 
nina32:

but currently it seems that there is no way to do a visual backtest of an ea with ATTACHED mtf indicators...

Summarising what you have been told so far, MT4 is not designed to do what you want to do. (I don't think this is a "bug". I don't think that adding multi-timeframe indicators to an EA's backtesting chart has ever been intended to work properly. That's one of the reasons why there is the separate mechanism for backtesting an indicator.)

However, try the following .mqh header file - not fully tested, but may work provided that you follow the guidelines explained in the header file.

If you take the simple example which I gave at https://www.mql5.com/en/forum/160159, then this indicator does work correctly if you #include the header file at the start of the code:

#include <timeseries-visualmode-fixup.mqh>

#property indicator_chart_window

int OnCalculate(const int rates_total, const int prev_calculated, const datetime &time[], const double &open[], const double &high[], 
const double &low[], const double &close[], const long &tick_volume[], const long &volume[], const int &spread[])
{
   // Shows the last real close price from the live market, before the backtesting starts.
   // Does NOT show the constantly changing latest price within the backtesting environment.
   Comment("Close price: ", iClose(Symbol(), PERIOD_D1, 0));
   return(rates_total);
}


[EDIT: file re-attached, fixing a bug with iBarShift with exact=true. Almost certainly not the only bug.] 

 
jjc: Summarising what you have been told so far, MT4 is not designed to do what you want to do. (I don't think this is a "bug". I don't think that adding multi-timeframe indicators to an EA's backtesting chart has ever been intended to work properly. That's one of the reasons why there is the separate mechanism for backtesting an indicator.)

However, try the following .mqh header file - not fully tested, but may work provided that you follow the guidelines explained in the header file.

If you take the simple example which I gave at https://www.mql5.com/en/forum/160159, then this indicator does work correctly if you #include the header file at the start of the code:

@JJC: You have outdone yourself! Although I have no need for it myself (as I follow a MQL5-like style of coding that does not use functions like iBarShift), I have to thank you for your contribution!

Irrespective of what the OP may eventually say, I have to take my hat off to you for truly going above and beyond of what may be expected of posters on the forum (because it seems that you came up with this code specifically for this thread).

PS! Just don't overdo it, or users will end up exploiting your goodwill!

 
FMIC:

Although I have no need for it myself

Me neither. It's a matter of intellectual curiosity. One of the things it explores is copying an array onto itself. The documentation describes the behaviour of this as "undefined". It does in fact seem to work reliably, which is what I'd expect because I'd assume that ArrayCopy gets translated pretty directly into a call to the Win32 RtlMoveMemory, which does allow overlapping source and destination parameters.

 

FMIC:

as I follow a MQL5-like style of coding that does not use functions like iBarShift

The problem with MT4 backtesting applies both to the "old-style" functions such iBarShift and also the "new-style", MT5-compatible timeseries functions such as CopyRates.
 
jjc: The problem with MT4 backtesting applies both to the "old-style" functions such iBarShift and also the "new-style", MT5-compatible timeseries functions such as CopyRates.
I meant, that since MQL5 does not support many of the "iFunctions", such as iBarShift, I adopted a different style/method of my own so as to be compatible to both MQL4 and MQL5 and which is not susceptible to this multi-time-frame problem in MT4 back-testing.
 
############## offtopic #########################
what you are trying to do is beyond the limitations
@ jjc, i like people who don't accept limits ;)

you can't imagine how often i get statements like this... in about 90% i found a solution by myself or in the internet. in one of my last projects, a own trading platform (in free-pascal) to analyze strategy's with full 4-core support of the cpu - i was searching for a way to get the history, the quotes and account details of mt4 to the outside world in REAL-TIME. people told me -> there is no way, big delay, you can't do this and so on, accept the limits! i asked in some forums, there was no solution and people often get me wrong because of my bad english ;(.... so i decided to go my own way, i used memory mapped files (i never have seen mmf before), copied pointers / records to the mmf and after some hours i was able to get the pointers and values from my external platform... now i have access to everything in mt4, account details, trades, history, mql rates and realtime changes, all marketinfo values and so on... the copy process is done in milliseconds, i can copy 100000 of mql rates in a few ms... for multiple currencys...

i did it with the kernel32, in mql4 there really was no solution, that does not mean that there is no way ;)

#import "kernel32.dll" 

 int  CreateMutexW(int attr, int owner, string mutexName);   
 int  WaitForSingleObject(int hnd, int dwMilliseconds);    
 int  ReleaseMutex(int hnd);   

 int  CreateFileMappingW(int hFile, int lpAttributes, int flProtect, int dwMaximumSizeHigh, int dwMaximumSizeLow, string lpName);  
 int  MapViewOfFile(int hFileMappingObject, int dwDesiredAccess, int dwFileOffsetHigh, int dwFileOffsetLow, int dwNumberOfBytesToMap);  
 int  UnmapViewOfFile(int lpBaseAddress);   

 void RtlMoveMemory(int DestPtr , const STRUCT_ACCOUNT       &ACCOUNT    , int Length);      
 void RtlMoveMemory(int DestPtr , const MqlRates             &MQLRATES[] , int Length); 
 void RtlMoveMemory(int DestPtr , const STRUCT_SYMBOLS_ARRAY &SYMBOL[]   , int Length); 
 
 void RtlZeroMemory(int DestPtr , int Length);
 void RtlSecureZeroMemory(int DestPtr , int Length);
 
 int  GetLastError();         // call Print(kernel32::GetLastError());  
 void SetLastError(int _err); // call Print(kernel32::SetLastError(0));   
 int  CloseHandle(int hwnd);    
 
#import

so what i want to say, never trust people who tell you that you're unable to do something. sometimes they simply don't know a way. and that was the reason i was asking the same question again and again, why does it work for short times ? that means that there is maybe a hack possible, if you understand how it works you can maybe fix it, but i was missing the internals, i had no idea - what they do or don't and why it fails... jjc, keep it on ;)

############## offtopic #########################

great work, i have downloaded your mqh file and i will read it today, i just had a quick view. i have not yet tested it. but thanks for making it public! you proved what i assumed, the needed data is there but something does not work...

now you have done what i was thinking about the last few days, a workaround that fixes the problem, well done! BUT!

i don't want to go on and just use it. i will read every letter until i have the full understanding what you're are doing in every single piece of your code... your work is worth it ;) it will take some time to test and understand it, and maybe i will ask some questions ;)

one final question: jjc can you explain just for for me ;) - in easy words - why IBarShift fails? somewhere i was reading something about the problem is because of TimeCurrent() or something is wrong? what is exactly going on when it fails and why does it work when i for example reload the indi or recompile it ?

it seems that refreshing or reloading it also refresh the internal array buffers of all timeframes - and so it works for the candle i was doing it. if the visual test is running, they do shift the current timeframe buffer for attached indicators, but the don't do it for all other timeframes in the attached indicator...
if you now use iBarShift for other timeframes, it will fail because the chart candle does not exits in the other timeframe buffer arrays? is that right ? or can you explain the reason, what is exactly the problem - i still not understand it completely.

and guys sorry for my bad english, it is not my native language and i only had a few lessons at school ;)
 
nina32: one final question: jjc can you explain just for for me ;) - in easy words - why IBarShift fails? somewhere i was reading something about the problem is because of TimeCurrent() or something is wrong? what is exactly going on when it fails and why does it work when i for example reload the indi or recompile it ?
He has has already answered that question in the comments of his code, and I quote:

...

The timeseries functions such as iBarShift do give the indicator access to historic data up to the start of the backtesting period. The problem is that the historic data on other timeframes does not then update.

As an extra note: iBarShift etc do work in a multi-timeframe indicator if the requested timeframe is the same as the backtesting timeframe. What is clearly happening is that MT4 detects that it can use the main backtesting data rather than timeseries access.

What this code does is maintain its own timeseries data, and then do lookups on that instead of using the normal iBarShift, iClose etc. It grabs the static history data which is available and correct at the start of the backtesting (see note above), and then adds in new and updated bars from each simulated tick in the backtesting.

...

 
i was now able to take a deeper look into its mqh file and now i have seen it too ;)

but thanks for adding it FMIC ;) and nice to see you back here ;)!


so the conclusion is, the needed data is there, even when the buffer of other timeframes is not correctly shifted, if i'm testing on H1 and all the data of other timeframes is there - from M1 to MN1...

jjc does a copy of it, he implemented its own functions to replace the internal functions, he creates a wrapper... nice and easy solution with the injection - by the way ;)

i have still not tested it, i need to eat now but maybe today or tomorrow i'll find some time to check it in full ;)

so forgive me FMIC that i'm asking some more questions, you don't need to answer ;)

at the moment it seems like they have all internal structures to make it working, jjc has access to every candle value in every timeframe.

that means the data is there but the mt4 internal arrays are not updated / refreshed for other timeframes... and if i reload/recompile the attached indis it seems to refresh all the timeframes...

before you posted that - i was thinking if it is maybe possible to refresh the other timeframes with some of the not / bad documented functions like MetaTrader4_Internal_Message / PostMessageW() or something ;)

 
jjc:

Summarising what you have been told so far, MT4 is not designed to do what you want to do. (I don't think this is a "bug". I don't think that adding multi-timeframe indicators to an EA's backtesting chart has ever been intended to work properly. That's one of the reasons why there is the separate mechanism for backtesting an indicator.)

However, try the following .mqh header file - not fully tested, but may work provided that you follow the guidelines explained in the header file.

If you take the simple example which I gave at https://www.mql5.com/en/forum/160159, then this indicator does work correctly if you #include the header file at the start of the code:


[EDIT: file re-attached, fixing a bug with iBarShift with exact=true. Almost certainly not the only bug.] 

 

Excellent work JJC

I tested it with our MTF indicators and it appears to work very well. I also profiled it and the overhead is smaller than I expected.

The only issue I spotted is the allocation of M1 minutes into the higher timeframes using the RoundTime method.

datetime RoundTime(datetime tm) {
   return (datetime)MathFloor(tm / (Timeframe * 60)) * Timeframe * 60;      
}

This works well for the low /medium timeframes, however I'm not sure it will map correctly to the Weekly and Monthly boundaries

 
jamescater:

This works well for the low /medium timeframes, however I'm not sure it will map correctly to the Weekly and Monthly boundaries

You're right. It won't work with MN1. It should work with W1.
123
New comment
Reason: