Scripts: Tick Audit - check real tick history and backtest quality month by month - page 2

New comment
 
Michael Charles Schefe #:
i meant to simply report what ticks i already have; ie to not download further/more ticks.
Michael, thanks for pinning it down. That is the harder of the two readings.
Reading the stored ticks and fetching them turn out to be the same call.
CopyTicks and CopyTicksRange both begin synchronising the symbol before they
answer, and a request already in flight runs to completion. So there is no
version of this that reports only what is on disk and asks for nothing.
What can be fixed is how much it asks for. The script samples six days from
every month, so a month you do not hold is requested six separate times over.
Stopping a month at the first miss, and putting a time budget on the whole
run, turns that into one request and a report that says which months it did
not reach instead of calling them empty. That is going on the list for the
next version.
Being straight about what that still will not do: the first request for a
month you do not hold will still pull that month down. A budget stops it
going further, it does not make the first one free.
 
Ryan L Johnson #:

In that case, I was mistaken.

Anyway, isn't that the purpose of the input, "How many months back to audit"?

And how would the script report empty/sparse M1 bars/months?

Ryan, on the months input, it sets how far back to look rather than whether
anything gets fetched. Ask for three months and it will still pull those
three if the terminal does not hold them. It bounds the range, not the
download, which is why Michael's request needs a separate control.
On sparse and empty months, the table keeps those counts apart. Every month
line carries the days it found data on, how many of those had bars, and how
many had ticks. A month with bars and no ticks reads 6 and 0 and gets the
bars-only verdict. A month where the terminal holds nothing at all is printed
with zeros rather than dropped, because a gap in a table reads as an
oversight. A day with neither bars nor ticks is not counted at all, which is
what keeps a crypto week at seven days and stops a holiday looking like a
hole.
What it cannot do is tell you why a day is missing. A closed market and a gap
in your own history look identical from outside, so a month with sparse bars
shows up as a low day count and the reason is yours to judge.
 
Ayaz #:
Ryan, on the months input, it sets how far back to look rather than whether
anything gets fetched. Ask for three months and it will still pull those
three if the terminal does not hold them. It bounds the range, not the
download, which is why Michael's request needs a separate control.
On sparse and empty months, the table keeps those counts apart. Every month
line carries the days it found data on, how many of those had bars, and how
many had ticks. A month with bars and no ticks reads 6 and 0 and gets the
bars-only verdict. A month where the terminal holds nothing at all is printed
with zeros rather than dropped, because a gap in a table reads as an
oversight. A day with neither bars nor ticks is not counted at all, which is
what keeps a crypto week at seven days and stops a holiday looking like a
hole.
What it cannot do is tell you why a day is missing. A closed market and a gap
in your own history look identical from outside, so a month with sparse bars
shows up as a low day count and the reason is yours to judge.

Thank you for the explanation.

What I meant was... After the initial run of the script, the user has the numbers in the Experts tab. At that point, can't the user simply run the script again with the bogus month preceding the set start month?

(The script runs in about 10 seconds on my machine).
 
Ryan L Johnson #:

Thank you for the explanation.

What I meant was... After the initial run of the script, the user has the numbers in the Experts tab. At that point, can't the user simply run the script again with the bogus month preceding the set start month?

(The script runs in about 10 seconds on my machine).
Ryan, yes, and that is the sensible way to use it. Once the first run has
told you where the ticks begin, a second run aimed at just the boundary
month settles it in seconds.
The ten seconds is worth explaining though. A run is fast when the terminal
already holds those months, and after your first run it does. The one month
that takes real time is the one it does not hold, and on the boundary that
is exactly the month you are asking about, so a narrow re-run can still sit
there downloading it. That is the same wall Michael ran into from the other
side.
So it is quick where the answer is already local, and slow exactly where the
answer is still unknown. Which is also why the verdict can change between
runs, and why the description says to run it twice on a heavy symbol.
 
Ayaz #:

So it is quick where the answer is already local, and slow exactly where the
answer is still unknown. Which is also why the verdict can change between
runs, and why the description says to run it twice on a heavy symbol.

yes i can confirm this. i had to run it 3 times on gbpcad. first run said no ticks, even tho i have 180 for wait time. The 2nd run's result was "real ticks" on each line. i did it a 3rd time -- and the number of ticks went up by 500 again.

Maybe there could be a setting to run it X times? with another delay/pause between each run, of course.
 
Michael Charles Schefe #:

yes i can confirm this. i had to run it 3 times on gbpcad. first run said no ticks, even tho i have 180 for wait time. The 2nd run's result was "real ticks" on each line. i did it a 3rd time -- and the number of ticks went up by 500 again.

Maybe there could be a setting to run it X times? with another delay/pause between each run, of course.
Why then not to download the full history (or up to a specific date) once and only then run audit?
 
Michael Charles Schefe #:

yes i can confirm this. i had to run it 3 times on gbpcad. first run said no ticks, even tho i have 180 for wait time. The 2nd run's result was "real ticks" on each line. i did it a 3rd time -- and the number of ticks went up by 500 again.

Maybe there could be a setting to run it X times? with another delay/pause between each run, of course.
Michael, that is the same thing from the other side, and thank you for
putting a number on it. Three runs on GBPCAD, first one saying no ticks with
a 180 second wait, second one reading real ticks on every line, third one
adding another 500. Nothing changed at the broker between those runs. The
terminal was still filling its cache, and the script was reporting what it
had at that moment rather than what exists.
The 180 seconds is worth explaining. That wait only covers the warm-up before
the first request. It does not cover the months themselves, and a month that
is not held locally is fetched when it is asked for, which on a heavy symbol
takes minutes rather than seconds. So a long wait at the start does not save
you from a short read later.
Your suggestion of running it X times with a pause is the right instinct, and
it is close to what Stanislav is pointing at above. Rather than repeating the
whole audit, the cleaner shape is one pass that asks for each month once and
throws the data away, purely to fill the cache, and then the audit pass
reading from a warm cache. Same effect, one pass instead of three, and the
second pass is fast because everything is local by then. That is what goes on
the list for the next version.
 
Stanislav Korotky #:
Why then not to download the full history (or up to a specific date) once and only then run audit?

Stanislav, the reason it is not already built that way is that there is no
separate download step to call. Asking for the ticks is the download. There
is no MQL5 function that says fetch this symbol's tick history without also
handing the data back, so a download pass and a read pass are the same call.
Doing it as one request over the whole span runs into size. On this terminal
a single month of gold real ticks measured 15.2 million ticks, and the cache
files for a heavy symbol run 83 to 124 MB per month. A CopyTicksRange over
several years would be asking for an array on that scale in one go.
But the shape you are describing does work if the first pass discards what it
receives. One call per month, oldest to newest, ignoring the return value and
only noting that it completed, then the audit reading from a warm cache. That
is a download phase and an audit phase in the way you mean, without ever
holding more than one month at a time, and it also removes the retry logic
that is in there now. It is a better design than what I wrote and it is going
on the list.

12
New comment