MT5 and speed in action - page 63
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
Dear developers, could you please inform me how MQL_MEMORY_USED is calculated?
I made a calculation of memory that all EA variables occupy.
It is less than 10%. If I understand it correctly, MQL_MEMORY_USED contains the History cache and CopyTicks cache. But it is still much less.
At the same time, the parallel Expert Advisor consumes several times less. But the principle is the same.
In general, what is included to this value?
I have saved a template with Expert Advisor and applied it to the same chart by causing reloading. I have seen it like this.
The memory usage has changed by almost an order of magnitude. So far, it is difficult to explain what it means.
Many programs have cumulative effect of memory usage. This is a sin of browsers and sometimes even Word. The Terminal may also be guilty of it. Besides, all logs are written by default and it's easy to spend a week if you have too many actions in one giga. This is an evil which must be managed. But it is not clear how.
Maybe you know how to programmatically select a financial instrument and not get hung up for ages ?
Through an indicator
Forum on trading, automated trading systems and trading strategy testing
MT5 and Speed in Action
Renat Fatkhullin, 2020.10.14 04:15
For mass ticking put more memory.
4gb (price €20) is no good in 2020 when it comes to analytics and research.
For Market Products this approach is no good anywhere. We have to bypass the 10-second retention of unnecessary data through such a crutch.
Making a trivial Market Product in the form of a Market Screener is actually not an achievable task for MT5 due to excessive memory consumption.
Making a trivial Market product in the form of a Market Screener is actually not an achievable task for the MT5 due to excessive memory consumption.
The terminal eats so much memory after launching.
After execution of the screener it started to eat 2 gigs (TERMINAL_MEMORY_USED and did not decrease with time). This is with only one open chart for 5000 M1-bars.
Did not save a screenshot. But decided to throw in an example, which shows memory eating Terminal not in itself, where it's just stupid.
The script just makes copies of the original symbols from the Market Watch. I was supposed to add all the symbols on MQ-Demo and run this script first time on cold and then again on hot.
And after the hot run (when the ticks are already on SSD in the form of tkc-files) I will observe enormous memory exhaustion where it's not needed in a proper implementation.
However, running the script, I've encountered that it hangs on MQ-Demo. It can be unloaded only through Abnormal termination, after which the Terminal will not release more than 1 GB of memory.
It's easy to compare this screenshot with the one in the beginning.
Sorry, not sure if this is a bug or a feature. I haven't found an answer on my own. But the question is related to performance and I suppose it's better to ask it here.
If I add, for example, 22 buffers of DRAW_SECTION type to an empty indicator, then at startup of such an indicator for one chart containing 1000000 bars or more the terminal lags significantly (it causes significant CPU load) and eats significant amounts of memory, even if the indicator calculates nothing.
My interest was aroused by the fact that if you use buffers with other types, everything works without problems and such slowdown is not observed.
For example, I executed the following code on a single chart with 1000000 bars and it consumed about 500 MBytes and lags, especially the chart itself.
But if I change the buffers type to, say, DRAW_LINE, the load on the processor is reduced dramatically, lags are not observed, and memory consumes 5 times less (about 100 MBytes consumed).
Similar tests were done on different builds, up to 2019 builds and the situation is the same.
An individual citizen pulled outa duplicate of an invaluable load of MQL code from his wide trousers, which showed that under the same conditions the crutch worked faster than the regular function.
A very simple test:
20 Expert Advisors on EURUSD, i.e. all OnTick processes simultaneously. The build of the terminal is 2664. The test is running without optimization, compilation and other additional load at 100% CPU - you are not going to run a real "hft" trading on this background, are you?
Typical test log:
A very simple test:
20 EAs on EURUSD, i.e. all OnTick processes at the same time. Terminal build 2664. The test is running without optimization, compilation and other additional workload on 100% CPU - you are not going to run a real "hft" trading on this background, are you?
Typical test log:
You create hothouse conditions by doing 100K iterations for 1.5 seconds on the same tick. I, on the other hand, purposely waited for ticks that didn't generate an OnTick.
Looking through my trade logs, I notice the SymbolInfoTick execution within a few milliseconds. And I know for 100% that the CPU was at full Idle at that time.
It's very simple. There are conditionally 1 million ticks per day. Even 0.01% of ticks slowdown is 100 ticks per day with lags. You'll say that's nonsense. I'll say it's bad. If I run into a lag when I need to make an order, it's a potential monetary loss.
Very grateful that this branch doesn't go unnoticed, and on this feature in particular, a lot of work has been done to speed things up. However, it was a bit of a surprise to me that the awful crutch could outperform the regular function in certain situations. And perhaps if you sorted it out and eliminated that lag, 100 potential lags per day would turn into 10.
I say it's much better than it was at the beginning of the thread. But the fact remains that there are times when it's not good. And I can see that you don't want to look into it. I respect your choice.
Above cited the snapshot option of getting current prices. It would suit me completely if it didn't catch millisecond lags on the Idle-CPU machine.
I'm also worried not only about SymbolInfoTick's speed, but also about the relevance of the prices it returns. I see that you don't raise this question. Apparently you decided to look at it later.
You create hothouse conditions by doing 100K iterations within 1.5 seconds on the same tick. I, on the other hand, specifically waited for ticks that didn't spawn OnTick.
When reviewing my trade logs, I notice that SymbolInfoTick is running for a few milliseconds. I know 100% that the CPU was full Idle at that moment.It's very simple. In a day there are conditionally 1 million ticks. Even a 0.01% lag is 100 ticks per day with lags. You'll say that's nonsense. I'll say it's bad. If I run into a lag when I need to make an order, it's a potential monetary loss.
Very grateful that this branch doesn't go unnoticed, and on this feature in particular, a lot of work has been done to speed things up. However, it was a bit of a surprise to me that the awful crutch could outperform the regular function in certain situations. And perhaps if you sorted it out and eliminated that lag, 100 potential lags per day would turn into 10.
I say it's much better than it was at the beginning of the thread. But the fact remains that there are times when it's not good. And I can see that you don't want to look into it. I respect your choice.
Above cited the snapshot option of getting current prices. It would suit me completely if it didn't catch millisecond lags on the Idle-CPU machine.
I'm also worried not only about SymbolInfoTick's speed, but also about the relevance of the prices it returns. I see that you don't raise this question. Apparently you decided to look at it later.
These are not warm conditions at all. A loop for 100000 price requests without Sleep() and the like and in 20 threads simultaneously is an obvious stress test. Nothing like that should be even close in real conditions.
If you think there are no other ticks coming in 1.5 seconds - ok, make 10 million queries, it won't change anything, your "crutch" works worse:This your statement is 100% false:
Just like this previous statement of yours:
Accessing prices via SymbolInfoTick is now very fast and it's completely decoupled from tick stream processing. You're working with a ready-made price cache, which is updated very sparingly and quickly.
All "0.01% tick rate slowdowns" only show up under stress test conditions, and that's a great result. Under normal conditions, access is guaranteed to be very fast.
If you admit that "a lot of work has been done to speed up" SymbolInfoTick, then you should believe me that the "crutch" of getting prices via PositionSelectByTicket cannot be a better solution.
For one simple reason - the implementation of PositionSelectByTicket is completely identical to the original "slow" implementation of SymbolInfoTick.
As I wrote above, this implementation is not slow in the literal sense of the word, but under heavy CPU load, it has a higher probability of runtime outliers (which is clearly visible in my last test).
The timings here are highly dependent on the system task scheduler running under load, i.e. there can be differences from OS to OS and even from version to version.
And the heavier the load, the more you test the performance of the task scheduler, not the terminal.
That's the end of the SymbolInfoTick performance topic.
If your experts create load on the level of synthetic stress tests, then there is one solution: "the iron must match the tasks".
I have a question about the relevance of the ticks given by SymbolInfoTick.
Situation:
1. We do TimeCurretn(); we get time 18:00:00
2. Do SymbolInfoTick on an invalid symbol. We get a tick with the time 17:58:00.
3. Sleep(1)
4. Adds a SymbolInfoTick for the non-left symbol. We get a tick with the time 17:59:00.
I.e., in the fourth item we have a new tick, which is one minute different than TimeCurretn().
Do you see a problem in this situation?
How to get into this situation less often ?
I have a question about the relevance of the ticks given by SymbolInfoTick.
Situation:
1. We do TimeCurretn(); we get time 18:00:00
2. Do SymbolInfoTick on an unlabeled symbol. We get a tick with the time 17:58:00.
3. Sleep(1)
4. Adds a SymbolInfoTick for the non-left symbol. We get a tick with the time 17:59:00.
I.e., in the fourth item we have a new tick, which is one minute different than TimeCurretn().
Do you see a problem in this situation?
How to get into this situation less often ?
SymbolInfoTick sends the data received from the broker server. What the server sent is what you get.
If there are questions about the tick stream which is broadcasted by your broker, then you have to contact your broker.