MT5 and speed in action - page 17

 

Note that HistorySelect makes a physical snapshot of the selected history to the EA's local cache, so that you can go through them without fear. Without this you can get very unpleasant effects, as the account history is updated/synchronised asynchronously. Not to mention the fact that the user himself can manually change the depth of history from the interface.

Hence such copying costs. All the more so if you deliberately try to force copying of this history into the cache simultaneously from multiple threads.

We've already optimized a lot in sampling operations and now we're thinking about optimal cache update, when in reality 99% of samples will be completely useless and will be skipped on the fly.

That is, unless you specifically randomise the sampling limits, the cache will show hits close to 100%.

Most likely next week there will already be an effective solution.

 
Renat Fatkhullin:

Note that HistorySelect makes a physical snapshot of the selected history to the EA's local cache, so that you can go through them without fear. Without this you can get very unpleasant effects, as the account history is updated/synchronised asynchronously. This is not to mention the fact that the user can manually change the depth of history himself from the interface.

There is a table of orders and a table of deals. Why would we need to make physical snaps, when knowing the four indexes at the moment of HistorySelect call?

That's why the copying costs are so high. Especially if we specifically deal with simultaneous forced copying of this history into the cache from multiple threads.

We've already optimized a lot in sampling operations and now we're thinking about optimal update of cache, when in reality 99% of samples will be completely useless and will be skipped on the fly.

That is, unless you specifically randomise the sampling limits, the cache will show hits close to 100%.

Most likely next week there will already be an effective solution.

HistoryDealSelect and HistoryOrderSelect now destroy the relevant samples. Why isn't there, as in MT4, access to both tables by indexes? Introduce new functionality.

Forum on trading, automated trading systems and trading strategies testing

Bugs, bugs, questions

fxsaber, 2020.08.20 11:28

New in-house functions.
int OrderExist( const string symbol, ENUM_ORDER_TYPE type, ulong magic, ulong &tickets[] );

int PositionExist( const string symbol, ENUM_POSITION_TYPE type, ulong magic, ulong &tickets[] );

The indexes are asking for it. I don't understand why i should know the number of trades in my account through one place.

 

And these tables can change at any time. So can individual records in them.

No one can guarantee their immutability due to asynchronous operations, synchronisation processes and manual depth modes set by users.

As I wrote above, we'll apply intelligent caching techniques, which will reduce the cost of Select functions to zero. Unless, of course, you specifically randomise the sampling limits. The last date can be changed and it won't invalidate the cache if it's always in the future/last time. Recent transactions will be sparingly added to the cache.


In MT4 it works the same way, just cache creation is hidden. On every OnTick/OnStart of MT4, the terminal automatically and sparingly creates a snapshot of the market environment for each EA.

Therefore, you cannot assess the true latency of data preparation from the MQL4 code. Fortunately, in MT4 the data is small and simple.


In MT5 we gave up the costs of automatic creation of market environment to reduce delays and not to do unnecessary work. Instead we have given full cost control to robot developers who can request exactly what they need.

Note that the size of the market environment in MT5 is huge compared to MT4 - it simply cannot be replicated.

With your tests you have taken advantage of one of these costly samples.

We will fix that - it is in our interest.

 
Renat Fatkhullin:

And these tables can change at any time. So can individual records in them.

No one can guarantee their immutability due to asynchronous operations, synchronisation processes and manual depth modes set by users.

Are you saying that before the already last trade, another trade may appear in the trading history? If a trade has changed, another. But to wedge inside an already existing list - I haven't noticed that.

 

OrderExist and PositionExist are special optimised functions that avoid stupid looping through all orders or positions in search of entries by symbol, transaction type and medzhik.

The result is an array of tickets.


Programs may save a lot of money using these functions. Especially when calling open positions and orders in bulk, constantly and repeatedly in overshoot loops.

We will implement more effective functions to access massive trade data in the future.

The language will also be dramatically strengthened and simplified with more powerful functionality.

 
fxsaber:

Are you saying that there may be another trade in the trading history before the already last trade? If a deal has changed, another. But to wedge inside an already existing list - I haven't noticed that.

Theoretically, yes.

Don't forget about the synchronization processes. A huge number of processes in the platform are asynchronous.

For example, a gateway integration with an exchange or liquidity provider may send transaction reports with delays of seconds or even minutes. Often the api does not provide access to history for reconciliation at all, but provides slow and non-rhythmic report generators.

At market opening, or due to an unexpected gateway reconnect, reports may be delayed. They are replicated to the history on the server and immediately asynchronously sent to the terminals. Because of the sorting by date, they are inserted in the right places, and in the meantime you can open new trades.

Most of the integration APIs are so illiterate and dysfunctional that they almost make it impossible to make guaranteed gateways. Although there is an opinion that this is a product of deliberate sabotage by their developers.

 
Renat Fatkhullin:

OrderExist and PositionExist are special optimised functions that avoid stupid looping through all orders or positions in search of entries by symbol, transaction type and medzhik.

The result is an array of tickets.


Programs may save a lot of money using these functions. Especially when calling open positions and orders in bulk, constantly and repeatedly in overshoot loops.

We will implement more effective functions to access massive trade data in the future.

The language will also be dramatically strengthened and simplified with more powerful functionality.

Renat, a big request to have access to the quotes outside of TERMINAL_MAXBARS when using Copy... functions. At least just the minute timeframe. You can also make a separate function for this.
But to have access to this data you must always set TERMINAL_MAXBARS to unlimited (moreover, it overloads the terminal), which is very inconvenient because you do not need unlimited for all symbols.
As far as I understand, if you copy all little MN1 period bars, all M1 bars will still be downloaded, but there is no access to them.
Of course, you can download all ticks, because they are not subject to this restriction, but it is more time-consuming and unfortunately ticks don't always coincide with the entire minute history.

 
Nikolai Semko:

Renat, a big request to have access to quotes outside of TERMINAL_MAXBARS when using Copy... functions. At least just the minute timeframe. You can also make a separate function for this.
But to have access to this data you must always set TERMINAL_MAXBARS to unlimited (moreover, it overloads the terminal), which is very inconvenient because you do not need unlimited for all symbols.
After all, as far as I understand, if you copy all little MN1 period bars, all M1 bars will be downloaded anyway, but there is no access to them.
Of course, you can download all ticks because they are not subject to this restriction, but it is more time-consuming and unfortunately ticks do not always coincide with the entire minute history.

No, history outside of these limits is not picked up and checked for synchronisation. Moreover, there is no place to store them (we won't build additional invisible caches), we won't climb the disk in non-economical modes and won't build crutches.

Generally speaking, this is a harmful idea as developers will immediately start writing absolutely inefficient algorithms and advise traders to "set 5000, or better 1000 bars" and accuse us of slowness and ineffectiveness.

We have purposely allowed to control the amount of resources for charts. It's 64 bits and memory is cheap now - use the appropriate settings within the rules and architecture.

We will not change this behaviour.

 
Renat Fatkhullin:

No, history outside these limits is not raised and checked for synchronisation. Moreover, there is no place to store them (we won't build additional invisible caches) and we won't climb over the disk in non-economical modes.

In fact, it is a harmful idea, as developers will immediately start writing absolutely inefficient algorithms and advise traders to "set 5000 or better 1000 bars" and accuse us of slowness and ineffectiveness.

We have purposely allowed to control the amount of resources allocated to charts. It's 64 bits and memory is cheap now - use the appropriate settings within the rules and architecture.

We won't change this behaviour.

Ok. Got it. Thank you. Crossed out.
Though, of course, I'd like to argue about uneconomy. I'll have to leave unlimited and as a result, all open (for example I have 14 charts at the moment) will keep all the history, although I only need 5000 for most of them. Which on the contrary will be more uneconomical.
Also I don't need this data to be stored. I will take care of the storage. I've initiated the loading of all minute bars, got them into an array and I don't need it anymore and all caches can be deleted if their size exceeds TERMINAL_MAXBARS. So maybe that's what a separate function that doesn't store caches is for.

Although, of course, I agree that naughty hands can suspend the system with such periodic loads.

 

You only have two states 5000 and an unlimited?

You are the mastermind of your own happiness.

Reason: