Did Structs Change with the last MT5 Update? - page 9

New comment
 
Doerk Hilger #: So would it be possible to overcome that 25ms OnTimer() restriction with an MQL service app?

I believe so, but since it has been a long while, I no longer remember the specifics of what we had discussed.

Your project has since then also evolved, so I would need to be reprised on the details and then carry out some tests to confirm.

 
Doerk Hilger #:
So would it be possible to overcome that 25ms OnTimer() restriction with an MQL service app?
If you want this to be solved reliably, future proof, a full rewrite would be required. But I am 100% confident, this can be done in a way, you would never need to fight it again.

Anyone reading this, and having other opinion, please speak up, or contribute, if Doerk agrees.


EDIT:

We provide the MQL side of it, conceptually, and the C#, you would need to adopt. But by the same schema. So nothing to complex.
 
Dominik Egert #:
If you want this to be solved reliably, future proof, a full rewrite would be required. But I am 100% confident, this can be done in a way, you would never need to fight it again.

Why would you wanna rewrite a stable solution which is already high-end in view of performance, safety and sharing-fairness?  We are already somewhere between Unix UDS level and shared-memory. Safety? We have zero losses. 

At the moment I am facing only two (possible) issues, possible, maybe partly real:

1. The restriction within OnTimer(), while is not that big deal, since it mainly affects the ping-time. Once its reached, further messages can be handled instantly after this first access - <>5000 sync messages average. Nonetheless: MQL service, if it can provide something better than OnTimer(), maybe simply while (true) { if (server.Listen()) ....  Sleep(0); } that issue would be already be solved. 

2. Still the thing with the transferred data. At the moment it also works without problem, thats why I pointed out I´d be interested to see sth which is more safe in view of possible future changes within MQL.


Edit: Did some research with local, serverless SQL  The access/read/write time for one single value is already 1000 times more than the current solution needs for the whole struct.What am I missing here?

 
Doerk Hilger #:

Why would you wanna rewrite a stable solution which is already high-end in view of performance, safety and sharing-fairness?  We are already somewhere between Unix UDS level and shared-memory. Safety? We have zero losses. 

At the moment I am facing only two (possible) issues, possible, maybe partly real:

1. The restriction within OnTimer(), while is not that big deal, since it mainly affects the ping-time. Once its reached, further messages can be handled instantly after this first access - <>5000 sync messages average. Nonetheless: MQL service, if it can provide something better than OnTimer(), maybe simply while (true) { if (server.Listen()) ....  Sleep(0); } that issue would be already be solved. 

2. Still the thing with the transferred data. At the moment it also works without problem, thats why I pointed out I´d be interested to see sth which is more safe in view of possible future changes within MQL.


Edit: Did some research with local, serverless SQL  The access/read/write time for one single value is already 1000 times more than the current solution needs for the whole struct.What am I missing here?

Yeah, a service will solve your OnTimer issue, dont use an SQL, use a shared memory, like you (probably) already do.

For struct stability, you would need to either use a generic approach, or do a data dislocation approach. Both could be done, but introduce a copy step in any case.

Otherwise, dont use structs, but arrays of data rows. Like one array for open, one for close and so on.

Then use a descriptor struct to point you to the right location to get back the full struct you have in C#. (References as members)

EDIT:
Since they work via a pointer, an offset could be used, reducing the struct size. But I guess that's actually overkill in your use-case.
 
Dominik Egert #:
Yeah, a service will solve your OnTimer issue, dont use an SQL, use a shared memory, like you (probably) already do.
I totally like the SQL Memory idea, but unfortunately its at least factor 100 in view of access time. 

Besides this I handle already plenty of data within keyvaluepair-arrays and for 50-100 vars I have a timing of around 0,03 ms - for all. Thats aleady pretty good and used for data which is not that critical. 

But when it came to transferring tick data, i developed the struct transfer protocol. Nonetheless, Im still concerned about changes in MQL. Whenever they decide to any kind of alignment within structs, im doomed. 
 
Doerk Hilger #: Nonetheless, Im still concerned about changes in MQL. Whenever they decide to any kind of alignment within structs, im doomed. 

All the tick-data fields can in general all be converted to "long" data-types, and so you could transfer it as a simple array of long and not depend on "structs".

EDIT: The price fields can either be "union-ed" between double and long, or they can be scaled (divided by point size) to become true integer based values.

 
Doerk Hilger #:
Whenever they decide to any kind of alignment within structs, im doomed.
I'm not very knowledgeable about this and will probably say something stupid. But what if you explicitly set the alignment for your structures?
 
Vladislav Boyko #:
I'm not very knowledgeable about this and will probably say something stupid. But what if you explicitly set the alignment for your structures?
Its not stupid, but exactly what we are suggesting.

There are multiple ways of doing so, question is which is the most efficient.


 
Basically, you can use a transformer, copy from MqlStruct to your own. This way, you ensure the data format.

But that's most probably an inefficient way of handling data.

If I am not wrong, MT has internal arrays of prices, meaning all open prices are in a continuous memory, same for high, low and so on. So it is much more efficient to keep this around, because the CPU does this much better than jumping around in memory.

Tick data is differently organized. It seems to me it is stored in structs of MqlTick. This can be seen when calling very frequently CopyTicks. Whenever new ticks come in, a certain amount of last ticks cannot be received via CopyTicks due to memory locking, and CopyTicks will give less ticks than in a call before, missing some of the most recent, previously run ceived ticks.

An algorithm is required to keep all ticks up to date and have them be in order. To get away from the MqlTick structure a transformer into your own struct could be used.

For ChartGetXXX, you need some clever code to do caching, because these functions can take forever to execute. Problem here is to know when to update your cache. This can be done in an asynchonus service, providing these values to your app.

Reading out position data is either polling very quickly and presenting the data, or your own implementation of calculating all the values. Latter option is faster, but much more work to code.

All of this would now go into a shared memory area as result, with a well defined layout, such that your app can read any data at any time.

For the RPC aspect, I suggest to use a message system with return result handling. Could be done with a fifo buffer for the requests, and a hash map for the return results. As one way to go. Could also be done with an RPC + CallBack method. Or with two Fifo buffers. Depending on how you want to use it.

For the case of ChartGet and ChartSet, it could be done asynchronous.

Until now, I cannot see any mutex or semaphore requirements. All can be done with atomic increments and ring buffers. (BTW, ring buffers can be dynamic in size without the need for modulo operators. Thus very efficient, just use pow2 in size.)

Most of this doesn't even require polling-wait, it could all be done via on demand and processing loops, or an update signal, or a message in the messaging queue, to inform your app of an update.
 
Vladislav Boyko #:
I'm not very knowledgeable about this and will probably say something stupid. But what if you explicitly set the alignment for your structures?

Any factual comments are welcome, thank you. 

Of course, can be done like this. I even practice manual alignment, also in C#. That means, the struct itself is packed bytewise, but the data inside is organized in 4 byte alignment nonetheless. No bool, just int/uint, ulong/long, double. Nothing else. 

I should have rather written: " Whenever they decide to any kind of very special alignment within structs, im doomed.". Means, what if they decide to add a header before the first field or sth like that. You never know. 

12345678910
New comment