Did Structs Change with the last MT5 Update? - page 7

New comment
 
Doerk Hilger #:

I note your change in tone, but also noticed you mentioned “various reasons” for taking me down.

If there are actual technical reasons, I’m open to hearing them. If it’s just personal, then that’s not productive.

I don’t dismiss ideas, but they need a concrete, working implementation. If you can show a stable solution that matches or outperforms what I’ve built in real-world conditions, I’ll gladly take a serious look. Theory alone doesn’t help anyone.
Granted. But I currently have no incentive to implement this (again) in a way that suits your needs.

But what I can try to do is lay out the general concept on how it could be done and coded.

Therefore I would need to at least know the data flow model and the problematic requirement/critical section, you are worried about.
 
Doerk Hilger #:

I note your change in tone, but also noticed you mentioned “various reasons” for taking me down.

If there are actual technical reasons, I’m open to hearing them. If it’s just personal, then that’s not productive.

I don’t dismiss ideas, but they need a concrete, working implementation. If you can show a stable solution that matches or outperforms what I’ve built in real-world conditions, I’ll gladly take a serious look. Theory alone doesn’t help anyone.
I am interested too to know what is in play concretely. The discussion can become interesting only if it's more concrete. What data needs to be send through IPC ?
 
Doerk Hilger #:

I note your change in tone, but also noticed you mentioned “various reasons” for taking me down.

If there are actual technical reasons, I’m open to hearing them. If it’s just personal, then that’s not productive.

I don’t dismiss ideas, but they need a concrete, working implementation. If you can show a stable solution that matches or outperforms what I’ve built in real-world conditions, I’ll gladly take a serious look. Theory alone doesn’t help anyone.
Basically its all about locking, if the lock is reduced into the ring buffer and a smart memory management is put in place, you get exactly what I suggested initially.

The method includes a basic ring buffer. Imagine a read pointer that cannot push beyond the write pointer, now if a new dataset >has been written<, the write pointer can move forward. Any reader can maintain their own read pointer. The only atomic operation is the update to the write pointer. But even that is not (depending on the scenario) required to be atomic, since they only increase, like a counter. To make it a ring buffer, you mask out the ring buffer size from the pointer counter.

That's basically it.

Now, depending on your writers, you might want to maintain either multiple ring buffers or an atomic increment.

The buffer itself only holds pointers to memory areas with the payload.

This technique works both ways. On each connection. And it is thread safe and consistent, can use cache locality and efficient non fragmenting memory management.

EDIT:

Memory management is done via a FiLo Buffer, in case of known payload size, or via multiple FiLo buffers representing different memory area sizes. Like 1 MemPage filo-buffer, 4 MemPage buffer and so on.

This should give a very robust and simple full scale data handling system, bit like the Linux kernels new memory system, coming in one of the next releases.

The communication between the different executors/threads should be doable via a messaging system, such that this synchronization process can be handled on a side-channel.

For the payload, instead of using structs, ether you refer to an external library, like protobuf or you create your own data descriptor, iE one possible option would be to have a header in each payload that composes the "struct" in your target system (the reader) on the fly. After all, its just how the bits are being interpreted, so, what data type are they representing. There are not a lot of them, and since it almost makes no big difference between a ulong and an uint, they can be broken down to a very small common denominator.

This would reduce the size of the payload header significantly, and put more resources towards actual data handling, instead of metadata interpretation.
 

I have not been following the discussion too deeply, but I would like to offer an on-topic but 3rd party viewpoint to help it along.

I personally have been involved very superficially on-off with @Doerk Hilger's project, but it has been enough to understand what it entails and to know what difficulties he has faced regarding many of MQL5's quirks, and in this case, the lack of a proper IPC method.

With that I can say, that the methods suggested by @Dominik Egert, while theoretically plausible and practical to a degree, the truth is that when MetaTrader is overloaded, those methods can fail.

On the surface, there are many aspects of those IPC methods, that will work in MQL5 in practice in a controlled environment. However, @Doerk Hilger's project is quite intensive, and when MetaTrader is under load and stress, it does not behave as expected (i.e. as documented) and often acts erratically. That is why @Doerk Hilger is looking for a solution that can stand the test under heavy load.

I know both of your personalities to a degree and consider both of you "too proud" for your own good, but I consider both of you highly competent. So, if both of you can calm down and put your egos aside, I believe you could both gain and profit from a mutual professional relationship.

 
Fernando Carreiro #:

I have not been following the discussion too deeply, but I would like to offer an on-topic but 3rd party viewpoint to help it along.

I personally have been involved very superficially on-off with @Doerk Hilger's project, but it has been enough to understand what it entails and to know what difficulties he has faced regarding many of MQL5's quirks, and in this case, the lack of a proper IPC method.

With that I can say, that the methods suggested by @Dominik Egert, while theoretically plausible and practical to a degree, the truth is that when MetaTrader is overloaded, those methods can fail.

On the surface, there are many aspects of those IPC methods, that will work in MQL5 in practice in a controlled environment. However, @Doerk Hilger's project is quite intensive, and when MetaTrader is under load and stress, it does not behave as expected (i.e. as documented) and often acts erratically. That is why @Doerk Hilger is looking for a solution that can stand the test under heavy load.

I know both of your personalities to a degree and consider both of you "too proud" for your own good, but I consider both of you highly competent. So, if both of you can calm down and put your egos aside, I believe you could both gain and profit from a mutual professional relationship.

Am on it. He already got me with this interesting challenge.

Can you elaborate on what it is that is causing the grieving on MQL side?

So actually, what is it that MQL needs to provide consistently?

EDIT: I hope its not pride on my side, but I'll look into it. Thanks for pointing this out.

EDIT2: and how does MQL fail on delivering this consistency?
 
Dominik Egert #: Can you elaborate on what it is that is causing the grieving on MQL side? So actually, what is it that MQL needs to provide consistently?
I am unable to offer a detailed answer, because my involvement was on-off and related to side issues. @Doerk Hilger is much more qualified to answer that.
 
Fernando Carreiro #:
I am unable to offer a detailed answer, because my involvement was on-off and related to side issues. @Doerk Hilger is much more qualified to answer that.
I'll wait, thank you. And nice to have you back, btw.
 
Doerk Hilger #:

I note your change in tone, but also noticed you mentioned “various reasons” for taking me down.

If there are actual technical reasons, I’m open to hearing them. If it’s just personal, then that’s not productive.

I don’t dismiss ideas, but they need a concrete, working implementation. If you can show a stable solution that matches or outperforms what I’ve built in real-world conditions, I’ll gladly take a serious look. Theory alone doesn’t help anyone.

I would suggest, if you want to continue, we move some posts to a new thread and the following  conversation.


 
Fernando Carreiro #:

I have not been following the discussion too deeply, but I would like to offer an on-topic but 3rd party viewpoint to help it along.

I personally have been involved very superficially on-off with @Doerk Hilger's project, but it has been enough to understand what it entails and to know what difficulties he has faced regarding many of MQL5's quirks, and in this case, the lack of a proper IPC method.

With that I can say, that the methods suggested by @Dominik Egert, while theoretically plausible and practical to a degree, the truth is that when MetaTrader is overloaded, those methods can fail.

On the surface, there are many aspects of those IPC methods, that will work in MQL5 in practice in a controlled environment. However, @Doerk Hilger's project is quite intensive, and when MetaTrader is under load and stress, it does not behave as expected (i.e. as documented) and often acts erratically. That is why @Doerk Hilger is looking for a solution that can stand the test under heavy load.

I know both of your personalities to a degree and consider both of you "too proud" for your own good, but I consider both of you highly competent. So, if both of you can calm down and put your egos aside, I believe you could both gain and profit from a mutual professional relationship.

I do agree and that's why I asked to know more about what is really at play. What data, in which context, to do what, etc...?
 
Alain Verleyen #:
I am interested too to know what is in play concretely. The discussion can become interesting only if it's more concrete. What data needs to be send through IPC ?

Depends on what you wanna achieve. 

In our case, or maybe also in the interest of many professional coders, its necessary to move the most of the code to an IDE like VS with C#. Problem is, there is no API in MT and you need to create kinda your own. 

Mainly this means, you have to constantly update data which you have in MQL, but you need on the other side. Like account data, PnL, Quotes, Tickdata, etc. The list gets longer the more you wanna do outside MQL. Thats for example the reason, why this ring-buffer-thing is not the way to do. Why? Lets overdo a litte.

Imagine your have 100 open positions, 20 symbols. To manage this outside, you need to know about every position all the details, about all the symbols, the ticks, maybe last bars depending on what you wanna do on the other side. News come out. 20 symbols fire ticks same time, more than you maybe can handle, but also more than you maybe need. Do you really need every tick to update some drawings in C#? No. Do you need every tick to manage the positions? Also no, since the execution time, when closing a position is anyways always above 50 ms. It doesnt matter if you miss a tick - most of the time.

Would you use ring-buffers, every tick would go into that ring, would be stuck in queue and would not help you. Same time there is an overflow risk for such buffers. We talk only bout the server side in C# here, not yet about MQL server side. 

So what you rather need, is a mutual buffer, but still working like a pipe. The way I do it, is: Push the data to the pipe, and when the next tick comes, I check: Was it even read? If not, overwrite it. With ring buffers not possible. And here we also come to the this= thing. Such data, in structs, can be copied to the IPC buffer at once, the struct is taken from MQL via IntPtr and copied using Marshal.Copy to the same struct in C#, after it went through the IPC pipe / its buffer.

In general, you have 3 kinds of messages:

1. Push - Client sends, Server processes, one after the other

2. Request - Client sends, waits, server processes, returns results, client takes it

3. Update - like described above

NamedPipe and everything that deals with ringbuffers can only handle 1 and 2, but the 3rd is crucial for the usecase with data like ticks, but also any kind of data which is just accurate for the very moment. With classical methods, you just create your own bottleneck for nothing. 

Next problem, with classical methods is: You are restricted in the number of clients, since every client has its own instance. What if dispose didnt work for serveral times, what if clients need temporary access here and there? You will end up in a deadlock sooner or later. 

12345678910
New comment