Download MetaTrader 5

dll multithreading

To add comments, please log in or register
xx2
50
xx2  

After the last unsolved mystery of MT4 tester peculiarities here comes the next. Wonder if forum gurus could take a look and give their wise opinions.

After noticing some promising posts and links here on FFT and SSA and Alglib I decided to try myself as well. So, I integrated Alglib to my dll. Got also the commercial version which allows you to have Intel MKL (math & kernel library) easily included to boost the performance with multithreading. Got it running and it gave the same results as the free version. But there was no speedup in tester run or optimization. However, if I integrate their test code as such to my init routine I WAS able to see that when increasing the n (in matrix multiplication) and worker (how many cores included) amount the time spent (measured now with QueryPerformanceCounter) went down. This is again the same result I saw earlier with TBB. Whatever I try with multithreading no overall speedup can be gained!!!

So, if there's ANYONE on this forum who can tell pure facts how they got the overall tester execution (or optimization) faster with mulithreading I'd be greatly appreciate. I have seen discussions and reports but most of those don't show overall speedup. And I'm not interested in MT5 tester capabilities. I'd only want to know why mt4 tester cannot be speeded up with multithreading. Or if you have successully made CUDA integration (or OpenCL) and got it faster with that please let me and us know the numbers.

I used SSA from Alglib on three timeframes to give entry points.   

Anthony Garot
828
Anthony Garot  
Mikko Siltanen:

After the last unsolved mystery of MT4 tester peculiarities here comes the next. Wonder if forum gurus could take a look and give their wise opinions.

After noticing some promising posts and links here on FFT and SSA and Alglib I decided to try myself as well. So, I integrated Alglib to my dll. Got also the commercial version which allows you to have Intel MKL (math & kernel library) easily included to boost the performance with multithreading. Got it running and it gave the same results as the free version. But there was no speedup in tester run or optimization. However, if I integrate their test code as such to my init routine I WAS able to see that when increasing the n (in matrix multiplication) and worker (how many cores included) amount the time spent (measured now with QueryPerformanceCounter) went down. This is again the same result I saw earlier with TBB. Whatever I try with multithreading no overall speedup can be gained!!!

So, if there's ANYONE on this forum who can tell pure facts how they got the overall tester execution (or optimization) faster with mulithreading I'd be greatly appreciate. I have seen discussions and reports but most of those don't show overall speedup. And I'm not interested in MT5 tester capabilities. I'd only want to know why mt4 tester cannot be speeded up with multithreading. Or if you have successully made CUDA integration (or OpenCL) and got it faster with that please let me and us know the numbers.

I used SSA from Alglib on three timeframes to give entry points.   

My understanding of MT4 is that it's limited to a single core.

So, what does it mean to have multi-threading limited to one core? Context switching, sure, but no overall improvement of speed because the threads are not running in parallel.

Now, if your DLL can spawn to the other hardware cores, please disregard my rudimentary comment because I don't fully understand your setup.

xx2
50
xx2  
Anthony Garot:

My understanding of MT4 is that it's limited to a single core.

So, what does it mean to have multi-threading limited to one core? Context switching, sure, but no overall improvement of speed because the threads are not running in parallel.

Now, if your DLL can spawn to the other hardware cores, please disregard my rudimentary comment because I don't fully understand your setup.

yes it can and as I said it was proven. But why no overall speedup? Let's ask then the other way around ... is there anybody else admitting to have failed in trying it? 

Icham Aidibe
11450
Icham Aidibe  
Why do you use alglib as a dll ?
xx2
50
xx2  
Icham Aidibe:
Why do you use alglib as a dll ?

SSA was not in mql version. At least in the one I saw at first. Also, MSVC can produce more efficient and faster code.

Icham Aidibe
11450
Icham Aidibe  
Mikko Siltanen:

CPU load on taskmanager details page for MetaTrader is always 13 whatever you do ...

With TBB i got it higher but that's all, testing/optimization time didn't go down.

Task affinity settings dont make a difference though people on different threads might suggest those.

You can run multiple instances of MetaTrader yes and sometimes it works, sometimes it just makes trouble for you.

You solve trouble after trouble and almost forget the first trouble and end up nowhere. Nothing accomplished. Is this what this is?

Expert didn't work, you try next, and the next, wait here's something which could work, let's try that. And always it doesn't. Backtesting shows nice results and again forward testing doesn't.

I'd been developing these experts now for over two years and haven't made a cent out of it.

And my personality doesn't suit for manual trading. Too many feelings and those lead me to make wrong decisions so stopped doing that.

As a sw engineer with a lot of experience I'm supposed to succeed when wrinting code, right? But certainly not on this field. 

Are we doing this just to learn to code or why? if you learn to code something there's always something behind the corner you don't know. And then you find it from stackoverflow. Great. Just integrate and proceed to the next learning. Which will get you again nowhere.

Enough words to get more comments to get this solved ? Or at least a get-a-life comment ..

Another glass of wine at least for me. 

I love wine.

SSA & TBB, dude, you're working on the EA of the millenium !

So what I suggest - it's worth what it's worth - is to first code something very standard, something poor, basic but inescapably profitable. 

I'm sure you already know how to do it. 

Only after, once remunerated, you''ll be free to pursue your researches.

See it as a self financed project. 

aslkdjf
25
aslkdjf  

You could use opencl. Here's an opencl lib by dingmaotu. 

https://github.com/dingmaotu/mql4-lib/tree/master/OpenCL

Icham Aidibe
11450
Icham Aidibe  
Mikko Siltanen:

thanks for encouraging words. it's just that i don't believe anymore that such and inescapably profitable system exists, regardless whether its poor, simple, complex or whatever.

Have been optimistic for such for quite long but i guess i have to learn something at some point. Just hope I could train my brain like I seem to be able to train these artificial neural networks (which is not easy either). 

It's more rewarding usually to try to make something real work or work better (like this multicore-multithread thing). But on this field the universe is against you.

Precisely. I did work on neural networks with alglib. When it comes to dll, you're directly penalized by the portability, so I did it with the alglib available for mql5.

What I'm sure of - because I did it - is : 

  1. Or your neural network is pre-trained on 10+ years for a pair, OHLC + 10 indicators, hardcoded & run - there's plenty of that kind on the market (IP*)
  2. Or you want to self train it, where you'd require memory for calculation, but data selection should be the neatest the better - I mean you won't do it with OHLC price + the value of 10 indicators, and 10 neurons on the fly, with or without opencl, with or without dll, at each tick
  3. With the last 10 bars of M1, there's enough to do a very nice scalper
  4. Some may notices that neural network doesn't make the grail, you might want to give a try with fuzzy logic - it works differently - I worked a little on it, I got this, I'm sure I did not exploit the full potential but the advantage on neural network is that you won't need plenty of data for analysis : 

It's still in my task list, because it abuses of so much ordermodify/ordersend/orderdelete - which is extremely heavy for mem - i feel like it's random orders ! I definitively have to refine the algo.

Icham Aidibe
11450
Icham Aidibe  
Mikko Siltanen:

Interesting. Thanks for sharing. I'm using fann library - didn't use the mql2fann but used fann lib directly from my dll. Don't have any issues with performance as I could get forecasts from SSA only for new bar, couldn't replace the last which would blow the perf requirements in ctrl point and every tick testing (both for SSA and fann). But maybe it's no problem since I can do any amount of smaller timeframes anyway. Could try less than 1M but only if this would work on higher timeframes (and 1M). 

Can you show what kind of performance you were able to get in forward testing? 

Fuzzy logic - no idea - totally fuzzy for me. How did you applied it?

Everyone of course have their own issues but also in many cases it's easy to make wrong conclusions what takes time and CPU or mem. In my case I have not seen any issues with those orderhandling functions. I don't have any indicators included mainly because the interface to iCustom is so poor, maybe it's better in mql5 but didn't go there. I've moved SMI to dll and use that along with angles for order closing (and actually also for entries). 

Sure. Here's the forward test 1/2 from 2017.01.01 'til now.

 

It's stable. 

It's not well documented in mql, in finance generally talking so I got to watch out c/c# example to understand well how it could be applied and I said previously even if profitable I'm not satisfied on how it process. I leave it parked, test flight not for now.

To add comments, please log in or register