Strategy tester, DLLs, and FreeLibrary()..

New comment
 
I have seen this issue mentioned here and there, especially by 7bit, but I was never able to get around this problem:

The strategy tester does not properly decrement the reference count of the loaded library during deinit(). Because of this, the DLL file stays loaded in the system, and crashes when you attempt to run the tester again.

I tried to use GetModuleHandle() and FreeLibrary(), but that did not seem to solve the problem.

if (IsTesting()) {
int dll_handle = DLLHandle("C:\Program Files (x86)\MT\experts\libraries\SQLBridgeV2.dll");
Print(dll_handle);
if (dll_handle>0) FreeLibrary(dll_handle);
}
CORRECTION:

dll_handle returns a non-0 number, and FreeLibrary decrements it.

If I check GetModuleHandle() again, I get 0 - which should mean the library is unloaded.

However if I start the tester again it still crashes.

This is getting pretty annoying because I cannot use the optimization feature.

What exactly happens? Is it that the DLL's global variables remain initialized and polluted by the last run's data? Or does an error occur during the attempt to initialize a DLL over an already-initialized DLL?
 
mfurlend:
If I check GetModuleHandle() again, I get 0 - which should mean the library is unloaded.

I'd use something like procexp to check that the DLL really is getting unloaded.


[...] and crashes when you attempt to run the tester again.

There's no reason why a well-written DLL should crash just because it's being retained in memory. If it's not possible to do consecutive strategy tests with the DLL being held in memory between them, then the DLL probably isn't safe for most real-life uses because it is likely to encounter problems if you try to run more than one copy of your EA at once on different charts.

What exactly happens? Is it that the DLL's global variables remain initialized and polluted by the last run's data?

Yes, if the DLL is being held in memory, then any global variables are going to be retained. (A prime candidate for causing the crash is going to be something like allocating a handle for a connection to a database; storing that in a global variable; freeing the connection handle but without null-ing the global variable; and then subsequently trying to re-use the dead handle because the global variable is not null.)

However, it's again the case that if there are global variables which can be "polluted by the last run's data", then you are likely to have problems with using the DLL simultaneously from multiple copies of an EA in live trading. Even if you don't intend to run more than one copy of your EA per instance of MT4, it still doesn't sound as though the structure of the DLL is ideal.

 

Regarding the crash this must definitely be the fault of your DLL. Your DLL should be written in such a way that it can be loaded as many times as you want and also be used by an arbitrary number of threads in parallel. The easiest way to achieve this is that you DON'T use any global variables in your DLL. If you need to remember state or allocated resources between two function calls then allocate memory on the heap for it during init() and let the EA remember the pointer for you (and pass this pointer to the dll on every function call). on win32 you can simply cast the pointer to an int and treat it like a handle, this might produce compiler warnings about portability problems but you wont ever port it to a different architecture so this is perfectly ok. On deinit() you free the memory again.

***

Regarding the unload problem: The only real problem that arises when the dll is not unloaded by the tester is that during development you cannot compile and replace the dll with a new version while it is still loaded. For this you either need to restart MT4 before copying the new dll or manually unload it after testing.

I have never succeeded in making an automatic solution that really always works under all circumstances. I noticed the odd behavior regarding dll unloading in strategy tester and tried the appoach with manually unloading in the deinit() function but this seemed such an ugly and hackish solution to me that I stopped experimenting with it before I had a working solution, I finally simply used the kill MT4, copy DLL, restart MT4 approach (with the help of a small batch file during DLL development)

If you really intend to solve the unload problem once and for all (AFTER you fixed your DLL and did what I wrote in the first paragraph) then the next thing you could try is to write a small separate pogram that monitors and displays the reference count (and also lets you manually increase and decrease it) and use this to find out what MT4 is actually doing. Maybe the behavior of MT4 is intentional and itsomehow remembers that it did not unload it and does not load it again on the next tester run (maybe they tried to optimize the backtesting speed although this would be a totally idiotic way to do it but who knows, everything is possible given the overall quality of this software and the strange priorities and crazy ideas of its developers)

Once you have explored and fully understood the exact behavior under all possible circumstances it might be possible to find a way to automatically work around this unload problem without restarting MT4.

 
It looks like deinitializing dlls correctly on deinit() plus using FreeLibrary in case of testing unloads the dll correctly.

The problem is that if you try to run the expert again, the dll will not be loaded again and it won't work.

Fortunately recompiling the expert will suffice. This is not the way everyone would like to do things, but it's at least better than shutting down the platform and restarting it.

 

 #import "kernel32.dll"

int GetModuleHandleA(string lpString);

int FreeLibrary(int hModule);

// int LoadLibraryA(string lpString); // this won't work

#import


 .....

 


void support_deinit()

{

   // invoke destructors

   support_deinitialize();

   

   // do your mql crap cleanings ...

   if (IsTesting())

   {

      int HMOD = GetModuleHandleA("expert.dll");

      if (HMOD != 0) FreeLibrary(HMOD);

   }

}


HTH.

 
Thanks!  I was looking all over for this.  This works.
New comment
Reason: