Access Violation After GET Webpage
Bizarre, reproducible behaviour with Access Violation, related to GET of a webpage, but with a twist.
This code snippet is the basic case:
Webroot and GetWebPage are, for the purposes of this initial description at least, what one would expect from their names: they provide the root of the HTTPS query URL, and do the HTTPS GET.
The value from the query returned in licence is, for these tests, always the same, "ERROR: Not Registered [932]", and is exactly as expected.
The bizarre behaviour is actually reliably reproducible in a way that shows that the underlying functionality of those two methods works as expected (as far as this demonstration goes).
With the code exactly as above, the result is a valid (expected) outcome with the following:
>>> ERROR ERROR: NOT Registered! [932]
*** ERROR!
By merely commenting out the last line (return(false);), an Access Violation occurs. Here are the two cases (UNcommented, followed by Commented):
I recognise there is a memory leak flagged, but that started only AFTER the access violation manifested and I started adding debugging code.
Since the Prints above when completing without the accvio are as expected, and since the Print of "*** ERROR?????! HUH!!" never happens, it's clear the code never reaches there ... which means the following return(false) after that is also never reached.
What could possibly be going on???!
Thanks!
I understand. I'm looking for ideas from experienced EA developers. My code base is 12K+ lines, so not really feasible to post the whole thing, and even the relevant method is several hundred lines. It also contains proprietary code, as the extracts imply, related to licences.
I am hoping someone has seen something like this before and can offer suggested approaches. Frankly, it is bizarre that a line of code that never executes in the given example should cause an Access Violation when commented out.
I have seen this in a number of different examples of similar code. It seems to relate to compiled code emission in the vicinity of WinAPI calls to network methods. Interestingly, introducing additional code above the point of accvio that has zero relevance to the actual code (nothing in common) sometimes seems to "prevent" the accvio. In one case, simply adding several new string variable decalarations before the actual point of the accvio and calling some methods with those strings as parameters "stopped" the issue. That really is bizarre!
I've been a professional software developer for decades, so this is "interesting", but speaks of a compiler code emission problem rather than a "simple" logic or "improper" usage issue.
I'm aware of other forum posts re accvios and network methods (using NULL correctly, etc) and have no issues in that respect. This is proven by the fact the run with the UNcommented line works correctly and entirely as expected.
So, ideas appreciated. Even ones that relate to ways to further debug/identify the problem would be very much appreciated.
Thanks.
I understand. I'm looking for ideas from experienced EA developers. My code base is 12K+ lines, so not really feasible to post the whole thing, and even the relevant method is several hundred lines. It also contains proprietary code, as the extracts imply, related to licences.
I am hoping someone has seen something like this before and can offer suggested approaches. Frankly, it is bizarre that a line of code that never executes in the given example should cause an Access Violation when commented out.
I have seen this in a number of different examples of similar code. It seems to relate to compiled code emission in the vicinity of WinAPI calls to network methods. Interestingly, introducing additional code above the point of accvio that has zero relevance to the actual code (nothing in common) sometimes seems to "prevent" the accvio. In one case, simply adding several new string variable decalarations before the actual point of the accvio and calling some methods with those strings as parameters "stopped" the issue. That really is bizarre!
I've been a professional software developer for decades, so this is "interesting", but speaks of a compiler code emission problem rather than a "simple" logic or "improper" usage issue.
I'm aware of other forum posts re accvios and network methods (using NULL correctly, etc) and have no issues in that respect. This is proven by the fact the run with the UNcommented line works correctly and entirely as expected.
So, ideas appreciated. Even ones that relate to ways to further debug/identify the problem would be very much appreciated.
Thanks.
My further comments:
• As mentioned, I suspected it was a compiler issue, and potentially from optimisation, so thanks for that additional information that seems to support that
• I will "experiment" with code content to try to find some random "incantation" to avoid specific Access Violations
• Obviously, that is a most regrettable approach to be forced to take
To the MetaTrader 5 Platform Development Team:
• Thanks for your work to continually improve and enhance MT5
• Particular thanks for your work to harden the emitted code to reduce the risk of reverse engineering for the purposes of Intellectual Property theft, with further work in that area most welcome
• Work to optimise and harden code is essential, critical development work; but it must not come at the cost of disruptive failure from errors in executable code that demonstrably should not occur
• Source code that compiles differently based on the presence or absence of comments should never happen
• Currently, the MQL5 compiler seems to generate code based on code elements that are either irrelevant (comments, formatting, etc) or through optimisation can erroneously vary the execution from the programmed logic. This makes programs non-deterministic! That is the antithesis of all programming theories and recognised programming practices!
• I strongly urge the team to ensure that all compiler code emitted performs deterministically as per the logic of the source code, and that EA developers are not subjected to a lottery of potential Access Violations or other aberrant behaviour.
Thanks.
I hate to say it, but I'm rapidly beginning to believe that the MQL5 compiler (with optimisation?) is fairly frequently emitting faulty code.
Here's a new example of the inconsistencies of its behaviour.
Here's a simple method:
#import "advapi32.dll" int GetUserNameA(uchar &lpBuffer[], int &nSize); #import //+------------------------------------------------------------------+ //| WhoIsThis | //+------------------------------------------------------------------+ string WhoIsThis() { string who = ""; int size = 64; uchar user[]; ArrayResize(user,size); if (GetUserNameA(user,size)) { who = CharArrayToString(user); if (IsDebug) // Just a bool >>>>> This initial line of the if statement is referred to as "IfTest" Print(MsgHdr(strDebug),"Windows login username: ",who); // MsgHdr(strDebug) just returns a simple string >>>>> This whole Print statement is referred to as "PrintWithHdr" Print("Windows login username: ",who); // >>>>> This whole Print statement is referred to as "PrintNoHdr" } else { Print(MsgHdr(strError),"Anonymous login!"); // Never executed in the current circumstances } return(who); }
I include that method in a .mqh file with a lot of other code (1K+ lines). It is used as part of setting the value of a string with other elements. If I comment just it out, all works as expected without it.
I run it with the three statements annotated in the code above as "IfTest", "PrintWithHdr" and "PrintNoHdr" either Commented out or not, in the order as specified in the table that follows, with results as follows. (Note that the output lines for STARTED, COMPLETED, and "Validating" come from the associated code of the complete TEST EA, invoking the larger method which includes WhoIsThis.)
2024.01.23 11:33:41.776 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.23 11:33:41.776 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} Validating MGT5 licence ... 2024.01.23 11:33:42.375 MGT5 TESTS ONLY (AUDUSD,M1) Windows login username: Testuser 2024.01.23 11:33:42.375 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1003} === MGT5 TESTS ONLY has COMPLETED === 2024.01.23 11:34:25.036 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.23 11:34:25.036 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} Validating MGT5 licence ... 2024.01.23 11:34:25.341 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1003} === MGT5 TESTS ONLY has COMPLETED === 2024.01.23 11:34:25.342 MGT5 TESTS ONLY (AUDUSD,M1) 256 bytes of leaked memory 2024.01.23 11:34:57.614 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.23 11:34:57.614 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} Validating MGT5 licence ... 2024.01.23 11:34:57.915 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1003} === MGT5 TESTS ONLY has COMPLETED === 2024.01.23 11:34:57.915 MGT5 TESTS ONLY (AUDUSD,M1) 256 bytes of leaked memory 2024.01.23 11:35:27.461 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.23 11:35:27.461 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} Validating MGT5 licence ... 2024.01.23 11:35:27.765 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1003} >>> Windows login username: Testuser 2024.01.23 11:35:27.765 MGT5 TESTS ONLY (AUDUSD,M1) Windows login username: Testuser 2024.01.23 11:35:27.765 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1004} === MGT5 TESTS ONLY has COMPLETED === 2024.01.23 11:35:27.765 MGT5 TESTS ONLY (AUDUSD,M1) 1 leaked strings left 2024.01.23 11:35:59.128 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.23 11:35:59.128 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} Validating MGT5 licence ... 2024.01.23 11:35:59.431 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1003} >>> Windows login username: Testuser 2024.01.23 11:35:59.431 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1004} === MGT5 TESTS ONLY has COMPLETED === 2024.01.23 11:36:28.077 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.23 11:36:28.077 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} Validating MGT5 licence ... 2024.01.23 11:36:28.379 MGT5 TESTS ONLY (AUDUSD,M1) Windows login username: Testuser 2024.01.23 11:36:28.379 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1003} === MGT5 TESTS ONLY has COMPLETED ===
Summary:
| IfTest Used | PrintWithHdr Used | PrintNoHdr Used | Result | Nett Code Result is 1 or 2 Print statements? |
|---|---|---|---|---|
| Yes | Yes | Yes | No problem | Yes x 1 |
| Yes | Yes | No | 256 bytes leaked memory | No |
| Yes | No | Yes | 256 bytes leaked memory | No |
| No | Yes | Yes | 1 leaked strings left | Yes x 2 |
| No | Yes | No | No problem | Yes x 1 |
| No | No | Yes | No problem | Yes x 1 |
| No | No | No | 256 bytes leaked memory | No |
Essentially, it reveals that to avoid any error (leaked memory or string), I MUST have EXACTLY ONE Print statement in that area of the code!
EDIT: I did not originally include the case where all three lines were commented out, but I did the test with that in the above and included it in the table above. For the test below, it, like all of the others, worked perfectly as expected.
This is truly bizarre! Because if I extract that very same method and simply run it in its own EA (with STARTED, COMPLETED messages), and doing nothing else, the results are as follows:
2024.01.23 11:55:11.149 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.23 11:55:11.149 MGT5 TESTS ONLY (AUDUSD,M1) Windows login username: Testuser 2024.01.23 11:55:11.149 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} === MGT5 TESTS ONLY has COMPLETED === 2024.01.23 11:55:27.103 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.23 11:55:27.103 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} === MGT5 TESTS ONLY has COMPLETED === 2024.01.23 11:55:45.160 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.23 11:55:45.161 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} === MGT5 TESTS ONLY has COMPLETED === 2024.01.23 11:56:08.613 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.23 11:56:08.613 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} >>> Windows login username: Testuser 2024.01.23 11:56:08.613 MGT5 TESTS ONLY (AUDUSD,M1) Windows login username: Testuser 2024.01.23 11:56:08.613 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1003} === MGT5 TESTS ONLY has COMPLETED === 2024.01.23 11:56:24.044 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.23 11:56:24.044 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} >>> Windows login username: Testuser 2024.01.23 11:56:24.044 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1003} === MGT5 TESTS ONLY has COMPLETED === 2024.01.23 11:56:38.969 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.23 11:56:38.969 MGT5 TESTS ONLY (AUDUSD,M1) Windows login username: Testuser 2024.01.23 11:56:38.969 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} === MGT5 TESTS ONLY has COMPLETED ===
Summary: Works AS EXPECTED EVERY time!
So, unless there is some obvious issue I'm missing (always possible), my opinion is that in more than a few cases, the MQL5 compiler has SERIOUS problems with generating correct code!
Other opinions, please.
Thanks.
You might want to check this out:
Not sure what your point is, sorry.
Every variable I use is always initialised. What I have gone into detail about here is the apparently inconsistent (and therefore potentially faulty) code emitted by the MQL5 compiler.
If I have missed your point, please let me know.
Thanks.
Sadly, I am now in no doubt that the MQL5 compiler is generating faulty code is various situations. Some forty years in software engineering make this clear to me. What exactly is going on, though, is not at all clear.
The most recent evidence is in the method I have been testing all along, a 350 line method within a 610 line include file. The method returns a bool.
The situation is that a repeatable, known outcome now generates a "1 leaked strings left" runtime error. However, by the addition of a return(false); statement some 200 lines after the known exit point in the code (but still in the method), that error disappears. The added return statement is most definitely at a point way beyond where the code verifiably exits the method. Adding a Print statement immediately before that additional (redundant and inappropriate!) return statement confirms that it is not reached, as do a multitude of other possible actions along the way there that never happen! Commenting out the return statement (but leaving the trace Print) makes the "leaked string" error reappear. Interestingly, if I move the statement to the very end of the method it does not make the error disappear.
In the circumstances, this is very disturbing! It means one cannot be confident in the code generated and any errors that happen at runtime. It also may imply that some things that should cause errors do not!
I suspect the optimisation, to be honest. I have seen such faults in other compilers from time-to-time. Is there a way to disable optimisation, if for no other reason than testing this hypothesis?
Given that I've spent many days chasing down what manifested as errors, but what appear now to be artifacts of faulty compilation, I want to test ways to try to be more confident of the generated code. It will also help to identify if optimisation is indeed involved.
So, how can one disable optimisation?
And further thoughts from anyone?
Thanks.
I hate to say it, but I'm rapidly beginning to believe that the MQL5 compiler (with optimisation?) is fairly frequently emitting faulty code.
Here's a new example of the inconsistencies of its behaviour.
Here's a simple method:
I include that method in a .mqh file with a lot of other code (1K+ lines). It is used as part of setting the value of a string with other elements. If I comment just it out, all works as expected without it.
I run it with the three statements annotated in the code above as "IfTest", "PrintWithHdr" and "PrintNoHdr" either Commented out or not, in the order as specified in the table that follows, with results as follows. (Note that the output lines for STARTED, COMPLETED, and "Validating" come from the associated code of the complete TEST EA, invoking the larger method which includes WhoIsThis.)
Summary:
| IfTest Used | PrintWithHdr Used | PrintNoHdr Used | Result | Nett Code Result is 1 or 2 Print statements? |
|---|---|---|---|---|
| Yes | Yes | Yes | No problem | Yes x 1 |
| Yes | Yes | No | 256 bytes leaked memory | No |
| Yes | No | Yes | 256 bytes leaked memory | No |
| No | Yes | Yes | 1 leaked strings left | Yes x 2 |
| No | Yes | No | No problem | Yes x 1 |
| No | No | Yes | No problem | Yes x 1 |
| No | No | No | 256 bytes leaked memory | No |
Essentially, it reveals that to avoid any error (leaked memory or string), I MUST have EXACTLY ONE Print statement in that area of the code!
EDIT: I did not originally include the case where all three lines were commented out, but I did the test with that in the above and included it in the table above. For the test below, it, like all of the others, worked perfectly as expected.
This is truly bizarre! Because if I extract that very same method and simply run it in its own EA (with STARTED, COMPLETED messages), and doing nothing else, the results are as follows:
Summary: Works AS EXPECTED EVERY time!
So, unless there is some obvious issue I'm missing (always possible), my opinion is that in more than a few cases, the MQL5 compiler has SERIOUS problems with generating correct code!
Other opinions, please.
Thanks.
Thanks, Dominik.
As previously mentioned, the code is part of a large proprietary project, and the specific area of code is quite sensitive. I would, however, be willing to work with you, perhaps online together to demonstrate and to try to diagnose any issue or create code that can reproduce the problem. Would that be possible?
MT5 Version: 5.00 build 4153, 22 Jan 2024.
Windows 10 Pro 22H2, build 19045.3930 with Windows Feature Experience Pack 1000.19053.1000.0. [32 GB memory, Intel Core i7-3920XM CPU @ 2.90GHz, 3.10 GHz]
Thanks.
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
Bizarre, reproducible behaviour with Access Violation, related to GET of a webpage, but with a twist.
This code snippet is the basic case:
Webroot and GetWebPage are, for the purposes of this initial description at least, what one would expect from their names: they provide the root of the HTTPS query URL, and do the HTTPS GET.
The value from the query returned in licence is, for these tests, always the same, "ERROR: Not Registered [932]", and is exactly as expected.
The bizarre behaviour is actually reliably reproducible in a way that shows that the underlying functionality of those two methods works as expected (as far as this demonstration goes).
With the code exactly as above, the result is a valid (expected) outcome with the following:
By merely commenting out the last line (return(false);), an Access Violation occurs. Here are the two cases (UNcommented, followed by Commented):
2024.01.22 14:29:35.241 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.22 14:29:35.241 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} Validating MGT5 licence ... 2024.01.22 14:29:35.241 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1003} --- MGT5 SerA: 12345678901234567890123456789012 2024.01.22 14:29:37.938 MGT5 TESTS ONLY (AUDUSD,M1) >>> ERROR ERROR: NOT Registered! [932] 2024.01.22 14:29:37.938 MGT5 TESTS ONLY (AUDUSD,M1) *** ERROR! 2024.01.22 14:29:37.938 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1004} === MGT5 TESTS ONLY has COMPLETED === 2024.01.22 14:29:37.939 MGT5 TESTS ONLY (AUDUSD,M1) 256 bytes of leaked memory 2024.01.22 14:30:04.629 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1001} === MGT5 TESTS ONLY has STARTED === 2024.01.22 14:30:04.629 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1002} Validating MGT5 licence ... 2024.01.22 14:30:04.629 MGT5 TESTS ONLY (AUDUSD,M1) TESTING {0.1003} --- MGT5 SerA: 12345678901234567890123456789012 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) Access violation at 0x00000200803266DD write to 0x000001FFFFFFFFF8 in 'C:\Users\user\AppData\Roaming\MetaQuotes\Terminal\D0E8209F77C8CF37AD8BF550E51FF075\MQL5\Experts\MGT5 TESTS ONLY.ex5' 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) crash --> 00000200803266DD FF4A04 dec [rdx+0x04] 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 00000200803266E0 7516 jnz 0x00000200803266F8 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 00000200803266E2 48B9109E6BEB0002 mov rcx, 0x200EB6B9E10 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 0000 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 00000200803266EC 48B82064B496F67F mov rax, 0x7FF696B46420 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 0000 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 00000200803266F6 FFD0 call rax 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 00000200803266F8 83BC243005000000 cmp dword ptr [rsp+0x530], 0x00 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 0000020080326700 48C784247C040000 mov qword ptr [rsp+0x47C], 0x00 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 00000000 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 000002008032670C C784247804000000 mov dword ptr [rsp+0x478], 0x00 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 000000 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 0000020080326717 7430 jz 0x0000020080326749 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1) 00: 0x00000200803266DD 2024.01.22 14:30:05.580 MGT5 TESTS ONLY (AUDUSD,M1)I recognise there is a memory leak flagged, but that started only AFTER the access violation manifested and I started adding debugging code.
Since the Prints above when completing without the accvio are as expected, and since the Print of "*** ERROR?????! HUH!!" never happens, it's clear the code never reaches there ... which means the following return(false) after that is also never reached.
What could possibly be going on???!
Thanks!