Possible Preprocessor Bug

[Dominik Egert]

Possible error in the preprocessor stage of the compiler.

For some reason, the preprocessor keeps substituting the same macro over and over again, though, it should stop after first substitution.

To narrow down the issue, all relevant code is provided here.

Concerning the macro ___DBG_MSG_VAR()

Expected behaviour would be, and this was verified with g++ compiler. The parameter passed to the macro ___DBG_MSG_VAR() should only expand once. Even if it is another macro, which it is in this case.

But the result shows multitude of substitutions of the same macro, creating a substitution loop, which is in contrast to g++ compiler for sure not the expected or wanted behaviour.

Here is what actually is produced by the preprosessor stage:

printf("%*s%s", trace_call_depth * 2, "", StringFormat("  " + "  " + "%s >%s< %s(){ @%i: %s }", "",
   __FILE__, __FUNCTION__, __LINE__, var_out(strip_mql_api_trace(
   "dbg_mql_api_retval.msg(StringFormat(%*s%s,trace_call_depth*2,,StringFormat(  +  +%s >%s< %s(){ @%i: %s },((()==)?:()),
   __FILE__,__FUNCTION__,__LINE__,((StringFormat(%s => %s() %35s,MQL5-API Function,TimeCurrent,%s))))),datetime)*TimeCurrent()"),
   (dbg_mql_api_retval.msg(StringFormat("%*s%s", trace_call_depth * 2, "", StringFormat("  " + "  " + "%s >%s< %s(){ @%i: %s }", ((("") == "") ? "" : ("")),
   __FILE__, __FUNCTION__, __LINE__, ((StringFormat("%s => %s() %35s", "MQL5-API Function", "TimeCurrent", "%s"))))), "datetime") *
   dbg_mql_api_retval.msg(StringFormat("%*s%s", trace_call_depth * 2, "", StringFormat("  " + "  " + "%s >%s< %s(){ @%i: %s }", ((("") == "") ? "" : ("")),
   __FILE__, __FUNCTION__, __LINE__, ((StringFormat("%s => %s() %35s", "MQL5-API Function", "TimeCurrent", "%s"))))), "datetime") *
   dbg_mql_api_retval.msg(StringFormat("%*s%s", trace_call_depth * 2, "", StringFormat("  " + "  " + "%s >%s< %s(){ @%i: %s }", ((("") == "") ? "" : ("")),
   __FILE__, __FUNCTION__, __LINE__, ((StringFormat("%s => %s() %35s", "MQL5-API Function", "TimeCurrent", "%s"))))), "datetime") *
   TimeCurrent()), (StringLen(StringFormat("%*s%s", trace_call_depth * 2, "", StringFormat("  " + "  " + "%s >%s< %s(){ @%i: %s }", ((("") == "") ? "" : ("")),
   __FILE__, __FUNCTION__, __LINE__, ((""))))) - 2))));

And this is the result from the g++ compiler, using the -E option.

printf("%*s%s", trace_call_depth * 2, "", StringFormat("  " + "  " + "%s >%s< %s(){ @%i: %s }", "",
   __FILE__, __FUNCTION__, __LINE__, var_out(strip_mql_api_trace("TimeCurrent()"),
   (StringLen(StringFormat("%*s%s", trace_call_depth * 2, "", StringFormat("  " + "  " + "%s >%s< %s(){ @%i: %s }", ((("") == "") ? "" : ("")),
   __FILE__, __FUNCTION__, __LINE__, ((""))))) - 2))));

I have aligned the outputs, it is relevant to scroll all the way to the right to see the difference in results.

Anyways, the code produced by the MQL-Preprocessor lets the program crash harshly, resulting in:

Access violation at 0x00000273C7E21008 read to 0xFFFFFFFFFFFFFFFF in 'LibDebug.ex5'

Additionally, I noticed the macro-operator # for stringifying, does not work properly, as it removes all '"' instead of escaping them properly, also whitespaces are being removed, giving false results in the outcome.

As a note:

It was quite difficult to track down this issue, as there is very limited way of debugging macros in the first place, and MQL-Compiler not providing output of the produced results of the preprosessor-stage makes things even harder.

I tried best by applying the stringify-macro to the end-results, but this still could influence the interpretation of the results, as to above mentioned issue with the stringify-operator.

This (assumed) bug in the preprocessor stage of the compiler hinders me on finishing the project, which can be found here on code-base:

• https://www.mql5.com/en/code/32071

EDIT:

I have updated the code for better readability... - I hope it is good enough

And here is a stripped version of the embedded issue at hand:

    static class pass_through
    { public:
        string _type;
        string _msg;
        pass_through() : _msg (NULL)                            { };
        pass_through(const pass_through& s)                     { _msg = s._msg; _type = s._type; }
        pass_through* msg(const string in, const string t_in)   { 
                                                                    _type = t_in; 
                                                                    _msg = in; 
                                                                    ResetLastError(); 
                                                                    return(GetPointer(this)); 
                                                                }
        template <typename T>
        T operator*(T in)                                       {
                                                                    const int err_no = GetLastError();
                                                                    const string _typename = typename(T);
                                                                    printf("%s; %s", 
                                                                        _msg, 
                                                                        (err_no != ERR_SUCCESS) ? StringFormat("Error: %i", err_no) : "");
                                                                    _msg = NULL;
                                                                    return(in);
                                                                }
    } dbg_mql_api_retval;

    
    template <typename T>
    const string    var_out(const string name, T val, const int shift = 0, const string prefix = "", const int offset = 0)
    { return(StringFormat("%s[xxx]  %-" + IntegerToString(60 - ((offset < 60) ? offset : NULL)) + "s = '%s'", prefix, name, typename(val))); }


void OnInit()
{
    #define ____DBG_MSG_VAR(x)                          printf("   " + "   " + "%s>%s< %s(){ @%i: %s }", "", __FILE__, __FUNCTION__, __LINE__, var_out(#x, (x), 0))
    #define TEST_DBG_MSG_MQLFUNC_RETURN(x, y)           dbg_mql_api_retval.msg(x, y) *
    #define TimeCurrent                                 TEST_DBG_MSG_MQLFUNC_RETURN("TimeCurrent", "datetime") TimeCurrent
    ____DBG_MSG_VAR(TimeCurrent());


}

[Fernando Carreiro]

Due to text scrolling off screen, your comments text is difficult to follow, especially the example code.

So, I have reformatted your post and copied those comments to regular text

Please edit and correct them if I did not do them proper justice

[Dominik Egert]

Fernando Carreiro #:

Due to text scrolling off screen, your comment text are difficult to follow, especially the example code.

So, I have reformatted your post and copied those comments to regular text

Please edit and correct them if I did not do them proper justice

Well, yes, I suppose thats much better. - Thank you for the effort.

I myself edited the preprocessor outputs already, as it was way to much text on one line. 

Its difficult to keep track of all the brackets and "capsules" inside that one line of code.

[William Roeder]

Dominik Christian Egert #: Well, yes, I suppose thats much better.

Are your books one column but two feet wide? No because that is unreadable. They are six (6) inches, sometimes two columns, so you can read it easily. So should be your code. I'm not going to go scrolling (or moving my eyes) back and forth trying to read it. Don't copy and paste code, write self documenting code. Please edit your post and you might get additional help.

[Carl Schreiber]

Hmm,

the possibilities of #define should simplify the code and make it easier to read, you push it so far (too far?) that the opposite happens.

Besides, you could actually find the errors quite easily with search and replace.

[Fernando Carreiro]

Dominik Christian Egert #: Well, yes, I suppose thats much better. - Thank you for the effort. I myself edited the preprocessor outputs already, as it was way to much text on one line. Its difficult to keep track of all the brackets and "capsules" inside that one line of code.

You can use the "\" character to break-down your macros into multi-line and make them more readable. Here is an example:

// Define macro for error status
   #define MReportError( _text )                \
      PrintFormat( "%s — Error %d: %s (%s,%d)", \
         g_sProgramName, _LastError, _text,     \
         __FUNCSIG__, __LINE__ )

[Dominik Egert]

Carl Schreiber #:

Hmm,

the possibilities of #define should simplify the code and make it easier to read, you push it so far (too far?) that the opposite happens.

Besides, you could actually find the errors quite easily with search and replace.

I understand.

I don't see or find an error within my code, though.

The error is produced by the preprocessor stage of the MQL compiler.

If I write out the code properly, everything works as expected, but if I wrap it up into macros, the resulting code is faulty.

That's not by me doing something wrong, but by the preprocessor working differently than expected, or given by the expectancy if it behaving like a C/C++ compilers preprocessor.

Since this issue is inside the library, you are exposed to the inner guts of that library. In contrast, when using the library, everything is very neat and organized as well as readable, as it should be for macros.

[Dominik Egert]

William Roeder #:

Are your books one column but two feet wide? No because that is unreadable. They are six (6) inches, sometimes two columns, so you can read it easily. So should be your code. I'm not going to go scrolling (or moving my eyes) back and forth trying to read it. Don't copy and paste code, write self documenting code. Please edit your post and you might get additional help.

I will try to make it more readable, but please do not expect wonders, as it is quite complex anyways.

Hope to still get support from your side.

[Dominik Egert]

Fernando Carreiro #:

You can use the "\" character to break-down your macros into multi-line and make them more readable. Here is an example:

I know and I will do so, in the hope it gets more readable.

[Alain Verleyen]

Dominik Christian Egert:

Possible error in the preprocessor stage of the compiler.

For some reason, the preprocessor keeps substituting the same macro over and over again, though, it should stop after first substitution.

To narrow down the issue, all relevant code is provided here.

Concerning the macro ___DBG_MSG_VAR()

Expected behaviour would be, and this was verified with g++ compiler. The parameter passed to the macro ___DBG_MSG_VAR() should only expand once. Even if it is another macro, which it is in this case.

But the result shows multitude of substitutions of the same macro, creating a substitution loop, which is in contrast to g++ compiler for sure not the expected or wanted behaviour.

Here is what actually is produced by the preprosessor stage:

And this is the result from the g++ compiler, using the -E option.

I have aligned the outputs, it is relevant to scroll all the way to the right to see the difference in results.

Anyways, the code produced by the MQL-Preprocessor lets the program crash harshly, resulting in:

Access violation at 0x00000273C7E21008 read to 0xFFFFFFFFFFFFFFFF in 'LibDebug.ex5'

Additionally, I noticed the macro-operator # for stringifying, does not work properly, as it removes all '"' instead of escaping them properly, also whitespaces are being removed, giving false results in the outcome.

As a note:

It was quite difficult to track down this issue, as there is very limited way of debugging macros in the first place, and MQL-Compiler not providing output of the produced results of the preprosessor-stage makes things even harder.

I tried best by applying the stringify-macro to the end-results, but this still could influence the interpretation of the results, as to above mentioned issue with the stringify-operator.

This (assumed) bug in the preprocessor stage of the compiler hinders me on finishing the project, which can be found here on code-base:

• https://www.mql5.com/en/code/32071

EDIT:

I have updated the code for better readability... - I hope it is good enough

I would have similar request to what I already asked you in your library topic.

You provided a code with 1500 lines, a lot of them being commented. I opened it, check 2 minutes and I am discouraged, how could I know I will not waste 1 hour of my time reading, trying to understand, etc... to just discover that you made an error and it's actually not a preprocessor bug ? You compared to g++ preprocessor but who said mql5 preprocessor should work the same as g++ one ? This is for sure not the case and that's not bug but MQL limitations.

If you want MetaQuotes to consider it you would need to narrow down that more. As least it's my opinion, I can't help like it is currently, sorry.

[Dominik Egert]

Alain Verleyen #:

I would have similar request to what I already asked you in your library topic.

You provided a code with 1500 lines, a lot of them being commented. I opened it, check 2 minutes and I am discouraged, how could I know I will not waste 1 hour of my time reading, trying to understand, etc... to just discover that you made an error and it's actually not a preprocessor bug ? You compared to g++ preprocessor but who said mql5 preprocessor should work the same as g++ one ? This is for sure not the case and that's not bug but MQL limitations.

If you want MetaQuotes to consider it you would need to narrow down that more. As least it's my opinion, I can't help like it is currently, sorry.

Understood. I will try to break it down and see.

My assumption it behaves like a C/C++ compiler is this page from the docs.

Especially paragraph 2.

https://www.mql5.com/en/docs/basis

The fact that the code works on VC++ and g++ and produces expected results, has let me assume, it is a bug.

As said, I will try to break it down even more. Although it is already down to one line and a comparison with a different compiler.

I understand, the effort is to much to track all the code provided.

Most of it is for giving all details. Because that's what usually is asked. All relevant code.

I will do my best to narrow it down.