Discussion of article "MQL Parsing by Means of MQL"

 

New article MQL Parsing by Means of MQL has been published:

The article describes a preprocessor, a scanner, and a parser to be used in parsing the MQL-based source codes. MQL implementation is attached.

Programming is essentially formalizing and automating some processes using general-purpose or special-purpose languages. The MetaTrader trading platform allows applying programming to solving a wide variety of the trader's problems, using embedded MQL language. Normally, the coding process is based on analyzing and processing application data according to the rules specified in source codes. However, a need sometimes arises for analyzing and processing the source codes themselves. Here are some examples.

One of the most consistent and popular tasks is contextual and semantic search in source code base. Surely, you can search for strings in a source code as in a normal text; however, the semantics of the sought for is getting lost. After all, in case of source codes, it is desirable to distinguish among the specifics of using the substring in each specific case. If a programmer wants to find where a specific variable is used, such as "notification," then a simple search by its name can return much more than necessary, where the string occurs in other values, such as a method name or a literal, or in comments.

A more complicated and sought for, as a rule, at larger projects, is the task of identifying code structure, dependencies, and class hierarchy. It is closely connected to meta-programming that allows performing code refactoring/improving and code generation. Recall that MetaEditor provides some opportunities of generating codes, particularly, creating the source codes of Expert Advisors using Wizard or forming a header file by source code. However, the potentials of this technology are much more powerful.

Code structure analysis allows computing various quality metrics and statistics, as well as finding the typical sources of runtime errors that cannot be detected by the compiler. In fact, compiler itself is, of course, the first tool of analyzing a source code and returns warnings of many types; however, checking for all potential errors is not usually built into it — this task is too large, so it is normally assigned to separate programs.

Moreover, parsing source codes is used for styling (formatting) and obfuscation (scrambling).

UML-Diagram of MQL Parsing Classes

Author: Stanislav Korotky

 
The topic is complex and interesting, but what could be the practical application? Creating an editor? AN AI?
 

The article is written very well, qualitatively. It is logical to assume that the continuation of lexical parsing is "semantic" parsing, about which I propose to write the next article. And after the semantic parsing, we will be a stone's throw away from creating an AI. :)

ZY. Only artificial psyche with basic intellectual functions will be left to be written.

 

Serious question for the author:

I have my own markup language. It has a set of rules and consists of keywords, which are numbers.

Most of the keywords are interleaved with string words - names of groups or elements, and there is no need to analyse these string "tokens". The MQL compiler catches errors in the names of keywords, because their defines are connected in the file.

However, the user can commit violations of the language rules when compiling the markup code. It is necessary to check the code after compilation for errors related to rules, not grammar.

Is it possible to do it based on the parsing concept described in the article, or is it necessary to create a different mechanism?

I rely on your experience.

Thank you.

 
Реter Konow:

...

Can this be done based on the parsing concept outlined in the paper, or is a different mechanism required?

...

For myself, I have chosen the following solution: scan the code (essentially an array of values) for violations of the sequence of keywords (commands) and output messages about them into the log. In principle, it is simple, because you can monitor the sequence of numbers through a set of conditions, filters and flags.

Lexical, grammatical and syntactic parsing is much more difficult, as human words have no numerical equivalent, unlike commands. Words have properties that do not have a number, and these properties are many. Lexical parsing requires working with different properties of not only words, but also phrases or punctuation. And above all this "floats" the meaning embedded in the text (context), the extraction of which may become the task of semantic parsing in the future (I'm not sure that it exists).

For my tasks, a simple control of the sequence of commands is enough.

 
In time we will try to add some well-known static analyser.

We use PVS Studio ourselves
 

Probably, using this method it is possible to write a mq4->mq5 converter for the Market:

  • An ex5-advisor is launched, which is fed mq4-code.
  • The output is mq5-code.
There are such attempts in the Market, but they are weak.

 

I post updated sources.

Ensured compilability in the latest builds.

Fixed several hard-to-find bugs.

Files:
 
Stanislav Korotky:

I'm posting the updated source code.

Replaced in the article
 
That's cool. An example of refactoring would be nice.
 

Hi, I am a relative newbie here.

Having read the article assumed the RESULT would be a search tool to look up code generated by the wizard.

Downloaded the zip file. Installed the script mql.mq5 compiled it without errors.

Running mql.ex5 produces no result on screen.

I am in learn mode. Want to be able to use the wizard to use code out of the codebase then analyze and modify to get a usable EA.

Any help would be appreciated.

Thanks