Discussion of article "Universal Expert Advisor: the Event Model and Trading Strategy Prototype (Part 2)"

[MetaQuotes]

New article Universal Expert Advisor: the Event Model and Trading Strategy Prototype (Part 2) has been published:

This article continues the series of publications on a universal Expert Advisor model. This part describes in detail the original event model based on centralized data processing, and considers the structure of the CStrategy base class of the engine.

The article contains further description of the universal trading engine CStrategy. In the first article Universal Expert Advisor: Trading Modes of Strategies (Part 1), we have discussed in detail trading modes and functions that allow implementing them. We have analyzed a universal Expert Advisor scheme consisting of four methods, two of which open new positions and the other two methods close them. Different combinations of method calls define a particular trading mode. For example, an Expert Advisor can be allowed only to Sell or to Buy, can manage previously opened positions or wait. Using these modes, an Expert Advisor operation can be flexibly set up depending on the trading time or day of the week.

However, the trading modes are not the only thing that an Expert Advisor may need. In the second part we will discuss the event model of the CStrategy trading mode based on the centralized event handling. The proposed event handling scheme differs from the system events in that all the events are gathered in one place. The advantages of such an implementation will be considered later.

Also, this article describes two important classes of the trading engine — CStrategy and CPosition. The first one is the core of the whole EA trading logic, it unites events and modes into a single flexible framework that the custom EA inherits directly. The second class is the basis of universal trading operations. It contains actions applied to an open position (like closing of a position or modification of its Stop Loss or Take Profit). This allows formalizing all trading actions and making them platform-independent.

Access to Events Occurring on Other Instruments, the MarketEvent Structure

When designing a trading system which analyzes multiple symbols, you need to create a mechanism that can track changes in the prices of multiple instruments. However, the standard OnTick function is only called for a new tick of the instrument the Expert Advisor is running on. On the other hand, trading system developers may use the OnBookEvent function that responds to changes in the order book (Depth of Market). Unlike OnTick, OnBookEvent is called for any change in the order book of the instrument, to which you subscribed using the MarketBookAdd function.

Changes in the order book happen very often, that is why monitoring this event is a resource-intensive procedure. As a rule, monitoring changes in the tick stream of the required symbol is enough for Expert Advisors. On the other hand, the event of the order book change also includes the arrival of a new tick. Apart from OnBookEvent, you can set up calls of OnTimer at specified intervals and analyze price changes of multiple symbols in this function.

So in the system functions that react to NewTick, BookEvent and Timer events, you can add a call of some intermediate module (let's call it EventProcessor), which would simultaneously analyze changes in prices of multiple instruments and generate an appropriate event. Each event would have a unified description in the form of a structure and would be sent by the control methods of the strategy. Having received an appropriate event as a structure, the strategy would either react to it or ignore. In this case, the system function which actually initiated the event for the final Expert Advisor would be unknown.

Indeed, if an Expert Advisor receives a notification of a new incoming tick, it does not matter whether the information is received through OnTick, OnTimer or OnBookEvent. The only thing that matters is that there is a new tick for the specified symbol. One event handler can be used for many strategies. For example, if each strategy is represented as a custom class, multiple instances of these classes can be stored in the special list of strategies. In this case, any strategy from the list will be able to receive a new event generated by EventProcessor. The following diagram shows how events are generated and sent:

Fig. 1. Diagram of event generation and sending

Author: Vasiliy Sokolov

[Eugene Myzrov]

Thank you to the author. The material of the article is extremely useful and also excellently presented. I have long wanted to put my approaches to expert writing in order, and here the author of the article has already gone through all the pains of creativity and offers a ready-made solution. Super! I look forward to the continuation.

Let's hit the roadlessness of trading systems automation, tortures of creativity and developers' slackness with the " trading engine"!

[Ivan Negreshniy]

Thanks to the author for the informative article, but I have some doubts about the reliability of the software implementation.

In your proposed code, all event handler functions refer to the common event_type variable or to the m_event structure in a common instance of CStrategy.

But it can be dangerous, because events occur asynchronously and threads of their handlers can overlap in time, chaotically changing common data or causing failures when accessing the same memory areas.

Usually, thread synchronisation is performed using semaphores, mutexes, wait-functions, critical code sections, etc., but MQL has no such means, so it is obvious that in functions-handlers of asynchronous events it is necessary to minimise references to global instances of objects and variables, trying to use local ones.

On this basis, the combined event handler and data block proposed in the article, IMHO, is not a good idea.

[Vasiliy Sokolov]

Eugene Myzrov:
Thank you to the author. The material of the article is extremely useful and also excellently presented. I have long wanted to put my approaches to expert writing in order, and here the author of the article has already gone through all the pains of creativity and offers a ready-made solution. Super! I look forward to the continuation.

Let's hit the roadlessness of trading systems automation, the agony of creativity and developers' slackness with the "trading engine"!

Thank you!

Ivan Negreshniy:

Thanks to the author for the informative article, but I have some doubts about the reliability of the software implementation.

In your proposed code, all event handler functions refer to the common event_type variable or to the m_event structure in a common instance of CStrategy.

But this may be dangerous, because events occur asynchronously and their handlers' threads may overlap in time, chaotically changing common data or causing failures when accessing the same memory areas.

...

Thanks for the interesting remark. Indeed, events in MetaTrader are asynchronous. However, the execution of user threads is strictly sequential. It means that other events will not be processed until the Expert Advisor finishes processing the current event. The code below illustrates this point:

//+------------------------------------------------------------------+
//|TestAsynch.mq5 |
//|Copyright 2015, Vasiliy Sokolov. |
//|http://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2015, Vasiliy Sokolov."
#property link      "http://www.mql5.com"
#property version   "1.00"
bool need_check = true;
#define  EVENT_CHECK_ASYNCH 1
//+------------------------------------------------------------------+
//| Expert tick function|
//+------------------------------------------------------------------+
void OnTick()
  {
//---
   if(need_check)
   {
      EventChartCustom(ChartID(), EVENT_CHECK_ASYNCH, 0.0, 0.0, "Asynchrony check");
   }
   if(need_check)
      printf("The user thread is terminated first.");
  }
//+------------------------------------------------------------------+
//| ChartEvent function|
//+------------------------------------------------------------------+
void OnChartEvent(const int id,
                  const long &lparam,
                  const double &dparam,
                  const string &sparam)
  {
//---
   if(id == CHARTEVENT_CUSTOM+EVENT_CHECK_ASYNCH && need_check)
   {
      printf("Then the OnChartEvent event is called.");
      need_check = false;
   }
  }
//+------------------------------------------------------------------+

Its execution produces lines in the following sequence:

2016.01.15 13:51:31.724 TestAsynch (NZDUSD,W1)  Потом вызывается событие OnChartEvent
2016.01.15 13:51:31.722 TestAsynch (NZDUSD,W1)  Сначала завершается пользовательский поток

Thus, there are no errors caused by the joint change of the m_event shared variable, because the code is executed sequentially. I should also note that the structure itself is private and is accessible only by reference to special methods. Therefore, its actual location inside CStrategy does not matter, the structure can easily be created directly in event handler methods and also passed to strategy instances.

[Ivan Negreshniy]

Vasiliy Sokolov:

Thank you!

Thanks for the interesting observation. Indeed, events in MetaTrader are asynchronous. However, the execution of user threads is strictly sequential. This means that other events will not be processed until the Expert Advisor finishes processing the current event. The code below illustrates this point:

Its execution produces lines in the following sequence:

Thus, there are no errors caused by the joint change of the m_event shared variable, because the code is executed sequentially. I should also note that the structure itself is private and is accessible only by reference to special methods. Therefore, its actual location inside CStrategy does not matter, the structure can easily be created directly in event handler methods and also passed to strategy instances.

You are right, strictly sequential execution of event handlers can eliminate errors, but this is essentially a single-threaded mode where it is difficult to achieve high performance.

At the same time, there are time-critical tasks that require parallelisation, for example, calculations with large volumes of price and historical data performed with OpenCL or more specific tasks that are relevant for me - training of neural networks on different symbols and periods performed with the help of multithreaded engine described in my article.
https://www.mql5.com/en/articles/706

[Vasiliy Sokolov]

Ivan Negreshniy:

You are right, strictly sequential execution of event handlers can eliminate errors, but this is essentially a single-threaded mode where it is difficult to achieve high performance.

At the same time, there are time-critical tasks that require parallelisation, for example, calculations with large volumes of price and historical data performed with OpenCL or more specific tasks that are relevant for me - training of neural networks by different symbols and periods, performed with the help of the multithreaded engine described in my article.
https://www.mql5.com/en/articles/706.

Parallelisation in MetaTrader is done by the multi-threaded strategy tester. Thus, if you run a strategy written with the help of the described engine in the strategy tester, it will automatically become multithreaded. Since there are no third-party dlls, you can use MQL Networl Cloud to its full potential. What you write about refers to the MetaTrader itself. Yes, the engine, as well as any Expert Advisor or script in MetaTrader, is single-threaded, but the optimiser allows you to perform multi-threaded and cloud computing of the presented code.

As for the engine performance itself, everything is fine here. You can make sure of it by performing profiling. Infrastructure and event delivery costs are minimal. The main execution time is concentrated in calls of system functions and custom methods BuyInit, SellInit, BuySupport, SellSupport. Based on profiling calculations, the engine is capable of handling many dozens of events per second, redirecting them effortlessly to multiple strategies simultaneously.

What is offered with the article, I apologise, is not finger-pointing. It is a verified and tested solution for several years. All calls and internal procedures are optimal and important. The algorithms are tested in the profiler. There are no restrictions on the internal organisation of user code. In general, there is everything for complete happiness. Take it and use it.

[Mikhail Tkachev]

Many thanks to Vasiliy Sokolov for his efforts to promote good programming style in general and OOP in particular.
The article is clearly stated and very useful, at least for MQL beginners, which I am. :)
Please answer the following question:
In my trading algorithms, positions by market orders of Buy/Sell typeare opened in special situations - if the price has already passed the corresponding level or before the end of the trading session.
Generally positions are opened/closed on up/down level breakdown (for Long) by placing BuyStop/SelStop stop orders, which are recalculated at the beginning of a new bar.
Could you please suggest how it would be possible to introduce placement and change of stop orders in the described classes ?

[Vasiliy Sokolov]

Mike:
Many thanks to Vasiliy Sokolov for his efforts to promote good programming style in general and OOP in particular.
The article is clearly stated and very useful, at least for MQL beginners, which I am. :)
Please answer the following question:
In my trading algorithms, positions by market orders of Buy/Sell typeare opened in special situations - if the price has already passed the corresponding level or before the end of the trading session.
Generally positions are opened/closed on up/down level breakdown (for Long) by placing BuyStop/SelStop stop orders, which are recalculated at the beginning of a new bar.
Could you please suggest how I could introduce placing and changing stop orders in the described classes ?

Greetings. At the moment algorithms for managing pending orders are being worked out, but there is no concrete implementation in the current version of the engine yet. The only thing I can suggest at the moment is to place the code for placing pending orders in the BuyInit and SellInit methods. It is also necessary to enumerate pending orders there and all those actions that usually have to be done in regular Expert Advisors.

In the future it is planned to add methods for processing pending orders to the engine and make their corresponding description. They will work similarly to position management methods.

[Mikhail Tkachev]

Vasiliy Sokolov:

Greetings. At the moment algorithms for pending orders management are being worked out, but there is no concrete implementation in the current version of the engine. The only thing I can suggest at the moment is to place the code for placing pending orders in BuyInit and SellInit methods. It is also necessary to enumerate pending orders there and all those actions that usually have to be done in regular Expert Advisors.

In the future it is planned to add methods for processing pending orders to the engine and make their corresponding description. They will work similarly to position management methods.

Thank you, Vasily, we will be waiting. :)

[Viktor Vlasenko]

Vasily, thank you so much for the article(s)! Just great and very useful for a beginner!

[Mohammed Abdulwadud Soubra]

thanks

*****