Tips from a professional programmer (Part II): Storing and exchanging parameters between an Expert Advisor, scripts and external programs

Malik Arykov | 4 June, 2021

Introduction

In this article, we are going to discuss parameters which can be restored after terminal restart (shutdown). All examples are real working code segments from my Cayman project.

Parameter storage locations

Parameter examples

Zero bar time. For example, when detecting a candlestick pattern, it is logical to evaluate it once, after the emergence of a new bar on a given timeframe.
Trading level parameters. For example, you may select a trading level and use a script to set the time and size of a deal to be opened in case of a level breakout. The script passes parameters to the Expert Advisor. The Expert Advisor creates a level analyzer. The analyzer "turns on" only after the emergence of a new bar on the specified timeframe.
User preferences. These include color, trading rules, drawing methods and other parameters. Obviously, such parameters should be installed once, for example, in a set file.

Where to store these parameters?

Terminal global variables
Graphical objects
Order comments
Text files

Storage	Type	Scope	Lifetime
Terminal global variables	double	All charts	4 weeks after the last call
Graphical objects	Any. Strings <= 63 characters	Current chart	Chart lifetime
Order comments	Strings <= 23 characters	All charts	Terminal lifetime
Text files	Any. Unlimited	All charts	File lifetime

Terminal global variables

The terminal global variables are available from any chart. Their scope can be limited by including additional components to the variable name, such as ChartId, Symbol, or Period. What cannot be changes is the variable type. You cannot save the text.

There is a lifehack: pack/unpack integer values. As you know, double takes up 8 bytes (64 bits). Please check the following example: it shows how to store multiple integer values in one variable. The most important thing is to determine the bit size of their maximum values.

// -----------------------------------------------------------------------------
// Example of packing/unpacking integer values to/from a global variable       |
// using bitwise operations                                                    |
// -----------------------------------------------------------------------------
void OnStart() {
    
    int     value10 = 10; // max = 255 (8 bits)
    int     value20 = 300; // max = 65535 (16 bits)
    bool    value30 = true; // max = 1 (1 bit)
    
    // pack the values into 25 bits (8+16+1)
    // 39 bits (64-25) remain free
    ulong packedValue = 
        (value10 << 17) + // reserve space (16+1) for value20, value30
        (value20 << 1) + // reserve space (1) for value30
        value30;
    
    // save the global variable
    string nameGVar = "temp";
    GlobalVariableSet(nameGVar, packedValue);
    
    // read the global variable
    packedValue = (ulong)GlobalVariableGet(nameGVar);
    
    // unpack the values
    // 0xFF, 0xFFFF, 0x1 - bit masks of maximal values
    int value11 = (int)((packedValue >> 17) & 0xFF);
    int value21 = (int)((packedValue >> 1) & 0xFFFF);
    bool value31 = (bool)(packedValue & 0x1);
    
    // compare the values
    if (value11 == value10 && value21 == value20 && value31 == value30) Print("OK");
    else PrintFormat("0x%X / 0x%X /0x%X / 0x%X", packedValue, value11, value21, value31);
}

Graphical objects

Can you store script parameters in graphical objects? Why not. Set the object property OBJPROP_PRICE = 0 — in this case the object is visually "hidden" but is accessible within the program. For reliability, such an object can be saved in a chart template. Parameter accessing logic is as follows: if there is an object, extract the parameters; if there is no object, set the default values.

Order comments

The maximum order comment length is limited to 23 characters. What can be stored in a comment? For example, SOP/H1/SS/C2/Br/Br/Br. Where (left to right)

SOP — order sender (SOP – the SendOrderByPlan script)
H1 — order generation timeframe (H1)
SS — order type (SS – Sell Stop)
C2 — order closing algorithm
Br — D1 trend (Br – Bear)
Br — H4 trend (Br – Bear)
Br — trend at the order generation timeframe (Br – Bear)

Why do we need this? For example, this data can be used for analyzing deals. Here is how I use it: when a pending order triggers, I extract the value of the closing algorithm and create a virtual stop analyzer AnalyserVirtSL, which will then close the deal under certain conditions.

Text files

This is perhaps the most reliable and universal way to store recovery parameters. You can set up access classes once and then use them whenever and wherever you need.

Application settings

Part of the AppSettings.txt settings file:

# -------------------------------------------------------------------
# Expert Advisor and script settings
# File encoding = UCS-2 LE with BOM (required!!!) // it is Unicode
# -------------------------------------------------------------------
TimeEurWinter = 10:00 # European session beginning winter time (server time)
TimeEurSummer = 09:00 # European session beginning summer time (server time)
ColorSessionEur = 224,255,255 # European session color
ColorSessionUsd = 255,240,245 # American session color
NumberColorDays = 10 # the number of highlighted days (sessions)

The AppSettings.mqh class

#property copyright "Copyright 2020, Malik Arykov"
#property link      "malik.arykov@gmail.com"
#property strict

#include <Cayman/Params.mqh>

// application parameter names
#define APP_TIME_EUR_SUMMER "TimeEurSummer"
#define APP_TIME_EUR_WINTER "TimeEurWinter"
#define APP_TIME_TRADE_ASIA "TimeTradeAsia"
#define APP_COLOR_SESSION_EUR "ColorSessionEur"
#define APP_COLOR_SESSION_USD "ColorSessionUsd"
#define APP_NUMBER_COLOR_DAYS "NumberColorDays"

// -----------------------------------------------------------------------------
// General settings of the Expert Advisor and Scripts                          |
// -----------------------------------------------------------------------------
class AppSettings {
private:
    Params  *m_params;
public:
    // set in the AppSettings.txt file
    string  TimeEurSummer; // European session beginning summer time
    string  TimeEurWinter; // European session beginning winter time
    string  TimeTradeAsia; // Asian corridor trading end time
    color   ColorSessionEur; // European session color
    color   ColorSessionUsd; // American session color
    int     NumberColorDays; // Number of highlighted days

    // set by the program
    string  PeriodTrends; // Trend calculation periods (D1,H4)
    string  TradePlan; // Trading direction (brief plan)
    bool    IsValid; // Parameter validity
    
    // methods
    AppSettings();
    ~AppSettings() { delete m_params; };
    void Dump(string sender);
};

// -----------------------------------------------------------------------------
// Constructor                                                                 |
// -----------------------------------------------------------------------------
AppSettings::AppSettings() {

    IsValid = true;
    m_params = new Params();
    m_params.Load(PATH_APP_SETTINGS);
    if (m_params.Total() == 0) {
        PrintFormat("%s / ERROR: Invalid file / %s", __FUNCTION__, PATH_APP_SETTINGS);
        IsValid = false;
        return;
    }
        
    TimeEurWinter = m_params.GetValue(APP_TIME_EUR_WINTER);
    TimeEurSummer = m_params.GetValue(APP_TIME_EUR_SUMMER);
    TimeTradeAsia = m_params.GetValue(APP_TIME_TRADE_ASIA);
    ColorSessionEur = StringToColor(m_params.GetValue(APP_COLOR_SESSION_EUR));
    ColorSessionUsd = StringToColor(m_params.GetValue(APP_COLOR_SESSION_USD));
    NumberColorDays = (int)StringToInteger(m_params.GetValue(APP_NUMBER_COLOR_DAYS));
}

// -----------------------------------------------------------------------------
// Print settings parameters                                                   | 
// -----------------------------------------------------------------------------
void AppSettings::Dump(string sender) {
    PrintFormat("sender=%s / %s", sender, PATH_APP_SETTINGS);
    PrintFormat("%s = %s", APP_TIME_EUR_WINTER, TimeEurWinter);
    PrintFormat("%s = %s", APP_TIME_EUR_SUMMER, TimeEurSummer);
    PrintFormat("%s = %s", APP_TIME_TRADE_ASIA, TimeTradeAsia);
    PrintFormat("%s = %s / %s", APP_COLOR_SESSION_EUR, ColorToString(ColorSessionEur), ColorToString(ColorSessionEur, true));
    PrintFormat("%s = %s / %s", APP_COLOR_SESSION_USD, ColorToString(ColorSessionEur), ColorToString(ColorSessionEur, true));
    PrintFormat("%s = %i", APP_NUMBER_COLOR_DAYS, NumberColorDays);
}

Features

The AppSettings class declaration is located in the Uterminal.mqh file which is connected to an Expert Advisor and to any script via #include.

extern AppSettings  *gAppSettings; // application settings

With this solution you can:

Initialize gAppSettings once anywhere
Use gAppSettings in any class instance (instead of passing it as a parameter)

Analyzer parameters

The Cayman Expert Advisor manages various analyzers such as AnalyzerTrend, AnalyserLevel, AnalyserVirtSL. Each analyzer is linked to a specific timeframe. It means that the analyzer is only launched when a new bar emerges on the specified timeframe. Analyzer examples are stored in the text file, with the Key = Value strings. For example, the H4 trading level analyzer stores its parameters in the Files\Cayman\Params\128968168864101576\exp_05_Lev607A160E_H4.txt file

Cayman — project name
Params — subdirectory with analyzer parameters
128968168864101576 — chart ID // IntergerToString(ChartID())
exp_05_Lev607A160E_H4.txt — the name of the file containing analyzer parameters —

exp — prefix
05 — analyzer type
Lev607A160E — the name of the analyzer (trading level)
H4 — tracked timeframe.

Below is the file contents with comments (the real file has no comments)

// trading level parameters
nameObj=Lev607A160E // trading level name
kindLevel=1 // level type (1 - resistance)
riskValue=1.00 // deal volume upon level breakout (1)
riskUnit=1 // deal volume change unit (1 - % of funds for margin)
algClose=2 // deal closing algorithm (2 – two correction bars)
ticketNew=0 // ticket of a deal opened upon level breakout
ticketOld=0 // ticket to close a deal upon level breakout
profits=0 // planned profit in points
losses=0 // planned loss in points
// analyzer parameters
symbol=EURUSD // symbol name
period=16388 // analyzer period (H4)
time0Bar=1618603200 // zero bar time (sec)
typeAnalyser=5 // analyzer type
colorAnalyser=16711935 // color for analyzer results
resultAnalyser=Lev607A160E, H4, 20:00, RS // analyzer results

There is a base class Analyser which can save and restore parameters of any analyzer. When an Expert Advisor is restarted (for example, after switching timeframes), analyzers restore parameters from the relevant text files. If the time for a new bar has not yet come, the analysis is not restarted. Analyzer results (resultAnalyser, colorAnalyser) calculated at the previous bar are displayed in the Expert Advisor comments.

Passing script parameters to an Expert Advisor

The SetTradeLevel script allows setting the parameters of a trading level. One object (straight line, trend line or rectangle) is selected on the chart. The SetTradeLevel script finds the selected object (trading level) and sets its parameters.

SetTradeLevel script parameters

Next, the script saves the parameters to Files\Cayman\Params\128968168864101576\exp_05_Lev607A160E_H4.txt and sends the command and the path to the file via the SendCommand function.

// -----------------------------------------------------------------------------
// Send level parameters to the Expert Advisor                                 |
// -----------------------------------------------------------------------------
NCommand SendCommand() {

    // load level parameters (if any)
    Params *params = new Params();
    string speriod = UConvert::PeriodToStr(_Period);
    params.Load(PREFIX_EXPERT, anaLevel, gNameLev, speriod);

    // define the command
    NCommand cmd = 
        (gKindLevel == levUnknown) ? cmdDelete :
        (params.Total() > 0) ? cmdUpdate :
        cmdCreate;

    // save parameters
    params.Clear();
    params.Add(PARAM_NAME_OBJ, gNameLev);
    params.Add(PARAM_TYPE_ANALYSER, IntegerToString(anaLevel));
    params.Add(PARAM_PERIOD, IntegerToString(_Period));
    params.Add(PARAM_KIND_LEVEL, IntegerToString(gKindLevel));
    params.Add(PARAM_RISK_VALUE, DoubleToString(gRiskValue, 2));
    params.Add(PARAM_RISK_UNIT, IntegerToString(gRiskUnit));
    params.Add(PARAM_ALG_CLOSE, IntegerToString(gAlgClose));
    params.Add(PARAM_TICKET_OLD, IntegerToString(gTicketOld));
    params.Add(PARAM_PROFITS, IntegerToString(gProfits));
    params.Add(PARAM_LOSSES, IntegerToString(gLosses));
    params.Save();
    
    // send a command to the Expert Advisor
    params.SendCommand(cmd);
    delete params;
    
    return cmd;
}

The params.SendCommand(cmd) function is as follows:

// -----------------------------------------------------------------------------
// Send a command to the Expert Advisor                                        |
// -----------------------------------------------------------------------------
void Params::SendCommand(NCommand cmd) {
    string nameObj = NAME_OBJECT_CMD;
    ObjectCreate(0, nameObj, OBJ_LABEL, 0, 0, 0);
    ObjectSetString(0, nameObj, OBJPROP_TEXT, m_path);
    ObjectSetInteger(0, nameObj, OBJPROP_ZORDER, cmd);
    ObjectSetInteger(0, nameObj, OBJPROP_TIMEFRAMES, 0);
}

Every tick (OnTick), the Expert Advisor checks the existence of the object named NAME_OBJECT_CMD via the CheckExpernalCommand() function. If it exists, the command and the path to the file with the analyzer parameters are read, and the object is immediately deleted. Next, the Expert Advisor searches for a running analyzer by the file name. If cmd == cmdDelete, then the analyzer is deleted. If cmd == cmdUpdate, then the analyzer parameters are updated from the file. If cmd == cmdNew, then a new analyzer is created with parameters from the file.

Here is the full text of the Params class which encapsulates the logic for working with parameter files (Key=Value strings).

#property copyright "Copyright 2020, Malik Arykov"
#property link      "malik.arykov@gmail.com"

#include <Arrays/ArrayString.mqh>
#include <Cayman/UConvert.mqh>
#include <Cayman/UFile.mqh>

// -----------------------------------------------------------------------------
// Parameter class (key=value strings with # comments)                           |
// -----------------------------------------------------------------------------
class Params {
private:
    string  m_path; // path to parameter file
    NCommand m_cmd; // command for the Expert Advisor
    CArrayString *m_items; // array of pairs {key=value}
    int Find(string key);
public:
    Params();
    ~Params() { delete m_items; };
    void Clear() { m_items.Clear(); };
    int Total() { return m_items.Total(); };
    string Path() { return m_path; };
    CArrayString *Items() { return m_items; };
    void Add(string line) { m_items.Add(line); };
    bool Add(string key, string value);
    string GetValue(string key);
    void Load(string prefix, int typeAnalyser, string nameObj, string speriod);
    void Load(string path);
    void Save();
    void SendCommand(NCommand cmd);
    NCommand TakeCommand();
    void Dump(string sender);
};

// -----------------------------------------------------------------------------
// Default constructor                                                         |
// -----------------------------------------------------------------------------
Params::Params() {
    m_items = new CArrayString();
}

// -----------------------------------------------------------------------------
// Add a key=value pair                                                        | 
// -----------------------------------------------------------------------------
bool Params::Add(string key, string value) {

    int j = Find(key);
    string line = key + "=" + value;
    if (j >= 0) { // update
        m_items.Update(j, line);
        return false;
    }
    else { // add
        m_items.Add(line);
        return true;
    }
}

// -----------------------------------------------------------------------------
// Get the value of a parameter by key                                         |
// -----------------------------------------------------------------------------
string Params::GetValue(string key) {

    // find the key
    int j = Find(key);
    if (j < 0) return NULL; // no key
    
    // check the separator
    string line = m_items.At(j);
    j = StringFind(line, "=");
    if (j < 0) { // no =
        PrintFormat("%s / ERROR: Invalid string %s", __FUNCTION__, line);
        return NULL;
    }
    
    // return the value
    return UConvert::Trim(StringSubstr(line, j + 1));
}

// -----------------------------------------------------------------------------
// Find the value of a parameter by key                                        |
// -----------------------------------------------------------------------------
int Params::Find(string key) {

    int index = -1;
    for (int j = 0; j < m_items.Total(); j++) {
        if (StringFind(m_items.At(j), key) == 0) {
            index = j;
            break;
        }
    }
    return index;
}

// -----------------------------------------------------------------------------
// Load parameters                                                             |
// -----------------------------------------------------------------------------
void Params::Load(string prefix, int typeAnalyser, string nameObj, string speriod) {
    string nameFile = StringFormat("%s%02i_%s_%s.txt", prefix, typeAnalyser, nameObj, speriod);
    m_path = StringFormat("%s%s/%s", PATH_PARAMS, IntegerToString(ChartID()), nameFile);
    if (FileIsExist(m_path)) Load(m_path);
}

// -----------------------------------------------------------------------------
// Load parameters                                                             |
// -----------------------------------------------------------------------------
void Params::Load(string path) {
  
    m_path = path;
    if (!FileIsExist(m_path)) return;

    //PrintFormat("%s / %s", __FUNCTION__, m_path);
    string text = UFile::LoadText(m_path);
    if (text == NULL) return;
    
    // split the text into lines
    string line, lines[];
    int numLines = StringSplit(text, DLM_LINE, lines);
    for (int j = 0; j < numLines; j++) {
        line = lines[j];
        // delete the comment
        int k = StringFind(line, "#");
        if (k == 0) continue; // the whole string is a comment
        if (k > 0) line = StringSubstr(line, 0, k);
        // add a non-empty string
        if (line != "") m_items.Add(line);
    }
}

// -----------------------------------------------------------------------------
// Save parameters                                                             |
// -----------------------------------------------------------------------------
void Params::Save() {

    string text = "";
    for (int j = 0; j < m_items.Total(); j++) {
        text += m_items.At(j) + "\n";
    }
    // rewrite the existing file
    UFile::SaveText(text, m_path, true);
}

// -----------------------------------------------------------------------------
// Send a command to the Expert Advisor                                        | 
// -----------------------------------------------------------------------------
void Params::SendCommand(NCommand cmd) {
    string nameObj = NAME_OBJECT_CMD;
    ObjectCreate(0, nameObj, OBJ_LABEL, 0, 0, 0);
    ObjectSetString(0, nameObj, OBJPROP_TEXT, m_path);
    ObjectSetInteger(0, nameObj, OBJPROP_ZORDER, cmd);
    ObjectSetInteger(0, nameObj, OBJPROP_TIMEFRAMES, 0);
}

// -----------------------------------------------------------------------------
// Receive a command from the script                                           |
// -----------------------------------------------------------------------------
NCommand Params::TakeCommand() {
    string nameObj = NAME_OBJECT_CMD;
    if (ObjectFind(0, nameObj) < 0) return cmdUnknown;
    
    m_path = ObjectGetString(0, nameObj, OBJPROP_TEXT);
    m_cmd = (NCommand)ObjectGetInteger(0, nameObj, OBJPROP_ZORDER);
    ObjectDelete(0, nameObj);
    Load(m_path);
    
    return m_cmd;
}

// -----------------------------------------------------------------------------
// |Dump parameters                                                            |                                                             
// -----------------------------------------------------------------------------
void Params::Dump(string sender) {
    for (int j = 0; j < m_items.Total(); j++) {
        PrintFormat("%s / %s", sender, m_items.At(j));
    }
}

For MQL5 Fans: when changing the m_items type to CHashMap, the code of the Add, GetValue, Find functions will be significantly reduced. But the Params class is also used in MQL4. Furthermore, parameter access speed is not important in this case, as the parameters are read once to initialize local variables. Why haven't I remade the class for CHashMap for MQL5? Probably because I worked in a bank for a long time. Financial software developers have a very important principle: If it works, don't touch it! ;-)

Passing parameters to external programs

The data exchange unit between different systems is de facto a json file. Previously it was an xml file. The main advantages of json files are:

Ease of creation (generation/formatting)
Excellent support in all high-level languages
Readability

For example, there is a Bar class with the following fields: m_time, m_open, m_high, m_low, m_close, m_body. Where m_body is the candlestick color: white, black or doji. The Bar class has a ToJson() method which generates a json string

string Bar::ToJson() {
        return "{" +
        "\n\t\"symbol\":\"" + _Symbol + "\"," +
        "\n\t\"period\":" + IntegerToString(_Period) + "," +
        "\n\t\"digits\":" + IntegerToString(_Digits) + "," +
        "\n\t\"timeBar\":\"" + TimeToStr(m_time) + "\"," +
        "\n\t\"open\":" + DoubleToString(m_open, _Digits) + "," +
        "\n\t\"high\":" + DoubleToString(m_high, _Digits) + "," +
        "\n\t\"low\":" + DoubleToString(m_low, _Digits) + "," +
        "\n\t\"close\":" + DoubleToString(m_close, _Digits) + "," +
        "\n\t\"body\":" + IntegerToString(m_body) + "," +
        "\n}";
}

We could use StringFormat instead, but this would cause problems while rearranging or deleting values. Formatting “\n\t” could be deleted since there are quite a lot of online json formatting services. One of them is JSON Parser. Set the receiving of a valid json once and use the bar.ToJson() function whenever you need it.

An external program, for example a C# application, can convert a json file of any complexity into an object. How to transfer a json file from MQL? It is very simple. Load (save) the json file, for example, to the Files/Json terminal directory. An external program monitors this directory for new files. Having found a file, the program reads it, converts it into an object and immediately deletes the file or moves it to the archive (for statistics).

Receiving parameters from external programs

Connecting a json library (or reinventing the wheel) to MQL programs causes extra trouble. A better solution is to pass text files with Key=Value strings. Files can be processed using the Params class (see above). The Expert Advisor and the Indicator are candidates for receiving parameters from external programs or scripts. For example, you need to call the CheckExternalCommand() function in the OnTick handler, which will check the existence of files in the Files/ExtCmd directory. When a file is found, it should read, process (accept the parameters) and delete the file.

So, we have considered methods for receiving and passing parameters between MQL and external programs. Now think about the following: Why do MQL programs need DLLs? Such programs are not accepted by the MQL Market. There is only one reason — security, since you can access anything from a DLL.

Passing parameters to a smartphone

For further operations, I will use the Android app WirePusher. This is a wonderful service (free and with no adds). I do not know if there is something like this for iPhone. If there are any iPhone fans reading this article, please share in comments.

To start using the service:

Install WirePusher on your smartphone
Launch the application. You will see your id on the main screen
Add https://wirepusher.com to Terminal/Service/Settings/Experts/Allow WebRequest

Then launch the script (do not forget to write your id instead of asterisks in id = “********”

void OnStart() {
    string id = "**********"; // your smartphone id in WirePusher
    WirePusher("Profit $1000", "Deal", "Closed", id);
}

// ------------------------------------------------------------------------------------------------
// Send notification to smartphone via the WirePusher web service
// Add https://wirepusher.com to Terminal/Service/Settings/Experts/Allow WebRequest
// message - notification text
// title - notification title (for example, Attention / Alert / Deal)
// type - notification type (for example, Triggered pending order / Levels breakout / Closed)
// id - unique smartphone id from the WirePusher Android-app
// ------------------------------------------------------------------------------------------------
bool WirePusher(string message, string title, string type, string id) {

    char data[]; // HTTP message body data array
    char result[]; // Web service response data array
    string answer; // Web service response headers
    string url = "https://wirepusher.com/send?id={id}&title={title}&message={message}&type={type}";
    
    StringReplace(url, "{id}", id);
    StringReplace(url, "{type}", type);
    StringReplace(url, "{title}", title);
    StringReplace(url, "{message}", message);
    
    ResetLastError();
    int rcode = WebRequest("GET", url, NULL, 3000, data, result, answer);
    if (rcode != 200) {
        PrintFormat("%s / error=%i / url=%s / answer=%s / %s", 
            __FUNCTION__, GetLastError(), url, answer, CharArrayToString(result));
        return false;
    }
    PrintFormat("%s / %s / %s", __FUNCTION__, title, message);
    return true;
}

In the Cayman EA, the WirePusher function is called in AnalyserTrade when:

A pending order triggers
The price breaks though a trading level
A deal closes

An individual sound can be assigned to each notification type on WirePusher. Previously, I had a "ta-da" sound for deals closed with profit and a "bomb" sound for those closed with loss. But then I got tired of bombs.

Conclusion

The most reliable and convenient method for storing parameters is using text files. Moreover, file operations are completely supported/cached in any operating system (application).