Native Twitter Client for MT4 and MT5 without DLL

8 August 2020, 20:36
Soewono Effendi
3
18 331

Introduction


Tweeter provides free platform for anyone to post anything on their site. It can be as valuable as financial tips or as valueless as any prominent person can be in expressing her/his thoughts. Since this article primary focus on the media instead of its contents, let's get started.

Please sign-up on Tweeter to indulge yourself with a bunch of tokens required to access Tweeter API.

These tokens can be quite confusing at first, simply because there are a lot of them with similar names. Basically you need following tokens to be able to use Twitter API:
  • customer_token
  • customer_token_secret
  • access_token
  • access_token_secret

Note:

If you're familiar with public and private keys used for digital signing then you're on the right side.

The customer_token and customer_token_secret are public and private keys to identify your "Twitter app". A Twitter app is, simply said, an identification of your service and/or access utilizing Twitter API. 

The access_token and access_token_secret are public and private keys to identify you as "Twitter user", in Twitter's term it is called as "user auth" or "user context". Based on this access_token Twitter API can identify who is accessing it.

There exists another so called bearer_token which allow "anonymous" access using Twitter API. This method of access is called as "app auth" or "app context". Without "user context" some Twitter APIs are not accessible which are well documented on Twitter API reference

For those who can code in other programming languages might find these Twitter Libraries are useful for reference. They are great resources that provide great insight into implementation details which are sometime not obvious from simply reading API documentation only.


Tweeter API and authorization

We focus on using the above mentioned tokens:

  • customer_token
  • customer_token_secret
  • access_token
  • access_token_secret

There are plenty of guides and/or tutorials on YouTube on how to get these tokens.

OAuth is well-accepted and widely-used standard for authentication and authorization of web based API, which is Twitter API is also using.
Simply said, the OAuth is digital signature, a method to sign digital content so that any attempt to manipulate the content shall invalidate it.

To verify the content and its signature correctly, specific methods and process of creating that content shall be followed precisely.

Again, Twitter API documentation has done a great job in documenting this whole process.

To keep this process simple, Twitter API requires a specific method of encoding requests over HTTP. This method will be described in the next chapter.

URL encoding and parameters sorting

To assure absolute correctness of digitally signed content, the content itself shall be well defined. For that purpose Twitter API (to be precise, the OAuth method) requires HTTP POST and/or   HTTP  GET parameters to go through well defined steps before they are digitally signed.

It is imperative to encode HTTP POST / HTTP GET parameters as follows:

  • All characters shall be "percent encoded (%XX)", except: alphanumeric (0-9, A-Z, a-z) and these special characters ( ‘-‘, ‘.’, ‘_’, ‘~’ )
  • The hex value of the "percent encoding" shall use capital letter (0-9 A B C D E F)

Please note in the reference documentation and this documentation too regarding the "%" and "+" characters which shall be encoded correctly as well.

Please also pay attention regarding sorting of the parameters as quoted here from the reference documentation:

  • The OAuth spec says to sort lexicographically, which is the default alphabetical sort for many libraries.
  • In the case of two parameters with the same encoded key, the OAuth spec says to continue sorting based on value. However, Twitter does not accept duplicate keys in API requests

A simple implementation of this requirement is as follow:

string hex(int i)
  {
   static string h="0123456789ABCDEF";
   string ret="";
   int a = i % 16;
   int b = (i-a)/16;
   if(b>15)
      StringConcatenate(ret,ret,hex(b),StringSubstr(h,a,1));
   else
      StringConcatenate(ret,ret,StringSubstr(h,b,1),StringSubstr(h,a,1));
   return (ret);
  }

string URLEncode(string toCode)
  {
   int max=StringLen(toCode);

   string RetStr="";
   for(int i=0; i<max; i++)
     {
      string c = StringSubstr(toCode,i,1);
      ushort asc = StringGetCharacter(c, 0);

      if((asc >= '0' && asc <= '9')
         || (asc >= 'a' && asc <= 'z')
         || (asc >= 'A' && asc <= 'Z')
         || (asc == '-')
         || (asc == '.')
         || (asc == '_')
         || (asc == '~'))
         StringAdd(RetStr,c);
      else
        {
         StringConcatenate(RetStr,RetStr,"%",hex(asc));
        }
     }
   return (RetStr);
  }

string arrayEncode(string &array[][2])
  {
   string ret="";
   string key,val;
   int l=ArrayRange(array,0);
   for(int i=0; i<l; i++)
     {
      key = URLEncode(array[i,0]);
      val = URLEncode(array[i,1]);
      StringConcatenate(ret,ret,key,"=",val);
      if(i+1<l)
         StringConcatenate(ret,ret,"&");
     }
   return (ret);
  }

void sortParam(string&arr[][2])
  {
   string k1, k2;
   string v1, v2;
   int n = ArrayRange(arr,0);

// bubble sort
   int i, j;
   for(i = 0; i < n-1; i++)
     {
      // Last i elements are already in place
      for(j = 0; j < n-i-1; j++)
        {
         int x = j+1;
         k1 = arr[j][0];
         k2 = arr[x][0];
         if(k1 > k2)
           {
            // swap values
            v1 = arr[j][1];
            v2 = arr[x][1];
            arr[j][1] = v2;
            arr[x][1] = v1;
            // swap keys
            arr[j][0] = k2;
            arr[x][0] = k1;
           }
        }
     }
  }

void addParam(string key,string val,string&array[][2])
  {
   int x=ArrayRange(array,0);
   if(ArrayResize(array,x+1)>-1)
     {
      array[x][0]=key;
      array[x][1]=val;
     }
  }


An example on using the above functions is as follow:

   string params[][2];

   addParam("oauth_callback", "oob", params);
   addParam("oauth_consumer_key", consumer_key, params);

   sortParam(params);

Please note:
For simplification the parameters sorting is incomplete, it does not consider multiple same keys parameters. You might want to improve it in case you're using parameters with same keys, i.e radio buttons, check boxes on your html form.


HMAC-SHA1 as easy as possible

Another hurdle before we can create the OAuth signature is the lack of HMAC-SHA1 native support in MQL. It turns out that MQL CryptEncode() is of little use here, since it supports only building SHA1-HASH, hence the flag: CRYPT_HASH_SHA1.

So, let's code our own HMAC-SHA1 with help of CryptEncode()


string hmac_sha1(string smsg, string skey, uchar &dstbuf[])
  {
// HMAC as described on:
// https://en.wikipedia.org/wiki/HMAC
//
   uint n;
   uint BLOCKSIZE=64;
   uchar key[];
   uchar msg[];
   uchar i_s[];
   uchar o_s[];
   uchar i_sha1[];
   uchar keybuf[];
   uchar i_key_pad[];
   uchar o_key_pad[];
   string s = "";

   if((uint)StringLen(skey)>BLOCKSIZE)
     {
      uchar tmpkey[];
      StringToCharArray(skey,tmpkey,0,StringLen(skey));
      CryptEncode(CRYPT_HASH_SHA1, tmpkey, keybuf, key);
      n=(uint)ArraySize(key);
     }
   else
      n=(uint)StringToCharArray(skey,key,0,StringLen(skey));

   if(n<BLOCKSIZE)
     {
      ArrayResize(key,BLOCKSIZE);
      ArrayFill(key,n,BLOCKSIZE-n,0);
     }

   ArrayCopy(i_key_pad,key);
   for(uint i=0; i<BLOCKSIZE; i++)
      i_key_pad[i]=key[i]^(uchar)0x36;

   ArrayCopy(o_key_pad,key);
   for(uint i=0; i<BLOCKSIZE; i++)
      o_key_pad[i]=key[i]^(uchar)0x5c;

   n=(uint)StringToCharArray(smsg,msg,0,StringLen(smsg));
   ArrayResize(i_s,BLOCKSIZE+n);
   ArrayCopy(i_s,i_key_pad);
   ArrayCopy(i_s,msg,BLOCKSIZE);

   CryptEncode(CRYPT_HASH_SHA1, i_s, keybuf, i_sha1);
   ArrayResize(o_s,BLOCKSIZE+ArraySize(i_sha1));
   ArrayCopy(o_s,o_key_pad);
   ArrayCopy(o_s,i_sha1,BLOCKSIZE);
   CryptEncode(CRYPT_HASH_SHA1, o_s, keybuf, dstbuf);

   for(int i=0; i < ArraySize(dstbuf); i++)
      StringConcatenate(s, s, StringFormat("%02x",(dstbuf[i])));

   return s;
  }


To verify its correctness we can compare with the hash created on Twitter API documentation:

   uchar hashbuf[];
   string base_string = "POST&https%3A%2F%2Fapi.twitter.com%2F1.1%2Fstatuses%2Fupdate.json&include_entities%3Dtrue%26oauth_consumer_key%3Dxvz1evFS4wEEPTGEFPHBog%26oauth_nonce%3DkYjzVBB8Y0ZFabxSWbWovY3uYSQ2pTgmZeNu2VS4cg%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1318622958%26oauth_token%3D370773112-GmHxMAgYyLbNEtIKZeRNFsMKPR9EyMZeS9weJAEb%26oauth_version%3D1.0%26status%3DHello%2520Ladies%2520%252B%2520Gentlemen%252C%2520a%2520signed%2520OAuth%2520request%2521";
   string signing_key = "kAcSOqF21Fu85e7zjz7ZN2U4ZRhfV3WpwPAoE3Z7kBw&LswwdoUaIvS8ltyTt5jkRh4J50vUPVVHtR2YPi5kE";  
   string hash = hmac_sha1(base_string, signing_key, hashbuf);

   Print(hash); // 842b5299887e88760212a056ac4ec2ee1626b549

   uchar not_use[];
   uchar base64buf[];
   CryptEncode(CRYPT_BASE64, hashbuf, not_use, base64buf);
   string base64 = CharArrayToString(base64buf);
   
   Print(base64); // hCtSmYh+iHYCEqBWrE7C7hYmtUk=


WebRequest to the rescue


Thanks to WebRequest() it is now easily to access any REST API over web without using any external DLL.
Following code will simplify accessing Twitter API using WebRequest()

#define WEBREQUEST_TIMEOUT 5000
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
string SendRequest(string method, string url, string headers="", string params="", int timeout=WEBREQUEST_TIMEOUT)
  {
   char data[];
   char result[];
   string resp_headers;
   ResetLastError();
   StringToCharArray(params, data);
   ArrayResize(data, ArraySize(data)-1);
   int res = WebRequest(method, url, headers, timeout, data, result, resp_headers);
   if(res != -1)
     {
      string resp = CharArrayToString(result);
      if(verbose)
        {
         Print("***");
         Print("Data:");
         Print(CharArrayToString(data));
         Print("Resp Headers:");
         Print(resp_headers);
         Print("Resp:");
         Print("***");
         Print(resp);
        }
      return resp;
     }
   else
     {
      int err = GetLastError();
      PrintFormat("* WebRequest error: %d (%d)", res, err);
      if(verbose)
        {
         Print("***");
         Print("Data:");
         Print(CharArrayToString(data));
        }
      if (err == 4014)
      {
         string msg = "* PLEASE allow https://api.tweeter.com in WebRequest listed URL";
         Print(msg);
      }
     }
   return "";
  }

PLEASE READ THE DOCUMENTATION OF WEBREQUEST().


QUOTE:

To use the WebRequest() function, add the addresses of the required servers in the list of allowed URLs in the "Expert Advisors" tab of the "Options" window. Server port is automatically selected on the basis of the specified protocol - 80 for "http://" and 443 for "https://".


Helper functions to build Twitter REST API

Below some helper functions useful in building Twitter API signature.

string getNonce()
  {
   const string alnum = "abcdef0123456789";
   char base[];
   StringToCharArray(alnum, base);
   int x, len = StringLen(alnum);
   char res[32];
   for(int i=0; i<32; i++)
     {
      x = MathRand() % len;
      res[i] = base[x];
     }
   return CharArrayToString(res);
  }

string getBase(string&params[][2], string url, string method="POST")
  {
   string s = method;
   StringAdd(s, "&");
   StringAdd(s, URLEncode(url));
   StringAdd(s, "&");
   bool first = true;
   int x=ArrayRange(params,0);
   for(int i=0; i<x; i++)
     {
      if(first)
         first = false;
      else
         StringAdd(s, "%26");  // URLEncode("&")
      StringAdd(s, URLEncode(params[i][0]));
      StringAdd(s, "%3D"); // URLEncode("=")
      StringAdd(s, URLEncode(params[i][1]));
     }
   return s;
  }

string getQuery(string&params[][2], string url = "")
  {
   string key;
   string s = url;
   string sep = "";
   if(StringLen(s) > 0)
     {
      if(StringFind(s, "?") < 0)
        {
         sep = "?";
        }
     }
   bool first = true;
   int x=ArrayRange(params,0);
   for(int i=0; i<x; i++)
     {
      key = params[i][0];
      if(StringFind(key, "oauth_")==0)
         continue;
      if(first)
        {
         first = false;
         StringAdd(s, sep);
        }
      else
         StringAdd(s, "&");
      StringAdd(s, params[i][0]);
      StringAdd(s, "=");
      StringAdd(s, params[i][1]);
     }
   return s;
  }

string getOauth(string&params[][2])
  {
   string key;
   string s = "OAuth ";
   bool first = true;
   int x=ArrayRange(params,0);
   for(int i=0; i<x; i++)
     {
      key = params[i][0];
      if(StringFind(key, "oauth_")!=0)
         continue;
      if(first)
         first = false;
      else
         StringAdd(s, ", ");
      StringAdd(s, URLEncode(key));
      StringAdd(s, "=\"");
      StringAdd(s, URLEncode(params[i][1]));
      StringAdd(s, "\"");
     }
   return s;
  }


Sample script

Now we are ready to send our first Twitter API request.

void verifyCredentials()
  {

   string _api_key = consumer_key;
   string _api_secret = consumer_secret;
   string _token = access_token;
   string _secret = access_secret;

   string url = "https://api.twitter.com/1.1/account/verify_credentials.json";
   
   string params[][2];
   addParam("oauth_consumer_key", _api_key, params);
   string oauth_nonce = getNonce();
   addParam("oauth_nonce", oauth_nonce, params);
   addParam("oauth_signature_method", "HMAC-SHA1", params);
   
   string oauth_timestamp = IntegerToString(TimeGMT());
   addParam("oauth_timestamp", oauth_timestamp, params);
   
   addParam("oauth_token", _token, params);
   addParam("oauth_version", "1.0", params);
   
   sortParam(params);
   
   string query = getQuery(params, url);
   string base = getBase(params, url, "GET");
   uchar buf[];
   string key = URLEncode(_api_secret);
   StringAdd(key, "&");
   StringAdd(key, URLEncode(_secret));
   
   uchar hashbuf[], base64buf[], nokey[];
   string hash = hmac_sha1(base, key, hashbuf);
   CryptEncode(CRYPT_BASE64, hashbuf, nokey, base64buf);
   string base64 = CharArrayToString(base64buf);

   addParam("oauth_signature", base64, params);
   sortParam(params);
   string o = getOauth(params);

   string headers = "Host:api.twitter.com\r\nContent-Encoding: identity\r\nConnection: close\r\n";
   StringAdd(headers, "Authorization: ");
   StringAdd(headers, o);
   StringAdd(headers, "\r\n\r\n");
   
   string resp = SendRequest("GET", query, headers);
   Print(resp);

   // if everything works well, we shall receive JSON-response similar to the following
   // NOTE: Identity has been altered to protect the innocent.
   // {"id":122,"id_str":"122","name":"xxx","screen_name":"xxx123","location":"","description":"", ... 
  }


A sample Twitter client on a MT5 chart


The following picture shows a Twitter client displaying Tweets of an Indonesian news channel. I'm preparing a followup article with more Twitter API implementation, which I'm hoping to publish it as soon as possible.



Twitter client on MT5 chart

Figure 1. Tweets displayed on chart


Another screenshot showing tweet posted from MT5 terminal.



Tweet from MT5 terminal
Figure 2. A tweet posted from MT5 terminal 


Future enhancement


The above described method works well as it is, albeit it is far from complete to cover all Twitter API. It is left as good exercise for those who want to dive deeper into Tweeter API. Some more details on posting media, including chart screenshots will be described on next following article.

You might want to build a TwitterAPI class or even a general OAuth client class.

A note to 3-legged authorization and PIN based authorization


You might want to dive deeper into the so called 3-legged authorization and also the PIN based authorization.

It is beyond of this article to describe them, please feel free to contact me in case you need further guidance.

Conclusion

Twitter is a well accepted platform that provides anyone to publish almost anything.
With this article and its code described above, I'm hoping to contribute to MQL community with the result of my little adventure in understanding OAuth in accessing Twitter API .

Looking forward for encouraging and improving feedbacks. Feel free to use all code provide in any free and/or commercial projects of yours.

I want to thank Grzegorz Korycki for providing this library code (SHA256, SHA384 and SHA512 + HMAC - library for MetaTrader 4) which has inspired me in creating the HMAC-SHA1 function.

Let's tweet for fun and profit!

Enjoy.


Attached files |
tweeter_mql.mq5 (9.67 KB)
urlencode.mqh (3.55 KB)
Last comments | Go to discussion (3)
Yohana Parmi
Yohana Parmi | 9 Aug 2020 at 03:47
Thank you @Soewono Effendi,
 - your hard work is useful :)
Fatchur Rochman
Fatchur Rochman | 9 Aug 2020 at 11:45
Thanks you for your share... 
Icham Aidibe
Icham Aidibe | 10 Aug 2020 at 22:02
👍 thank you for that it was missing
Timeseries in DoEasy library (part 41): Sample multi-symbol multi-period indicator Timeseries in DoEasy library (part 41): Sample multi-symbol multi-period indicator

In the article, we will consider a sample multi-symbol multi-period indicator using the timeseries classes of the DoEasy library displaying the chart of a selected currency pair on a selected timeframe as candles in a subwindow. I am going to modify the library classes a bit and create a separate file for storing enumerations for program inputs and selecting a compilation language.

Continuous Walk-Forward Optimization (Part 7): Binding Auto Optimizer's logical part with graphics and controlling graphics from the program Continuous Walk-Forward Optimization (Part 7): Binding Auto Optimizer's logical part with graphics and controlling graphics from the program

This article describes the connection of the graphical part of the auto optimizer program with its logical part. It considers the optimization launch process, from a button click to task redirection to the optimization manager.

Timeseries in DoEasy library (part 42): Abstract indicator buffer object class Timeseries in DoEasy library (part 42): Abstract indicator buffer object class

In this article, we start the development of the indicator buffer classes for the DoEasy library. We will create the base class of the abstract buffer which is to be used as a foundation for the development of different class types of indicator buffers.

Native Twitter Client: Part 2 Native Twitter Client: Part 2

A Twitter client implemented as MQL class to allow you to send tweets with photos. All you need is to include a single self contained include file and off you go to tweet all your wonderful charts and signals.