Building a Social Technology Startup, Part I: Tweet Your MetaTrader 5 Signals
laplacianlab | 17 June, 2014
Introduction
This article aims to illustrate, through a practical example, how you can communicate an MetaTrader 5 terminal with an external web service. We are tweeting the trading signals generated by an Expert Advisor.
This idea comes from a particular conception of automatic trading called computer-assisted trading. In a nutshell, the computers of the XXI century do not have cognitive abilities, but they are very good at processing information and executing data. So why do not we build computer systems using human brains as filters to make decisions? This approach is inspired on the Human-based computation (HBC) paradigm, hence focuses on building decision support tools, rather than coding decision maker algorithms.
I had initially thought about creating an RSS feed with the trading signals generated by my EAs (it is assumed that there is a medium or long term underlying trading system, this idea is not valid for automatic scalping systems). A human with access to the feed should validate my robotic signals according to the circumstances of the moment, just before putting them on the market. However, I soon realized that everything could be even more social, and thought to myself, 'Why not publish my trading signals on the Twitter?' This led me to develop this Social Decision Support System.
Figure 1. SDSS Architecture
By the way, if you plan to create a technology startup related to FX trading this article may help you to take some ideas. It can be seen as a technical guide to build a commercial SaaS (Software as a Service) based on an SDSS.
This text is long, so I've decided to split it into two parts. The first one focuses on the web service architecture, the communication protocol used between the MetaTrader 5 terminal and the Twitter app, and finally the web app's integration with Twitter. The second part will cover the MQL5 layer of the network diagram shown above, which is intended to consume our Social Decision Support System's RESTful web service. Specifically, we will code an MQL5-RESTful framework in the same way as explained in the article MQL5-RPC. Remote Procedure Calls from MQL5: Web Service Access and XML-RPC ATC Analyzer for Fun and Profit.
This article is also social, so I encourage you to write your comments to help to continue with the second part.
1. Some Notes on the SDSS Architecture
1.1. MetaTrader 5 on a server
This is an MetaTrader 5 terminal running on a machine which is available 24 hours a day. This computer can be a Virtual Private Server (VPS) or a Dedicated Server (DS). MetaTrader 5 on a server communicates with the
Twitter application through a RESTful web service. The format for data interchange is JSON. In the second part of this article we will deploy the MQL5 layer consuming the web service, according to the communication protocol that we are defining later in this text.
1.2. Twitter app
As we said in the introduction, we are developing the Twitter app of the SDSS in the first part of this article. This is a PHP web application which basically receives signals from the MetaTrader 5 terminal, stores them in a MySQL database and then tweets them. However, this app can be extended in order to handle many problems that require human abilities to be solved.
For example, you would want to capture knowledge, and build a web ontology with the signals received from 'MetaTrader 5 on a server' which have been filtered by people, to see how reliable your EA is from the human perspective. We could calculate the reliability of a given signal (a tweet) from the retweets and favorites associated with it. This is just an idea that can not be covered in this article, of course, but it is theoretically possible. It is just a practical application based on the Human-based computation (HBC) paradigm.
1.3. Twitter
As Wikipedia says, Twitter is the famous online social networking and microblogin service that enables users to send and read short 140-character text messages, called tweets.
2. RESTful Web Service Specification
2.1. REST Overview
REST, SOAP and XML-RPC are the three most used architectural patterns for building a web service. As most modern Web 2.0 applications use this pattern to develop their services, we will also code our SDSS's on REST. I encourage you to learn more about this topic by reading the document entitled The Twitter REST API. In short, REST is an exchange of HTTP calls with JSON or XML as a data exchange format.
Unlike SOAP and XML-RPC, REST is very easy to implement and it is also the fastest. It is the simplest because there is no protocol specification in the same sense as there is in XML-RPC and SOAP. This means that the developer does not have to learn any XML pseudo-language in order to deploy the web service. On the other hand, REST can use JSON as the data exchange format between the client and the server, which allows to give a faster response. Indeed, any object represented in JSON always occupies fewer bytes than if represented in XML. This is because JSON is more semantic, or put another way, JSON has less syntactic elements than XML to represent information.
2.2. SDSS API Reference
Did you already read the document entitled The Twitter REST API? Now, with all that said, Twitter's API specification can serve to take some ideas and apply them in building our SDSS API. Recall that we are making things very simple because the goal of this article is not to fully develop the SDSS, which could be deployed by a technology startup, but show a practical application in real life where you can connect your MetaTrader 5 terminal with an external web service. Accordingly, in this section we are designing an extremely simple API with a single REST method which uses the HTTP verb POST. This is for receiving, storing into the database, and tweeting a given EA's trading signal.
2.2.1. General
The API should be hosted at http://api.sdss-your-startup.com, and should be accessed via HTTP or HTTPS. For POST requests, parameters are submitted in the POST body in JSON format.
2.2.2. REST resources
| Resource | URL | Description |
|---|---|---|
| POST signal/add | http://api.sdss-your-startup.com /signal/add | Sends an EA signal to the Twitter application. This is the request which is performed by the MetaTrader 5 terminal in order for the SDSS to store and tweet the trading signal. |
As indicated in the introduction this article is social, so I encourage you to list here the REST methods you would want to implement in your Social Decision Support System. Feel free to make your comments and share with the MQL5 community.
2.2.3. POST signal/add
| Parameter | Description |
|---|---|
| ea_id |
The ID of the EA which is sending the new signal. Example value: 12345 |
| symbol |
The symbol involved in this transaction. Example value: EURUSD |
| opeartion |
The operation to be performed. Example value: BUY or SELL |
| value |
The symbol value at the moment of the transaction. Example value: 1.3214 |
Example request:
{
"ea_id": 1,
"symbol": "AUDUSD",
"operation": "BUY",
"value": 0.9281
}
With this example in view, there are two important things to note. On the one hand, let's assume for the moment that there is only one MetaTrader 5 terminal sending signals to the Twitter application. For this reason there is no need for terminal ID. If then, later, we decide to extend the system and further develop our startup, we will probably want to connect multiple terminals with the Twitter app.
So in that case, we would add a field named mt5_id in order to identify the MetaTrader 5 terminal. On the other hand, note that the call above should be secured by some authentication mechanism (Basic HTTP Authentication over SSL, token based authentication or OAuth). However, we are skipping this part, but do not forget to point this issue. Please, note that without an authentication mechanism between the MetaTrader 5 terminal and the Twitter app, anyone who knows the communication protocol could send trading signals to the Twitter app in Figure 1. SDSS Architecture.
3. The Web App's Database Design
The REST API specification has helped us to see how the system works, therefore we can now understand the database design:
# MySQL database creation...
CREATE DATABASE IF NOT EXISTS sdss;
use sdss;
# Please, change the user's password in production
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES,
CREATE TEMPORARY TABLES ON `sdss`.* TO 'laplacianlab'@'localhost' IDENTIFIED BY 'password';
CREATE TABLE IF NOT EXISTS eas (
id mediumint UNSIGNED NOT NULL AUTO_INCREMENT,
name VARCHAR(32),
description TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (id)
) ENGINE=InnoDB;
CREATE TABLE IF NOT EXISTS signals (
id int UNSIGNED NOT NULL AUTO_INCREMENT,
id_ea mediumint UNSIGNED NOT NULL,
symbol VARCHAR(10) NOT NULL,
operation VARCHAR(6) NOT NULL,
value DECIMAL(9,5) NOT NULL,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (id),
FOREIGN KEY (id_ea) REFERENCES eas(id)
) ENGINE=InnoDB;
# Dump some sample data...
INSERT INTO eas(name, description) VALUES
('Bollinger Bands', '<p>Robot based on Bollinger Bands. Works with H4 charts.</p>'),
('Two EMA', '<p>Robot based on the crossing of two MA. Works with H4 charts.</p>');
4. Programming the PHP RESTful Web Service
In the words of its own developers, Slim is a PHP micro framework that helps you quickly write simple yet powerful web applications and APIs. Yes, you guessed it! We are going to write our RESTful API on Slim, with very few lines of code. Remember that we are only coding one HTTP POST method because of learning reasons and simplicity. Please, read the official documentation of Slim to know in detail how it works.
Figure 2. Directory structure of the PHP API based on Slim
config\config.php
<?php // Creating constants. // General define('BASE_URL', 'laplacianlab.local'); define('APPLICATION_PATH', realpath(dirname(__FILE__)) . '/../'); // Database connection define('DB_NAME', 'sdss'); define('DB_USER', 'laplacianlab'); define('DB_PASSWORD', 'password'); // Don't forget to change this in your production server! define('DB_SERVER', 'localhost');
model\DBConnection.php
<?php class DBConnection { private static $instance; private $mysqli; private function __construct() { $this->mysqli = new MySQLI(DB_SERVER, DB_USER, DB_PASSWORD, DB_NAME); } public static function getInstance() { if (!self::$instance instanceof self) self::$instance = new self; return self::$instance; } public function getHandler() { return $this->mysqli; } }
public\.htaccess
<IfModule mod_rewrite.c>
<IfModule mod_negotiation.c>
Options -MultiViews
</IfModule>
RewriteEngine On
# Redirect Trailing Slashes...
RewriteRule ^(.*)/$ /$1 [L,R=301]
# Handle Front Controller...
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^ index.php [L]
</IfModule>public\index.php
<?php // Bootstrap logic require_once '../config/config.php'; set_include_path(get_include_path() . PATH_SEPARATOR . APPLICATION_PATH . '/vendor/'); set_include_path(get_include_path() . PATH_SEPARATOR . APPLICATION_PATH . '/model/'); require_once 'slim/slim/Slim/Slim.php'; require_once 'DBConnection.php'; use \Slim\Slim; Slim::registerAutoloader(); $app = new Slim(); $app->response->headers->set('Content-Type', 'application/json'); // RESTful API methods // POST signal/add $app->post('/signal/add', function() { $request = Slim::getInstance()->request(); $signal = json_decode($request->getBody()); $sql = 'INSERT INTO signals(id_ea, symbol, operation, value) VALUES (' . mysql_real_escape_string($signal->ea_id) . ",'" . mysql_real_escape_string($signal->symbol) . "','" . mysql_real_escape_string($signal->operation) . "'," . mysql_real_escape_string($signal->value) . ')'; DBConnection::getInstance()->getHandler()->query($sql); $signal->id = DBConnection::getInstance()->getHandler()->insert_id; echo json_encode($signal); }); // More API methods here!.., according to your API spec $app->run();
composer.json
{
"require": {
"slim/slim": "2.*",
"abraham/twitteroauth": "dev-master"
}
}
With all this we can test the web service in our local development machine. By the way, don't forget (1) to add a new entry in your hosts file in order for your Windows to resolve the local domain name api.laplacianlab.local, and (2) create a new virtual host for your Apache.
C:\Windows\System32\Drivers\etc\hosts
::1 localhost # Copyright (c) 1993-2009 Microsoft Corp. # # This is a sample HOSTS file used by Microsoft TCP/IP for Windows. # # This file contains the mappings of IP addresses to host names. Each # entry should be kept on an individual line. The IP address should # be placed in the first column followed by the corresponding host name. # The IP address and the host name should be separated by at least one # space. # # Additionally, comments (such as these) may be inserted on individual # lines or following the machine name denoted by a '#' symbol. # # For example: # # 102.54.94.97 rhino.acme.com # source server # 38.25.63.10 x.acme.com # x client host # localhost name resolution is handled within DNS itself. # 127.0.0.1 localhost # ::1 localhost 127.0.0.1 localhost 127.0.0.1 api.laplacianlab.local
httpd-vhosts.conf
# Virtual Hosts # # Required modules: mod_log_config # If you want to maintain multiple domains/hostnames on your # machine you can setup VirtualHost containers for them. Most configurations # use only name-based virtual hosts so the server doesn't need to worry about # IP addresses. This is indicated by the asterisks in the directives below. # # Please see the documentation at # <URL:http://httpd.apache.org/docs/2.4/vhosts/> # for further details before you try to setup virtual hosts. # # You may use the command line option '-S' to verify your virtual host # configuration. # # VirtualHost example: # Almost any Apache directive may go into a VirtualHost container. # The first VirtualHost section is used for all requests that do not # match a ServerName or ServerAlias in any <VirtualHost> block. # <VirtualHost *:80> ServerAdmin webmaster@laplacianlab.local DocumentRoot "c:/wamp/www/laplacianlab/public" ServerName api.laplacianlab.local ErrorLog "logs/api.laplacianlab.local-error.log" CustomLog "logs/api.laplacianlab.local-access.log" common <directory "c:/wamp/www/laplacianlab/public"> Options FollowSymLinks AllowOverride all Order Deny,Allow Deny from all Allow from 127.0.0.1 </directory> </VirtualHost>Now we are testing our first REST resource with RESTClient, which is a Firefox complement for debugging RESTful web services. If everything goes ok the web service must run the logic that we coded in the Slim method above and send back an HTTP 200 result along with a JSON response.
Figure 3. Sending a POST signal/add request to http://api.laplacianlab.local
The HTTP response header for the POST signal/add request above is this:
Status Code: 200 OK Connection: Keep-Alive Content-Length: 70 Content-Type: application/json Date: Mon, 07 Apr 2014 18:12:34 GMT Keep-Alive: timeout=5, max=100 Server: Apache/2.4.4 (Win64) PHP/5.4.12 X-Powered-By: PHP/5.4.12
And this is the HTTP response body:
{
"ea_id": 1,
"symbol": "AUDUSD",
"operation": "BUY",
"value": 0.9281,
"id": 22
}
We got it! We have just implemented the part of our Social Decision Support System that corresponds to the figure below. But remember!, there should be an authentication layer to protect such calls in a real scenario.
Figure 4. MetaTrader 5 and the Twitter app talking to each other through a RESTful webservice
5. The Web App's Integration with Twitter
Now let's deal with that part of the network diagram which tweets trading signals from the Slim PHP Twitter app to Twitter.
Figure 5. Tweeting trading signals from the Twitter app to Twitter
What's next? Once again, you've guessed it! First of all you need a Twitter application. So please go to Twitter Developers, sign in with your Twitter account (you first need to have a Twitter account so that you can create a Twitter app), go to "My Applications" and click on "Create New App".
Figure 6. Create a New App in Twitter Developers
You have to fill your new app's details and accept the Developer Rules of the Road. For example, the details of my new application are the following:
Name: Laplacianlab's SDSS
Description: Social Decision Support System
Website: http://api.laplacianlab.local
Callback URL: http://api.laplacianlab.local/twitter/oauth_callback
Please, now go to the tab named "Permissions" and update permissions to "Read and Write". Finally, after creating your Twitter application, write down your API key, your API secret and your Oauth callback in order for you to use them as constants in your config\config.php file:
// Twitter OAuth define('API_KEY', 'akMnfXR45MkoaWbZoPiu3'); define('API_SECRET', '45Mkoa54NcvQRBbf119qWerty0DnIW45MncvFgqw'); define('OAUTH_CALLBACK', 'http://api.laplacianlab.local/twitter/oauth_callback');
At this point we are communicating the Slim PHP micro web app with Laplacianlab's SDSS (the Twitter app we created above). To do this you need to know the basics of the OAuth 2.0 protocol so, please, do not miss the opportunity to read the official documentation. You should be familiar with the OAuth flow diagram.
Figure 7. OAuth Flow Diagram
Consequently, according to this diagram, we must now create the following MySQL table to store the resource owners (the twitterers who want to tweet the trading signals):
CREATE TABLE IF NOT EXISTS twitterers ( id mediumint UNSIGNED NOT NULL AUTO_INCREMENT, twitter_id VARCHAR(255), access_token TEXT, access_token_secret TEXT, created_at DATETIME DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (id) ) ENGINE=InnoDB;
Specifically, we are using TwitterOAuth, which is a PHP library for working with Twitter's OAuth API. If you installed Slim via Composer using the composer.json file that I attached in the previous section, then you don't have to worry about installing this library because TwitterOAuth will already be copied on your PHP app's vendor folder.
Otherwise, you can always download this PHP component from GitHub. With the OAuth flow in view, we only have to program the method for tweeterers to apply permissions to Laplacianlab's SDSS and the oauth callback. For now our SDSS is so simple that we can write everything in the index.php file.
public\index.php
<?php // Bootstrap logic require_once '../config/config.php'; set_include_path(get_include_path() . PATH_SEPARATOR . APPLICATION_PATH . '/vendor/'); set_include_path(get_include_path() . PATH_SEPARATOR . APPLICATION_PATH . '/model/'); require_once 'slim/slim/Slim/Slim.php'; require_once 'abraham/twitteroauth/twitteroauth/twitteroauth.php'; require_once 'DBConnection.php'; use \Slim\Slim; session_start(); Slim::registerAutoloader(); $app = new Slim(); $app->response->headers->set('Content-Type', 'application/json'); // RESTful API methods // POST signal/add $app->post('/signal/add', function() { $request = Slim::getInstance()->request(); $signal = json_decode($request->getBody()); $sql = 'INSERT INTO signals(id_ea, symbol, operation, value) VALUES (' . mysql_real_escape_string($signal->ea_id) . ",'" . mysql_real_escape_string($signal->symbol) . "','" . mysql_real_escape_string($signal->operation) . "'," . mysql_real_escape_string($signal->value) . ')'; DBConnection::getInstance()->getHandler()->query($sql); $signal->id = DBConnection::getInstance()->getHandler()->insert_id; echo json_encode($signal); }); // More API methods here!.., according to your API spec // Twitter OAuth flow // This method is for users to give permissions to Laplacianlab's SDSS to tweet // on their behalf. $app->get('/tweet-signals', function() use ($app) { if (empty($_SESSION['twitter']['access_token']) || empty($_SESSION['twitter']['access_token_secret'])) { $connection = new TwitterOAuth(API_KEY, API_SECRET); $request_token = $connection->getRequestToken(OAUTH_CALLBACK); if ($request_token) { $_SESSION['twitter'] = array( 'request_token' => $request_token['oauth_token'], 'request_token_secret' => $request_token['oauth_token_secret'] ); switch ($connection->http_code) { case 200: $url = $connection->getAuthorizeURL($request_token['oauth_token']); // redirect to Twitter $app->redirect($url); break; default: echo '{"error":{"text":"Connection with Twitter failed"}}'; break; } } else { echo '{"error":{"text":"Error Receiving Request Token"}}'; } } else { echo '{"message":{"text":"Everything is ok! Laplacianlab\'s SDSS ' . 'can now tweet trading signals on your behalf. Please, if you no ' . 'longer want the SDSS to tweet on your behalf, log in your Twitter ' . 'account and revoke access."}}'; } }); // This is the OAuth callback $app->get('/twitter/oauth_callback', function() use ($app) { if(isset($_GET['oauth_token'])) { $connection = new TwitterOAuth( API_KEY, API_SECRET, $_SESSION['twitter']['request_token'], $_SESSION['twitter']['request_token_secret']); $access_token = $connection->getAccessToken($_REQUEST['oauth_verifier']); if($access_token) { $connection = new TwitterOAuth( API_KEY, API_SECRET, $access_token['oauth_token'], $access_token['oauth_token_secret']); // Set Twitter API version to 1.1. $connection->host = "https://api.twitter.com/1.1/"; $params = array('include_entities' => 'false'); $content = $connection->get('account/verify_credentials', $params); if($content && isset($content->screen_name) && isset($content->name)) { $_SESSION['twitter'] = array( 'id' => $content->id, 'access_token' => $access_token['oauth_token'], 'access_token_secret' => $access_token['oauth_token_secret'] ); // remove the request token from session unset($_SESSION['twitter']['request_token']); unset($_SESSION['twitter']['request_token_secret']); // Twitter's OAuth access tokens are permanent until revoked so // we try to update them when a tweeterer tries to give access // permissions again $sql = "SELECT * FROM twitterers WHERE twitter_id='$content->id'"; $result = DBConnection::getInstance()->getHandler()->query($sql); if($result->num_rows) { $sql = "UPDATE twitterers SET " . "access_token = '" . mysql_real_escape_string($access_token['oauth_token']) . "', " . "access_token_secret = '" . mysql_real_escape_string($access_token['oauth_token_secret']) . "' " . "WHERE twitter_id ='" . $content->id . "'"; } else { $sql = "INSERT INTO twitterers(twitter_id, access_token, access_token_secret) " . "VALUES ('" . mysql_real_escape_string($content->id) . "','" . mysql_real_escape_string($access_token['oauth_token']) . "','" . mysql_real_escape_string($access_token['oauth_token_secret']) . "')"; } DBConnection::getInstance()->getHandler()->query($sql); echo '{"message":{"text":"Everything is ok! Laplacianlab\'s SDSS ' . 'can now tweet trading signals on your behalf. Please, if you no ' . 'longer want the SDSS to tweet on your behalf, log in your Twitter ' . 'account and revoke access."}}'; } else { echo '{"error":{"text":"Login error"}}'; } } } else { echo '{"error":{"text":"Login error"}}'; } }); $app->run();
The most imporant thing here is storing in the database the two pieces of data that allow the SDSS to make authenticated calls to Twitter on behalf of the users who provided the access permissions. These two pieces are the access token and the access token secret, which are permanent until revoked by users. Thus, the Social Decision Support System never stores user credentials, and on the other hand, users can remove their access permissions whenever they want through revocation.
While the PHP-based SDSS have the correct access token and access token secret it can perform Twitter calls like the following:
// Let's assume there's an object named $user to access the tokens... $connection = new TwitterOAuth( API_KEY, API_SECRET, $user->getAccessToken(), $user->getAccessTokenSecret()); $message = "Hello world! I am Laplacianlab's SDSS and I am tweeting on behalf of a tweeterer."; $connection->post('statuses/update', array('status' => $message));
We have all the ingredients required. Now it is very easy to complete the SDSS API's resource POST signal/add that we started to code in 4. Programming the PHP RESTful Web Service, so it is left as an exercise for you to add the PHP code necessary to tweet some sample data.
Conclusion
This article has showed how to connect an MetaTrader 5 terminal with Twitter in order for you to tweet your EAs' trading signals. We have developed a computer-assisted system which combines the processing power of robots with the cognitive abilities of individuals. We want people to validate robotic trading signals which otherwise would be automatically placed on the market by the Expert Advisors. Thus,
we have opened a door to a new and exciting subject, which is capturing knowledge about automatic trading systems when people retweet or fav trading signals. This is a practical application based on the Human-based computation (HBC) paradigm.
Deploying a complete Social Decision Support System is time consuming, so this exercise can be seen as an idea for building a technology startup. We have started developing an SDSS which consists of three main parts:
- MetaTrader 5 terminal running on a VPS or a DS;
- PHP Twitter web app based on Slim;
- Twitter;
Specifically, in this first part we have implemented the architecture of the RESTful web service that communicates MetaTrader 5 with the Slim PHP web app, and we have also linked the PHP web app with Twitter through the OAuth protocol. In the next part we will code an MQL5-RESTful framework in the same way as explained in the article MQL5-RPC. Remote Procedure Calls from MQL5: Web Service Access and XML-RPC ATC Analyzer for Fun and Profit.