Retrieving a price stream via WebSocket in C#.

New comment
 

Hello, Dear Sirs. Interested in receiving quotes from different sources (including the LMAX exchange). Since different brokers are connected to different ECNs, liquidity providers, it is better to get quotes directly from the ECNs themselves. But there is a limitation there - most providers will not share their quotes with us unless we connect to them directly. But there are a couple of places where we can get quotes from, including "market depth". For example LMAX is streaming its liquidity in widgets like https://s3-eu-west-1.amazonaws.com/lmax-widget3/website-widget-vwap.html (it is inactive on weekends because there are no quotes, on weekdays the logo will disappear and a market place will appear). There are also a couple of ECNs that stream their quotes with market depth.

Quotes in widget are streamed via websocket, i.e. they cannot be obtained via WebRequest() directly, it is necessary to subscribe to websocket events. And here is where I'm in dark forest, as I'm not (nearly) familiar with C# and moreover with web technologies :) found 2 examples of getting data through web sockets, but none of them works properly.

Example: create client and connect to socket, looks like this. I received it (after digging in widget sources and trying to understand something), I can send a quote to widget, but not receive :) i.e. it's not clear which request should be sent to server to receive a quote. If we send the quotes themselves, the server will reply positively and they will appear in the widget. In the screenshots you can see the server's reply, if you look through the socket's thread in Chrome, you will see information about the connection. And when the market is active, the quotes sent will be downloaded and displayed in the widget :) I have another example, connection via socket, but it does not work so far.

If anyone has any idea how to receive a quote from widget (may be directly from MT and winapi, or C#), I would be thankful :)

using System;
using System.Threading.Tasks;
namespace SenseConsoleApp
{
    class Program
    {
        static void Main(string[] args)
        {
            GetDocList();
            Console.ReadLine();
        }
        static async Task<string> GetDocList()
        {
            var client = new SenseWebSocketClient(new Uri("wss://data-fix.smt-data.com/lmax"));
            Console.WriteLine("Connecting to Qlik Sense...");
            Console.WriteLine("Getting document list...");
            var docs = await client.GetDocList();
            Console.WriteLine(docs);
            return docs;
        }
    }
}

using System;   
 using System.Net.WebSockets;  
 using System.Text;  
 using System.Threading;  
 using System.Threading.Tasks;

 namespace SenseConsoleApp  
 {
   
   public class SenseWebSocketClient  
   {  
     private ClientWebSocket _client;
    
     public Uri _senseServerURI;  
     public SenseWebSocketClient(Uri senseServerURI)  
     {  
       _client = new ClientWebSocket();
       _senseServerURI = senseServerURI;  
     }  
     public async Task<string> GetDocList()  
     {
            
       string cmd = "{\"channel\":\"/fixprof/depthmax/EURUSD\",\"data\":[\"EURUSD\",0,0,0,0,0,0,0,0,0,0,[[0,0],[0.0,0.0],[0,0],[0,0],[0,0],[0,0],[0,0],[0.0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0]],[[0,0],[0,0],[0,0],[0,0],[0.0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0]]],\"id\":\"56kcb\"}";  // это строка с запросом к серверу, т.е. по факту мы отправляем на него котировки
       await _client.ConnectAsync(_senseServerURI, CancellationToken.None);  
       await SendCommand(cmd);  
       var docList = await Receive();  
       return docList;  // ответ сервера
     }  
     private async Task ConnectToSenseServer()  
     {  
       await _client.ConnectAsync(_senseServerURI, CancellationToken.None);  
     }  
     private async Task SendCommand(string jsonCmd)  
     {  
       ArraySegment<byte> outputBuffer = new ArraySegment<byte>(Encoding.UTF8.GetBytes(jsonCmd));  
       await _client.SendAsync(outputBuffer, WebSocketMessageType.Text, true, CancellationToken.None);  
     }  
     private async Task<string> Receive()  
     {  
       var receiveBufferSize = 1536;  
       byte[] buffer = new byte[receiveBufferSize];  
       var result = await _client.ReceiveAsync (new ArraySegment<byte>(buffer), CancellationToken.None);  
       var resultJson = (new UTF8Encoding()).GetString(buffer);  
       return resultJson;  
     }  
   }  
 }  

This frame displays a stream of quotes and other info like my connection at active time:

 

I don't understand everything. But let's wait for Monday, and quote activity. I will try to look too. Other direct sources of quotes are also interesting. No delays, by the way?

About 5-6 years ago, I was getting real-time information (not the market) from some website, through VBA Excel. Now I don't remember the details of how it was done. I connected via my browser window with a script that interacted with the information page. The script was communicating with the page, and then it was sent to Excel.

Actually, it's an interesting question.

 
Yuriy Asaulenko:

I don't get it all. But let's wait for Monday, and quote activity. I'll try to see. Other direct sources of quotes are also interesting. Are they without delays, by the way?

About 5-6 years ago, I was getting real-time information (not the market) from some website, through VBA Excel. Now I don't remember the details of how it was done. I connected via my browser window with a script that interacted with the information page. The script was already working with the page and was feeding it to Excel.

Actually, it's an interesting question.

Yes, I have to wait for Monday. There you will immediately see what you can cram into the widget zeros through the proposed query :) Seems to be without delays, compared with quotes in mt4, come quickly...

Interesting question as it is possible to pull not only from widgets but also from web terminals...

 
Maxim Dmitrievsky:

The question is interesting, since you can pull not only from widgets, but also from web terminals...

If possible, please send me some other links. Not every page can be used to pull information. Connect - you connect, but the format of the data is written in, say, Flash. And paragraph.
 
LMAX has an API - you don't need any widgets.
 
Dmitriy Skub:
LMAX has an API - you don't need any widgets.
so it's all for dough. Any market date from a vendor costs dough, or an invoice from 10k.
 
Yuriy Asaulenko:
If possible, please send me some other links. Not every page can be used to pull information from it. Connect - you connect, but the data format is stored in, say, Flash. And paragraph.
I've done it. Well, here we check it... Basically, a socket is visible in this case, a thread is visible... The question is how to parse it correctly. Of course, I'm not an expert but I believe that all that can be seen with the eyes can be obtained by software :) Flash will also pull data from somewhere...
 
Dmitriy Skub:
LMAX has an API - you don't need any widgets.
I suspect you need an account for the API.
 
Maxim Dmitrievsky:
I've thrown it. Well here we check it... in principle a socket is visible in this case, a thread is visible... Question how to parse it correctly, of course I'm not an expert but I have an opinion that all that is visible to the eyes can be obtained by software :) Flash will also pull data from somewhere...
maybe obtained with a known exchange protocol. You won't get it from Flash, or it's akin to cryptography. :)
 

Connection example number 2. According to the rules, in order to connect to a socket, you need to send headers to switch from http to wss, and then already communicate with the socket via tcp... I don't understand why such a connection is made via Socket when it's much easier via WebSocket. The error here is incorrect request, i.e. no handshake happens, I think I filled in all the headers correctly. Maybe the error is that connection comes with ssl, and it's necessary to get GetStream, not GetSslstream... Since ws is an unsecured connection, and wss is secured. And there are many other "maybes", it's hard to understand without a bottle, or without your help :) At the top of the code is a link to the original example, which I've already modified somewhat.

//stackoverflow.com/questions/2064641/is-there-a-websocket-client-implemented-for-net
using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Text;

namespace ssoc
{
    class Program
    {
        static void Main(string[] args)
        {
            Uri LMAXuri = new Uri("wss://data-fix.smt-data.com/lmax"); // путь к сокету
            Dictionary<string, string> headers = new Dictionary<string, string>(); //список хедеров для передачи серверу

            headers.Add("Host", "data-fix.smt-data.com");
            headers.Add("Connection", "Upgrade");
            headers.Add("Pragma", "no-cache");
            headers.Add("Cache-Control", "no-cache");
            headers.Add("Upgrade", "websocket");
            headers.Add("Origin", "https://s3-eu-west-1.amazonaws.com");
            headers.Add("Sec-WebSocket-Version", "13");
            headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36");
            headers.Add("Accept-Encoding", "gzip, deflate, sdch");
            headers.Add("Accept-Language", "ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4");
            headers.Add("Cookie", "connect.sid=s%3ALwAlO60b6DkqtFUZldvHp-kR.K3qfytDjKCo4cz0%2FxxHpZk1qGCyS9AWFPDv3ro2Yu%2BU");
            headers.Add("Sec-WebSocket-Key", "Kd6XNibByshdJKZA8qWmDA==");
            headers.Add("Sec-WebSocket-Extensions", "permessage-deflate; client_max_window_bits");

            WebSocket ws = new WebSocket(LMAXuri);
            ws.SetHeaders(headers); // установим свои хедерыб иначе будут исп. умолчания
            ws.Connect();
            string result = ws.Recv();
            Console.WriteLine(result);
            Console.ReadLine();

        }
    }

    public class WebSocket
    {
        private Uri mUrl;
        private TcpClient mClient;
        private NetworkStream mStream;
        private bool mHandshakeComplete;
        private Dictionary<string, string> mHeaders;

        public WebSocket(Uri url)
        {
            mUrl = url;

            string protocol = mUrl.Scheme;
            if (!protocol.Equals("ws") && !protocol.Equals("wss"))
                throw new ArgumentException("Unsupported protocol: " + protocol);
        }

        public void SetHeaders(Dictionary<string, string> headers)
        {
            mHeaders = headers;
        }

        public void Connect()
        {
            string host = mUrl.DnsSafeHost;
            string path = mUrl.PathAndQuery;
            string origin = "http://" + host;
            
            mClient = CreateSocket(mUrl);
            mStream = mClient.GetStream();

            int port = ((IPEndPoint)mClient.Client.RemoteEndPoint).Port;
            
            if (port != 443)
                host = host + ":" + port;

            StringBuilder extraHeaders = new StringBuilder();
            if (mHeaders != null)
            {
                foreach (KeyValuePair<string, string> header in mHeaders)
                    extraHeaders.Append(header.Key + ": " + header.Value + "\r\n");
            }

            string request = "GET " + path + " HTTP/1.1\r\n" + extraHeaders.ToString() + "\r\n";
            byte[] sendBuffer = Encoding.UTF8.GetBytes(request);

            mStream.Write(sendBuffer, 0, sendBuffer.Length);

            StreamReader reader = new StreamReader(mStream);
            {
                string header = reader.ReadLine();
                Console.WriteLine(header);
                if (!header.Equals("HTTP/1.1 101 Web Socket Protocol Handshake"))
                    throw new IOException("Invalid handshake response");

                header = reader.ReadLine();
                if (!header.Equals("Upgrade: WebSocket"))
                    throw new IOException("Invalid handshake response");

                header = reader.ReadLine();
                if (!header.Equals("Connection: Upgrade"))
                    throw new IOException("Invalid handshake response");
            }

            mHandshakeComplete = true;
            Console.WriteLine("Сервер пожал нам руку: ", mHandshakeComplete);
        }

        public void Send(string str)
        {
            if (!mHandshakeComplete)
                throw new InvalidOperationException("Handshake not complete");

            byte[] sendBuffer = Encoding.UTF8.GetBytes(str);

            mStream.WriteByte(0x00);
            mStream.Write(sendBuffer, 0, sendBuffer.Length);
            mStream.WriteByte(0xff);
            mStream.Flush();
        }

        public string Recv()
        {
            if (!mHandshakeComplete)
                throw new InvalidOperationException("Handshake not complete");

            StringBuilder recvBuffer = new StringBuilder();

            BinaryReader reader = new BinaryReader(mStream);
            byte b = reader.ReadByte();
            if ((b & 0x80) == 0x80)
            {
                // Skip data frame
                int len = 0;
                do
                {
                    b = (byte)(reader.ReadByte() & 0x7f);
                    len += b * 128;
                } while ((b & 0x80) != 0x80);

                for (int i = 0; i < len; i++)
                    reader.ReadByte();
            }

            while (true)
            {
                b = reader.ReadByte();
                if (b == 0xff)
                    break;

                recvBuffer.Append(b);
            }

            return recvBuffer.ToString();
        }

        public void Close()
        {
            mStream.Dispose();
            mClient.Close();
            mStream = null;
            mClient = null;
        }

        private static TcpClient CreateSocket(Uri url)
        {
            string scheme = url.Scheme;
            string host = url.DnsSafeHost;

            int port = url.Port;
            if (port <= 0)
            {
                if (scheme.Equals("wss"))
                    port = 443;
                else if (scheme.Equals("ws"))
                    port = 80;
                else
                    throw new ArgumentException("Unsupported scheme");
            }
            if (scheme.Equals("ws"))
            {
                throw new NotImplementedException("SSL support not implemented yet");
            }
            else
                return new TcpClient(host, port);
        }
    }
}
 
Maxim Dmitrievsky:
so it's all for dough. Any market date from a vendor costs dough, or an invoice of 10k or more.
Well, yes, they are greedy, what can you do)). But if you want to arbitrate, api is the only option available.
123456789
New comment
Reason: