SocketTlsHandshake fails against server with self-signed certificate

tandoori_mql 2026.09.06 12:11 #1

Hello again,

it seems like this topic is a bit of a dead spot in the community...

And unfortunately the error message I get from MT5 is not terribly helpful either. It is just error 5274, which is a generic handshake error, so I really do not get any insight as to what the certificate lacks in order to be accepted by MT5. Are there maybe any devs here, that can enlighten me? At the moment the only thing I could do is brute force through different settings for certificates and hope that at one point I get a certificate that fits the expectations, but since there are virtually thousands if not millions of combinations, I would like to circumvent that at all costs. Is there really nobody that can give a small clue as to where one might search for a solution?

Cheers,

tandoori

Preparing a secure socket Accounts - MetaTrader 5 How to Use -

Ebunoluwa Abimb Owodunni 2026.09.06 13:19 #2

MT5 can only be as a socket client, you may need to create a socket server bridge with python or node before it goes to your app.

Stanislav Korotky 2026.09.06 15:34 #3

Did you try to connect to your server with TLS from a browser first? Make sure the browser is satisfied with your sertificate, only then ask questions about MQL5, and provide a code fragment.

Some time ago I generated a selfsigned serificate (by external tools, without any trials and errors) and used it for the algotrading book - it worked normally from a browser and from MQL5. Of course, now it's outdated and marked by browsers by red color, but technically should work.

MQL5 Book: Server component of web services based on the WebSocket protocol / Advanced language tools

www.mql5.com

To organize a common server component of all projects, we will create a separate folder Web inside MQL5/Experts/MQL5Book/p7/ . Ideally, it would be...

How to connect to MetaEditor - Professional editor When MetaTrader 5 Web

tandoori_mql 2026.09.11 08:54 #4

@Ebunoluwa Abimb Owodunni:

I run MT5 as a client. The server is on another machine and runs via Python. And I don't understand what you mean by a "socket server bridge". Do you mean a standard TCP connection? If yes, that is what I started out with via the SocketConnect function, which works perfectly fine.

@Stanislav Korotky:

Do you mean connect by browser as to "connect to a HTTPS site"? Because I just want to establish a TLS-encrypted connection to a general server, not an HTTPS-server via port 443. I did connect to the Python-server on the other machine succesfully. I did this from the same machine that MT5 runs on, but not via MT5 but by another Python shell (a client script). All of this worked fine, even the encryption, so I assume that the problem lies in MT5 not being able to validate my certificate. With the Python-client I can import the certificate from the general certificate store and then successfully connect to the server and encrypt the connection.

What kind of code fragment do you want? From the expert advisor? It would just be the two functions SocketConnect and SocketTlsHandshake run after each other, but I can provide that if you want.

Thanks for your replies, guys :)

Cheers,

tandoori

Ebunoluwa Abimb Owodunni - assetbase - Trader's profile

www.mql5.com

Trader's profile

Preparing a secure socket Checking socket status External Subversion client -

Stanislav Korotky 2026.09.11 18:27 #5

tandoori_mql #:

Do you mean connect by browser as to "connect to a HTTPS site"? Because I just want to establish a TLS-encrypted connection to a general server, not an HTTPS-server via port 443. I did connect to the Python-server on the other machine succesfully. I did this from the same machine that MT5 runs on, but not via MT5 but by another Python shell (a client script). All of this worked fine, even the encryption, so I assume that the problem lies in MT5 not being able to validate my certificate. With the Python-client I can import the certificate from the general certificate store and then successfully connect to the server and encrypt the connection.

What kind of code fragment do you want? From the expert advisor? It would just be the two functions SocketConnect and SocketTlsHandshake run after each other, but I can provide that if you want.

If you run a minimal web-server (powered by python or other publicly accepted engine, such as Node.js) with your TLS keys attached - can you connect to it by means of a browser? When you are paring 2 custom python scripts (client and server), this does not confirm standards. If it works, then - yes - we need to see your MQL5 code - I gave you a refernce for existing implementation which worked for me some time ago. But I'm not a python enthusiast, so can't help in this direction.

Encrypting My Work---Howto? Manual Martiangle possible to send MT4

tandoori_mql 2026.10.03 19:44 #6

Hello again,

Stanislav Korotky #:

If you run a minimal web-server (powered by python or other publicly accepted engine, such as Node.js) with your TLS keys attached - can you connect to it by means of a browser? When you are paring 2 custom python scripts (client and server), this does not confirm standards. If it works, then - yes - we need to see your MQL5 code - I gave you a refernce for existing implementation which worked for me some time ago. But I'm not a python enthusiast, so can't help in this direction.

I did that with the additional effort of building my own CA, signing a certificate request for my server, then importing the CA certificate into the certificate store of the machine that has MT5 on it and then trying to connect and using the newly signed certificate for the server. I tried to connect with my Firefox to the minimal web server I ran with the built-in python http.server package. It did connect via https without complaining.

Now the expert advisor code that still fails with error 5274 on the MT5 side and with "ssl.SSLError: [SSL: SSLV3_ALERT_ILLEGAL_PARAMETER] sslv3 alert illegal parameter (_ssl.c:1032)" on the python side:

int OnInit() {
    uint sock = SocketCreate(SOCKET_DEFAULT);
    SocketConnect(sock, "10.10.10.10", 12345, 1000);
    SocketTlsHandshake(sock, "10.10.10.10");
    return INIT_SUCCEEDED;
}

10.10.10.10 is the IP address that also appears in "X509v3 Subject Alternative Name" in the certificate and 12345 is the port number.

And I tried the STARTTLS message as described in the wiki page of the SocketTlsHandshake function but this only seems to be an option for SMTP connections and it fails with another error (python declines because of TLSv1) anyways.

Cheers, tandoori

Preparing a secure socket Extended Authentication - Accounts Network Functions - SocketTlsHandshake

Stanislav Korotky 2026.10.04 10:57 #7

tandoori_mql #:

Hello again,

I did that with the additional effort of building my own CA, signing a certificate request for my server, then importing the CA certificate into the certificate store of the machine that has MT5 on it and then trying to connect and using the newly signed certificate for the server. I tried to connect with my Firefox to the minimal web server I ran with the built-in python http.server package. It did connect via https without complaining.

Now the expert advisor code that still fails with error 5274 on the MT5 side and with "ssl.SSLError: [SSL: SSLV3_ALERT_ILLEGAL_PARAMETER] sslv3 alert illegal parameter (_ssl.c:1032)" on the python side:

10.10.10.10 is the IP address that also appears in "X509v3 Subject Alternative Name" in the certificate and 12345 is the port number.

And I tried the STARTTLS message as described in the wiki page of the SocketTlsHandshake function but this only seems to be an option for SMTP connections and it fails with another error (python declines because of TLSv1) anyways.

If it fails in python as well, then this is not an issue in MT5.

Please note, that I did not use certificates via importing them into the store, but attach them directly to the web-server, so I can't say if the store could somehow interfere while working with a custom certificate.

It would be better to handle errors on all steps, for example:

      handle = SocketCreate();
      timeout = t;
      TLS = scheme == "wss";
      if(handle != INVALID_HANDLE)
      {
         Print("Connecting to ", host, ":", port);
         if(SocketConnect(handle, host, port, timeout))
         {
            SocketTimeouts(handle, timeout, timeout);
          
            string subject, issuer, serial, thumbprint;
            datetime expiration;
            if(TLS)
            {
               if(!SocketTlsHandshake(handle, host))
               {
                  Print("TLS handshake error: ", _LastError);
                  TLS = false;
               }
               else
               {
                  Print("TLS enabled");
               }
               if(SocketTlsCertificate(handle, subject, issuer, serial, thumbprint, expiration))
               {
                  Print("TLS:");
                  Print("  Owner:      ", subject);
                  Print("  Issuer:     ", issuer);
                  Print("  Number:     ", serial);
                  Print("  Signature:  ", thumbprint);
                  Print("  Expiration: ", expiration);
               }
            }
         }
         else
         {
            Print("Can't connect: ", GetLastError());
         }
      }
      else
      {
         Print("Can't create socket: ", GetLastError());
      }

Currently it's not clear if you have a valid handle, and if it's really connected before the call to SocketTlsHandshake.

Also I'm not sure if the IP is actually operational - how did you acquire this exact IP?

Preparing a secure socket Network Functions Extended Authentication - For

tandoori_mql 2026.10.04 15:07 #8

Hi there,

first off: It did not fail in Python. As you suggested, I could connect to the python minimal web server with the self-signed certificate from a browser. I then tried to connect to the Python (not web- but MT5-) server that I want to connect to in the end but with another Python instance instead of via the MT5 expert advisor and this also works with TLS without any problem. The only time it fails is if I try to connect to the Python server with an MT5 expert advisor.

And I have a valid handle, since the Python server gave an SSL error, which must mean that the standard TCP connection was established successfully, but for your sake I made the same experiment with your modified code:

int OnInit() {
    uint handle = SocketCreate();
    uint timeout = 1000;
    string host = "10.10.10.10";
    uint port = 12345;
    bool TLS = true;
    if (handle != INVALID_HANDLE) {
        Print("Connecting to ", host, ":", port);
        if (SocketConnect(handle, host, port, timeout)) {
            SocketTimeouts(handle, timeout, timeout);
            string subject, issuer, serial, thumbprint;
            datetime expiration;
            if (TLS) {
                if(!SocketTlsHandshake(handle, host)) { 
                    Print("TLS handshake error: ", _LastError);
                    TLS = false;
                }
                else {
                    Print("TLS enabled");
                }
                if (SocketTlsCertificate(handle, subject, issuer, serial, thumbprint, expiration)) {
                    Print("TLS:");
                    Print("  Owner:      ", subject);
                    Print("  Issuer:     ", issuer);
                    Print("  Number:     ", serial);
                    Print("  Signature:  ", thumbprint);
                    Print("  Expiration: ", expiration);
                }
            }
        }
        else {
            Print("Can't connect: ", GetLastError());
        }
    }
    else {
        Print("Can't create socket: ", GetLastError());
    }
    return INIT_SUCCEEDED;
}

This code also gives back the same Python SSLV3_ALERT_ILLEGAL_PARAMETER error and the "experts" rider inside MT5 logs the following:

2026.10.04 16:38:49.493 test (EURUSD,M1) Connecting to 10.10.10.10:12345
2026.10.04 16:38:49.501 test (EURUSD,M1) TLS handshake error: 5274

And about the IP addresses: This is just an IP address out of the range of addresses reserved for private use (e.g. as in my case in my local network/LAN) and can be found under https://en.wikipedia.org/wiki/List_of_reserved_IP_addresses

Basically it's the same as the usual private LAN-network address range 192.168.0.0/16. I just configured my LAN so that it uses the range 10.0.0.0/8 (or, to be more precise, 10.10.10.0/24) instead of the standard from the sentence before which most routers use.

And here I would apply the same argument as above: If there would be any problem with the IP address, then the browser could not connect via https without any complaint, but this works.

About the certificate being imported into the store instead of directly attached to the server: First off, I could connect via browser, so this means that the certificate is accessible in the store. Second of all I would say that MT5 also uses this store (since I can't find a separate built-in store inside MT5).

To test this, I tried to connect to https://www.mql5.com:443 for fun and this worked as the log shows:

2026.10.04 17:02:26.785 test (EURUSD,M1) Connecting to https://www.mql5.com:443
2026.10.04 17:02:27.109 test (EURUSD,M1) TLS handshake error: 5274
2026.10.04 17:02:27.110 test (EURUSD,M1) TLS:
2026.10.04 17:02:27.110 test (EURUSD,M1) Owner: /CN=*.mql5.com
2026.10.04 17:02:27.110 test (EURUSD,M1) Issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=RapidSSL TLS ECC CA G1
2026.10.04 17:02:27.110 test (EURUSD,M1) Number: 02:8d:32:95:51:7f:76:ec:ec:cc:a2:a1:e8:84:da:69
2026.10.04 17:02:27.110 test (EURUSD,M1) Signature: 8afe3c3fa3ab2d02d112347ca46b49654bc375bf
2026.10.04 17:02:27.110 test (EURUSD,M1) Expiration: 2026.10.26 23:59:59

Here the handshake fails for the reason that when you connect to port 443 the connection is automatically upgraded to TLS by SocketConnect(), which means that another handshake fails, but of course we then get the info of the certificate. So what this means is that MT5 is capable of connecting to the certificate store, since otherwise we could not connect via TLS, since the root certificate that signed the certificate of mql5.com is stored in the certificate store.

I in the meantime assume that not Python is the problem, but that the SocketTlsHandshake function uses legacy protocols? How else should a SSLV3 error happen? SSLV3 is deprecated since 2015, TLS1.0 and TLS1.1 since 2021 (https://en.wikipedia.org/wiki/Transport_Layer_Security#History_and_development), so the ssl module in Python, which uses openssl can not establish a connection with a client that "only speaks" SSLV3. And openssl also tries to use the most modern protocol, so I would say that there is no other possibility of why an SSLV3 error should happen, except when the MT5 client can not speak any more modern protocols than SSLV3. Could this be the problem? It would certainly fit the message "illegal parameter", since SSLV3 is illegal security-wise. I am almost ready to assume that since most website providers don't care too much about security and prefer many people being able to use their service, they just allow deprecated protocols like SSLV3 to be used by their clients and this is why you can connect MT5 to most websites but not to my Python server?

Cheers, tandoori

Preparing a secure socket Network Functions - SocketTlsHandshake Network Functions

Alain Verleyen 2026.10.04 16:08 #9