SocketTlsHandshake fails against server with self-signed certificate

[tandoori_mql]

Hi there,

I am currently trying to connect my expert advisor to a remote machine via the network functions SocketCreate, SocketConnect and secure it with SocketTlsHandshake. For that I created a self-signed certificate. Currently, the handshake fails with the expected error 5274 and I want to find out why that is.

The certificate I created has the following attributes (read out with openssl):

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            f7:51:c0:bf:ad:5a:56:03
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: C=CH, O=self-signed certificate, CN=self-signed certificate
        Validity
            Not Before: Jun  4 04:00:38 2026 GMT
            Not After : Jun  1 04:00:38 2036 GMT
        Subject: C=CH, O=self-signed certificate, CN=self-signed certificate
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (384 bit)
                pub:
                    04:b2:aa:48:bf:5c:06:33:48:9b:6f:98:f4:16:cc:
                    43:99:90:fb:ad:5f:ea:e8:6e:c0:87:35:94:a0:91:
                    55:ed:ef:8b:34:44:7a:aa:2d:8d:3d:22:bf:1d:f9:
                    0e:61:e9:b2:3b:60:dd:2e:5b:aa:64:89:7b:ef:06:
                    1f:ca:54:c9:1a:f1:03:13:ae:ab:ab:c2:2a:95:b9:
                    c8:61:9b:dc:ad:88:28:21:09:74:4e:e8:a1:62:dd:
                    41:84:c4:f9:4c:1d:71
                ASN1 OID: secp384r1
                NIST CURVE: P-384
        X509v3 extensions:
            X509v3 Key Usage: 
                Digital Signature, Data Encipherment, Key Agreement
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Basic Constraints: 
                CA:TRUE
            X509v3 Subject Alternative Name: 
                IP Address:127.0.0.1, IP Address:*another local ip address*
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:66:02:31:00:e5:0a:2c:a3:9f:5f:f4:dd:1f:34:0c:ce:92:
        78:d9:a8:43:66:32:2e:9b:ac:2b:22:44:c8:3b:87:3f:78:46:
        b3:ac:a6:29:77:9f:8c:b3:f0:f4:93:e5:01:ce:53:2b:6c:02:
        31:00:b2:4e:38:72:f9:f9:ea:c5:47:1e:f3:b9:88:db:6f:d3:
        e9:85:ab:db:47:b0:be:3e:29:88:d0:67:21:00:2b:42:0d:2a:
        cf:25:06:d7:62:ad:ab:cb:64:b4:ca:d3:09:06
-----BEGIN CERTIFICATE-----
*certificate information* 
-----END CERTIFICATE-----

I just basically copied how the google CA certificate looks like and added TLS authentication to the extended key usage. I then imported the certificate into the "Trusted Root Certification Authorities" certificate storage inside the Windows machine on which I run the MetaTrader5 instance.

My server runs on python, so I can see that the handshake fails because of an SSL error called "SSLV3_ALERT_ILLEGAL_PARAMETER", which is an error that was forwarded to python right over cpython directly from openssl, so the error does not happen inside the python code, but when openssl code is invoked. Another small remark is that if I try to connect to the python server using a python client, the handshake is successful, even when I'm connecting from a remote computer, so it has to be something in the implementation of the SocketTlsHandshake function.

Anyways, the certificate seems to have illegal parameters and I wanted to ask you guys if you know about the prerequisites of certificates in order for the expert advisor to be able to connect to the server. Is it maybe because it does not want to use CA certificates directly for authentication but rather a non-CA "server TLS certificate" and I got to have some kind of a certificate chain for it to work? Or are just some parameters inside the certificate given above missing? help.

Cheers,

tandoori

[tandoori_mql]

Hello again,

it seems like this topic is a bit of a dead spot in the community...

And unfortunately the error message I get from MT5 is not terribly helpful either. It is just error 5274, which is a generic handshake error, so I really do not get any insight as to what the certificate lacks in order to be accepted by MT5. Are there maybe any devs here, that can enlighten me? At the moment the only thing I could do is brute force through different settings for certificates and hope that at one point I get a certificate that fits the expectations, but since there are virtually thousands if not millions of combinations, I would like to circumvent that at all costs. Is there really nobody that can give a small clue as to where one might search for a solution?

Cheers,

tandoori

[Ebunoluwa Abimb Owodunni]

MT5 can only be as a socket client, you may need to create a socket server bridge with python or node before it goes to your app.

[Stanislav Korotky]

Did you try to connect to your server with TLS from a browser first? Make sure the browser is satisfied with your sertificate, only then ask questions about MQL5, and provide a code fragment.

Some time ago I generated a selfsigned serificate (by external tools, without any trials and errors) and used it for the algotrading book - it worked normally from a browser and from MQL5. Of course, now it's outdated and marked by browsers by red color, but technically should work.

[tandoori_mql]

@Ebunoluwa Abimb Owodunni:

I run MT5 as a client. The server is on another machine and runs via Python. And I don't understand what you mean by a "socket server bridge". Do you mean a standard TCP connection? If yes, that is what I started out with via the SocketConnect function, which works perfectly fine.

@Stanislav Korotky:

Do you mean connect by browser as to "connect to a HTTPS site"? Because I just want to establish a TLS-encrypted connection to a general server, not an HTTPS-server via port 443. I did connect to the Python-server on the other machine succesfully. I did this from the same machine that MT5 runs on, but not via MT5 but by another Python shell (a client script). All of this worked fine, even the encryption, so I assume that the problem lies in MT5 not being able to validate my certificate. With the Python-client I can import the certificate from the general certificate store and then successfully connect to the server and encrypt the connection.

What kind of code fragment do you want? From the expert advisor? It would just be the two functions SocketConnect and SocketTlsHandshake run after each other, but I can provide that if you want.

Thanks for your replies, guys :)

Cheers,

tandoori