Market Simulation (Part 12): Sockets (VI)

Introduction

In the previous article, "Market  Simulation (Part 11): Sockets (V)", we explained how to create a Python application for use in Excel. The purpose of this application was to demonstrate how to build an echo server in Python. Its distinguishing feature was that data related to connection and disconnection events were displayed directly in Excel.

In truth, this server is not particularly useful for us, mainly because it allows only a single connection—and servers designed to handle only one connection are of limited practical value. However, I do not want you to dwell too much on this detail. The intent was to show how a script written in Python can operate transparently within Excel. But for our actual needs, the server must be somewhat more advanced. To achieve this, we will need to implement several additional features.

The goal here is not to create a production-ready application. As previously mentioned, sockets are a highly complex topic that requires a significant amount of time to study and explore. Do not expect to be able to develop something truly robust and polished in just one day. When dealing with sockets, it is necessary to delve into many details—some of them are simpler, while others are considerably more intricate.

However, since our application requires a more sophisticated solution to enable communication between Excel and MetaTrader 5, it was decided to ease the learning curve slightly. This is because the aim is to show everyone who is just beginning to explore the topic of sockets how things can actually work in practice.

I apologize to those who are more experienced if this seems like repetition or if the article does not provide new insights. But this material will be very useful for beginners, especially since we will be doing everything in pure Python.

In this article, we will not be working directly with Excel or MQL5. However, in the case of MQL5, we will be able to use it. More precisely, we will make use of what has been developed earlier in this same series. For a complete understanding of the material, it will also be helpful to review what has already been implemented in MQL5.

If you are interested, you may read the following articles:

• Market Simulation (Part 08): Sockets (II)
• Market Simulation (Part 09): Sockets(III)

It was in these two articles that we created what will serve as the foundation for this one. In those articles, we developed a mini-chat that allows MetaTrader 5 users to exchange information via text messages. The thing is, back then the server code was written in C++. Now we will develop a similar server, but this time in Python.

Let's first clarify a few moments

Converting an echo server into a mini-server does not initially seem like a complicated task. Similarly, adapting an echo server to the form we need for communication between Excel and MetaTrader 5 is also not particularly difficult — at least for those who already possess sufficient experience and knowledge. However, when it comes to sockets, especially when multiple clients need to use the same server simultaneously, the situation becomes somewhat more complex. This is because many people, particularly beginners, have no real idea what actually needs to be done.

Let's go back to the Python code discussed in the previous article. It can be seen below:

01. import socket
02. import xlwings as xw
03. 
04. def Echo():
05.         wb = xw.Book.caller()
06.         server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
07.         server.bind(("127.0.0.1", 27015))
08.         wb.sheets[0]['A1'].value = "Waiting connection..."
09.         server.listen(1)
10.         client, addr = server.accept()
11.         client.send("Welcome to Server in Python\n\r".encode())
12.         wb.sheets[0]['A2'].value = str(f"Client connected by {addr}")
13.         while True:
14.             info = client.recv(512)
15.             if not info:
16.                 break
17.             client.send(b"Echo Server:" + info)
18.         wb.sheets[0]['A3'].value = "Client disconnected"
19.         client.close()

Python Server

Although this code will not work without the material presented in the previous article, it can be made functional and subsequently used as a module. Given that Excel will not be used in conjunction with Python, certain parts of the code will need to be removed in order to implement these changes. The code will then look like this:

01. import socket
02.  
03. def Echo():
04.     server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
05.     server.bind(("127.0.0.1", 27015))
06.     server.listen()
07.     client, addr = server.accept()
08.     client.send("Welcome to Server in Python\n\r".encode())
09.     while True:
10.         info = client.recv(512)
11.         if not info:
12.             break
13.         client.send(b"Echo Server:" + info)
14.     client.close()
15. 
16. if __name__ == "__main__":
17.     Echo()

Python Server

Well. If you're already somewhat familiar with Python, you’ve probably noticed what’s going on here. But if you're not familiar with Python, you might be thinking: «What is this madness?» In fact, there’s nothing insane about it. It's simply that the previous code, which could not be executed directly by the Python interpreter, can now do so. This became possible with the addition of lines 16 and 17—yet the code can still be used inside a module, precisely because of these lines.

But that’s not what we’re discussing here. The question is how to make the presented code capable of accepting multiple clients, even though, in principle, it currently cannot.

Well. If you've studied the reference materials included in the previous articles, you should already understand some of the details and issues present in this code. This is because it is essentially designed to accept only one connection at a time. But how is that possible? It’s due to the fact that the accept function in line 07, once executed, will not run again. This prevents additional clients from connecting. Even if the same client closes the connection, they will not be able to reconnect.

The reason is the same: the accept function will no longer be executed. «So, is it possible to add another loop to this code so that the content between lines 07 and 14 runs again if the client decides to return?» Yes, it is possible. And indeed, in some simpler cases, that’s exactly what a programmer would do. However, let’s think a bit further ahead, because even if we implement that solution, we would still only have one connected client. And for the purpose of our mini-chat, that will not be sufficient.

In addition to this issue, there is another one. If you tested what was presented in the previous two articles, you may have noticed that Excel becomes practically unresponsive when the server is in use. Among the mentioned articles are the following:

• Market Simulation (Part 10): Sockets (IV)
• Market Simulation (Part 11): Sockets (V)

This is for those who haven’t yet read those articles. And now the question arises: why does Excel become difficult to use while the echo server is running? To answer this, it is necessary to understand what is happening inside the server code. One must keep in mind that Excel calls, via VBA, the Python code that needs to be executed. When the Python code starts running, it begins competing with Excel for CPU usage.

To the operating system, the Python code is not a separate process. In fact, it is more like a thread of Excel. The word "thread" is not actually the most appropriate term to use in this case, since a thread does not typically compete with the main program for CPU time. However, because closing Excel also terminates the Python code, we can cautiously use the word "thread" here. But one should be careful with this terminology, and the reason for this will be explained a little later.

Nevertheless, a fair question remains: why does the Python code compete for CPU usage when Excel has its own time window for processing? This is due to two specific points in the code. Let’s return to the original echo server code. Notice that in line 10 there is a call to the accept function. This function is getting blocked. That is, when executed, the code stops at this point and does not proceed until a connection is established. This function represents the first bottleneck—the first instance where the Python script will compete with Excel for CPU usage.

Once a connection is established, a second problem arises. This occurs exactly at line 14, where the Python script again competes with Excel. This competition repeats with every interaction involving line 14. As a result, Excel receives less attention than the Python script, making it difficult to use while the server is running.

But is there a way to solve this problem? Yes, there are several options. The first is to make the Python script a true thread of Excel. That way, when Excel enters its execution window, both Excel and the Python script would stop competing for CPU usage. This happens because the OS schedules tasks so that each receives a portion of processing time. In the case of multi-core processors, the OS could allow Excel to use one core and the Python script another. However, for this to happen, Excel must inform the OS that the Python script is to become a thread.

But what kind of thread are we actually talking about? To understand this, let’s move on to a new topic.

Let's understand what a thread is

Roughly speaking, a thread is like another program existing inside the main program. Indeed, that is a simple way to describe a thread. A thread exists only as long as the main program is running. When the main program—in our case, Excel—ceases to exist, everything associated with it also terminates.

The concept of threads emerged alongside parallel processing. In fact, that is precisely the purpose of a thread: it helps the main program execute tasks that would be very difficult to implement directly within the main code. To achieve this, we place a portion of the code into a segment that runs in parallel with the main program.

Once the thread finishes its work, the main program can use the computed or received data. This data will be available in the area we reserved for it. This is a quite interesting form of programming, because by using a thread, you can perform a task while allowing the main program to continue running, even if the task assigned to the thread takes several minutes to complete.

This capability—both in Python and in other languages—is very compelling. However, to truly understand what is happening and what the advantage of using threads is, it is necessary to see it in action. In order not to leave the topic without a practical example, we will look at a case of using threads in Python.

01. import socket as sock
02. import threading as thread
03. 
04. def NewClientHandler(conn, addr):
05.     print(f"Client [%s:%s] is online..." % addr)
06.     while True:
07.         msg = conn.recv(512).decode().rstrip(None)
08.         if msg:
09.             print(f"Message from [%s:%s]: {msg}" % addr)
10.             if msg.lower() == "/see you later":
11.                 break
12.     print(f"Client [%s:%s] is disconnected..." % addr)
13.     conn.close()
14. 
15. server = sock.socket(sock.AF_INET, sock.SOCK_STREAM)
16. server.bind(('localhost', 27015))
17. server.listen()
18. print("Waiting connections...")
19. while True:
20.     conn, addr = server.accept()
21.     conn.send("Welcome to Server.\n\r".encode())
22.     thread.Thread(target=NewClientHandler, args=(conn, addr)).start()

Python Server

This server, whose code is provided above, uses threads to handle multiple clients simultaneously. Please note a few important details. First, the server does not forward information provided by one client to others. Second, it is intended for educational purposes only—that is, to demonstrate the use of threads. However, if you understand what is happening, you will be able to create many interesting things simply by modifying the code above.

But let's first understand what is happening inside this code. If you are already familiar with Python, then you may need to be a little patient with beginner programmers. Remember that someone, at some point, taught you—or you learned from documentation written by others. Please be patient.

In line 01, we import the socket package. Notice that we are presenting things a bit differently. In line 02, we import the package that allows us to create threads. As with line 01, there is something here that may seem unusual, but don’t be alarmed. We are creating an alias to make things in Python more pleasant and understandable.

Between lines 04 and 13, there is a procedure. We will leave it aside for now and move to line 15. Notice that between lines 15 and 18, the code is almost identical to the previous examples in today’s article. This part is responsible for creating the server socket, so the code is the same. One important point: in these examples, we are using the TCP protocol. But it’s worth remembering that there are other network protocols, so the code may vary slightly from server to server, mainly due to certain aspects we will discuss later.

In any case, in line 18, we indicate that the server is ready to accept connections. Remember the detail mentioned earlier—the need to place a loop before the accept call? Well, here we are doing exactly that. Thus, even if a client disconnects, it will be able to reconnect to the server, because the loop allows multiple interactions with the accept call. Now let’s recall another detail: the accept call blocks the code and forces it to wait for a connection. When this happens, execution continues to line 21, which sends a welcome message to the new client, acknowledging that the connection has been established. Then, in line 22, a thread is created in Python.

This is where things get truly interesting, and where something very important emerges. If we placed the code between line 04 and line 13 here instead of at line 22, the server’s behavior would be completely different. That’s because the code would no longer allow more than one connection—it would accept only one.

«Wait, how is that possible?» To understand this, you need to realize that line 22 is not a magical line. In fact, it calls the procedure defined in line 04. But there is one key difference between a call that becomes a thread and one that does not. Once the call becomes a thread—as shown in this code—the OS does the following: it takes the thread code, allocates it in memory, and divides the execution time or window of the main code between the main code itself and the thread.

At first, this may be a bit confusing. But think of it this way: the time the OS allocates for program execution is like a chocolate bar. When a program has no threads, it can take the whole bar. But when it has threads, that bar is split into increasingly smaller pieces. Thus, the main program will only be able to consume one of these pieces, and the rest will be consumed by the threads. This would be a fairly simple explanation if our processor only had one core.

Returning to our code, note that after executing line 22, the program "splits" into two parts. One part immediately returns to line 19, where it waits again at line 20. It’s important to remember that the accept function blocks the code, preventing further execution. The other part will be directed to line 04.

Well. «Now I’m completely confused—how can the code in line 04 run if the code is blocked at line 20? This is very confusing». Calm down, dear readers—there’s no reason to be confused. Remember the chocolate bar analogy. The main code will be blocked at line 20, waiting for a new client connection. However, the code between lines 4 and 13 will execute independently of the new client connection. The important point is that the server will begin listening for information from the connected client. Therefore, we introduce a new loop in line 06. Pay attention to this, because if you don’t understand this part, you will become completely lost in how threads work.

This loop, which starts at line 06 and ends at line 11, will exist only within the thread. And each client—and this is important to understand—will have its own loop. Thus, even if two or more clients are connected, one will not interfere with the operation of another. Now imagine that each client has its own server, dedicated solely to that client. In other words, each client that is connected or will be connected will create a server for itself. And this server will respond only to that client.

That is the magic of threads. However, not all these wonders are perfect. There are problems that arise when using threads. Many of these problems appear precisely when we, as programmers, start using them indiscriminately. Always remember that a thread will take time away from the main program, even if it seems to be running in parallel. This is because no matter how many cores our processor has, they are not infinite. Therefore, excessive use of threads may consume all available cores, leading to increased execution time and, consequently, diminishing the advantage of using threads.

You may also be wondering something else. Although this code can handle multiple clients, it does not allow them to communicate with each other. Can’t information be passed between clients using threads? If that thought crossed your mind, it means that you still don’t fully understand how threads work. But don't be discouraged. On the contrary, you should feel satisfied, because you asked this question. If, while looking at the code and reading the article, you thought exactly that, it shows that you truly understood how the server works. However, the very fact that the question arose indicates that you haven’t yet fully mastered the concept of threads. So let's resolve these doubts. To separate the ideas, we will examine this in the next section.

Client-to-client communication is managed by the thread

To understand how this works, you need to grasp what has been covered up to this point. If you didn’t fully understand the previous topics, go back and read them again. It is important to comprehend them—otherwise, you will become even more confused in this section, which, although relatively short, may leave you bewildered and, even worse, without any understanding of what is actually happening.

Let's take a look at the code.

01. import socket as sock
02. import threading as thread
03. 
04. CONN_LIST = []
05. 
06. def NewClientHandler(conn, addr):
07.     global CONN_LIST
08.     CONN_LIST.append(conn)
09.     print(f"Client [%s:%s] is online..." % addr)
10.     while True:
11.         msg = conn.recv(512).decode().rstrip(None)
12.         if msg:
13.             print(f"Message from [%s:%s]: {msg}" % addr)
14.             for slave in CONN_LIST:
15.                 if slave != conn:
16.                     slave.send(f"[{addr[0]}]: {msg}\n\r".encode())
17.             if msg.lower() == "/see you later":
18.                 break
19.     print(f"Client [%s:%s] is disconnecting..." % addr)
20.     conn.close()
21.     CONN_LIST.remove(conn)
22. 
23. server = sock.socket(sock.AF_INET, sock.SOCK_STREAM)
24. server.bind(('0.0.0.0', 27015))
25. server.listen()
26. print("Waiting connections...")
27. while True:
28.     conn, addr = server.accept()
29.     conn.send("Welcome to Server.\n\r".encode())
30.     thread.Thread(target=NewClientHandler, args=(conn, addr)).start()

Python Server

«But what is this? You’ve only added a few lines of code—are you sure this allows clients to send messages to each other?» Yes, I assure you, this code not only enables information to be forwarded between clients, but also makes the server display those very messages. This can be seen in the command prompt window where the server is running.

However, there is one important aspect to note in this code: old messages will not be accessible. This means that only those clients currently connected to the server will receive messages. «But how exactly does this work? I’m looking at the code and I just don’t get it».

To understand how the server manages to transmit messages sent to other clients, you need to understand how threads work. We covered this in the previous section. If you’ve grasped how threads function, we can now move on to understanding how they communicate with each other. That's right. In order for a thread to send messages to other clients, they need to share information. But how is this done? In the simplest case, this happens through a shared memory area—this shared portion is explicitly defined in line 04. In other words, the list declared in line 04 will be visible to all threads.

To ensure everything runs smoothly, in line 07 of the thread we declare the list as global. This prevents Python from creating it locally. If Python were to create it locally, the memory would not be shared between threads, which is exactly what we need. In line 08, we add the connection that created the thread to our list. This step is crucial, because each new connection will generate a new thread and add a new entry to the list.

When line 11 is executed, it blocks while waiting to receive data from the connection—and this happens in each thread independently. As soon as a client sends something to the server, we enter a small loop at line 14. This is where the “magic” happens. Notice that we iterate through the list of connections that have been created by the individual threads. However, the loop is agnostic to this—since memory is shared between threads, it will traverse all connections one by one.

If the condition in line 15 is false (i.e., if the connection matches the connection belonging to the current thread’s loop), we do nothing. But if the test is true, line 16 executes. It sends to the connection being managed by another thread whatever data was received by this thread. I hope you understood this part. Remember: each thread corresponds to a connection. When a connection receives data, it triggers the loop to forward that data to other connections. However, this does not make the other threads become active.

«Wait a minute. How is that possible? If one thread sends data to other connections, that should make the other threads become active». No, dear reader. Your reasoning is flawed. Not entirely wrong, but you’ve overlooked the following: the server does not listen to what is happening inside the threads. It listens to what is happening on the client side. To each client, the server appears to be exclusively connected to a dedicated server. And the client has no idea that it is inside a thread.

Finally, when a client disconnects, we remove from the list the connection identifier that was added in line 21. As a result, all other threads will no longer see the recently closed connection.

Concluding thoughts

Although this may seem very confusing, the topic of threads is just as serious as the topic of sockets. However, this solution using threads is only one of many possible approaches. In the next article, we will discuss another way to solve this problem, since this thread-based solution does not resolve the Excel issue. This is because we still have the problem with the accept function blocking the server, making the experience of using Python in Excel not particularly pleasant—at least for those who need sockets.

File	Description
Experts\Expert Advisor.mq5	Demonstrates interaction between Chart Trade and the expert advisor (requires Mouse Study for interaction).
Indicators\Chart Trade.mq5	Creates a window for configuring the order to be sent (requires Mouse Study for interaction).
Indicators\Market Replay.mq5	Creates controls for interacting with the replay/simulation service (requires Mouse Study for interaction).
Indicators\Mouse Study.mq5	Provides interaction between graphical controls and the user (required for both the replay system and real market operation).
Services/Market Replay.mq5	Creates and maintains the market replay/simulation service (main file of the entire system).
Code VS C++ Server.cpp	Creates and maintains a socket server developed in C++ (mini-chat version).
Code in Python\Server.py	Creates and maintains a Python socket for communication between MetaTrader 5 and Excel.
ScriptsCheckSocket.mq5	Allows testing of the connection with an external socket.
Indicators\Mini Chat.mq5	Implements a mini-chat via an indicator (requires the server to function).
Experts\Mini Chat.mq5	Implements a mini-chat in an expert advisor (requires the server to function).