Can't use pipes in Strategy Tester when using Network Farm Agents?

 

Hi All,

Sorry for another pipe vs. http dll question, but I don't think it's been asked before whether you can use pipes in Strat Tester on remote agents in your own network farm.

I've been trying to run an EA that communicates through a named pipe (using the stdLib CFilePipe), and it all works in Strategy Tester as long as you use 'Local Agents'.

However, when I try to run them using 'Local Network Farm' agents, the pipe fails. I have the pipe servers running on the other machines in the Network farm, so you would think it would connect. They aren't trying to connect to pipes remotely, just a local pipe named '\\\\.\\pipe\\fx-queue' on each machine.

Even if I set up my local machine as using Local Network Farm agents, (by enabling them locally in MetaTester and disabling the 'Local Agents', it still won't open the pipes when it runs. It just seems like we can't open pipes in a Network Farm.

Has anyone seen this or been able to get around it? Am I missing something basic here?

I thought I was 'doing the right thing' using pipes instead of http or an  MQ client with a dll, so this is a bit of a killer for me. I might have to go back to that.


Appreciate any input,

Regards.

 
I don't know if it is possible, but

- What's the error message from File open?
- Have you checked the Firewall of the server?
- What are your permissions on the pipe provided by your server?

As far as I am aware, the pipe needs to be set up properly for remote access.

Also, maybe you can share the relevant code.
 

Hi,

I actually got it working, but with a very simple version. Network Farm Agents on the remote machine talking to a pipe server on the remote machine.

I think the problem was too many connections to the pipe server from 8xAgents starting one time, and it just messed everything up. And I only had 1 pipe server instance running. Every now and then I would get the following error in the remote machines agent's logs:

NM      0       13:26:05.159    127.0.0.1       tester forced to stop
GF      0       13:26:05.160            optimize Experts\PipeTest.ex5 on AUDUSD.a,M1 thread finished
EJ      0       13:26:05.164    127.0.0.1       prepare for shutdown
JE      0       13:26:22.715    127.0.0.1       login (build 3180)
QS      0       13:26:22.724    127.0.0.1       tester forced to close
OE      0       13:26:22.727    127.0.0.1       stop immediately
PN      0       13:26:22.937    Server  MetaTester 5 stopped
The the only way to fix it was to restart the machine. The agents would even stop writing to their logs, so I think it's something to do with messing up their file writing (a Pipe is an in-memory file I suppose). Restarting the services or the pipe server wouldn't fix it. Maybe the pipe name became corrupt and a reboot fixed it, don't know. However once it got working on a reboot, it would keep working.


I'm still having trouble getting it working with my main pipe server, it's a bit more complicated. It's all a bit flaky, so I'm thinking of changing it over to mq. My pipe servers were only just creating messages and sending them to RabbitMQ anyway, so I might as well start the message directly from a dll rather than pipes.


Thanks for your response though,

Regards.

 

Hi Trev, sorry to resurect this thread, but can you explain how you got it working? How were you able to connect a remote agent to a remote pipe server?

 
You might be facing "blocking behaviour" of named pipes.

For deep technical information about named pipes please refer to:
 
Soewono Effendi #:
You might be facing "blocking behaviour" of named pipes.

For deep technical information about named pipes please refer to:

Thanks for your reply, but as stated in my post here https://www.mql5.com/en/forum/475801, my issue is MT5 giving me an error 4014 (ERR_FUNCTION_NOT_ALLOWED) on the FileOpen call. And because I have successfully communicated with the pipe server1 from server 2 from a Nodejs script, I know it's not an access issue, or a pipe configuration issue.


 

Using named pipes in remote agents on the same network
Using named pipes in remote agents on the same network
  • 2024.11.03
  • Mike D-rock
  • www.mql5.com
Hello everyone, My setup : Server1, has 74 local agents, this is where I start the optimization process Server2, has 74 local agents, which are add...
 
Mike D-rock #:

Thanks for your reply, but as stated in my post here https://www.mql5.com/en/forum/475801, my issue is MT5 giving me an error 4014 (ERR_FUNCTION_NOT_ALLOWED) on the FileOpen call. And because I have successfully communicated with the pipe server1 from server 2 from a Nodejs script, I know it's not an access issue, or a pipe configuration issue.

Please show exact problematic code with FileOpen.

 
Stanislav Korotky #:

Please show exact problematic code with FileOpen.

    ResetLastError();
    int handle = FileOpen(in_pipe, FILE_WRITE | FILE_READ | FILE_BIN);
    if (handle == INVALID_HANDLE)
    {
        Print("Failed to connect to the named pipe:", GetLastError());
        return "";
    }

In this context, in_pipe == "\\DESKTOP-9ERDSAF\pipe\MT5ConfigPipe".   DESKTOP-9ERDSAF being the name of Server1 that started the pipe as \\.\pipe\MT5ConfigPipe

 
Mike D-rock #:
FileOpen

Try to add FILE_SHARE_READ | FILE_SHARE_WRITE flags.

Also make sure your pipe server creates pipes in duplex mode.
 
Mike D-rock #:

In this context, in_pipe == "\\DESKTOP-9ERDSAF\pipe\MT5ConfigPipe".   DESKTOP-9ERDSAF being the name of Server1 that started the pipe as \\.\pipe\MT5ConfigPipe

You probably need to escape the backslashes:

\\\\DESKTOP-9ERDSAF\\pipe\\MT5ConfigPipe
 
Stanislav Korotky #:

You probably need to escape the backslashes:

Yes they are in the actual code.

Stanislav Korotky #:

Try to add FILE_SHARE_READ | FILE_SHARE_WRITE flags.

Also make sure your pipe server creates pipes in duplex mode.

Tried it and I get the same error code 4014 (ERR_FUNCTION_NOT_ALLOWED) on the FileOpen call.

This is how I create the named pipe in C++

HANDLE hPipe = CreateNamedPipeW(
            PIPE_NAME,
            PIPE_ACCESS_DUPLEX,
            PIPE_TYPE_BYTE | PIPE_READMODE_BYTE | PIPE_WAIT,
            PIPE_UNLIMITED_INSTANCES,
            BUFFER_SIZE,
            BUFFER_SIZE,
            0,
            &in_sa
        );

in_sa is the security attributes I create specifically for anonymous access (which allowed me to talk to the pipe server from Server2 in a nodejs script) :

SECURITY_ATTRBIUTES CreateSecurityAttributes() {
    SECURITY_ATTRIBUTES sa;
    sa.nLength = sizeof(SECURITY_ATTRIBUTES);
    sa.bInheritHandle = FALSE;

    // Define a security descriptor string that allows access for Everyone and Anonymous Logon
    LPCWSTR sd = L"D:(A;OICI;GRGW;;;AN)(A;OICI;GRGW;;;WD)";
    
    // Convert the string to a security descriptor
    if (!ConvertStringSecurityDescriptorToSecurityDescriptorW(sd, SDDL_REVISION_1, &sa.lpSecurityDescriptor, NULL)) {
        std::cerr << "Failed to create security descriptor. Error: " << GetLastError() << std::endl;
        sa.lpSecurityDescriptor = NULL;
    }

    return sa;
}