Errors, bugs, questions - page 2377

New comment
 
Slava:
Yes. Any Print from OnInit

Thank you. Interesting, if I didn't happen to notice it, how would it be possible to find out about it...


ZS I'd leave it for local Agents only. In the Cloud, you can easily spam the log in this way.

 
fxsaber:

Thank you. Interesting, if I didn't happen to notice it, how would it be possible to find out about it...


ZS I'd leave it for local Agents only. In the Cloud, you can easily spam the log in this way.

In Cloud, it won't show up. Because there's no reason to.
 
Slava:

When you run genetics, do you optimise according to your custom criterion?

Based on the logs presented, OnTester returned 0 in all cases

I usually optimise according to my criterion, but here I tried the custom criterion as well. The result is the same.

OnTester returns 0, that's why it returns zeros in the results - that's understandable. The question is why it returns "0" on general run (optimization) but on single run from "zero results" (with the same parameters) it returns normal result, graph, etc.? I.e. something is not working in "Full overshoot" and yet genetics works fine. Any other thoughts/ideas?

 
Kuzmich:

Any other thoughts/ideas?

Pull all information of optimization pass in this way

Forum on trading, automated trading systems & strategy testing

MT5. STRATEGY TESTER. Divergence of testing and optimization results.

fxsaber, 2017.08.22 11:06

Insert these lines into the EA

#define  REPORT_TESTER // В тестере будут автоматически записываться отчеты
#include <Report.mqh>

And run Optimization. Then run a mismatched single run.

Then compare the saved two reports of the corresponding pass from Optimisation and the single pass.

The result of comparing the two reports will quickly reveal the causes.

 
Within the framework of getting acquainted with the Socket* functionality a number of questions to the current implementation arose.
The goal is to improve as much as possible what is done, I ask developers not to be offended by possible criticism.



1. I do not understand the reasons for such strong differences in "interfaces" for socket read functions
: a) For encrypted connection there are two functions for reading, and for unencrypted - one.
b) InSocketRead it is necessary to specifytimeout_ms explicitly, and inSocketTlsRead andSocketTlsReadAvailable
there is
no such parameter at all (set by separate function SocketTimeouts)
.
int  SocketTlsRead(int socket, uchar& buffer[], int buffer_maxlen);
int  SocketTlsReadAvailable(int socket, uchar& buffer[], int buffer_maxlen);

int  SocketRead(int socket, uchar& buffer[], int buffer_maxlen, uint timeout_ms);


2. The name of the function SocketIsReadable has nothing to do with what it actually performs:

bool  SocketIsWritable(const int  socket); // Return true if writing is possible, otherwise false.
uint  SocketIsReadable(const int  socket); // Number of bytes that can be calculated. In case of an error, 0 is returned.
bool  SocketIsConnected(const int socket); // New function without description. May be, it returns true if connection is not closed.

In fact, SocketIsReadable is analogous to ioctlsocket() function with FIONREAD flag in Ws2_32.dll


3. How can a user using Socket* functionality over a non-encrypted connection get a response from a server with minimum time delay, if the server doesn't break the connection after data transfer?

- The SocketIsReadable function without explicitly using a time delay (e.g. without Sleep) returns 0.
- the SocketRead function doesn't know how much to read, you specifybuffer_maxlen with a reserve - you'll have to wait fortimeout_ms

Yeah, that's how it's done:

- wait for 1 byte of data in SocketRead;
- then find out the size of the whole response using SocketIsReadable;
- read the remaining length into SocketRead;
- merge results by copying arrays:

#define  PRINT(x) Print(#x, ": ", string(x))
                
void OnStart() {
   string domain = "www.mql5.com";
   int port = 80;
 
   string request = "GET / HTTP/1.1\r\nHost: " + domain + "\r\n\r\n";
   char req[];
   
   int socket = SocketCreate();
   PRINT(SocketConnect(socket, domain, port, 5000));
   int len=StringToCharArray(request,req)-1;
   PRINT(SocketSend(socket,req,len));
   
   
   
   uchar resp[];
   uchar result[];
   
   int resp_result;
   uint resp_len;
   int start_write;
   
   
   resp_len = 1;
   resp_result = SocketRead(socket, resp, resp_len, 5000);
   if (resp_result <= 0){
      PRINT(GetLastError());
      return;
   }
   start_write = ArraySize(result);
   ArrayResize(result, start_write + resp_result);
   ArrayCopy(result, resp, start_write);
   
   
   resp_len = SocketIsReadable(socket);
   resp_result = SocketRead(socket, resp, resp_len, 5000);
   if (resp_result <= 0){
      PRINT(GetLastError());
      return;
   }
   start_write = ArraySize(result);
   ArrayResize(result, start_write + resp_result);
   ArrayCopy(result, resp, start_write);
   
   
   PRINT(CharArrayToString(result));
};

Isn't this too much code?


4. SocketIsReadable returns false information.
Turn off Internet and execute above code.
As a result, SocketIsReadable returns a sane value of 1. Wonders.


I managed to describe about one third of all questions and problems related to Socket*.
Unfortunately, I needed a lot of time to check, describe and double-check everything... (so that it's not a fact that there will be a sequel)

The general impression is that either everything was done in a big hurry, or the Socket* functionality got to the junior developer.
In any case, the current solution is very crude and covers rather narrow approach of using sockets.

 
MQL5\Include\Math\AlgLib\dataanalysis.mqh - CLinReg::LRLine does not work for 1M or more values?
 
Kuzmich:

I usually optimise according to my own criteria, but here I tried the standard ones as well. The result is similar.

OnTester returns 0, that's why there are zeros in the results - that's understandable. The question is why it returns "0" on general run (optimization) but on single run from "zero results" (with the same parameters) it returns normal result, graph, etc.? I.e. something is not working in "Full overshoot" and yet genetics works fine. Any other thoughts/ideas?

Can you share an EA (ex5 in private) and optimization conditions?

We want to reproduce the problem you mentioned.

After research the EA will be irrevocably erased

 
Slava:

Can you share the EA (ex5 in a private message) and the optimisation conditions?

We want to reproduce the problem you mentioned.

After research the EA will be irrevocably erased

Would you look at my EA? I have a similar problem - profit is not counted, consequently the optimisation does not work.
 
Slava:

Can you share the EA (ex5 in a private message) and the optimization conditions?

We want to reproduce the problem you mentioned.

EA will be irretrievably erased after research

Replyed in a private message.

 
Sergey Dzyublik:
Within the framework of getting acquainted with the Socket* functionality a number of questions to the current implementation arose.
The goal is to improve as much as possible what is done, I ask developers not to be offended by possible criticism.



1. I do not understand the reasons for such strong differences in "interfaces" for socket read functions
: a) For encrypted connection there are two functions for reading, and for unencrypted - one.
b) InSocketRead it is necessary to specifytimeout_ms explicitly, and inSocketTlsRead andSocketTlsReadAvailable
there is
no such parameter at all (set by separate function SocketTimeouts)
.


2. The name of the function SocketIsReadable has nothing to do with what it actually performs:

In fact, SocketIsReadable is analogous to ioctlsocket() function with FIONREAD flag in Ws2_32.dll


3. How can a user using Socket* functionality over a non-encrypted connection get a response from a server with minimum time delay, if the server doesn't break the connection after data transfer?

- SocketIsReadable function without explicit use of time delay (e.g. without Sleep) returns 0.
- SocketRead function doesn't know how much to read, if you specifybuffer_maxlen with reserve - you will have to wait fortimeout_ms

Yeah, that's how it's done:

- wait for 1 byte of data in SocketRead;
- then find out the size of the whole response using SocketIsReadable;
- read the remaining length into SocketRead;
- merge results by copying arrays:

Isn't this too much code?


4. SocketIsReadable returns false information.
Turn off the internet and execute the above code.
As a result, SocketIsReadable returns a sane value of 1. Wonders.


I managed to describe about one third of all questions and problems related to Socket*.
Unfortunately, I needed a lot of time to check, describe and double-check everything... (so that it's not a fact that there will be a sequel)

The general impression is that either everything was done in a big hurry, or the Socket* functionality got to the junior developer.
In any case, the current solution is very crude and covers rather narrow approach of using sockets.

1. This is the interface.

TLS functions are auxiliary to support complex cases. No problem with setting SocketTimeouts - these are the best ones to use.


2. It performs its function correctly.

You must not be aware of the problems with TCP connection breaking detection. It's quite difficult (resource intensive at the cost of extra calls) to detect that a connection is guaranteed to be broken correctly. All network implementations suffer from this problem.

Our implementation of SocketIsReadible is smart enough and has a break detector. When it detects a clean 0 bytes, it does the extra work of checking that the socket is complete:

   //+------------------------------------------------------------------+
   //| Доступно для чтения?                                             |
   //+------------------------------------------------------------------+
   UINT32 IsReadible(void)
     {
      unsigned long size=1;    // специально, чтобы убиться при попытке чтения, если сокет мертв
      //--- проверка
      if(m_socket!=INVALID_SOCKET)
        {
         //--- считаем количество доступных для чтения байт
         if(ioctlsocket(m_socket,FIONREAD,&size)!=0)
           {
            Close();
            return(1);        // вернем 1, чтобы убиться при попытке чтения
           }
         //--- если нет данных, проверим сокет на завершенность
         if(size==0)
           {
            timeval wait_time;
            fd_set  fd;
            //--- ждём
            FD_ZERO(&fd);
            FD_SET(m_socket,&fd);

            wait_time.tv_sec =0;          // секунды
            wait_time.tv_usec=1000;       // микросекунды
            //--- ждём
            if(select(0,&fd,NULL,NULL,&wait_time)>0)
               return(1);                 // вернем 1, чтобы убиться при попытке чтения
           }
        }
      //--- размер
      return(size);
     }

Since it returns the number of bytes without a termination flag, it outputs 1 byte so that a subsequent/imminent SocketRead read attempt will normally return an error.

Why is this correct? Because most of the code is written by programmers in this way:

if(SocketIsReadible(...)>0)
  {
   if(SocketRead( )<1)
     return(false);
   ...
  }
... уходим на следующий круг ожидания

the actual result of the operation is checked on a direct read attempt.


3. it needs to do SocketIsReadible() before the actual read, if you don't know the exact size of the data to be read.

The SocketisReadible/SocketRead bind gives you the ability to not lose control (minimize to almost zero loss of control) over the execution flow of your program. This avoids flying into network timeouts.

Yes, a few lines more code, but you won't lose control for a millisecond (roughly). It's up to you to decide what to do in the intervals of no network data.


4. explained in the second paragraph.

Issuing 1 for the sake of read and output stimulation as a read error.



Your conclusions are wrong.

This is the nature of TCP/IP transport, where there are no guarantees at all. You can get into network black holes there too on filters/firewalls when there is no TCP signalling part. Raw timeout and data flow control allows you to detect them and terminate connections yourself.

We've given a raw/direct access interface to network functions, including TLS implementations. If you use them, you are the one who needs to properly wrap the raw functions in a secure/controlled SocketIsReadible/SocketRead handler.

If you want to make high-level requests without having to think about the minutiae, there are WebRequest functions. All the protections are built in there.

1...237023712372237323742375237623772378237923802381238223832384...3189
New comment
Reason: