Find average value over number of bars? - page 4

New comment
 
R4tna C #:

You are worried about a few microseconds? Nothing more to be said.

And PS, why on earth would you switch off the optimizer? Not something I would ever do...

 
R4tna C #:

You are worried about a few microseconds? Nothing more to be said.

Right, nothing more to be said here...

Considering its just a function call to make that difference. 


To me that is relevant, but I am german, and germans usually over-engineer things, dont they? 

Like cars, ohh, or ISDN telephony, or, or, or... I think it was even a german who came up with O-Notation for algorithms... 

https://en.wikipedia.org/wiki/Big_O_notation


But, yes, you are right, its irrelevant for making an algorithm work... 

 
Dominik Christian Egert #:

Right, nothing more to be said here...

Considering its just a function call to make that difference. 


To me that is relevant, but I am german, and germans usually over-engineer things, dont they? 

Like cars, ohh, or ISDN telephony, or, or, or... I think it was even a german who came up with O-Notation for algorithms... 

https://en.wikipedia.org/wiki/Big_O_notation


But, yes, you are right, its irrelevant for making an algorithm work... 

Actually I don't agree - I like the way German's engineer stuff - from the cars I have owned to the software I have worked with. 

The company I did the most work with was SAP and their engineering approach in the 90's was absolutely fantastic - I learnt so much from them (especially the science of perf optimization which they had down to a tee). Of course after the late 90's with newer companies like Google the game changed, and that makes older companies like SAP look a little dated in their technology stack, but it has stood the test of time and much of the older tech which made them a world beater still runs today.

Some design classics are ageless

 
R4tna C #:

And PS, why on earth would you switch off the optimizer? Not something I would ever do...

Because compiling a large project, something aling the size of 8 MB of source takes a considerable amount of time to complete, while without the optimizer, its within a few seconds...

For debugging, or fast verification of syntax, its much better this way. - And as already stated, it makes in fact no difference for my code to use the optimizer, as it as shown (in previous post) its not really working... Although it could be considered "margin of error" a difference of 12 microseconds, but still is around 20% runtime difference.

 
Dominik Christian Egert #:

Because compiling a large project, something aling the size of 8 MB of source takes a considerable amount of time to complete, while without the optimizer, its within a few seconds...

For debugging, or fast verification of syntax, its much better this way. - And as already stated, it makes in fact no difference for my code to use the optimizer, as it as shown (in previous post) its not really working... Although it could be considered "margin of error" a difference of 12 microseconds, but still is around 20% runtime difference.

Yeah maybe, but ultimately if you need fast runtime you have to wait for the optimizer to do its thing - no free lunch (unless you work at SAP that is.. LOL)

20% of the runtime of this small program segment maybe, but in reality one would have a much larger code base and larger execution times, with waits from client/server traffic which would eclipse CPU/memory wait events. Anyway, enough said, I think this thread is as complete as can be

 
R4tna C #:

Actually I don't agree - I like the way German's engineer stuff - from the cars I have owned to the software I have worked with. 

The company I did the most work with was SAP and their engineering approach in the 90's was absolutely fantastic - I learnt so much from them. Of course after the late 90's with newer companies like Google the game changed, and that makes older companies like SAP look a little dated in their technology stack, but it has stood the test of time and much of the older tech which made them a world beater still runs today.

Some design classics are ageless

Yes, as it is af today, most electronics in cars is engineered at Bosch and they care alot about performance and optimization. BTW, Tesla does as well, running a neural network for their autonomous driving agent is a considerable amount of resource, they need to use. - Every nanosecond they can extract is beneficial for them. - They even went as far as developing their own NN-Chip to enahnce claculations. Also they have figured a different way of activation function calculations, just to slice off another few micros. 

Considering that such functions get called millions of times per second, nad the fact a nanosecond is a considerable amount of time for a CPU, it is not irrelevant, as you believe. and there are so many use cases where exactly this is so important.

Like network latency in datacenters, they use special time syncronisation protocols which feed directly into the network chip on the network card to reduce transfere latencies....

There are som many cases where it is important to get these micros. 


When comparing algorithms on a large scale, it makes huge differences, especially when they are computationally focused. - Else there would not be a need to constantly increase the power of modern CPUs, or would there?

 
R4tna C #:

Yeah maybe, but ultimately if you need fast runtime you have to wait for the optimizer to do its thing - no free lunch (unless you work at SAP that is.. LOL)

20% of the runtime of this small program segment maybe, but in reality one would have a much larger code base and larger execution times, with waits from client/server traffic which would eclipse CPU/memory wait events. Anyway, enough said, I think this thread is as complete as can be

OK, lets take it to the next level:

    ulong start = GetMicrosecondCount();
    for(int cnt = ArraySize(test_arr) - 2; (cnt >= NULL) && !_StopFlag; cnt--)
    { 
        if( (test_arr[cnt + 1] % 2) == NULL)
        { test_arr[cnt] = cnt; }
        else
        { test_arr[cnt] = cnt; }
    }
    printf("Time in microseconds used: %llu", GetMicrosecondCount() - start);


    start = GetMicrosecondCount();
    for(int cnt = ArraySize(test_arr) - 2; (cnt >= NULL) && !_StopFlag; cnt--)
    { test_arr[cnt] = ( (test_arr[cnt + 1] % 2) == NULL) ? cnt : cnt; }
    printf("Time in microseconds used: %llu", GetMicrosecondCount() - start);


Non optimizer results:

2022.08.27 11:38:05.747 Playground (GBPUSD,M1) Time in microseconds used: 58

2022.08.27 11:38:05.747 Playground (GBPUSD,M1) Time in microseconds used: 34


Optimizer results:

2022.08.27 11:38:33.578 Playground (GBPUSD,M1) Time in microseconds used: 58

2022.08.27 11:38:33.578 Playground (GBPUSD,M1) Time in microseconds used: 35


BTW, the issue with the code being slower, is given by the fact there is a conditional execution path within the loop, using if, while the ternary operator does not have that issue...

now lets add both together.

    ulong start = GetMicrosecondCount();
    for(int cnt = 1; (cnt < ArraySize(test_arr)) && !_StopFlag; cnt++)
    { 
        if( (test_arr[cnt - 1] % 2) == NULL)
        { test_arr[cnt] = cnt; }
        else
        { test_arr[cnt] = cnt; }
    }
    printf("Time in microseconds used: %llu", GetMicrosecondCount() - start);


    start = GetMicrosecondCount();
    for(int cnt = ArraySize(test_arr) - 2; (cnt >= NULL) && !_StopFlag; cnt--)
    { test_arr[cnt] = ( (test_arr[cnt + 1] % 2) == NULL) ? cnt : cnt; }
    printf("Time in microseconds used: %llu", GetMicrosecondCount() - start);



Non optimizer results:

2022.08.27 11:42:56.235 Playground (GBPUSD,M1) Time in microseconds used: 63

2022.08.27 11:42:56.235 Playground (GBPUSD,M1) Time in microseconds used: 38


Optimizer results:

2022.08.27 11:43:44.796 Playground (GBPUSD,M1) Time in microseconds used: 27

2022.08.27 11:43:44.796 Playground (GBPUSD,M1) Time in microseconds used: 19



Interesting, isnt it? - Results depend very much on the code, but not in any way I understand.




 
R4tna C #:

I outlined the 3 operations in maintaining a sum, count and mean. You could also have variables for min and max - just compare these with the incoming number and decide whether it is a min or max and keep it if so. A fixed number of operations... two ternary statements would do it

No, they dont. - I disagree. - Simply because you do not know the max value as soon as you drop your current max value from your scope of consideration. - You would be forced to seearch for the new highest value throuout your current scope. - If things were that easy...


Even if you keep track of every value in your scope, and maintain a ordered/sorted list (maybe even using a linked list because of memory management and copying around poortions of the array), you still have to iterate through all of the current values of your scope of interest.

There is no way to do it in a deterministic way, except of course you define your deterministic approach as iterating all of the elements every time, which is, what I did in my finctions min()/max(). - So they are, ginven the length of the mean, in fact deterministic, because their execution time is predefined before interation. But it is not an O 1 operation.

It is not possible, no matter how you do it. - Mathematically unsolvable.


Just like this algorithm: 

 https://en.wikipedia.org/wiki/Ackermann_function&nbsp;


int ack(int m, int n) 
{ 
    if (m == 0){ 
        return n + 1; 
    } 
    else if((m > 0) && (n == 0)){ 
        return ack(m - 1, 1); 
    } 
    else if((m > 0) && (n > 0)){ 
        return ack(m - 1, ack(m, n - 1)); 
    } 
} 
  
int main() 
{ 
    int A; 
    A = ack(1, 2); 
    printf("%i", A); 
    return 0; 
}
 
Dominik Christian Egert #:

No, they dont. - I disagree. - Simply because you do not know the max value as soon as you drop your current max value from your scope of consideration. - You would be forced to seearch for the new highest value throuout your current scope. - If things were that easy...


Even if you keep track of every value in your scope, and maintain a ordered/sorted list (maybe even using a linked list because of memory management and copying around poortions of the array), you still have to iterate through all of the current values of your scope of interest.

There is no way to do it in a deterministic way, except of course you define your deterministic approach as iterating all of the elements every time, which is, what I did in my finctions min()/max(). - So they are, ginven the length of the mean, in fact deterministic, because their execution time is predefined before interation. But it is not an O 1 operation.

It is not possible, no matter how you do it. - Mathematically unsolvable.


Just like this algorithm: 

 https://en.wikipedia.org/wiki/Ackermann_function&nbsp;


So if you initialize the max variable to some small value (e.g. DBL_MIN) and simply collect any value which is higher than the previous value, no matter how many numbers you feed into the series, the the max variable hold hold the highest value detected in the sequence right?

Opposite logic for collecting the min value.

You don't need to keep the history - just process the stream of numbers.

 
Dominik Christian Egert #:

It is not possible, no matter how you do it. - Mathematically unsolvable.


If you need a code example of what I am suggesting... 

Output sample (random data):

2022.08.27 18:21:06.408 //--- Sum = 184190.00 Count=9.00 Mean = 20465.56 Max = 32233.00 Min = 1851.00

2022.08.27 18:21:06.408 Check ArrayMax[2]  true  = 32233.00

2022.08.27 18:21:06.408 Check ArrayMin[3]  true  = 1851.00

2022.08.27 18:21:06.408 29554.00000 17037.00000 32233.00000  1851.00000 22728.00000 31879.00000 25929.00000  6003.00000 16976.00000


And please note - this is just an example, not a finished version. But using this approach one could build a function which accepts a stream or set of numbers - the sum, count, mean, max, min could be retained so each time the same number of operations are executed.


//+------------------------------------------------------------------+
//|                                      430138-MathMeanOverBars.mq5 |
//|                                  Copyright 2022, MetaQuotes Ltd. |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2022, MetaQuotes Ltd."
#property link      "https://www.mql5.com"
#property version   "1.00"
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
   testMeanMaxMin();
  }

//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
void  testMeanMaxMin(int dataSampleSize = 9)
  {
   double mean = 0, max = DBL_MIN, min = DBL_MAX, sum = 0, newdata = 0;
   int count   = 0, maxIndex = 0, minIndex = 0;
   double newDataHist[] = {};

   for(int i = 0; i < dataSampleSize; i++)
     {

      newdata =  double(MathRand());

      sum   += newdata;
      count++;
      max = (newdata > max) ? newdata : max;
      min = (newdata < min) ? newdata : min;
      ArrayResize(newDataHist, count, dataSampleSize);
      newDataHist[count - 1] = newdata;
     }

   mean = (count != 0) ? sum / count : 0;

   PrintFormat("\n//--- Sum = %.2f Count=%.2f Mean = %.2f Max = %.2f Min = %.2f", sum, count, mean, max, min);

   maxIndex = ArrayMaximum(newDataHist);
   minIndex = ArrayMinimum(newDataHist);

   PrintFormat("Check ArrayMax[%d]  %-5s = %.2f", maxIndex, string((newDataHist[maxIndex]==max)), newDataHist[maxIndex]  );
   PrintFormat("Check ArrayMin[%d]  %-5s = %.2f", minIndex, string((newDataHist[minIndex]==min)), newDataHist[minIndex]  );

   ArrayPrint(newDataHist);

  }
123456
New comment