Bug: function MathSample() does not generate correct results
Thank you for your message
You are right, calculations of source index are incorrect
1) MathSample w/o replace
IMHO correct and faster way is
int ind=int((ulong)size*MathRand()/32768);multiplication overload is not possible and division will be replaced with shifting at runtime
Thank you for getting attention to this bug.
I need to clarify the difference between sampling schemes.
Sampling With Replacement
You replace back the first item you draw to the list before you draw a second. So, if you had a list of names to choose two people from (sample of 2), you could choose the same person twice. If the list had unique items, then our sample would contain duplicates.
//sample output using the fixed header "Math_fix.mqh"" /* MathSample_demo (CHFJPY,M1) MathSample(population=9,samplesize=4,replacement=true) MathSample_demo (CHFJPY,M1) 6 5 1 7 MathSample_demo (CHFJPY,M1) 4 2 1 6 MathSample_demo (CHFJPY,M1) 3 3 7 1 MathSample_demo (CHFJPY,M1) 9 5 4 2 MathSample_demo (CHFJPY,M1) 2 7 3 5 MathSample_demo (CHFJPY,M1) 5 3 5 4 MathSample_demo (CHFJPY,M1) 3 7 3 8 MathSample_demo (CHFJPY,M1) 9 9 4 9 MathSample_demo (CHFJPY,M1) 7 1 2 5 MathSample_demo (CHFJPY,M1) 5 7 6 5 */
Sampling Without Replacement
You don’t replace back the first item you draw before you draw a second. So, if you had a list of names to choose two people from, you couldn’t choose the same person twice. If the list has unique items then our sample is unique also (no duplicates).
Reference:
- stats.libretexts.org
2) MathSample with replace
Yes, Fisher-Yates algorithm gives much better shuffle
But using it for indices... why not!
we can little bit change it because we need only first count elements from indices array
For sampling without replacement, Fisher-Yates algorithm shuffles the whole array (population) via indices because the array is const, then we only draw the first count elements on the top (sample) without replacement back into array.
Edit:For sampling without replacement, Fisher-Yates algorithm shuffles the whole array (population) via indices because the array is const, then we only draw the first count elements on the top (sample) without replacement back into array.
Edit:Thank You
For sampling without replacement, Fisher-Yates algorithm shuffles the whole array (population) via indices because the array is const, then we only draw the first count elements on the top (sample) without replacement back into array.
Edit:That is what I mean "little change"
bool MathSample(const int &array[],const int count,const bool replace,int &result[]) { //--- unique values not needed if(!replace) return MathSample(array,count,result); int size=ArraySize(array); if(size==0 || count>size) return(false); //--- prepare target array if(ArraySize(result)<count) if(ArrayResize(result,count)!=count) return(false); //--- unique values needed, prepare indices int indices[]; if(!MathSequenceByCount(0,size-1,size,indices)) return(false); for(int i=0; i<count; i++) { //--- select random indices and swap them int j=i+int((ulong)(size-i)*MathRand()/32768); if(j!=i) { int t=indices[i]; indices[i]=indices[j]; indices[j]=t; } } //--- select data according to indices for(int i=0; i<count; i++) result[i]=array[indices[i]]; //--- return true; }
I suspect this shuffling is good enough and will be work faster because it does not shuffle whole indices array, only first count elements which will be used in the next loop
Thank you for getting attention to this bug.
I need to clarify the difference between sampling schemes.
Sampling With Replacement
You replace back the first item you draw to the list before you draw a second. So, if you had a list of names to choose two people from (sample of 2), you could choose the same person twice. If the list had unique items, then our sample would contain duplicates.
Sampling Without Replacement
You don’t replace back the first item you draw before you draw a second. So, if you had a list of names to choose two people from, you couldn’t choose the same person twice. If the list has unique items then our sample is unique also (no duplicates).
Reference:
we will think what to do, there might be some codes use existing opposite meaning of replace
might be it better to move replace parameter at the end of parameters lists (it will be helpful to revise existing codes, because compiling errors will be occurred) and rename it to replacement
have you read header of MathSample function ?
//+------------------------------------------------------------------+ //| MathSample | //+------------------------------------------------------------------+ //| The function takes a sample of the specified size from the | //| elements of the array[] with replacement. | //| | //| Arguments: | //| array[] : Array with double values | //| count : Number of elements to choose | //| replace : If true, only unique values are placed to result[] | !!! //| result[] : Output array | //| | //| Return value: true if successful, otherwise false. | //+------------------------------------------------------------------+
We made decision to change behavior
New code
//+------------------------------------------------------------------+ //| MathSample | //+------------------------------------------------------------------+ //| The function takes a sample of the specified size from the | //| elements of the array[] with replacement. | //| | //| Arguments: | //| array[] : Array with double values | //| count : Number of elements to choose | //| replace : If true, Sampling with Replacement will be used | //| result[] : Output array | //| | //| Return value: true if successful, otherwise false. | //+------------------------------------------------------------------+ bool MathSample(const double &array[],const int count,const bool replace,double &result[]) { //--- Sampling with Replacement if(replace) return MathSample(array,count,result); ...
Yes, I like this idea. It is in fact more efficient to use a partial-shuffle Fisher algorithm here for samples without replacement.
Visual verification:
https://bost.ocks.org/mike/shuffle/compare.html
In Choose an algorithm: custom
Enter JavaScript code:
// sampling without replacement function partial_shuffle(array, count = 20) { var size = array.length, i, j; for (i = 0; i < count; i++) { j = i + Math.floor(Math.random() * (size - i)); // i <= j <= size-1 //if (j != i) { t = array[j]; array[j] = array[i]; array[i] = t; //} } }
Click refresh button few times to check randomness in each selected sample (count = 20), on the left upper part of graph.
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
This a fix for the include file <Math\Stat\Math.mqh>:
According to the online documentation:
The replace=true argument allows performing random sampling of the elements with replacement back to the original sequence.
This is a script file to test the original and fixed include files.