All Blogs / Analytics & Forecasts

Thread Swarm with leader and global variables , for 1 terminal

13 December 2023, 14:14

140

Let's assume you have data that must be downloaded on a server .

This data concerns multiple symbols not just one and you are not making one request per symbol separately.

You need to be running multiple "stuff" (indicator/ea) in one terminal but you don't want all of them to try and perform this one unique but similar task (whatever it is) at the same time . You want one of them to go fetch the data and inform everyone else to open it.

So , we need a swarm .

We will call swarm the set of threads (indicators/eas) that are running the code and each and every one of them a swarm unit.

This swarm will be active for as long as there is at least one swarm unit still active (i.e. an open chart)

Let's assume 3 datetime variables :

Last check for data : This is the last time "anything" from this terminal asked the server the question "is there new data?"
Last data time : This is the last time "anyone" from this terminal downloaded new data.
Last update time : This is the time of the "LastDataTime" this swarm unit is running with.

These are used directly for all the checks we need , and these are the checks regarding the data.

When was the last time we checked , so that we don't flood the server.
When was the last time we downloaded.
The data this swarm unit has , when was it downloaded.

Also you will need to indentify whether or not the data you are getting is the same as the data you already have , but you do that in your server or in your files , or whatever constitutes the "external" thing you want to work with.

So let's dive right in :

First the code for one swarm unit

class swarm_unit{
         public:
    long chart_id;//the chart id 
    long leader_id;//the leader id
    long others[];//ids of other open charts
datetime last_check_for_data; 
datetime last_data_time;
datetime last_update_time;

The chart id is the id of the chart the unit is running on .

Then we need a list of all the other units in the swarm that are still active.

And lastly we need to know who the leader is , again a chart id.

then the 3 times for checking and interacting with the "external" packages as mentioned above

Time to start deploying our functions .

What are we going to need ? A way to find the other swarm units that are still active.

We will loop through the open charts
We will look for an ea that has a swarm unit
If we don't find it we will search for an indicator with a swarm unit
we will collect the ids of the charts in the others[] array

   void get_others_from_charts(){
        ArrayFree(others);
        long read_chart=ChartFirst();
        while(read_chart>=0){
               bool has_swarm_unit=false;//if this chart has a swarm unit 
             //if not this chart 
               if(read_chart!=ChartID()&&read_chart!=0){
               //look for ea
                 if(ChartGetString(read_chart,CHART_EXPERT_NAME)==SWARM_EA_NAME){
                   has_swarm_unit=true;
                   }
                 else{
               //look for indicator
                 //get windows 
                   int windows=(int)ChartGetInteger(read_chart,CHART_WINDOWS_TOTAL);
                   //loop to windows of that chart
                     for(int w=0;w<windows;w++){
                        //indicators on that window 
                          int indicators=ChartIndicatorsTotal(read_chart,w);
                          //loop to indicators 
                            for(int i=0;i<indicators;i++){
                               string indicator_name=ChartIndicatorName(read_chart,w,i);
                               if(indicator_name==SWARM_INDY_SHORTNAME){
                                 has_swarm_unit=true;
                                 break;
                                 }
                               }
                          //loop to indicators ends here 
                          if(has_swarm_unit){break;}
                        }
                   }
                   //look for an indicator ends here
                   //loop to windows of that chart ends here 
                   if(has_swarm_unit){
                     add_to_others(read_chart);
                     }
                 }
               //if not this chart ends here
             if(read_chart!=0){
               read_chart=ChartNext(read_chart);
               }else{
               read_chart=ChartNext(ChartID());
               }
             }
        }

So our unit knows itself , knows the IDs of the other units now it needs to know who the leader is.

To facilitate these checks we will deploy a small "bridge" using global variables.

We need 4 things :

To be able to lock and unlock the bridge
To store the id of the leader
To store the last check time
To store the last data time

The lock works as follows :

If the bridge is not locked lock it and assign the chart id of the swarm unit that locked it to it.

If the bridge is locked :

If the swarm unit that locked the bridge is no longer active , lock the bridge
If the swarm unit that locked the bridge is active do nothing

/* lock unlock systems for GV */
//check if common global swarm variable space is locked
bool canOccupyGSVS(swarm_unit &self,bool &reLock){
  reLock=true;//this means , write new lock
//check lock global variable 
  long gv=GvToLong(GVSS_LOCK);
  //this is the id of the chart that locked it essentially
  if(gv>0){
  if(gv==ChartID()){reLock=false;return(true);}//if self
  else{
   reLock=false;
   //so is the chart that locked the swarm open ?
   for(int i=0;i<ArraySize(self.others);i++){
      if(self.others[i]==gv){
        return(false);
        }
      }
   //if we reach this point it means the chart that locked the swarm is gone
     reLock=true;//so relock
     return(true);//and we can occupy!
   }
  }
return(true);
} 

void UnlockGVSS(){
LongToGV(GVSS_LOCK,0);
}

Above is the check lock function

We pass the swarm unit (of this chart) and a reference bool for whether or not we can relock the bridge.

This function will return true if we can go on interacting with the bridge from this swarm unit , and also an additional true/false if the bridge must be locked.

it pulls the id of the locking unit
if there is no id we can interact and we must relock
if there is an id and it is actually of this chart , we can proceed but we don't relock
if we find the id in the others[] we don't proceed and we don't relock
if we dont find the id in the others[] we can proceed and we can relock

And we package all the above in one swarm maintainance function :

    void swarm_maintain(){
         get_others_from_charts();
         //is the Swarm system occupied ? 
          bool reLock=false;
          if(canOccupyGSVS(this,reLock)){
          //relock if needed
            if(reLock){
              LongToGV(GVSS_LOCK,chart_id);
              }
          //is there a leader ? 
            long new_leader=GvToLong(GVSS_LEADER);
          //if there is no leader , become it
            if(new_leader==0){
                leader_id=chart_id;
                LongToGV(GVSS_LEADER,chart_id);
              }
          //if there is a leader 
            else{
              bool is_on=false;
              for(int i=0;i<ArraySize(others);i++){if(others[i]==new_leader){is_on=true;break;}}
              //if leader is active
                if(is_on){
                leader_id=new_leader;
                }
              //if leader closed
                else{
                //become the leader 
                  leader_id=chart_id;
                  LongToGV(GVSS_LEADER,chart_id);
                }
              } 
          //if this chart is leading 
            if(chart_id==leader_id){
            //acquire last check time 
              last_check_for_data=(datetime)GvToLong(GVSS_LAST_CHECK_FOR_DATA_TIME);
              long elapsed=TimeGMT()-last_check_for_data;
              if(elapsed>=SecsForNewAnalysisCheck){
                //im placing a bool check here 
                  bool gotnew=false;
                //if you get new data update the last_data_time
                  if(gotnew){
                  last_data_time=last_check_for_data;
                  //we also write to the GVs to inform the other units theres new data
                  LongToGV(GVSS_LAST_DATA_TIME,((long)last_data_time));
                  }
                }
            }
            //we need to know if there is new data to load
              last_data_time=(datetime)GvToLong(GVSS_LAST_DATA_TIME);
              if(last_data_time>last_update_time){
                //load the fetched data here 
                //and update the update time
                  last_update_time=last_data_time;
                }

          UnlockGVSS();
          }
        //is the Swarm system occupied ? ends here 
        }

we get a list of all the other active swarm units
we check if we can interact with the bridge

If we can interact with the bridge :

we check if there is a leader
if there is no leader this unit becomes the leader
if there is a leader we check whether or not it is still active.
if it is not active this unit becomes the leader

If there is a leader and is active

if this unit is the leader it manages interaction with the external data source
if this unit is not the leader it is checking the latest data time in order to decide if it must load new data that the leader received

And this is it more or less

I'm attaching the test code for an indicator

If you find any issues let me know

#property indicator_chart_window
#define SWARM_INDY_SHORTNAME "SwarmTest"
#define SWARM_EA_NAME "SwarmTest"
#define GVSS_NAME "SWARMGV"
#define GVSS_LOCK GVSS_NAME+"_LOCK"
#define GVSS_LEADER GVSS_NAME+"_LEADER"
#define GVSS_LAST_CHECK_FOR_DATA_TIME GVSS_NAME+"_LAST_CHECK"
#define GVSS_LAST_DATA_TIME GVSS_NAME+"_LAST_TIME"
input long SecsForNewAnalysisCheck=60;//Seconds interval for checking for new data
//you should place this inside your data structures or smth , here is by itself:
int OnInit()
  {   
  SWU.reset(true,SWARM_INDY_SHORTNAME);
  SWU.setup(ChartID());
  EventSetMillisecondTimer(1000);       
  return(INIT_SUCCEEDED);
  }
int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime &time[],
                const double &open[],
                const double &high[],
                const double &low[],
                const double &close[],
                const long &tick_volume[],
                const long &volume[],
                const int &spread[])
  {
  return(rates_total);
  }
void OnTimer()
  {
  SWU.swarm_maintain();
  SWU.display_swarm();
  }


/* SWARM 
*/
class swarm_unit{
         public:
    long chart_id;//the chart id 
    long leader_id;//the leader id
    long others[];//ids of other open charts
datetime last_check_for_data; 
datetime last_data_time;
datetime last_update_time;
         swarm_unit(void){reset();}
        ~swarm_unit(void){reset();}
    void reset(bool is_indicator=false,string indyshortname=NULL){
         last_check_for_data=0;
         chart_id=-1;
         leader_id=-1;
         ArrayFree(others);
         if(is_indicator){
           IndicatorSetString(INDICATOR_SHORTNAME,indyshortname);
           }
         }
    void setup(long _id){
         chart_id=_id;
         }
    void swarm_maintain(){
         get_others_from_charts();
         //is the Swarm system occupied ? 
          bool reLock=false;
          if(canOccupyGSVS(this,reLock)){
          //relock if needed
            if(reLock){
              LongToGV(GVSS_LOCK,chart_id);
              }
          //is there a leader ? 
            long new_leader=GvToLong(GVSS_LEADER);
          //if there is no leader , become it
            if(new_leader==0){
                leader_id=chart_id;
                LongToGV(GVSS_LEADER,chart_id);
              }
          //if there is a leader 
            else{
              bool is_on=false;
              for(int i=0;i<ArraySize(others);i++){if(others[i]==new_leader){is_on=true;break;}}
              //if leader is active
                if(is_on){
                leader_id=new_leader;
                }
              //if leader closed
                else{
                //become the leader 
                  leader_id=chart_id;
                  LongToGV(GVSS_LEADER,chart_id);
                }
              } 
          //if this chart is leading 
            if(chart_id==leader_id){
            //acquire last check time 
              last_check_for_data=(datetime)GvToLong(GVSS_LAST_CHECK_FOR_DATA_TIME);
              long elapsed=TimeGMT()-last_check_for_data;
              if(elapsed>=SecsForNewAnalysisCheck){
                //im placing a bool check here 
                  bool gotnew=false;
                //if you get new data update the last_data_time
                  if(gotnew){
                  last_data_time=last_check_for_data;
                  //we also write to the GVs to inform the other units theres new data
                  LongToGV(GVSS_LAST_DATA_TIME,((long)last_data_time));
                  }
                }
            }
            //we need to know if there is new data to load
              last_data_time=(datetime)GvToLong(GVSS_LAST_DATA_TIME);
              if(last_data_time>last_update_time){
                //load the fetched data here 
                //and update the update time
                  last_update_time=last_data_time;
                }

          UnlockGVSS();
          }
        //is the Swarm system occupied ? ends here 
        }
   void get_others_from_charts(){
        ArrayFree(others);
        long read_chart=ChartFirst();
        while(read_chart>=0){
               bool has_swarm_unit=false;//if this chart has a swarm unit 
             //if not this chart 
               if(read_chart!=ChartID()&&read_chart!=0){
               //look for ea
                 if(ChartGetString(read_chart,CHART_EXPERT_NAME)==SWARM_EA_NAME){
                   has_swarm_unit=true;
                   }
                 else{
               //look for indicator
                 //get windows 
                   int windows=(int)ChartGetInteger(read_chart,CHART_WINDOWS_TOTAL);
                   //loop to windows of that chart
                     for(int w=0;w<windows;w++){
                        //indicators on that window 
                          int indicators=ChartIndicatorsTotal(read_chart,w);
                          //loop to indicators 
                            for(int i=0;i<indicators;i++){
                               string indicator_name=ChartIndicatorName(read_chart,w,i);
                               if(indicator_name==SWARM_INDY_SHORTNAME){
                                 has_swarm_unit=true;
                                 break;
                                 }
                               }
                          //loop to indicators ends here 
                          if(has_swarm_unit){break;}
                        }
                   }
                   //look for an indicator ends here
                   //loop to windows of that chart ends here 
                   if(has_swarm_unit){
                     add_to_others(read_chart);
                     }
                 }
               //if not this chart ends here
             if(read_chart!=0){
               read_chart=ChartNext(read_chart);
               }else{
               read_chart=ChartNext(ChartID());
               }
             }
        }
   void display_swarm(){
        string swarmids="[SELF] "+IntegerToString(chart_id);
        if(leader_id==chart_id){swarmids+=" <LEADER>";}
        swarmids+="\n";
        for(int i=0;i<ArraySize(others);i++){
           swarmids+="["+IntegerToString(i)+"] "+IntegerToString(others[i]);
           if(others[i]==leader_id){
             swarmids+=" <LEADER>";
             }
           swarmids+="\n";
           }
        Comment(swarmids);
        }
 string bool_to_string(bool _bool){if(_bool){return("True");}return("False");}
        private:
   void add_to_others(long id){
        ArrayResize(others,ArraySize(others)+1,0);
        others[ArraySize(others)-1]=id;
        }
};

swarm_unit SWU;

/* lock unlock systems for GV */
//check if common global swarm variable space is locked
bool canOccupyGSVS(swarm_unit &self,bool &reLock){
  reLock=true;//this means , write new lock
//check lock global variable 
  long gv=GvToLong(GVSS_LOCK);
  //this is the id of the chart that locked it essentially
  if(gv>0){
  if(gv==ChartID()){reLock=false;return(true);}//if self
  else{
   reLock=false;
   //so is the chart that locked the swarm open ?
   for(int i=0;i<ArraySize(self.others);i++){
      if(self.others[i]==gv){
        return(false);
        }
      }
   //if we reach this point it means the chart that locked the swarm is gone
     reLock=true;//so relock
     return(true);//and we can occupy!
   }
  }
return(true);
} 

void UnlockGVSS(){
LongToGV(GVSS_LOCK,0);
}


long GvToLong(string name){
if(GlobalVariableCheck(name+"_A")&&GlobalVariableCheck(name+"_B")&&GlobalVariableCheck(name+"_AFrontDigits")&&GlobalVariableCheck(name+"_BFrontDigits")&&GlobalVariableCheck(name+"_ABackDigits")&&GlobalVariableCheck(name+"_BBackDigits")){
  int digits_front_a=(int)MathFloor(GlobalVariableGet(name+"_AFrontDigits"));
  int digits_back_a=(int)MathFloor(GlobalVariableGet(name+"_ABackDigits"));
  int digits_front_b=(int)MathFloor(GlobalVariableGet(name+"_BFrontDigits"));
  int digits_back_b=(int)MathFloor(GlobalVariableGet(name+"_BBackDigits"));
  long p1=(long)MathFloor(GlobalVariableGet(name+"_A"));
  long p2=(long)MathFloor(GlobalVariableGet(name+"_B"));
  return(assembleParts(p1,p2,digits_front_a,digits_back_a,digits_front_b,digits_back_b));
  }
return(0);
}

void LongToGV(string name,long value){
long part_a=0,part_b=0;
 int part_a_front_digits=0,part_a_back_digits=0,part_b_front_digits=0,part_b_back_digits=0;
 long_to_parts(value,part_a,part_a_front_digits,part_a_back_digits,part_b,part_b_front_digits,part_b_back_digits); 
 GlobalVariableSet(name+"_A",part_a);
 GlobalVariableSet(name+"_B",part_b);
 GlobalVariableSet(name+"_AFrontDigits",part_a_front_digits);
 GlobalVariableSet(name+"_ABackDigits",part_a_back_digits);
 GlobalVariableSet(name+"_BFrontDigits",part_b_front_digits);
 GlobalVariableSet(name+"_BBackDigits",part_b_back_digits);
}

long assembleParts(long part_a,
                   long part_b,
                    int part_a_front_digits,
                    int part_a_back_digits,
                    int part_b_front_digits,
                    int part_b_back_digits){
string partA=longToStringWithDigits(part_a,part_a_front_digits,part_a_back_digits);
string partB=longToStringWithDigits(part_b,part_b_front_digits,part_b_back_digits);
string full=partA+partB;
long result=(long)StringToInteger(full);
return(result);
}

string longToStringWithDigits(long l,int front_digits,int back_digits){
string front="";
for(int i=0;i<front_digits;i++){
   front+="0"; 
   }
string back="";
for(int i=0;i<back_digits;i++){
   back+="0";
   }
string middle=IntegerToString(l);
return(front+middle+back);
}

void long_to_parts(long original,
                   long &part_a,
                    int &part_a_front_digits,
                    int &part_a_back_digits,
                   long &part_b,
                    int &part_b_front_digits,
                    int &part_b_back_digits){
string full=IntegerToString(original);
  part_a=0;
  part_a_front_digits=0;
  part_a_back_digits=0;//this may be useless but roll with it now
  part_b=0;
  part_b_front_digits=0;
  part_b_back_digits=0;
//count digits
  int alldigits=StringLen(full);
//split
  int alength=(int)MathFloor(((double)alldigits)/((double)2.0));
  int blength=alldigits-alength;
  string partA=StringSubstr(full,0,alength);
  string partB=StringSubstr(full,alength,blength);
//---
  /*
  okay so what is the fuss about ?
  lets take this long number 12345600002446231000
  length is 20
  kay
  so part a is 10 digits 
  and part b is 10 digits
  okay but 
  when we store it in GV the leading zeroes may get evaporated
  and we dont want that so , scan for 0
  */
  //part a 
    for(int i=0;i<alength;i++){
    string ch=StringSubstr(partA,i,1);
    if(ch=="0"){part_a_front_digits++;}
    else{break;}
    }
    for(int i=alength-1;i>=0;i--){
    string ch=StringSubstr(partA,i,1);
    if(ch=="0"){part_a_back_digits++;}
    else{break;}
    }
  //part b
    for(int i=0;i<blength;i++){
    string ch=StringSubstr(partB,i,1);
    if(ch=="0"){part_b_front_digits++;}
    else{break;}
    }
    for(int i=blength-1;i>=0;i--){
    string ch=StringSubstr(partB,i,1);
    if(ch=="0"){part_b_back_digits++;}
    else{break;}
    }   
part_a=(long)StringToInteger(partA);
part_b=(long)StringToInteger(partB); 
}


void OnChartEvent(const int id,const long& lparam,const double& dparam,const string& sparam){

}

void OnDeinit(const int reason)
{

}