GZIP - Decode - library for MetaTrader 5

The library allows you to decrypt GZIP archives from *.gz files or responses from sites compressed with this format. Tested on files with up to 0.5 GB of text packed in them.

The library can automatically check by the flag in the 4th byte whether the archive is a compressed file or compressed data from the site, in the second case there is no file name in the data.

Input data for decompression should be represented in an array of char type.

The terminal function CryptDecode(CRYPT_ARCH_ZIP, tmp, key, tx) is used for decompression;

The GZIP class contains methods:

isGZIP(char& gz[])

- checks against the first three characters of the received data to see if it is compressed in GZIP format.

After this check, one of the overloads of the unGZIP method can be called:

bool unGZIP(char& gz[], char& tx[], string& fname, bool printTime=false){}

the preferred use case in terms of speed and memory consumption. During decompression, the char array tx is filled. Then data from it can be processed by CSV or JSON parser. This way we get the result without gluing the array into a string and then dividing it by strings or other characters.

If you still need to get a string, you can use one of the overloads:

bool unGZIP(char& gz[], string& out, bool printTime=false, uint codepage=CP_ACP){}

return true if unpacking is successful, the unpacked string is in the string out

or

string unGZIP(char& gz[], bool printTime=false, uint codepage=CP_ACP){}

return string. If unpacking fails, the string will be empty.

Care should be taken when using concatenation of data into a single string.
If you unpacked an archive with 0.5GB of data, the string will require another 0.5GB, then you can split it into an array of strings, which will require another 0.5GB. Plus the time cost of all these operations.

Example use case for reading compressed data from a website and then reading from a file: (this script is attached for download)

#property copyright "Copyright 2023, Forester"
#property link      "https://www.mql5.com"
#property version   "1.00"

#include <GZIP.mqh>
GZIP gzip;

void OnStart(){
      Print("GZipped webpage read sample:");
      string url="https://............";
      char out[]; 
      if(http_req(url, out,"GET","Accept-Encoding: gzip, deflate\r\n", 1,0)){
          if(gzip.isGZIP(out)){
            Print("GZIP detected");
            string txt; if(!gzip.unGZIP(out, txt, 1, CP_UTF8)){return;}//CP_ACP
            Print(txt);
         }else{
         Print("No GZIP");Print(CharArrayToString(out, 0, WHOLE_ARRAY, CP_UTF8));
         }
      }
      
      Print("File read sample:");
      bool is_common=1;
      string fn="3.csv.gz";
      bool is_zip=1;

      int h;
      if(is_zip){
         h = FileOpen(fn,FILE_READ|FILE_BIN|FILE_READ|(is_common?FILE_COMMON:0));
      }else{
         h = FileOpen(fn,FILE_READ|FILE_CSV|FILE_ANSI|(is_common?FILE_COMMON:0));
      }
      if( h!=INVALID_HANDLE){
         Print ("Start gzip");
         char gz[],tx[];
         FileReadArray(h, gz); FileClose(h); 
         
         if(gzip.isGZIP(gz)){
            Print("GZIP detected");
            string fname;
            if(!gzip.unGZIP(gz, tx,fname,1)){return;}
            //string txt; if(!unGZIP(gz, txt, 1, CP_ACP)){return;}//CP_UTF8
            //string txt=unGZIP(gz, 1, CP_ACP);if(txt==""){return;}
            if(fname!=NULL){Print("File name: ",fname,", file size: ",ArraySize(tx));}
            string t=  CharArrayToString(tx,0,100);Print("Filr content: ",t);t=  CharArrayToString(tx,ArraySize(tx)-100,100);Print("... ",t);

 
            //ulong mcs = GetMicrosecondCount();
            //string csvRow[];int columns=0, next=0; char sep=',', newLine='\n';
            //while(getCSVRow(tx, csvRow, columns, next, sep, newLine) && !IsStopped()){//get next row
              // for(int c=0; c<columns; c++){
               //Print();
              // }
               //Print(next);
            //}
            //Print("CSV parse time: ",(GetMicrosecondCount()-mcs)," mcs");
            //ArrayPrint(csvRow);

        }else{}
      }
}

int getCSVRow(char& csv[], string& outAr[], int& cols, int& next, char& sep, char& newLine){//get next row
   for(int i=next; i<ArraySize(csv); i++){
      if(csv[i] == newLine){
         cols=StringSplit(CharArrayToString(csv,next,i-next),sep,outAr);// process 1 line separately - 5-10% faster than parseCSV and e has memory for 1 large array.
         next=i+1;
         return true;
      }
   }
   return false;
}



bool http_req(string url, char& out[],string method="GET",string headers_request=NULL, bool printTime=false,bool debug=false){
   ulong mcs = (printTime?GetMicrosecondCount():0);
   bool retVal=false;
   ResetLastError();
   char data[];string headers=NULL;
   int r=WebRequest(method, url, headers_request, 5000, data, out, headers);
   if(r==-1){Print("Error in WebRequest. Err code =",GetLastError());MessageBox("It is necessary to add the address '"+url+"' to the list of allowed URLs in the 'Advisors' tab","Error",MB_ICONINFORMATION);}
   else{if(r==200){ retVal=  true; }else{PrintFormat("Download error '%s', responce code %d",url,r);Print("Error in WebRequest. Err code =",GetLastError());}}if(debug){PrintFormat("Responce headers: %s\r\nResponce:",headers);Print(CharArrayToString(out));}//
   if(printTime){Print("WebRequest time: ",(GetMicrosecondCount()-mcs)," mcs");}
   return retVal;
}

This example reads a file from the files folder, and decompresses it into an array.
The first and last 100 characters from the unpacked file are printed.
The array can then be sent to a CSV or JSON parser. For JSON there is JASON library in kodobase. It can parse data received as a char array.

Example of output when unpacking a compressed CSV file of 120 MB, the source file is 463 MB:

Unpacking took about 1.2 seconds.