Discussion of article "Handling ZIP Archives in Pure MQL5"

 

New article Handling ZIP Archives in Pure MQL5 has been published:

The MQL5 language keeps evolving, and its new features for working with data are constantly being added. Due to innovation it has recently become possible to operate with ZIP archives using regular MQL5 tools without getting third party DLL libraries involved. This article focuses on how this is done and provides the CZip class, which is a universal tool for reading, creating and modifying ZIP archives, as an example.

Once this article's author got attracted by an interesting feature of CryptDecode function, in particular its ability to decompress a ZIP array transferred to it. This modifier was introduced by the developers of MetaTrader 5 trading platform in order to extract answers from several servers using the standard WebRequest function. However, due to certain features of the ZIP file format, it was impossible to use it directly.

Additional authentication was required: to decompress an archive it was required to know its hash sum before compressing - Adler-32 which, clearly, wasn't available. However, when discussing this issue, the developers met the request to overload CryptDecode and CryptEncode, its mirror-image twin, with a special flag that allowed to ignore Adler32 hash when decompressing the transmitted data. For inexperienced from technical perspective users this innovation could be easily explained: it enabled full functionality of ZIP archives. This article describes the ZIP file format, its specifics of data storage and offers convenient object-oriented CZip class to operate with an archive.

Each ZIP archive is a binary file that contains an ordered sequence of bytes. On the other hand, every file of a ZIP archive has a name, attributes (such as file modification time), and other properties, that we are used to seeing in a file system of any operating system. Therefore, in addition to the compressed data, each ZIP archive stores the name of a compressed file, its attributes, and other service information. The service information is placed in a very specific manner and has a regular structure. For example, if an archive contains two files (File#1 и File#2), then it will have the following scheme:


Fig. 1. Schematic representation of a ZIP archive that contains two files: File#1 and File#2

Author: Vasiliy Sokolov

 

Has anyone tested the code ? Does it work in mt 4?

 
The impressive compression of historical data, together with the idea of applying it to RAM contents discussed above, suggests that there is hope for a radical reduction in the memory resources required by the MT5 terminal. Will it happen?
 
Vladimir Pastushak:

Has anyone tested the code ? Does it work in mt 4?

Not only does it work, but it is used to the fullest in such projects as MetaCOT 2.
 
Vladimir:
The impressive compression of historical data, together with the idea of applying it to RAM contents discussed above, suggests that there is hope for a radical reduction in the memory resources required by the MT5 terminal. Will it happen?
MT5 itself requires very few resources. But when requesting quotes, memory is used depending on the "max bars in history" variable. I don't know if this failed bug has been fixed, but they promised to remove it.
 

Something is broken, this is what happens when compiling.

The ZipHeader.mqh has an error.

 

The notorious casting of structures is swearing. Keep the corrected version in the attachment.

Files:
Zip.zip  11 kb
 
Vasiliy Sokolov:

The notorious casting of structures is swearing. Here is the corrected version in the attachment.

Should it work under MT4? I have this situation.
'm_array' - structures containing objects are not allowed       Dictionary.mqh  252     25
cannot cast 'DoubleValue' to 'ULongValue'       Dictionary.mqh  209     14
 
Stanislav Korotky:
Should it work under MT4? I have this situation.

Does#property #strist help?

 

The new version adds an additional mechanism for extracting data structures from the archive if the structure fields are not filled in. (It was noticed that Microsoft products spit on the standard from a high tower and simply do not fill in fields like "file size"). The new version now works with such problematic archives.

I've already added the new version to the archive to the article. It seems to have been updated already, so you can download the attachment to the article.

 
Stanislav Korotky:
Should it work under MT4? I have the same situation.

Now I have carefully looked at your error - the same situation with casting structures, but in a different container - CDictionary. Replace Include\Dictionary.mqh with the variant attached below.

And yes, everything works fine under MT4 and I will say more CZip is a workhorse in my projects on MT4.

Files:
Dictionary.mqh  38 kb