Need help! Can't solve the problem, I'm hitting hardware limitations - page 20
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
komposter, can you use a DLL in your EA?
If so, you can do the following:
Pack the data into a header table file, using ZLIB
(http://www.zlib.net/)
It will work very fast (you'll be surprised how fast.
The DLL will be ready to work in 3 msec. Everything will work in "real time").
Data will be reduced 5-8 times and at the end of the same file (packed)
will contain table with ID, offsets and data length.
If you have very big amount of records in source file then you have to compile
into a subtable (several subtables), indicating the offsets in the main table, so you don't need to
so that you don't have to look through the whole table, but only a small part of it.
For example: USD data are stored from 0 offset to 1023,
EU data from 1024 to 2047 etc.
If the data is not packed into one file (it will be huge), there will be
one more (tiny) subtable, in which the packer will indicate the file number.
And when DLL loads the files, it will create a common subtable from the subtables of
of all the files. Better yet, all of the offsets are stored in the first file, and if
we "step outside" of 1 file, the data is taken from the second file, etc.
I forgot...
If you take my advice, I recommend that you pack your
text data with Zlib string packing function (not binary data, it works faster).
I think the function is called ZCompressString...
Zipping, as well as encryption, can already be standard:
Data encryption methods
To specify method of data conversion (encryption and hash calculation), enumeration ENUM_CRYPT_METHOD is used in functions CryptEncode() and CryptDecode().
ENUM_CRYPT_METHOD
Constant
Description
CRYPT_BASE64
BASE64 encryption (transcoding)
CRYPT_AES128
AES encryption with 128 bit key (16 bytes)
CRYPT_AES256
256 Bit (32 bytes) AES Encryption
CRYPT_DES
DES encryption with key 56 bits (7 bytes)
CRYPT_HASH_SHA1
Calculate HASH SHA1
CRYPT_HASH_SHA256
Calculate HASH SHA256
CRYPT_HASH_MD5
HASH calculation MD5
CRYPT_ARCH_ZIP
ZIP archiving
Zipping, as well as encryption, can already be standard:
Data encryption methods
To specify method of data conversion (encryption and hash calculation), enumeration ENUM_CRYPT_METHOD is used in functions CryptEncode() and CryptDecode().
ENUM_CRYPT_METHOD
Constant
Description
CRYPT_BASE64
BASE64 encryption (transcoding)
CRYPT_AES128
AES encryption with 128 bit key (16 bytes)
CRYPT_AES256
256 Bit (32 bytes) AES Encryption
CRYPT_DES
DES encryption with key 56 bits (7 bytes)
CRYPT_HASH_SHA1
Calculate HASH SHA1
CRYPT_HASH_SHA256
Calculate HASH SHA256
CRYPT_HASH_MD5
HASH calculation MD5
CRYPT_ARCH_ZIP
ZIP archiving
The question is not about encryption, it's about how to access the data quickly.
The purpose of archiving is to reduce the size of the data and allow the fast transfer of offsets
not the main (20GB) file, but a 5-8 times smaller one.
But it is not enough to pack, you should also have a mechanism for fast access to data.
P/S Zlib has functions for fast string compression and decompression.
I pointed out that it is no longer necessary to have third-party dlls to pack or encrypt the data. As opposed to your DLL method.
I didn't talk about solving the topstarter's problem.
I pointed out that it is no longer necessary to have third party dlls to pack or encrypt the data. As opposed to your DLL method.
I didn't say anything about solving the problem of the topicstarter.
A DLL is not a data unpacker, but a mechanism for quickly extracting data from
a file packed according to a certain scheme.
Well, all this is now easily done using the language tools. The compression is available from the standard.
Great, now the topicstarter will probably solve his problem.
I've never worked with files in MQL5, I'll see if it's possible to open
I have never worked with files in MQL5.
Yes, it is :)
Everything works, and it's fast. I have described above methods to make file operations more efficient in our implementation.
I'm not understating the abilities and capabilities of the terminal, but
when I needed to extract data from a 1.21Gb file, with 21,345,728(!) lines, a couple of years ago,
http://ftp.micex.com/pub/info/historical_data/Securities_market/OrderBook20130206.rar
data of the form:
NO,SECCODE,BUYSELL,TIME,ORDERNO,ACTION,PRICE,VOLUME,TRADENO,TRADEPRICE
21345728,USD000UTSTOM,B,235000002,3568,0,29.6095,300000,,
Then by the method I indicated, the search time was 35-45 MICROSECUNDS,
It is true to say that the file was prepared for more than 2 days :(
P / S It's not about what to use (terminal or DLL), but how to prepare the data.
And the fact that there are new features in the terminal is very welcome!
I'm not underestimating the abilities of the terminal, but
when I needed to extract data from a 1.21GB file a couple of years ago, with 21,345,728(!) lines,
http://ftp.micex.com/pub/info/historical_data/Securities_market/OrderBook20130206.rar
data of the form:
NO,SECCODE,BUYSELL,TIME,ORDERNO,ACTION,PRICE,VOLUME,TRADENO,TRADEPRICE
21345728,USD000UTSTOM,B,235000002,3568,0,29.6095,300000,,
Then by the method I indicated, the search time was 35-45 MICROSECUNDS,
It's true to say that the file was prepared for more than 2 days :(