Imported Tick Data is Removed on Start-up

 

Problem Summary:

When I close MetaTrader 5 and reopen later, the imported tick data for my custom financial instrument has been eliminated. I must reload it. This takes quite a bit of time. Actually, MetaTrader 5 seems to also delete the TKC files when the application is open. It just did this after I ran optimization.


Details:

I am importing 10+ gigabytes of tick data for a custom financial instrument as described here:

https://www.metatrader5.com/en/terminal/help/trading_advanced/custom_instruments

Everything works great in the strategy tester. I am very happy with this. However, when I close MetaTrader 5 and reopen later, the tick data is no longer available in the Symbols utility's tick section for the custom instrument (toolbar icon "Show symbol list, Ctrl-U"). Also, if I try to start the Strategy Tester, it does indicate the start date of the data in the Journal, but the tester never begins any obvious testing operations. In fact, the Start button immediately flips back to Start. Normally, it changes to Stop to indicate there is something running. Therefore, it is either failing or not finding any tick data. There are no errors in the journal. The only way to resolve the issue is to import the data again. Then, everything works as usual.

I found the folder with the TKC files:
C:\Users\ugliestUserEver\AppData\Roaming\MetaQuotes\Terminal\D0E8209F77C8CF37AD8BF550E51FF075\bases\Custom\ticks\MY-CUSTOM-INSTRUMENT

If I check the history folder, TKC files are there before I close Metatrader 5. Upon restart, they are no longer there. Due to this, I have to import it again. MetaTrader also seems to delete the TKC files while open, as this just happened to me after running optimization.

I have searched quite a bit to determine if this is a known issue, and I have found nothing. Has anyone else heard about this or experienced it?

Wish List for MetaQuotes Developers:

It takes a while to load decades of tick history. Therefore, I would like for MetaTrader 5 to keep what I import. If that is not standard functionality (i.e. MetaTrader intentionally deletes tick data), I would like to request that the development staff add an option to "keep tick history forever." There can be a nice pop-up when this is clicked that the user is in danger of filling up their hard-drive, or there can be a cut-off where the data rolls off if the hard-drive space gets below a pre-set value, such as 100 GB.

Workaround:

I tried saving the TKC files before MetaTrader 5 deletes them. However, if I copy them into the folder before MetaTrader 5 starts, it deletes the TKC files. Therefore, I have to do the following:

1. Save the TKC files before MetaTrader 5 deletes them. I put mine in a .ZIP file.

2. Start up MetaTrader 5.

3. Unzip/copy them to the same folder.

4. Proceed with testing.

The above took about 1-2 minutes instead of the 1+ hours it takes to import the data.

Custom Financial Instruments - For Advanced Users - MetaTrader 5
Custom Financial Instruments - For Advanced Users - MetaTrader 5
  • www.metatrader5.com
The trading platform allows creating custom financial symbols. You can view charts of such symbols and perform technical analysis, as well as use them for testing trading robots and indicators in the Strategy Tester. If your broker does not provide the instrument, on which you want to test your strategy, or the provided history depth and the...
 

Just a heads up to anyone else experiencing this problem... MetaTrader 5 has deleted my custom instrument's TKC files while in the middle of an overnight optimization. It definitely deletes them on start-up, every time. However, it also deletes them at various times throughout the day/night, while the application is open. I have experienced this frequently.

Another thing that I seem to have experienced is issues with data synchronization, when using the local network feature. I have another computer on the network, and I used it to perform optimizations. However, the tick data did not seem to synchronize properly. My results were way off, and when I tried to reproduce them using only 1 computer, I could not. I decided just to turn off the local network feature entirely. I have no certainty that this is occurring, but I have enough certainty that I will no longer use the feature with tick data. In fact, I would not even venture to use the cloud network with tick data, due to this experience. When I have more time, I may try to reproduce this and nail it down for certain. It is not a high priority for me.

 
I ran a long-time optimization with tick data on a custom financial instrument that took over 2 days (single computer). While it was running, MetaTrader 5 (build 1755) deleted the TKC files.  I suspected this skewed my results, but I waited until the end. I then ran a single run on one of the results to determine if the deletion skewed the data. It appears the tick data for optimization is cached at the beginning. Therefore, deletion of the TKC files does not appear to affect running optimizations. I thought I would post this, in case others found themselves in similar circumstances.
 

I have the same problem. I`ve imported ticks and M1 bars in previously created custom symbol and then I made few backtests on real ticks without any problems. I even restarted the MT5 Terminal during the same day and the ticks were still there. But at the next day, all the ticks just disappeared. The Bars are still there, but...no more ticks.
However I just read that developers knows about this bug: https://www.mql5.com/en/forum/240592

 
I got the same problem, really annoying..
 

For everyone that need a solution, here is one:

Go to the folder where ticks are saved and remove the permissions to delete the folder or the file inside of it.

  • Right click on the folder (for me the folder is in C:\Program Files\MetaTrader 5\Bases\Custom\ticks) and then Properties
  • Security tab
  • Advanced
  • Edit authorizations
  • Select the proper user or user group
  • Edit the authorizations as needed. I removed the permission to Delete the folder or the files inside it

Problem solved..

Reason: