data integrity check

 

Do any of you have any routines for checking to ensure the integrity of data? For example, I noticed last night a hole in my daily bar data from FXDD on June 22nd (incomplete data for that day) which completely threw off the pivot levels for the next day. Fixing that made the levels work perfectly.

I was thinking of a script or EA which would check the daily bars and send out an alert (email to admin) if the volume on the day was significantly lower than the average. I'd then go in a manually examine the situation and make any corrections as needed. Such a check would have easily caught the problem with the data described above.

Just curious how you go about dealing with data integrity.

Thanks in advance for any pointers.

Bill

 
billworld wrote:

Do any of you have any routines for checking to ensure the integrity of data? For example, I noticed last night a hole in my daily bar data from FXDD on June 22nd (incomplete data for that day) which completely threw off the pivot levels for the next day. Fixing that made the levels work perfectly.

I was thinking of a script or EA which would check the daily bars and send out an alert (email to admin) if the volume on the day was significantly lower than the average. I'd then go in a manually examine the situation and make any corrections as needed. Such a check would have easily caught the problem with the data described above.

Just curious how you go about dealing with data integrity.

Thanks in advance for any pointers.

Bill


Follow up on this. Well, even though I manually fixed the data, looks like it reverts back to the wrong data when the platform is re-launched. How do I prevent this? I've notified the broker about the bad data, but, I need to be able to work off of the correct data. Do I fix the data and then only work offline untilt the broker fixes the bad bar?
Reason: