Discussing the article: "Data Science and ML (Part 36): Dealing with Biased Financial Markets"

[MetaQuotes]

Check out the new article: Data Science and ML (Part 36): Dealing with Biased Financial Markets.

Financial markets are not perfectly balanced. Some markets are bullish, some are bearish, and some exhibit some ranging behaviors indicating uncertainty in either direction, this unbalanced information when used to train machine learning models can be misleading as the markets change frequently. In this article, we are going to discuss several ways to tackle this issue.

Different forex markets and financial instruments exhibit different behaviors at different times. While some financial markets such as stocks and indices are often bullish in the long run, others such as forex markets often display bearish behaviors and much more, this uncertainty adds complexity when trying to predict the market using Artificial Intelligence (AI) techniques and Machine Learning models (ML).

Author: Omega J Msigwa

[linfo2]

Thanks, You Omega , Appreciate you putting this together , Bais is something we all fear. I have downloadted the attachments , Could I suggest it includes all the required componets . Thankfully you have the github so I was able to find and install the prerequisites (preprossing.mqh, plots.mqh ,Matrixextend.mqh, metrics.mqh and Random Forext.mqh). Unfortunately I then am stuck with the message ' Init - Undeclared Identifier  ' from the line if (!random_forest.Init(StringFormat("%s.%s.%s.onnx", symbol_, EnumToString(timeframe_), technique_name), ONNX_COMMON_FOLDER)) //Initializing the RFC in ONNX format from a common folder. I checked and I do have USDJPY.PERIOD_D1.randomundersampling.onnx in MQL5\Common folder

  

[Omega J Msigwa]

linfo2 #:
Thanks, You Omega , Appreciate you putting this together , Bais is something we all fear. I have downloadted the attachments , Could I suggest it includes all the required componets . Thankfully you have the github so I was able to find and install the prerequisites (preprossing.mqh, plots.mqh ,Matrixextend.mqh, metrics.mqh and Random Forext.mqh). Unfortunately I then am stuck with the message ' Init - Undeclared Identifier  ' from the line if (!random_forest.Init(StringFormat("%s.%s.%s.onnx", symbol_, EnumToString(timeframe_), technique_name), ONNX_COMMON_FOLDER)) //Initializing the RFC in ONNX format from a common folder. I checked and I do have USDJPY.PERIOD_D1.randomundersampling.onnx in MQL5\Common folder

  

The required components are the latest version of everything imported inside the notebook, you can do pip install without worrying about the versions conflicts. Alternatively, you can follow the link on the attachments table, it takes you to Kaggle.com where you can edit and modify the code.

Undeclared identifier, could mean a variable or an object isn't defined. Inspect your code or DM me send me a screenshot of the code.

[zhainan]

The test results are not credible. The test time period includes the time period of the training data. The model will perform well on the training set and poorly on the test set. You can see that many test results are in the time range of 2024.07.06 to 2025.01.01. The model performs poorly, while the model performance in the previous time is excellent. I calculated that the training set accounts for 80%. The data is not disrupted for random allocation, and the training set and test set are directly allocated in order. The model recognizes the previous training data but not the subsequent test data. You used the training data in the test, which will lead to inflated overall results, and the performance on the subsequent test data is very poor, with losses. Then the cluster centroid is also unreliable. The clustering uses all the data, and the overall data will be used for clustering and then training. Therefore, the training data used by this method includes the time range of 2023.01.01~2025.01.01. It can be said that this is a test completely on the training data, without testing on unknown data. The training data and the test data must be strictly distinguished in chronological order. Only test data can be used to test the performance of the model, such as 2023.01.01~2024.01.01 as training data, and 2024.01.01~2025.01.01 as test data. The test model can only be tested using test data. There should be no risk of information leakage, otherwise all the good performance of the model will be unreliable.