Discussing the article: "Benchmark Champions vs. Real-World Optimization Problems"

[MetaQuotes]

Check out the new article: Benchmark Champions vs. Real-World Optimization Problems.

We analyze why ratings may be inflated due to the alignment of algorithm trajectories with benchmark diagonals, and we supplement the testing methodology with a requirement to keep the global extremum away from the diagonals. We update Forest and Megacity, and perform RAW verification and calibration using VerifyExtremes.mq5. A drop in HHO and DOAm performance serves as a practical indicator of false leaders.

You run dozens of population-based optimizers on the same test bench and see a neat table of top performers. The problem is that some of these victories do not reflect the algorithm’s true search power: a high result may stem not from the algorithm’s ability to find the optimum, but from a coincidence between the geometry of the algorithm’s steps and the geometry of a specific test function. A maximum located on the boundary, at the center, or near the diagonal of the normalized space gives certain classes of methods a systematic advantage.

In this article, we formalize this risk, especially the “diagonal effect,” provide a measurable diagnostic criterion, and show how to eliminate structural advantages from a benchmark set. After reading this, you will get not an abstract diagnosis, but concrete tools: a rule for checking the position of an extremum, a procedure for verifying RAW extrema (VerifyExtremes.mq5), and an example of practical correction of test functions.

Author: Andrey Dik