Familiar scenario: an optimizer has tried several hundred parameter combinations, the best showed 300% on historical data. The strategy goes live — and after a couple of weeks, the account is in the red. The test hasn't "broken" and the market hasn't "gone crazy." Most likely, this is over-optimization.

What is over-optimization

The market history consists of patterns and noise. When we test many parameters on the same part of the history, sooner or later a combination is found that fits not a pattern, but noise — random coincidences of that particular slice of data.

Such a strategy "knows" nothing about the market. It has memorized the past — and falls apart on any new data. The more parameters and the more optimizer runs, the higher the chance of getting exactly this result.

Signs that a strategy is overfitted

  • The result lives on a peak, not a plateau. The best parameter set gives 300%, while a nearby (slightly shifted) value gives −20%. Real patterns are stable to small parameter changes; noise is not.
  • Few trades. With 30–50 trades, statistics mean almost nothing: a few lucky coincidences make any curve look good.
  • All profitability comes from a couple of trades. Remove one or two best trades — and nothing remains of the result.
  • Too perfect a curve. Steady growth almost without drawdowns on a volatile market is a reason not to rejoice, but to be cautious.
  • No explanation of "why it works." If the strategy lacks a clear reason to earn money (participant behavior, market structure, costs of others), then most likely it doesn’t.

How to test a strategy honestly

Out-of-sample test. Basic hygiene: history is split into two parts. The strategy is tuned on the first part, and then only tested on the second, without tweaking. The result on the second part is the honest assessment.

Walk-forward. A stricter version: the "tuned on one segment — tested on the next" window slides over the entire history. The strategy repeatedly encounters unseen data. If the result holds from window to window, it's a sign of a pattern, not overfitting.

Parameter stability. Look not only at the best parameter set but also around it: if small parameter shifts don’t ruin the result, you have found a plateau, not a random peak.

Honest simulation conditions. Commissions, slippage, funding, liquidity. Without these, even the correct method tests the strategy in a fairy tale world — but trading will happen in the real one.

Where this is already built in

We built XTester around exactly this problem: the tester should help catch overfitting, not help ignore it. Data splitting for out-of-sample testing, honest accounting of costs in simulation, metrics to show result stability — these are parts of the engine, not add-ons.

Early access is open: xtester.pw.

Frequently asked questions

How much history is needed for testing?
There is no single figure — the number of trades and market condition variety matter more: the strategy should pass through trend, sideways, and high volatility. A hundred or two trades in varying market regimes tells more than thousands of candles in a quiet year.

What’s better: simple history split or walk-forward?
Splitting is the minimum requirement, walk-forward is a stricter time stability check. If the strategy is aimed at real money, walk-forward is worth the time spent.

The strategy failed the out-of-sample test. Fix it or discard it?
Beware: if "fixing" means tweaking parameters until it passes — that is just manual over-optimization. You should return to the hypothesis: why should the strategy work at all? No answer — no strategy.