A familiar scenario: the optimizer tried several hundred parameter combinations, the best showed 300% on historical data. The strategy goes into live trading — and after a couple of weeks the account is in the red. The test didn’t "break" and the market didn’t "go crazy." Most likely, this is overfitting.

What is overfitting

The market history consists of patterns and noise. When we iterate over many parameters on the same segment of history, sooner or later a combination is found that fits not a pattern, but noise — random coincidences specifically in that piece of data.

Such a strategy “knows” nothing about the market. It has memorized the past — and falls apart with any new data. The more parameters and the more optimizer runs, the higher the chance of getting exactly this result.

Signs that the strategy is overfitted

  • The result lives on a peak, not a plateau. The best parameter set gives 300%, but a neighboring one (slightly shifted value) already yields −20%. A real pattern is stable against small parameter changes; noise is not.
  • Few trades. With 30–50 trades, statistics mean almost nothing: a few lucky coincidences make any curve look good.
  • All profitability comes from a couple of trades. Remove one or two best and the result disappears.
  • Too perfect a curve. Smooth growth with almost no drawdowns in a volatile market is a reason for caution, not celebration.
  • No answer to "why does it work". If the strategy has no clear reason to earn (participant behavior, market structure, costs of others), most likely it doesn’t.

How to test a strategy honestly

Out-of-sample test. Basic hygiene: history is divided into two parts. On the first, the strategy is tuned; on the second, it is only tested without adjusting anything. The result on the second part is the honest assessment.

Walk-forward. A stricter variant: the window “tuned on this segment — tested on the next” moves along the entire history. The strategy repeatedly encounters data it “hasn’t seen.” If the result holds from window to window — this is a sign of a pattern, not overfitting.

Parameter stability. Look not only at the best set but at the neighborhood: if small parameter shifts do not ruin the result — you have found a plateau, not a random peak.

Honest simulation conditions. Commissions, slippage, funding, liquidity. Without these, even a correct method tests the strategy in a fairy tale world — but it will trade in the real one.

Where this is already built-in

We built XTester around this very problem: the tester should help catch overfitting, not help ignore it. Data splitting for out-of-sample testing, honest cost accounting in simulation, metrics that show result stability — these are part of the engine, not an add-on.

Early access is open: xtester.pw.

Frequently asked questions

How much history is needed for the test?
There is no universal number — what matters more is the number of trades and diversity of market conditions: the strategy should pass through trend, sideways, and high volatility. A hundred or two trades across different market regimes say more than thousands of candles from one calm year.

Which is better: simple history split or walk-forward?
Splitting is the bare minimum; walk-forward is a stricter test of stability over time. If a strategy aims for real money, walk-forward is worth the time spent.

The strategy failed the out-of-sample test. Fix it or discard it?
Caution: if “fixing” means tweaking parameters until it passes — that is the same overfitting, only done manually. Better return to the hypothesis: why should the strategy work at all? No answer — no strategy.