We overfit a strategy on purpose

Here's the winner of a parameter sweep. A moving-average crossover on Bitcoin, eighteen months of hourly data, honest fills and fees: +17.96%, a Sharpe ratio of 0.88, a worst drawdown of 12.6%, across 91 trades. It beat 71 other parameter combinations on Sharpe, which is the risk-adjusted measure a careful trader would rank by rather than raw return.

Then we ran that exact configuration on the next thirteen months, which it had never touched: -14.32%, Sharpe -1.21, drawdown 18.1%.

This is the most common way retail traders lose money on a system that "backtested well." It's worth watching closely, because the usual explanation for it is only half right.

The protocol

We fixed all of this before running anything:

• Strategy: EMA crossover on BTCUSDT, 1-hour candles
• In-sample: 1 January 2024 to 30 June 2025, the only data used to choose parameters
• Out-of-sample: 1 July 2025 to 1 August 2026, never touched during selection
• Grid: 72 fast/slow combinations (fast 5–50, slow 20–200, keeping slow at least double fast)
• Costs: full realism on every run. Taker fees, spread, slippage, intra-bar fill resolution, funding.

Then we ran every combination on both windows. That last detail matters. We didn't just test the winner out-of-sample, we tested all 72, and that's what makes the rest of this article possible.

The champion, and the nine runners-up

(That's ordered by in-sample Sharpe, so the second row's slightly higher return isn't a typo. Ranking by raw return swaps the top two and changes nothing that follows.)

Ten best-in-sample configurations. Ten out-of-sample losses. Whichever one you'd picked, by whatever selection rule, you would have lost money.

Across the whole grid: 14 of 72 combinations were profitable in-sample. Two of 72 were profitable out-of-sample, and they returned +1.33% and +0.50%. After a year of screen time, that's a rounding error.

The part everyone gets wrong

Here's the obvious objection. Bitcoin rose 153% during the in-sample window and fell 41% during the out-of-sample one. Of course everything broke. That's regime change, not overfitting.

Fair objection, and the full grid answers it. Compare the distributions instead of the winner:

The median combination performed about the same in both regimes: -17.69% against -16.04%. The typical EMA crossover was a mediocre loser in the bull market and a mediocre loser in the bear market. What collapsed wasn't the middle of the distribution. It was the top. The best result fell from +18.06% to +1.33%.

If regime change were the whole story, the median would have moved too. It barely did. The damage lands precisely where we went looking for a winner, and that's the signature of selection rather than of markets.

Which is what overfitting actually means. Not "the market changed" but "we searched a noisy surface for its highest point, and height on a noisy surface is mostly noise."

In-sample ranking isn't worthless, it's just not enough

Here's the nuance we didn't expect. If in-sample results were pure noise, the in-sample and out-of-sample rankings would be unrelated. They aren't. The Spearman rank correlation between them is +0.394 on return and +0.279 on Sharpe.

So the sweep did learn something real. Slower parameter sets genuinely behaved better than faster ones in both windows, mostly because they trade less and pay less (see our cost ladder experiment for why that dominates everything).

But a positive rank correlation isn't the same thing as a profitable selection rule. The two configurations that made money out-of-sample ranked 13th and 17th in-sample. No sane process picks the 13th-best backtest. The ranking carried a little signal, the level carried none, and the level is what you trade.

Does forward-walking fix it?

The standard answer to overfitting is to stop picking parameters once and start re-picking them on a rolling basis. Optimise on a recent window, trade the next one, repeat. We ran exactly that over the same out-of-sample period: 3-month optimisation windows, 3-month trading windows, the engine's default parameter grid.

Total: -20.35%, against the frozen champion's -14.32%.

Forward-walking didn't save it. We're including this because the marketing version of this article would have stopped one section earlier, and because the result teaches two things.

First, look at the third row. The optimiser examined its window, decided no parameter set had positive expectancy, and refused to trade for a quarter. That's the mechanism working exactly as designed. When there's nothing to fit, the honest answer is cash.

Second, look at what it chose: 12/50, then 30/100, then nothing, then 12/50, then 30/100. The "best" parameters flipped between two distant regions every three months. Parameters that genuinely capture market structure don't oscillate like that. This is a system re-fitting noise on a fresh schedule, and forward-walk analysis is honest enough to show you that instead of hiding it.

The lesson isn't that rolling re-optimisation is useless. It solves one problem, stale parameters, and it can't solve the other one. No validation method manufactures an edge the rule family doesn't have. A crossover on one asset doesn't become profitable because you re-fit it more often. It just loses money with better hygiene.

What to do with this

Hold out data before you start, not after. Your out-of-sample window has to be data you've never looked at, chosen before you know what you want it to say. Once you've peeked, it's in-sample forever.

Judge a sweep by its distribution, not its maximum. If 14 of 72 combinations are profitable and the median loses 18%, the honest reading is "this rule family loses money and a few cells got lucky," not "I found the good one." Look at the median and the spread before you look at the top.

Be suspicious in proportion to how hard you searched. One backtest of a hypothesis you formed in advance is weak evidence. The best of 72 is much weaker evidence, because you've given noise 72 chances to look like skill.

Check whether the winners cluster. Genuine parameter effects show up as broad plateaus of similar results. A single sharp peak surrounded by losers is noise wearing a hat.

Honest caveats

One asset, one strategy family, two windows. EMA crossovers are widely copied and easy to verify, which is why we used one. They aren't good strategies, and a rule family with a real edge would degrade less than this.

That said, "the strategy had no edge to begin with" isn't a defence of the method. You can't know that before you run the test, and the sweep's +17.96% winner is exactly what "it has an edge" looks like from inside the in-sample window.

The finding we'd defend: the top of a parameter sweep is the least trustworthy number in it, and the more combinations you try, the less it means.