A backtest is a simulation: you take a trading rule and run it over historical data to see how it would have performed. Done honestly, it’s a useful tool. Done the way most of the industry does it, it’s a marketing machine that makes dead ideas look alive.
Trick one: testing on the data you tuned
The most common trick is the simplest. You adjust the strategy until it fits the historical data perfectly, then present those results as if they were a prediction. Of course it looks good — you built it to look good on exactly that data. This is the in-sample problem, and it’s why we insist on out-of-sample testing.
Trick two: parameter mining
Try fifty variations of a strategy, keep the one that worked best, and quietly forget the other forty-nine. With enough attempts, something will look great purely by luck. Presenting that single winner as if it were chosen in advance is one of the most seductive forms of self-deception in all of trading.
Trick three: ignoring costs
Many backtests assume trading is free. In reality, every trade pays fees and slippage. A strategy that trades often can look profitable on paper and lose money the instant real costs are applied. We model realistic costs on every trade for exactly this reason.
Our entire method is built to neutralize these tricks: out-of-sample testing defeats trick one, committing to one parameter choice in advance defeats trick two, and modeling real costs defeats trick three. When a strategy still survives all three, the result means something.
A backtest doesn’t prove a strategy works. It only shows what would have happened if you’d known the future — which you didn’t.
- 1Testing on the same data you tuned the strategy on guarantees flattering results.
- 2Trying many variations and keeping the best one manufactures luck that looks like skill.
- 3Ignoring fees and slippage can turn a real loss into a fake profit.
parameter mining — a failure mode we explicitly guard against.