Here’s the most important idea in honest strategy testing, and it’s simpler than it sounds. When you build a trading strategy, you tune it using historical data — you adjust the rules until it performs well. That period of data is called in-sample. It’s the data the strategy got to “study.”
The problem is obvious once you see it: of course the strategy looks good on the data you used to build it. You shaped it to fit. Showing off in-sample results is like bragging about acing an exam when you wrote the questions yourself.
The real test
Out-of-sample data is the data the strategy never saw during design. You lock the rules, then run them on a fresh, untouched period. If the strategy still works here, you might have found something real. If it falls apart — and most do — you’ve just been saved from betting on an illusion.
We split every dataset in two before we start. We design only on the in-sample years, then freeze everything and run once on the out-of-sample period. We never go back and adjust after seeing the result — that would defeat the entire purpose.
Why this is the whole game
Almost every strategy looks profitable in-sample. That’s not a sign of skill — it’s the default. The strategies worth anything are the rare ones that keep working out-of-sample. This single distinction separates serious research from marketing, and it’s why our Strategy Graveyard exists: most ideas simply don’t survive the real test.
In-sample is the practice exam you wrote yourself. Out-of-sample is the real one. Only the second one counts.
- 1In-sample data is what a strategy is built and tuned on — it almost always looks good.
- 2Out-of-sample data is fresh and untouched — it’s the only honest test.
- 3Adjusting a strategy after seeing out-of-sample results quietly destroys the test.
out-of-sample (OOS) — the core of our entire methodology.