BUILD: COURSE 3 | LESSON 5
Backtesting and forward-testing on demo
Learning objectives
Run an honest manual backtest — hidden-right-edge replay, costs included, every signal counted — and name the five biases that fake most published results.
Split data into design and validation periods, and explain why a strategy only gets one shot at its out-of-sample data.
Forward-test a frozen spec on demo with pre-registered success criteria, and make a pass/revise/kill decision from the numbers.
Back and forward testing explained
You now hold three strategy skeletons and, from P3.1, a one-page spec of your own. Every walkthrough ended the same way: the expectancy of your version, on your market, is unknown until you test it.
This lesson is that test. The procedure that turns "I like this idea" into "over 143 signals across two regimes, this spec earned +0.14R/trade after costs, with a max drawdown of 9R."
The order is fixed and each stage exists to catch a specific way traders fool themselves: backtest → out-of-sample check → forward test on demo → decision.
Stage 1: The honest manual backtest
A manual backtest replays history candle by candle and records every trade your spec generates. Mechanics: in MT4/MT5, open your instrument and timeframe, scroll back 12+ months, and step forward bar by bar (F12 in MT4/MT5) — the critical discipline being that you never see the right side of the chart before deciding. For each valid setup, record in a sheet: date, direction, entry, stop, size per your risk rule, exit, result in R, and the rule that closed it. Simulate the whole spec, including filters and partials, and subtract realistic costs from every trade — spread plus slippage, e.g. ~0.05–0.10R for the trend spec's wide stops, more for tight-stop styles (P2.2/P3.4 arithmetic).
Target at least 50 trades, ideally 100+ (P2.1: fewer is noise), across visibly different conditions — a trending stretch and a ranging one. Then compute the report card: win rate, average win/loss in R, expectancy (P2.1), max drawdown in R and longest losing streak (P2.3 — this is where you check the streaks you must plan for), and profit share of the top 5 trades (if 3 trades made all the money, the edge is fragile — one missed signal live and the year is gone).
Now the five biases that quietly fake results — audit your own test against each:
- Look-ahead bias: deciding with information unavailable at the time (the classic: reading the right edge; the subtle: using a candle's close to enter at that candle's open).
- Survivorship/selection bias: testing only the pair and period where you already saw the strategy working — you sampled your winners.
- Cherry-picking signals: skipping "obviously bad" setups in hindsight. The rule is brutal: every valid signal counts, or the test measures nothing.
- Ignored costs: mid-price fills, free spreads, no slippage — decisive for thin edges (a +0.15R system with 0.08R costs is a coin toss).
- Curve-fitting: tuning parameters until history looks great. Nine filters and "ATR × 2.35" are its signatures (P3.1's filter budget exists for this reason). The next stage is the antidote.
Stage 2: Out-of-sample — the one-shot rule
Curve-fitting can't be avoided entirely — any design decision uses the past — so we contain it. Split your history: design and tune on one period (say 2024, "in-sample"), then run the frozen spec once on a period it has never seen (2025, "out-of-sample"). In-sample results are always flattering, by construction: you kept the versions that suited that data. Out-of-sample performance is the first genuine evidence. Expect degradation; the question is how much survives. A useful bar: out-of-sample expectancy should stay positive and retain a meaningful fraction of in-sample performance. In-sample +0.35R decaying to out-of-sample +0.18R is a robust result; decaying to −0.02R means the "edge" was fitted noise.
The discipline that makes this stage real: the out-of-sample data can be used once. If you peek, tweak, and re-run on the same validation period, it has silently become in-sample data and you are curve-fitting with extra steps. One test per spec version; a failed spec goes back to design with fresh validation data (another year, another correlated-but-different pair) or dies. Traders ruin more validation sets by "just checking one small change" than by any other route.
Stage 3: Forward-testing on demo — and what it tests that backtests can't
A passed out-of-sample check earns the spec a forward test: trading it on your demo account, in real time, every signal, for a pre-committed window — 8–12 weeks or 30+ trades, whichever takes longer. This is slower than backtesting and irreplaceable, because it tests four things history files can't:
- Real execution: actual spreads at your trading hours, slippage on stops, fills at news (P3.4's tax, measured on your account instead of assumed). Log intended vs actual fills on every trade.
- Real-time ambiguity: on the hard right edge, "3 touches, zones ≤ 0.25 × ATR" turns out to have judgement in it. Forward-testing calibrates your rule interpretations while errors are free — and every ambiguity you hit is a spec bug to fix in writing.
- You: taking the 6th signal during a 5-loss streak at 11pm is a different activity from clicking through it in replay. Demo P&L doesn't replicate live emotions fully — everyone trades braver on paper money — but process adherence under mild pressure is measurable, and if you can't follow the spec on demo you have your answer about live.
- The current regime, not last year's.
Before the first trade, pre-register the pass criteria in your journal — deciding after seeing results is bias #3 wearing a suit. A reasonable set: (a) ≥ 90% of valid signals actually taken; (b) expectancy after real costs > 0; (c) max drawdown within 1.5× the backtest's; (d) no rule ambiguity left unresolved in writing. Then the decision is mechanical. Pass all four → the spec graduates to consideration for small live size (that conversation — and whether trading live is right for you at all — belongs to the Professional level, A3's business plan). Fail (b) or (c) with good adherence → back to design; your next idea now starts from evidence. Fail (a) → the strategy may be fine and unsuited to you — a finding worth the whole exercise. Kill without guilt. Most first specs fail forward tests. That is the system working: every failed strategy retired on demo is an account-sized loss you never took, and the trader who has honestly killed three specs is measurably ahead of the one still "confident" in an untested first draft.
The Practitioner finish line
This closes both the course and the level. You can measure an edge (P2.1), size and survive it (P2.2–P2.3), manage and diversify it (P2.4–P2.5), specify it (P3.1), draw on three strategy families (P3.2–P3.4), and — as of now — test it without lying to yourself. Complete the demo task below to finish the Practitioner certificate. The Professional level assumes everything here as vocabulary.
Key takeaways
Test in fixed order — backtest (50–100+ trades, costs in, right edge hidden) → one-shot out-of-sample validation → 8–12 week demo forward test — each stage catching a distinct self-deception.
The five result-fakers: look-ahead, selection bias, cherry-picked signals, ignored costs, curve-fitting. Every published equity curve you ever see should be interrogated against them — especially your own.
Out-of-sample data is single-use: peek-and-tweak converts validation into curve-fitting with extra steps. Expect degradation; demand survival.
Forward-testing measures what history can't — real fills, rule ambiguity, and your own adherence — against pass/kill criteria pre-registered before trade one.
Most specs should die on demo; killing them there is free. An honestly killed strategy is evidence and progress, not failure.