How it tells skill from luck

An optimisation always has a best row. These are the checks Strategywright runs to see whether that row found something real, and what none of them can tell you.These checks show whether it found something real.

  1. 01Count the triesMany settings tested means a lucky winner is likely.
  2. 02Check the neighboursTrust a plateau, drop a lone spike.
  3. 03Group and trade offClusters and the Pareto front.
  4. 04Walk it forwardTune on the past, test on the months after.
  5. 05Reshuffle the tradesMonte Carlo shows the range of falls.
  6. 06Mix with a limitNo two strategies that move together.

01

The chance problem

An optimisation tests thousands of settings. Test enough of them and one will look great by chance alone. With five years of data, about 45 truly different tries are almost sure to turn up a winner with no edge at all.

So the top row of a sorted optimisation is the row most likely to be luck. Every check below asks the same question of it: would a small change, or a period it never saw, still agree?

Bailey, Borwein, López de Prado and Zhu, “Pseudo-Mathematics and Financial Charlatanism”, Notices of the AMS, 2014.

The example file, sorted by profit
RankFast MASlow MAProfit
141260+47,313
2985+39,108
333215+34,216
and 2,497 more rows

Sorted by profit, the top row looks like an edge. Every example on this page is made up, for illustration.

02

Neighbours and plateaus

A real edge rarely lives on one exact setting. If Fast MA 41 works, 40 and 42 should work nearly as well. Luck has no such reason to spread.

  • The neighbour check scores each top setting from 0 to 1 against the settings around it: 0.75 and up is a broad plateau, 0.5 to 0.75 is moderate, under 0.5 is a lone spike.
  • The plateau analysis starts from the whole file and finds the regions where good settings gather: the place to pick from.
  • Gaps in the grid. A genetic optimisation tests only a sample, so the checks that need the neighbours show a notice when your file is missing too much of the grid. Every other tool works as normal.
A grid of settings: one lone bright square on the right, a broad bright patch on the left5152535455560120180240300Fast MASlow MA
  • Lone spike: the brightest square, its four neighbours dark. Score 0.07.
  • Broad plateau: about as bright as every neighbour. Score 0.95.
Brighter means more profit.

03

Clusters and the Pareto front

Profit is one measure. Drawdown, profit factor and trade count are others, and the best on one is rarely the best on all.

  • Cluster analysis sorts every setting into a few groups that behave alike. Several healthy groups suggest the strategy has more than one way to work.
  • Pareto analysis removes the settings that are worse on every measure you chose. What is left is the Pareto front, and the trade-off along it is yours.
Settings plotted by drawdown and profit, in three groups, with the Pareto front along the top left edgeMax drawdown (lower is better)Net profit
  • Pareto front: no other setting beats these on both at once.

04

Walk-forward, WFO and RWFO

A walk-forward test (WFO) tunes the settings on one stretch of the past, then tests the pick on the period after it. Those months played no part in choosing it, so they are a fairer test than the backtest itself. A rolling walk-forward (RWFO) repeats that, period after period, and records how often the pick held up, on a track made only of months the settings never saw.

Walk-forward is part of the Strategy creation and backtesting module, which is being built now.

One walk-forward window, then six rolling windows, each tuned on the past and tested on the months afterWFORWFOheldhelddid notheldheldheldStartYear 1Year 2Year 3Year 4
  • Tuned on
  • Tested on months it never saw
  • Test period that did not hold up
In this example the pick held up in five of the six test periods.

05

Monte Carlo

A backtest is one order of trades. The same trades in another order can dig a much deeper hole before they recover.

Monte Carlo runs a strategy’s own trades many times, drawn at random or played in a new order, each with its swap and commission.

You read the spread, not one line: the middle profit, the worst and best cases, the share of runs that end in a loss, and how deep the falls go.

Forty equity lines from the same trades in different orders, fanning out0Trades, in the order each run drew them
  • The middle run
  • A run near the bottom 5%
Forty of the runs shown.

06

The portfolio correlation limit

Two strategies that lose on the same days are one risk taken twice. The Portfolio Optimizer measures how each pair’s daily profit moves together, drops any mix with a pair above the correlation limit you set, and ranks the rest by Return/Drawdown, profit factor, win rate, Sharpe or Sortino. It chooses which strategies go together and never weights them.

EURUSD trendXAUUSD swingGBPUSD rangeUSDJPY trendEURGBP trend
EURUSD trend10.120.310.220.86
XAUUSD swing0.121-0.050.180.09
GBPUSD range0.31-0.0510.140.27
USDJPY trend0.220.180.1410.19
EURGBP trend0.860.090.270.191
  • 0.86 is over the limit: no mix may hold both EURUSD trend and EURGBP trend.

07

What it cannot tell you

  • It cannot promise a winning strategy. It can show that a result is likely luck. Passing every check makes luck less likely; it does not make an edge certain.
  • It only sees what the test saw. Costs, spreads and fills in your test set the limits of every result built on it.

Each term used here is defined in the glossary. The steps to run each check are in the docs.

Test your next strategy before it trades.