This backtest made +935%.Its real edge was zero.
Most people who try quant build a bot that hunts for trading edges. Quant Red Team does the opposite. It plants fake edges on purpose, then measures which audits catch them, how often each one wrongly rejects a real edge, and how much access an auditor needs.
Data table
Backtests lie.
Nobody can grade the lie detectors.
A backtest can look brilliant because you tried 120 variants and kept the best, because your code peeked at the future, because you forgot trading costs, or because your data only contains the companies that survived.
Statisticians have built many tests to catch this: the Deflated Sharpe Ratio, the Probability of Backtest Overfitting, White's Reality Check, Hansen's SPA, holdouts and placebo re-runs. But in real markets nobody knows which strategies truly work, so nobody can score the tests. You can't measure a detector without an answer key.
So this project builds the answer key. Every market is simulated, every flaw is planted on purpose, and an oracle knows the true edge of every claim.
Plant the crime.
Then send in the detectives.
One experiment, run once with its rules fixed in advance. The config and code were hashed before the final run; the results are locked.
Markets
Four simulated markets with fat tails and volatility clustering (GARCH, Student-t). The truth of each is known: no edge, a real trend, a reversal that costs destroy, or crashing stocks that get delisted.
Researchers
Ten complete research processes. Two are honest; eight carry one classic flaw each. Each turns a dataset into a claim.
Claims
Only backtests with t ≥ 2 count as claims, because flawed research is only ever shown when it looks good. 300 claims per researcher.
Audits
Fourteen audits in three access tiers judge every claim: credible or rejected.
Oracle
The ground truth: the same rule, implemented honestly, run on ~2,300 years of fresh data from the same market. A true Sharpe ≤ 0.05 means fake; ≥ 0.20 means real.
Ten researchers.
Eight are lying.
Each card shows the average Sharpe ratio the researcher claimed and the true Sharpe the oracle measured for the same rules.
The more access,
the more you can test.
Access tiers are enforced in the code: a tier-1 audit is literally handed nothing but the returns. No audit ever sees the oracle.
Who caught whom.
Each cell is the share of a researcher's claims that an audit rejected. For the eight flawed researchers a rejection is a catch, so darker is better. For the two with real edges it is a false alarm, so darker is worse. Hover a cell for counts and 95% intervals.
Hatched cells: not applicable (PBO needs more than one trial). Arrow keys move between cells. Download: matrix.csv · auc.csv · site-data.json
No audit sits
in the safe corner.
The ideal audit is top-left: it catches every fake and never rejects a real edge. The audits that catch the most do it by rejecting almost everything, including about four in five genuine edges.
Data table
Delay, Cost×2 and the point-in-time universe look weak here only because each catches exactly one flaw, almost perfectly, and almost never raises a false alarm. That is Exhibit C.
The Deflated Sharpe with an assumed 100 trials and the forward tests both catch over 85% of fakes. They also reject 65–83% of real edges. A smoke alarm that goes off every time you cook doesn't tell you much.
The Probabilistic Sharpe Ratio sits at zero on both axes. Every claim already passed a significance test, so asking the same question again with a fat-tail correction changes nothing.
One flaw, one antidote.
Never the same one twice.
The statistical audits in tiers 1 and 2 caught 0% of these four flaws: each researcher ran a single, honest-looking test, so there is no multiple testing to correct. Each flaw falls instantly to one cheap re-run attack, and every other attack misses it.
What do you need
to see?
For each flaw: the lowest access tier at which a usable audit catches at least 80% of fakes. Usable means it wrongly rejects at most 20% of real edges. That cap was set after the final run as a presentation choice, and it is logged as decision D12.
PBO is a
coin flip.
Forget pass/fail thresholds. Using each audit's continuous score, how well does it separate a researcher's fake claims from all real ones? 1.0 is perfect separation; 0.5 is a coin toss. The Probability of Backtest Overfitting lands at 0.50.
Data table
PBO asks whether the in-sample winner keeps winning across 12,870 ways of splitting history in half. A strategy that was lucky across its whole history looks consistent, so PBO passes it.
PBO grades the selection procedure, not whether this particular winner is real. Useful for research hygiene; useless as a verdict on one claim.
The Deflated Sharpe with the true trial count had the opposite problem. When the strategies tried really do differ, it counts the real differences as noise and rejected 100% of genuine mined edges.
Even real edges
are oversold.
Every faint dot is one claim's reported Sharpe. Filtering backtests through t ≥ 2 inflates everything: genuine mined edges were claimed at 1.22 against a true 0.69, and honest real edges at 1.02 against 0.72.
Data table
Paper trading
is painfully slow.
An honest researcher who simply got lucky is invisible to every usable audit. Only new data exposes luck, and new data is weak. The experiment's forward tests rejected 79% of genuine edges after one year and 62% after three.
Data table
A Sharpe ratio of 0.7 is a good strategy. Proving it from scratch takes over a decade of live results, and a weaker one takes longer. A year of paper trading can neither confirm nor refute a bot.
Run it again.
Same answer.
v1 could have been one lucky seed. So the whole experiment ran again on a fresh seed: 3,000 new claims from 35,312 new markets. Two new audits, designed after seeing v1, came along. The hypotheses were written down before the run and are reported pass or fail.
Data table
All 134 cells replicated within sampling error. The detection matrix is a property of the audits, not of one set of random numbers.
The two failures matter just as much. Pre-registering means a hypothesis can fail in public, and that is what makes the passes believable.
Two fixes.
Half-fixed.
DSR-eff stops the Deflated Sharpe treating real differences between strategies as luck. Under the null every trial's true Sharpe is zero, so its bar uses that variance (1/T) and counts correlated trials as an effective number. Full history re-runs the claimed rule from the earliest available date, so a cherry-picked start can't hide the bad years.
Data table
Data table
DSR-eff is now usable: false alarms on real mined edges fell from 100% to 15%. But it catches 70% of miners, where the Reality Check catches 87%. Full history never wrongly rejected a real edge, but it caught only half the window pickers, because the best start date usually covers most of the history anyway.
Then we tried it
on real markets.
The same researchers and audits ran on 20 liquid US ETFs: research on 2005–2018, then a look at what happened in the 2019–2026 "future". There's no oracle in real markets, so each rule's real future stands in for the truth. That's one draw per researcher: a story, not statistics.
Data table
The look-ahead bug produced the best-looking backtest in the study, with a Sharpe of 1.33. Only the placebo and delay attacks rejected it, exactly as the simulation predicts. Then it lost money.
The miner found a long-only trend filter that really did keep working, but it trailed simply holding all 20 ETFs. Its "edge" was mostly the market going up. The placebo rejects it for exactly that reason: sign-randomised data has no upward drift.
Most flawed researchers never even reach a publishable claim on real data. Look-ahead does.
Thirteen findings.
- Returns alone tell you almost nothing.The PSR caught 0%. The Deflated Sharpe with assumed trials caught most fakes only by rejecting 91% of honest real edges, and it caught 0% of look-ahead, whose fake Sharpe of 3.5 clears any bar.
- Disclosing every trial fixes selection bias, and only that.On the 120-rule miner: Bonferroni 96%, Reality Check 90%, SPA 89%. Every tier-2 audit scored 0% on look-ahead, leakage, costs and survivorship.
- PBO cannot judge a single claim.It caught 25% of mined fakes and 8% of asset-picked fakes, with an AUC of 0.50. It measures how consistent a selection process is, not whether this winner is real.
- The Deflated Sharpe punishes genuine variety.With the true trial count it rejected 100% of real mined edges, because real differences between strategies look like noise to it.
- Each implementation flaw has its own antidote.Look-ahead: delay 96%. Leakage: placebo 95%. Costs: cost×2 100%. Survivorship: point-in-time universe 96%. No single attack catches all four.
- The placebo is the best all-rounder, but it is blind to data and economics.It caught 85–98% of selection and leakage flaws but only 4% of survivorship and 8% of ignored costs, and it wrongly rejected 22% of real mined edges.
- Window picking is the hardest selection flaw.Choosing the best of 11 start dates turned zero edge into a claimed Sharpe of 1.05. The best usable audit caught 36%.
- Honest luck is invisible without new data.No usable audit catches a lucky honest test. Forward tests catch it, but they also reject 79% of real edges after one year.
- Selection inflates even real edges.Real mined edges were claimed at 1.22 against a true 0.69. The t ≥ 2 filter alone took honest real edges from 0.72 to 1.02.
- It replicates.On a fresh seed, all 134 cells landed within 10 points of v1 (largest gap 8). Pre-registered, and it passed.
- The Deflated Sharpe can be half-fixed.Counting correlated trials as an effective number cut false alarms on real mined edges from 100% to 15%, but it then caught only 70% of mined fakes. Pre-registered, and it failed the 80% bar.
- Window picking survives everything usable.Re-running from the earliest date caught 50% with zero false alarms: better, but still short. Pre-registered, and it failed.
- Real markets agree.On 20 real ETFs, the look-ahead bug's Sharpe of 1.33 got past every statistical audit. Only the placebo and delay attacks caught it, and it then lost money.
Practical rule: ask for every trial, and also re-run the pipeline with a one-day delay, double costs, delisted names restored, and placebo data. Statistics on the returns alone will not save you.
Pre-registered.
Hashed. Locked.
Every threshold was fixed before any run; dev runs used a different seed; the final run executed once and wrote a lock file. Every decision, including the ones made after seeing results, is logged with who made it and why.
Integrity
- Final config hash b39faa28…
- Code hash at run time a8897be1…
- Seed 314159265, distinct from dev
- v2: config d35a77ab…, code be2b188f…, seed 271828182
- 84 automated tests, including causality checks proving the honest engine never sees the future and the planted bugs really do
- Size checks: RC, SPA and the placebo reject ≤ 8% when there is truly no edge
Caveats
- The ground truth is exact for these simulated markets, which have no regimes, crowding or structural breaks.
- Conventional thresholds (α = 5%, PBO > 0.5, DSR < 0.95). The AUC exhibit is threshold-free.
- One family of simple rules, with 300 claims per researcher (95% intervals of about ±3–6 points).
- Trading cost is 2 bps per unit of turnover, a liquid futures-style universe (decision D5).
- Simulation research only. Nothing here is financial advice.
Reproduce
Python 3.14 with numpy and scipy only.
python -m pytest -m "slow or not slow" python scripts/calibrate.py python -m qrt run --config configs/dev.toml python -m qrt report results/final
Read: the paper (PDF) · RESULTS.md · DECISIONS.md · design spec · v2 pre-registration