Backtesting & Replay5 min read

Making Your Replay Test Reproducible

Replay Trader Editorial Team

Educational content for Indian traders on deliberate practice, chart replay, and setup discovery. Not investment advice.

A replay month that cannot be rerun is not a test. It is a story you will misremember after the next ugly week. Reproducibility here does not mean a Python seed and a tick database. It means a record complete enough that next month’s you — delayed, biased, tempted to “tweak one thing” — can load the same object and walk it again.

The one-page strategy spec freezes the stub. Testable sentences make the stub unambiguous. This page is the experiment wrapper: the test record that says which history, which fills, which speed, which peek policy sat around that stub.

If two people (or two Tuesdays) would not know they had run the same test, you cannot compare blocks. You can only collect vibes.

The Test Record Template (copy the block)

Create one record per experiment. New record if any circled field changes. Rename the file when you rename the stub.

TEST RECORD
id:              TR-YYYYMMDD-##
stub_id:         (must match the dated spec name)
spec_freeze:     YYYY-MM-DD
product:         NSE cash intraday / delivery swing / index chart  [pick one]
timeframe:       e.g. 5-min decision
session_window:  09:15–15:30 IST  or a written sub-window
universe_rule:   one sentence (not a live buy list)
universe_list:   file name or “randomised platform starts”
date_source:     randomised / listed / holdout-sealed
date_list_file:  dates-v#.txt  (hash or row count)
start_policy:    hidden-date Y/N; peek = automatic keep?=N
speed_mode:      freeze / session-real / compressed
fill_rule:       next-bar open / close / worst-of-bar / other (write it)
cost_note:       practice fills ignore brokerage, STT, slippage unless haircut
haircut:         none / tagged conservative  (no fake bps)
sample_card:     link or “six-bucket v1”
holdout_id:      none / HO-YYYYMMDD-##
log_file:        sheet tab name
n_planned:       decisions (not “until I feel done”)
n_keep_rule:     exclude keep?=N from setup-fit
edit_policy:     zero edits during block / one dated edit at review
operator:        you
started:         YYYY-MM-DD
ended:           YYYY-MM-DD

You do not need cryptographic hashes. You need a date list you did not silently extend, a spec you did not silently rewrite, and a fill rule you did not upgrade on winners only.

Protocol: freeze, run, seal, rerun

Step 1 — Freeze the object. Spec + test record filled. If a field is “TBD,” you are not in a test. You are exploring. Exploring is allowed; calling it a test is not.

Step 2 — Seal the date list. Write the dates (or the platform’s randomisation policy) before session one. Adding a famous week because the first ten days were choppy is a new record, not a continuation. If the platform randomises starts, write that: “Replay Trader randomised NSE starts, hidden dates, block TR-…” That sentence is the date_source.

Step 3 — Run with the log attached. Every decision points at id. Process scores follow the logging workflow. Peek rows stay in the file with keep?=N so you can see contamination, not so you can average them in.

Step 4 — Close the record. End date, actual n, count of keep?=N. One paragraph: what the test was for (usually process and classification, not a published edge). No win-rate headline.

Step 5 — Rerun drill (the point). At least once per quarter, pick an old TR-id. Do not open the old P&L first. Load spec + dates + fill_rule. Walk a subset (even five sessions). Compare process tags, not rupees. If you cannot find the date list, the original “test” was not reproducible. If you “remember how it went” and skip the walk, you are using narrative memory — the cousin of look-ahead.

What must be identical on a rerun

Must match May differ Must not silently differ
stub sentences your mood stop distance “because this name is wilder”
session_window platform UI 09:15 filter dropped after a gap
fill_rule whether you pause longer filling at the wick you already saw
universe_rule a name the randomiser draws adding a hero stock mid-block
peek policy life interruptions replaying until the day looks friendly

A rerun that “improves” because you now skip the losers you remember is not a rerun. It is a contaminated sequel. Put sequels in a new TR-id and keep the old one sealed.

Indian data hazards to write down, not assume

Corporate actions and index reconstitution change what a symbol was. If the universe_list is “today’s index,” write that limitation on the record; do not discover it in the recap. Split-adjusted charts versus the print a trader saw that morning are different objects — pick one and name it.

Liquidity: a large-cap now may have been a different tape then. If you cannot know, shrink the claim on the record: “current liquid names, randomised dates,” not “the NSE.”

Limits of a tidy record

Reproducible practice is still practice. It does not become a coded backtest. It does not survive live brokerage, STT and slippage unless those are in the fill_rule as a deliberate haircut — and even then it is a humility device, not a forecast.

A 30-day block with ~300 decisions can be reproducible and still too small to crown a stub. Reproducibility stops you from moving the goalposts. It does not enlarge the sample.

Holdout dates belong in a separate record. Mixing them into a rerun of the train block is how people fake out-of-sample. See holding out data.

Write the test so delayed-you can load it without asking present-you what you “meant.” Join the waitlist.

Educational note

Educational practice content only. Not investment advice. No guaranteed returns. Practice ≠ live profitability.