A Backtest Experiment Log for Every Strategy Trial

Why log every strategy trial?#

A backtest report that shows only the winning version leaves out an important part of the research: how many alternatives were tried before that version was selected. When several candidate signals or strategy variations are compared on historical data, choosing the strongest-looking result can create selection bias if the search process is ignored. Backtesting Strategies Based on Multiple Signals | NBER Work on backtest overfitting likewise describes trying multiple strategy variations on historical data as part of the problem. https://sdm.lbl.gov/oapapers/ssrn-id2507040-bailey.pdf

A backtest experiment log template makes that search visible. It is a record of what you proposed, what you changed, what you examined, and what you decided—including tests that failed, produced no useful result, or were abandoned. It is not a test of whether a strategy is sound, and filling it in does not eliminate overfitting. Its purpose is narrower and practical: preserve a traceable account of the research so the final summary does not quietly become a story about only the successful trials.

The backtest experiment log template#

Use one row for each meaningful trial. If you change a parameter, rule, filter, instrument, or simulation assumption and run the test again, create a new row rather than overwriting the old one. Give each trial a unique ID and connect it to the saved configuration or output. The columns below work in a spreadsheet, database, or research notebook.

Copy these fields into your preferred tool. The exact labels can change; completeness and consistency matter more than the software.

A simple operating procedure helps prevent the ledger from turning into a retrospective reconstruction. Research-methods guidance emphasizes disclosing the hypotheses explored and analyses conducted; a contemporaneous record makes those decisions easier to report accurately. Full article: The ASA Statement on p-Values: Context, Process, and Purpose

1. Write the hypothesis before running the variation. State what should change and why, even if the explanation is provisional.
2. Record the baseline and exact change. Note the old value, new value, and whether any other rule or assumption changed.
3. Identify the run. Add the date, unique ID, parent trial, strategy version, and a reference to the dataset and configuration.
4. List what you inspected. Include the originally planned metric and any additional metric examined after viewing results; label them separately.
5. Enter the observed result before the decision. Preserve the values and definitions rather than replacing them with a summary such as “better.”
6. Make the decision explicit. Choose keep, reject, or investigate further, and write the reason or criterion in plain language.
7. Record the next step and save the artifact. Link the output and create a new row for any follow-up variation.
8. Retain every row. Keep unsuccessful, inconclusive, abandoned, and failed runs in the same history as successful-looking trials.

This example is a blank-style record, not a report of test results or a claim that a longer lookback is preferable. Replace the placeholders with actual observations. If you inspected an extra metric after seeing the run, add it and mark it as post-result exploration instead of implying it was planned.

The ledger is only useful if it captures uncomfortable results as readily as attractive ones. Undisclosed flexibility in analyses and selective presentation can distort how findings are interpreted; documenting the choices helps make that flexibility visible, but the documentation itself does not correct the analysis. False-Positive Psychology - Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn, 2011

• Do not delete a rejected trial because it made the final strategy look worse. It is still part of the search history.
• Do not overwrite earlier parameter values or results when a run is repeated. Preserve the earlier record and link the rerun.
• Do not use “optimized” as the only description of a change. Name the parameter, old value, new value, and any other edits.
• Do not report only the metric that favored the selected version. Record every metric you inspected and whether it was planned or added later.
• Do not treat “keep” as proof. It describes a research decision, not a conclusion about future performance.

Use the template as a lightweight research record. Keep the fields stable across trials, make each meaningful change a new entry, and save the underlying artifacts. The final summary can then describe both the chosen version and the path taken to choose it—without presenting the ledger as proof that the strategy will work.

Frequently asked questions#

Should I log every parameter change?

Create a new row for every meaningful variation, rerun, or changed assumption. Give it a unique ID and link it to the prior trial or baseline so the sequence remains visible.

Do rejected or failed trials belong in the log?

Yes. Record unsuccessful, inconclusive, abandoned, and failed runs as part of the search history. A final report that omits these trials can conceal how many alternatives were explored.

What if I checked a metric I had not planned to inspect?

Record the metrics you inspected, including measures added after seeing results, and label planned versus post-result inspection. Broader research guidance calls for disclosure of hypotheses explored and analyses conducted. Full article: The ASA Statement on p-Values: Context, Process, and Purpose

Does keeping an experiment log prevent overfitting?

No. The log documents the search; it does not establish that a strategy is valid or remove overfitting. Selection among candidate strategies can bias results when the search is not accounted for. Backtesting Strategies Based on Multiple Signals | NBER https://sdm.lbl.gov/oapapers/ssrn-id2507040-bailey.pdf

Keep the record of each run alongside your research workflow.

Scope and limitations#

This article addresses only how to document your search history while iterating on backtests. It does not explain validation methods or claim that recordkeeping prevents selection bias or overfitting.

Appendix: experiment log fields recap#

At a glance, your ledger should capture a unique trial ID, date/time, parent trial, hypothesis, exact variation, implementation version, test setup reference, metrics inspected, observed results, decision and reason, next step, and a link to the run artifact.

Sources#

• Backtesting Strategies Based on Multiple Signals | NBER — nber.org
• False-Positive Psychology - Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn, 2011 — doi.org
• Lawrence Berkeley National Laboratory — escholarship.org
• Full article: The ASA Statement on p-Values: Context, Process, and Purpose — doi.org
• https://sdm.lbl.gov/oapapers/ssrn-id2507040-bailey.pdf — sdm.lbl.gov
• https://escholarship.org/uc/item/4w1110bb — escholarship.org
• https://academic.oup.com/jrssig/article/18/6/22/7038278 — academic.oup.com
• https://arxiv.org/abs/2608.23808 — arxiv.org
• https://research.aalto.fi/en/publications/quantifying-backtest-overfitting-in-alternative-beta-strategies/ — research.aalto.fi
• https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3177057 — papers.ssrn.com
• https://senzoukria.com/glossary/research-record — senzoukria.com
• https://ideas.repec.org/p/arx/papers/2608.27734.html — ideas.repec.org
• https://www.frontiersin.org/articles/10.3389/fpsyg.2016.01832/pdf — frontiersin.org
• https://www.nber.org/papers/w21329.pdf — nber.org
• https://www.backtestlabs.com/docs — backtestlabs.com
• https://statmodeling.stat.columbia.edu/2012/11/01/researcher-degrees-of-freedom/ — statmodeling.stat.columbia.edu