Trading Backtests: Where Paper Profits Meet Costs, Timing and Evidence

A backtest is a reconstruction of decisions that could have been made with the information and resources available at the time. It becomes unreliable when a later observation influences an earlier signal, when a favourable price is assumed without execution evidence, or when repeated experimentation is presented as one independent test.

This reference report consolidates the strategy-lab presentations and expanded teaching notes into a single validation framework. The original numerical settings and illustrative performance claims are retained as research examples. They are not evidence of a currently profitable strategy.

Specify the experiment before computing its return

A strategy definition needs a universe, data frequency, information timestamp, signal rule, position-size rule, execution assumption and exit policy. It should also identify the benchmark and the economic reason the signal might work. A collection of profitable parameter combinations is not a substitute for that specification.

The source lab uses three different ideas: a moving-average crossover for trend, a reversal rule for mean reversion and a breakout rule adjusted by recent range. This is useful because the rules rely on different market behaviour. It does not imply they will be uncorrelated or collectively profitable in every regime.

Fast and slow moving-average settings in the same source version

Another version explores a wider grid. These are separate search spaces; a final report should disclose the actual number of configurations tried rather than showing only the winner.

Reconstruct when a decision was possible

The basic engine observes completed bars and executes at the next bar’s open. This avoids assuming that a signal using today’s final close could already have traded at that same close. A different execution convention can be valid, but it must correspond to an order process that could actually have obtained the price.

The source allocates 95% of available cash to an entry and leaves 5% as a buffer. That buffer is a teaching parameter, not proof that costs and margin can always be funded. Position rounding, fees, adverse open gaps and partial fills still need explicit accounting.

Event-driven architecture helps make the timeline visible, but it does not eliminate look-ahead bias automatically. A precomputed feature column can contain future information even when the trade loop advances one bar at a time. Conversely, a carefully lagged vectorised calculation can be valid. The control is the information set and execution model, not the programming style alone.

For multi-day labels, overlapping future-return intervals create additional leakage. Purge training observations whose label windows overlap the test interval and apply an appropriate embargo where dependencies warrant it. A simple chronological split is not always sufficient when labels or features span the boundary.

Build costs as dated rules, not one universal percentage

The teaching examples include a flat order charge of 20 and slippage assumptions ranging from two to five basis points per side. They also list historical transaction-tax, fee and stamp-duty inputs that differ between companion versions. These are retained implementation examples, not a current fee schedule.

The cost function should depend on the instrument, trade side, date, quantity and execution type. Apply each charge to the correct base: turnover, premium, order count or applicable service fee. A price-point slippage term must be multiplied by quantity before adding it to currency fees. A buy-only charge should not silently be levied on both sides.

Separate bid-ask spread, impact and delay costs where the data support it. Stress them when volume falls or the market gaps. A strategy that remains attractive at its central cost estimate but fails with slightly worse execution needs a different confidence level from one whose edge is large relative to its trading friction.

Use a complete account and return series

Track cash, positions, marks, funding, realised profit and unrealised profit at every observation. Equity should reconcile from one period to the next. Transfers of cash, additional margin and changes in capital should not masquerade as trading profit.

For returns $r_t$, cumulative wealth is the product of $1+r_t$ when those are simple period returns. Summing log returns and then exponentiating is a different but consistent route. Mixing the two methods can distort compounded results, particularly in volatile periods.

The source’s statement that a 40% drawdown requires “2.5 times upside” is incorrect. After falling from 100 to 60, recovery to 100 requires a gain of $100/60-1=66.67\%$. More generally, a fractional loss $d$ requires a subsequent gain of $d/(1-d)$. This arithmetic is central to understanding why loss depth and recovery time matter.

A performance report needs several complementary measures

Performance measures and their limits of interpretation

Sharpe can be calculated for non-normal returns; the calculation itself does not require normality. Normality and independence enter particular inferential approximations and interpretations. A high Sharpe can still conceal negatively skewed option-selling risk.

Expected shortfall normally exceeds the corresponding value-at-risk loss threshold even for many ordinary distributions. The inequality alone does not prove that the observed data have unusually fat tails. Compare the magnitude with the relevant benchmark distribution and acknowledge estimation uncertainty.

The win-rate examples show why outcome size matters. With an average win of two units and average loss of one, a 40% win rate produces an expected 0.2 units per trade before costs. With an average win of half a unit and average loss of one, a 60% win rate produces an expected loss of 0.1 units. Neither calculation assumes the trades are independent; it simply identifies the average-payoff arithmetic.

Separate parameter selection from evaluation

The teaching lab begins with a 70/30 chronological split. Optimisation occurs in the first 70%; locked parameters are then evaluated on the remaining 30%. This is a useful introductory holdout design. A full walk-forward process repeatedly trains on past information and evaluates subsequent observations according to a predeclared schedule.

An expanding window uses more accumulated data; a rolling window discards older observations and may adapt more quickly. Neither dominates in every market. The choice changes the bias-variance trade-off and should be tested without repeatedly inspecting and redesigning the final holdout.

Parameter heatmaps help identify whether a result is concentrated in one fragile setting. A broad region of acceptable outcomes can support robustness, but a plateau selected after a large search is still selected. Disclose the exploration rather than treating a visually broad green region as independent statistical confirmation.

The source gives illustrative in-sample/out-of-sample Sharpe pairs such as 1.8 to 1.1 and 2.0 to 0.4. These are diagnostic examples, not universal acceptance rules. A decline can reflect overfitting, a changed regime, sampling error or costs absent from the training model. Investigate rather than automatically assigning one explanation.

Preserve time dependence when estimating uncertainty

The advanced lab uses 2,000 bootstrap replications to estimate a Sharpe interval and compare a strategy with a benchmark. That is a useful computational exercise, but resampling individual daily returns destroys serial dependence. A block or stationary bootstrap can better preserve some dependence when the method’s assumptions are appropriate.

For a paired comparison, resample the strategy and benchmark together using the same dates or blocks. Independently resampling them changes the relationship between the two return series. The resulting difference distribution may answer the wrong question.

A 95% interval above zero is evidence under the resampling design; it does not prove that an edge is permanent or that future profit has a 95% probability. It also does not imply that every bootstrap sample was positive. Repeated strategy selection requires additional consideration because the chosen result benefited from the search.

Permutation tests have similar constraints. Shuffling a strategy’s realised returns is not the same as shuffling its signals and rerunning execution, costs and sizing. A valid null experiment should preserve the structures not being tested while breaking the specific relationship under investigation. The hypothesis must be stated before the randomisation.

Attribution distinguishes exposures from unexplained returns

A regression of strategy returns on selected factor returns can identify exposures and a residual intercept. The advanced teaching example reports market beta around 0.35 and an explanatory $R^2$ around 0.28. Those are source-reported illustrative results, not independently reproduced estimates.

An $R^2$ of 0.28 means the selected regressors explain about 28% of the sample variation under that fitted model. The remaining 72% is not automatically skill. It can include omitted factors, nonlinear exposure, measurement error and noise. A significant intercept is evidence relative to the chosen model and standard-error treatment, not proof that every known risk premium has been excluded.

Use returns of investable factors when attributing return premia. A raw volatility level or a cumulative signal is not automatically a factor return. Such regressors can still help describe behaviour, but the interpretation of the intercept changes.

Sizing rules need their own validation

The source combines a 15% volatility target, a 1.5-times leverage cap and quarter-Kelly sizing, taking the smaller exposure. These are useful research settings. They do not guarantee survival, especially when expected returns and variances are estimated from short windows.

The source’s claim that quarter Kelly retains approximately 85% of full-Kelly growth is not correct under the standard quadratic approximation. If $a$ is the fraction of the ideal full-Kelly allocation, the approximate growth ratio is $2a-a^2$. At $a=0.25$, it is 43.75%. The rationale for fractional Kelly is reduced sensitivity to estimation error and lower risk, not the quoted 85% figure.

The study should show what sizing adds separately from the signal. Compare a fixed-notional version, the risk-scaled version and the combined result under identical execution assumptions. Otherwise a better Sharpe caused by lower exposure can be mistaken for a stronger forecast.

Compare like periods and like capital

Long indicator windows create warm-up intervals. If a strategy begins only after 252 observations but the benchmark is measured from the first observation, their reported returns refer to different periods. Align the comparison or show both periods explicitly. The same problem arises when a model drops difficult rows and the benchmark retains them.

Capital definitions deserve similar care. A fully funded cash strategy, a futures strategy quoted on initial margin and an option strategy quoted on premium use different denominators. Comparing their percentage returns without a common capital policy can reward leverage rather than investment skill. Present currency profit, committed capital, maximum funding need and any target-volatility scaling alongside the performance ratio.

Finally, keep a record of failed experiments. Discarded signals, alternative filters and revised holding periods are part of the research search. A clean final notebook does not make that search disappear. An experiment register helps a reviewer assess whether the final result was a durable economic idea or the survivor of many opportunities to fit noise.

Before allocating capital, reconcile the equity account, inspect several individual trades and confirm that every feature and fitted parameter existed before the relevant decision. Establish model-health, data-health and execution-health limits. A failure in any one can invalidate the strategy even when its last historical Sharpe remains attractive.

Paper trading and limited deployment then reveal fill quality, latency, monitoring requirements and operational failure modes that a historical file cannot capture. Keep a versioned change log. Once a rule is changed after seeing new outcomes, evaluate the revised strategy as a new experiment rather than attaching the old performance record unchanged.

The aim is not to accumulate favourable metrics. It is to produce a strategy whose historical decisions can be reconstructed, whose uncertainty is honestly represented and whose live failures can be detected before they become unexplained losses.