VaR Backtesting: What an Exception Count Can and Cannot Prove

Seven exceptions in 250 days can mean several different things. It can reject a statistical coverage hypothesis, select a capital multiplier in a historical regulatory schedule and still leave a trading desk below a separate model-eligibility threshold. These are different decisions with different scopes. A validation report should preserve the chain from the daily observations to each decision rather than compressing everything into a single statement that the model passed or failed.

REFERENCE · NO CALL

The numerical case in this report is an illustrative equity-and-rates desk from September 2026 teaching material. It is not a disclosure about a real bank. Regulatory schedules are described in the historical framework used by the material; they are not a statement of the rules currently effective in any jurisdiction.

Build the exception file before selecting a test

Let $v_t$ be the one-day VaR forecast available before day $t$, and $L_t$ the corresponding realised loss. Define the exception indicator

$$I_t=\mathbf{1}\{L_t>v_t\}.$$

The indicator equals one for a breach and zero otherwise. The strict inequality, the P&L definition and the dating convention should be explicit. If the system stores P&L with losses negative, the equivalent comparison is against the negative VaR threshold.

Each row needs its valuation date, forecast-production time, portfolio snapshot, model version, loss number, VaR number, exception flag and investigation status. The loss must not include an unacknowledged change in portfolio after the forecast. Actual and hypothetical P&L are separate data products, even when they happen to produce identical counts.

In the teaching case, the 99% backtest records seven exceptions against both P&L definitions over 250 days. The 97.5% backtest records 17 against each. Under the promised coverage rates, the expected counts are 2.5 and 6.25 respectively. The second confidence level provides more tail observations and can reveal weaknesses that a very sparse 99% test struggles to detect.

Missing observations should enter a visible control workflow. A failed valuation batch must not disappear through deletion of its row. The treatment under the applicable rule must be recorded separately from the statistical analysis of otherwise complete data.

The Kupiec test measures frequency

Let $T$ be the number of observations, $N=\sum I_t$ the exception count and $p$ the promised exception probability. The null sets $p=0.01$ for 99% VaR. The unrestricted best-fitting rate is $\hat p=N/T$.

The log-likelihoods are

$$\ell_0=(T-N)\log(1-p)+N\log p,$$

$$\ell_1=(T-N)\log(1-\hat p)+N\log\hat p.$$

The unconditional-coverage likelihood-ratio statistic is

$$LR_{uc}=2(\ell_1-\ell_0).$$

The binomial coefficient cancels if one starts with the probability of the total count instead of the probability of a particular sequence. The test deliberately ignores the sequence order and breach magnitude.

For $T=250$, $N=7$ and $p=0.01$, the retained calculation gives:

Coverage-test outputs for the retained exception case

The null is rejected at the 5% significance level and not rejected at the 1% level. Those statements are consistent. The second test requires stronger evidence. Neither statement identifies the economic cause of the miscalibration.

Zero exceptions expose the importance of test direction

A formal two-sided coverage test can reject a model for producing too few exceptions as well as too many. At zero exceptions, the limiting likelihood calculation is

$$LR_{uc}=-2T\log(1-p).$$

With 250 observations and a 1% tail probability, the result is approximately 5.025, above 3.841. Thus zero exceptions rejects the formal two-sided test at 5%. An early teaching summary described zero through six as passing while focusing on excessive losses; its later detailed treatment corrects that distinction.

The practical lesson is to state whether the exercise tests calibration in both directions or focuses only on underprediction of loss frequency. A traffic-light table can still place zero exceptions in a favourable regulatory zone. Statistical calibration and regulatory classification answer different questions.

Boundary samples also weaken simple asymptotic approximations. Exact binomial calculations or simulation can help assess small-sample behaviour. They should be specified before inspecting the outcome, rather than selected afterwards to produce a more comfortable verdict.

A non-rejection has limited power

At a genuine 2% exception probability, a 250-day sample averages five exceptions. A model whose tail probability is twice its advertised level can therefore produce an apparently ordinary count. The historical teaching calculation estimates that such a model lands in the favourable zero-to-four zone about 43.9% of the time.

That figure concerns a particular count-based zone, not the power of every possible backtest. Its value is conceptual: sparse tail data cannot support very strong assurance. More observations help only when they remain comparable. Pooling years across model changes or structurally different market regimes can add apparent sample size while changing the hypothesis being tested.

A complete outcomes report should show the count, the forecast confidence level, uncertainty, breach sizes, timing and the relevant model versions. Four exceptions do not establish that missing volatility skew or basis risk has been represented correctly. A generous volatility calibration can temporarily conceal those defects.

Translate counts into capital without mixing regimes

The source material compares a legacy multiplier schedule with an FRTB schedule. Seven exceptions select 3.65 in the former and 1.83 in the latter. These numbers multiply different capital constructions and cannot be exchanged merely because they come from the same count.

For the legacy arithmetic illustration, the displayed 60-day average VaR is ₹7.82 crore. Holding the relevant capital leg fixed gives:

Legacy capital arithmetic using the displayed inputs

The original headlines were ₹23.47 crore and ₹28.55 crore, consistent with rounding from a more precise base. The increase is about 21.7% for this capital leg. The difference should be disclosed as rounding rather than silently changing the source record.

This is not a complete legacy market-risk capital reconstruction. Additional components, applicable horizons and the comparison with the latest risk measure depend on the actual framework. It is also not valid to multiply ₹7.82 crore by the FRTB coefficient and call the result an FRTB charge: the latter framework is built around different inputs, including Expected Shortfall.

Keep bank-wide and desk-level decisions separate

A bank-wide multiplier and a desk’s continued eligibility to use an internal model are distinct controls. The historical teaching framework places the illustrative desk’s seven 99% exceptions and 17 97.5% exceptions below its stated separate desk triggers. A favourable answer to that eligibility question does not remove the multiplier consequence or cure a coverage rejection.

Similarly, P&L attribution is another independent gate. A desk may reproduce the overall scale of losses while failing to represent their drivers. That is why a validation conclusion should identify precisely which requirement is satisfied and which is not.

The same discipline applies to any allowed exception relief. A narrative that a non-modellable factor caused a loss is not enough. The historical framework requires causal evidence, a quantitative capital comparison and supervisory conditions. An analyst should preserve the original breach and any separately authorised treatment, creating an audit trail rather than rewriting history.

Investigate the mechanism behind the count

Seven breaches can arise from slow volatility adjustment, omitted factors, inaccurate repricing, poor data or genuinely rare market movements. Those explanations imply different fixes. Higher volatility may reduce breaches but leave a missing-factor problem untouched. A data repair may correct a contaminated observation without improving the economic model.

A useful investigation starts with controlled reproduction of the daily result, decomposes the loss into risk factors, tests the proposed cause and records an owner for the corrective action. The hypothesis should predict something beyond the already observed breach: for example, missing skew should generate residual P&L on skew-steepening days, while slow volatility adjustment should produce short waiting times between exceptions after a regime shift.

The final report should therefore have several sentences, not one badge. State the observed count and sample; report the coverage statistic and its uncertainty; identify the applicable regulatory consequence; describe the economic diagnosis; and specify what evidence would close the finding. That sequence turns backtesting from a colour on a dashboard into a usable control.