Backtesting Exception Clusters: When the Right Count Hides the Wrong Model

An annual exception count discards the calendar. A model that breaches five times in isolated months and one that breaches five times during a short volatility episode receive the same unconditional-coverage statistic. Their operational implications can be very different. The second pattern may indicate that the forecasting process does not adapt when risk changes.

REFERENCE · NO CALL

This report develops the sequence-based tests that complement exception counting. Its numerical exhibit comes from a September 2026 teaching case with seven exceptions over 250 days. Those observations are illustrative training data, not a performance disclosure about an identified institution. The purpose is to explain what each diagnostic can and cannot establish.

Preserve the ordered binary sequence

Write $I_t=1$ when day $t$ breaches its pre-existing VaR forecast and zero otherwise. The sequence must preserve every observation’s date. Sorting losses by size is useful for quantile estimation, but it destroys the timing information needed here.

A sequence-based test also needs a clear treatment of missing days. Removing an unavailable observation can make two nonconsecutive calendar dates appear consecutive in the test. Retaining a placeholder without a defined rule can create a different distortion. The backtesting calendar, operational outages and any formal exception treatment should therefore remain visible in the evidence file.

Model versions matter too. A dependence test crossing a material model change may detect a change in the forecasting process rather than market clustering under one stable model. That can still be important, but its interpretation differs from the test’s simplest null hypothesis.

The lag-one transition table

Christoffersen’s independence construction compares the chance of a breach after a quiet day with the chance after a breach. Let $n_{ij}$ count transitions from state $i$ on one day to state $j$ on the next. There are $T-1$ observed transitions for $T$ daily indicators.

The teaching example gives:

Observed transitions between quiet days and exceptions — Previous day: Quiet

Observed transitions between quiet days and exceptions — Previous day: Exception

Observed transitions between quiet days and exceptions — Previous day: Total

The estimated conditional exception probabilities are

$$\hat\pi_{01}=\frac{n_{01}}{n_{00}+n_{01}}=\frac7{242}=0.02893,$$

In words: Divide exceptions following quiet days by all observed transitions starting with a quiet day.

$$\hat\pi_{11}=\frac{n_{11}}{n_{10}+n_{11}}=0.$$

The pooled probability under independence is

$$\hat\pi=\frac{n_{01}+n_{11}}{T-1}=\frac7{249}=0.02811.$$

The denominator is 249 for this transition experiment, not 250. Replacing it mechanically with the total number of daily observations makes the reported likelihood inconsistent with the table.

Compare one transition probability with two

Under the restricted model, the next-day breach probability is the same after either state. Under the unrestricted first-order Markov model, it can differ. Their likelihoods are

$$L_0=(1-\hat\pi)^{n_{00}+n_{10}}\hat\pi^{n_{01}+n_{11}},$$

$$L_1=(1-\hat\pi_{01})^{n_{00}}\hat\pi_{01}^{n_{01}} (1-\hat\pi_{11})^{n_{10}}\hat\pi_{11}^{n_{11}}.$$

In words: Multiply the chances of the observed transitions, allowing a different next-day exception chance after each previous state.

Zero counts are handled by the limiting convention that a zero-exponent term contributes one; in log-likelihood form, its contribution is zero. An implementation that evaluates zero times the logarithm of zero directly can return a numerical error even though the likelihood has a well-defined limit.

The independence statistic is $LR_{ind}=2(\log L_1-\log L_0)$. The teaching calculation yields approximately 0.4050, comfortably below the conventional 5% chi-square critical value of 3.841 with one degree of freedom.

The correct conclusion is narrow: the sample does not reject the specified lag-one independence restriction under that calibration. It does not prove that exceptions are independent at every lag, that the volatility model is adequate or that the portfolio’s tail risk is fully represented.

No adjacent breaches does not mean no clustering

The example has no back-to-back exceptions, yet four exceptions occur within a nine-week region. A first-order transition test has little direct sensitivity to this pattern because each exception is followed by a quiet day.

Consider an exception every fifth day during a volatile month. The immediate post-exception transition is always quiet, so the statistic may look reassuring even though the local rate of exceptions has risen sharply. The relevant dependence is spread across several days. A test built only on adjacent pairs cannot manufacture information about every possible longer-lag structure.

This is why a dated exception chart remains useful alongside the formal statistic. Visual inspection is not a substitute for inference, but it can identify the kind of inference needed. Waiting-time diagnostics, multi-lag regressions and regime-conditioned analyses ask different questions from the lag-one table.

Joint coverage can be less decisive than a component test

The conditional-coverage statistic combines the unconditional-coverage and independence components:

$$LR_{cc}=LR_{uc}+LR_{ind}.$$

The teaching case retains $LR_{uc}=5.4970$, producing $LR_{cc}=5.9020$. The usual joint reference distribution has two degrees of freedom, with a 5% critical value of 5.991. Its approximate p-value is 0.0523.

Coverage tests under the stated teaching approximation — Test: Unconditional coverage

Coverage tests under the stated teaching approximation — Test: Lag-one independence

Coverage tests under the stated teaching approximation — Test: Joint conditional coverage

There is no mathematical contradiction. The joint alternative frees an additional parameter and uses a different critical value. Adding a component with little evidence against its restriction can leave the larger test just below its threshold.

Reporting only the joint result would hide the significant frequency evidence. A complete report shows all components, their hypotheses and p-values. It should avoid describing 0.0523 as proof of adequacy merely because it sits on one side of an arbitrary significance threshold.

Waiting times examine a different failure signature

Under independent daily breaches with constant probability $p$, the time between exceptions follows a geometric distribution. If $D$ counts days through the next breach, then

$$\Pr(D=d)=(1-p)^{d-1}p, \qquad E[D]=\frac1p.$$

At $p=0.01$, the expected waiting time is 100 days. That does not mean every gap should be close to 100; the geometric distribution is highly variable. It gives a reference against which the entire observed duration pattern can be assessed.

The retained sequence of waiting times in days

Its specified time-between-failures calculation produces 17.914 against a quoted 14.067 reference value, with p-value 0.01236. This is a retained result of that particular teaching implementation, not a universal critical-value table for every duration test.

Several details require care in a production implementation. The first observed wait may begin before the sample starts, making it left-censored or dependent on a convention. The last incomplete wait contains information too. Parameter estimation, discrete waiting times and small event counts affect reference distributions. A specialist reviewer should reproduce the exact statistic and its null calibration before relying on the headline.

What remains robust is the diagnostic distinction: short nonadjacent gaps can motivate evidence against a constant independent breach process even when the lag-one table does not.

Translate patterns into competing explanations

Clustering is consistent with a volatility model that reacts too slowly, but it is not uniquely caused by one. Repeated stale prices, a series of related positions entering the book or a missing factor that becomes active can create a similar pattern. The mechanism needs further evidence.

Observed exception patterns and targeted follow-up

The analyst should define these follow-ups before adjusting the model. Otherwise a remediation exercise can turn into repeated tuning until one set of p-values becomes favourable.

A practical reporting standard

The deliverable should contain the dated indicator sequence, transition table, likelihood convention, test statistics, sample boundaries, significance choices and any simulation method. It should then explain which alternative mechanisms remain plausible. Graphs and durations should preserve calendar meaning, especially when holidays or missing valuations complicate the sequence.

A model with too many isolated breaches and one with a normal total concentrated in a short episode require different attention. Combining the count, transition and duration views gives a more informative diagnosis. None is a certificate of correctness; together they help identify whether the forecasting process responds to the risks it is supposed to control.