Validation creates value when a finding becomes a controlled change and the change survives daily production. A favourable test result after an undocumented recalibration is weak evidence. A reproducible change that removes a specific defect, improves the intended measure and leaves an auditable history is much stronger—even if it raises reported risk.
REFERENCE · NO CALL
This report consolidates remediation, supplementary testing and production-control material from a September 2026 teaching series. Its before-and-after figures are illustrative source results. They do not describe a named bank, demonstrate a live regulatory approval or establish that the revised model would perform the same way out of sample.
Turn a finding into a falsifiable change
A useful finding names the defect, the affected use and the evidence supporting it. Missing basket beta, missing volatility skew and slow volatility adjustment are different findings. They should not be grouped under a general instruction to improve the model because each predicts a different observable failure.
Findings, targeted changes and expected evidence
| Finding | Targeted change | Evidence expected after the change |
|---|---|---|
| Cash basket mapped one-for-one to an index | Correct the systematic exposure mapping | Residual scale mismatch falls on index-driven days |
| Basket-specific basis absent | Add a supported residual factor or treatment | Composition-related P&L is better represented |
| Volatility skew absent | Introduce the necessary surface dimension | Skew-driven residuals fall |
| Large-shock option approximation error | Use a validated revaluation method | Benchmark error decreases across the shock ladder |
| Slow volatility response | Reassess conditional-variance dynamics | Fewer clustered forecast failures, without look-ahead |
| Snapshot mismatch | Repair data alignment under controlled rules | Operational component of the residual is explained |
This matrix prevents a common failure: changing several parts of the process and attributing every improvement to the most appealing explanation. A staged experiment can isolate effects; a bundled release still needs component-level evidence and regression checks.
Preserve the baseline
Before changing the model, archive the code version, configuration, market snapshots, positions, data transformations and output tables needed to reproduce the original finding. The baseline is the reference against which the change is assessed.
Re-running the historical window with a new model is useful, but it is a backcast. It should not replace the original forecasts in the live backtesting record. A model that would have predicted a past loss correctly was not necessarily the model operating before that loss occurred.
This distinction is central to honest remediation. The institution should retain both what was known and produced at the time and what a later reconstruction shows. An audit trail that preserves only the improved history can make a failure appear never to have existed.
Figure 01 · Conceptual framework
From a validation finding to controlled closure
Read the before-and-after evidence narrowly
The source case reports seven original 99% exceptions and three after remediation. Its KS distance falls from 0.096 to 0.024. The revised model clears the teaching example’s specified metrics, while recognising more of the portfolio’s risk.
Retained model-remediation results and interpretation — Metric: 99% exceptions in the example window
| Measure | Value |
|---|---|
| Original model | 7 |
| Revised model | 3 |
| Interpretation | Historical coverage improves |
Retained model-remediation results and interpretation — Metric: KS distance for P&L attribution
| Measure | Value |
|---|---|
| Original model | 0.096 |
| Revised model | 0.024 |
| Interpretation | Distributional mismatch falls |
Retained model-remediation results and interpretation — Metric: Simplified source capital proxy
| Measure | Value |
|---|---|
| Original model | ₹14.32 cr |
| Revised model | ₹17.28 cr |
| Interpretation | Better recognition raises the measured base |
The source quotes the capital-proxy increase as ₹2.97 crore; subtraction of the rounded displayed figures gives ₹2.96 crore. This is a rounding difference, not a new economic result.
More significantly, the proxy applies a simplified multiplier to an average VaR measure. It is not a complete FRTB capital calculation and should not be compared directly with the source’s separate ₹53.72 crore capital-stack illustration. Retaining the two examples without explaining their different bases would create a false before-and-after claim.
The robust message is that a lower penalty coefficient can coexist with a higher recognised risk base. Passing more tests does not guarantee lower capital, nor should capital reduction be the acceptance criterion for a model fix.
Independent approval is a separate event
A developer’s successful rerun is evidence for a release, not the release decision itself. Independent validation should reproduce material outputs, challenge the assumptions, examine the change’s intended scope and assess new limitations. Governance then records whether the model is fit for its intended use, fit with conditions or unsuitable.
Conditions should be specific: an exposure cap, a valuation overlay, a restricted product set, additional monitoring or a dated remediation action. A condition with no owner or measurable closure evidence is unlikely to remain effective.
Any regulatory recovery rule also follows its own calendar. A technically fixed model does not erase the historical observations used by a rolling assessment. The date of implementation, the date of independent approval and the date of formal status recovery should therefore be recorded separately.
Figure 02 · Qualitative comparison
Three events that must not be conflated
| Approach or concept | What it contributes | What still matters |
|---|---|---|
| Code change | Implements a proposed technical remedy | Does not establish the remedy is correct |
| Independent validation | Assesses evidence and residual limitations | Does not prove production is using the approved version |
| Production control | Monitors actual daily behaviour | Requires ownership, release records and escalation |
Daily production is part of the model
A forecasting method depends on a repeatable operating process. Missing market data, stale trades or inconsistent closing snapshots can change the output more than a sophisticated model enhancement. A production failure is therefore relevant model-use evidence, even if the mathematical specification is sound.
The daily process should establish position completeness, market-data validity, successful valuation, generation of P&L variants, exception detection and reconciliation. A failure should identify the affected outputs and the permitted fallback. An unexplained carry-forward number should never look identical to a newly calculated one.
Batch completion alone is not proof of correctness. A system can finish successfully using stale inputs. Controls should compare data age, record counts, material position movements and distributional changes against expected bounds. Threshold breaches should generate an investigation rather than an automatic suppression of the output.
Build a calendar with control purposes
Illustrative control calendar and retained evidence
| Frequency | Control purpose | Evidence retained |
|---|---|---|
| Daily | Produce controlled risk and P&L measures | Inputs, version, results, reconciliations and failures |
| Rolling monitoring | Detect deteriorating attribution or coverage early | Metrics, trends and escalation records |
| Monthly | Track observation eligibility and evidence expiry | Factor counts, continuity and source-quality issues |
| Quarterly under the historical framework | Refresh formal classifications and stressed inputs | Dated assessments and consistent reference periods |
| Annual | Reassess conceptual soundness and outcomes | Independent replication, benchmarks and findings |
| Event driven | Reassess material change or new use | Change impact, approval and compensating controls |
The calendar is an example of an operating design, not a claim that every frequency is universally prescribed. The institution must map controls to its actual model uses and governing requirements.
Figure 03 · Conceptual framework
Can the remediation finding be closed?
How the parts connect
- Remediation evidence packPath AApproved fix operates as intended
- Remediation evidence packPath BApproval or production evidence missing
- Approved fix operates as intendedReviewSustained control
- Approval or production evidence missingReviewSustained control
Investigate every exception with the same vocabulary
A standard cause taxonomy makes patterns visible across events. Useful categories include genuine tail movement, missing factor, volatility-regime response, valuation approximation, data issue and product outside approved scope. Several causes can coexist, so the classification should not force a complex event into a single convenient explanation.
The investigation begins by reproducing the result. It then decomposes P&L, evaluates candidate causes and identifies what would have changed under the proposed correction. A stale-data explanation needs evidence of the stale input and the corrected output. A missing-factor explanation needs a causal relation to the unexplained movement, not merely the observation that the factor also moved that day.
Any approved adjustment or exclusion should be recorded alongside the original event. The record should identify the authority, conditions and quantitative effect. Editing the exception flag without retaining the original state breaks the evidence chain.
Supplement minimum metrics with decision-relevant tests
The teaching series adds duration diagnostics, tail-severity backtests and uncertainty analysis to the count-based framework. Each contributes evidence about a different weakness. They should be selected because they address a concrete limitation, not because more tests automatically make a validation stronger.
In one source example, realised losses on tail-breach days average ₹8.50 crore against average predicted ES of ₹7.56 crore, a gap of ₹0.94 crore, or about 12.4% of the forecast. The associated specialised statistic uses simulated critical values under the model’s predictive distribution. That null specification is part of the test.
The material reports a Monte Carlo p-value displayed as 0.0000 using 6,000 simulations. This should be described as no simulated tail result as extreme at the reported resolution, not a literal probability of zero. A common finite-simulation correction would bound the reported value away from zero; the exact method should be stated.
Bootstrap uncertainty is similarly conditional on the resampling design. It can show fragility near a threshold but does not automatically forecast next quarter’s zone. A strong reviewer preserves these qualifications rather than turning a technical diagnostic into a stronger claim.
Manage the whole model inventory
Spreadsheet calculations and local scripts can become production models through use, even if nobody formally designated them as such. The inventory should follow economic function: what decision depends on the output, how material it is and what happens if it fails.
The inventory entry should connect owner, approved purpose, inputs, implementation, limitations, controls and validation status. A duplicate tool using a different market-data convention can undermine a controlled central model if it becomes the de facto decision source. Reconciliation and retirement plans are therefore part of remediation.
Closure requires evidence of use
A finding is ready to close when the specified change is implemented, independently assessed, operating under its approved controls and supported by the promised evidence. Where outcomes need time to accumulate, the closure decision should say so and retain monitoring conditions.
The purpose of governance is to preserve the connection between model claims and actual decisions. Better statistics help. Reproducibility, honest history, operational discipline and clearly bounded approvals are what make those statistics dependable in daily use.
