Machine Learning for Finance: Why Accuracy Can Mislead

Supervised learning uses examples with known outcomes to estimate a mapping from inputs to a target. Its usefulness depends on whether that mapping continues to work on genuinely unseen cases. A sophisticated algorithm cannot repair an ill-defined target, leaked future information or labels that measure the wrong outcome.

This reference consolidates regression, nearest neighbours, naive Bayes, support vector machines and decision trees. It focuses on the analytics process and its limitations rather than claims about any named platform or institution.

Define the learning problem before selecting a model

A feature vector $x$ contains the inputs available for one observation. A target $y$ is the outcome to predict. Classification estimates a class or class probability; regression estimates a numerical quantity.

The observation unit matters. A transaction, customer, issuer-month and market day create different datasets. Multiple rows from the same underlying entity can be strongly related. Splitting them randomly may place near-identical information in both training and test samples.

Labels should have a clear event definition and horizon. “Default within twelve months” differs from “currently impaired”; “document concerns a regulation” differs from “document recommends a policy change.” A model can achieve high accuracy against weak labels while failing the business purpose.

A data dictionary should state units, missing-value meaning, timestamp and available history for each feature. A missing balance may mean zero, unavailable data or an inapplicable field; those interpretations should not be merged without thought.

Scaling changes distance and optimisation

Distance-based methods compare feature vectors numerically. If one feature is measured in millions and another between zero and one, raw Euclidean distance can be dominated by the first. Standardisation and domain-specific scaling change the geometry of the problem.

The scaling parameters must be estimated using training data and then applied unchanged to validation and test data. Fitting them on the full dataset leaks information about the test distribution. The same rule applies to imputation, feature selection and dimensionality reduction.

Categorical labels require suitable encoding. Assigning arbitrary integers to unordered categories can create a fictitious numerical distance. Ordered categories may justify an order but still not equal spacing between levels.

Regression provides a transparent baseline

Linear regression estimates

$$\hat y=\beta_0+x^\top\beta,$$

where $\beta_0$ is an intercept and $\beta$ contains feature coefficients. Ordinary least squares minimises squared residuals. A linear model can include nonlinear transformations of features, but increasing flexibility also increases the opportunity to fit noise.

Ridge regression adds a squared-coefficient penalty; lasso adds an absolute-coefficient penalty. Their regularisation strengths control a bias-variance trade-off and should be selected through an appropriate validation process.

A coefficient describes a conditional fitted association, not automatically a causal effect. Correlated features can make individual coefficients unstable even when the combined prediction is relatively stable. Extrapolating far outside observed feature ranges can be especially unreliable.

Nearest neighbours use local similarity

For a new observation, $k$-nearest-neighbours finds the $k$ closest training observations. A classifier can use a majority vote; a regressor can average their outcomes. Distance weighting gives closer observations more influence.

Small $k$ produces a local, flexible fit with high sensitivity to noise. Larger $k$ smooths predictions but can obscure local structure. The distance metric and feature scaling are therefore as important as the choice of $k$.

The method has little explicit parameter fitting, but it still learns through the stored training examples and chosen representation. It is inaccurate to say that it contains no learning in any meaningful sense. Prediction can also become expensive because it requires finding neighbours among many stored observations.

In high dimensions, observations become sparse and distance distinctions can weaken. The retained conceptual example uses binary features: $M$ such features permit $2^M$ possible patterns. At 20 features there are more than one million patterns, so a few thousand observations cannot densely cover the space.

Naive Bayes separates a probability rule from an approximation

Bayes’ rule gives

$$P(c\mid x)\propto P(c)P(x\mid c),$$

where $c$ is a class. Naive Bayes approximates the likelihood through conditional independence of features given the class:

$$P(x\mid c)\approx\prod_jP(x_j\mid c).$$

The independence assumption is often false, particularly for words in text, yet the resulting classifier can remain useful. Its probability outputs may need calibration even when its class ranking works reasonably well.

Text preprocessing defines the features. Tokenisation, case handling, stemming, stop-word treatment and count-versus-presence representation can all change results. Smoothing prevents unseen words from assigning a whole document zero likelihood, while logarithms avoid numerical underflow from multiplying many small probabilities.

One historical teaching example classified a speech into a political category inconsistent with its author’s known position. The useful lesson is methodological: the classifier may have learned rhetorical style or vocabulary rather than the intended ideology. Personal identities and the original speech text are omitted here; the failure mode remains relevant to any document classifier.

Support vector machines control the separating margin

A linear classifier uses the sign of $w^\top x+b$, where $w$ defines the separating direction and $b$ the intercept. A hard-margin support vector machine seeks a separating hyperplane with maximum margin when the classes are separable.

A soft-margin formulation allows violations and trades them against margin size. In one common representation,

$$\min_{w,b}\ \frac12\|w\|^2 +C\sum_i\max\{0,1-y_i(w^\top x_i+b)\},$$

In words: Choose the separating boundary by balancing a wider gap between classes against penalties for cases inside the gap or on the wrong side.

where labels $y_i$ are minus one or plus one and $C$ controls the penalty on violations. The hinge loss penalises observations inside the margin or on the wrong side.

Kernel methods represent nonlinear separation through inner products in a transformed feature space. Greater flexibility can improve fit but adds hyperparameters and overfitting risk. A classifier score is not automatically a calibrated class probability.

Decision trees make local rules explicit

A decision tree repeatedly splits data using feature thresholds or category rules. Classification splits can reduce entropy or Gini impurity; regression splits can reduce within-node squared error.

For class proportions $p_j$, entropy is

$$H=-\sum_j p_j\log_2p_j.$$

A balanced binary node has entropy one bit; a pure node has entropy zero. Information gain compares parent entropy with the sample-weighted entropy of child nodes.

Deep trees can memorise training observations. Maximum depth, minimum leaf size and pruning constrain this flexibility. Small data changes can alter early splits and produce a different-looking tree, so apparent interpretability does not guarantee stability.

Trees can handle nonlinear interactions and mixed features with limited scaling requirements. Their local thresholds can also generate abrupt prediction changes near a split. Whether that behaviour is acceptable depends on the downstream decision.

Classification quality needs more than accuracy

For true positives TP, false positives FP and false negatives FN,

$$\operatorname{Precision}=\frac{TP}{TP+FP}, \qquad \operatorname{Recall}=\frac{TP}{TP+FN}.$$

In words: Precision is the share of flags that are correct, while recall is the share of actual positive cases found.

Precision measures how many flagged cases are correct; recall measures how many actual positives are found. A model can obtain high accuracy by predicting the majority class when the important event is rare.

For an illustrative 1,000-case dataset with ten actual positives, predicting every case negative gives 99% accuracy and zero recall. This arithmetic explains why an operational screening process must choose thresholds using error costs and workload, not accuracy alone.

Probability calibration asks whether cases assigned a given probability occur at approximately that frequency. Ranking performance and calibration are different properties. A model may rank cases well while systematically overstating probabilities.

Validation should reproduce the intended use

Use separate training, validation and final test roles. Hyperparameters are selected using validation data; the final test should be protected from repeated tuning. Time-dependent applications need chronological splits, while repeated entities may need group-based separation.

Assess performance across relevant subgroups, periods and data-quality conditions. A model that fails on sparse or newly appearing categories may need a fallback decision process. Monitoring after deployment should track input drift, missingness, prediction distributions and realised outcome quality.

Supervised methods and the limitations to inspect

The strongest supervised-learning workflow begins with a precise target and a defensible dataset, then compares methods under the same honest evaluation. Model complexity earns its place only when it improves the decision on genuinely unseen cases.

Base rates change the meaning of a positive flag

Consider an illustrative screening system with 1,000 cases, ten true events, 80% sensitivity and a 5% false-positive rate among nonevents. It finds eight true positives and generates about 49.5 false positives in expectation. Only about 13.9% of positive flags are true events, despite the seemingly strong sensitivity.

The result follows from the low base rate. It is not a contradiction between model metrics. Raising the decision threshold may improve precision while reducing recall; the preferred threshold depends on the cost of missed events, review capacity and the consequences of false alarms.

If the event base rate changes after deployment, precision can change even when conditional sensitivity and specificity remain stable. Monitoring only the model’s score distribution can miss that shift until outcomes mature.

Probability calibration changes downstream decisions

Suppose two groups are each assigned predicted probability 20%, but the event occurs in 5% of one group and 35% of the other. Overall calibration can appear acceptable if the groups are similarly sized, while each group is poorly calibrated.

Calibration should therefore be examined where the decision is actually made. A portfolio expected-loss calculation, a review queue and a binary acceptance rule use probabilities differently. A model that is adequate for ranking may require recalibration before its scores are used as literal probabilities.

Calibration methods themselves need held-out data. Fitting a flexible calibration curve on the same observations used to judge it creates another opportunity for overfitting. Sparse extreme-score regions also need caution because few outcomes support the estimated probabilities.

Feature importance is not a causal explanation

A tree’s split frequency or a model’s performance deterioration after a feature is perturbed can indicate predictive reliance. It does not prove that changing that feature would change the outcome. Correlated features can share or obscure importance, and a proxy can carry information about a variable absent from the dataset.

For document classification, repeated organisation names or template phrases can become shortcuts. A model may learn which source produced a document rather than what the document argues. Removing such identifiers and testing on unseen sources is a useful way to assess whether the classifier understands the intended distinction.

This problem is particularly relevant when historical reports are reorganised for publication. Labels should follow the report’s conclusion and reader purpose, while titles and author or client identifiers should not become accidental substitutes for substantive classification.

Missing-data handling should survive deployment

An imputation rule fitted on training data should be reproducible when a new case arrives. Using the eventual outcome or future observations to fill a missing input can make historical performance unattainable in practice.

Missingness indicators can help when absence carries information, but they can also encode a temporary operational process that later changes. A model relying heavily on such indicators should be tested under plausible changes in data collection.

The final operating design should define what happens when required features are unavailable, a category is unseen or a case lies far outside the training range. A controlled fallback is part of the analytics system. Returning an unqualified prediction for every input is not evidence that the model is reliable for every input.