Hypothesis Testing: When a Convincing Business Story Is Just Noise

A hypothesis test asks whether an observed result is difficult to reconcile with a specified statistical model. It does not remove judgement from a decision, prove a business mechanism or estimate the probability that an attractive explanation is true. Its value comes from making the question, evidence threshold and consequences of error explicit before the result is known.

Reference report · No directional market call

This synthesis joins the introductory presentation, complete pre-read, hypothesis-building examples and practice workbooks. Numerical cases are teaching inputs. They should not be read as observed client results or current thresholds mandated for any industry.

Begin with a parameter and a decision

“Did the redesign work?” leaves too much unspecified. Define the outcome, population, comparison period and effect worth detecting. For a checkout redesign, the parameter might be the probability that an eligible session ends in abandonment. For a manufacturing change it could be mean time per completed unit, with product mix and quality standards held comparable.

The null hypothesis describes the benchmark being tested; the alternative identifies the relevant departure. If $p$ denotes the true abandonment probability and the baseline is 0.72, a pre-specified reduction question can be written as:

$$H_0:p\geq0.72,\qquad H_1:p<0.72.$$

The boundary $p=0.72$ is commonly used to calculate the one-sided test. Writing only equality is a classroom shorthand, but the composite null makes the decision logic explicit. A two-sided question would instead compare equality with any difference. Choose direction before inspecting the results; changing to a one-sided test after seeing a favourable movement changes the error rate.

Statistical error has a business consequence

A Type I error rejects a true null. A Type II error fails to reject a false null. The significance level $\alpha$ controls the first error under the specified model and procedure. Power, equal to $1-\beta$, is the probability of detecting a specified alternative when it is true; $\beta$ denotes the Type II error probability for that alternative.

False-positive and false-negative costs depend on the decision

The source examples use significance levels from 0.10 to 0.001 to discuss different error costs. These are pedagogical choices, not verified regulatory standards. No universal threshold converts a low-stakes product experiment into the equivalent of a safety-critical decision.

Sample design should specify a minimum relevant effect and desired power. With too little data, a non-significant result may simply be inconclusive. With enormous data, a negligible difference can be statistically significant. Neither circumstance is resolved by quoting the p-value alone.

Choose the test from the design

The outcome type and sampling design matter more than familiarity with a formula. Comparing a mean with a benchmark differs from comparing two independent groups. Measuring the same worker before and after a change produces paired observations. Comparing proportions requires success and failure counts. Comparing several means raises a joint-testing question.

Choose the statistical method from the question and sampling design

Thirty observations are not a universal switch from t to z. If the population standard deviation is unknown, estimating it still matters with a sample larger than thirty. A large-sample approximation may be reasonable, but its justification comes from the estimator and data, not the label “large sample.”

Accident counts require particular care. Accidents per worker-year are rates, and a worker may have more than one accident. Treating that rate automatically as a Bernoulli probability discards the event-count structure and exposure time. The appropriate model can differ from a test of the proportion of workers with at least one accident.

A paired operational example

The teaching material describes 30 workers measured before and after an assembly-layout change. Mean time falls from 22.5 to 20.1 minutes and the standard deviation of paired differences is 4.2 minutes. Define each difference as before minus after, so a positive difference indicates time saved.

The observed mean difference is 2.4 minutes. Its estimated standard error is $4.2/\sqrt{30}$, approximately 0.767 minutes. The paired t-statistic is therefore approximately 3.13, with 29 degrees of freedom under the usual assumptions.

The statistical evidence concerns a change in mean paired time. A causal interpretation still needs the design. Workers may have learned through repetition, changed product mix or used new equipment during the same period. Randomising rollout order or using an appropriate comparison group can strengthen identification. Also examine defect rates: faster assembly that raises rework may not improve the process.

A proportion example and its boundary

Another teaching case starts from abandonment of 72% and observes 66.5% in 800 sessions. Under a fixed benchmark and independent Bernoulli-session approximation, the null standard error is:

$$SE_0=\sqrt{\frac{0.72(1-0.72)}{800}}\approx0.01587.$$

The corresponding standardised difference is approximately $(0.665-0.72)/0.01587=-3.46$. That is evidence against the specified benchmark under the model. It does not itself show that the redesign caused the difference. A historical benchmark also has estimation error unless it is treated as a fixed target rather than an estimate.

Repeated sessions from the same customer, marketing campaigns that alter visitor composition and a changed definition of abandonment can undermine the independent-session interpretation. An experiment should randomise at a unit consistent with the intended inference and avoid contamination when customers can experience both versions.

Interpret the p-value precisely

A p-value is the probability, under the null model and test procedure, of a result at least as extreme as the one observed. It is not the probability that the null is true. It is not the probability that the result “happened by chance” across every possible explanation. A small p-value can arise from a real effect, a violated assumption, selection, leakage or an inappropriate model.

The decision language should reflect the actual evidence. “Fail to reject” is different from “prove there is no effect.” If the objective is to establish practical equivalence, an equivalence or non-inferiority design with an explicit margin may be appropriate. An ordinary test of zero difference does not establish equivalence merely because its p-value is large.

Report effect size, uncertainty and economic relevance alongside the test. A five-percentage-point retention improvement has different value at different margins, acquisition costs and service costs. Statistical significance is one input to the commercial decision.

Multiple testing and repeated peeking

At a 5% test level, 100 independent tests of true nulls produce five false rejections on average. The realised number can differ. If only the most attractive results are reported, readers cannot assess the evidence without knowing how many opportunities there were to find them.

Define the family of questions, distinguish exploratory screens from confirmatory tests, and use an adjustment or design appropriate to the purpose. A false-discovery-rate approach and familywise-error control solve different problems. Neither repairs poor data or a question chosen after observing the outcome.

Repeatedly checking an experiment and stopping as soon as an ordinary p-value falls below 0.05 also alters its interpretation. Pre-specify a fixed horizon or use a sequential procedure designed for interim looks. Retain the record of analyses performed; otherwise an apparently single test may conceal many chances to declare success.

Make the experiment operationally reproducible

An experiment record should specify eligibility, randomisation unit, allocation rule, outcome window, exclusions, handling of missing values and planned subgroup analyses. Check that sample allocation and logging worked. A clean statistical formula cannot repair a broken experiment assignment.

Use guardrail metrics where the primary objective can be improved at another stakeholder’s expense. A service intervention might improve speed while worsening quality. A pricing test might increase revenue while increasing returns or reducing retention. Specify what constitutes an unacceptable deterioration before examining the outcome.

Data exclusions need documented reasons. Removing unusually large orders because they change the conclusion is different from excluding a corrupted record under a pre-existing rule. Where an unusual observation is valid, show sensitivity with and without it and explain why the preferred specification answers the question.

Compare more than two alternatives without losing the question

The hypothesis-building material includes three operational teams with mean resolution times of

Listed operational-team mean resolution times

minutes.

These means alone are insufficient to establish a meaningful difference: sample sizes, within-team variation and comparable case mix are missing. A joint test can ask whether the population means are all equal, but rejection does not identify every pair as different.

If the business needs to choose between particular teams or operating methods, specify those contrasts and account for the comparison structure. A series of unadjusted pairwise tests can inflate the chance of a false discovery. Follow-up analysis should also examine whether differences arise from work allocation rather than the operating method itself.

The same caution applies to tests of independence in categorical data. A statistically significant table can contain an effect too small to matter commercially or reflect different customer mixes. Show the actual rates and denominators, not just the test statistic. Counts that are too small for the approximation require a more suitable method or a redesigned evidence-gathering exercise.

From a test result to a management statement

A strong result statement identifies the population and comparison, quantifies the observed effect, names the method and uncertainty, and limits the causal claim to what the design supports. It then states the proposed action and the conditions under which the decision should be revisited.

For the assembly example, the complete judgement is not simply “the layout works.” It is that the measured paired sample shows a 2.4-minute average reduction under stated assumptions; concurrent changes and quality effects require review; a controlled rollout can test whether the saving persists. This preserves both the numerical finding and the uncertainty around its business meaning.

Hypothesis testing is most useful when it disciplines a decision rather than decorates a preferred conclusion. The essential work happens before the calculation: define the parameter, choose the comparison, price the errors and specify what evidence would lead to action.