Statistical inference

Statistical inference is the process of analysing data to draw conclusions about the properties of an underlying probability distribution. It enables researchers to use information from a sample to make generalisations about a wider population. This approach contrasts with descriptive statistics, which focuses exclusively on summarising observed data without making assumptions about a broader population. In modern machine learning the term inference is often used differently to refer to the act of generating predictions from a trained model, whereas training or learning refers to estimating model parameters—highlighting the distinction between statistical and computational terminology.

Introduction

In traditional statistics, inference relies on the idea that data obtained from a sample represent a larger population. The inferential process begins with selecting an appropriate statistical model that describes how the data were generated and proceeds by deducing propositions about population parameters. Model selection is central to this process: the translation of a real-world question into a statistical model is often the most crucial step in an analysis.
Statistical propositions resulting from inference take several forms, including:

• Point estimates, which provide a single best estimate of a parameter.
• Interval estimates, such as confidence intervals or set estimates, which provide a range of plausible values.
• Credible intervals, which quantify posterior belief in Bayesian frameworks.
• Hypothesis tests, which assess the plausibility of specific claims about population characteristics.
• Classification statements, which assign observations to groups based on statistical rules.

Models and Assumptions

Statistical inference depends critically on assumptions about how data are generated. A statistical model embodies these assumptions, describing the probability structure underlying the observations and the quantities of interest. Descriptive statistics can serve as preliminary tools, but formal inference requires precise modelling choices.

Degrees of Modelling Assumptions

Statistical models vary in complexity and strength of assumptions:

• Parametric models fully specify the distributional form of the data using a finite number of parameters. For instance, assuming normality with unknown mean and variance, or adopting a generalised linear model.
• Nonparametric models make minimal assumptions about the form of the distribution. Estimators such as the sample median or Hodges–Lehmann estimator are typical tools.
• Semiparametric models combine parametric structure for some components and nonparametric forms for others. The Cox proportional hazards model is a prominent example, using a parametric structure for covariate effects but leaving the baseline hazard unspecified.

Importance of Valid Assumptions

Inference is reliable only when a model’s assumptions reflect the true data-generating process. Violations of assumptions—such as incorrect treatment of sampling design, misspecified functional forms or unjustified distributional claims—can invalidate results. For example, incorrect application of the Cox model may lead to misleading survival analyses, and assumptions of normality may undermine regression-based inference for heavily skewed or heavy-tailed populations.
Sampling statisticians are often cautious about parametric assumptions when working with human populations, preferring large-sample methods justified by the central limit theorem. Even then, such justifications require careful consideration, especially when real-world populations exhibit complex distributions.

Approximate Distributions

Exact distributions of sample statistics are often difficult to obtain, prompting the use of approximation methods.

With Finite Samples

Approximation theory quantifies how closely a limiting distribution approximates the true distribution of a statistic. Results such as the Berry–Esseen theorem provide bounds on the accuracy of normal approximations for sample means. In practice, simulation studies and experience show that normal approximations are often reasonable with as few as ten independent observations for many types of data.
Modern approaches employ functional analysis and metrics such as Kullback–Leibler divergence, Bregman divergence and Hellinger distance to quantify errors and compare distributions. These mathematical tools formalise the accuracy of approximations and help assess their suitability.

With Indefinitely Large Samples

Asymptotic theory studies the behaviour of estimators and test statistics as sample size tends to infinity. Key results include the central limit theorem and laws of large numbers. However, asymptotic conclusions do not directly describe the behaviour of finite samples. Limit theorems serve as conceptual guides, but their practical relevance depends on how well large-sample approximations capture finite-sample properties.
Scholars have cautioned that asymptotic statements may be logically irrelevant to any specific finite sample and that the applicability of limit results must be validated through context-specific assessment. Approximations derived from asymptotic theory, such as those underlying generalised estimating equations or the generalised method of moments, require evaluation through simulation or empirical diagnostics to ensure accuracy.

Inference in Practice

Because practical inference relies on both modelling and approximation, an awareness of model quality, sampling design and distributional behaviour is essential. Modern statistical tools blend parametric, nonparametric and semiparametric ideas, often guided by computational simulations that assess estimator performance under realistic conditions. The balance between theoretical justification and empirical validation is central to contemporary statistical practice.