Bayes' Theorem: Updating Beliefs with Evidence

Your ability to do good science depends on your ability to update your beliefs in the face of evidence.

— E. T. Jaynes [1]

The Problem of Inverse Probability

Suppose you test positive for a rare disease. The test is 99% accurate. Should you panic?The intuitive answer is “yes” — but as we’ll see, the correct answer depends critically on how rare the disease is.

This is a question of inverse probability: given an observed effect (positive test), what is the probability of the underlying cause (having the disease)? For centuries, this kind of reasoning was considered impossible or meaningless. It took a Presbyterian minister named Thomas Bayes to crack the problem.

Bayes’ Theorem

Bayes’ theorem relates the conditional probability of a hypothesis $𝐻$ given evidence $𝐸$ to the reverse:

where:

• $𝑃\left(𝐻|𝐸\right)$ is the posterior — our updated belief after seeing evidence
• $𝑃\left(𝐸|𝐻\right)$ is the likelihood — how probable the evidence is if the hypothesis is true
• $𝑃\left(𝐻\right)$ is the prior — our belief before seeing evidence
• $𝑃\left(𝐸\right)$ is the marginal likelihood — the total probability of the evidence

A Simple Example

Consider a medical test for a disease that affects 1 in 1,000 peopleThis base rate is crucial. Many real-world errors in reasoning come from ignoring it — a phenomenon called base rate neglect.:

• Prevalence: $𝑃\left(\text{disease}\right)=0.001$
• Sensitivity: $𝑃\left(\text{positive}|\text{disease}\right)=0.99$ (true positive rate)
• Specificity: $𝑃\left(\text{negative}|\text{no disease}\right)=0.95$ (true negative rate)

What is $𝑃\left(\text{disease}|\text{positive}\right)$?

First, compute the marginal probability of testing positive:

Then apply Equation 1:

The result is shocking: despite the test being 99% accurate, a positive result only means a  2% chance of actually having the disease. This counterintuitive result arises because the false positives from the large healthy population (5%) vastly outnumber the true positives from the tiny sick population (0.1%).

Visualizing the Update

The shift from prior to posterior is best seen visually. Figure 1 shows how the distribution concentrates around the true value as evidence accumulates:

The following table tracks how our belief updates through each piece of evidence:

This is why doctors order confirmatory tests. A single positive is weak evidence; multiple independent positives compound into strong evidence.

The Odds Form

Bayes’ theorem has an elegant equivalent in odds form that makes sequential updating trivial:

where the likelihood ratio $\text{LR}=𝑃\left(𝐸|𝐻\right)/𝑃\left(𝐸|\neg 𝐻\right)$ and $\text{odds}\left(𝐻\right)=𝑃\left(𝐻\right)/\left(1-𝑃\left(𝐻\right)\right)$.

This is the form used in practice The odds form avoids computing the marginal probability $𝑃\left(𝐸\right)$ entirely — the likelihood ratio handles it implicitly. because it factors out $𝑃\left(𝐸\right)$ and lets us chain updates:

Figure 2 illustrates the likelihood ratio concept: when the evidence is much more likely under the hypothesis than under the alternative, the LR is large and the update is strong.

Implementation

The code is straightforward. Here’s a Python implementation that handles both single and sequential updates:

class BayesUpdater:
    """Bayesian belief updater using the odds form."""

    def __init__(self, prior: float):
        self.odds = prior / (1 - prior)
        self.history = [prior]

    def update(self, lr: float) -> float:
        """Update belief with a new likelihood ratio."""
        self.odds *= lr
        p = self.odds / (1 + self.odds)
        self.history.append(p)
        return p

    @property
    def probability(self) -> float:
        return self.odds / (1 + self.odds)

# Medical diagnosis example
belief = BayesUpdater(prior=0.001)

# First positive test (LR = sensitivity / (1 - specificity))
belief.update(lr=0.99 / 0.05)   # P ≈ 0.0194
print(f"After 1st positive: {belief.probability:.4f}")

# Second independent positive test
belief.update(lr=0.99 / 0.05)   # P ≈ 0.284
print(f"After 2nd positive: {belief.probability:.4f}")

# Third independent positive test
belief.update(lr=0.99 / 0.05)   # P ≈ 0.867
print(f"After 3rd positive: {belief.probability:.4f}")

Output:

After 1st positive: 0.0194
After 2nd positive: 0.2841
After 3rd positive: 0.8673

Visualizing the sequence: Figure 3 shows how the probability climbs with each independent positive test — starting from near zero and crossing 50% after just three tests.

Applications

Spam Filtering

Email spam filters were one of the first killer apps of Bayesian reasoning. Paul Graham’s 2002 essay A Plan for Spam [2] showed that a simple Bayesian classifier could achieve >99.5% accuracy.

The idea: each word has a spam probability. Given the words in an email, combine them using Bayes’ theorem:

where ${𝑝}_{𝑖}=𝑃\left(\text{spam}|{𝑤}_{𝑖}\right)$ and $𝑠$ is the prior spam rate.

Modern spam filters (Gmail, etc.) use more sophisticated models, but the core insight remains Bayesian.

A/B Testing

In product development, Bayesian A/B testing provides a principled way to decide whether variant A or variant B is better:

Unlike frequentist hypothesis testing, this directly answers the question product managers actually care about: “what is the probability that B is better?”

Bayesian Neural Networks

Deep learning has embraced Bayesian ideas. Instead of learning a single weight vector $𝑤$, a Bayesian neural network learns a distribution over weights $𝑃\left(𝑤|\text{data}\right)$, enabling principled uncertainty estimates Uncertainty quantification is critical for safety-sensitive applications like autonomous driving and medical diagnosis..

Figure 4 compares traditional and Bayesian approaches to neural network weights:

The Philosophy of Priors

The choice of prior is the most controversial aspect of Bayesian reasoning. Critics argue it introduces subjectivity; proponents counter that all reasoning involves assumptions, and Bayes just makes them explicit.

There are several schools of thought:

• Objective Bayes: use “uninformative” priors (Jeffreys, maximum entropy) that let the data speak
• Subjective Bayes: encode genuine prior knowledge — this is a feature, not a bug
• Empirical Bayes: estimate the prior from the data itself Empirical Bayes is widely used in genomics and other fields with many similar hypotheses — see Efron’s work on large-scale inference [3].

The pragmatic view: the prior matters most with little data. With enough evidence, all reasonable priors converge This is a consequence of the “washing out of priors” — Bayesian posteriors are asymptotically dominated by the likelihood..

Conclusion

Bayes’ theorem is not just a formula — it is a framework for rational thought. It tells us how to update our beliefs when new evidence arrives, how much weight to give that evidence, and when to be skeptical of our own intuitions.

From spam filters to medical diagnosis, from A/B testing to artificial intelligence, Bayesian reasoning is everywhere — often hiding in plain sight. The next time you encounter surprising evidence, ask yourself: what is the likelihood ratio, and what was my prior?