Naive Bayes, Explained

There is a machine learning algorithm that makes an assumption so wrong it has the word “naive” baked right into its name, and yet it still cheerfully sorts your spam, classifies your documents, and refuses to die. Meet Naive Bayes: the classifier that is technically incorrect about how language works and somehow correct about which emails to bin.

Let’s unpack why something so simple, and so openly delusional, keeps earning its keep.

Bayes’ theorem, in plain words

At its heart sits Bayes’ theorem, which is really just a formula for updating a belief when new evidence shows up. You start with a hunch about how likely something is, you observe some clue, and you revise.

For classification the question is: given the words in this email, what’s the probability it’s spam? Bayes lets us flip a hard question into easier ones. Instead of asking “how likely is spam given these exact words” (basically unanswerable, since you’ve probably never seen this exact email before), we ask “how likely are these words if it’s spam” and multiply by “how common is spam in general.” The word “viagra” rarely shows up in your mum’s emails; it shows up in spam constantly. Bayes turns that lopsidedness into a verdict.

The “naive” part

Here’s the catch, and it’s a big one. To compute “how likely are all these words together if it’s spam,” you’d need to account for how words interact: “new” next to “york” means something different from “new” next to “offer.” Modelling every dependency is hopeless.

So Naive Bayes shrugs and assumes every word is conditionally independent of every other word, given the class. In plain English: it pretends that once you know an email is spam, the presence of “free” tells you nothing about whether “money” also appears. They just float around independently like strangers in a lift.

This is, of course, nonsense. Real language is dripping with dependencies. “Conditional independence” is the polite academic phrase for “we are knowingly lying to make the maths tractable.” And because of that lie, the probability of all the words is just the product of each word’s individual probability, which is the difference between a calculation you can do and one you can’t.

Why a wrong model gives right answers

If the assumption is so false, why does it work? The trick is that classification doesn’t need accurate probabilities. It only needs the right answer to win. As the scikit-learn docs put it, Naive Bayes is “a decent classifier” but “a bad estimator”, so don’t trust its confidence scores, but its rankings are often spot on.

Think of it like a doctor who is hilariously bad at saying “you have a 73% chance of flu” but reliably gets the diagnosis right. The exact numbers can be wildly miscalibrated, yet as long as the score for the correct class stays higher than the others, the argmax lands on the truth. The independence errors tend to inflate or deflate both classes together, so the ordering survives even when the magnitudes are garbage. This is why it has worked, in scikit-learn’s words, “in many real-world situations, famously document classification and spam filtering.”

The variants

Naive Bayes isn’t one thing, it’s a small family, each making a different assumption about how features are distributed:

• Multinomial (MultinomialNB): counts how often each word appears. The default workhorse for text.
• Bernoulli (BernoulliNB): cares only whether a word appears at all, not how many times. It explicitly penalises absent words, which can help on short documents.
• Gaussian (GaussianNB): for continuous features, assuming each follows a bell curve. Reach for this when your inputs are measurements, not word counts.
• Complement (ComplementNB): a tweak that behaves better on imbalanced datasets.

Show me the code

Here’s a complete spam-style text classifier in scikit-learn. Note how little there is:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

texts = [
    "win a free prize now",
    "claim your free money today",
    "lunch meeting at noon tomorrow",
    "can you review the report draft",
]
labels = ["spam", "spam", "ham", "ham"]

model = make_pipeline(CountVectorizer(), MultinomialNB())
model.fit(texts, labels)

print(model.predict(["free money waiting for you"]))  # ['spam']
print(model.predict(["are we still on for lunch"]))    # ['ham']

Vectorise the words, count them, fit. Training is essentially counting, which is why Naive Bayes can be, in scikit-learn’s phrasing, “extremely fast compared to more sophisticated methods.” It learns from a handful of examples and shrugs off high-dimensional data that would choke fancier models.

The takeaway

When you face a new classification problem, especially text, start with Naive Bayes before you reach for anything heavier. It trains in seconds, needs little data, gives you a real accuracy number to beat, and tells you fast whether the problem is even learnable. If a deep model can’t comfortably outscore your Naive Bayes baseline, the model isn’t the bottleneck, your features or labels are. Just don’t believe its probabilities, only its choices.

Sources: scikit-learn: Naive Bayes, Cornell CS4780: Bayes Classifier and Naive Bayes, UW CSE312: Notes on Naive Bayes for Spam Filtering.