Spam Filter and Text Classifier From Scratch: Naive Bayes to Transformers

Every day, your inbox quietly rejects millions of messages you never see. Behind that is a classifier, and the surprise is that a very simple one can do astonishingly well. In this capstone we build a spam filter from scratch, then a multi-class text classifier, and finally ask whether a Transformer is actually worth its price.

• Spam filtering at scale
• Simple models, serious results

In the last lesson we built customer segmentation and recommenders. Today is capstone five and six. We start with Naive Bayes from lesson 18, add TF-IDF with logistic regression, study errors and costs, then fine-tune a small pretrained Transformer using ideas from lessons 54 to 57.

• Naive Bayes from scratch
• TF-IDF and logistic regression
• Fine-tune a small Transformer

First, frame the problem. Each email is text, and the label is spam or not spam, which we call ham. We want a function that maps raw words to a probability of spam. Notice that this is binary classification, so everything from lessons 15 and 16 about probabilities and thresholds applies here.

• Label: spam or ham
• Output: probability of spam

Models need numbers, so we use a bag of words. Build a vocabulary, then count how often each word appears in each message. Word order is thrown away. The message free money now becomes a count vector with ones at free, money and now, and zeros everywhere else.

• Build a vocabulary
• Count words per message
• Order is discarded

Before modeling, decide what mistakes cost. A spam message in your inbox is annoying, costing maybe one unit. A real job offer sent to the spam folder could cost ten units or far more. Because the costs are unequal, plain accuracy is the wrong yardstick, and we will come back to this.

• False negative: cost 1
• False positive: cost 10

Naive Bayes starts with Bayes theorem from lesson 8. The probability of spam given the words is proportional to the prior probability of spam times the probability of the words given spam. The naive part assumes words are independent given the class, so we simply multiply one probability per word.

• Posterior ∝ prior × likelihoods
• Words independent given class

Here is a tiny training set. Ten emails, four spam and six ham. So the prior for spam is point four and for ham is point six. Across the spam emails there are twenty words in total, and across the ham emails thirty. Our vocabulary has ten distinct words.

• 4 spam, 6 ham: priors 0.4, 0.6
• 20 spam words, 30 ham words
• Vocabulary size 10

Now estimate word likelihoods by counting. The word free appears six times in spam and once in ham. The word meeting appears zero times in spam and five times in ham. The naive estimate for meeting given spam is zero divided by twenty, which is exactly zero. That is a disaster waiting to happen.

• free: 6 spam, 1 ham
• meeting: 0 spam, 5 ham
• 0 / 20 = 0

Why a disaster? Because we multiply. One zero wipes out every other word in the product, so any email containing meeting would have spam probability exactly zero no matter how many spammy words it contains. A single unseen word should not get veto power over all the evidence.

• Product with a zero is zero
• Unseen word vetoes all evidence

The fix is Laplace smoothing. Pretend you saw every vocabulary word one extra time in each class. The estimate becomes count plus one, divided by total words in the class plus the vocabulary size. For meeting in spam that is one over thirty, about zero point zero three three, small but not zero.

• Add one to every count
• Denominator adds vocabulary size V

Compute the rest. Free given spam is six plus one over thirty, which is seven thirtieths, about zero point two three three. Free given ham is one plus one over forty, which is zero point zero five. Meeting given ham is five plus one over forty, which is zero point one five.

• free | spam = 7/30 ≈ 0.233
• free | ham = 2/40 = 0.05
• meeting | ham = 6/40 = 0.15

Classify the message free meeting. Spam score: point four times point two three three times point zero three three, about point zero zero three one. Ham score: point six times point zero five times point one five, which is point zero zero four five. Normalizing, spam gets about forty-one percent. The filter says ham, because meeting outweighs free.

• Spam score ≈ 0.0031
• Ham score = 0.0045
• P(spam) ≈ 0.41

A practical warning. With hundreds of words, multiplying many small probabilities underflows to zero in floating point, as we saw in the numerical stability discussion. The standard fix is to work in logs: add the log prior and the log likelihoods, compare the sums, and never multiply raw probabilities.

• Products underflow
• Sum log probabilities instead

In code this is short. Count words per class with a dictionary, compute log probabilities using the smoothed formula, and for a new message sum the log prior plus the log likelihood of each known word. Skip words outside the vocabulary. The whole classifier fits in about thirty lines of NumPy and Python.

• Count, smooth, take logs
• Sum scores, pick the larger

Raw counts treat every word as equally informative, yet the word the tells you nothing. TF-IDF fixes this. Term frequency measures how often a word appears in this document. Inverse document frequency is the log of total documents divided by documents containing the word, so common words get weights near zero.

• TF: frequency in this document
• IDF: rarity across all documents

A worked number. With one thousand documents, free appears in one hundred of them, so its IDF is the natural log of ten, about two point three. If free makes up three of a hundred words in an email, TF is point zero three, and the TF-IDF weight is about point zero six nine. The word the scores exactly zero.

• IDF = ln(1000/100) ≈ 2.30
• TF = 3/100 = 0.03
• TF-IDF ≈ 0.069

Now feed these vectors into logistic regression from lessons 15 and 41. The model learns one weight per word, adds them up with a bias, and squashes the result through a sigmoid. Unlike Naive Bayes, it does not assume independence, so correlated words like free and winner share credit instead of double counting.

• One weight per word
• No independence assumption

The weights are readable. Sort them and the top positive words, such as free, winner and claim, push toward spam, while negative words like meeting and thanks push toward ham. Because there are many more features than examples, regularization from lesson 17 matters. Tune the strength C on validation data, not on the test set.

• Top positive: spam words
• Tune regularization on validation

The model outputs a probability, but you still choose a threshold. Suppose a false positive costs ten and a false negative costs one. Flag spam only when p times one exceeds one minus p times ten. Solving, p must exceed ten elevenths, about zero point nine one. Not zero point five.

• Flag if p > 10/11 ≈ 0.91
• Not the default 0.5

Compare two models on one thousand emails. Model A makes five false positives and twenty false negatives, for a cost of fifty plus twenty, seventy. Model B makes one false positive and forty false negatives, costing ten plus forty, fifty. Model B has more total errors, yet it is the cheaper model.

• A: 25 errors, cost 70
• B: 41 errors, cost 50

Now do error analysis, starting with false positives, the expensive mistakes. Pull every legitimate email the model flagged, sort by predicted probability, and read them. You will often find patterns: newsletters with promotional language, receipts that say free shipping, or short messages with unusual words the model rarely saw.

• Sort errors by confidence
• Look for repeating patterns

Each pattern suggests a fix. Receipts might need sender features. Newsletters may need a separate category or character n-grams to catch tricks like f r e e. Short messages may simply lack signal. Fix the biggest group first, retrain, and check that the cost, not just accuracy, truly went down.

• Group errors, fix the largest
• Track cost after each change

One more check: spam is an arms race. Spammers adapt, so the data distribution drifts, a topic from lesson 62. A filter trained last year will quietly degrade. Evaluate on the most recent data, split by time rather than at random, and plan to retrain regularly.

• Split by time, not randomly
• Expect drift and retrain

Now the second project: multi-class text classification. Imagine a news dataset with four topics: world, sports, business and science and technology. Each headline gets exactly one label. This is softmax territory from lesson 15, where the model outputs four probabilities that sum to one.

• Softmax over four classes
• Probabilities sum to one

Always begin with a baseline. A TF-IDF vectorizer plus logistic regression trains in seconds on a laptop CPU. It gives you a reference number, a feel for how hard the task is, and a model that is cheap to run. Any fancier model must beat this baseline by enough to justify its extra cost.

• TF-IDF plus logistic regression
• Fast, cheap, a reference line

Evaluate with more than accuracy. For many classes, use a confusion matrix, plus per-class precision and recall, and macro F1, which averages F1 equally over classes. Macro F1 matters when classes are imbalanced, because accuracy can look great while a small class is ignored entirely.

• Per-class precision and recall
• Macro F1 for imbalance

Next, fine-tune a small pretrained Transformer such as DistilBERT, which has about sixty-six million parameters. Recall lesson 54: text is split into subword tokens and mapped to embeddings. Lesson 55 gave us attention, so each token's representation now depends on its whole context, not just on counts.

• Subword tokens
• Context-aware representations

Fine-tuning, from lesson 57, means we keep the pretrained weights and add a small classification head, a linear layer from the summary vector to four logits. Train everything with cross-entropy and a tiny learning rate, around two times ten to the minus five, for a few epochs. Large rates can destroy the pretrained knowledge.

• Pretrained body plus new head
• Small learning rate, few epochs

Practical details matter. Truncate to a maximum length, say one hundred twenty-eight tokens, pad within batches, and use a GPU if possible. Watch validation loss each epoch and keep the best checkpoint. Set random seeds, because fine-tuning results can vary from run to run, so report averages over several seeds when the differences are small.

• Truncate, pad, batch
• Keep best checkpoint, vary seeds

Now compare honestly on three axes. Accuracy and macro F1 on the same held-out test set. Training cost: seconds on a CPU versus minutes or hours on a GPU. Latency: a linear model answers in well under a millisecond per text, while a Transformer takes far longer, especially on a CPU. Measure these yourself on your hardware.

• Accuracy and macro F1
• Training cost
• Latency per request

Typically the Transformer wins on accuracy, especially when meaning depends on context, but the gap can be modest on easy topic tasks. Ask whether a small gain is worth a hundred times more compute. For spam at huge volume, a cheap model with a good threshold may win. For subtle tasks, the Transformer earns its keep.

• Diminishing returns on accuracy
• Match model to the stakes

Look at the confusion matrices side by side. Both models will probably confuse business with science and technology, because a headline about a chip maker's earnings truly belongs to both. The Transformer shrinks the confusion but rarely removes it. When humans would disagree, no model can be expected to agree.

• Business versus Sci/Tech
• Ambiguous labels cap accuracy

Now two traps that inflate scores. First, duplicates. Text datasets are full of near copies: reposted articles, forwarded emails, templates. If a copy lands in train and another in test, the model just memorizes it, and your score is fake. Deduplicate before you split, as lesson 35 taught for leakage.

• Near copies cross the split
• Deduplicate before splitting

Second, label noise. Human labels are imperfect, and some are simply wrong. A good trick: list the test examples where the model is highly confident but disagrees with the label. Often the label is the mistake. Fix those, but never use the test set to tune the model; that would turn it into training data.

• Confident disagreements are suspects
• Audit labels, do not tune on test

Other leaks lurk too. Fitting the TF-IDF vectorizer on the whole dataset lets test document frequencies influence training. Always fit it inside a pipeline on training data only. Also watch for shortcuts, like email headers or signatures that reveal the label without any real language understanding.

• Fit vectorizer on train only
• Beware shortcut features

Let us recap. Naive Bayes with Laplace smoothing and log probabilities gives a strong, tiny baseline. TF-IDF with logistic regression learns better weights. Cost asymmetry moves the threshold, so judge models by cost, not accuracy. Error analysis tells you what to fix, and leakage checks keep your numbers honest.

• Smoothing, logs, TF-IDF
• Cost-based thresholds
• Errors and leakage checks

For multi-class text, start with a TF-IDF baseline, then fine-tune a pretrained Transformer and compare accuracy, cost and latency on the same clean test set. The best model is not always the biggest one. It is the one that meets the need at a price you can afford.

• Baseline first
• Compare on equal footing

Next time, in lesson 67, we move to unsupervised work: customer segmentation with clustering and a recommendation system.

• Lesson 67 preview
• Subscribe and build it yourself