Week 3 · Lesson 3 · 50-60 minutes
Confuse and Improve the Model
Find the edge cases that fool a classifier, read its mistakes in a confusion matrix, and propose changes that would make it better.
- Materials
- This lesson in a web browser · Paper and pencil, or a notes app · The built-in mistakes set and confusion matrix shown in the lesson
- This lesson
- In progress
Auto-saves on this device
What you'll be able to do
- Tell a false positive from a false negative for a category.
- Identify edge cases that are likely to confuse a classifier.
- Read a confusion matrix to see which categories get mixed up.
- Propose a change, such as adding examples, that would reduce a specific mistake.
Think about it
A pet classifier keeps calling one specific fluffy little dog a 'cat'. It is confident every time. Is this random bad luck, or could there be a pattern in its mistakes you could actually find and fix?
This lesson is about finding the pattern in a classifier's mistakes, not just noticing that it makes them.
Make a prediction
Predict: for a cat-dog-rabbit classifier, which two categories do you think get confused with each other most often, and why?
Vocabulary
See the full course vocabulary- False positive
- When the classifier says something IS in a category but it is not — for example, calling a photo 'cat' when it is really a dog.
- False negative
- When the classifier says something is NOT in a category but it really is — for example, missing a real cat and calling it a dog.
- Edge case
- An unusual or tricky input near the boundary between categories, like a fluffy dog that looks a bit like a cat, which is easy to get wrong.
- Confusion matrix
- A table that lines up the true labels against the classifier's predictions, so you can see exactly which categories get mixed up and how often.
False positives and false negatives
Not all mistakes are the same. A false positive for 'cat' is when the classifier says 'cat' but the photo is not a cat. A false negative for 'cat' is when the photo really is a cat but the classifier says it is something else. Every category has both kinds.
The difference matters because the two mistakes can have very different costs. Missing a real fire alarm (a false negative) could be far more serious than a false alarm (a false positive), so people often care which kind of mistake a system makes.
- False positive for 'cat': a rabbit photo labeled 'cat'.
- False negative for 'cat': a real cat photo labeled 'dog'.
- A spam filter marking a real message as spam is a false positive for 'spam'.
Edge cases sit near the boundary
Classifiers make most of their mistakes on edge cases: unusual inputs that sit near the boundary between two categories. A fluffy small dog shares features with cats, a rabbit photographed with its ears down loses a key feature, and a dark blurry photo hides features from every category.
Edge cases are not random. If you can predict which inputs are near the boundary, you can predict where the classifier will struggle — and gather better examples for exactly those cases.
- A hairless cat that lacks the usual fur texture.
- A dog wearing a costume that hides its shape.
- A photo taken in very low light where colors and edges are unclear.
Reading a confusion matrix, then improving the model
A confusion matrix lays the true labels against the predictions in a grid. Numbers on the diagonal (true cat predicted cat) are correct; numbers off the diagonal are mistakes, and each off-diagonal cell tells you exactly which mix-up happened and how often. A big number in the 'true dog, predicted cat' cell means dogs are often mistaken for cats.
Once you can see the biggest mistake, you can improve the model on purpose. The most common fix is adding more, better examples of the confused case — such as more photos of fluffy small dogs — so the classifier learns the features that tell them apart. You improve where the matrix points, then measure again.
- The cell 'true rabbit, predicted cat' being large means rabbits get called cats a lot.
- Adding clearer rabbit photos targets that exact mistake.
- After a fix, you re-check accuracy and the matrix to confirm the mistake shrank.
Worked example
From a confusing mistake to a targeted fix
- Look at the confusion matrix and find the largest off-diagonal number: 'true rabbit, predicted cat' with 9 mistakes.
- Name the mistake type: for the cat category these are false positives (called cat, but really rabbits); for the rabbit category they are false negatives (real rabbits missed).
- Ask why: rabbits and cats can share fur texture and rounded shapes, and rabbits with ears down lose the feature that sets them apart — classic edge cases.
- Propose a fix: add more rabbit photos to the training set, especially rabbits with ears down and at cat-like angles, so the classifier learns the differences.
- Re-measure: test again and check whether the 'true rabbit, predicted cat' cell got smaller and rabbit accuracy went up.
Takeaway: Find the biggest mistake in the confusion matrix, name what kind it is, explain the edge case behind it, add targeted examples, then measure again.
| Predicted | |||
|---|---|---|---|
| Actual | Cat | Dog | Rabbit |
| Cat | 47 | 2 | 1 |
| Dog | 3 | 44 | 3 |
| Rabbit | 9 | 4 | 37 |
A 3 by 3 confusion matrix for a cat, dog, and rabbit classifier on the test set. Rows are the true label, columns are the predicted label. True cats: 47 predicted cat, 2 dog, 1 rabbit. True dogs: 3 cat, 44 dog, 3 rabbit. True rabbits: 9 cat, 4 dog, 37 rabbit. The diagonal (47, 44, 37) is correct. The largest mistake is 9 in the 'true rabbit, predicted cat' cell, showing rabbits are most often mistaken for cats — the same weak rabbit category from the accuracy chart.
Diagonal cells are correct; the 9 in 'true rabbit, predicted cat' is the biggest mistake to fix.
Before: the training set had few photos of rabbits with their ears down, so many such rabbits were predicted as cats, and rabbit accuracy sat at 74 percent. After: more rabbit photos were added, especially edge cases with ears down and at cat-like angles; the classifier learned the difference, the 'true rabbit, predicted cat' mistakes dropped, and rabbit accuracy rose. The fix targeted the exact mistake the confusion matrix pointed to.
Adding examples aimed at the biggest confusion is how you improve on purpose.
Activity
Confuse and Improve
Try to confuse the classifier with edge cases, then improve a weak model by adding a few varied training pictures and comparing the first and improved models.
- Part 1 — Confuse the model: predict what it will do on each tricky picture, run it, and see the source of the confusion.
- Part 2 — Improve it: a weak starter model has only ever seen circles and squares, never a triangle. Add a few varied pictures and retrain.
- Compare the first and improved models: what got fixed, what broke, and why.
Part 1 · Confuse the model
Each picture is a known edge case. Predict the model's answer first, then run it.
Geometric shapes: A square rotated 45 degrees, so it looks like a diamond.
Edge case: rotated
Geometric shapes: A circle with its right half hidden behind a block.
Edge case: obstructed
Geometric shapes: A square on a noisy, speckled background.
Edge case: unusual background
Geometric shapes: A blurred triangle with soft, fuzzy edges.
Edge case: blurred
Geometric shapes: A very small circle in the middle of the picture.
Edge case: very small
School supplies: A very thick pencil that looks more like a bar.
Edge case: shared features
Geometric shapes: A square with a small circle overlapping one corner.
Edge case: mixed category
Part 2 · Improve the model
The starter model has only ever seen circles and squares — it has never seen a triangle, so it misses every one. Add up to 4 more varied pictures (try some triangles!) and retrain to see if it improves.
Add pictures (0/4 used)
Add at least one picture to build an improved model.
Check your understanding
Answer these to check that you can read a classifier's mistakes and improve it.
Question 1. What kind of mistake is this for the 'cat' category?
A photo is really a rabbit, but the classifier predicts 'cat'.
Question 2. Which of these are good ways to reduce a specific mistake a confusion matrix reveals? Choose all that apply.
Select all that apply.
Question 3. Decide if the statement is true or false.
In a confusion matrix, the numbers on the diagonal are the correct predictions.
0 of 3 answered
Try it yourself
Hunt the confusion
Pick a category pair you expect a classifier to confuse and plan how you would find and fix that confusion.
- Choose two similar categories (for example, muffins vs. cupcakes, or wolves vs. dogs).
- Describe one edge case near the boundary and predict whether it becomes a false positive or false negative for each category.
- Say which cell of a confusion matrix would grow, and what examples you would add to shrink it.
Success looks like
- A believable confused pair with a boundary edge case.
- The mistake correctly framed as a false positive or false negative.
- A targeted fix tied to a specific confusion-matrix cell.
Reflection
Saved on this device as you type
Please don't include private information. Your answers are saved only in this browser and are never sent anywhere.
Recap
A confusion matrix shows exactly which categories a classifier mixes up, and edge cases explain why; you improve the model by adding targeted examples and measuring again.
- False positives and false negatives are different mistakes with different costs.
- Edge cases near a category boundary cause most confusions.
- Read the confusion matrix, add examples for the biggest mistake, then re-measure.
Grades 7–8 extension
Which mistake would you rather make?
For many real systems, you cannot remove all mistakes, so designers choose which kind to lean toward. A medical screening tool might accept more false positives (extra check-ups) to avoid false negatives (a missed illness), while a spam filter might accept more false negatives (some spam gets through) to avoid false positives (losing a real message).
Pick a real classifier and argue which mistake it should avoid most. How could its makers shift the model or its threshold to trade one kind of error for the other?
Finish this lesson
- Attempt the knowledge check — not done
- Save a reflection (recommended) — not done
Attempt the knowledge check to finish this lesson.