Week 3 · Lesson 2 · 45-55 minutes
Train and Test a Classifier
Learn why a classifier is checked on photos it never trained on, and measure how well it does using accuracy on a fair test set.
- Materials
- This lesson in a web browser · Paper and pencil, or a notes app · The built-in labeled test set shown in the lesson
- This lesson
- In progress
Auto-saves on this device
What you'll be able to do
- Explain why a classifier is tested on separate photos it did not train on.
- Calculate accuracy as correct predictions out of total predictions.
- Read category-level accuracy and spot which category is weakest.
- Explain why one overall accuracy number can hide a weak category.
Think about it
A classifier gets every single training photo right. Its makers say it is 'perfect'. Then it meets brand-new photos and misses many of them. What went wrong with the way they measured it?
Think about the difference between memorizing the answers and actually learning the pattern.
Make a prediction
Predict: if a classifier scores 90% accuracy overall on cats, dogs, and rabbits, do you think it does equally well on all three categories?
Vocabulary
See the full course vocabulary- Accuracy
- How often a classifier is correct, usually the number of correct predictions divided by the total number of predictions, shown as a percent.
- Category-level accuracy
- The accuracy measured separately for each category, so you can see which categories the classifier handles well and which it struggles with.
- Test set
- A group of labeled photos kept aside and never used in training, used to fairly check how well the classifier does on new images.
- Training set
- The labeled photos the classifier learns its patterns from before it is tested.
Why you test on photos the classifier never trained on
A classifier learns from a training set. If you then test it on those same photos, it can look great just by having memorized them — like a student who saw the exact test questions in advance. That score would not tell you how it does on new photos.
So we keep a separate test set: labeled photos held back and never used in training. Testing on new-to-the-classifier photos is the fair way to measure whether it really learned the pattern.
- Train on 800 pet photos, then test on 200 different pet photos.
- A high training score plus a low test score is a warning sign of memorizing.
- The test photos must be labeled too, so you can check each prediction.
Accuracy: how often the classifier is right
Accuracy is the simplest score: the number of correct predictions divided by the total number of predictions, written as a percent. If the classifier gets 90 out of 100 test photos right, its accuracy is 90%.
Accuracy is easy to understand, which is why it is so common. But one single number covers everything at once, so it can hide problems — and that is where category-level accuracy comes in.
- 45 correct out of 50 photos is 90% accuracy.
- Two classifiers can share the same accuracy but make very different mistakes.
- Accuracy only makes sense on a labeled set, where you know the right answers.
Category-level accuracy tells a fuller story
Category-level accuracy measures the score for each category on its own. Overall accuracy might be 90%, but the classifier could get cats right 98% of the time and rabbits right only 74% of the time. The overall number quietly averaged the weak category away.
Breaking accuracy down by category shows you exactly where to improve. It is often the first step to finding out which category the classifier confuses — the focus of the next lesson.
- Overall 90%, but cats 98%, dogs 92%, rabbits 74%.
- A weak category can be hidden if it has only a few photos in the test set.
- Fixing the weakest category usually raises the overall accuracy the most.
Worked example
Measuring a pet classifier fairly
- Split the photos: 800 go into the training set, 200 different ones into the test set.
- Train: the classifier learns cat, dog, and rabbit patterns from the 800 training photos only.
- Test: run all 200 held-back photos through and compare each prediction to the true label.
- Overall accuracy: 180 of the 200 were correct, so 180 ÷ 200 = 90% accuracy.
- Category-level: cats 98%, dogs 92%, rabbits 74%. The rabbits are dragging things down, even though the overall 90% looked strong.
Takeaway: Test on held-back photos, compute accuracy as correct over total, then break it down by category to find the weak spot the overall number hides.
Values shown in % accuracy.
A bar chart of accuracy per category, in percent, on the 200-photo test set. Cats are 98 percent, dogs are 92 percent, and rabbits are only 74 percent. A dashed reference line marks the 90 percent overall accuracy. The chart makes it obvious that rabbits are the weakest category and are pulling the overall score down, even though the single overall number looked strong.
The overall 90% hides that rabbits are far weaker than cats and dogs.
Before (unfair): the classifier is tested on the very same photos it trained on, so it can score high just by memorizing them, and the score does not predict how it handles new photos. After (fair): the photos are split so training and testing use different photos; the classifier is scored only on photos it never saw in training, which shows whether it truly learned the pattern.
Testing on new-to-the-classifier photos is what makes the score trustworthy.
Activity
Train and Test a Classifier
Pick a topic, choose the training pictures, really train an on-device classifier, and read overall accuracy, category-level accuracy, and a confusion matrix on unseen pictures.
- Pick a topic and read the categories.
- Choose which pictures to train on (keyboard or tap), and watch the per-category counts.
- Predict the easiest and hardest category, train, then read the results, confusion matrix, and category accuracy.
2 · Categories
- Circle
- A round shape with no corners.
- Triangle
- A shape with three straight sides and three corners.
- Square
- A shape with four equal straight sides and four corners.
This classroom model is a nearest-neighbor classifier over simple shape features — it runs fully on your device with no camera, internet, or big AI service. It is deliberately simple, so it makes the same kinds of mistakes you can see and explain here.
Check your understanding
Answer these to check that you can measure a classifier fairly with accuracy.
Question 1. A classifier gets 45 out of 50 test photos correct. What is its accuracy?
Question 2. Which classifier was measured in a fair way?
Team A tests its classifier on the exact same 500 photos it trained on and reports 99%. Team B holds back 100 photos, trains on the rest, and tests only on the held-back 100.
Question 3. Decide if the statement is true or false.
One overall accuracy number can hide that the classifier does poorly on a particular category.
0 of 3 answered
Try it yourself
Split it fairly
Design a fair train/test plan for a classifier that sorts photos of three kinds of fruit, and describe what you would measure.
- Decide how many photos go into the training set and how many are held back for the test set.
- Explain why the test photos must not be used during training.
- List the accuracy numbers you would report: the overall accuracy and the accuracy for each fruit.
Success looks like
- A clear split of photos into training and test groups.
- A reason the test photos must be kept out of training.
- Overall accuracy and per-category accuracy are both named as things to report.
Reflection
Saved on this device as you type
Please don't include private information. Your answers are saved only in this browser and are never sent anywhere.
Recap
A classifier is measured fairly on a held-back test set using accuracy, and breaking accuracy down by category reveals weak spots the overall number hides.
- Test on a separate set of photos the classifier never trained on.
- Accuracy is correct predictions divided by total predictions.
- Category-level accuracy shows which category is weakest, even when the overall score looks strong.
Grades 7–8 extension
When accuracy can mislead
Imagine a test set that is 95% cats and only 5% rabbits. A lazy classifier that guesses 'cat' every single time would score 95% accuracy while getting every rabbit wrong. The high accuracy hides a total failure on rabbits.
Explain why the mix of categories in a test set matters. What could you measure or change so that a classifier cannot 'cheat' by ignoring a rare category?
Finish this lesson
- Attempt the knowledge check — not done
- Save a reflection (recommended) — not done
Attempt the knowledge check to finish this lesson.