Week 2 · Lesson 2 · 45-55 minutes

Training Data Versus Testing Data

See why a model is taught on one set of examples and checked on another, and how testing on unseen examples measures whether it really learned the pattern.

Materials: This lesson in a web browser · Paper and pencil, or a notes app · The built-in animal dataset shown in this lesson
This lesson: In progress
Auto-saves on this device

What you'll be able to do

Explain the difference between training data and testing data.
Describe why a model must be tested on examples it did not learn from.
Define generalization and accuracy and connect them to testing.

Think about it

A teacher hands out a practice worksheet, then gives a quiz with the exact same questions and answers. If everyone scores 100 percent, does that prove they learned the material — or just that they memorized those answers?

Hold onto this — a model checked on the very examples it learned from has the same problem.

Make a prediction

Predict: if you test a model only on the same examples it trained on, will its score tell you how well it handles brand-new examples?

Vocabulary

See the full course vocabulary

Training: Showing a model many labeled examples so it can learn the pattern that links features to a category.
Testing: Checking a model on new labeled examples it did not learn from, to see how well it really works.
Training data: The examples a model learns from during training.
Testing data: A separate set of examples, held back from training, used only to check the model.
Generalization: How well a model works on new examples it has never seen, not just the ones it learned from.
Accuracy: The share of test examples a model labels correctly, often written as a fraction or a percent.

Training: learning the pattern from examples

Training is the step where a model looks at many labeled examples and learns a pattern that connects features to categories. The examples used for this are the training data.

During training, the model gets to see both the features and the correct label for each example, so it can adjust until its guesses tend to match the labels.

Showing a model 100 labeled animal rows so it learns 'has wings usually means bird'
Feeding a spam filter thousands of emails already marked spam or not spam

Testing: a fair check on unseen examples

If you only checked a model on the same examples it learned from, it could just repeat answers it already memorized — like a student quizzed on the exact worksheet they practiced. That would not prove it learned anything useful.

So we hold back some examples as testing data. The model never trains on them. Testing means running the model on this separate, unseen set and comparing its guesses to the true labels. That is a fair check of what it actually learned.

Keep 20 animal rows aside; train on the other 80; then test on the 20 held-back rows
A practice set to study from and a separate quiz to be graded on

Generalization and accuracy

Generalization is how well a model does on new examples it has never seen. A model that generalizes well learned the real pattern, not just the exact training rows. Testing on unseen data is how we measure generalization.

Accuracy is a simple score for that check: out of all the test examples, what share did the model label correctly? If it labels 18 out of 20 test animals right, its accuracy is 18/20, or 90 percent. High accuracy on unseen test data is the sign that a model truly generalizes.

16 of 20 test examples correct → accuracy 80 percent
A model that scores high on training rows but low on test rows did not generalize

Worked example

Splitting 20 animals into training and testing

Start with a dataset of 20 labeled animals (each row has features and a 'bird' or 'not bird' label).
Split it: put 15 rows in the training set and hold back 5 rows as the testing set.
Train: let the model learn the pattern using only the 15 training rows.
Test: show the model the 5 held-back rows without their labels and record its guesses.
Score accuracy: compare its 5 guesses to the true labels. If 4 are right, accuracy is 4/5, or 80 percent — a measure of how well it generalizes.

Takeaway: Train on one part of the data, test on a held-back part, and measure accuracy on the unseen part to judge generalization.

Split the data, then check

Two stages. Stage 1 (Training): the training data — most of the rows — flows into the model, which learns a pattern. Stage 2 (Testing): the held-back testing data, which the model never saw, flows in; the model's guesses are compared to the true labels to get an accuracy score. The key point: the testing rows are kept completely separate from training.

The model learns from training data and is graded on separate testing data.

Same model, two scores

On training data98

On unseen test data82

Values shown in % correct.

A bar chart comparing one model's accuracy on the data it trained on versus on unseen test data, measured in percent. Bar 1, 'On training data', is 98 percent. Bar 2, 'On unseen test data', is 82 percent. The test score is the honest measure of how well the model generalizes; a much lower test bar warns that the model mostly memorized.

Activity

Train, Test, and Change the Data

Split a space-fruit dataset into training and testing sets, run a transparent model, then experiment with how balanced, unbalanced, and mislabeled data change the results.

Split the 16 fruits into a training set (to learn from) and a testing set (held back to check the model).
Every fruit goes in exactly one set. Keep some of each category in the test set.
Run the model, read its accuracy, then try the three data experiments below.

Predict first: why must some examples be hidden from training?

Assign each fruit

Training set

Safe: 6 · Not safe: 6

Testing set (held back)

Safe: 2 · Not safe: 2

Sixteen space fruits. For each, a toggle assigns it to the training set or the testing set, and its correct label is shown.
Fruit	Correct label	In which set?
DawnberryA 10 cm crimson round fruit with smooth skin and 6 seeds, sweetness 8 of 10, glow level 0 of 5, growing in a canopy.	Safe
SandplumA 8 cm amber oval fruit with bumpy skin and 4 seeds, sweetness 7 of 10, glow level 1 of 5, growing in a dune.	Safe
TidefruitA 12 cm teal round fruit with fuzzy skin and 8 seeds, sweetness 9 of 10, glow level 0 of 5, growing in a reef.	Safe
EmberpearA 9 cm crimson oval fruit with smooth skin and 5 seeds, sweetness 6 of 10, glow level 2 of 5, growing in a canopy.	Safe
DunedropA 7 cm amber round fruit with bumpy skin and 3 seeds, sweetness 7 of 10, glow level 1 of 5, growing in a dune.	Safe
CanopyfigA 11 cm teal oval fruit with fuzzy skin and 7 seeds, sweetness 8 of 10, glow level 0 of 5, growing in a canopy.	Safe
RedgourdA 10 cm crimson round fruit with smooth skin and 6 seeds, sweetness 5 of 10, glow level 2 of 5, growing in a dune.	Safe
ReefplumA 8 cm amber oval fruit with fuzzy skin and 4 seeds, sweetness 9 of 10, glow level 1 of 5, growing in a reef.	Safe
StarflareA 5 cm violet star fruit with spiky skin and 2 seeds, sweetness 3 of 10, glow level 4 of 5, growing in a crater.	Not safe
GlowcoilA 4 cm azure spiral fruit with bumpy skin and 1 seeds, sweetness 2 of 10, glow level 5 of 5, growing in a cavern.	Not safe
BrightstarA 6 cm violet star fruit with smooth skin and 0 seeds, sweetness 4 of 10, glow level 3 of 5, growing in a crater.	Not safe
ThornstarA 5 cm azure star fruit with spiky skin and 3 seeds, sweetness 2 of 10, glow level 2 of 5, growing in a cavern.	Not safe
SpikecoilA 4 cm violet spiral fruit with spiky skin and 1 seeds, sweetness 3 of 10, glow level 1 of 5, growing in a crater.	Not safe
CavernglowA 7 cm azure star fruit with bumpy skin and 2 seeds, sweetness 1 of 10, glow level 4 of 5, growing in a cavern.	Not safe
VoidstarA 5 cm violet spiral fruit with smooth skin and 0 seeds, sweetness 3 of 10, glow level 5 of 5, growing in a crater.	Not safe
BarbstarA 6 cm azure star fruit with spiky skin and 4 seeds, sweetness 2 of 10, glow level 3 of 5, growing in a cavern.	Not safe

Experiment: change the data, keep the test the same

Each experiment trains on a different version of the data and is checked on the same 8 held-back test fruits. Predict what will happen, then run it.

A · Balanced, correct data

All 24 examples, evenly split between Safe and Not-safe, every label correct. This is the fair starting point.

Predict: how will accuracy compare to the balanced data, and why?

Write a prediction first.

B · Unbalanced data

Every Safe example but only 2 Not-safe examples, so the model barely sees the Not-safe category.

Predict: how will accuracy compare to the balanced data, and why?

Write a prediction first.

C · Data with incorrect labels

The same 24 examples, but 5 of them have their labels flipped to the wrong answer.

Predict: how will accuracy compare to the balanced data, and why?

Write a prediction first.

This is a simplified classroom model (k-nearest neighbors). It compares a new fruit to the most similar training fruits and copies the most common label. Real AI systems are far larger — but the same ideas about good data still apply. The made-up rule is: A space fruit is Not safe to eat if it glows brightly (glow level 3 or more) OR it is spiky. Otherwise it is Safe to eat.

Check your understanding

Answer these to check that you understand training, testing, and accuracy.

Question 1. Why do we test a model on examples it did not train on?
Question 2. Whose test result is a fair measure of learning?
Ava studies a practice sheet, then is quizzed on brand-new questions of the same kind. Ben studies a practice sheet, then is quizzed on the exact same questions with the answers shown during practice.
Question 3. A model labels 15 out of 20 test examples correctly. What is its accuracy?

0 of 3 answered

Try it yourself

Plan a fair test

Design a fair training-and-testing plan for a model that sorts messages into 'question' or 'not a question'.

Decide how many example messages you would collect in total.
Choose how to split them into a training set and a testing set, and say why.
Explain how you would compute accuracy from the test results.

Success looks like

A clear total number of examples and a training/testing split.
A reason the testing set is kept separate from training.
A correct description of how accuracy would be measured.

Reflection

Saved on this device as you type

Please don't include private information. Your answers are saved only in this browser and are never sent anywhere.

Why can a model score very high on its training data yet still fail on new examples?

How is holding back testing data like the way a teacher writes a quiz that is different from the practice sheet?

Recap

Models train on one set of examples and are tested on a separate, unseen set; accuracy on that unseen test data measures how well the model generalizes.

Training data teaches the model; testing data checks it.
Testing rows must be held back and never seen during training.
Generalization is doing well on new examples; accuracy is the share of test examples labeled correctly.

Grades 7–8 extension

When training looks great but testing looks bad

Sometimes a model scores almost perfectly on its training data but poorly on the test data. This usually means it memorized the exact training rows instead of learning the general pattern.

Describe a case where this could happen — for example, a model that memorized specific fruits by their exact weight. What could you change about the data or the split to get a more honest picture of how it will perform?

Finish this lesson

Attempt the knowledge check — not done
Save a reflection (recommended) — not done

Attempt the knowledge check to finish this lesson.