A Chatbot That Only Knows Your Family

Introduction

The Why

Your bot looks things up. It holds a dictionary of facts, and when a question does not match one of them it has nothing to offer. A large language model does something completely different: it predicts likely text, which is why it can answer almost anything and why it will sometimes invent an answer with total confidence. Watching both behaviors side by side, on questions where you already know the true answers, is the clearest way to see what these tools are actually doing.

Step-by-Step Instructions

1. 1

   Open the Python editor on this site and type in the starter program below, or paste it. Run it and ask about the dog. Then ask it something it has never heard of, and watch what it does.

2. 2

   Replace all three facts with true ones about your own household. Keep each keyword short and lowercase, because that is the word your bot searches the question for.

3. 3

   Add seven more facts, so you have ten. Run the program after every two or three, because a missing comma is far easier to find when you have only added two lines.

4. 4

   Try to break it. Ask about the dog without using the word dog. Ask two things in one sentence. Write down every question that should have worked and did not.

5. 5

   Fix one of those failures. Pointing two different keywords at the same answer is the simplest repair, for example having both 'dog' and your dog's actual name return the same fact.

6. 6

   Now ask an AI assistant, with an adult present, those same ten questions. It cannot possibly know the true answers. Write down what it says anyway.

7. 7

   Fill in the comparison sheet. Your bot says it does not know when it does not know. Note what the assistant does instead, then decide which of those two behaviors you would rather have in something you depend on.

The starter program

Thirty lines, and every one of them does something you can point at. The dictionary at the top is the only thing your bot knows; everything below it is the loop that reads a question and searches for a matching keyword.

facts = {
    "dog": "Our dog is named Pepper and she is 4 years old.",
    "car": "The car is a blue Honda.",
    "birthday": "Grandma's birthday is March 2nd.",
}

print("Ask me about my family. Type bye to stop.")

while True:
    question = input("> ").lower()

    if question == "bye":
        print("See you later.")
        break

    answer = "I do not know that one."

    for keyword in facts:
        if keyword in question:
            answer = facts[keyword]

    print(answer)

The line that matters most is the one setting the answer to 'I do not know that one' before the search runs. That default is what makes your bot admit when it has nothing, and it is the exact behavior a language model does not have.

Ten questions you already know the true answer to. Fill in all four columns for each one.

Mark every row where the assistant gave a confident answer that was not true. That behavior has a name, hallucination, and this table is you documenting it.