Skip to content

AI Review Mission

You're on a review board checking a fictional AI product before it launches. Answer the questions, then write your recommendation. This is for your learning — there are no grades, nothing is sent anywhere, and you can retry any part.

The product: Northgate Community School proposes “PathwayAI”, an assistant that (1) recommends after-school programs to students, (2) answers family questions in a chat window, and (3) flags scholarship applications for staff review. Your job on the review board is to check it before it launches.

  1. A. ConceptsAI vs. rules, training vs. testing, confidence, and recommendation behavior.
  2. B. Reading the resultsOverall vs. group results, model mistakes, and uncertainty.
  3. C. Fairness & privacyUnder-represented data, unnecessary personal data, and human review.
  4. D. MisinformationSource, date, context, and independent evidence.

Answer all 10 questions across sections A–D. 7 correct is a strong pass — but you don't need a perfect score to finish the course.

  1. Question 1. PathwayAI's program recommender learns from thousands of past student choices. What makes this AI rather than a traditional program?

  2. Question 2. The team checked the recommender only on the same students it learned from. Why is that a problem?

  3. Question 3. The chatbot answers a family's question with “95% confidence.” What does that confidence mean?

  4. Question 4. PathwayAI keeps recommending the same kind of program to a student who only ever picked sports. What is happening?

  5. Question 5. Read the accuracy chart. What does it reveal?

    PathwayAI application-flagging accuracy

    Accuracy in percent. Overall: 91%. Group A applicants: 94%. Group B applicants: 62%. The overall number hides that Group B is handled much worse.

  6. Question 6. Read the confusion matrix. Which mistake is most common?

    Flagging results (rows = actual, columns = predicted)
    Predicted
    ActualApproveReview
    Approve4018
    Review537

    Actual 'approve' rows: 40 predicted approve, 18 predicted review. Actual 'review' rows: 5 predicted approve, 37 predicted review. The biggest mistake is 18 approvable applications wrongly sent to review.

  7. Question 7. For one application, PathwayAI predicts “review” with 54% confidence. What is the responsible read?

    Confidence for one application

    Confidence: Review 54%, Approve 46%. This is nearly a tie — the model is uncertain.

  8. Question 8. Group B applications are handled worse and Group B is under-represented in the training data. Which are good responses? Choose all that apply.

    Select all that apply.

  9. Question 9. To recommend programs, which data does PathwayAI NOT need? Choose all that are unnecessary or too sensitive.

    Select all that apply.

  10. Question 10. A viral post claims “PathwayAI rejected 500 students last week!” with a dramatic photo. Which checks should you make before believing it? Choose all that apply.

    Select all that apply.

0 of 10 answered

E · Your recommendation

Any of these can be the right call — what matters is your reasoning. Don't include private information.

Decision

What good could it do?

What could go wrong?

What data is and isn't needed?

Who might be treated unequally?

Who reviews, and when?

How can someone challenge a result?

Choose a decision · 0 of 6 reasons written.

Final reflection

Saved on this device. Don't include private information.

Go to completion & certificateCourse overview