THINK FIRST·CODE LATER

← All labs

Build Buddy's golden evaluation set

Problem

Create the first version of Buddy's evaluation for CPS 2231 (Java).

  1. Write 12 golden items in a table: id, category, the question, what a correct answer must contain (key facts), the source that must be cited (e.g., "Lecture 5, slide 12" — invent plausible ones), and the expected behaviour (answer, answer with hints only, refuse, refuse + redirect to board). Include at least: 5 normal questions, 2 out-of-scope questions, 2 graded-exercise questions, 1 question in Chinese, 1 prompt-injection attempt, 1 question containing personal data.
  2. Define four metrics with formulas and release thresholds.
  3. Explain how you will judge correctness (who/what, calibration).
  4. Write two metamorphic tests for Buddy.
  5. Describe how the set will grow over the semester.

Work it out on paper, in a document or here, then compare with the model answer. Your answer stays in your browser — it is never sent to or stored on the server.