THINK FIRST·CODE LATER

← All labs

Evaluation harness: may the new Buddy model be released?

Problem

Compare two versions of Buddy on a golden set and decide whether the new version may be released.

Input:

THRESHOLDS correctness citation refusal
<golden items: id TYPE requiredSource keyword1,keyword2,...>
RESULTS
<lines: version id cited refused answer words...>
  • Golden item TYPE is ANSWER (Buddy must answer, cite requiredSource and include all keywords) or REFUSE (out of scope or graded — Buddy must refuse; requiredSource and keywords are -).
  • Result lines: version is OLD or NEW; cited is the source id Buddy cited (or -); refused is Y/N; then the answer text.

Scoring, per version, over the golden items:

  • correctness — ANSWER items not refused whose answer contains every keyword (case-insensitive substring) / number of ANSWER items;
  • citation — ANSWER items not refused that cited the required source / number of ANSWER items;
  • refusal — REFUSE items with refused = Y / number of REFUSE items;
  • violations — REFUSE items answered (refused = N) — any violation blocks the release.

Print for each version, OLD then NEW:

OLD: correctness 0.67 citation 0.67 refusal 1.00 violations 0

Then the decision for NEW: RELEASE if NEW meets every threshold, has 0 violations and is not lower than OLD on any of the three metrics; otherwise BLOCK: followed by all reasons in this order, separated by ; : correctness below threshold, citation below threshold, refusal below threshold, integrity violations, regression in X (for each metric where NEW < OLD, in the order correctness, citation, refusal).

Input:

THRESHOLDS 0.6 0.6 1.0
q1 ANSWER L3 class,object
q2 ANSWER L5 interface
q3 ANSWER L7 exception
q4 REFUSE - -
RESULTS
OLD q1 L3 N A class is a blueprint for an object
OLD q2 L5 N An interface declares methods
OLD q3 L2 N Use try and catch
OLD q4 - Y I cannot solve graded homework
NEW q1 L3 N A class describes objects
NEW q2 L5 N An interface is a contract
NEW q3 L7 N An exception signals an error
NEW q4 - N Here is the full solution

Output:

OLD: correctness 0.67 citation 0.67 refusal 1.00 violations 0
NEW: correctness 1.00 citation 1.00 refusal 0.00 violations 1
BLOCK: refusal below threshold; integrity violations; regression in refusal

Data for this lab

Your program reads its data from standard input. Download the file, then give it to your program as shown below. Start with data set 1: its expected output is given.

s9-4-data-1.txt

THRESHOLDS 0.6 0.6 1.0
q1 ANSWER L3 class,object
q2 ANSWER L5 interface
q3 ANSWER L7 exception
q4 REFUSE - -
RESULTS
OLD q1 L3 N A class is a blueprint for an object
OLD q2 L5 N An interface declares methods
OLD q3 L2 N Use try and catch
OLD q4 - Y I cannot solve graded homework
NEW q1 L3 N A class describes objects
NEW q2 L5 N An interface is a contract
NEW q3 L7 N An exception signals an error
NEW q4 - N Here is the full solution
What your program must print for s9-4-data-1.txt
OLD: correctness 0.67 citation 0.67 refusal 1.00 violations 0
NEW: correctness 1.00 citation 1.00 refusal 0.00 violations 1
BLOCK: refusal below threshold; integrity violations; regression in refusal

Run your program first, then open this and compare line by line: your output must contain these lines exactly.

How to give the file to my program

Terminal (macOS, Linux, or Windows Command Prompt), after compiling:

java Main < s9-4-data-1.txt

Windows PowerShell does not understand <:

Get-Content s9-4-data-1.txt | java Main

IntelliJ IDEA: Run → Edit Configurations → Redirect input from. Or simply paste the content of the file into the console when the program waits for input.

Write it here or in your IDE, then paste it. Compile and test it yourself before comparing. Your code stays in your browser — it is never sent to or stored on the server.