Buddy uses a detector that decides whether a question is about a graded exercise. The requirement is:
On the instructor-labelled test set, recall for "graded" shall be at least R and precision at least P.
Input: first line R P (e.g. 0.95 0.80). Then lines actual predicted until the end of input, each G (graded) or N (not graded).
Compute TP, FP, FN, TN (positive = G), then print (values with three decimals):
TP=.. FP=.. FN=.. TN=..
Precision: x.xxx
Recall: x.xxx
F1: x.xxx
Accuracy: x.xxx
Requirement: PASS
or Requirement: FAIL (recall x.xxx < R) / Requirement: FAIL (precision x.xxx < P) / both reasons joined by ; , recall first, e.g. Requirement: FAIL (recall 0.900 < 0.95; precision 0.750 < 0.8) — print R and P as Java prints a double.
Conventions: if TP+FP = 0, precision is 0; if TP+FN = 0, recall is 0; if precision+recall = 0, F1 is 0.
Input:
0.95 0.80
G G
G G
G N
N G
N N
N N
Output:
TP=2 FP=1 FN=1 TN=2
Precision: 0.667
Recall: 0.667
F1: 0.667
Accuracy: 0.667
Requirement: FAIL (recall 0.667 < 0.95; precision 0.667 < 0.8)