THINK FIRST·CODE LATER

← All labs

Evaluate NL2VIS outputs: exact vs. execution accuracy

Problem

Input: a small CSV dataset, a line ---, then one JSON object per line: {"q": question, "gold": spec, "pred": spec} (Vega-Lite specs with mark and encoding).

For each pair: components = the mark plus every channel present in either spec; a channel matches if (field, type, aggregate) are equal; the mark matches if equal. Exact = all components match. Same data = executing both specs gives the same table: group the data by the fields of the non-aggregated channels (sorted names), apply the aggregate of the aggregated channel (average = mean; count counts rows), and compare the sorted rows (values rounded to 6 decimals); a spec that cannot be executed counts as different.

Print average pm25 by district: components 1/3, exact no, same data yes per pair and finally exact-match accuracy 1/4 = 25%; execution accuracy 3/4 = 75%.

Data for this lab

Your program reads its data from standard input. Download the file, then give it to your program as shown below. Start with data set 1: its expected output is given.

v12-3-data-1.txt

district,pm25,day
Lucheng,40,Mon
Lucheng,50,Tue
Ouhai,30,Mon
Ouhai,34,Tue
---
{"q": "average pm25 by district", "gold": {"mark": "bar", "encoding": {"x": {"field": "district", "type": "nominal"}, "y": {"field": "pm25", "type": "quantitative", "aggregate": "mean"}}}, "pred": {"mark": "bar", "encoding": {"y": {"field": "district", "type": "nominal"}, "x": {"field": "pm25", "type": "quantitative", "aggregate": "mean"}}}}
{"q": "total pm25 by day", "gold": {"mark": "bar", "encoding": {"x": {"field": "day", "type": "ordinal"}, "y": {"field": "pm25", "type": "quantitative", "aggregate": "sum"}}}, "pred": {"mark": "bar", "encoding": {"x": {"field": "day", "type": "ordinal"}, "y": {"field": "pm25", "type": "quantitative", "aggregate": "sum"}}}}
{"q": "readings per district", "gold": {"mark": "bar", "encoding": {"x": {"field": "district", "type": "nominal"}, "y": {"type": "quantitative", "aggregate": "count"}}}, "pred": {"mark": "bar", "encoding": {"x": {"field": "district", "type": "nominal"}, "y": {"field": "pm25", "type": "quantitative", "aggregate": "sum"}}}}
{"q": "average pm25 by district (line)", "gold": {"mark": "bar", "encoding": {"x": {"field": "district", "type": "nominal"}, "y": {"field": "pm25", "type": "quantitative", "aggregate": "mean"}}}, "pred": {"mark": "line", "encoding": {"x": {"field": "district", "type": "nominal"}, "y": {"field": "pm25", "type": "quantitative", "aggregate": "average"}}}}
What your program must print for v12-3-data-1.txt
average pm25 by district: components 1/3, exact no, same data yes
total pm25 by day: components 3/3, exact yes, same data yes
readings per district: components 2/3, exact no, same data no
average pm25 by district (line): components 1/3, exact no, same data yes
exact-match accuracy 1/4 = 25%; execution accuracy 3/4 = 75%

Run your program first, then open this and compare line by line: your output must contain these lines exactly.

How to give the file to my program

Terminal (macOS, Linux, or Windows Command Prompt), in the folder that holds both files:

python my_lab.py < v12-3-data-1.txt

Windows PowerShell (the default terminal of VS Code on Windows) does not understand <:

Get-Content v12-3-data-1.txt | python my_lab.py

PyCharm: Run → Edit Configurations → tick Redirect input from and choose the file.

Jupyter: put this at the top of the cell that holds your program:

import sys
sys.stdin = open("v12-3-data-1.txt", encoding="utf-8")

Write it here, in Jupyter or in your IDE, then paste it. Run it on the sample input yourself before comparing. Your code stays in your browser — it is never sent to or stored on the server.