Before the questions, make sure you can: describe the main kinds of AI support for visualization and name representative systems; explain the steps of a natural-language-to-visualization pipeline and where ambiguity arises; build guardrails that validate an AI-generated specification against the data schema; evaluate NL2VIS outputs with exact-match, component and execution accuracy and explain what each metric misses; fact-check AI-generated captions against the data; and design prompts and workflows that protect privacy and resist prompt injection.
"Show me average PM2.5 by district" is now a valid way to make a chart. Language models translate requests into code or specifications, recommend charts, answer questions about chart images and write captions. They are fast, often right, and wrong in ways that look right. This chapter treats AI as a component of a visualization system — one that must be specified, constrained, tested and evaluated like any other. The engineering pattern is simple: let AI generate, make the output checkable, check it automatically, and keep a human in charge of meaning.
What AI does for visualization today
| Capability | Examples | Typical failure |
|---|---|---|
| Chart recommendation | Voyager, Draco (rules + optimization), VizML (learned from millions of charts) | suggests common, not task-appropriate charts |
| Natural language → chart (NL2VIS) | NL4DV (VIS 2020), Microsoft LIDA (2023), Data Formulator, chat assistants with code execution, BI copilots | ambiguous requests, wrong fields or aggregations, misleading defaults |
| Code assistance | Copilot-style completion of pandas/Altair/Plotly code | deprecated APIs, silent type errors |
| Chart question answering | ChartQA benchmark (2022); multimodal models answering "which district is highest?" from an image | misreading values, especially from area/colour |
| Captions, alt text, summaries | generated descriptions and "insights" | overstated or invented claims |
Inside an NL2VIS pipeline
LIDA is a good mental model. Its stages are a data summarizer (column names, types, example values, semantic descriptions), a goal explorer (proposes questions), a visualization generator (writes chart code or specifications) and an infographer (styles the result). In general:
- Understand the data — types and meanings of columns (Chapter 2). Many errors start here: an id taken as a number, a date as text.
- Interpret the request — which attributes, which task (trend, comparison, distribution: Chapter 3), which filters ("for Lucheng"), which aggregation ("average", "total", "number of").
- Generate a specification — Vega-Lite JSON (or Altair/Plotly code).
- Render and explain.
Requests are often ambiguous: "show me the data" names no quantity; "average PM2.5" names no grouping; "busiest stations" could mean trips started, ended or both. Good systems ask a clarifying question instead of guessing (Lab V12.1 asks "which quantity?" and "by which attribute?").
Why Vega-Lite is a good target
Generating a declarative specification instead of arbitrary Python code has big advantages: the output can be parsed as JSON, validated against the Vega-Lite schema, checked against the dataset's columns and types, linted for design problems (Labs V1.3, V5.4) and executed safely (no arbitrary code). Generated Python code, by contrast, must be sandboxed before running.
Guardrails
A robust AI-chart pipeline:
- Prompt with the schema, not the data — column names, types, a few example values; this improves accuracy and protects privacy.
- Parse the output (valid JSON?).
- Validate against the data schema — every field exists (case matters:
District≠district), types match (dateis temporal,station_idis categorical), aggregations are legal (avgis not a Vega-Lite aggregate;mean/averageare) and meaningful (no mean of an id) — Lab V12.2. - Lint the design — zero-based bars, colour schemes, too many categories, dual axes.
- Execute and compare — compute the numbers the chart will show with pandas; compare with known values.
- Human review for meaning and context; record provenance (prompt, model, output, checks) and disclose AI use.
Evaluating NL2VIS
Research benchmarks pair questions with reference ("gold") specifications (e.g., nvBench for NL2VIS, ChartQA for chart question answering). Common metrics:
- Exact match — the predicted specification equals the gold one. Strict: swapping x and y in a bar chart (horizontal vs. vertical bars) counts as wrong.
- Component accuracy — mark, each channel's field/type/aggregate compared separately.
- Execution accuracy — the data the chart would show is the same. Lenient: it accepts a line chart over categories that shows the right numbers but is the wrong idiom.
In Lab V12.3, exact match is 25 % and execution accuracy 75 % on the same four predictions — the truth is in between, and human judgment of task suitability remains necessary. In user-facing products, measure task success: did users get the right answer faster?
Chart question answering and captions
Multimodal models can read chart images and answer questions or write summaries. They do better on clean charts with labels and position encodings, worse on area/colour encodings, many categories and dense charts. Generated captions and "insights" must be fact-checked: in Lab V12.4, "Lucheng fell by 23 %" is supported (−23.1 %), "Ouhai fell by 20 %" is not (−5.3 %), and "improved because of the new bike lanes" is a causal claim no rule can verify — a human must check it, and the data alone cannot support it.
Privacy and security
- Do not paste personal or confidential data into public AI services; send the schema and aggregates, or use approved/self-hosted models.
- Prompt injection through data: if raw data is sent to a model, a cell containing "Ignore previous instructions and …" can manipulate the output. Treat data as data: send summaries, escape or filter text fields, constrain outputs to a schema, and validate.
- Keep logs of prompts and outputs for reproducibility, and state AI use in reports and dashboards.
BI copilots (Power BI Copilot, Tableau AI, Looker with Gemini) generate visuals and narratives from natural language over semantic models — curated, certified definitions of metrics and relationships. The semantic model is the guardrail: the AI may only combine approved measures, which is exactly the "validate against the schema" step of this chapter at enterprise scale.
Open research questions include: evaluating NL2VIS beyond exact match (does the chart support the user's task?), handling ambiguity through dialogue, making AI-generated analyses verifiable (linking each claim to the data that supports it), studying over-reliance on AI suggestions, and accessibility — can AI descriptions make charts usable for blind readers without introducing errors they cannot detect?
Key takeaways
- AI supports recommendation, NL2VIS, code, chart QA and captions — each with characteristic failures.
- NL2VIS: data summary → intent/task → specification → rendering; ambiguous requests deserve clarifying questions.
- Prefer declarative specs (Vega-Lite) as the AI's output: parseable, validatable, lintable, safely executable.
- Guardrails: schema-only prompts, JSON parse, field/type/aggregate validation, design lint, execution check, human review, provenance.
- Metrics: exact match (strict), component accuracy, execution accuracy (lenient) — plus human judgment and task success.
- Fact-check generated captions; causal claims need humans and better evidence; protect privacy; beware prompt injection through data.
Ready? Close the notes and practise.
30 questions. Predict the output before you check — that is the skill the exam measures.