THINK FIRST·CODE LATER

← Interactive Information Visualization
Chapter 14 · Week 14

Evaluation, Storytelling and Reproducibility

Before You Start: What You Must Be Able to Do

Before the questions, make sure you can: choose an evaluation method that matches the claim and the nested-model level; compute usability measures including the System Usability Scale; design a small controlled comparison and report it with a confidence interval and effect size rather than a p-value alone; recognize Simpson's paradox and confounding in your own charts; turn an analysis into a narrative visualization with message titles and annotations; communicate uncertainty; and make your visualization work reproducible.

The Big Idea

A visualization is a claim: "this view helps these people do this task better". This chapter is about showing that the claim holds — with users, numbers and honest statistics — and about communicating what the data says so that others can understand and check it. In an era where AI can generate a chart and a confident paragraph for anything, evidence, transparency and reproducibility are what make your work trustworthy and distinguish an engineer or researcher from a prompt.

Choosing an evaluation method

Match the method to the claim and to the nested-model level (Chapter 3):

Method What it shows Level Typical size
Heuristic / expert review violations of known principles (Chapters 5–6) idiom 2–5 experts
Think-aloud usability test where and why users struggle idiom/abstraction ~5 users per round (formative)
Controlled experiment which design is faster/more accurate for defined tasks idiom 12–40+ participants
Crowdsourced study perception and simple tasks at scale idiom 100s (e.g., Heer & Bostock 2010)
Field / case study whether real users get real work done abstraction, domain a few teams, weeks
Insight-based evaluation number and depth of insights in open exploration abstraction small groups
Benchmarks speed, memory, scalability algorithm —

Small formative tests early and often (a handful of users per iteration finds many major problems) beat one big test at the end. Summative comparisons need careful design.

Measures

  • Task completion (success rate), time (often skewed — report medians, or analyse log times), errors.
  • System Usability Scale (SUS): ten statements rated 1–5; odd (positive) items score (rating − 1), even (negative) items score (5 − rating); the sum × 2.5 gives 0–100. Around 68 is average; above ~80 is good. In Lab V14.1, dashboard B beats A on completion (89 % vs. 67 %), time (median 42.5 s vs. 55 s), errors and SUS (87.5 vs. 71.7) — and "locate" is the hardest task for both, which tells you what to fix next.
  • Insights and confidence for exploratory tools; adoption in the field.

Designing a comparison

  • Within-subjects (each participant uses both designs): more sensitive with fewer people, but learning carries over → counterbalance the order (half A-then-B, half B-then-A; Latin squares for more conditions) and use equivalent but different task instances.
  • Between-subjects (each participant uses one design): no carry-over, needs more participants.
  • Representative participants (city officials are not students) and tasks (from the task analysis), a pilot run, and a pre-registered analysis plan reduce bias.

Honest statistics

Report effect sizes with uncertainty, not just "p < 0.05". In Lab V14.2, 8 participants answered questions with a table and with a chart: the chart was faster by 10.6 s on average, 95 % CI [6.4, 14.9] s, paired t(7) = 5.93, effect size dz = 2.10, and all 8 participants were faster. With 4 participants and a 1.2 s difference, the CI [−6.7, 9.2] includes 0: no reliable difference — which does not prove the designs are equal. Bootstrap confidence intervals work for medians and other statistics without distribution assumptions (Lab V14.4), but they depend on random resampling — fix and report the seed to make them reproducible.

Threats to validity: small or unrepresentative samples, tasks that favour one design, novelty effects, experimenter bias, measuring at the wrong level, and multiple comparisons (test 20 metrics and one will "succeed" by chance).

Confounding and Simpson's paradox

Aggregates can reverse within groups. In Lab V14.3, app-based van routing succeeds more often than manual routing on weekdays (93.1 % vs. 86.7 %) and on weekends (73.0 % vs. 68.8 %), yet manual routing looks better overall (82.6 % vs. 78.0 %) — because the app was used mostly on difficult weekends. Any chart comparing groups should consider such confounders (weekday, weather, district size); show the breakdown, or compare like with like.

Telling stories with data

Exploration (discover) and communication (present) need different artefacts (Chapter 3). Narrative visualization (Segel & Heer, 2010) identified genres — magazine style, annotated chart, partitioned poster, flow chart, comic strip, slide show, film/animation — and a spectrum from author-driven (linear, heavy messaging) to reader-driven (free exploration). Common structures: the martini glass (an author-driven narrative that opens into free exploration), the interactive slideshow and the drill-down story; scrollytelling sequences views as the reader scrolls.

Practical rules:

  • Titles state the message ("Weekend van trips fail more often — app routing helps in both cases"), not just the topic ("Success rate by routing").
  • Annotate the important points directly on the chart (events, thresholds, the one data point that matters); highlight one series and grey the context.
  • One idea per chart, sequenced into a logical argument; remove what does not support it.
  • Show uncertainty in a form the audience understands (ranges, intervals, "between 3 and 14 µg/m³ higher"), state data sources, definitions and limitations.
  • Stories must not cherry-pick (Chapter 6): the narrative is a guided path through honest evidence.

AI tools can draft narrative text and suggest annotations; every claim must still be checked (Chapter 12), and the choice of what matters is the author's responsibility.

Reproducible visualization

A chart someone cannot regenerate is a claim they cannot check. Make your work reproducible:

  • Code, not clicks: scripts and notebooks that run from raw data to final figures ("restart and run all" must work).
  • Pinned environment: requirements.txt/environment.yml with versions (Altair 5.5 behaves differently from Altair 4).
  • Data versions: raw data kept unchanged, with source and fetch date; processed data regenerated by code; large data versioned (e.g., DVC) or archived.
  • Seeds for anything random (sampling, bootstrap, t-SNE/UMAP, k-means).
  • Provenance and disclosure: record AI assistance (tools, prompts, what was verified) and decisions (excluded data, chosen thresholds).
  • Open materials where possible: code, (anonymized) data, study materials, so others can audit and build on your work.

Ethics

Evaluations with people need informed consent, anonymity, minimal data collection and the right to withdraw (university ethics procedures apply). Report honestly — including results that did not support your design.

Industry spotlight

Product teams run A/B tests on dashboards and data products (e.g., two layouts shown to different users, measuring task success or engagement), use analytics of feature use, and keep design systems of validated chart components. Data-journalism teams (e.g., at the Financial Times or Reuters) publish code and data behind their graphics so readers can check them.

Research spotlight

Visualization research increasingly emphasizes transparent statistics (confidence intervals, effect sizes, pre-registration, open materials) after replication concerns in many fields. Evaluating AI-assisted visualization adds new questions: do users over-trust AI-generated charts and captions, and how should interfaces communicate the uncertainty of AI outputs?

Key takeaways

  • Match evaluation methods to the claim and level: expert review, think-aloud, controlled/crowdsourced experiments, field/case studies, benchmarks.
  • Measures: completion, time (medians), errors, SUS (0–100, ~68 average), insights, adoption.
  • Within-subjects + counterbalancing, representative participants and tasks, pilots.
  • Report effect sizes with confidence intervals (paired t, bootstrap with fixed seeds); a CI including 0 is not proof of equality; consider threats to validity.
  • Check for confounding and Simpson's paradox before comparing aggregates.
  • Narrative visualization: genres, author- vs. reader-driven, message titles, annotation, uncertainty.
  • Reproducibility: code, pinned environments, data versions, seeds, provenance, AI disclosure; ethics in studies.

Ready? Close the notes and practise.

30 questions. Predict the output before you check — that is the skill the exam measures.