Before the questions, make sure you can: choose a strategy when there are too many items (filter, aggregate, bin, sample) and say what each hides; detect and fix overplotting; explain why random sampling can delete exactly the outliers you care about; choose views for many attributes (SPLOM, parallel coordinates, table heat maps); run and interpret PCA (standardization, explained variance, loadings); read t-SNE/UMAP plots without over-interpreting cluster sizes and distances; and use k-means clustering as a form of aggregation, choosing k with care.
Every technique so far assumed the data fits on the screen. Real data does not: millions of trips, thousands of sensors, dozens of attributes, embeddings with hundreds of dimensions. Reduction — filtering, aggregating, binning, sampling, projecting — is unavoidable, and every reduction hides something. The skill of this chapter is choosing what to hide on purpose, telling the reader what was hidden, and checking that the patterns you see are in the data rather than in the reduction. AI pipelines increasingly hand you reduced views (embedding maps, cluster labels); you must know how to read and question them.
Too many items
| Strategy | Example | What it hides |
|---|---|---|
| Filter | only March, only Lucheng | everything else (context) |
| Aggregate | daily means, district totals | variability, outliers, individual items |
| Bin | 2D histogram / hexbin of trips by distance × duration | positions inside bins |
| Sample | 1 % of readings | rare items — often the important ones |
Overplotting: with 200,000 points, a scatter plot becomes a black blob — density is invisible, outliers hidden among thousands of overlapping marks. Remedies, by size (a rule of thumb): under ~1,000 points a plain scatter; up to ~10,000 transparency (opacity=0.1), smaller marks and jitter for discrete values; beyond that 2D binning (heat map of counts, hexbins) or datashader, which aggregates millions or billions of points into pixels before drawing.
Sampling is dangerous for outliers. In Lab V11.4, a 10 % random sample of 200 readings kept none of the 4 outliers: the sample's maximum is 45 µg/m³, the true maximum 140. If the task is discover outliers, sample the normal data but keep every outlier (stratified or outlier-preserving reduction). Means are more robust to sampling than extremes — but even they shift (36.1 vs. 34.4).
Too many attributes
- Scatterplot matrix (SPLOM): all pairwise scatter plots in a grid (
alt.Chart(df).mark_point().encode(alt.X(alt.repeat("column"), type="quantitative"), alt.Y(alt.repeat("row"), type="quantitative")).repeat(row=cols, column=cols)). Good up to ~6–8 attributes. - Parallel coordinates (Inselberg): one vertical axis per attribute, each item a polyline. Shows correlations between adjacent axes (parallel lines = positive, crossing = negative) and clusters; axis order matters; clutters with many items (use transparency, brushing).
- Table heat map: items × attributes, cells coloured by standardized value, rows ordered by a cluster — compact for dozens of attributes.
Dimensionality reduction
Project items from many attributes to 2 dimensions for a scatter plot:
- PCA (principal component analysis): linear; finds orthogonal directions of maximal variance. Report the explained variance of each component and read the loadings (weights of the original attributes). Standardize first when attributes have different units: in Lab V11.2, without scaling PC1 explains 99.9 % of variance and is simply "trips per day" (the attribute with the largest numbers); after standardization PC1 (84 %) contrasts busy central commuter stations with long, casual, peripheral trips, and PC2 (14 %) captures distance from the centre.
- MDS (multidimensional scaling): preserves pairwise distances as well as possible.
- t-SNE and UMAP: non-linear; excellent at revealing local neighbourhoods and clusters (e.g., in image or text embeddings), but cluster sizes and distances between clusters are not meaningful, results depend on parameters (perplexity, number of neighbours) and random seeds, and apparent clusters can appear in random data (Wattenberg et al., "How to use t-SNE effectively", 2016). Run several settings, colour by known labels, and never measure distances on a t-SNE plot.
Embedding plots are now common in AI work: language-model embeddings of documents, image features, user vectors. A 2D map of 10,000 customer embeddings is a reduced, distorted view — useful for exploration, dangerous as evidence.
Clustering as aggregation
Clustering groups similar items so you can visualize groups instead of individuals: k-means partitions standardized data into k groups minimizing the within-cluster sum of squares (inertia). Choosing k: the "elbow" where adding clusters stops reducing inertia much (Lab V11.3: 32.0 → 6.7 → 2.3 → 1.0 for k = 1…4 — a clear drop at k = 2, a smaller one at 3) — plus domain sense: k = 2 separates central commuter stations from leisure/peripheral ones; k = 3 isolates the airport. Cluster labels are arbitrary numbers, and k-means always returns k clusters even if none exist — validate with domain experts and stability checks.
Reporting reduced views honestly
State what was reduced and how: "daily means of 1.2 million readings", "2D binning, 50 × 50 cells", "PCA on standardized attributes, PC1 + PC2 = 98.5 % of variance", "t-SNE, perplexity 30, distances not meaningful", "10 % sample plus all outliers". Without that sentence, readers cannot judge what the chart can prove.
Companies with billions of events (ride-hailing, telecom, web analytics) rely on pre-aggregated tables, approximate query engines and tile-based rendering (datashader, deck.gl, Kepler.gl) to keep dashboards interactive. Data scientists use embedding projectors (TensorBoard, Nomic Atlas) to explore model data — with the same caveats about t-SNE/UMAP distances.
Visual analytics research studies how to scale interaction to big data (progressive visualization that refines results while computing), how trustworthy dimensionality-reduction maps are (quality metrics such as trustworthiness and continuity), and how to explain what an AI model has learned by visualizing its internal embeddings (Chapter 13).
Key takeaways
- Too many items: filter, aggregate, bin, sample — each hides something; say what.
- Overplotting: transparency/jitter (≤ ~10k points), 2D binning/hexbin/datashader beyond.
- Random sampling can remove all outliers; keep outliers when they matter.
- Many attributes: SPLOM, parallel coordinates (axis order matters), table heat maps.
- PCA: standardize, report explained variance and loadings. t-SNE/UMAP: local structure only; sizes and inter-cluster distances meaningless; parameters matter.
- k-means as aggregation: standardize, choose k with elbow + domain sense, labels are arbitrary.
Ready? Close the notes and practise.
30 questions. Predict the output before you check — that is the skill the exam measures.