Visualization 1: Principles

The Purpose of Data Visualization

datasaurus_dozen

	dataset	x	y
0	dino	55.384600	97.179500
1	dino	51.538500	96.025600
2	dino	46.153800	94.487200
3	dino	42.820500	91.410300
4	dino	40.769200	88.333300
...	...	...	...
1841	wide_lines	33.674442	26.090490
1842	wide_lines	75.627255	37.128752
1843	wide_lines	40.610125	89.136240
1844	wide_lines	39.114366	96.481751
1845	wide_lines	34.583829	89.588902

1846 rows × 3 columns

datasaurus_dozen.groupby("dataset").size()

dataset
away          142
bullseye      142
circle        142
dino          142
dots          142
h_lines       142
high_lines    142
slant_down    142
slant_up      142
star          142
v_lines       142
wide_lines    142
x_shape       142
dtype: int64

This dataset includes multiple observations for each of the listed datasets. It’s hard to see any patterns because there are so many observations.

There 142 observations for each of 13 datasets.

Let’s look at summary statistics

selected_datasets = datasaurus_dozen[datasaurus_dozen['dataset'].isin(["away", "bullseye", "dots", "star", "dino"])]
selected_datasets.groupby("dataset").mean()

	x	y
dataset
away	54.266100	47.834721
bullseye	54.268730	47.830823
dino	54.263273	47.832253
dots	54.260303	47.839829
star	54.267341	47.839545

Interestingly, the means, std-deviations, and Pearson correlations are largely identical. And yet…

Visualize

px.scatter(selected_datasets, x="x", y="y", facet_col="dataset", 
           width=1000, height=300)

See: https://cran.r-project.org/web/packages/datasauRus/vignettes/Datasaurus.html

Same Stats, Different Graphs: Generating Datasets with Varied Appearance and Identical Statistics through Simulated Annealing

Never trust summary statistics alone. Graphics can help distill information from data.

“The simple graph has brought more information to the data analyst’s mind than any other device.” — John Tukey, cited in R4DS J. Tukey’s oft-cited quote

References: - http://www.thefunctionalart.com/2016/08/download-datasaurus-never-trust-summary.html - https://en.wikipedia.org/wiki/Anscombe%27s_quartet

Same Stats, Different Graphs: Generating Datasets with Varied Appearance and Identical Statistics through Simulated Annealing

The Foundations of Data Visualization

Data Visualization uses visual representations to help data scientists discover and present patterns in data. It takes advantage of the well-developed human visual system.

Designing an effective visualization requires that we determine:

• The purpose of the visualization by identifying:
  • the key question
  • the context
• The most appropriate visual composition using:
  • The structure of our data (e.g., types of variables)
  • How the human visual system works (e.g., visual cues)

Summary: Novartis Graphics Principles Cheat Sheet

• Purpose/Function/Strategy (see: Cheatsheet “Planning”)
  • Identify the key question (cf. Question-driven scientific inquiry)
    • Communicating vs Exploring
  • Know the context (e.g., audience, domain, …)
• Composition/Form/Tactics
  • Principles of effective graphic design (for information)
    • Review: Cheatsheet “Principles of Effective Graphic Design”.
    • Review: Cheatsheet “Selecting the right base graph” (n.b., no pie charts)
    • Review: Cheatsheet “Proximity improves association”.
  • Yau’s taxonomy (MDSR2.2 focuses on these.)
    • Visual cues: Cheatsheet “Effectiveness Ranking”.
      • Color: Cheatsheet “Color for emphasis or distinction”.
    • Coordinate system: Cartesian; polar; geographical
    • Scale: numeric (linear/logarithmic/percentage); categorical; time
    • Context: titles/labels

References: - See: Vandemeulebroeck, Figure 1 - See: Vandemeulebroeck, Figure 2 - Graphics Principles - Website: https://graphicsprinciples.github.io - Paper: https://ascpt.onlinelibrary.wiley.com/doi/full/10.1002/psp4.12455 - DSBox Visualisation Tips - Key researchers: J. Tukey, E. Tufte, W. Cleveland [& R. McGill]

Cf.: M. Vandemeulenbroecke et al, Effective Visual Communication for the Quantitative Scientist

Exploratory Data Analysis (EDA)

https://learningds.org/ch/01/lifecycle_cycle.html

Exploratory Data Analysis

EDA is the process by which a data scientist explores a dataset in a systematic ways looking for patterns. The process is iterative, focused on the data, and generally involves:

1. Asking questions about your data.
2. Searching for answers using visualization, transformation, and modelling of your data.
3. Using what you learn to refine your questions and/or generate new questions.

One additional virtue: curiosity

This is how the scientific method plays out in DS. It’s the “understand loop” in the DS diagram on slide 1.

References: - https://rstudio.cloud/learn/primers/3.1 & https://r4ds.had.co.nz/exploratory-data-analysis.html - https://en.wikipedia.org/wiki/Exploratory_data_analysis - R4DS 7: EDA uses “visualisation and transformation to explore your data in a systematic way”. - MDSR 10: EDA identifies “patterns and multivariate relationships in data”. - IDS 2: EDA analyses “analysing data sets to summarize its main characteristics”. https://ids-s1-20.github.io/slides/week-02/w2-d02-data-viz/w2-d02-data-viz.html#8 - ModernDive 5: “EDA gives you a sense of the distributions of the individual variables in your data, whether any potential relationships exist between variables, whether there are outliers and/or missing values, and (most importantly) how to build your model.”

Plots to Communicate

• Purpose
  • Question
  • Context
    
• Composition
  • Graphic Design
  • Visual Cues and scales

1. Purpose:

• To communicate the relationship between child health an over time across the world.
• Narrowly, for his world health course; generally, for everyone

2. Composition

• General principles of graphic design (see below)
• N. Yau’s taxonomy
  • Visual cues:
    • Position (Scatter plot) shows correlation
    • Color (continent) & area (population) for less important data
  • Coordinate system
    • Cartesian (x-y) for most important numerical correlation
    • Categorical (color) for continent
  • Scale: numeric (logarithmic)
  • Context: titles/labels

Analyzing Visualizations

Describe the design of each of the following visualizations.

• In the Barrel…

• Four score and seven years…

• To truncate or not to truncate

  • taxes
  • temperatures

• Snow on Cholera

• Napoleon’s long, cold Russian winter

Note that a picture can be worth a thousand words, either a thousand valuable words or a thousand malicious lies.

Notes: - OPEC and the energy crisis - Tufte’s lie factor - Dig up my old notes on this example.

• Gettysburg address
  • Useless graph
  • See: http://www.norvig.com/Gettysburg/
• Scaling examples
  • Reference: https://engineering.tableau.com/truncating-the-y-axis-threat-or-menace-d0bce66d4d08
• J. Snow’s cholera map
  • famous
• C. Minard’s visualization of Napoleon’s army in the Russian campaign, 1812-1813
  • famous
  • variables:
    • Location: x/y geographical points (see rivers and towns)
    • Army Size: line width (422K at start near Niemen; 100K in Moscow; 10K on leaving Russia)
    • Time: left-two right in brown; right to left in black.
    • Temperature: height (i.e., length) of temperature line at bottom
• Consider the scaling used in the following two visualizations.

  • Bush Tax Cuts Visualization (Fox News)

  • That climate change temperature chart that looked flat. Where did I see it?
• The text’s Challenger example
  • If we do this, keep Norman’s critique in mind (see https://jnd.org/in_defense_of_powerpoint/).

Here are some more ugly chart examples:

• Flowing Data - https://flowingdata.com/category/visualization/ugly-visualization/
• Reddit - https://www.reddit.com/r/dataisugly/
• DataVis - https://www.datavis.ca/gallery/missed.php
• (Mostly Bad) Graphics and Tables - http://users.stat.umn.edu/~rend0020/Teaching/STAT8801-resources/graphics/index.html

References: - E. Tufte, lie factor, data density - D. Huff, How to Lie with Statistics - H. Wainer, How to display data badly - https://www.wisdom.weizmann.ac.il/~zvika/course2015/announcements/WainerAmericanStatistician1984.pdf

Logistics

Quiz 1

• Quizzes in general:
  • Last 20-30 minutes on Friday
  • Some weeks will be on paper, others on Moodle
  • This week: Moodle
• Topics
  • Anything from CS 104/6/8 (since it’s a prerequisite)
  • Anything you’ve seen more than once so far (e.g., exercises, slides, reading, …)
• Paper notes allowed
• Memorization of syntax not required, but concepts behind the syntax are required

Other Assignments

• Midterm Project first milestone posted this week, due next week
• Exercise 1
• Reading 3