Messy Data:
Teaching Students Early on About
the Realities of Data
Cornell College
• Small liberal arts college (1100
students)
• Mathematics and Statistics Department with
4.5 tenure track lines
• Teach on the block plan
Statistics History at Cornell
•
•
•
•
Intro stat
Probability/Math Stat
Stat 2
“New Frontiers”
• Epidemiology
• Dealing with Data: Data Manipulation, Data
Visualization, and Big Data
Data Course
• Team taught with computer scientist
• Prerequisite either intro stat or CS 1
• Focused on hands-on
• Morning was two hours of lecture
• Afternoon was two hours of computer lab
Data Course - Plan
• 1/3 of the course on each topic
• Data Cleaning
• Data Visualization
• Big Data
• Relevant computer science
fundamentals addressed in a just-intime fashion
• Use R as the software tool
Data Course - Reality
• 1/3 Data Cleaning
• 1/2 Data Visualization
• 1/6 Big Data
Daily Structure
• Morning – 2 hours
• M-Thur: Lecture
• 1 hour stat
• 1 hour CS
• Fri : Student presentations
• Afternoon – 2 hours
• Computer lab
Data Cleaning
• Simple issues
• Clearly wrong entries
• Potentially wrong entries
• Functions of a variable
Data Cleaning
• More complex issues
• Combining data sets
• Linking variable issues
• Making sure data sets are combined properly
• Different variable formatting in different data sets
Data Visualizations
• Look at published visualizations
• Discuss ways to improve published
visualizations
• Specific visualizations created:
• Stream graphs
• Tree graphs
• Maps
Big Data
• Described “big data”
• Volume
• Velocity
• Variety
• Discussed computer science issues
• MapReduce
• Hadoop
Projects
• 3 Projects
• Chapter 2 of Data Science in R: A Case Studies
Approach to Computational Reasoning and
Problem Solving by Deborah Nolan and
Duncan Temple Lang
• Twitter project
• Group project
Project 1
• Introduce students to R
• 10 years of data from the Cherry
Blossom Road Race in DC
• Lots of data cleaning
• Introduced some visualization
issues with larger data sets
• Introduced the idea of smoothing
Project 1
• Done in pairs
• Deliberately formed with one “stat” and one
“cs” student
• In class work following the steps
given for the men’s data
• Written report due for women’s data
• Includes both code and statistical report
Project 2
•
•
•
•
•
Download public tweets
Filter for a query term
Assign a sentiment score
Aggregate tweets by state
Produce geographic visualization of
data
Project 2
• Again done in cs/stat pairs
• Final report
• Required an extension of the basic lab
• Required both code and statistical report
Project 3
• Term-long 4-person group project
• First week
• Individual brainstorming about topics
• Friday morning – elevator pitches
• Second week
• Teams find data and refine goals
• Friday morning – check-in report from all
teams – class feedback provided
Project 3
• Third week
• Lab time devoted to project
• Finish data cleaning and do much of the
analysis
• Friday morning – check-in report from all
teams – class feedback provided
Project 3
• Last 3 days of class
• Finishing touches on the analysis
• Create project website
• Final presentation to both class and other
visitors
Examples of group projects
Examples of group projects
Examples of group projects
Lessons Learned
• Slower introduction to R
• Small individual assignments as we
go
• More faculty input for statistical
analysis of group projects
For more information
Ann Cannon
Department of Math and Stat
Cornell College
600 First St SW
Mt. Vernon, IA 52314
(319) 895-4461
acannon@cornellcollege.edu