Educational Data Sciences
and the Need For
Hermeneutic Principles
An Interdisciplinary Perspective
Educational Data Sciences
and the Need For
Hermeneutic Principles
Applied Epistemology
An Interdisciplinary Perspective
Educational Data Sciences
and the Need For
Hermeneutic Principles
Applied Epistemology
Interpretive Skills
An Interdisciplinary Perspective
Educational Data Sciences
and the Need For
Interpretive Skills
An Interdisciplinary Perspective
Some Driving Questions
• What counts as good learning analytics?
• What kind of profession will data sciences
be?
• What are its ancestor, sister, and adjoining
disciplines?
• Which kinds of skills and dispositions are
important for preparing future practitioners
and scholars?
Our Sociotechnical Thesis
• Data exist inside a social context;
shaped by and shaping that context.
Our Sociotechnical Thesis
• Data exist inside a social context;
shaped by and shaping that context.
• Interpretation is not technical. It is
itself socially situated with goals,
predispositions/ biases, and norms.
Our Sociotechnical Thesis
• Data exist inside a social context;
shaped by and shaping that context.
• Interpretation is not technical. It is
itself socially situated with goals,
predispositions/ biases, and norms.
• Professional communities have
developed valuable ways to reason
from imperfect evidence. We can
leverage/translate them to this new
sociotechnical terrain.
Overview
1. Quantitative shifts in evidentiary artifacts (a
digital ocean) in education
2. Qualitative shifts in educational focus
3. Some contributing/relevant disciplines
4. How to approach analysis, what kind of
science/craft/skill/briciolage, etc. is it?
QUANTITATIVE AND QUALITATIVE
SHIFTS IN EDUCATIONAL EVIDENCE
Dramatic Growth in Artifacts
1850s
Computing
Technology
Classroom
Technology
Assessment
Technology
1900 1910 1920 1930 1940 1950 1960
1970
1980
1990
2000
2010
Cloud
Technology
Services
Tabulating Technology
Central “Mainframe“
Computing
Digital
Classroom
Technology
Analog Paper-based (Textbooks, worksheets, and manual classroom tools)
Distributed
Integrated Assessment
Systems
Traditional fixed response, short
task assessments
The Digital Ocean
•
•
•
•
•
•
•
•
•
•
Test scores
Interim assessments
In class, formative assessments
Growth models
Student collaboration
Conversation records from
classroom talk and online tools
Student work, including rich and
multimodal demonstrations of
knowledge and competency
(essays, presentations, etc.)
Records of after-school
experiences
Records of informal learning
Activity traces from digital media
(in school, out of school, etc.)
•
•
•
•
•
•
•
•
•
•
•
Demographics
Student-teacher relationships (TSDL)
School improvement plans/goals
Classifications (ex: proficiency
groups)
Video records of teaching
Annotated/evaluated records of
teaching
Teacher evaluations
Individual Education Plans (IEPs) and
personalized learning maps
Geospatial information
(mapping and trends)
Attendance and rosters (more
important than you think!)
FERPA/privacy blocks
Studying Oceans
Studying Oceans
Studying Oceans
Structures &
Interrelationships
Variations in
Affordance
Diachronic/Change
Processes
Qualitative Shift in Emphasis
Institutional Center
Individual Student Nexus
Social
Networks
&Teams
Learning
Communi
ties.
Mobile
Technology
Families
Open Ed.
Resources
Institution Focus
Evidence and
Transparency
Social
Networks
Learning
Networks
Teacher Control
Expert
Sources
Related to the Educational Data Movement
Qualitative Shift in Emphasis
•
•
•
•
Critical thinking
Information literacy
Reasoning
Innovation
• Intellectual openness
• Work ethic and
conscientiousness
• Positive core selfevaluation
•
•
•
•
Flexibility
Initiative
Appreciation for
diversity
Metacognition
Interpersonal
•
•
•
•
Communication
Collaboration
Responsibility
Conflict resolution
• Cognitive processes
and strategies
• Knowledge
• Creativity
Intrapersonal
• Teamwork and
collaboration
• Leadership
Digital Mediation
Cognitive
Artifacts
Qualitative Shift in Emphasis
• Danny Hillis story;
Oscon July 2012
Black Boxes Model
Qualitative Shift in Emphasis
• Danny Hillis story;
Oscon July 2012
Black Boxes Model
Sociotechnical way of
thinking about an
educational system
Explicit and Interrelated Components Model
THE DATA SCIENCES
Six Adjoining Disciplines
Educational Data Sciences
Education
Data
Sciences
1. A new field with growing
interest from leading
universities, foundations,
USED
2. Journals, conferences,
and programs now
emerging
3. What is the disciplinary
focus, what counts as
rigor and success?
Statistical Data Analysis
Statistical
Data
Analysis
Education
Data
Sciences
• Much of the digital ocean is
compatible with statistical
analysis.
• Exploratory data analysis (ex:
Tukey with satellite data in 70s
asked many questions that are
being asked today about “big
data”
• Already established (entrenched)
in educational power structures
• Can produce strong claims
Learning Technology
Statistical
Data
Analysis
Education
Data
Sciences
Classroom/
Learning
Technology
• This is where the
data we want most
often come from…
• This area is seeing
an explosion in
media/learning
resources and
classroom
management tools
Learning Sciences
Statistical
Data
Analysis
Education
Data
Sciences
Classroom/
Learning
Technology
Learning
Sciences
• What does the data
mean for multimodal,
sociotechnical learning?
• How do socio-cultural
and cognitive theories
influence and be
informed by data
technologies?
• A design science for
educational practice with
iterative experiment,
evaluate, refine process
Information Sciences
Statistical
Data
Analysis
Classroom/
Learning
Technology
Education
Data
Sciences
Information
Sciences
Learning
Sciences
• Visualizations and
Human Computer
Interface
• The information
architectures that
undergird data systems
• Codes, classifications
• Infrastructures and
boundary objects
• Media centers and
educational resources
Organization/Management Sciences
Statistical
Data
Analysis
Classroom/
Learning
Technology
Education
Data
Sciences
Organization
& Mgmt
Sciences
Information
Sciences
Learning
Sciences
• Education is full of
processes that can be
designed
• Blended learning models
are essentially restructuring of
organizational practices
• Inter-organizational
functions are changing:
• States-districts
• Special education
Educational Data Sciences
Statistical
Data
Analysis
Decision
Sciences
Classroom/
Learning
Technology
Education
Data
Sciences
Organization
& Mgmt
Sciences
Information
Sciences
Learning
Sciences
• Established field that
uses large bodies of
information to support
organizational decisions
• As the volume and
quality of educational
data increase, more
situations where
decision sciences can be
applied will emerge.
THE DATA SCIENCES
The Seventh and Generative Discipline
Computer Science and EDM
Computer Science and EDM
Computer Science and EDM
REASONING FROM DIGITAL AGE
EVIDENCE
Approaching Digital Age Data Analysis
•
•
•
•
What counts as rigor and success?
Which parts of what disciplines are needed?
What methods are best?
What kinds of processes will make good,
great, and poor educational data analysts?
• How much of the requirements are technical
versus attitudinal ?
Core Principles
1. All analytic processes are socially situated and
iterative
2. Data is a mediational tool in an iterative process
of discovery
3. Data is an imperfect lens for context and for
interactions within that context
4. Organizational/systems thinking helps expand
the reach of educational data science
5. Ethical as well as legal considerations are
important.
Three Research Traditions
Evidence
Centered
Design (ECD)
Exploratory
Data Analysis
(EDA)
Linked Activity
Systems
Framework
(LASF)
Some Insights from ECD
• Arguments and Layers
• Domain Analysis: What
is important in the
domain?
• Domain Modeling: The
Structure of Assessment
Arguments
• Student, Evidence, and Task
Models
Some Insights from CHAT/LASF
District Data
Warehouse
Engeström Cultural
Historical Activity
Theory (CHAT)
Traditional Model
Blended Model
•Teacher and school
characteristics
•Home and
demographic data
Elem.
Teachers
Teachers
Schools
Teachers
Teachers
Teachers
Teachers
Teachers
Teachers
Teachers
Teachers
Teachers
Teachers
Teachers
Teachers
Teachers
Teachers
Teachers
• Budgets and
operating costs
•Student and parent
surveys
•Census and geospatial data
•Comparable data from SLDS
District
Analytics
•Programmatic Evaluation
•Quality of online tools
•Professional
Development
•School Feedback
•Student Performance Data
•Demographics
•Rosters and assignments
•Logs from technology
tools
Piety and Behrens Linked Activity Systems Framework
•
Data analysis generally involves more than one activity system.
•
The technology plays a linking/mediating role across contexts as well as
within
•
Framework allows for conceptualizing privacy/authorized access space
Some Insights from EDA
Underlying Heuristics (4Rs)
• Revelation (graphics)
• Residuals (models)
• Resistance /robustness
• Re-expression: scale
Broader Meaning
• What do we “see” in the data?
• What in our data fits/does not fit
with our emergent model?
• Do we have a summary that is
not easily fooled by unusual
distributions/examples
• How do the explanations we see
apply more broadly.
What Kinds of Skills/Aptitudes?
• Broad fluency with a range of
qualitative/quantitative methods
• Ethics, privacy, and confidentiality (FERPA+)
• Technology accumen
The Educational Data Movement
• Systemic viewpoint: across
silos and interorganizational
understanding
Though a famous and successful statistician, Tukey wanted to create a field that dealt with all data, even when it came in such
poor shape that it was not amenable to statistical analysis. He called it “data analysis” and created the field called “Exploratory
Data Analysis”. My undergraduate degree was in Psychology and Philosophy. I thought if I knew the logic of how we know things
(epistemology) and understood the human lens through which all perception and thought occurs (psychology) I would have the
fundamental layers of knowing from which to acquire more knowledge. After serving as a social worker and studying special
education, I sought my Ph.D in Educational Psychology with a cognate called “Measurement, Statistics & Methodological Studies”. I
would approach it as applied epistemology: How do we learn from data?
When I discovered Tukey’s writings I knew I had found the right place. I conducted psychological studies on perception of
statistical graphics and wrote about the logical foundations of data analysis. When I wrote such a chapter called “Data and Data
Analysis” [6] people told me it was a silly title – data wasn’t a subject, it’s only a piece of the background to other sciences.
Philosophy is concerned with understanding meaning and the application of logic. The philosopher asks What do we mean by
‘data’? What do we mean by ‘analysis’? If data are symbols that point to elements in the world, what kind of logic do we need to
understand that linkage? Like very good scientists, philosophers question the obvious. Such questioning may not be essential for
what you do today, but it may open the door to do new ways of thinking you never imagined.
The successful learning analyst will avoid two common errors: Failure to understand the context and failure to become intimately
familiar with the data.
1. The first error is caused by lack of contextual knowledge. Studying the learning sciences, education, and related disciplines will
help.
2. The second is error is caused by a substitution of complex statistical or computational models for detailed mental models. We
only build computational models or display to help our mental models.
Question the assumptions of your work deeply. It is important that analysts understand their work is about “revelation” or
“unveiling” the reality of the world. It is a special (at times prophetic) role in society and should be taken very seriously.
Do not think of data science as a set of techniques but as a collection of viewpoints (epistemic positions) and habits of mind.
To undertake good visualization we need to know the techniques of data display, but also the psychology of perception, the
anthropology of semiotics, the mathematics of fluctuation and the philosophy and art of aesthetic engagement. We will always
Drivers of Educational Data Mining
• Personalized Learning
• College Going
• Human Capital
Big Question
•
•
•
•
What is the new role for the teacher?
Privacy
Getting designation as school vendors
Funder issues (12-24 months 2,3,5 yrs)
• Speak Gate-ish
• Issue of interoperability…
• Continuous Improvement from ECD
• Activity theory tensions
• Activity systems in coherence/contrast
• Innovations that help resolve Activity System
tensions (Engestrom, end of life care).
• More data about the context