Lecturing with Digital Ink
Richard Anderson
University of Washington
Lessons learned from the
Classroom Presenter project

Classroom Pedagogy


HCI


Teaching with ink
Ink based presentation
Multimedia

Analysis of lecture artifacts
Classroom Presenter


Integration of slides and digital ink
using Tablet PC
Key ideas:



Ink overlay on images
Distributed application
Many other systems also support ink
and slides
Presenter Features






Tablet PC Ink
Multiple Colors
Highlighter
Stroke Erase
Page Erase
Undo







Slide Minimize
Whiteboard
Multiple Decks
Filmstrip Navigation
Slide previews
Ink Export
Instructor notes
Classroom Presenter as a
distributed application


Designed as distributed
application for distance
learning
Enables many scenarios

Mobility


Sharing materials with
students


Walking and talking
Note taking
Classroom interaction

Student submissions
Deployments




Estimated use in at least 100 courses
Wide use inside of computer science
Push for adoption outside of CS
Lecture archives from UW Professional
Master’s Program

Several hundred hours of recorded audio,
video, and ink.
Distance Learning Classes
Ink based pedagogy
…in the winter of 1813 & '14 … I attended a
mathematical school kept in Boston…On
entering his room, we were struck at the
appearance of an ample Black Board
suspended on the wall, with lumps of chalk
on a ledge below, and cloths hanging at
either side. I had never heard of such a
thing before. [Samuel J. May, 1855]
Mediating artifact

Traditional lecture


Speaker
Display surface




Technologies for display


Information
Shared context
Images
Whiteboard, overhead, data projector, …
Importance / use of display varies
Instructional practices

Inking with traditional slides



Digital ink used to augment slides
Whiteboard style or marking on text
Premeditated use of ink

Slides designed to include ink
“Typical ink usage”
Best practices

Focus on clarity


Plan for ink usage


Design with ink in mind
Use ink to convey meaning


Legibility, Use of space, Color choice for
contrast, Avoid clutter
Attentional ink, color correspondence
Take advantage of the form factor
Planning for ink usage
Ink use in presentation

Cognitive load


Only limited attention available for
computer while lecturing
Linkage with speech

Close tie between ink and speech
Cognitive load

Limited feature use




Even color change unusual
User interface must be simple (and robust)
Cannot give feedback to user
Many actions appear to minimize mental
effort


Color change only for contrast
Reliance on screen erase
Understanding Attentional
Marks

Properties





Brief, simple markings
Occur with speech
Augment meaning of speech
Ad hoc form
Is there a linguistic context in which to
understand these marks?
Spontaneous Hand Gestures
Spontaneous Hand gestures [McNeill]:



are synchronous w/speech
are co-expressive w/speech
lack standard of form
Attentional marks share these properties.
Gesture Types: Iconic
Gesture Types: Deictic &
Cohesive
Analysis of digital ink



Understand ink usage
Motivation: inform development of ink
based applications
Archiving


Search, Summarization, Transcription
Lecture based

Improved rendering, note taking,
accessibility
Ink classification



Textual
Diagrammatic
Attentional
% of strokes
B
C
% of episodes
B+C
B
C
B+C
Attentional
49
53
51
77
74
76
Diagram
9
7
8
8
8
8
Writing
41
38
40
14
16
15
Other
1
2
1
2
2
2
Coding of six hours of lecture
Goals




Understand usage “in the wild”
Cannot expect lecturers to modify
behavior
Determine opportunities for automatic
analysis
Identify challenges
Methodology


Study of recorded classes
Best data set: Professional Master’s
Program



Distance courses
Audio, Video, Ink archives
HCI, Compilers, Programming Languages,
AI, Transaction Processing
Attentional ink

Problem – content
matching


Identify slide content
referred to by ink
Study


Implement basic
algorithms to match
attention marks to slide
content
Compare results with
human coders
Attentional ink

Determine the lecturer’s intent:

Determine level to parse the content
Attentional ink

Challenges


Recognition of attentional ink on text
Difficult example:
Handwriting

How well does handwriting recognition
work on “typical” instructor writing?

Domain has many challenges
Recognition Study




Studied isolated
words/phrases written
on slides
Removed non-textual
ink
Fed through the
Microsoft Handwriting
Recognizer
No training
Recognition Examples

The Good:

The Bad:

The Ugly:
Handwriting Reco Results
Exact
Alternate
Close
None
1 (6%)
0 (0%)
1 (6%)
Prof. B 146 (59%) 26 (10%)
6 (2%)
71 (29%)
Prof. C 18 (42%)
5 (11%)
1 (3%)
19 (44%)
Prof. D 262 (61%) 45 (11%)
9 (2%)
111 (26%)
2 <(1%)
58 (11%)
18 (1%)
260 (21%)
Prof. A 16 (88%)
Prof. E 408 (79%)
Total
46 (9%)
850 (68%) 123 (10%)
Joint Writing and Speech
Recognition


Can we use handwriting recognition with
speech recognition together to improve
accuracy?
Co-expression of ink and speech

Are written words spoken as well?

Can speech disambiguate handwriting?

Can handwriting disambiguate speech?
Examples

Difficult for Speech and Ink Recognition

Difficult Written Abbreviations

Speech/Ink Used to Disambiguate Ink/Speech
Experiment





Examined instances of isolated word writing
Selected word writing episodes at random
but uniformly from the various instructors
Generated transcripts manually from the
audio
Checked whether the instructor spoke the
exact word written
Measured the time between the written and
spoken word
Speech/Text Co-occurrence
Results
Exact
Approx
None
Simul
A
B
C
1 (100%)
9 (75%)
9 (82%)
0 (0%) 0 (0%) 1 (100%)
3 (25%) 0 (0%) 12 (100%)
2 (18%) 0 (0%) 10 (91%)
D
E
12 (86%)
9 (56%)
2 (14%) 0 (0%)
7 (44%) 0 (0%)
Total 40 (74%) 14 (26%) 0 (0%)
0-2s
> 2s
0 (0%)
0 (0%)
1 (9%)
0 (0%)
0 (0%)
0 (0%)
10 (71%) 4 (29%) 0 (0%)
7 (44%) 4 (25%) 5 (31%)
40 (74%) 9 (17%)
5 (9%)
Activity Recognition

Identifying slide corrections
Example Results
Diagrammatic ink



How do instructors use diagrams
Basic legibility
Observed behaviors


Diagram phasing
Locality of expression
Typical diagram



Basic, irregular
shapes
Difficult labels
Attentional ink
More examples
Zipf diagram
Stroke order
Diagram phasing
More phasing
Locality in diagrams
Separate wins indicated
together
Top arrows: “Not there”
Summary

Pedagogy with ink


Presentation with ink


How is ink used in conjunction with content
and speech to express information
Low attention task
Analysis of ink usage

Extracting meaning from archived lectures
Resources

cs.washington.edu/education/dl/presenter/



Software Downloads
Papers
Contact info



Richard Anderson, anderson@cs.washington.edu
Ruth Anderson, ruth@cs.virginia.edu
Craig Prince, cmprince@cs.washington.edu