Large Effects With Small Efforts:
What Does it Take to Leverage User Data?
Joe Zucca, Director for Planning and Organizational Intelligence, Penn Libraries
SLA, Philadelphia, April 2, 2014
Prologue
Producing Large Effects With Small Effort?
How Well Are They Managing Complexity (in the LMS)?
Count of User Type
Column Labels
Row Labels
FAC
Canvas Features
217
Access
200
Migrate content
41
Settings
29
Course content
27
Other
Assignments
Staff
Copy content
Site Request
103
2
Other (Features)
17
53
8
Account Request
1
Other
1
14
41
11
Multimedia
1
Quizzes/surveys
530
7
69
329
144
Modules
4
4
16
6
6
1
Learning Objects
6
288
13
Gradebook
Discussions
16
8
Turnitin
Grand Total
15
Announcements
2
58
Conversations/messa
ges
LTI Tools (other)
Administration
Student
Grand Total
21
2
3
5
1
163
Courseware Problem Incidents: Frequency by Incident & User Group
1
85
784
How Well Are We Managing Complexity (in the LMS)?
Average of lag
Row Labels
Administration
Column
Labels
FAC
3.0
Canvas Features
1.9
Site Request
1.4
Access
1.3
Account Request
0.0
Other
0.0
Grand Total
1.6
Quizzes/surveys
9.2
Multimedia
4.0
Gradebook
3.8
Assignments
2.4
Other
Grand Total
Migrate content
2.0
Copy5.4
content
1.8 5.0
Course content
1.6
Staff Student
4.0
Settings
0.6
Turnitin
1.0
1.8
Modules
LTI Tools (other)
0.4
27.5
1.6
0.4
2.4
1.5
0.5
1.0
3.9
1.5
1.1
Other (Features)
0.4
Announcements
Conversations/mess
ages
0.4 7.0
3.5
0.3
Discussions
0.0
3.2
3.3
E-reserves
2.2
Learning Objects
0.9
0.0
11.7
0.0
Courseware Problem Incidents: AVG resolution time by Incident & User Group
1.8
The Challenge: The Scope & Complexity of Service
 Acquisition and licensing services
 Access to massive e-content
 Discovery service





Supply chain services [local circulation & third party - ILL]
Inventory
control
What
products
and services does your library
Researchor
| Instructional
services for the university?
directly support
Courseware service (e.g., Blackboard)
Learning environments (e.g. MOOCs)




Author service
Copyright compliance service
Technology support services (e.g., desktops, digital media, scanning)
Digitization services (image production, description, curation)
offer
 Metadata service
 Data services
 Repository services (internal and public-facing digital archives)
 Researcher intelligence services (e.g., VIVO)
SpringShare
FootPrints
HR db
Relais
Analytics
Repository
Google Analytics
STAFF
Banner IDM
SFX
AEON
ARES
Fedora Repo
ILLIAD
VIVO
Canvas LMS
SOLR
log
EZPROXY
ILS
APACHE
log
log
COUNTER
BePress
A B.I. Framework
Analytics
Repository
Extract
Enterprise Products &Services
Trnsfrm
Data Mart
Resolvers
Load
Analysis
a framework for:
Extracting activity data from systems
Transforming those data into readable,
normalized formats
Loading transformed/normalized
payload into a repository
Supporting analysis through local and
collaborative dissemination channels.
https://github.com/upenn-libraries/metridoc-app
MetriDoc Tiered Architecture
Abstracts 4 key functions, exposes interfaces for interoperability
1. Extract
Target Source,
e.g. Relais,
Illiad, ILS
2. Transform
Resolution
Sources
e.g. IdM, WorldCat
Resolve
Codes & IDs
Normalize
Format
Refined
output
4. Query
Results
Document
Query Srvc
Ingest Log
Parse
3. Load
User
Interface
Data
Repo
Refined
output
Query
Document
Local
Data
Stores
Economic failure of German colonies in Africa
Using Google Fusion Tables
Papers on reaction kinetics on ethylene oxide hydration
History of marketing listerine
monetization of contemporary art especially digital art
The folklore of the supernatural in the media
Arab nationalism at the beginning of the 20th century, and relations with the Ottoman Empire.
Medicaid eligibility rules by state data
Using GIS for studying health in neighborhoods
Researching Democratic party's views on charter schools and high-stakes testing.
Health of pregnant women in prison
Locate the full list of proscribed authors under the Yuanyou ban 20thh-century China.
Pediatric Tuberculosis Caregivers Attitudes
history of Chinatown developments in the U.S./Canada
How the Illiad reflects political realities in Bronze Age Greece.
Economic effects Sykes-Picot agreement in Syria.
The effect of the Nike settlement on labor relations in the footwear industry around the world.
educational linguistics, semiotics, reflectivity and meta-language
type 1 diabetes and telecommunications
Native American slaveholders (specifically the Cherokee)
Import substitution in Brazil under Vargas
household interventions for asthma prevention in urban setting
data sets on toilets in Mozambique
The changing Confucian-influenced perceptions among Song-dynasty writers of the Kitans.
Familial lineages in Commedia del Arte
Walter Mischel and original marshmallow, gratification studies
Research on democratically elected Presidents of Brazil since the 1980s.
Why Employment Non-Discrimination Act is taking so long to pass.
Economic consequences of French colonialism in North Africa.
Raster creation in arc gis
Possible use of Shoah VHA in class about the Holocaust in Italy.
Relation between U.S. intervention in Philippines and Cuban independence.
glycogen storage disease and mortality
Putting together an annotated bibliography on the Cristero rebellion.
Researching medieval pilgrimages to Hebron
Lliterature by Christian missionaries in the Middle East during the late 19th and early 20th centuries.
Epilogue
Raising the Organizational I.Q.
The O.I.Q. NOT about
Software, IT Systems, Network Plumbing, Assessment
Plans, or Culture…
…it IS about creating an infrastructure for learning.
Analytics
Repository
Services
Analytics
Organizational
Knowledge
And What Is It We’re Learning to Do?
Heighten the observational power of our organizations.
Understand how users work, what they want, need, and expect,
and what’s expected of them
Guide the library through the morass of strategic choices…
Improve our response to change and the value of our services
“…the key requirement for institutional success is to move from
scalable efficiency to scalable learning.
Said differently, the rate of learning, innovation, and performance
improvement within the institution must match (or exceed) that of
the surrounding environment
if the institution is to survive (or thrive).”
-- John Hagel III and John Seely Brown
The New Organization Model: Learning at Scale
HBR Blog Network
Producing Large Effects With Small Efforts
MetriDoc: https://github.com/upenn-libraries/metridoc-app
PENN Libraries
Joe Zucca | zucca@upenn.edu
With the support of