An Outcome Study on the Learning Programs
at the Novartis Knowledge Center
9th Northumbria International Conference
on Performance Measurement in Libraries and Information Services
York, UK. Aug. 22nd – 26th, 2011
Leifang He,
Novartis Knowledge Center, New Jersey, USA
Introduction (1) - Novartis, AG
 Novartis is a global company with more than 100,000
employees in 140 countries. Headquarter is in Basel,
Switzerland
 Novartis offers a wide range of health products through
Pharmaceutical, Vaccines and Diagnostics, Sandoz and
Consumer Health divisions
2
Introduction (2) - Novartis Knowledge Center (NKC)
 NKC is the only library & information service hub in the
global Novartis community.
 NKC has 5 physical locations- Basel in Switzerland, New
Jersey, Massachusetts, California in USA, and Horsham
in United Kingdom
 Novartis associates can access e-library services 24 x 7
from anywhere in the world.
 NKC collection consists of 3000+ electronic journals,
4000+ e-books and nearly 200 databases
3
The goal of NKC learning programs
enable users to better utilize the e-library for their job
performance as learning has such an important role in
Knowledge Economy
Learning
contributes to
building the
competency
Essential
competency of
best knowledge
obtaining
Actionable
Knowledge for
execution
Business
growth in
Knowledge
Economy
[1] the definition, http://www.businessdictionary.com/definition/knowledge-economy.html
[2] Amy C Edmondson, “The Competitive Imperative of Learning”, Harvard business Review, July-August 2008
4
Two types of NKC learning programs
 NKC Orientation for new hires
• This provides an overview of NKC information resources for
different job functions and disciplines.
 NKC database (DT) targeted learning
• This is customized for specific customer groups for their specific
information needs
5
Methodology (1) - The Kirkpatrick's model
for evaluation effectiveness
Level 1- Reactions, which answers to the course perceptions e.g. reactions to the improvement of a
learning program
Level 2 –
 Learning, which measures advancing skills, knowledge or attitude
Level 3 –
 Transfer, which measures learners’ behavior change due to a learning program, the newly
acquired knowledge and skills (or attitude) are being used
Level 4 –
 Results, which measures outcomes/business impacts
http://coe.sdsu.edu/eet/Articles/k4levels/start.htm
6
Methodology (2) – Hypothesis and
Key Performance Indicators (KPIs)
Hypothesis on after engaging in the NKC learning
programs
 Users come back more often to utilize NKC resources and to search them more
efficiently
 The number of users, who rely on information for decision making, correlates
positively with an improvement in their job performance
 KPIs used to measure against the hypothesis
 Information Searching efficiency (Level 2.skills, knowledge)
 Job /business impact (level 4, outcomes)
 Behavior change (level 3, attitude)
 Improvement in learning programs (level 1, reactions)
7
Methodology (3) - NKC survey questionnaire design

Sample core questions reflecting
 Attitude –
 “Before this session, on average, I used this database”
 “After this session, on average, I am likely to use this database”
 Behavior –
 After this session, on average, I am likely to use this database”
 Outcome –
 “In my job, I rely on information for decision making”
 This session improved my searching skills in this database”
 After this session, on average, I am likely to use this database”
One to many relations—one question with multi answers
 “My overall experience of the session was positive”





Strongly agree
Agree
Neutral
Disagree
Strongly disagree
8
Methodology (4) – Survey data
 507 of 671 associates, who participated the NKC learning
programs in 2009, were surveyed via a web based survey
 Response rate is 70.87% (360/507)
 The collected data were mapped to the participants’
business functions, divisions and geographical locations
for result analysis
9
Key findings and discussions (1) - delivery types
 40.2% NKC learning programs were delivered via
distance learning. It indicates that distance learning is the
most utilized way of delivering NKC global learning
programs
10
Key findings and discussions (2)- Learner
Division distribution
 The learner are from more than 30 locations across 20 countries.
Research has the most learners (39%) and followed by Development
(17%), Marketing (11%) and Generic (4%)
11
Key findings and discussions (3) - Overall
positive experience
 From NKC Orientation group, 93% (152/163) responded
to Agree (or Strongly Agree) to “My overall experience of
the session was positive” and 6% (10/163) claimed
neutral
 From DT group, 82% (145/176) responded to Agree (or
Strongly Agree), and 16% (28/176) claimed Neutral
12
Key findings and discussions (4) - Improve
information seeking efficiency and effectiveness
 NKC learning programs improve the participants’ information seeking
efficiency and effectiveness.
• 90%(147/164)of NKC Orientation participants learned how to find useful information
for their job needs. They claimed that they would spend 66%(3.14/4.76) of their
information gathering time in using NKV information resources (see the table below)
• 88% (124/146) of DT group indicated that they have improved their information
resource searching skills
13
Key findings and discussions (5) - Generate
business impact and job performance
 NKC learning programs improve generate business impact and
improve job performance
• 93% (290/311) of the participants claimed that they need information for their job
performance. It means that NKC learning programs have an excellent potential base
to drive business impact
• 90% (148/164) of the Orientation participants responded that NKC resources will be
useful for their job. And 85% (124/146) of DT participants responded that the NKC
recourses presented to them were relevant to their jobs. These imply a strong
alignment between NKC information sources to company’s objectives
• DT participants responded that they would be using NKC resources more
significantly (P<0.01) after the learning programs (See table below)
14
Key findings and discussions (6.1) – higher business
impact (results) on DT group compared to the Orientation group
 On average, DT participants spent significantly more time per week,
in the past 12 month on information gathering to the Orientation
participants (P<0.001, see table below) This implies that NKC should
engage more with DT groups as potentially, they have more
information needs
15
Key findings and discussions (6.1) – higher business
impact (results) on DT group compared to the Orientation group
 There is a significantly higher potential business impact from DT
learning programs compared to NKC Orientation (p<0.05) as DT
group rely on more information for their decision making (see Table
below)
Q. “ I rely on information for decision making”
16
Key findings and discussions (6.3) - higher business
impact on DT group compared to the Orientation group
 The amount of DT group time spending on NKC resources correlates
positively to their job performance (R=0.90). The more NKC provide
the learning programs, the more business impact it produces (See Figure
below with Spearman Non-parametric test on Co-relation)
Positive co-relation between on dependency on information for
decision making and using NKC resources (R=0.90, Nx=153; Ny=140)
120
Responses
100
80
After this session, on
average, I am likely to use
this...(Y)
60
40
Database/Targeted rely on
info for decision making (X)
20
0
Not at all
Less
than
once a
month
Once a Several Several
month
times times a
a month week
17
Conclusions
 The study results disclosed that the goal of NKC learning
program was achieved and the hypotheses were
confirmed by the changes in behavior, attitude and
results:
 NKC learning programs improve the participants’ information
seeking efficiency and effectiveness
 NKC learning programs generate business impact and improve job
performance
 NKC drive higher business impact in Targeted learning programs as
they utilize more NKC resources
18
Next steps
 Invest more resources in developing distance learning
programs as participants come from more than 30 sites
and 20 countries.
 Develop internal talent to get better understanding of
different business functions to design and deliver superior
quality DT learning programs.
 Seek new seek new learning market potential in
Emerging Markets to keep pace with Novartis business
ventures.
 Reduce “Neutral or Disagree” from all categories by
reviewing the “Comments” field and Integrating the users’
voice into the learning program design
19
Questions?
20