AI in Corporate Training Program (e)Valuation
Consultants: Bernard Mohr and Jane Magruder Watkins, The Synapse Group, Inc.
Description of the Organization
SmithKline Beecham, (now GlaxoSmithKline) a transnational pharmaceutical company,
developed and implemented a new process for drug discovery in its research division.
The implementation effort included the design and introduction of an intensive computerbased simulation program to help scientists and leaders operate effectively within the new
system by understanding the impact of actions over extended time periods. The training
program was called the SB Discovery Simulation, referring to a simulation of the
company’s research process, and was developed and managed by an internal organization
development department.
Focus of the Appreciative Inquiry
The client organization wished to evaluate the impact of this computer-based simulation
training program. When the project began in 1997, 480 senior scientists out of
approximately 2000 scientists from the drug research division had completed the
program.
Program evaluation is generally one of the most critical and reductionist of the deficitoriented processes we find ourselves using in training and organization improvement
efforts. An appreciative process turned conventional approaches to program evaluation
on its head, with surprising and rewarding results.
This process inquired into what worked well, what had positive impact on the
participants, and what positive impact was carried into the organization.
Organizational Objectives
The internal Organization Development group who had designed and delivered the
program wished to do an in-depth evaluation to ascertain the degree to which the program
had made an impact on the organization (Level 3 of the Kirkpatrick evaluation model):
did program learning result in workplace behavior changes? Were there other
applications and uses for the program? What follow-up activities might be helpful to the
business? Should it be offered again, what parts should be redeveloped and why? The
OD group also wanted to complete the process in time to use the results in their decisions
about renewal of the program.
What was Done
Bernard Mohr and Jane Magruder Watkins from The Synapse Group (Portland, Maine)
responded to an RFP by proposing an appreciative process that they felt would be most
likely to move the program in the direction of it’s best practices. At the time, this
approach had never been tried, but the client was open to innovation and willing to work
as a collaborative partner.
Once contracts to proceed were in place, the actual evaluation activities evolved over
about three months as adaptations were made to accommodate the clients business
realities:

Having the consultants and the three members OD group help a small core group
conduct the interviews. Usually the consultants do not do the interviews.

Conducting interviews with 20% of the program participants (instead of 100%) to
accommodate time pressures of the scientists.

The project was also done without a steering committee.

The clients also adjusted their expectations about the time it would take to
complete the project in order to collect data on both sides of the Atlantic.
Phase 1: Information and Planning

Educated the clients (the internal OD group) about Appreciative Inquiry and the
consultants about the training program.

Role, function and possible membership of a core group was established.
Phase 2: Select and Train Interviewers

Established a 10 person core group composed of participants in the program, the
group which had created the training program and members of the client’s OD
function

Core group members agreed to also be the interviewers and were trained in two
workshops, one in the U.S. and one in the U.K.

The interview protocol was drafted piloted and revised in these workshops
Phase 3: Conduct Interviews (Collect Data)

In one-hour face-to-face interviews conducted by core group members, the
process reinforced the participants’ learning from the workshop.

Scientists responded well to, and were quickly engaged in the appreciative
questions. Approximately 100 of the 480 program participants were interviewed.
Phase 4: Preliminary Data Analysis

At end of each interview, interviewers completed and e-mailed an electronic
interview summary to a central administrative location.

We collated the data by person and by question.
Phase 5: Identify Major Learnings and Preliminary Recommendations

Held a two-day report-writing workshop with the whole core group to create first
draft of a report.

Worked to a table of contents provided by the consultants.

Findings were sorted into key themes with supporting examples under each
theme.

The core group wrote recommendations in form of provocative propositions.

A draft report was finished after the workshop by means of an extended
teleconferences that included whole core group conversations and geographically
based break out sessions in five locations.- while the teleconference line was kept
open.
Phase 6: Finalize Recommendations and Write the Report

Core group members identified audiences and tailored the report to the audiences
with support of the consultants.
Outcomes of the Initiative
The AI approach was broad enough to allow the company to achieve all it’s goals with
the study. The stories and examples illustrated the findings in compelling, persuasive
ways beyond what is possible from traditional evaluation data. The study served as a
follow-up event for the participants, reinforcing their learnings. The key elements about
what worked were incorporated into future simulations with the company. A broad base
of support for the program emerged, especially within the top-level leadership team.
What We Learned

An AI approach can be acceptable and even welcomed by scientists.

AI works particularly well for evaluating, while simultaneously reinforcing,
behavior changes resulting from training.

Joint partnership between the client and consultant(s) creates a more robust and
synergistic approach.

The usefulness and diffusion of the findings about organizational level data would
have been enhanced with a steering committee that included management.
Additional Resources: Appreciative Inquiry and Learning Assessment: An Embedded
Evaluation Process in a Transnational Pharmaceutical Company, by Bernard J. Mohr,
Elizabeth Smith, Jane M Watkins, In Appreciative Inquiry: An Emerging Direction for
Organization Development by Cooperrider, D; Sorensen, PF; Yaeger, TF; Whitney, D;
Stipes Publishing, 2001