Paper Title: Successful people and organizational development in a service company:
the processes, results and the underpinning guidance framework.
Authors: Kesiena Mercy Clement-Okooboh and Professor Bill Olivier
Affiliation: University of Bolton
Institute for Educational Cybernetics
Corresponding author: kesienaokooboh@gmail.com
1
Abstract
Purpose
This paper presents a work-based action research case study undertaken in a national
branch of a large multinational company to help move towards its strategic objective of
becoming a learning organization.
Design/methodology/approach
It describes how training programs in the organisation were enhanced through a series
of organisational processes and practices to become integrated with workplace learning.
This paper draws on the evaluation and learning organization literature to present the
linkages made in practice between the training process, procedures and the business
goals.
Findings
Findings in the literature identified the varying impact of training on different
stakeholders and these were used to integrate higher levels of evaluation into a work
embedded feedback loop not only to improve training programs, but to transition
towards a learning organization that is adaptive to its changing operating environment.
Social/Practical implications
This case study’s value lies in its proposal of a guidance framework generalised from
the concrete instance of implementing post-training performance support, that can be
used to enable the creation of a learning organization, a task that has been found
difficult to do in practice.
Originality/value
This case study’s value presents a widespread perspective in providing a concrete
instance of adopting training to enhance performance in the organization thus enabling
the creation of a learning organization, a task found difficult to do in practice.
Keywords:
Learning organization; higher-level evaluation; training evaluation; feedback loop;
training; informal learning; work-based learning
Article Classification: Working Paper
(239 words, including keywords and Classification)
Introduction
2
This work-based action research study presents an overview of the organizational
context, the aims and objectives, the research questions and action research cycles that
constituted my work-focused PhD. It explores how this study identified four approaches
to successful people and organizational development in a service industry.
The first needs based approach centers on meeting individual, unit and organisational
needs, whilst delivering immediate and future benefits to our clients.
The second performance enhancement approach focuses on the need to enhance
organisational performance of individuals and teams through transfer of knowledge,
skills and attitudes gained from training programs to the workplace. The measure of
success of a training program lies in determining the changes in work place behavior,
unit performance enhancement, and customer satisfaction.
The third informal learning approach introduces informal workplace learning, using the
70:20:10 framework and performance support into the work-based.
Finally, the fourth evaluation and feedback approach involves summarizing and
communicating the findings and results of the evaluation to both internal and external
stakeholders, to establish the benefits of what has been learned and what further
learning is then needed.
In summary, the needs based approach drives the formal and informal learning;
performance enhancement seeks to ensure this is carried through into practice;
evaluation establishes the extent to which this has happened; the evaluation results feed
into further learning and improved performance.
Organizational Context
The context of this case study is in an energy & utilities company, based in Dublin
operating across the Republic of Ireland and Northern Ireland, with over 350
employees. Its parent company is a large French multinational with operations in 42
countries. The company provides solutions to support the sustainable growth of cities
and businesses. Two-thirds of its’ workforces are technicians and engineers.
Aims and Objectives
At the outset of this study then, there were two major, open and interrelated questions
facing the company:
1. How to achieve its strategic aim of becoming a “Learning Organization”
2. How to achieve the “Engineers Ireland CPD Accreditation”
3
The goal of this work-focused action research project was then to facilitate the strategic
transition of this company from a training-based one towards a full learning
organization.
Background of the study
The study stemmed from the evaluation of three training courses, namely supervisory
management, certified energy management and boiler operative accreditation scheme
in the organisation. The reason for selecting only three training courses to be evaluated
at these higher levels was because it is not practical to evaluate every course at these
levels. These three courses were deemed ideal for evaluating at the results level of the
Kirkpatrick model because they were expensive, strategic and operationally focused
(Phillips, 2007). According to Phillips (2007) he asserts, “Only a few select programs
should be the subject of evaluation at the ROI level”. The purpose of the study was to
address the evaluation of its training programs to achieve a sustained performance
improvement of its process that adds value to the organisation. Evaluation at the
different levels is an important influence in training. For performance improvement to
be achieved, learning and development practitioners must first identify the reason for
evaluating training that ultimately leads to a high performance culture. This led to the
focus of the four approaches that included other factors like mentoring, informal
learning and performance support aimed at achieving sustained performance
improvement.
The Four Approaches
An important issue informed the four approaches. For the evaluation of training to be
part of a dynamic process in which “sustained performance results” can be achieved it
has to be accepted as an organisational challenge (Brinkerhoff, 2005).
As well as ensuring that appropriate training is provided, the critical evaluation
measures employed in all four approaches also enabled the impact of the training on
performance of employees and the work unit to be enhanced through additional
workplace measures. Evaluation was also extended to include informal work-based
learning in the evaluation and feedback loop, thus moving from the evaluation of
training, to the evaluation of learning as a whole.
First Approach: Needs based
4
The first approach required getting line management involvement in the learning
process. The mechanisms put in place included line managers and employees attending
pre-training meetings where a unit’s operational goals and the employees
corresponding learning needs are established. The first step was to identify the needs
base, as this drives both the required training and the evaluation process, as the needs
of the different stakeholders involved provide the evaluation criteria.
Training measurement and evaluation can be a painstaking task for HRD professionals.
The aim of measurement and evaluation is to continuously improve the training
interventions. The need to determine the impact of training programs led to a review of
the training evaluation literature. It became clear that the Kirkpatrick Training
Evaluation method (Kirkpatrick, 1959) was the most widespread in use, and that a
number of others were variations on this approach (Hamblin, 1974; Phillips, 1996;
Holton, 2005; Brinkerhoff, 2003). To evaluate both individual and unit performance, it
was necessary to establish both the unit’s unmet or insufficiently met goals, the tasks it
would need to perform to achieve them, and the corresponding needs of the training
delegates would be in order to perform the required tasks. In establishing a process for
making effective use of the evaluation data being gathered, it became clear that their
primary function was to provide a feedback loop both to help assess whether the goals,
as set out in the pre-course discussion meetings, have been met, and to enhance the
training and learning processes themselves. In attempting to examine the impact of our
learning and organisational development practices, a critical early finding from the
evaluation of a selection of our training programmes was that while 80 per cent of
training delegates could successfully repeat what they had learned, relatively few,
approximately 20 per cent, were able to translate this into changed performance in the
workplace. This confirms a corresponding gap, identified in the literature, between the
learning gained from the training, and its transfer to the workplace (Baldwin and Ford,
1988, p.63; Detterman and Sternberg, 1993; Saks and Belcourt, 2006). An important
point of debate in the literature indicates a general lack of higher-level evaluation.
Kirkpatrick and Kirkpatrick (2006) highlights that “Evaluating results, Level four,
provides the greatest challenge to training professional” (p. 69). Kirkpatrick further
(2007) reiterated, “the four levels are well known and poorly implemented” (p, 4).
There is a consensus in the literature on the evaluation challenges. According to Sugrue
and Rivera (2005) their ASTD review of trends in workplace learning and performance,
training evaluations occur in organizations at the following percentages: level one
5
(reaction of learners) 91%; level two (learner knowledge) 54%; level three (transfer of
learning on the job) 23%; level four (impact of learning on the business) 8%; and level
five (return on investment) 3%. The reason for the low evaluation rate at the higher
levels can be linked to the findings of Devins and Smith (2013). They argue that line
managers perception of evaluating at higher levels may seem too time-consuming and
of minimal interest. According to Brinkerhoff (2006), “When managers support
training and learners, it works. When they do not, it does not.” Given these findings, in
order to address the issue of taking this less travelled full evaluation route, it is essential
to begin with a description of the purpose of evaluating learning programs and their
benefits to organizations. Many studies in training evaluation have shown the need for
measurement and evaluation both to account for training investment and to drive
organizational results. In order to convince organizational leaders that learning
programs are effective, the need for accountability has led learning professionals to
embark on measurement and evaluation of organizations training. This feedback
process comprises both qualitative and quantitative as well as financial and nonfinancial data (Philips, 2010).
Second Approach: Performance Enhancement
The second approach identifies the aspects of the evaluation process that feeds into
improved performance. Learning and development programs incorporate a huge
investment from a company’s perspective (Calhoun et al, 2006). To thrive in their
competitive ventures, organisation leaders must be able to change and adapt to the
marketplace (Calhoun et al, 2006). To show how their strategies contribute to
improving business results, learning and development professionals need to
demonstrate that they own the process before, during and after the learning initiatives
employed (Calhoun et al, 2006). Achieving this purpose is to define their programs’
objectives in business terms.
It is well established that half of corporate expenditure on training is wasted, because
the training of employees is undertaken far too long before the first opportunity trainees
have to apply the skills and knowledge on the job (Cross, 2007). Kirwan (2009; 2013)
found that between1% and 4% of payroll cost is spent on formal learning and
development in Ireland, but only between 34% of this investment is applied on the job
by employees after training (Saks and Belcourt, 2006). Taking into consideration the
complexity of evaluation in an organizational context, a systems thinking approach was
6
adopted to determine the other organizational factors that may have both contributed to
the results and may have also themselves been accelerated through the learning transfer
and change in behavior. To foster best practices for learning effectiveness within
organizations, which ultimately leads to performance enhancement, organizations need
to carry out training evaluation (Kraiger et al., 2004; Collins, 2002). In other to
determine the effectiveness of learning and development practices, measurements of
these practices must be conducted. Training outcomes are determined by a combination
of mechanisms that influence how people process information, focus their attention and
direct their effort during learning (Ackerman & Kanfer, 2004; Ackerman et al, 1995;
Kanfer et al, 1996; Kozlowski, Toney, et al., 2001). According to Robinson (2007),
“training is only one method of improving an organizations performance”. However,
Cross (2010), in his summary of learning in the 21st century workplace, demonstrates
convincingly that “work and learning are converging into the new conceptual work”
This tells us that work and learning cannot be separated from each other. This
interdependence has given rise to informal learning and the 70:20:10 framework that
will be discussed in the third approach. Developing and measuring the delegates’ skills
depends on other inherent organisational factors and initiatives namely, performance
support, mentoring, informal learning, and other initiatives like development plans, and
their associated measurement practices, and these constituted a major part of the
transition program.
Third Approach: Informal work based learning
The evaluation of learning, particularly at the higher levels of individual, unit and
organisational performance, is important as it informs the work based learning
approach. This in turn is a key aspect of our people development approach as it puts the
responsibility for learning primarily in the hands of the units and the learners. This
approach led to the adoption of the 70:20:10 concept Eichinger and Lombardo, (1996),
which is based on findings that, on average, 70 percent of all learning within an
organisation is informal, and happens on the job, 20 percent from semi-formal peer
review, feedback, mentoring and coaching and only 10 percent of learning is formal
(Australian Industry Group, 2006; Cross, 2006; Jennings and Wargnier, 2010; Meister
and Willyerd, 2010). Simply raising awareness of this concept within the organisation
had a significant impact on attitudes towards learning. This approach led to the
introduction of performance support mechanisms that ensured line managers and
7
delegates planned the support that will be available after the training. The support
provided by in-house mentors alongside the level four evaluation of training
programme, facilitated the translation of performance results into organisational
benefits. The performance support mechanisms provided a good platform for the
development of informal workplace learning. The relationship between training and
performance enhancement systems in organisations’ is one of mutual interdependence.
This is taking an important role in the education and training of the workforce (Johnston
and Hawke, 2002).
Taking this philosophy of workplace learning in the workplace at the outset of this
study. First, employees were of the opinion that they only learn when they attend formal
classroom training. Second, employees were not taking responsibility for their own
learning. Third the organisation wanted to record all learning in a central hub in order
to carry out analytics and to facilitate referencing and sharing of key information with
the senior management team.
Our intention in this study was to create a sustainable learning process that incorporates
informal workplace learning that translates into change in behavior and performance
improvement of individuals and business units in the organisation.
Fourth Approach: Evaluation feedback
Once all three approaches were put in place, the focus of the fourth approach was to
communicate all of these findings and results of the evaluation to the wider
organization. Sloman (2003) advocates the communication of learning and
development practices across the organisation. This programme adopted and adapted
the Kirkpatrick four–phase framework for evaluating learning effectiveness and builds
on its evaluation taxonomy, namely 1. Reaction to the learning event, 2. The learning
as tested verbally or in writing after the event, 3. behavior change (or individual
performance) and Results (or unit/organisational performance). The adoption of a
cognitive approach by organizational theorists has identified a number of principles that
encourage and enhance learning, such as motivation. The objective of learning is to
change the experience by impacting behavior and cognition (Williams, 1998), to
improve performance and to successfully transfer the learning to the workplace
(Clifford and Thorpe, 2007). Senge (2006) argues that the most powerful learning
comes from direct experience. Once the results from the training programs and semistructured interview were analysed and collated, the findings were shared with the
8
different stakeholders – individuals, line managers, clients’, trainers and the senior
management team via CPD meetings, the monthly Learning Talk newsletter and
workshops.
The Underpinning Guidance Framework
To help communicate the whole process to colleagues and to make it more usable by
others, both with in the organisation and externally, a guidance framework was
developed which is outlined in the following overview diagram and the accompanying
description.
Figure 1: Overview of the unit level learning framework
The new process starts (top left in the diagram) with managers and training delegates
attending a pre-training meeting, where the unit’s operational needs and the training
delegates’ corresponding learning needs are agreed. These then inform those who are
providing the training/mentoring/performance support. Critically, they are also used to
provide the criteria for the follow-on personal and unit performance evaluations. The
training or other learning event then takes place and is initially evaluated at level 1, the
learners’ reactions to the event, and level 2, the test of what has been learned. After the
learning event, the delegates should begin to change how they perform their tasks, and
9
this in turn should bring about changes in the unit’s operations, helping it to better achieve
its goals. Later, perhaps some months later, level 3, the actual changes in the delegates’
task performance, and level 4, the unit’s operational performance, are also evaluated. The
collected results of the evaluations are provided as feedback to the post-training meeting
of the managers and training delegates and compared with the original aims. This review
then provides further feedback to the trainers, mentors and those providing workplace
support. New or unmet goals and needs are then considered and become inputs to future
learning events and evaluations. Improvements are also proposed for the unit’s internal
mentoring, performance support and other informal learning activities.
The framework, initially developed around training, was later extended to include
performance support, mentoring and informal learning, and, after the first few
iterations, the pre- and post-course meetings were merged into unit learning meetings.
These are held at regular intervals and review the unit’s goals, the learning that has
taken place, and what learning is still needed.
An important aspect of these unit learning meetings is that they put the responsibility
for learning primarily in the hands of the operational units and the learners. They set
the goals, which inform the evaluation, which then provides them with the feedback
needed to assess and improve their learning.
However, the limitations on generality inherent in a specific case study are
acknowledged and the framework is offered to others wishing to follow a similar path
towards establishing a learning organisation, in the hope that it will prove useful and
that it will become clear whether and how if so how much it needs to be adapted to fit
different circumstances.
Research Methodology
This programme adopted both qualitative and quantitative methods, use of
questionnaires, observation and semi-structured interviews. The analyses were
conducted on twenty-three delegates that attended three different training courses. We
employed data from three different questionnaires that assigned a range of instruments
namely:

Pre-course questionnaire – Before commencing the training, this questionnaire
was sent to both the delegates and the line managers to set the goals and
10
objectives of the training, the needs of the individual, team and manager are
outlined.

Level one evaluation questionnaire – This questionnaire employed likert scale
and open-ended questions to gather findings on the delegates perception of the
training courses, the trainers facilitation skills and the overall programme. This
questionnaire is administered at the end of the training.

Level three-evaluation questionnaire – This post-course questionnaire is sent to
the delegates six to nine months after completing the training course to
determine if the delegates have changed their behavior and applied the new
knowledge and skills in the workplace to enhance their performance.

Semi-structured interviews (Yin, 1989) were conducted with line managers of
the training delegates to assess the impact of the training on the unit’s
performance. The interviews were carried out six to nine month’s period after
the training.

Line managers and also the trainer conducted observation of the delegates
during and after the course.
The data from each of the instruments employed was analysed and compared to each
other to determine the factors that may have achieved the performance results from the
training. The evidence from this case study includes comparing the goals and objectives
set during the pre-training meetings and the outcomes during the post-training
meetings. Once the results from these meetings were determined, a review of the other
factors that may have contributed to the performance improvement results of the
training courses was established through the semi-structured interviews with the line
managers. In the interviews questions were asked by the line managers of the delegates
how the delegates have applied the new skills/knowledge to their job. This question
was asked to gain insight in the application of learning and development process. Next,
questions were asked about the impact of training on the different stakeholders.
Findings and Discussion
In the first approach, it became apparent that there was a need to carry out a pre-course
discussion between delegates and their line managers to set the goals and outcome
before the training commenced. It was also determined during pre-course discussions
that there should be post-training discussions/ meetings of the delegates and their line
11
managers with the L&D Unit to determine if the pre-course goals had been achieved.
Aspects of the second approach on the rate of training transfer were investigated, had
been carried out prior to the commencement of this study as well as during it. The
results of the prior evaluation showed that most of the delegates sampled did not have
a clear understanding of the objectives of the specified training, or even why they
attended. Some also mentioned that they did not have the opportunity to apply the new
knowledge and skills back in the workplace. Line managers also felt that allowing the
delegates to attend the training programme would increase their morale in the team,
where it had been identified that the training delegates were demotivated. The revealed
lack of successful transfer from the training to the workplace was a concern to all.
Subsequently, when delegates were surveyed after these new processes and procedures
were implemented in the organisation, the rate of transfer from learning changed
performance in the workplace. This was found to have increased significantly from
20% to 95%. It also became apparent that the pre-course discussion had encouraged
both delegates and line managers to plan performance support mechanisms to put in
place after the training. The introduced changes thus had a huge impact on the
individual, team and organisational performance. In the third approach, the
development of an interrelated strategy encompassing performance support, informal
learning, and the introduction of the 70:20:10 concept and the in-house mentoring
programme helped to strengthen and integrate with the organisational learning strategy.
This informal learning part of the programme was deemed a critical factor as the line
managers now set out work related activities that gave the training delegates the
opportunity to apply and practice what they had learned, thus not just preventing loss
of the new knowledge and skills, but establishing them more firmly. The goal of higherlevel evaluation is to encourage better ways of ensuring the change of behavior of the
delegates that attended training programs. The output of the training had a higher
effectiveness rate and this was of importance to the organisation. The final approach
highlighted the summary and communication of the different processes to the different
stakeholders across the organisation. Using this approach ensured that these new
processes are understood by all to drive a process of continuous improvement. The
main aim of adopting this strategy was to enhance the adoption of the processes to
ensure its effectiveness in achieving the business results.
During the semi-structured interview process, nine line managers were interviewed as
part of this study. Firstly, they were interviewed to determine if the delegates had
12
changed their behavior or performance subsequent to the training. Secondly, they were
interviewed to determine how the training had impacted on the unit, clients or business.
It was confirmed that evaluating performance at the levels of
individual and
unit/organisational performance is more difficult than the levels of reaction and
learning evaluation. As a result, relatively few organizations evaluate the impact of
training (Brinkerhoff and Gill, 1994; Rothwell and Benkowski, 2002). Clearly, the line
managers play an integral and critical role in facilitating knowledge transfer as, in
addition to performance support, they need to provide the right conditions, within a
reasonable timeframe, for delegates to develop the use of their new skills on the job
(Gottfredson and Mosher, 2011). Furthermore, the learning and development unit
established a process for making effective use of the evaluation data being gathered.
The data enables improvement of the training courses and also to inform what needs to
happen in the workplace. As a result, promoting evaluation at these higher levels
enabled the organisation to move from viewing learning as something that happens in
the classroom to something that mainly happens in the workplace. This indicates that,
both learning in the classroom and the workplace are complimentary.
Conclusion
As with any HRD initiatives, the transfer of learning back to the workplace has been
the challenge for HRD professionals. The responsibility of assessing, measuring and
evaluating of any HRD activities relies solely on processes and procedures inherent in
the organization. The results from this study highlight strong links between our
approach to people development and the inter-woven goal of single feedback loop of
the new processes, procedures and frameworks introduced in the organisation. The
focus for us was to align our people development practices closely to enhance the
performance of training delegates. This has enabled employees to pull information and
learning from across the organisation into a positive, productive and supported learning
transfer in the workplace. Finally, this paper sets out how all of these practices are now
considered together with formal learning in the learning reviews.
13
Reference:

Ackerman, P.L. and Kanfer, R. (2004). Cognitive, affective, and conative
aspects of adult intellect within a typical and maximal performance framework.
In D.Y. Dai and R.J. Stenberg (Eds.), Motivation, emotion, and cognition:
Integrated perspectives on intellectual functioning (pp. 119-141). Mahwah, NJ:
Lawrence Erlbaum Associates.

Ackerman, P.L., Kanfer, R., and Goff, M. (1995). Cognitive and non-cognitive
determinants and consequences of complex skill acquisition. Journal of
Experimental Psychology: Applied, 1, 270-304

Australian Industry Group (2006); World Class Skills for World Class
Industries: Employers Perspectives on Skilling in Australia. Australian Industry
Group, Sydney

Baldwin, T. T., and Ford, S. K. (1988). Transfer of training: A review and
directions for future research, Personnel Psychology, 41, (1) 63-105

Brinkerhoff, R.O and Gill, S.J. (1994). The Learning Alliance: Systems
Thinking in Human Resource Development, Wiley, Chichester

Brinkerhoff, R. O. (2003). The success case method. Find out quickly what’s
working and what’s not. San Francisco: Berrett-Koehler.

Brinkerhoff, R.O. (2005). The success case method. A strategic evaluation
approach to increasing the value and effect of training. In G.G. Wang and D.R.
Spitzer (Eds.), Advances in HRD measurement and evaluation: Theory and
practice. Thousand Oaks, CA: Sage

Brinkerhoff, R.O. (2006). Telling training’s story: Evaluation made simple,
credible, and effective. San Francisco: Berrett-Koehler

Calhoun, W. et al (2006) The Six Disciplines of Breakthrough Learning: How
to Turn Training and Development into Business Results. Library of Congress
Cataloging-in-Publication Data.

Clifford, J. and Thorpe, S., (2007) Workplace Learning & Development.
Delivering Competitive Advantage for your Organizations. Kogan Page.
London
14

Collins, M.E. (2008). Evaluating child welfare training in public agencies:
status and prospects. Evaluation and Program Planning, 31, 241-246

Cross, J. (2006) Informal Learning: Rediscovering the Natural Pathways That
Inspire Innovation and Performance. Pfeiffer Publication

Cross, J. (2007) Informal Learning. Rediscovering the Natural Pathways That
Inspire Innovation and Performance. Pfeiffer Publication.

Detterman, D.K. and Sternberg R.J (1993), Transfer on trail: Intelligence,
cognition, and instruction. Ablex Publishing, New York, NY

Devins, D. and Smith, J. (2013). Chapter 8 Evaluation of HRD, in God, J.
Holden R., Stewart, J., Iles P., Beardwell, J. Ed (2013) Human Resource
Management, Theory and Practice, 2nd edn. Palgrave MacMillian, Basingstoke

Eichinger, R. and Lombardo, M. (1996) The Career Architect Development
Planner: A Systematic Approach to Development Including 103 Researchbased and Experience-tested Development Plans and Coaching Tips: for
Learners, Managers, Mentors, and Feedback Givers. Lominger Limited,
Minneapolis

Ford, J.K. and Weissbein, D.A. (1997). Transfer of training: An update review
and analysis. Per

Gottfredson, C., and Mosher, B., (2011). Innovative Performance Support:
Strategies and Practices for Learning in the Workflow. McGraw Hills
Companies

Groat, L. and Wang, D. (2002). Architectural research methods. New York.
John Wiley

Hamblin, A. (1974), Evaluation and Control of Training, McGraw-Hill,
London

Holton, E. F., III. (2005). Holton’s evaluation model: New evidence and
construct elaborations. Advances in Developing Human Resources, 7(1), 37-54.

Jennings, C. and Wargnier, J. (2010) “Effective Learning with 70:20:10: The
new
frontier
for
the
extended
enterprise.
Available
at:
www.crossknowledge.com/ (accessed 6 July 2014)
15

Johnston, R. and Hawke, G. (2002), Case Studies of Organizations with
Established Learning Cultures, NCVER, Adelaide.

Kanfer, F.H., Reinecker, H., and Schmelzer, D. (1996). Selbstmanagementtherapie. Ein lehrbuch fur die klinische praxis [Self-management therapy. A
training guide for clinical practice]. Berlin: Springer.

Kew, J. and Stredwick, J. (2013) Human Resource Management in a Business
Context. United Kingdom: Chartered Institute of Personnel & Development

Kirkpatrick, D.L. (1959). Techniques for evaluating training programs. Journal
of the American Society for Training Directors, 13(11), 3-9.

Kirkpatrick, D.L. (1996). Great Ideas Revisited. Training & Development,
50(1), 54-59.

Kirkpatrick, D.L., and Kirkpatrick, J.D. (2006). Evaluating training programs:
the four levels (3rd ed.). San Francisco, US: Berrett-Koehler.

Kozlowski, S.W.J., Toney R.J., Mullins, M.E., Weissbein, D.A., Brown, K.G.,
and Bell, B.S. (2001) Developing adaptability: A theory for the design of
integrated-embedded training systems. In E. Salas (Ed.) Advances in human
performance and cognitive engineering research (Vol. 1. Pp. 59-123).
Amsterdam: JAI/Elsevier Science.

Kraiger, K., McLinden, D., and Casper, W.J. (2004). Collaborative planning for
training impact. Human Resource Management, 43(4), 337-351

Kraiger, K. Salas E., Cannon-Bowers, J.A. (1995). Measuring knowledge
organisation as a method for assessing learning during training. Human Factors,
37, 804-816.

Meister, J.C., and Willyerd, K. (2010) The 2020 Workplace: How innovative
companies attract, develop and keep tomorrows employees today. Harper
Collins Publishers

Phillips, J.J. (1996) Accountability in Human Resource Management. Houston,
Texas. Gulf Publishing

Phillips, J.J. (1996). Measuring ROI. The fifth level of evaluation. Technical
and Skills Training, 4, 11-13
16

Phillips, J.J. (1997a). Handbook of training evaluation and measurement
methods (3rd ed.). Boston: Butterworth-Heinemann

Phillips, J.J. (1997c). Return on investment in training and performance
improvement programs. Boston, MA: Butterworth-Heinemann.

Phillips, J.J., and Phillips, P.P (2010). Measuring for Success: What CEOs
Really Think about Learning Investments. Alexandria, VA: ASTD Press

Robin, P.L (2007). A Level 4 Evaluation of a structured on-the-job operator
training intervention at a North American Paper Mill: A Dissertation presented
in partial fulfillment of the requirements for the degree of Doctor of Philosophy.
Capella University

Rothwell, W. and Benkowski, J. (2002). Building Effective Technical Training:
How to Develop Hard Skills Within Organizations, Pfeiffer, San Francisco, CA

Rummler, G.A., and Brache, A.P, (1995). Improving performance: How to
manage the whit space on the organization chart (2nd ed.) San Francisco:
Berrett-Koehler

Saks, A., and Belcourt, M. (2006). An investigation of training activities and
transfer of training in organizations. Human Resource Management, 45(4), 629648

Senge, P.M., (2006) The Fifth Discipline, The Art & Practice of The Learning
Organisation, London, Random House

Sloman, M., (2003) Training in the Age of the learner: Chartered Institute of
Personnel Development (CIPD) British Library Cataloguing, Great Britain

Sugrue, B., and Rivera, R.J. (2005). State of the industry: ASTD’s annual review
of trends in workplace learning and performance. Alexandria, VA: ASTD.

Tashakkori, A. and Teddlie, C. (2003). Handbook of Mixed Methods in Social
& Behavioral Research. Sage Publications

Tessmer, M., and Richey, R. (1997). The role of context in learning and
instructional design. Educational Technology, Research, and Development,
45(3), 85-115

Yin, R.K. (1989) Case Study Research, Sage: London
17