Ref 8.10
The evaluation of training and HRD: A critical study with applications.
Referred paper
Eduardo Tomé,
Universidade Lusíada de Famalicão
Email: eduardo.tome@clix.pt
Keywords: Training, Evaluation, HRD, Economics.
Abstract:
In this paper we review the evaluation methods used on training and HRD
programs. Specifically we address two questions in succession. First we try to assess if
some methods are better than others on defining the true impact of HRD and training
investments in the economy. Secondly we use a very specific methodology (Gramlich
1990) to evaluate some very well known experiences on the evaluation of training. In
that methodology, the benefit derived from the operations depends on three factors: 1)
the probability that the methodology defines correctly the program impact; 2) the
probability that the administrator takes on account the evaluators’ conclusions; 3) the
total value of program expenses.
1
Introduction
Societal background:
In the 21st century economy it is well understood that Human Resources are a
major factor of economic and social wellbeing. In world terms, the Education Index,
computed by the United Nations Development Program (UNDP) shows the clear
relation between Education, income and Human development (Table 1). The Human
Development Report (HDR) Education Index is obtained computing the adult literacy
rate (% for the population aged 15 years old and above) and the combined gross
enrolment ratio for primary, secondary and tertiary education in percentage.
Table 1 - Education, Income and Human Development – 2006
Countries
Income
Human
Development
33082
0.936
High Income
7416
0.776
Middle Income
2631
0.570
Low Income
9543
0.743
World
Source: United Nations, 2008
Education Index
0.937
0.843
0.589
0.750
We must note, that the HDR Education Index is not only a measure of the level
of Education in countries, but it is also a result of decades of investment in Education
and in basic HRD in those countries.
In what concerns specifically training, it is well known that the more developed
countries are also those that invest more in private training (funded by companies and
by individuals) and in public training (funded by the state). Usually those investments
are measured by the share of adults that participate in life long learning, and by the
share of individuals that undergo training, or by the funds spent in active labour market
policies (see Table 2).
Table 2 - Investment in Training: Basic Statistics
Life Long Learning (2006)
Public
Euros
7.5
608
Belgium
1.3
14
Bulgaria
13
Czech Republic 5.6
29.2
1059
Denmark
7.5
5580
Germany
6.5
4
Estonia
7.3
342
Ireland
1.9
71
Greece
10.4
1334
Spain
7.6
4927
France
6.1
2839
Italy
6.9
13
Latvia
4.9
11
Lithuania
3.8
34
Hungary
2
Funds
(2005)
Million
15.6
723
Netherlands
13.1
800
Austria
4.7
250
Poland
4.2
431
Portugal
1.3
10
Romania
15.0
12
Slovenia
4.1
9
Slovakia
23.1
581
Finland
32.0
981
Sweden
United
26.6
1559
Kingdom
18.7
891
Norway
Source: Eurostat 2007
Note: Life Long Learning refers to the percentage of the population aged 25 to 64
participating in education and training over the four weeks prior to the survey. Public
funds refers to the Public expenditure on training for unemployed and groups at risk
The data related training are usually collected regarding the OECD (OECD, 2006), the
EU (Eurostat, 2007) and the other developed countries, with few exceptions (UNESCO
2006).
However, even if the investment in HRD and training is seen as an obvious
necessity, the evaluation of that investment poses many crucial questions. First of all
resources are scarce and must be used with the best return possible. Secondly, in order
to guide policies, it is essential to evaluate investments. That evaluation might be made
in macroeconomic terms, if the program has a macroeconomic impact, but must be
made at micro level in any case. Therefore, it is crucial to know which the best methods
available to evaluate training and HRD are. The evaluation problem has a particular
significance, because with the emergence of a Knowledge Based Economy (KBE) the
societal needs for training and HD worldwide are not decreasing, quite on the contrary,
they are increasing. Finally, every person who had some important position in the
administration of HRD and training programs will tell you that the evaluation processes
put in place, are dependent on the assumptions and feelings the administrators have on
the programs. Putting it very basically, we only evaluate when we have doubts. We
don’t evaluate if we are sure that the program is very good (or very bad).
Research Questions:
Given what was written above, this paper will address two basic research
questions:
1)
What are the best methods available to define the impact of training / HRD in
the economy?
2)
How can we evaluate the evaluations that have been made, having in
consideration the dimension of the program, the evaluation method that is used, and the
aprioristic conceptions of the administrators ?
Plan of the paper:
In the first part of the paper we define the theoretical background. Namely, we
review the basic economic theories on training and HRD (1.1) and we introduce the
evaluation problem (1.2) and describe the most important evaluation methods on the
topic (1.3), dividing them in result methods (1.3.1), and impact methods (1.3.2). In the
second part of the paper we present a model (Gramlich, 1990, 160-4) that we consider is
3
fundamental to answer to the two questions we posed. In the third part we describe data
we collected on evaluation experiences from some of the most important HRD and
training programs all over the world. In the fourth part, first (4.1) we analyse the
methods described in 1.2 according to the probability of defining the true impact of the
program. Therefore we answer to question 1. Secondly, in section (4.2) we apply the
Gramlich’s method to the experiences we described in section 3. In section 5 we discuss
the paper findings (5.1), point the papers’ limitations (5.2) and suggest further studies
(5.3). Finally, in section 6 we present the paper’s conclusions.
Theoretical background
Theories
The positive influence of training and HRD in the situation of individuals,
companies, and nations was noticed by the most important thinkers in the economic
world, since the dawn of Economics. Indeed, Adam Smith, was the first to notice that
skills and competences are an important factor in the development of countries (Paul,
1989, 59-60). The other most important classical economists agreed with Smith’s idea:
Marshall (Paul, 1989, 59-60), Stuart Mill (Shackleton 1995, 10), Pigou (Stevens, 1996,
21). Keynes, also gave competences and skills an important place in the economic
process (Keynes, 1990, 191). But it is very well known that the basic economic theory
that is nowadays considered on HRD, and E & T saw the light with the works of neoclassical economists in the 50s, 60s and 70s: Becker, 1962; Becker 1976; Becker, 1980;
Ben-Porath, 1967; Mincer 1958; Mincer 1962; Mincer 1974; Oi, 1962 and Schultz,
1971. According to those studies, HRD should improve labour productivity, and in
consequence the wages of qualified people should be higher than those of non qualified
people. Furthermore, HR being a form of investment, its returns should essentially be
perceived in the long run. And, given that according to Human Capital Theory (HCT)
the process of economic development was to be based in HR, at least in the long run a
higher level of HR was considered to be a condition for employment.
Those analysis were quite strong and valid, but also a bit simplistic. And it is
quite interesting to note that in the last decades, HCT theories have been challenged in
significant a number of ways, and that, as a consequence of the critical contributions
that challenged the HCT, the corpus of economic theories on HRD grew in stature and
the real world was much better explained. Some of the major advances relate to the
explanation of how discriminative factors like race, gender and company dimension
may be incorporated in the analysis, to the public intervention in the labour market, and
to the role of knowledge in the economy. We will describe succinctly these
contributions in the next paragraph.
Regarding the discriminative factors, critical analysts explained why some
classes (among them ethnic minorities, adult women,, disabled persons) are given less
HRD opportunities because they are perceived beforehand (screened) by companies as
being less productive; and on the contrary people with diplomas and “politically
correct” status are given all the training opportunities by organizations because they are
perceived as productive. As a consequence, inefficiencies and inequalities are originated
(Spence, 1973; Stiglitz, 1975). This analysis originated a vast bibliography on contracts
and asymmetric information in the labour market, to deal with that market imperfection.
Other important fact is that not all the companies that act in the labour market are equal.
Quite on the contrary there is a tendency for the market to be quite dualistic, with small
4
SMEs having difficulties training employees, and big companies offering solid career
perspectives and big training opportunities. As a consequence training tends to be better
remunerated in big companies than in small ones (Chapman, 1994; Doeringer and Piore,
1985).
Regarding the public intervention in the labour market, for HCT theorists, HR
investments were mainly private investments, made by workers and companies; but,
quite a number of studies have since suggested that the public intervention in the HR
arena should be made, at least to eliminate some market failures related to externalities,
market imperfections, and weak private capacity to train (Middleton, Ziderman and
Adams, 1991, 35) funding difficulties (Katz and Ziderman, (1990), transferable training
(Stevens, 1996), the bad job, low skills trap (Snower, 1996), and individual myopia
(Dougherty and Tan, 1997, 38). Those studies also pointed out that in case of equity
problems the public intervention could be justified (Booth and Snower, 1996).
Of course, as a general golden rule, the public intervention in the HRD market should
only be made if the dimension of the government failure it carries with, is smaller than
the dimension of the public failure that is to be solved (Booth and Snower, 1996).
It is also very curious to note, that several philosophic schools exist in social policy, and
we can assume that they differ in the relative dimension they give to market failures
versus government failures (see Table 3) (Alcock, Erskine and May, 2003; Esping
Andersen, 1990; Ferrera and al, 2000; Deacon, 2000). An application of those
differences of conception to the Human Resource Management reality was already done
(Ignatovic and Svetlik, 2003).
Finally, a vast number of studies have been done on Knowledge Management,
that in some way extend the analysis of the use of HR in companies, following the
studies of Nonaka and Takeuchi (1995), and Kaplan and Norton (1996).
Table 3: Public Intervention versus Private Intervention in HRD
Type of Social Market
Government
Consequence
System
Failure
Failure
for HRD
Minimal
Maximal
Privatized
Liberal
Moderate
Considerable
Subsidiary
Conservative
Considerable
Moderate
State led
Social
Democratic
Maximal
Minimal
Nationalization
Socialist
Example
UK, USA
Central Europe
Nordic
Countries
Eastern
Countries
Source: Own analysis of Alcock, Erskine and May, 1997; Esping Andersen, 1990;
Ferrera and al, 2000; Deacon, 2000
The problematic about evaluation
Program evaluation is a complex question. At least 10 questions can be put when
evaluating a program (see Table 4).
Table 4: Questions that might be addressed when evaluating a program
Question
How ?
When ?
Where ?
Meaning
Methodology
Timing
Sampling, Target Group, Organization,
5
Country
Rationale
Planner, Evaluator
Objectives
Decision Maker
Decisions
Criteria
Beneficiary
Why ?
Who ?
What ?
Whom ?
What decisions ?
Which criteria ?
Which benefit ?
Source: Chinapaj and Miron 32, 1990.
In this paper we concentrate ourselves in the first and in the last question,
namely the methodology and the benefit from the evaluation. However, some other
questions will also be important in the discussion, namely number 3 (Where, When).
The second question (Why) was analysed in the first section (Theories).
Evaluation Methods
Economists and managers have developed many important evaluation methods
in order to evaluate training and HRD. Those may be analysed in the context of the
evaluation of social policies (Tomé 2005). Accordingly, some of those methods relate
basically to occurrences (they may be called results methods), and some of those
methods relate to the perceived consequences of the training programs (they may be
called impact methods). We will present result methods (1.3.1) and impact methods
(1.3.2) in order to analyse the methods used in evaluations (see 4).
Result Methods
The easiest way to evaluate HRD investments or policies is to count the level of
investment in money terms (financial indicator) and the number of people that
participate in the operation, namely the trainees (physical indicator). However result
indicators only analyse costs and participants, and are not related with benefits. At least
in theory it is absolutely possible that a large program (regarding costs and participants)
may have very low or even negative consequences for the people involved. But, result
methods tend to be used to evaluate policies because they are cheaper, less risky, less
random and much easier to command than impact methods (Tomé 2005).
Impact Methods
These type of methods try to define the difference the program makes to the
society in which it is implemented (Tomé 2005). There are several ways of putting that
idea in practice.
In economics and social policy, these methods involve computing the difference
between the evolution of a group of participants and a group of non participants. In a
weaker form, some studies only analyse the difference “before-after” regarding the
group of participants. When two groups are compared, selection problems may exist.
Therefore two main sub-groups of impact methods exist: experimental studies in which
the selection to the program is done randomly, and quasi-experimental studies in which
the selection to the program is not done randomly but a selection rule is estimated to
eliminate the selection bias. If we consider that there is not a selection bias (as in
experimental studies) we use simple control function estimators. If we consider that the
selection bias is only originated by the observable variables (as in non experimental
6
studies), we may use first difference estimators. If we consider that the selection bias is
originated by non observable variables (also in non experimental studies) we have to
use difference on difference estimators (Heckman and Hotz, 1989). Impact methods are
very important because they are the finest way to detect the program benefits, and
therefore to make cost benefits analysis. But they present quite a number of difficulties:
they are expensive (in money, time, resources, decisions), they are risky (because they
may expose the programs’ and the administrators failures), and they are random
(because there is no agreement within the scientific community about what are the best
estimators, and in the past different methods have originated different impact
estimators) (Tomé, 2005).
HR scholars have tried to measure the impact of a program by interview and
questionnaire methods, above all by the Kirkpatrick model of four levels (Kirkpatrick,
1998). The four levels are immediate reaction, learning, behaviour and organizational
results. Traditional managers and accountants try to define the impact by converting
HRD in a Return of Investment (ROI) perspective (Fits-zen, 2000). And in the last
decade knowledge managers have evaluated HR investment in the perspective of
Intellectual Capital investment using the Balance Scorecard methodology (Kaplan and
Norton, 1994).
Regarding all those methodologies (Schellhass, 1991) it is important to stress
that from there is a difference between the optimal impact of the program (A) which
tend to be less then the real effects (B). However, for political and administrative
purposes the predicted effects (C) may be estimated be default originating that in the
end the real effects tend to be higher than the predicted ones (C > B). However, the
evolution of the control group or comparison (D)has to be taken into account, and the
impact of the program (E) is given by B – D.
Different perspectives of the evaluation
It is quite important to verify that not all the methods that are used evaluating
training and HRD programs have the same perspective. In our opinion at least five
different perspectives exist, and every one of them results in different evaluation
methods, namely the following: 1) the public administrator perspective; 2) the private
manager perspective; 3) the HR expert perspective; 4) the accountant perspective; 5) the
economist perspective:
1)
In the public administrators’ perspective the best way of evaluating a program is
to use basic result methods. These methods provide the administrator with the
description of the operation. If the operation was received by a large number of people
and organizations, this is meant to be a success. According to this way of thinking the
final report should include financial and physical results. The public administrator may
also sense the importance of assessing the employability of the program. Other two very
important notions are the difference between the expected evolution and the real
evolution, and the ex-post assessment that the program really met social needs. That is,
public administrators are concerned with the execution of the program and the
correctness of the planning. The definition of the impact of the program is only a second
thought for the administrators;
2)
In the private managers’ perspective HRD and training programs are looked as
ways of spending own money; so, the evaluation of that investment might be made
almost intuitively by the evolution of the company’s overall performance. However,
sometimes the manager may want to test strictly the “value for money” aspect of the
7
investment in HR and in these circumstances ROI evaluation methods are the most
preferred. The basic handicap with those methods is that they overlook all the evolution
outside the company, and a similar value of profitability may mean very different things
in times of crisis or in times of growth;
3)
In the HRD expert perspective, the best ways of evaluating the programs, are
those that deal more immediately with the training scene; there fore HRD managers
tend to prefer Kirkpatrick reaction models or competence based models of evaluations
in which questionnaires or interviews are applied or even the methods in which the
observer participates closely in the operation; these models have also been developed in
the knowledge management analysis; A very significant branch of science was
developed analysing knowledge creation, knowledge sharing, knowledge transfer, and
knowledge audit;
4)
The accountancy perspective began to be very important in the last two decades,
when it became obvious that the company value was not dependent only and essentially
on the tangibles, but was mostly dependent on the value of intangibles; the Balance
Scorecard methodology was made in order to solve the equation: Market Value – Book
Value. Another branch of analysis was developed around the concept of Intellectual
Capital;
5)
Finally, in the economist perspective, in particular in the HRD economists
perspective, the most important is to define the impact of private or public policies, and
microeconomic impact models, or macroeconomic input-output models are the way to
detect the program impacts.
Table 5: A typology of evaluation methods
Perspective
Method
Classification
Participants,
Costs, Results
Public
administrator Direct Employability
ROI
Impacts for company
Private
manager
Interviews,
Impacts
for
agents
HRD expert
involved
Questionaire,
Participant – Observer
Kirkpatrick,
Finances, Structural Impacts for company
Accountant
Capital,
Social (finance,
structures,
Capital,
Human costumers,
human
Capital
capital)
Control Group
Impacts for society
Economist
Input Output Methods
Branch of science
Public
administration
Private Finance
Knowledge
Management
Intellectual Capital
Macroeconomics,
Human Resource
Economics
How to evaluate an evaluation
We think that nowadays evaluation is a common procedure in developed
countries and in conscious and prosperous institutions. However, we also think that not
all the efforts made to evaluate training and HRD have the same quality, and therefore,
the evaluation of the evaluation is a very important scientific topic. A very interesting
model to evaluate evaluations was presented some years ago (Gramlich, 1990, 160-4).
In this model the benefit derived from an evaluation is defined as a function of three
variables:
8
BEN = p * q * PV
The meaning of each one of those variables is the following:
1) p is the probability of discovering the true impact of the program;
2) q is the probability that the government / private authority will pay attention to the
result of the evaluation and take it into account.
3) PV is the program value measured in monetary units.
Using this method of evaluating evaluations we arrive to the following conclusions
concerning the priorities regarding program evaluations :
1)
it is better to evaluate using very fine, even if extremely costly impact methods
(for which p is large);
2)
it is better to evaluate large programs (for which PV is large);
3)
it is better to evaluate programs when there is a doubt about the program
usefulness (and therefore q is large).
It is also interesting to understand that, in a complementary perspective (Burtless
and Orr, 1986) the benefit of an evaluation can be measured as the costs (F) that are
saved if the program is stopped minus the benefits that cease to exist if the program is
stopped (G) plus the evaluation costs (H). It is clear that if F is higher than G + H the
evaluation is socially beneficial. Therefore, high values of p, q and PV may generate a
high F, that will ensure that G plus H are offset. In this circumstance the evaluation is
worth. However, a costly evaluation (with an high H) and a high F, may identify an
even higher G and in this hypothesis, H is justified because it showed that G exists and
G is an important and positive social benefit.
Known experiences of evaluation analysed and evaluated
We will separate the evaluations on training geographically. In section 3.1 we
will describe the American cases, in section 3.2 we will deal with the European cases,
and in section 3.3 we will address cases from other continents.
The variables that we will use to study PV, q and p are basically described in the
following Table 6:
Table 6: Elements of the model and relevant proxys
Concept
Relevant proxys
Physical indicators
PV
Financial indicators
Political discourse
Q
Type of methodology put in place
Detailed analysis of the methodologies
P
used
The American case
Public sponsored training and HRD programs
Very important evaluations on public sponsored training programs have been
made in the USA since the sixties (Tomé, 2001). Initially those evaluations were above
all made using result studies. But since the final years of the seventies impact studies
began to be performed on the those programs (Ashenfelter, 1978). The programs that
9
were evaluated include those made under the Manpower Development Training Act
(MDTA) and the three Acts that followed that piece of legislation in succession: the
Comprehensive Employment Training Act (CETA), the Job Training Partnership Act
(JPTA) and the Work Investment Act (WIA). A detailed list of the evaluations made to
the training programs in the United States is included in (Lalonde 1996). It is quite
interesting to study the evolution of the evaluation of those programs, in special those
four major programs, because in the mid eighties the CETA program was subjected to a
theoretical discussion in the Academy of Sciences of the United States (Ashenfelter and
Lalonde 1996). The most important part of the discussion was related with the fact that
very different methods of non-experimental evaluation had originated very different
estimates of the program impact. That troubling finding had a crucial consequence: the
JTPA program was evaluated with an experimental study (Orr, 1996). In consequence
we may say with a very high degree of certainty that in relation with the American case
of public sponsored training and HRD programs, p was large, q was large, PV was also
large and the benefit from that evaluation was also very big.
Privately sponsored training and HRD programs
The evaluation of HRD in the private programs is a common occurrence. Private
training and HRD programs have been made in the USA by the most influential
companies and organizations since WWII. Evaluation of HR investment has been also
made by many smaller companies whenever they want to assess HR investments. HR
and training have been evaluated using a plethora on methods, namely the following
(Mc Lean 2005, Smith and Piper, 1990): questionnaires: testing and griding; Kirkpatrick
four levels; trained observers: interviews; return on investment methods; impact
microeconomic methods; Balance Scorecard; systemic approaches. It is very curious to
note that training was widely evaluated, and that the values of the programs in question
were very high. Those programs were evaluated using pedagogic methods, traditional
management methods and new accountancy methods. So, in relation to Gramlich’s
methodology, PV, b, q and BEN are considered to by high, even if impact studies were
seldom used, because the evaluation approach was company based and not program /
society based.
The European Case
The programs funded by the European Union
The EU began funding training and HRD in 1960. However, until 1989 the
programs funded by the European Social Fund were only evaluated by result indicators.
This happened because from the perspective of the European Union (at the time known
as European Community) the simple fact of awarding money to the less developed
regions of the EU, and to some target groups was seen as beneficial to those regions and
persons. Therefore physical and financial indicators were seen as good indexes of the
benefits derived from the programs.
Until 1972, each country new the difference between the money received and the
money spent paying the ESF operations, and the operations were only funded ex-post,
meaning that there was a direct link between the money spent and the level of
employability: only successful operations were funded.
From 1973 to 1989, with the establishment of ex-ante payments, the awarding of
funds was money according to some “Orientation Guidelines”. In consequence when
10
the demanded amounts exceeded the available amounts, the proposed value of funds
was reduced in percentage even if they were eligible. The evaluation of the operations
was made by accounting the number of persons supported and the money spent. No
impact studies were performed. The ESF also asked for the situation of the trainees six
months after the operations but those data were not made public. Calculations were
possible on the relation between the amount of money awarded in the candidature phase
and the amount of money. It was also possible to analyse the number of trainees per
1000 habitants at the EU level and the number of ECUs per inhabitant at EU level also.
We call that way of analysing the “absorption analysis”. In consequence we must say
that before 1989, regarding the EU, PV was increasing, put p was small, and q was even
smaller. The potential benefit derived from that type of evaluation was very small.
Table 7: Evaluation experiences according to the Gramlich model
PV
P
Q
Country Funding Programs
Public
MDTA,
High:
High:
Very
High:
USA
CETA.
Billions of Impact
Concern
CETA,
dollars
studies
over
WIA
evaluation
reaches
Academy of
Sciences
Private Main private Very High : High:
High:
companies
Many
Plethora of Concern of
Billions of scientific
and
with
methods but economic
organizations. dollars
Small
few impact impact was
companies
studies.
evident.
when
Concern of
concerned
impact
as
with
HR
calculated
development.
by
Kirkpatrick,
ROI
and
Balance
Scorecard
was evident
too.
However,
few impact
studies.
Until 1990
High
and Small
– Small –
Europe Public
EU
increasing – results and Spending
budget
absorption
and
augmenting
absorption
are
indicators of
benefits.
From 1990 to High
and Rare – very Medium –
real Legal
2000
increasing – few
budget
evaluation
concern
11
BEN
Very high:
billions of
dollars
invested and
assessed.
Very High:
Huge
investment
assessed on
daily basis.
Very small –
the
evaluation
was
not
valid.
Very small –
in practice
almost
augmenting
Other
studies
performed
Increasing –
impact
studies
made on the
European
Strategy for
Employment
High:
impact
studies
in
growing
number of
cases
Since 2000
Very high
and
still
augmenting
Public
Sweden,
France,
Spain, UK
Very high
and
increasing
through time
Private
Major private Very high: High
–
companies
billions
Economic
and
since WWII and
organizations.
economic
Some small
methods
companies.
even if not
using impact
studies
Eastern
High
– Small
–
countries
Training
Existing as a
was
a form
of
widespread positive
phenomenon evaluation
Public
with
evaluation
High
–
Legal
concern
with
evaluation.
Evaluation
reports
High – need
to
assess
scientifically
the use of
public
money and
its economic
impact
High
–
Always
evaluated,
even if with
company
based
concerns,
case studies
Very small –
Taking for
granted that
to
the
program
was positive
nothing
changes
Medium –
countries
that
use
impact
studies are
not
the
majority
High
–
Investment
scientifically
assessed.
High
–
Investment
frequently
assessed.
Very small –
Almost
impossible
to detect any
impact
Public
Brazil,
Canada,
Australia
High
High
– High
– High
socioconcern
policy
economic
with public guidance
studies
money
World
Bank
and
NGOs
Less
Developed
Countries
Important
Small
– Small
– Small
–
result
beneficial
international
indicators
instincts
aid concerns
dominate
prevail
Private
Brazil,
Canada,
Australia,
Japan
High
High
– High
– High
–
company
concerns
guidance of
based
about
private
methods
usefulness
investment
12
–
In 1989, the European Commission tried to reform the Structural Funds (SFs).
As a consequence a new procedure of implementation was defined. The SFs began to be
organized in programs. For each program four operational phases were defined: ex-ante
appreciation, monitoring, control, and ex-post evaluation. In consequence, the
importance of the evaluation of programs was reinforced, at least in theory. And we
may say that from 1990 to 1999, at the epoch of the first two “programming” phases
from the Structural Funds reform (phase I 1990-3; phase II 1994-9), the value of PV
was increased because the funds available for the ESF were augmented, and the value of
q was also increased because the concern with the evaluation of programs was clearly
shown in the legislation. However, in practice the desire of evaluation was expressed in
evaluation reports that had the following structure: 1) Description of the situation before
the program and justification for the program; 2) Description of the program and
description of the differences between the expectations and the reality; 3) Inquiry in the
people (trainees and trainers) and in the organization of evolution of the situation
“before-after”. Therefore, the value of p was not considerably augmented, even if it was
expressed that the effect of the programs was going to be detected.
It is a clear sign of the situation that existed in the EU regarding the evaluation
of training and HRD that in 2001, the European published the collection “MEANS”,
named “Measures to Evaluate the Effectiveness of Structural Funds” (EU, 1999). That
book was composed of five volumes and in them the European Commission made a
summary of the various types of methods that were available to study the Structural
Funds. As a consequence, it was expectable that those methods would be extensively
put in use in the third programming phase (2001-6). And indeed, for some countries
(namely Spain, Greece, France, Netherlands, Austria, Sweden) the investment in
training by the EU was evaluated in 2002 using impact studies, as a part of the
European Strategy for employment, and in some cases (Spain, Greece and Austria) the
impact was found to be positive (EU 2002). One very interesting characteristic of the
evaluations made by the ESF is that they are made ex-post on the whole program and
not only ex-post on one of the years of the program. This characteristic in turn implies
that it is almost impossible to perform a meaningful impact study because impact
studies should be made when the program is still operating.
So, as a general conclusion, we must say that in the European Union funded
projects, the value of PV has been increasing steadily since 1958, and that increase was
a consequence of the increasing importance HR, HRD and training had in the European
economy. However, the value of p has not increased very much, in all the countries,
because we do not think that the methodologies of evaluation that were used find the
true impact of the program in socio-economic terms. Furthermore, we also think that
regarding the other aspects of the evaluation (namely the pedagogical and knowledge
based ones) p was low. Finally, we think that even if the concerns of the administrators
rose with the time, the fact the methodologies used in the evaluation are basically basic
result ones, that the evaluations compared the situation of the participants in the
program in a before-after comparison, and that the retroactivity of the evaluation was
severely diminished, means that q is growing and is high in the discourse but low in
practice. The exception to this rule is the evaluation made in the scope of the European
Strategy for Employment (EU, 2002). As a rule, BEN is small.
13
Programs funded by the National Budgets
Those type of programs were subjected to a large set of different types of
evaluation:
1)
Basic result methods were widely applied in the generality of the programs put
in place;
2)
Methods related to the pedagogical aspects of HRD and training were also used
in many cases;
3)
Impact methods were used in some cases, in different countries, most of all in
Sweden, Norway, France, the UK, Netherlands, Spain.
The concern with the evaluation of training in those countries and regarding
public funded programs was clear from the since the eighties, and since than more
sophisticated evaluation methods were implemented. It is also clear that the value of the
PV variable has been increasing with the passage of the time, because the economic
changes meant that the public presence in the labour market, regarding training and
HRD was widened and reinforced. So, we consider that in this case the values for the
three variables that constitute the model are high, and BEN is also high.
The programs funded by the private organizations
Training and HRD practices have been a frequent practice in Europe since
WWII and mainly since the sixties. Most of the privately funded programs have been
submitted to simple forms of evaluations. Many programs have been evaluated on
pedagogical and reactive grounds. Some programs have been evaluated with ROI
techniques and with Balance Scorecard techniques. A small number of programs have
been evaluated with impact studies.
It seems that even if the organizations were quite concerned about training and
HRD, a fact that is evident by the increasing number of workers involved in those types
of operations, by the increase in the money spent, and by the large number of
evaluations made. However, it is also quite interesting to see that most of the
evaluations made were related to the company own results.
It seems that the European private companies rely on the fact the HRD and
training are useful, and that they expect that usefulness to be shown in terms of
competences, financial results, routines or costumer satisfaction. So in this case, PV, p
and q look to be really big even if impact studies of social scope are not used. The rare
use of impact studies is justified by the fact that companies have a more inner oriented
approach and that approach is more easily described by Kirkpatrick, ROI or Balance
Scorecard methods.
Other cases
Public funded cases
Important HRD and training programs funded by the public administration took
place outside the USA and Europe in countries like Brazil, Australia, and the Eastern
Countries of Europe. In what concerns the cases of the Eastern countries little statistical
information is known, with the exception that in those countries training was a part of
the educational system, and that in those countries the level of training and education
was high, when compared to the world average. We also don’t know of any experience
of evaluation in those countries, and we are led to the conclusion that training was made
14
by the public organizations because it was meant to be beneficial for the country, and
that fact was enough to justify the occurrence of training. The only evaluation method
was the existence of training itself. So, PV was large, p was very small and q was even
smaller. The BEN derived from that type of evaluation was very small and that
situation led to massive changes in the administration of training when the socialist
regimes were replaced by democracies.
In Australia, Brazil and Canada, large training and HRD programs funded by the
public authorities existed since the sixties. Those programs were submitted to impact
evaluations in some specific cases. In those cases it was quite clear that the program
administrator was clearly concern and doubtful about the program impacts and wanted
to check them. So, p was high, q was high, PV was high, and BEN was also high.
World Bank and NGO funded cases
In some LDC countries, the HRD and training programs have been done with
the support of the World Bank and the other NGOs. Those programs were usually
evaluated using result indicators, and competence based indicators. We don’t know of
any deep and fine economic analysis regarding the use of those programs. We consider
that in these cases PV was small, and p and q were also small, even if there was some
justification in the fact that the promoters thought they were providing much needed
social support to those societies. It is also possible that in the future in the context of the
Millennium Goals those programs will be increased.
Privately funded programs
Privately funded training and HRD was also a common characteristic of
developed countries, like Japan, Canada, Australia, and New Zeeland. As it happened in
the USA and in Europe, those programs had an important dimension, and were
frequently evaluated using complex scientific techniques. That fact happened because
the companies were concerned with the return of the investment they made.
Concluding comments
Conclusions
The use of Gramlich’s methodology to evaluate training methodologies and their
practical use led use to three main conclusions:
1)
Some very basic and administrative methods are clearly less effective detecting
the true impact of the program;
2)
However a very wide of scientific methodologies are available with the intention
of detecting the “true” effect the program has; and we found that rather surprisingly all
those methodologies are similar, because they try to answer the basic question on the
“program” difference / consequence, put in different perspectives;
3)
When we analysed the practical use of evaluations on HRD and training
programs we found that State funded programs are evaluated with “socio-economic”
impacts studies and privately funded programs are evaluated with indicators related to
the company’s situation. The exception are EU programs in which the reliance on
“European money” and “absorption” concepts decreases the level of impact analysis.
15
References
Alcock, P.; Erskine A. and May M. (2003) The Student’s Companion to Social Policy.
Second Edition. Blackwell Publishing.
Ashenfelter, O. (1978), "Estimating the effect of training programs on earnings",
Review of Economics and Statistics, Vol. LX pp.47-57
Ashenfelter, O., LaLonde, R. (Eds),The Economics of Training, Volume I, Theory and
Measurement, Edward Elgar Publishing, Aldershot, pp.ix-xvi.
Becker G. (1962) Investment in Human Capital: A Theoretical Analysis - The Journal
of Political Economy – volume LXX - 5 - part 2 - Supplement - October – pp.9-49.
Becker, G. (1976) The Economic Approach to Human Behaviour - The University of
Chicago Press - Chicago.
Becker , G. (1980) Investment in Human Capital: Effects on Earnings - in Human
Capital : A theoretical and empirical analysis whit special reference to education Chicago - University of Chicago Press - Chapter II - pp.15-44.
Ben Porath, Y. (1967) The production of human capital and the life cycle of earnings Journal of Political Economy - 75 - August - pp.352-65.
Booth, A.L., Snower, D.J. (1996), Acquiring Skills, Cambridge University Press,
Cambridge
Burtless, G., Orr, L. (1986), "Are classical experiments needed for manpower policy?",
Journal of Human Resources, Vol. 21 No.4, pp.606-39.
Chapman, P. (1993), The Economics of Training, Harvester, Manchester.
Chinapah, V., & Miron, G. (1990). Evaluating educational programmes and projects:
Holistic and practical considerations. Socio-economic studies, No. 15. Paris:
UNESCO.
Deacon, B (2000) ‘Eastern European Welfare States: the impact of the politics of
globalization.’, Journal of Social European Policy, Vol 10 (2) pp. 146-161
Doeringer, P.B., Piore, M.J. (1985), Internal Labour Markets and Manpower Analysis,
Sharp Armonk, New York, NY, .
European Union (1999), Socio-economic Programs Evaluation, Collection MEANS,
Luxembourg,
European Union (2002) Impact Evaluation of the European Employment Strategy:
Technical
Analysis.
http://ec.europa.eu/employment_social/employment_strategy/eval/papers/technical_anal
ysis_complete.pdf
Ferrera, M,; Hemerijck A.; Rhodes M. (2000) The Future of Social Europe. Recasting
Work and Welfare in the New Economy Oeiras (Portugal) Celta Editora.
Fits-zen, J (2000) The ROI of Human Capital: Measuring the Economic Value of
Employee performance. AMACON.
Gramlich E. (1990) A Guide to Cost Benefit Analysis. Second Edition, Prentice Hall.
New Jersey
Heckman J.; Hotz J. (1989) Choosing Among Alternative Nonexperimental Methods
for Estimating the Impact of Social Programs: The Case of Manpower Training Journal
of the American Statistical Association, Vol. 84, No. 408 (Dec., 1989), pp. 862-874
Ignjatovic, M, Svetlik, I. (2003) European HRM clusters. EKK Toim., Autumn 2003,
no. 17, str. 25-39.
Kaplan R.; Norton D (1996) The Balance Scorecard. Harvard.
Katz, E, Ziderman, A (1990), "Investment in general training: the role of information
and labour mobility", Economic Journal, Vol. 100 No.403, pp.1147-58.
16
Keynes, J.(1990) Teoria Geral do Emprego, do Juro e da Moeda - São Paulo - Editora
Atlas - Translated from the 1973 edition of the McMillan Press.
Kirkpatrick D. (1998) Evaluating Training Programs: The Four Levels. (2nd edition) San
Francisco. Berrett-Kohler.
LaLonde, R. (1995), "The promise of public sponsored training programs", Journal of
Economic Perspectives, Vol. 9 No.2, pp.149-68.
Mc Lean G. (2005) Examining Approaches to HR evaluation; Strategic Human
Resources Volume 4, Issue 2, January – February.
Mincer, J.(1958) Investment in On Job Training and Personal Incomes Distribution Journal of Political Economy - 66 - 4 - pp.281-392.
Mincer, J. (1962) On-the-job training: costs, returns, and implications. - The Journal of
Political Economy - volume LXX - 5 - part 2 - Supplement - October - pp.50-80.
Mincer, J. (1974) Schooling, Experience and Earnings – NBER – Columbia University
Press – New York.
Nonaka I.; Takeuchi H. (1995) The Knowledge Creating Company – Oxford University
Press.
OECD (2006) Education at glance. Paris.
Oi, W. (1962) Labour as a quasi-fixed factor – The Journal of Political Economy - 70 –
pp.538-55.
Orr, L. (Eds) (1996) Does Training for the Disadvantaged Work? Evidence From the
National JTPA Study, Urban Institute Press, Washington, DC, .
Paul J.(1989) La relation formation - emploi: un défi pour l' économie, Economica
Paris.
Shackleton, J (1995) Training and Unemployment in Western Europe and the United
States -Edward Elgar , Aldershot, England.
Schultz T. (1971) Investment in Human Capital: The Role of Education and of
Research, New York: Free Press.
Spence, A M. 1973. Job Market Signaling. Quarterly Journal of Economics, 87(3), pp.
355-374.
Stevens, M. (1996) Transferable training and poaching externalities - in Alison Booth
and Denis Snower editors - Acquiring skills: market failures, their symptoms, and
policies responses - CEPR - Cambridge University Press - pp.19-40.
Stiglitz, J (1975). The Theory of "Screening," Education, and the Distribution of
Income. American Economic Review, 65(3), pp. 283-300.
Tomé E. (2001) The Evaluation of Vocational Training: A Comparative Analysis
Journal of European Industrial Training, volume 25 number 7 pags. 380-88 2001
Tomé E. (2005) Evaluation Methods in Social Policy – Bits – Boletin informativo de
trabajo social – nº 8.
UNESCO (2006) Participation in formal technical and vocational education training
programmes worldwide. An initial statistical study. UNEVOC International Centre for
Technical and Vocational Education and Training. Bonn, Germany.
United Nations (2008) Human Development Report 2007/8- Fighting climate change:
Human solidarity in a divided world. New York.
Internet websites:
Eurostat (2007) Labour market policy expenditure in active measures by type.
17