Integrating Needs Analysis,
Assessment & Evaluation
of Training
Catherina Opperman & Marius Meyer
2009
www.roionline.co.za 2008
FOCUS
D
E
S
I
G
N
NEEDS ANALYSIS
Skills Audit
ASSESSMENT
T
R
A
I
N
I
N
G
www.roionline.co.za 2008
EVALUATION
STRATEGIC LEARNING
ALIGNED TO TRAINING
Identify skills gaps:
• Current gaps that need further
development or are lacking
• Future skills that the organisation
will need in the next
two to five years
www.roionline.co.za 2008
STRATEGIC LEARNING ALIGNED
TO TRAINING
Skills gaps & training needs:
• Corporate training needs across entire
organisation, example communication
skills
• Job-specific training needs specific to a
particular area or position, example
product knowledge or marketing skills
www.roionline.co.za 2008
STRATEGIC LEARNING
ALIGNED TO TRAINING
Identify strategic learning objectives from:
• Business Plan
• Workplace Skills Plan
• Strategic HRD Plan
Ensure that all job titles are represented on
the Plan
www.roionline.co.za 2008
STRATEGIC LEARNING ALIGNED
TO TRAINING
Record and categorise these needs in a
Strategic learning and development Plan:
Example:
• BS: Business specific, product knowledge
• IP: Interpersonal: communication skills
• IT: computer skills: Excel
• ML: Management & leadership skills
www.roionline.co.za 2008
MAIN ELEMENTS OF A JOB
ANALYSIS
• Purpose of a job: job demands,
responsibilities, accountability,
activities, procedures, processes
• Tasks: how often performed,
sequence of tasks, equipment
needed, how do tasks relate to other
jobs
continue
www.roionline.co.za 2008
DEVELOP THE JOB PROFILE
• Focus on the job position, not the
employee
Divide the position into knowledge,
skills & behaviours
that are essential
www.roionline.co.za 2008
METHODS TO COLLECT DATA
•
•
•
•
•
Job interviews
Observing employees in the workplace
Interview individuals
Interview groups
Interview supervisors
& technical specialists
• Questionnaires
• Self assessment questionnaires
www.roionline.co.za 2008
JOB PROFILE ANALYSIS &
PERFORMANCE MANAGEMENT
The next step is to link the job profile to
performance management to assess
specific competencies & address gaps that
require training
• Identify gaps, assess & evaluate by linking
output requirements to the individual’s
Performance management plan
www.roionline.co.za 2008
TERMINOLOGY
Needs assessment refers to different types
of analysis that are referred to as needs
analysis:
• Performance analysis
• Gap analysis
• Target population analysis
www.roionline.co.za 2008
NEEDS ASSESSMENT
Needs assessment refers to the three
elements of:
organisational-, person- and task analysis
www.roionline.co.za 2008
DIFFERENT CATEGORIES OF TRAINING
NEEDS
• Macro level: sector skills needs
• Meso level: a single organisation
• Micro level: individual needs
www.roionline.co.za 2008
PERFORMANCE GAPS
Gaps exist as a result of:
• Performance gaps in the way individuals
work
• Management gaps in the way people are
managed
• Organisation gaps in the manner in which
organisations are designed
www.roionline.co.za 2008
STAKEHOLDERS IN HRD
& ASSESSMENT
•
•
•
•
•
•
•
•
•
•
Government
External
Management
Training committee
Trade unions
ETD providers
Critical interest groups
Community groups
Assessors
Moderators
•
•
•
•
•
•
•
•
Learners
Mentors/coaches
Customers
Past participants
Professional bodies
SDF
Business partners
Suppliers
www.roionline.co.za 2008
THE NEEDS ANALYSIS PROCESS
Obtain data from two sources:
• The business plan: key strategies &
broad competencies
• Skills of individual employees: assessed
against the business plan will identify
gaps in performance
www.roionline.co.za 2008
REPORT RESULTS AND MAKE
RECOMMENDATIONS
Feedback will depend on the nature and
extent of the issue/problem and could be
done verbally or:
• A comprehensive document with charts &
graphs presented to top management
• Recommendations to take into account:
Training strategy, methods & costs
• Be aware of the influence of culture &
management buy-in to training
www.roionline.co.za 2008
SELECT OR DESIGN A LEARNING
INTERVENTION
If recommendations are accepted, use the identified needs
for selecting or designing a learning intervention:
•
-
Learners need to be aware of the objectives of training
related to the purpose and expected outcomes of
learning activities:
What they should be able to know and do
Criteria that indicates acceptable level of performance
What criteria they will be assessed against
Scope of learning
www.roionline.co.za 2008
DESIGNING A LEARNING
PROGRAMME
Take the following into account:
• Long term vision & mission of
organisation
• Strategic short term goals & strategies
• Key skills competencies to meet those
needs
• Skills priorities of the organisation
• Objectives of learning
continue
www.roionline.co.za 2008
DESIGNING A LEARNING
PROGRAMME
Take the following into account:
• Learning outcomes
• Assessment instruments
• Assessment procedures
• Numeracy & literacy levels
of learners
www.roionline.co.za 2008
OTHER TYPES OF ANALYSIS
i) Organisation or situation analysis
ii) Target population analysis
iii) Task analysis
iv) Skills & competency analysis
v) Skills Audit –
Skills Gap Analysis (SGA)
www.roionline.co.za 2008
OTHER TYPES OF ANALYSIS:
Task analysis
Task analysis form:
• Five levels of function:
- Job title
- Specific duty
- Tasks under each duty
- Sub tasks
- Knowledge
www.roionline.co.za 2008
SKILLS AUDIT PROCESS
Steps in the Skills audit process:
1.Identify current & future priorities from the
Business Plan; core work & future strategies
2. Identify essential skills for current & future
business
3. Assess current skills level, use a rating scale,
example: expert/satisfactory/limited/no
experience
4. Identify action to address skills gap
continue
www.roionline.co.za 2008
2.4 OTHER TYPES OF ANALYSIS
Assessment methods:
• Employee self assessment
• 360 degree
• Panel assessments
• Formal assessment:
- Diagnostic, formative, summative &
recognition of prior learning (RPL)
www.roionline.co.za 2008
COMPETENCE
• Perform whole work roles
• Against standards of performance
• Ability to work with your hands (skills),
head (knowledge) & heart (attitude, values
& behaviour)
www.roionline.co.za 2008
ASSESSMENT STRATEGY
• Analysis of source documents (policy & plan)
• Assessment context
(environment/circumstances)
• Assessment activities
• Logistics
• Scope: range of assessment
• Instructions to assessors
& candidates
• Evidence requirements
• Methods to gather evidence
www.roionline.co.za 2008
OVERALL ASSESSMENT
STRATEGY
•
•
•
•
•
•
•
Plan for the assessment
Prepare the learner
Conduct assessment
Assess evidence & store it
Evaluate the process
Provide feedback
Review at regular intervals
www.roionline.co.za 2008
COMPETENCY-BASED ASSESSMENT
OBE is a learner-centred process:
•
What learners are to learn is clearly defined
•
Learners’ progress is based on demonstrated
achievement
•
Learners’ needs are accommodated through
multiple teaching and learning strategies and
assessment tools
•
Each learner is provided the time and
assistance to realise his/her potential
www.roionline.co.za 2008
PLAN THE ASSESSMENT PROCESS
•
•
•
•
The candidate & assessor meet
Draw up an Assessment Plan
Conduct assessment
Evaluate the evidence
www.roionline.co.za 2008
PLAN THE ASSESSMENT PROCESS
•
•
•
•
•
Record the decision
Provide feedback
Plan reassessment
if necessary
Certification
Record the results
www.roionline.co.za 2008
PLAN THE ASSESSMENT PROCESS
Stages of planning the assessment:
i) Initial planning
ii) Pre-assessment meeting
with learners
i) Draft an Assessment Plan
ii) Prepare the candidates
iii) Collect the evidence
iv) Give feedback
continue
www.roionline.co.za 2008
PLAN THE ASSESSMENT PROCESS
Special education needs examples:
• Illiterate learners
• Semi-literate learners
• Innumerate learners
• Physical disabled learners
• Blind learners
• Deaf learners
www.roionline.co.za 2008
PLAN THE ASSESSMENT PROCESS
•
•
•
•
Plan against contingencies that may
arise
Obtain results of previous assessments
Design activities & instruments that are
appropriate to the outcomes & resources
Assessment documentation is prepared
for the recording of information
continue
www.roionline.co.za 2008
PLAN THE ASSESSMENT PROCESS
•
•
Potential unfair barriers are identified
and plans are made to address these
barriers without compromising the
validity of the assessment.
The required physical and human
resources are to be ready and available
for use, and logistical arrangements are
confirmed
continue
www.roionline.co.za 2008
PLAN THE ASSESSMENT PROCESS
•
•
Provision for moderation is made in
accordance with relevant assessment
policies and ETQA requirements.
A variety of assessment methods are
described and compared i.t.o. strengths,
weaknesses and applications
www.roionline.co.za 2008
PLAN THE ASSESSMENT PROCESS
•
•
•
•
The Appeals procedure
The re-assessment procedure
Follow up and support
The signatures of all the relevant parties
& dates
www.roionline.co.za 2008
PLAN THE ASSESSMENT PROCESS
Prepare the candidates:
• Use appropriate language level
• Carry out checks to ensure candidates
are ready for assessment
• Ensure assessments are in line with
policies
• Provide opportunities for input from
candidates
www.roionline.co.za 2008
ASSESSMENT PRACTICES
Validation of assessment:
•
Verification of the process
•
Statistical analysis
•
Examination of the assessment instrument
•
Sampling of evidence of applied competence
•
Observation of processes
•
Site visits
•
Interviews
www.roionline.co.za 2008
ASSESSMENT PRACTICES
The principles of good assessment
practice:
• Fairness
• Validity
• Reliability
• Practicability
www.roionline.co.za 2008
ASSESSMENT PRACTICES
Quality of evidence refers to ‘VACS’
criteria:
•
•
•
•
Valid
Authentic
Current
Sufficient
www.roionline.co.za 2008
ASSESSMENT PRACTICES
Types of evidence:
Refers to the method that will be used to
collect evidence.
• Will it be collected by directly observing
the learning?
• By making use of secondary sources of
evidence? or
• Historical evidence?
www.roionline.co.za 2008
ASSESSMENT PRACTICES
Appeals can be brought against:
•
•
•
•
•
•
Unfair assessments
Invalid assessments
Unreliable assessments
The assessors’ judgement; if considered
biased
Inadequate expertise and experience of the
assessor if it influenced the assessment
decision
Unethical assessment practices
www.roionline.co.za 2008
ASSESSMENT PRACTICES
Guide and support learners:
• Advise
• Coach
• Tutor
• Counsel
• Mentor
continue
www.roionline.co.za 2008
ASSESSMENT INSTRUMENT
Design the assessment instrument according to
the assessment strategy that includes the
following elements:
• Determine the purpose of learning
• Analyse the needs of learners
• Decide on the learning objectives and outcomes
• Select the content that will support the
achievement of outcomes
• Decide on the activities, methods, and media for
learning and development
www.roionline.co.za 2008
ASSESSMENT PROCESS
•
•
•
•
•
Apply integrated assessment strategy
Include Policy documents
Provide for RPL, Reassessment, Special
needs, language requirements
Identify the level of learner’s competence
Activities to include knowledge, skills &
attitudes
continue
www.roionline.co.za 2008
ASSESSMENT PROCESS
•
•
-
Identify methods:
Knowledge
Observation
3rd form of evidence
Design instruments:
Case study
Discussions
Portfolio
continue
www.roionline.co.za 2008
COLLECT THE EVIDENCE
Identify the sources to collect evidence from:
• Direct evidence
• Indirect evidence
• Historical evidence
www.roionline.co.za 2008
MARKING MEMORANDUM
Marking Memorandum is used to score the
outcomes of assessment
Include:
• Marks assigned per question
• The total marks of the assignment &
paper
• The marking memorandum can be
combined with model answers
www.roionline.co.za 2008
PURPOSE OF EVALUATION
• Establish the value of learning
• Decide to continue with a programme
• Obtain info on how to improve future
learning programmes
• Improve delivery mechanisms to be more
efficient and less costly
www.roionline.co.za 2008
WHAT IS EVALUATION?
“The collection of, analysis, and
interpretation of information about any
aspect of a programme of education or
training as part of a recognised process of
judging its effectiveness, its efficiency, and
any other outcomes it may have.”
(Mary Thorpe)
www.roionline.co.za 2008
ELEMENTS OF AN EVALUATION STRATEGY
•
•
•
•
Background: purpose
The scope of evaluation: levels & activities
Output: measure for evaluation
Methodology: integration with other
processes
continue
www.roionline.co.za 2008
ELEMENTS OF AN EVALUATION STRATEGY
• Work plan: roles & responsibilities
• Evaluation instruments: questionnaires,
etc.
• Resources: human, technology, budget
• Reporting & communication
www.roionline.co.za 2008
AREA OF MEASUREMENT
EXAMPLES
•
Customer support
•
Systems
•
Product knowledge
•
Technology
www.roionline.co.za 2008
MEASUREMENT IMPACT
•
Quality – How good? Standards? Criteria?
•
Quantity – How many?
•
Cost - What are the cost implications? Savings?
•
Time – How quick / long? When? Faster?
What can you increase / decrease?
www.roionline.co.za 2008
TRAINING COSTS
•
•
•
•
•
•
•
Gap analysis
Design & Development
Facilitation fees
Assessment
Evaluation
Fixed costs
Other costs
www.roionline.co.za 2008
EVALUATION MODELS
•
•
•
•
•
•
Kirkpatrick
Phillips
Nadler
Brinkerhoff
Holton
Coetsee and Van Zyl
www.roionline.co.za 2008
EVALUATION PROCESS:
DATA COLLECTION METHODS
•
•
•
•
•
•
•
Checklists
Questionnaires
Surveys
Needs Analysis
Performance Audit
Case studies
Focus groups
continue
www.roionline.co.za 2008
EVALUATION PROCESS:
DATA COLLECTION METHODS
•
•
•
•
•
•
Meetings
Interviews
Observation
Role plays
Simulations
Work samples
www.roionline.co.za 2008
ROI FORMULA
R
R
R
ROI% = (benefits – costs) x 100
___________
costs
R
www.roionline.co.za 2008
R
NON-TRAINING VARIABLES
External influences that had an impact on
the measurement are referred to as nontraining variables.
External influences such as new systems
and products, trends and business cycles
pose threats to validity when measuring
ROI.
www.roionline.co.za 2008
NON-TRAINING VARIABLES
•
•
•
•
Organisational influences
Technology
Human influences
Other influences
www.roionline.co.za 2008
REPORTING
•
•
•
•
•
•
•
Executive summary
Background of project
Purpose, intent and design of evaluation study
Results
Discussion
Costs and benefits
Conclusions,
recommendations
and options
www.roionline.co.za 2008