Common Tools in Vocational
Evaluation for Marginalized
Client Populations
Phillip W. Boswell MA, B.Ed, RRP
R. Stephen Russell, MA. Sc, P. Eng
Learning Objectives







Introduction
Identify the difference between vocational
assessment and vocational evaluation
Recognize the three levels of vocational assessment
Ascertain the factors involved in the appraisal of
clients
Identify useful voc assessment/evaluation resources
Develop a working knowledge of common tools
used in vocational evaluations
Work samples
Assessment - Defined
Systematic procedures to obtain information
from a variety of sources to draw inferences
about people
(Standards for Educational and Psychological Testing, 1999)
Five Phases of Vocational Rehabilitation





Same job with the same employer.
Same employer, modified job (light duties)
Different employer, similar or closely related
job
Different employer, alternative employment in
the same or related industry
Development of new occupational skills
What can be assessed?

Functional skills (basic
skills & functional capacity)





Interest
Achievement
Aptitude
Intelligence
Learning style







Work Samples
Personality
Assistive technology
Neuropsychological
Sensory impairments
Career Assessments
Dexterity
Career maturity, competency, employment barriers, emotional and multiple intelligence,
job readiness, personal factors, personal style, prior learning and values
Factors in Appraisal of the Individual
Interests
Social &
Economic
Factors
Education
& Training
Acquired
Skills
Potential
Skills
Pearl E.
White
Leisure
Time
Activities
Physical
Capacities
Personal
Traits
Vocational Evaluation Tools
Interest
Inventories
Dexterity
Tests
Achievement
Tests
Work
Samples
Aptitude
Tests
Pearl E.
Gates
Neuropsychological
Assessment
Functional
Skills
Assessment
Personality
Assessments
Individual Appraisal/Tools
Interest
Inventories
Interests
Social &
Economic
Factors
Education
& Training
Acquired
Skills
Potential
Skills
Pearl E.
White
Leisure
Time
Activities
Physical
Capacities
Personal
Traits
Achievement
Tests
Work
Samples
Aptitude
Tests
Dexterity
Tests
Pearl E.
Gates
Neuropsychological
Assessment
Functional
Skills
Assessment
Personality
Assessments
Vocational Assessment - Defined
A general term for the process of identifying
and appraising an individual’s level of
functioning in relation to vocational
preparation and employment decision making
Why Vocational Assessment?

Clients often do not have precise
occupational goals and aspirations. They
frequently have insufficient or inaccurate
information on the labour market and limited
self-knowledge of their own interests and
abilities.
Purpose of Vocational Assessment



Plan a course of action
Enhance client self-knowledge and vocational
decision-making abilities
Predict realistic employment outcomes that
result in successful client vocational
rehabilitation
Role of the Rehabilitation Professional

Collect enough information or ‘data’ about the
client to diagnose and make predictive
statements about his/her potential to obtain a
successful rehabilitation outcome
Levels of Vocational Assessment
Level 1 – Screening/Needs Assessment
Level 2 – Clinical or Exploratory
Level 3 – Comprehensive Vocational Evaluation
Level 1 – Screening/Needs Assessment
Initial Process


Needs Assessment
Determine what is necessary to develop a plan of
action (e.g. vocational plan)
Level 1 – Screening/Needs Assessment
Includes



Interview with client
Collect routine background information (demographics)
Reliance on subjective consumer statements





Vocational choice/interest
Self-estimates of competence
Reported work history
Functional Assessment (e.g. personal capacity questionnaire)
Limited, if any, standardized testing (e.g. interest)
Level 2 – Clinical or Exploratory
Intermediate process
In depth exploration or case study approach to the client and
vocationally related circumstances. Builds on level 1 information
through the use of:
 Additional interviews
 Collect/analyze documents (schools records, medical records, etc.)
 Career exploration
 Vocational and/or adjustment counselling
 Psychometric/standardized testing
 Transferable skills analysis
 Computerized job matching
 Job Analysis and/or environmental assessment
 Assistive technology considerations
Level 3 – Vocational Evaluation
“Final” Process



Comprehensive individualized holistic process of assessment
that utilizes specific instruments, procedures, and behavioural
observation
Designed to measure, observe, and document interests,
values, temperaments, work-related behaviours, aptitudes,
skills, and physical capacities, in order to predict viable
employment and/or training outcomes
Used when more in depth information about the client is
necessary and not available from information in level 1 and 2
Level 3 – Vocational Evaluation
Real or simulated work used as the focal point of the evaluation
 Systematic observation and recording of work behaviour &
performance
 Occurs over time and uses multiple methods and techniques to
validate finds. Some combination of the following methods are used:
 Work samples, job samples
 Situational assessments, community based assessments
 Standardized testing
 Interview
 Transferable skills, job matching
 Analysis of background information
 Career exploration/job shadow
 Assistive technology considerations
 Prescriptive recommendations
Distinction Between VA & VE
VE is holistic: considers disability/medical, psychological, social, vocational,
educational, cultural and economic issues
Ideally conducted by a certified vocational evaluator (CVE)
VE occurs over time (days/week +)

evaluated over time with varying work-related demands and environments
Systematic Behaviour Observation and Recording

Work performance

Work behaviour
Adaptive and/or transferable skills are questionable or unknown
Work is the specific focus of VE

Work samples

Situational assessments

Community-based assessments

Occupational resources

Career exploration
If we are facing in the right
direction, all we have to do is
keep on walking
Buddhist proverb
Vocational evaluation identifies….




Transferable skills
Vocational Interests
Worker Characteristics
and Behaviours
Academic Abilities and
Potential



Learning Styles
Work Aptitudes and
Abilities
Suitable Employment
Alternatives
Evaluating Assessments
When assessing an individual, the first question
is always……Should we use tests at all?
Testing may not always be appropriate
 Extreme anxiety
 Sensory impairments
 Minimal education
 Limited language abilities
Questions to ask




Is the test reliable (repeatable) and valid
(does it measure what its suppose to?)
Does my client meet the reading level
required by the test?
Is my client disadvantaged in any way by
completing this test?
Will the test provide the information that I am
looking for?
Questions to ask



Can the test administration procedures be
modified to accommodate the client?
Can clients answer the question on the test
form rather than a separate score sheet?
Are all the questions relevant and correct for
my client?
Questions to ask



Are there relevant norms/criteria to compare
my client’s performance?
Issue in rehabilitation is the lack of
representation of marginalized individuals in
sample populations
Marginalized client interested in competitive
employment is better served using “general
working population norms” rather than
“disabled norms”
Transferable Skills




Can be achieved through formal or informal
training
Is work related
Should have a measurable component
Should have the capacity to be matched to
other/alternative work options
Vocational Interests

Subjective measure of an individual’s
preferences in work activities

Can add validity by incorporating variety of
surveys with similar outcomes

Does not confirm that skills exist
Worker Characteristics and
Behaviours



Vocational Evaluation relies heavily on the
observational skills of the Counsellor
The Counsellor “takes on” different roles
throughout the evaluation
Allows the inclusion of subjective information
in the process
Academic Abilities and Potential


Understanding current skills
Can evaluate need for upgrading



Can predict potential


To competitively seek employment
To pursue further training
Timed vs. untimed testing
Needed to insure proper skill profile is
created
Learning Styles

Generally subjective in nature

Can be helpful in identifying training
environment

Can be helpful in identifying suitable work
environment
Work Aptitudes and Abilities





Computerized assessment
Work Samples
Academic Achievement
Aptitude testing (paper/pencil)
Work history (transferable skills)
All are used to create a composite profile
Worker Qualification Profile

General Education Development (GED)


Reasoning
Math
Language
Aptitudes

General Learning, Verbal Aptitude, Numerical
Aptitude, Spatial Aptitude, Form Perception,
Clerical Perception, Motor Co-ordination, Finger
Dexterity, Manual Dexterity, Eye-Hand-Foot Coordination, Colour Discrimination
General Education Development-GED

Embraces those aspects of education (formal
or informal) which contribute to a a worker’s
reasoning development and ability to follow
instructions; and, the acquisition of “tool
knowledge” such as language and
mathematical skills. (Revised Handbook for Analyzing
Jobs - 1991)
GED FACTORS



Reasoning Development
Math Development
Language Development
1 to 6 scale
1 = Low
6 = High
Aptitudes


The capacity to acquire proficiency in and
activity with a given amount of formal or
informal training
Aptitudes may be general, such as learning
aptitude or special, such as mechanical
aptitude (Pruitt, 1986)
Aptitudes







General Learning
Verbal Aptitude
Numerical Aptitude
Spatial Aptitude
Form Perception
Clerical Perception
Motor Co-ordination




Finger Dexterity
Manual Dexterity
Eye-Hand-Foot Coordination
Colour Discrimination
1-5 scale
1=High 5=Low
The Tools of Vocational Evaluation






Interest Surveys
Academic Achievement and Aptitude Testing
Computerized Assessment
Work Samples
Observation
Other Surveys
Interest Surveys






Strong Interest inventory (SII)
Canadian Work Preference Inventory (CWPI)
Jackson Vocational Interest Survey (JVIS)
Career Occupational Preference System
Interest Inventory (COPS)
Reading Free Vocational Interest Inventory: 2
(R-FVII:2)
Career Assessment Inventory (CAI)
Ability=Achievement+Aptitude

Achievement Tests - What a person has
already done!



Canadian Academic Achievement Test (CAAT)
Wide Range Achievement Test (WRAT-4)
Aptitude Tests – What a person has the
potential to do!


General Aptitude Test Battery (GATB)
Differential Aptitude Test (DAT)
Work Samples






Simulated work activity
Criterion Referenced Test (CRT)
Method Time Measurement (MTM)
Standardized tasks
Directly related to aptitude factors
Valpar has over 30 work samples for
functional and job specific assessments
Valpar Work Samples

VCWS03 Numerical Sorting

VCWS06 Independent Problem Solving

VCWS07 Multi-Level Sorting
Norm-Referenced test & scores



Compares the evaluee’s performance to the
performance of members of a particular norm group
Selected norm group population should be well
defined, relatively current and appropriate to the
evaluee’s goal
Results are reported in terms of percentile rankings
Norms

Frequently referred to as ‘moving targets’ as
an individual’s ‘relative’ rank can shift
significantly depending on the norm group
Pearl scored at the 87th percentile in arithmetic
(grade 4 students)
Pearl scored at the 11th percentile in arithmetic
(grade 10 students)
When norms are not enough

The world of work expects people to be able
to perform some job tasks based on criterion
of success or precision rather than compared
only to how well others perform

Expect airline pilots to take off and land an
airplane with 100% level of success, not just a
little better than their pilot classmates!
Criterion references tests & scores


Criterion-referenced tests have a predetermined
external standard or criterion
Work related criterion generally come from industry
standards





Number of widgets assembled per hour
Keyboarding speed 60 wpm 0 errors
MTM (Valpar)
Measures a specific knowledge, skill, or trait and
compares person’s performance against the
criterion
Focus and importance of on performance rather
than relative rank of how others performed
Norm vs criterion

The “Standards for Educational and
Psychological Testing” provide a basis on
which to evaluate tests. established by APA, National Council on
Measurement in Education, & American Education Research Association

The samples used for test validation and
norming must be of adequate size and must
be sufficiently representative to substantiate
validity statements, to establish appropriate
norms, and to support conclusions regarding
the use of the instrument for the intended
purpose
A Huge Issue---Norm vs. Criterion



Most norm referenced tests do not use
representative norm samples that include
individuals with disabilities
This often results in labels of disability and
the identification of weakness
Important to note that in some instances it is
helpful to understand how an individual is
functioning compared to the “average”
population when assessing for competitive
employment
Norms




Competitive: compares
people to other people
Relative skills reported
Content difficulty fixed
High scores inferred
from high performance
on tasks with lower
scores
Criterion




Non-competitive:
compares people
directly to jobs
Actual skills reported
Content difficulty
adjusts to evaluee’s
skills
Scores measured
directly at
corresponding content
difficulty
Norms



Validity depends on
appropriateness of
norm group
Strictly valid only for
members of the norm
group
Translation from
percentiles by
traditional methods
distorts DOT levels
Criterion



Validated by Method
Time Measurement
(MTM). No norm
groups
Accuracy verified by
data from over
thousands of
assessments
Measures DOT levels
directly: no translation
required
Best Norm Group (when available)

Norm to the environment you are predicting to




e.g. employed workers, general working population,
general
population, education, training norms
62nd percentile Form & Spatial GATB (GWP)
12th percentile MN Paper Form Board (employed drafting
technicians)
Obviously, criterion-referenced tests are better
indicators of performance, so if you have a choice
between a norm referenced and criterion referenced
measure – choose criterion-referenced.
Computerized Assessment




Bridges Ability Profile Multi-Aptitude
Assessment
Bridges Interest Profiler
Career Cruising - Career Matchmaker
Valpar Professional (PRO 3000)
(Magellan and Aviator)
PRO3000








System Manager
Computerized Assessment (COMPASS) with 3
specialized work samples
DOT Descriptions and database
Work History for transferable skills analysis
Pictorial/Audio Interest Survey
Spatial/Non-verbal
PET Survey
Norm Referenced Test Conversion (TECO)
Suitable Employment Alternatives



Counsellor directed selection of vocational
options
Objective listing based on testing results
Can be combined with career exploration
The Client Perspective






Generally 3 to 6 hours
Includes intake session
Pre-screening
Academic achievement testing
Interest / Aptitude Testing
Discussion of test results
VALPAR International








Started in 1973 in Tucson, Arizona
Product Developer with over 6000 customers world
wide
Component Work Samples Series (VCWS)
JOULE Functional Capacity Evaluation
PRO3000 Professional Vocational Assessment
MAGELLAN Career Self Assessment/Explorer
AVIATOR Aptitude & Interest Assessment
SIGI PLUS Values-Based Career Planning
Career Systems Canada Limited
Steve Russell
P. O. Box 255
Buckhorn, ON. K0L1J0
tel: 705 657-8646
fax: 705 657-1572
cell: 416.721.5759
steverussell@necicom.net
www.valparint.com
P.W. Boswell & Associates
Phil Boswell
9534 Riverbend Road
Black Creek, BC V9J 1E8
tel 250.337.5087
fax 250.337.5867
toll 877.790.7074
info@employment-training.org
www.employment-training.org