Assessing the Impact of Labour Market Information: Progress and

advertisement

Assessing the Impact of Labour

Market Information: Progress and Challenges

A Model and Sample Research

Design

Kris Magnusson

Lynne Bezanson

Reginald Savard

A Challenge by Canadian

Policy Makers:

“You haven’t made the case for the impact and value of career development services”

• A research team formed in 2004 to follow-up on recommendations from “Working Connections”:

– The Canadian Research Working Group for Evidence-

Based Practice in Career Development

LMI Impact Assessment 2

What is the CRWG?

• A pan-Canadian group of researchers (8 researchers; 6 universities; coordinated by CCDF)

• Formed as a recommendation from the pan-

Canadian Symposium on Career Development,

Lifelong Learning and Workforce Development

• Working on projects which advance the evidence base for career development practice

• Part of an international network on evidence based practice

LMI Impact Assessment 3

First Project: State of

Practice Survey

• What is the impact of career services?

• What evidence do we have of that impact?

LMI Impact Assessment 4

A Problem with Terminology

• Funders, including governments, often act as if they are measuring job placement as the only measure of career accomplishments

• Most practitioners believe in and provide services for career development

• Example: Let’s measure the impact of libraries in our school system

LMI Impact Assessment 5

Challenge

• The general goal of career services is client change

• There are a multitude of factors that may be related to client change

• Very few of those factors are ever directly measured

LMI Impact Assessment 6

What outcomes are being achieved that are going unreported or unmeasured?

– Client empowerment

– Client skill development (e.g., personal selfmanagement skills),

– Client increased self-esteem,

– Client changes in attitudes (e.g., about their future, or about the nature of the workforce),

– Client knowledge gains

– Financial independence

– Creation of support networks

– More opportunities for clients and marginally:

– Community benefits;

– Client satisfaction; Increased queries;

– Political lobbying (agency) 7

What Did We Learn?

1.

Agencies, practitioners, policy makers and employers agree:

– Impact assessment is important

– Current evaluation practices are inadequate

– Important outcomes are not measured and reported

– Want sophisticated evaluation procedures

2.

The importance of evaluation is related to the type of organization providing services

LMI Impact Assessment 8

Key question for LMI …

• What role does LMI play in attaining the

“measured” outcomes, and also in influencing all of the “unreported” or

“unmeasured” outcomes?

LMI Impact Assessment 9

CRWG Response 1:

Create a Framework for

Evaluation

General Approach to

Evidence-Based Practice

Input  Process  Outcome

Context

LMI Impact Assessment 11

General Approach to

Evidence-Based Practice

Input  Process  Outcome

Indicators of client change

1. Learning outcomes

2. Personal attribute outcomes

3. Impact outcomes

LMI Impact Assessment 12

General Approach to

Evidence-Based Practice

Input  Process  Outcome

Indicators of client change

1. Learning outcomes

• Knowledge and skills linked to the intervention

2. Personal attribute outcomes

• Changes in attitudes,

• Intrapersonal variables (self-esteem, motivation),

3. Impact outcomes

• Impact of #1 & #2 on client’s life, e.g., employment status, enrolled in training, etc.

• Social and relational impact

13

General Approach to

Evidence-Based Practice

Input  Process  Outcome

Resources available

1. Staff

• number of staff, level of training, type of training

2. Funding

• Budget,

3. Service guidelines

• Agency mandate.

LMI Impact Assessment 14

General Approach to

Evidence-Based Practice

Input  Process  Outcome

Activities that link to outputs or deliverables

1. Programs offered by agency

• Number of staff,

• Level of training,

Type of training

2. Interventions used by service providers

• Skills used by service providers

• Home practice completed by clients,

3. Involvement by 3 rd parties

4. Quality of service indicators

Stakeholder satisfaction.

15

General Approach to

Evidence-Based Practice

Input  Process  Outcome

Intervention

=

Process + Outcome

LMI Impact Assessment 16

General Approach to

Evidence-Based Practice

Input  Process  Outcome

Impact

=

Quality of Input +

Effectiveness of Implementation

LMI Impact Assessment 17

CRWG Response #2:

Sample Project that Implements the Framework: LMI Impact

Research Design

Background to CRWG Involvement in LMI

Project

• Evaluation of HRSDC LMI Products and

Services (HRSDC, January, 05)

• Impact of LMI on the Career Decision-

Making Process Literature Review

(Université de Sherbrooke, September, 05)

• Specific recommendations from the

Literature Review

• Meeting of two members of FLMM LMI

Working Group and the CRWG (Calgary,

February, 06)

LMI Impact Assessment 19

CRWG General Research Question:

What clients, under what circumstances, use and/or benefit from what LMI?

• Who seeks and why?

• How is LMI converted into personally relevant information?

• How is personalized information used?

• What are the impacts (learning, attitudinal, behavioural) of applied personalized information?

LMI Impact Assessment 20

Research Focus:

• Known:

– Unassisted LMI (computer assisted excepted) appears to have minimal impact on career decision making

– LMI accompanied by counsellor intervention

(computer assisted included) appears to have significant impact on career decision making

• Unknown:

– What is the winning combination of LMI and counsellor intervention (both quality and quantity) which results in significant positive impact on career decision making and gives the best return on investment?

LMI Impact Assessment 21

Three Core Research

Constructs:

1. Identify employability dimensions

• What is a working framework that can be used to identify and sort client needs and what LMI resources are available to meet each need (inputs)?

2. Apply Evaluation Framework

• What specific outcomes can be used as measures of

“success”?

3. Process of Intervention: Continuum of

Assistance

• What impact does level of help provided to clients have?

LMI Impact Assessment 22

Construct #1:

Employability Dimensions and Client

Needs

Client needs for LMI can be identified by using a framework that includes:

– Career Exploration and Decision Making

– Skill Enhancement

– Job/Work Search Enhancement

– Employment Maintenance

LMI Impact Assessment 23

LMI Alignment to Needs:

• Career Decision Making and Career Exploration:

– www.JobFutures.ca

: Know Yourself Quiz

– www.CanLearn.ca

: On-line career counsellor

– www.Jobset.ca

: 10 Steps to better decision-making

– www.ALIS.gov.ab.ca

: OCCInfo: Information on over 530 occupations

– www.careerccc.org

: Information on occupations requiring post-secondary education other than university

• Skill Enhancement:

– www15.hrdc-drhc.gc.ca

: Essential Skills: Case Studies for over 150 occupations

– www.CanLearn.ca

: Programs Search; Schools Search

• Job/Work Search:

– www.Jobset.ca

: Jobs Site – Job Banks and Job Tips

– www.Monster.ca

: Guides to interviews, resume preparation

• Employment Maintenance:

– www.Monster.ca

: Career Centre: Managing your Career articles

– www.CCDF.ca

: Circuit Coach Modules: Tips on job maintenance; Adding value to your employer; Workability Skills; How to stay employed.

LMI Impact Assessment 24

Construct 2: Apply These Dimensions to the Evaluation Model

• For each dimension, what are the outcomes that should be/could be measured?

LMI Impact Assessment 25

Linking Outcomes for Each Dimension:

Career Decision Making:

– Establish priorities and preferences and use past experience to inform current decisions (K)

– Identify transferable skills (K)

– Recognize and accurately assess work opportunities

(B)

– Understand personal strengths and skills and make connections to work choices (K+A)

Job Search:

– Demonstrate knowledge and skills in a number of job seeking strategies (K+B)

– Demonstrate skills to identify job openings and work opportunities (K+B)

– Cope with stress of job/work seeking (A)

LMI Impact Assessment 26

Linking Outcomes (cont.):

Skills Enhancement:

– Understand requirements and how to qualify for and access specific skill training (K)

– Explore and undertake learning activities to support learning and increased employability (K+B)

– Explore and identify skills and knowledge which are transferable from one work role to another (K+B)

Employment Maintenance:

– Demonstrate ways to deal effectively with stress (A+B)

– Understand how changes (technology, demand) influence requirements for work success (K+A)

– Demonstrate self-management skills in times of transition (job shifts, unexpected unemployment) (B)

LMI Impact Assessment 27

Construct 3: Process of

Intervention

• Two categories of service:

– Non- or minimally assisted LMI; and

– Assisted LMI

• Self-serve clients not included in design

• LMI is not treated as an isolated variable; rather, the experimental factor is the “treatment” of LMI

LMI Impact Assessment 28

Variables:

• Psychosocial and sociodemographic variables (e.g., educational levels, work attachment, reasons for seeking assistance, level of decidedness, optimism, skills with information)

• Employability needs (e.g., from dimensions)

• Criteria variables (e.g., outcome measures)

LMI Impact Assessment 29

General Experimental Design

• Repeated pre-test post-test (Impact over time of using LMI)

• Within group and between group

(assisted vs non-assisted) differences

• Tests of relationship between demographic/need variables and criteria variables

• Large N (400 to 600)

LMI Impact Assessment 30

Data Collection and Intervention:

Both Groups

• Questionnaire: to capture psychosocial and sociodemographic variables to provide indicators of cognitive complexity (i.e. educational levels, work attachment, reasons for seeking assistance, level of decidedness, optimism, skills with information)

• Employability Needs Assessment to determine LMI needs

• Random assignment to non-assisted or assisted group

• Take-home booklet with specific LMI websites/resources highlighted according to identified needs

• Orientation to using the self-serve information services

• Semi-structured telephone interview

72 hours after orientation (first data collection on outcome measures)

LMI Impact Assessment 31

Data Collection and Intervention:

Non-Assisted Group

• Follow-up (one month and 3 months after first telephone interview) semi-structured interviews to gather information on client change - processes and outcomes

(e.g., how clients are using LMI websites/resources; impact of LMI; what else clients are doing in addition to specific LMI usage; learning, attitudinal, behavioural changes, progress on specific employability dimensions, progress towards career/educational/employment goals)

LMI Impact Assessment 32

Data Collection and Intervention:

Assisted Group

• Within one month of initial interview, conduct 2 followup interventions with counsellor

– to continue to work towards goal,

– make best use of LMI matched to employability dimensions, and

– problem-solve/coach on issues related to making best use of

LMI resources

• Follow-up (one month and 3 months after first telephone interview) semi-structured interviews to gather information on client change - processes and outcomes

LMI Impact Assessment 33

Research Design Summary

LMI Needs

Assessment

LMI

Treatment

Outcomes

T1 (72 hrs)

Outcomes

T2 (1 mth)

Outcomes

T2 (3 mth)

Questionnaire Booklet of

LMI to

Needs

Nonassisted

Random

Assignment

Counsellor

Assisted

Interview Interview Interview

D ecision,

S kills,

W ork Search

M aintenance

D ecision,

S kills,

W ork Search

M aintenance

D ecision,

S kills,

W ork Search

M aintenance

D ecision,

S kills,

W ork Search

M aintenance

D ecision,

S kills,

W ork Search

M aintenance

D ecision,

S kills,

W ork Search

M aintenance

LMI Impact Assessment 34

Inferences may be made about:

• Impacts of client variables including level of cognitive skills on use and usefulness of LMI;

• Information on the reasons why individuals seek out LMI;

• Information about how adults use LMI;

• Specific information on the ways in which assisted and nonassisted LMI users actually use LMI;

• Impacts of two levels of assisted and unassisted LMI on different employability dimensions;

• Impacts of LMI on client learning, attitudinal and behavioural outcomes including short term (one and three months) specified economic and social indicators (e.g., time spent on work-search; training enrolment/completion; employment; time on benefits).

LMI Impact Assessment 35

Thank you!

Kris Magnusson kris.magnusson@uleth.ca

Download