DESPFRWG 25 09 2012
Agenda Item 2 – DES Post-Program Monitoring Survey
DES Performance Framework Review Working Group
Agenda item:
Date of Meeting:
Title of Discussion Paper:
2
25 September 2012
Post-Program Monitoring Survey
Purpose

To inform the Working Group about the DES Post-Program Monitoring survey and the
collection of JSA participant experience data
Background
Post-Program Monitoring Survey
 The Post-Program Monitoring (PPM) survey has operated since 1987 and is primarily used to
measure the labour market status of job seekers who received assistance through DEEWRfunded employment programs, including DES, with around 400,000 surveys sent to job
seekers each year.
 PPM surveys are sent to DES participants throughout the year with more than 50,000
surveys sent in 2011–12 alone. Two PPM survey types are currently used for DES,
1) Employment Assistance/Post-Placement Support, and
2) Ongoing Support.
 DES achieves a relatively high 37.5 per cent response rate across both DES PPM survey types
(see Attachment A: Post-Program Monitoring survey overview for further details).
 During the development of the DES PPM surveys, the department engaged an external
consultant with experience in the disability services sector to undertake cognitive testing
with a range of DES participants to ensure the appropriateness of the questions asked of the
DES client base (see Attachment B: Cognitive Testing for DES PPM survey for further details).
Job Services Australia Participant Experience Measure (KPI 3)
 For Job Services Australia (JSA), the established PPM survey was selected as the most
appropriate method to collect data for the JSA Quality KPI measure of Participant
Experience. The PPM survey underwent cognitive testing and a pilot test to ensure the
appropriateness of the additional questions asked and that they were understood by
participants.
 Currently, the scores derived for the JSA KPI3 Participant Experience Measure are based on
job seeker responses to questions about the services they’ve received as set out in the JSA
Service Guarantee and the job seekers’ extent of satisfaction or dissatisfaction with their
Employment Pathway Plan being tailored to their needs. The method for calculating the final
scores for the JSA KPI3 Participant Experience Measure were developed by an external
consultant, with expertise in developing experience measures (see Attachment C: Job
Services Australia KPI3 Participant Experience measure for further details).
Issues

The production of robust DES provider-level PPM results is currently limited. Any increase to
the existing sample sizes will increase survey costs. Given that a number of DES Contracts
Page 1 of 6
DESPFRWG 25 09 2012
Agenda Item 2 – DES Post-Program Monitoring Survey
are relatively small, it cannot be guaranteed that all Contracts will be able to get a
statistically reliably and robust assessment through the PPM survey.
Other attachments



Attachment D: Sample DES PPM survey cover letter
Attachment E: Sample DES Employment Assistance/Post Placement Support PPM survey
Attachment F: Sample DES Ongoing Support PPM survey
FOR NOTING AND DISCUSSION
Page 2 of 6
DESPFRWG 25 09 2012
Agenda Item 2 – DES Post-Program Monitoring Survey
Attachment A: Post-Program Monitoring survey overview
The Post-Program Monitoring (PPM) survey has operated since 1987. The survey is used to measure
the labour market status of job seekers who received assistance through DEEWR-funded
employment programs with around 400,000 surveys sent to job seekers each year.
Aggregate survey results are used for:
 reporting against the effectiveness indicators for employment services in the Portfolio
Budget Statements and the department’s Annual Report
 reporting in the Labour Market Assistance Outcomes report
 input into the KPI 3 (Quality) under the Job Services Australia Performance Framework
 employment program evaluation
 policy development.
The PPM survey is conducted on an ongoing basis and every two weeks a new cohort of participants
is selected for inclusion in the survey. This approach makes the large sample easier to administer,
and allows seasonal variation to be accounted for. Additionally, it decreases the likelihood of running
out of samples as individuals enter and leave the system over the course of the year.
The following process has been adopted to achieve the highest possible response rate:
 A survey is mailed to participants with a reply paid envelope and a letter from the
department explaining the purpose of the survey and encouraging response
 Three weeks following the first mail-out, a reminder letter (including a replacement
questionnaire and reply paid envelope) is sent to participants who have not yet responded
 Three weeks following the second letter, non-responding participants are called and invited
to complete the survey over the telephone.
Special arrangements are in place for visually impaired participants with a paper-based survey sent,
allowing for them or a nominee to complete on their behalf. Reminder notices are not sent to
visually impaired participants, but they are called sooner if a survey hasn’t been received.
Participants are considered in-scope for inclusion in the survey three months after various events,
eg. participation in an activity, exiting a stream, exiting assistance, being placed in a job, etc... though
it varies between employment programs. Participants may be included in the PPM survey while they
are still receiving assistance from a provider. For DES:
 A one-in-three sample selection is applied with two survey types used:
1. A job seeker will be recorded as being in-scope for a DES Employment Assistance/Post
Placement Support PPM survey if they:
o exit from DES, including achieving a 26 week outcome and exiting as a fully
independent worker;
o achieve a 26 week outcome and progress into Ongoing Support; or
o reach 12 months in DES assistance without having progressed into Ongoing Support.
2. A job seeker will be recorded as being in-scope for a DES Ongoing Support PPM survey if
they:
o exit from DES (and were participating in the Ongoing Support phase); or
o reach 12 months in Ongoing Support.
Page 3 of 6
DESPFRWG 25 09 2012
Agenda Item 2 – DES Post-Program Monitoring Survey
More than 50,000 DES PPM surveys were sent in 2011–12 with relatively high response rate. More
than 37 per cent of Employment Assistance/Post Placement support surveys sent were completed
by participants. This level of response is largely consistent across most disability types. PPM surveys
sent to Ongoing Support participants have a slightly higher response rate of 41 per cent.
Tables A.1 and A.2 provide further detail.
Table A.1: DES Employment Assistance/Post Placement Support response rates – 2011–12
Disability type
Psychiatric
Response
rate
38.6%
9,835
Return to
sender
404
Total in-scope
population
83,110
Surveyed
Responses
25,914
Physical/diverse
34.7%
18,126
6,172
348
58,371
Intellectual
36.5%
1,998
715
41
6,700
Learning disability
31.6%
3,063
951
56
9,960
Sensory/speech
35.9%
2,842
1,007
37
9,126
All participants
37.1%
45,644
16,625
790
147,075
Total in-scope
population
8,413
Table A.2: DES Ongoing Support response rates – 2011–12
Disability type
Psychiatric
Response
rate
40.1%
Surveyed
Responses
2,270
898
Return to
sender
32
Physical/diverse
38.4%
1,758
665
24
6,361
Intellectual
39.5%
362
141
5
1,337
Learning disability
35.6%
481
167
12
1,757
Sensory/speech
38.5%
364
139
3
1,283
All participants
41.2%
5,586
2,268
78
19,980
Page 4 of 6
DESPFRWG 25 09 2012
Agenda Item 2 – DES Post-Program Monitoring Survey
Attachment C: Job Services Australia KPI3 Participant Experience measure
Given that the PPM survey is a well-established process, it was decided that gauging a measure of
JSA participant experience through the PPM survey was the most appropriate method.
Development
The department engaged an external statistical consultant, with expertise in developing experience
measures. The consultant advised on the appropriateness and weighting of the Quality KPI questions
to provide the best possible indication of the participant’s experience. To enable calculation of final
measures that are equitable to providers and allow the compilation of responses from different
streams, statistical methods are used. These sophisticated methods utilise international best
practice for these types of measures.
The survey questions and calculations of the final scores were developed by the consultant with
input from departmental statisticians. Furthermore, as the survey is used to measure participants’
experiences and is included in the Quality Framework data, detailed investigation about the actual
sample design, questionnaire design, participant recall and other methodological factors were taken
into consideration. This helps to ensure the collection is undertaken in a uniform manner, on a level
playing field, and that results obtained from the PPM Survey are statistically robust and consistent
for all providers.
The additional survey questions that were included in the PPM survey underwent a two stage
development phase. The development phase involved cognitive testing of the questionnaire and an
eight week pilot test of the entire survey process.
Assessment
The objective of the PPM survey is to be able to give accurate feedback of the receipt of services by
participants in accordance with the Service Guarantees and Code of Practice on an ESA level.
Providers receive two scores against the Participant Experience measure and are based on survey
responses for that Provider, derived from the:
 Participants surveyed at the ESA level that articulate they have received services as set out
in the relevant Service Guarantee; and
 Participants surveyed who state their EPP was satisfactorily tailored to their needs.
The two scores are calculated independently and are represented on a 0 to 100 point scale. It is
important to note that the scores are calculated through a statistical process, and are not
percentages.
The results obtained from the survey are used for comparative purposes allowing providers to
benchmark themselves against other providers and the industry as a whole. The scores are also
comparable over time, and across Job Services Australia contracts.
Specifically for the KPI3 Participant Experience measure, a ‘top up survey’ is conducted for
participants that may have not otherwise triggered a PPM survey to ensure that statistically valid
results are available for each provider in each ESA. This survey sampling methodology has also been
reviewed by an external statistical consultant.
Page 5 of 6
DESPFRWG 25 09 2012
Agenda Item 2 – DES Post-Program Monitoring Survey
The number of responses required for a particular provider at the ESA level varies depending on the
number of participants who receive services from that provider within the ESA. The department
only assesses providers on this measure where the results obtained through the survey are
statistically robust.
THE NEXT FEW QUESTIONS ARE ABOUT <PROVIDER NAME>
F1 Did <Provider Name> provide or offer to provide you with any of the following?
Yes
(Tick “No, not needed” if not provided and not needed.)
No
No, not
needed
Information about job vacancies in your local area






Access to computer facilities



Advice about how to look for work






Yes
No
No, not
needed















Help to prepare or update your resume
Access to counselling or other professional support
F2 And did staff at <Provider Name>:
(Tick “No, not needed” if not provided and not relevant for your situation.)
Clearly explain the types of help they will give you?
Discuss any jobs you have had in the past?
Discuss the types of jobs that you might like in the future?
Discuss your skills and education?
Discuss extra skills and education that might help you get work?
 Yes
 No  GO TO F5
 Don’t know  GO TO F5
F3 Did you sign a document called an Employment Pathway Plan (EPP) with
< Provider Name >?
(A sample EPP has been provided on the back of the attached letter.)
F4 How strongly do you agree or disagree that the Employment Pathway Plan (EPP) you signed with them was
tailored to you?
Strongly agree
Agree


Neither agree
nor disagree

Disagree

F5
Overall, how satisfied or dissatisfied are you
with the assistance they gave you in terms of:
Very
satisfied
Satisfied
Providing help suited to your circumstances?
Staff treating you as an individual?
Staff treating you with respect?






Overall quality of service?


Strongly
Don’t know
disagree


Neither
satisfied
Very
Dissatisfied
nor
dissatisfied
dissatisfied











Page 6 of 6
