Making it Real:
a Proposed Career
Readiness Measure
Svetlana Darche, WestEd
David Stern, UC Berkeley Graduate School of Education
Prepared for presentation at PACE Policy Seminar
Sacramento, CA
April 5, 2013
What we’ll talk about
 Context: SB 1458
 A proposed measure of career readiness
to include in new API
 Rationale for proposed measure
 Rating student performance at work
 Implementation issues
Context: SB 1458
 For high schools, up to 40% of API may consist of
information other than STAR and CAHSEE scores
 The non-test 40% must include graduation rates
 “the Superintendent, with approval of the state board,
may incorporate into the index for secondary schools
valid, reliable, and stable measures of pupil
preparedness for postsecondary education and career”
(EC 52052 (a) (4) (F) (ii))
PSAA Advisory Committee
framework for CCI
 On Feb 12 the Public Schools Accountability Act
Advisory Committee considered a framework for a
college and career indicator (CCI), recommended by
the Technical Design Group (TDG)
 CCI would be a composite consisting of a number of
different measures
 Each measure would be scored at five levels
 Each student’s score would be the highest score
attained on any of the measures
Definitional Premise
“Career readiness” in our view
means having the foundational
skills, knowledge, and
dispositions needed for long-term
career success – irrespective of
the postsecondary path taken to
get there.
CRPC Definition
A career-ready person effectively navigates pathways that
connect education and employment to achieve a
fulfilling, financially-secure and successful career. A
career is more than just a job. Career readiness has no
defined endpoint. To be career ready in our everchanging global economy requires adaptability and a
commitment to lifelong learning, along with mastery of
key academic, technical and workplace knowledge,
skills and dispositions that vary from one career to
another and change over time as a person progresses
along a developmental continuum. Knowledge, skills
and dispositions are inter-dependent and mutually
reinforcing.
~Career Readiness Partner Council
Endorsement
“The Common Core State Standards (Common Core) articulate what
students need to know and be able to do in English language
arts/literacy and mathematics for success in college and careers.
The college readiness benchmarks are the same as the career
readiness benchmarks in the Common Core State Standards. The
Career Readiness Partner Council’s statement contributes to the
national dialogue on career readiness by affirming that career
readiness includes the Common Core State Standards while
painting the broader picture of the range of knowledge, skills, and
dispositions that students must exhibit beyond literacy and
mathematics. Implementation of the Common Core State
Standards will be strengthened in schools, districts, and states by
looking at the Common Core in the context of the career readiness
definition outlined in this paper.”
~Gene Wilhoit, Executive Director, CCSSO
A proposed measure of
career readiness
Supervisor’s rating of a student’s performance in a
paid job, paid or unpaid internship, school-based
enterprise, or other qualifying experience that allows
application and demonstration of important careerrelated transferable skills.
Some guiding principles
 Simplicity – employ a measure that can apply to any student
 Validity – measure real career readiness, not a proxy
 Appropriateness of incentives – measure what we value
 Authenticity – measure actual performance
 Complementarity – add value and provide a full picture of
student performance
 Feasibility – employ a measure that can be captured before
students leave 12th grade
In accordance with these
principles, the rationale for
our proposed measure is
as follows:
Rationale for proposed
measure (1)
A direct measure of students’ career success would
be too late to be useful:
A direct way to determine whether a high schools’
students are well prepared for careers would be to
measure occupational success after students have
graduated. However, to give an accurate picture these
surveys would have to wait until graduates have reached
age 30 or so, which would be too late to inform current
practice.
Rationale for proposed
measure (2)
Our proposed measure offers appropriate timing and
face validity:
A student’s successful performance in a job, internship,
school-based enterprise, or other qualifying experience
can be determined at the end of senior year, and has face
validity as an indication that students have already begun
to develop career-related capabilities in real productive
settings.
Rationale for proposed
measure (3)
It measures skills that are not easily developed and
measured in classrooms, and that affect long-term
career success:
Working in a job, internship, or school-based enterprise
can develop abilities and behaviors that are not usually
developed in regular school classrooms (e.g.,
collaboration with people of different ages).
There is evidence that these abilities and behaviors affect
career success later in life, over and above the impact of
cognitive skills and years of schooling.
Rationale for proposed
measure (4)
Assessment of student’s performance must explicitly rate
important skills and behaviors:
The judgment that a student has performed satisfactorily in a
job, internship, school-based enterprise or other qualifying
experience must involve explicit indications of whether the
student has developed these work-related abilities and
behaviors. For instance, the NAF Supervisor Assessment form
lists 7 foundational skills, 7 applied workplace skills, 9 skills
reflecting self-management and personal responsibility, 4
indicators of knowledge of the field, as well as position-specific
technical skills that are named by the supervisor.
Rationale for proposed
measure (5)
Only high-quality work-based learning experiences
would count:
Including such work-related skills, abilities, and behaviors
in the definition of satisfactory performance ensures that
the job, internship, school-based enterprise, or other
qualifying experience has provided opportunities to
develop these.
Rationale for proposed
measure (6)
This measure gives schools incentive to provide
access to experiences that develop career readiness:
A high-quality work-based learning experience helps
students develop these abilities and behaviors that are
related to long-term career success.
Rewarding high schools where more seniors have
performed satisfactorily in a job, internship, school-based
enterprise, or other qualifying experience will encourage
schools to develop these forms of work-based learning as
part of the curriculum, and will thus promote development
of abilities and behaviors that contribute to students’ later
career success.
Rationale for proposed
measure (7)
It measures what can be validly measured:
It is better to measure students’ career-related abilities
and behaviors in the context of actual work performance
than by giving students a battery of tests. There is good
evidence that abilities and behaviors developed by late
adolescence strongly affect career success later in life,
but there has not been enough research to determine
exactly which of these abilities and behaviors are most
important, and in which combinations.
Research on predicting
long-term career success
 Years of schooling and cognitive ability are both strong
predictors of later earnings, but leave most variance
unexplained
 In addition, various abilities and behaviors, other than
knowledge of school subjects, also predict earnings
 Variables measured around age 20 that predict earnings
decades later include: planful competence, self-esteem,
locus of control, work habits, leadership skills, teamwork and
other sports-related skills, discipline problems in high school,
ability to function in a cohesive small group, independence,
persistence, initiative, outgoing character, and emotional
stability
Evidence that working
helps teenage development
 Skills and abilities develop through practice
 Working while in high school leads to higher earnings after
high school –– and does not interfere with academic
achievement if less than ~20 hrs/wk
 High school work experience has been found to help
develop positive work values, self-confidence, sense of
efficacy, planfulness, commitment, and identity formation,
among other qualities
 Quality of work experience matters
 School-supervised work experience has been found to offer
more opportunity than non-school supervised experiences to
learn new things, work in teams, perform a variety of tasks,
and take initiative
Rating student performance
at work
B. Applied Workplace Skills
Systems thinking
1
2
3
4
N/A
Creativity and innovation
1
2
3
4
N/A
Information technology application
1
2
3
4
N/A
Teamwork/collaboration
1
2
3
4
N/A
Ability to work with diverse individuals
1
2
3
4
N/A
Ethical behavior
1
2
3
4
N/A
Flexibility/adaptability
1
2
3
4
N/A
C. Self-Management and Personal Responsibility
Part II. Knowledge of the Field and Organizational Context
Please rate the student’s knowledge of the industry/field, occupation, and organizational context
according to the rating scale below.
1
Level of
Knowledge Falls
Below
Expectations
2
Level of
Knowledge
Approaches
Expectations
3
Level of
Knowledge Meets
4
Level of Knowledge
Exceeds
Expectations
Expectations
N/A
No
Opportunity to
Observe
Understands career opportunities/requirements in the
industry or field overall
1
2
3
4
N/A
Understands career opportunities/requirements in the
specific occupational area related to the internship or
student project
1
2
3
4
N/A
Understands the culture, etiquette, and practices of the
workplace or the project client’s organization and knows
how to navigate the organization
1
2
3
4
N/A
Knows how to interact with supervisors, clients, and
teammates
1
2
3
4
N/A
For each of the ratings above in Parts I and II, evidence (examples of performance) can be
provided for any rating in the online form; examples must be provided for lowest ratings
(skills rated at level 1) and highest ratings (skills rated at level 4). If using the paper form
to complete the assessment, please add required examples on additional pages marked to
coincide with the rated dimension.
Implementation issues
• Fitting the PSAA
framework
• Reliability
• Reporting
Fitting the framework
 This measure could be one of several included in the
CCI
 In our PACE brief “Making it Real: How High Schools
Can Be Held Accountable for Developing Students’
Career Readiness”
(http://www.edpolicyinca.org/publications/making-itreal-how-high-schools-can-be-held-accountabledeveloping-students-career-readiness), we proposed
using percentage of seniors who had obtained a
satisfactory performance rating from a work
experience, but scoring a student’s performance at
one of five levels is also possible
Reliability of supervisor
ratings
 In many work settings, performance appraisal is a
routine procedure, and must be sufficiently reliable to
withstand legal challenge
 Studies of inter-rater reliability of NAF supervisor
assessment have been proposed, including test of
whether training may improve reliability
 Judgments of five levels of performance may be less
reliable than judgment of satisfactory vs. nonsatisfactory
Reporting results to CDE
 Schools can report results as part of CALPADS: could
be called assessment of student performance in
demanding work
 Districts would keep records, just as they have kept
records for Work Experience in the past
 CDE could conduct occasional spot checks