Pradeep Kotamraju, Ph.D.
Deputy Director
Principal Investigator
Oscar Aliaga Ph.D.
Program Director
Emily Dickinson
Graduate Assistant
The work reported herein was supported under the National Research Center for Career and Technical Education,
PR/Award (No. VO51A070003) as administered by the Office of Vocational and Adult Education, U.S. Department of
Education. However, the contents do not necessarily represent the positions or policies of the Office of Vocational and
Adult Education or the U.S. Department of Education and you should not assume endorsement by the Federal
Government.
Figure 1:The Work So Far
Technical
Skills
Year 2
Year 3
Year 4
Developed
Inventory
Template
Populated
Inventory
Template
with Data
Submitted
by States
Linked 3rd
Party
Industry
Certification
to LMI
Crosswalk Validation
NCES Data Analysis
ROI
Neutral
Intermediary
Built a Common
Postsecondary CTE
Data Dictionary
Analyzed Data To
Test Common
Postsecondary CTE
Data Dictionary
An NRCCTE Report:
Common
Postsecondary CTE
Data Dictionary
Jointly with the
Center for
Education and the
Workforce (CEW),
Georgetown U,
Producing a Report
on 2018
Employment
Projections by
Career Clusters
Community College
Completions Using
IPEDS Data;
Published in
Community College
Journal of Research
& Practice
Working on NCES
Restricted Sample
Survey Data to
Analyze CTE
Engagement,
Achievement, and
Transitions
Made a Presentation
on ROI at the ACTE
Annual Conference
Produced a Brief on
ROI for the USA
TVET Consortium
Developing an ROI
Guidebook
Chaired NSWG on
POS Measurement,
Accountability, and
Evaluation
Figure 2:The Work For Year 5 (and Beyond?)
Technical
Skills
Crosswalk
Validation
Produce a
Final
Inventory
Develop and
Validate
Existing
OVAE/Career
Clusters
Crosswalk
Conduct Two
Sub-Analyses
(one on Soft
Skills and the
Other on
Technical Skills
and Connect to
CTE
Engagement,
Achievement,
and Transitions
Using State-level
Data
State-Level
Analysis on
2018
Employment
Projections by
Career Clusters/
Pathways
NCES Data Analysis
Continue Work on
NCES Restricted
Sample Survey Data
to Analyze CTE
Engagement,
Achievement, and
Transitions
ROI
Neutral
Intermediary
Refining the ROI
Guidebook
As requested by
Center Partners
and by External
Stakeholders
Produce
Presentations,
Reports,
Monographs, Journal
Article Submissions
Preliminary
Analysis of Using
Existing Data Sets
to Develop ROI for
CTE
Technical Skills Assessment
 In Center Years 2 and 3 (August 1, 2008-July 31, 2010), data collection began and
continued in which states were asked to submit information that corresponded
with the elements of the template. Fourteen states submitted information that
has been reformatted and entered into the template. Analysis of the
information points to the following:
 The top seven program areas (based on two-digit CIP codes) are IT, Health, Business,
Manufacturing, Construction, Communications, and Agriculture.
 Third-Party Certification/Technical Exams, State-Developed Program of Study
Exams, and State-Developed End-of-Program Exams appear to be most commonly
used in states.
 Outstanding issues to be resolved include:
 Status of Postsecondary TSAs (no information has been provided as yet to the
Center).
 Program of Study level assessments – the extent to which the postsecondary partner
has a role and input.
 The shape, form and extent of the relationship of states with third-party TSA
organizations.
4
Technical Skills Assessment
In Center Year 4(August 1, 2010-July 31, 2011), the NRCCTE was requested
by the Office of Vocational and Adult Education (OVAE), U.S.
Department of Education (USDE) to connect industry skill certifications,
currently being offered within different state educational programs, to
available labor market information (LMI).
 The analysis required obtaining data on industry skill certifications,
educational program and CIP codes, labor market information, and
a crosswalk connecting CIP codes and occupational (SOC) data.
The methodology used:
 Converted the data states submitted to NRCCTE as part of the
technical skills assessment (TSA) inventory template project into a
database that could be used in the analysis.
 The template data was collected in a Microsoft Excel spreadsheet.
The spreadsheet information was reformatted so that it could be
imported into a Microsoft Access database.
5
Technical Skills Assessment
 The crosswalk information, which was only available as a
Microsoft Excel spreadsheet, was also reformatted for
import into the same Microsoft Access database.
 The wage information was downloaded as a Microsoft Excel
workbook from the U.S. Department of Labor and
appropriate wage information was reformatted for import
into the same Microsoft Access database. A database with
all connected information was then assembled.
The database contains 139 third-party, industry-based
technical skill certifications, 111 secondary education
programs, 56 CIP codes, and 110 SOC codes for the five
states.
6
Tables 1: Relating CIP and Occupation Families: No. of 3rd Party Industry Certifications
CIP Family
AGRICULTURE, AGRICULTURE
OPERATIONS, AND RELATED
SCIENCES.
Arts, Design,
Healthcare
Entertainment, Construction
Practioner and
Sports and and Extraction
Technical
Media
1
Healthcare
Support
Installation,
Maintenance,
and Repair
1
1
Production
4
2
2
10
112
1
5
2
8
13
18
8
1
3
1
6
3
2
HOMELAND SECURITY, LAW
ENFORCEMENT,
FIREFIGHTING AND RELATED
PROTECTIVE SERVICES.
9
4
MECHANIC AND REPAIR
TECHNOLOGIES/
TECHNICIANS.
PERSONAL AND CULINARY
SERVICES.
5
31
PRECISION PRODUCTION.
10
127
4
24
5
4
7
145
1
HEALTH PROFESSIONS AND
RELATED PROGRAMS.
All CIP Family
Transportation Total Industry
and Material
Skill
Moving
Certification
22
7
FAMILY AND CONSUMER
SCIENCES/HUMAN SCIENCES.
TRANSPORTATION AND
MATERIALS MOVING.
VISUAL AND PERFORMING
ARTS.
Protective Sales and
Services
Related
22
CONSTRUCTION TRADES.
ENGINEERING TECHNOLOGIES
AND ENGINEERING-RELATED
FIELDS.
Personal
Care and
Service
2
BUSINESS, MANAGEMENT,
MARKETING, AND RELATED
SUPPORT SERVICES.
COMMUNICATIONS
TECHNOLOGIES/
TECHNICIANS AND SUPPORT
SERVICES.
COMPUTER AND
INFORMATION SCIENCES AND
SUPPORT SERVICES.
Office and
Administrative
Support
1
2
7
22
22
1
1
7
38
33
5
61
81
4
3
2
13
11
4
317
7
Table 2: Relating CIP and Occupation Families: Average Entry-Wage for 3rd Party Industry Certifications
CIP Family
AGRICULTURE,
AGRICULTURE
OPERATIONS, AND
RELATED SCIENCES.
Arts, Design,
Healthcare
Construction
Entertainment,
Practioner and
and Extraction
Sports and Media
Technical
$19,630
Healthcare
Support
Installation,
Maintenance,
and Repair
Office and
Administrative
Support
$17,190
Personal
Care and
Service
Production
Protective Sales and
Services
Related
$16,010
Transportation
and Material
Moving
Total Wage
$15,680
$17,128
$15,730
$21,162
BUSINESS, MANAGEMENT,
MARKETING, AND RELATED
SUPPORT SERVICES.
COMMUNICATIONS
TECHNOLOGIES/
TECHNICIANS AND
SUPPORT SERVICES.
COMPUTER AND
INFORMATION SCIENCES
AND SUPPORT SERVICES.
CONSTRUCTION TRADES.
ENGINEERING
TECHNOLOGIES AND
ENGINEERING-RELATED
FIELDS.
FAMILY AND CONSUMER
SCIENCES/HUMAN
SCIENCES.
$20,210
$24,835
$22,920
$18,195
$18,860
$23,925
$25,453
$23,925
$29,249
$19,390
$29,448
$24,620
$16,230
$21,610
HOMELAND SECURITY, LAW
ENFORCEMENT,
FIREFIGHTING AND
RELATED PROTECTIVE
SERVICES.
MECHANIC AND REPAIR
TECHNOLOGIES/
TECHNICIANS.
$19,322
$16,230
$20,410
$20,447
$23,990
$28,922
$23,005
PERSONAL AND CULINARY
SERVICES.
$22,344
$23,298
PRECISION PRODUCTION.
$26,104
$24,620
$22,270
$25,885
$24,620
HEALTH PROFESSIONS AND
RELATED PROGRAMS.
TRANSPORTATION AND
MATERIALS MOVING.
VISUAL AND PERFORMING
ARTS.
All CIP Family
$19,088
$26,826
$27,220
$18,256
$26,127
$22,563
$16,120
$25,373
$16,890
$20,094
$21,820
$21,820
$29,500
$21,610
$23,990
$21,517
$23,990 $15,705
$18,855
$24,820
$25,174
$24,620
$21,833
8
Crosswalk Validation Project
 The NRCCTE convened a workgroup consisting of researchers, data experts, and
policymakers who met on January 12th and 13th, 2010, at the University of Louisville.
The purpose of this meeting was to capture the evolving multi-state, multiinstitutional collaborative efforts seeking greater consistency and clarity in Perkins
secondary and postsecondary data collection and reporting, The goal of this
meeting was to continue to move toward a common data crosswalk that links
educational programs to occupations with career clusters and career pathways and
create a foundation for more standardized accountability requirements in later
iterations of the Perkins legislation. Concluding the meeting, participants
recommended that the Center undertake the following tasks:
 Document and validate the decision rules that connect occupations to educational
programs, career clusters, and career pathways in several of the existing crosswalks.
 Analyze and discuss the different units of analyses that are currently operational within
several crosswalks, and outline for a plan for incorporating programs of study as a unit of
analysis for crosswalks.
 Develop a plan for disseminating the procedures, processes and products to data experts,
researchers and policymakers.
 The project continues into Year 4 and is being jointly led by the University of
Louisville and the NASDCTEc.
 Jointly with the Center for Education and the Workforce (CEW), Georgetown U,
Producing a Report on 2018 Employment Projections by Career Clusters
9
The NRCCTE CTE TYPOLOGY
 Traditionally, only the total number of CTE credits enrolled
and completed either across all areas or within one area are
used to identify the extent of participating or
concentrating in CTE. A three-credit threshold is used to
divide CTE students into participants and concentrators
 The NRCCTE has begun exploring, examining, and
analyzing credits taken within an occupational area,
across all occupational areas, and the number of
occupational areas in which high school graduates
participate and concentrate
 The main source is the US Department of Education
sample survey data, specifically the 2005 NCES High
School Transcript Studies (HSTS) data and the
Education Longitudinal Study (ELS) data
10
Conventional Way of Defining A CTE student
 Different denominations: concentrator, participant,
investor …
 “Traditional” classification in large surveys:
 Academic track
 Vocational track
 Neither and both (dual)—as defined in the NCES
sample survey
 NCES: Fulfillment of an occupational area if 3 or more
credits are taken in that area (“occupational
concentrator”)
 At the State Level: different levels, different goals
11
A Methodological Rationale for
Measuring, Accounting, and Evaluating CTE
Reformulating CTE Course Taking
 The purpose is to understand, the level, mix, and
intensity of CTE course-taking within and across
different occupational areas.
 The NRCCTE has expanded the participant-concentrator
dichotomy by creating a taxonomy that more accurately
represents CTE course-taking.
 Using the Classification of Secondary School Courses
(CSSC) different parts of the high school graduates’
transcripts are analyzed
Reformulating CTE Course Taking
• Following along the lines of NCES, the basic steps are
as follows:
 Identify CTE and Academic courses
 Divided CTE into 13 occupational areas
 Divided Academic into different subject areas,
including Math and Science
 Categorized Math & Science courses by three
levels: Less than Basic, Basic, and Advanced
 Identified Academic and CTE courses which
are in sequence and further divided by No
Sequence/Sequence
 Determined the last grade year Math & Science
course taken
14
Reformulating CTE Course Taking
 Dividing CTE course taking patterns of high school
graduates (students) into three facets:
 Identify how many credits high school graduates takes and
completes within each occupational area
 Determine the total number of CTE credits enrolled and
completed across all occupational areas
 Count the number of occupational areas in which each
graduate enrolls and completes courses
 A crosstab has been built that relates the above three
facets creating the following (shown on the next slide)
Reformulating CTE Course Taking
Participant
CTE Credits within an Occ. Area
Total CTE credits 0 - < 3
>0 -< 1
1
Zero Occ. Area
Not Applicable
Single Occ. Area
1
Multiple Occ. Area
2
>1 and <3
3
>3
Not Applicable
CTE Credits within an Occ. Area
Total CTE credits >=3
No of Occ. Areas
No CTE
Course
Taking
No of Occ. Areas
0
Single Occ. Area
0
>0 -< 1
1
>1 and <3
3
>3
Not Applicable
4
Multiple Occ. Area
3
Concentrator
0 is No CTE Course Taking; 1,2, and 3 are classified as Participants; 4 is classified as Concentrators
16
Reformulating CTE Course Taking
Participant
CTE Credits within an Occ. Area
Total CTE credits 0 - < 3
>0 -< 1
1
Zero Occ. Area
Not Applicable
Single Occ. Area
19%
Multiple Occ. Area
30%
>1 and <3
3
>3
Not Applicable
CTE Credits within an Occ. Area
Total CTE credits >=3
No of Occ. Areas
No CTE
Course
Taking
No of Occ. Areas
0
Single Occ. Area
8%
>0 -< 1
1
>1 and <3
3
>3
Not Applicable
17%
Multiple Occ. Area
27%
Concentrator
0 is No CTE Course Taking; 1,2, and 3 are classified as Participants; 4 is classified as Concentrators
17
Frequencies of CTE Credit Categorizations (n=2,698,609)
CTE Credit Categorization
%
No CTE credits
8.0
More than 0 and less than 1 CTE credits, no occupational area fulfilled
7.5
1 CTE credit, no occupational area fulfilled
10.7
More than 1 and less than 3 CTE credits, no occupational area fulfilled
29.9
3 CTE credits, no occupational area fulfilled
7.6
More than 3 CTE credits, no occupational area fulfilled
19.6
3 CTE credits, 1 occupational area fulfilled
0.8
More than 3 credits, at least 1 occupational area fulfilled
16.0
18
Frequencies of Gender, Race and Socioeconomic Status
by CTE Credit Categorizations for Public High School Students
0
1
2
3
4
5
6
7
Weighted n
Female
Male
9.0
6.9
8.1
6.8
11.4
9.9
30.8
28.9
7.6
7.6
18.4
20.8
0.9
0.7
13.8
18.3
1,378,033
1,320,577
White
Black
Hispanic
Asian
Other
8.3
8.0
7.7
10.3
7.5
7.4
5.7
9.2
9.8
9.8
10.1
8.9
12.2
17.0
21.0
27.8
33.5
31.6
34.9
70.2
7.1
7.8
8.8
7.2
16.5
21.2
17.9
17.7
12.0
46.8
0.8
1.3
0.8
0.3
1.3
17.4
16.9
12.1
8.4
26.7
1,633,287
379,133
421,325
122,181
142,684
SES-Q1 (Low)
SES-Q2
SES-Q3
SES-Q4 (High)
5.9
5.7
8.7
11.7
6.0
6.6
6.4
11.0
9.7
8.6
10.3
14.1
29.5
28.9
30.7
30.4
7.8
7.2
8.2
7.1
20.6
23.6
19.5
14.5
0.9
0.5
0.8
1.0
19.6
19.0
15.3
10.2
640,172
687,844
703,344
667,249
Group 0: No CTE credits
Group 1: More than 0 and less than 1 CTE credits, no occupational area fulfilled
Group 2: 1 CTE credit, no occupational area fulfilled
Group 3: More than 1 and less than 3 CTE credits, no occupational area fulfilled
Group 4: 3 CTE credits, no occupational area fulfilled
Group 5: More than 3 CTE credits, no occupational area fulfilled
Group 6: 3 CTE credits, 1 occupational area fulfilled
Group 7: More than 3 credits, at least 1 occupational area fulfilled
19
Preliminary Results
 Students taking more than 3 CTE credits, with or without a
focus on an occupational area, are 36% to 59% less likely to
drop out of high school than those taking fewer CTE
credits.
 Students who have taken 3 or more CTE credits, fulfilling
the requirements of an occupational area or not, are at least
43% more likely to attend 2-year colleges than those taking
fewer CTE credits.
 Based on high school graduate transcripts that describe
CTE and academic course taking, and using conventional
definitions of college, career, and college and career
readiness, early estimates for each are 42%, 28%, and 18%,
respectively.
20
21
Stakeholder Contacts: Publications
 Kotamraju, P. (2011, June). A new taxonomy of high school career and technical education
(CTE): Describing the engagement, achievement, and transitions of CTE students. Paper
presented at the 15h Annual Career Clusters Institute, Atlanta, GA.
 Kotamraju, P. (in press—expected publication date, Summer 2011). Vocational education
and training quality and evaluation: Its place in the U.S. community college. In A.
Barabasch & F. Rauner (Eds.), The art of integration: Work and education in America.
Berlin, Germany: Springer Verlag Press.
 Kotamraju, P. (2011). Meeting the 2020 American Graduation Initiative (AGI) goal of
increasing postsecondary graduation rates and completions: A macro perspective of
community college student educational attainment. Community College Journal of
Research and Practice, 35(1-2), 1-18.
 Kotamraju P., Richards, A., Wu, J., & Klein, S. (2010, April). A common postsecondary
data dictionary for Perkins accountability. Louisville, KY, and Berkeley, CA: National
Research Center for Career and Technical Education, University of Louisville, and MPR
Associates, Inc. Retrieved from
http://136.165.122.102/UserFiles/File/Tech_Reports/Postsecondary_Data_Dictionary_Rep
ort_WEB.pdf
22
Stakeholder Contacts: Paper Presentations
 Kotamraju, P. (2011, May). CTE Measurement, Accountability and Evaluation:
The NRCCTE’s Comprehensive Strategy for Technical Assistance. Paper To Be
Presented at the National Association of Career and Technical Education
Information annual Meeting, Philadelphia, Pennsylvania.
 Kotamraju, P. (2011, May). A New Taxonomy for Career and Technical
Education (CTE): Measuring Engagement, Achievement, and Transitions of
CTE Students. Paper To Be Presented at the National Association of Career and
Technical Education Information annual Meeting, Philadelphia, Pennsylvania.
 Kotamraju, P. (2011, April). Getting Timing and Sequencing of Math and
Science Course-Taking Right: A Possible Answer to the College and Career
Readiness of High School Graduates Concentrating on Career and Technical
Education (CTE). Paper presented at 2011 Council for the Study of Community
Colleges Annual Conference. New Orleans.
 Aliaga, Oscar. (2011, April). Career and Technical Education Graduates and
Enrollment in 2-Year Colleges. Paper presented at the annual meeting of the
Council for the Study of Community Colleges. New Orleans, Louisiana.
 Kotamraju, P. (2011, April). The college and career readiness of U.S. high school
graduates: From concept to measurement. Paper to be presented at the annual
convention of the American Educational Research Association. New Orleans,
LA.
23
Stakeholder Contacts: Paper Presentations
 Kotamraju, P. (2010, December). A tool kit for measuring CTE effectiveness using
return on investment and other related techniques: A basic introduction. Paper
presented at the annual convention of the Association for Career and Technical
Education, Las Vegas, NV.
 Kotamraju, P. (2010, October). Meeting the 2020 AGI completion goal: The role of
the community college. Paper presented at the annual convention of the National
Council of Workforce Education, Washington, DC.
 Kotamraju, P. (2010, June). Building a Technical Skills Inventory Database One
State at a Time,” 14th Annual Career Clusters Institute, Denver, Colorado.
 Kotamraju, P. (2010, May). A Cross-State Comparison of Postsecondary CTE
Student Graduation Rates and Completions: Determining the Efficacy of Using
IPEDS Data for Perkins Reporting,” National Association of Career and Technical
Education Information Annual Meeting, St. Louis, Missouri.
 Kotamraju, P. (2010, May). Building a Useable Inventory Template for Collecting
State and Local Information for Meeting the Technical Skills Accountability.
National Association of Career and Technical Education Information Annual
Meeting, St. Louis, Missouri.
 Kotamraju, P. (2010, April). A Cross-State Comparison of Postsecondary Student
Graduation Rates: A Macro Perspective of Educational Attainment,” Annual
Meeting, Council for the Study of Community Colleges, Seattle, Washington.
24
Stakeholder Contacts: Paper Presentations
 Kotamraju, P. (2009, December). The Minnesota FastTRAC Project:
Solving the Career Pathways Puzzle. 15th Annual Data Quality Institute,
Baltimore, Maryland.
 Kotamraju, P. (2009, December). Building a Technical Skill Inventory
Database One State at a Time,” 15th Annual Data Quality Institute,
Baltimore.
 Kotamraju, P. (2009, May). A Repository Template of Technical Skills
Assessment: Collecting, Accessing, Reporting and Using Information
and Resources for Meeting the Perkins Technical Skill Attainment
Indicator,” National Association of Career and Technical Education
Information Albuquerque, New Mexico.
 Kotamraju, P. (2009, May). Building a Data Dictionary Template from
State Administrative Record Data Systems,” Albuquerque, New Mexico.
25
Correspondence should be sent to:
Pradeep Kotamraju, NRCCTE, College of
Education and Human Development
Room 351
University of Louisville, Louisville, KY 40292.
Phone: 502-852-7714. Email:
pradeep.kotamraju@nrccte.org.
26