Ohio 2015 ACTE Conference
Pat Huston (Patricia.Huston-Holm@education.ohio.gov)
James T. Austin, austin.38@osu.edu
Brooke N. Parker (parker.1036@osu.edu)
http://news.webxam.org (posted)
 Poll
you: knowledge of testing & field contributions
 Test
you: 12-item test about standards, course outlines,
& test development
 Share
with you: “test key”, pictures, item writing-review
process
 Listen
to you: Q & A period for your inquiries
 Please
try the questions on this short quiz to see the
types of items used in technical tests (5 minutes)
 Nine
questions are standard multiple choice, three
are linked to a scenario (short paragraph)
• Circle correct answers
• Estimate your # right at the end of the test (1 item=8.33%)
 Hand
out “key” near end, discuss if time permits …
 Quality
test question banks link to OHIO course outlines—
high school students will be test-takers
• Planners considered Design-Development-Delivery
• Design: summative, criterion-referenced, multiple choice
• Development: Subject Matter Experts (HS, PS, BIL) and facilitators
• Delivery using WebXam: two 40-item forms created after QA
 Posttests: Required after course instruction (Perkins IV funding)
 Pretests: OPTIONAL way to show student gain (pre-to-post)
 Articulation
credit – iff available & approved (CT2, SCTAI)
Where do tests come from?
Ohio Subject-Matter Experts are Key!
Test Development Cycle

Strategies developed by team with ODE-ODHE input
• Targeted communications (directors, supervisors; DTC) & presentations like this
• WebXam news portal updates, newsletters, & blogs
• “Call for Item Writers” lists career fields + expected schedule
• Document interest, update ODE, invite to workshop (hope for 100% attendance)
• Fall 2015: Hospitality (Culinary, Lodging) & Family-Consumer Sciences
• Winter-Spring 2016: Ag-Env Systems (25+) & other revisions (Health, Engineering)

Web videos used to 1) orient prospective item writers & 2) thank item writers
• “Context” gives big picture, “Item Writing” & “Item Review” provide details
• “Thank you” video recognizes participants during 2014-15 (YouTube)
Stage 1: Design
Define Test Purpose(s)
•
•
•
•
•
CTE End-of-Course (EOC) Test Structure
Summative assessments
Assess knowledge-skill learned in
courses (posttest) or gain (with pretest)
Support federal and state reporting
Support articulation (some PS courses)
Program improvement at local level
Career Field
CFTCS (Revised) • Strands
• Outcomes (subset)
• Competencies (associated)
Pathway courses drawn from CFTCS Strands
Define Content Domain
•
•
•
Course outlines drawn from CFTCS
Course = Outcomes-Competencies
PS learning outcomes where defined
Test Blueprint
•
Plan item distribution across outcomes,
(ensuring key competencies covered)
Course 1
Course 2
Course 3
Course 4
EOC Test 1
EOC Test 2
EOC Test 3
EOC Test 4
Take an EOC test for each course taken
Stage 2: Develop
Write Items
• Facilitated, face-to-face workshop
• Small groups (2-3)
• Items stored instantly in the CAT
software tool (shown below)
Review Items
• Large group format, facilitated
• First stage QA: technical
accuracy (key, formulas, level)
• Modify if possible, get in shape
for question-by-question
ratings
Item Bank Features
• 4-option multiple choice
• ~95 items each EoC test bank
• ~40% higher level (application,
analysis, & evaluation)
• ~40% scenario-based items
• Learning outcomes in articulation
used when CTAN exists and
linkage provided
Item Ratings & Performance
Standards
• Question essentiality, quality
• Performance standards:
Proficient, Advanced
• Judgments on iPads on final
day of workshop (10 SME)
• Conduct open sessions to
obtain additional input
Admin
Content
Admin
SCENARIO: Agribusiness-Mathematics
SCENARIO: Your herd of cattle {Charolais/Angus/Polled Hereford} consists
of 5/10/15 (or 3/7/11) animals. A daily ration was established in
consultation with the county extension agent at 3/5/7 lb of grain/hay
grazing fodder per day/week. You are creating an annual budget.
ITEM 1: If grain/hay is priced at $2.25/2.50/3.50 per lb., what would it
cost to feed the herd for one/two/three months?
•$
•$$
•$$$
•$$$$
ITEM 2: If you are feeding a composite ration (grain and hay), priced
as follows, what would the costs be for a {time unit: month / year}?

Which of the following formats for graphic files is the newest? (or if
turned around: the ‘earliest’ or ‘oldest’)
• A. JPG; B. TIFF; C. PNG; D. GIFV
 Simple
project calculations (area, volume, perimeter)
 License
: Certification:: ______ : ______
• Correct: Mandatory:Voluntary
 The
injected mixture in hydraulic fracturing is mostly:
• A. sand. B. chemical additives. C. water. D. silica suspension.
How could this chart be
used to create C2 items?
SUMMARY: All TTP FY 2014 Pathways
C1
(Recall)
C2
(Analysis)
%
Scenario
%
Total
Health Information Management
209
140
40.1%
129
37.0%
349
Exercise Science and Sports Medicine
427
257
37.6%
244
35.7%
684
Allied Health and Nursing
801
574
41.7%
503
36.6%
1375
Networking
464
305
39.7%
254
33.0%
769
Information Services and Support
469
300
39.0%
258
33.6%
769
Interactive Media
475
294
38.2%
263
34.2%
769
Programming and Software Development
433
337
43.8%
284
36.9%
770
Ground Transportation
648
460
41.5%
375
33.8%
1108
Air Transportation
532
296
35.7%
284
34.3%
828
Ag Bioscience
430
257
37.4%
246
35.8%
687
Cosmetology
485
288
37.3%
279
36.1%
773
5373
3508
39.5%
3119
35.1%
8881
56
35
38.5%
32
35.2%
91
Totals by Pathway
+MEP: Powerline/Hi-Voltage Power Transmission
C1=recall-factual question (item);
C2=apply-analyze-evaluate question;
Scen=scenario (entry-level workplace)
Stage 3: Test
Conduct Field Test
•
•
•
•
More quality assurance for test question bank
Field testing uses all items written for EOC test
All juniors; given post-instruction (or as optional pretest)
Adequate sample of test-takers (1.5 x number of items)
Conduct Psychometric Analyses
• Industry guidelines followed in evaluating items
• Examples: item difficulty and discrimination
Assemble Test Forms
•
•
•
•
Industry best-practices used to select questions for final forms
Two unique 40-item forms: pretest and posttest
Common ‘anchor’ items
Balance content coverage & psychometric characteristics
ODE-CTE assessment matrix (underclass, seniors) – is definitive policy
guidance – always check & use current version
 New
field testing next year + “stragglers” in Year 2 (Ag,
IT, Transportation, etc.) until ~5-6-16 (first Friday)
• Arts & Communication (Visual Design-Imaging, Media Arts, &
Performing Arts courses)
• Business cluster (Business-Finance-Marketing courses)
• Agriculture-Environmental Systems “New Courses” (~13 Year 1)
plus Biotechnology & Food (~7 in Year 2)
 Question
writing work (prior year) determines specific
courses posted for field testing (rule: 80-100 items)
Stage 4: Release
Deliver Tests
• Operational forms (rule: 40 items)
• Posttest immediately following instruction
• Optional pretest windows in fall and winter (as rolled out)
Report Scores
• Immediately following operational tests
• Mid-level feedback (strand-outcome groups; not competency)
Maintain Tests
• Annual psychometric analysis & maintenance (swap questions)
• Revision cycle follows standards revision (~3 years)
90000
80000
70000
# Learners Tested
60000
50000
40000
30000
20000
10000
0
2002
2003
2004
2005
2006
2007
2008
2009
Year
2010
2011
2012
2013
2014
2015
TOP 10 LIST (Our Apologies to D. Letterman, Retiree)
Course
Course Name
Students
010105
Agriculture, Food and Natural Resources
7851
175001
Engineering Design
5173
072150
Medical Terminology
4683
072050
Patient Centered Care
3707
010910
Animal Science and Technology
3514
178000
Core-Sustainable Construction
3271
072040
Human Anatomy and Physiology
3254
072035
Principles of Allied Health
3126
175002
Engineering Principles
2593
010110
Communications and Leadership
2490
010115
Business Management for Agricultural-Environmental Systems
2245
010155
Plant and Horticultural Science
2229
072015
Nutrition & Wellness
2142
145005
Information Technology
2030
12 Districts Testing > 1000 Learners Through WebXam
CTPD Name
Students Tested
Great Oaks Institute Of Technology
4293
Butler County JSVD
3018
Ohio Hi-Point JSVD
2716
Eastland VSD
2659
Penta County JSVD
2605
Miami Valley Career Technology Center
2592
Upper Valley JSVD
2199
Four County JSVD
2151
Akron City
1966
Columbus City
1884
Pickaway-Ross County JSVD
1657
Pioneer JSVD
1624
 Achievement
defined in 3 ordered categories, used in
reporting to Federal & state governments
• NON-PROFICIENT means LIMITED-NOVICE level (not mastered)
• PROFICIENT means INTERMEDIATE level (mastery)
• ADVANCED means HIGHEST level (above proficient mastery)
 Next
slide illustrates performance standard judgments
Performance Standards
Validity Ratings

Pretesting Windows (40 questions)
•
August 17 to September 18, 2015
•
January 4-29, 2016
•
Post-tests complete by April 15, 2016 (for reporting)

Field-Test Window (80-100 questions)
•
~October 5, 2015
•
Closes May 6th, 2016

Standard-Test Window (40 questions)
•
~October 5, 2015
•
Closes June 3rd, 2015

Continue to emphasize test security for score credibility
• Increase use of test security analytics (tracking over time)
 Caveon firm provides services across globe (CETE exploring partnerships)
• Changes in procedures: acceptable use policies change (at first log-in)
 Document exceptions (“do-overs” frowned upon due to score credibility)

Increase rigor of items & # items on forms (>40 = higher reliability)
• “Crowdsourcing”: Teacher-written items valuable for practice tests and eventually,
for secure EoC item banks if quality can be assured
• Explore distance item writing, scenario generation, etc.

Validate interpretations of gain/growth for other purposes
• Correlate parts of evaluation system: self- & peer ratings, observations, SGM
 How
can we be sure of the currency of the content?
• Course outlines (CFTCS) created by ODE or vendor, given to CETE
• Content advisory committees could be formed to update yearly ($$)
 Something
going on with Welding tests (Manufacturing)
• Check counts of test-takers, performance standards, & levels (A-P-N)
 Pretests
for non-OTES use
• At present time NOT able to accomplish this, but CETE will investigate