Future of Psychometric
Analysis
Certification Exam Preparation
January 21, 2016
F. Sam Griggs, Jr., CCP, Cost Engineering Consultant
Introduction
What Is It?
How Did We Get Here?
What Is Its Language?
What is the Process?
Vision of the Future
How Can You Pass the Exam?
Your Technical Paper
Psychometrics
Technique
Theory
Measurement
Psychometrics
Statistical
Research
Effect on
Measurement
Objective
Measurement
of Skills and
Knowledge
Psychometrics
Key Concepts
are Reliability
& Validity
How Did We Get Here?
8 Hour Hand
Written Exam
Computerized
Testing
Statistical
Data
Psychometrics
Review
Removed
Difficult
Questions Most
Got Wrong
Removed
Easy
Questions
Most Got
Right
Grading:
 To determine a passing grade of any test, it must be justified
with empirical data.
 It should be legally defensible.
 Meet the Standards for Educational and Psychological
Testing.
Glossary of Terms
 Alpha Coefficient
 Angoff Method
Distractor
Item Difficulty
Item Distraction
Item Discrimination
 Key
Glossary of Terms
 Mean
 Mean P
Mean Rnbis
P-value
Point Biserial Correlation (Rbpis)
Standard Deviation
 Pre-test or Probationary Question
William H. Angoff
Distinguished research scientist at
ETS for more than 40 years.
Angoff Method: One of his many
contributions to psychometrics.
Author of “Scales, Norms, and
Equivalent Scores”.
Angoff Method:
Most widely
used
Relies
on SME
Goal
• Determines the passing % or cutscore.
• SME are subject matter experts who examine each question.
• Predict how many minimally qualified candidates would answer
correctly.
• The average of the SME’s predictions becomes its predicted
difficulty.
• The sum of the predicted difficulty values averaged is the
recommended cut score
• Rate each test item
• To determine whether a minimally qualified candidate
would answer the item correctly or incorrectly.
Mechanics of
Psychometrics
Reliability
Validity
Statistics
Reliability
 A Reliable Measure: Measures test items consistently
across time, individuals, and situations.
 Reliability is necessary, but not sufficient for validity.
Pearson
Correlation
Coefficient
Test
Retest
Validity
 Valid Measure: Measures what it is intended to
measure.
Sample question: What is the capital of Texas?
A valid answer is Austin.
This question is a reliable measure but would not be a
valid measure for the CCP exam.
A valid measure for a CCP exam must measure only CCP components such as
Cost Estimating, Scheduling, Project Management, etc.
Statistics
 Both Reliability & Validity can be assessed Statistically.
Statistical Assessments:
P-Value (Pass Rate or Item Difficulty)
Point Biserial Correlation (Rbpis)
Negative Correlation
N (Number of Exams Completed)
Easy Test Question
 90% tested answer correctly.
Sample Question:
What is the Capitol of Florida?
a. Miami
b. Tallahassee
c. Jacksonville
d. Orlando
Easy Test Question
Sample Question:
What is the Capitol of Florida?
a. Miami
b.Tallahassee
c. Jacksonville
d. Orlando
Hard Test Question
 Over 50% tested answer incorrectly.
Sample Question:
In which state is Jacksonville State
University located?
a. Florida
b. North Carolina
c. Alabama
d. Illinois
Hard Test Question –Change
Distractors to make easier
Sample Question:
In which state is Jacksonville State
University located?
a. Hawaii
b. Alaska
c. Alabama
d. Illinois
Changing Distractor Answers
to make Question Easier
Steps in Making the Question Easier or Harder
 Changed “Florida” to Hawaii. Jacksonville is largest city
in FL and more people would recognize it and may
mark it as correct.
 Changed “North Carolina” to Alaska. Jacksonville, NC
is also home to large military installation and more
people would possible recognize it and may mark it as
correct
 Changing two of the more obvious answers would
increase changes of candidate marking correct answer.
Certification Board
Review:
Steps in Making the Question Easier or Harder
 Review the question to confirm that the marked answer
is correct.
 Modify the distractor to add or decrease the difficulty.
 Make no changes.
 Retire the question.
The Testing Dilemma
Lower
Test
Scores
Harder
Exams
The Testing Dilemma
 Newspaper headline stated that “With Tougher
Exams, more than 40% could fail”..
This would be common mis-conception.
Make questions more difficult = Lower Pass
Rate.
A study of Psychometrics shows that this is not
necessarily the case.
Note unrelated byline “Project paved with good intentions”
This is conundrum?
 or difficult problem; a dilemma: " a puzzling question or
problem
 1. A confusing and difficult problem or question ...
 2. A paradoxical, insoluble, or difficult problem; :.
Vision of the Future
PMI Exam Grading
CPA Exam Grading
Use of Pre-test Questions:
 Also known as Probationary Questions .
 Pretest questions are not scored.
 These questions are mixed into the Exam and are not identified as
pretest questions.
 From the candidate’s perspective, pretest questions are
indistinguishable from operational questions.
 Pretest questions that meet certain statistical criteria are used as
operational questions on future exams.
Imagining the Future in
Psychometric Testing
Incorporation
Of Current
Knowledge
Technological
Advances
Development
Of
Informative
Item Sets
Technological Advances
Revise
Questions
Generate
Statistics
Computerized
Testing
Determine # of
Difficult Questions
Add New
Questions
Incorporation of Current
Knowledge
Question
Design
11R88
Document
Selection
Process
Cover All
Sections
Relevant &
Inclusive
Exam
AACE® International Recommended Practice No.
11R-88
REQUIRED SKILLS AND KNOWLEDGE OF COST
ENGINEERING
 This recommended practice has the following purposes:.
 define what core skills and knowledge of cost engineering a
person is required to have in order to be considered a
professional practitioner, and in doing so,
 establish the emphasis of core subjects for AACE
International education and certification programs.
Informative Item Sets
Includes all
Aspects of 1188
Balance of Easy
and Difficult
Questions
Future Testing Method
New
Technology
No Set
Pass Rate
New
Psychometric
Testing
Future Testing Method?
 The Certification Exams may Not have a set Pass rate;
 Pass Rate could depend on the difficulty of the individual
questions on the Exams.
 Higher score if they can answer the harder questions
correctly.
 Lower score for the easier questions.
 Exams may include Pretest (Probationary) questions as well as
Operational questions.
 Technology makes it easier to generate Statistics. Use to determine proper
mix of Questions.
Top Tips to Get a Great Score
Purpose
Expect
Attitude
• Understand the purpose of the
exam.
• Know what to expect in the
timed testing environment.
• A positive attitude is a most.
Top Tips to Get a Great Score
Practice
• Especially necessary for anyone who has never taken a Psychometric Exam.
• Find websites & books that have practice test.
• Focus on tests that specialize in your area(s) of weakness.
• Test yourself working against the clock.
Time
Yourself
Broaden
Sample Test
• Don’t rely only on the study guides.
• Use AACE Recommended Practices (RP) on their web page.
• Many of the new questions on exams come from the new RPs.
Top Tips to Get a Great Score
Refresh Math
Skills
• Fact 1:
• Fact 2:
AACE has Math Questions that other exams do not.
Cost Engineers are expected to know the Rate of Return and other Math
problems.
• Make sure you get a good night’s sleep before the exam so you are fully alert.
Get a Good
Night’s Sleep
Follow
Instructions
• Carefully read or listen to the instructions on taking the exam.
Top Tips to Get a Great Score
Answering Questions
• Trust yourself…Usually your first answer is correct.
• Answer easy questions first, skip those you get stuck on.
• At the end, return to the questions you skipped.
• Do not leave any questions unanswered. Eliminate all obviously wrong choices.
• Last resort…Guess if you need to as you have a 25% chance of getting it right.
• Complete the Feedback Section at the end of the exam.
• Your comments or concerns help the Reviewers modify and improve future exams.
Complete the Feedback
Section on Exam
Don’t Give Up
• Keep trying until you Pass!
• Identify what areas that kept you from passing.
• Study these problematic areas to improve your skills.
Additional Tips –Beware of
Info on Internet
Misinformation
Candidate Feedback
Study AACE Suggested Reference Material
Review AACE Study Guides
Attend AACE Test Prep Seminar
AEP Formal Study Program
On-line Study Courses
Conclusion:
Learn the
Language
Use
Angoff
Method
Implement
Processes
Achieve
Certification
Study &
Prepare for
Exam
Technical Paper
Important information you should know before writing your paper
Must Submit Your Technical Paper Before You Can Take the Exam
Graded Independently of the Certification Exam
CCP’s Need Passing Grades on the Certification Exam and the
Technical Paper for Diploma
Know the Basic Guidelines used to determine whether your
Technical Paper is acceptable
Tips for a
Winning
Paper
Make sure your paper Includes:
Table of Contents
1
2
Abstract
3
4
5
Introduction: Purpose & Background
Basis & Analysis of Your Data
References
Tips for a
Winning
Paper
Make sure your paper Includes:
Graphs or Tables to Support Premise
6
7
Organization & Clarity
8
9
10
Logical Conclusion
Do Word Count: Minimum 2,500 Req.
Extra Points for Appendix
Questions??