CAS EXAMINATION
PROCESS
Julie Stenberg, FCAS
CANE Meeting
March 20, 2007
CAS Admissions
Process Audit
• Fall 2000 – CAS Issues RFP for External
Review of Admissions Processes
• The Chauncey Group (Subsidiary of ETS)
Selected
• Spring 2001 – Chauncey Group Conducts
Audit of CAS Admissions Processes
2
Audit Findings
The CAS Does Many Things Well:
• Good Communication with Candidates
• Sound Procedures for Maintaining
Confidential Information
• Exams are Administered with
Appropriate Controls and Standardized
Procedures
3
Audit Findings
Several Areas for Improvement:
• Need Better Link Between Learning
Objectives and Exams/Readings
• Learning Objectives and Exam
Blueprints Should be Published
• Need Better Training of Item Writers
• Need to Consider Alternative Processes
for Selecting Pass Marks
4
Major Objectives
The Chauncey Group Engaged to Help CAS
with Three Issues:
• Write Better Learning Objectives and
Establish Links to Readings/Exams
• Develop a Process for Training Item
Writers
• Pilot an Alternative Process for
Selecting Pass Marks
5
Major Milestones
• August 2001 – Chauncey Began Facilitating
Meetings to Write Learning Objectives
• October 2001 – Piloted Pass Mark Panel
Process for Exams 6 & 9
• March 2002 – Piloted Item Writer Training
Classes for Exams 6 & 9
• April 2002 – Pass Mark Panels Meet for Exams
5, 7 & 8
• June 2002 – Item Writer Training for Exams 5,
7&8
6
Major Milestones
• October 2002 – Pass Mark Panels Meet for
Exams 6 & 9
• February 2003 – Executive Council Agrees to
Fund Item Writer Training and Pass Mark Panels
as Ongoing Processes
• April 2003 – Executive Council Approves New
Learning Objectives for Exams 3, 5-9
7
Recent Improvements
•
•
•
•
•
Evolution of CBT
Improvement to Pass Mark Panel Process
Expanded Sample Answer Sets
Increased Communication
CAS Board White Paper
8
Learning Objectives
The way things were
– What topics should
successful candidates
understand
– What readings should
they know?
The way things are now
– What should
successful candidates
be able to DO?
9
Learning Objectives
The way things were
– Individual topics and
readings were the
basis for assigning the
writing of exam
questions
The way things are now
– Learning Objectives
are the basis for
assigning the writing of
exam questions
10
Learning Objectives
The way things were
– Syllabus “blueprints”
were the documents
governing the review
of the Syllabus and the
construction of Exams
The way things are now
– Learning Objective
Documents are the
basis for the review of
the Syllabus and the
construction of Exams
11
Learning Objectives
The Syllabus Committee has developed
Learning Objective Documents for CAS
Exams 3, 5, 6, 7-US, 7-Canada, 8 and 9
12
Learning Objective Documents
Five Elements
• Overview Statement for a Group of
Learning Objectives
• Learning Objectives
• Knowledge Statements
• Syllabus Readings
• Weights
13
Learning Objective Documents
Overview Statements
• Certain Syllabus Sections Can Have
Multiple Learning Objectives (e.g.,
Ratemaking)
14
Learning Objective Documents
Learning Objectives
• What successful candidates should be able to
do
• Learning Objectives Should:
Clearly state a main intent
Reflect a measurable outcome
Support an attainable behavior
Relate to the learner’s needs or job
function
Have a definitive time frame
15
Learning Objective Documents
Knowledge Statements
• Support Learning Objectives
• In order to accomplish the objective,
what does the candidate need to know?
16
Learning Objective Documents
Readings
• An individual reading may be listed
under more than one learning objective
• Readings listed under multiple
objectives may facilitate more
synthesis/reasoning/cross-topic Exam
questions
17
Learning Objective Documents
Weights (by Learning Objective)
• Shown as ranges
• The ranges are guidelines and are not
intended to be absolute
• Ended practice of candidates calculating
de facto weights by reading or topic
from past Exams
18
IV. Ratemaking
This section contains objectives covering ratemaking in broad, general principles, as well as
specific detail. Candidates should have a thorough understanding of the basic principles of
ratemaking, so that they can analyze data, select an appropriate technique, and develop a
solution to a numerical problem. In addition, the candidate should be able to compare specific
ratemaking techniques in terms of advantages and disadvantages as they are applied to specific
situations and for different lines of business.
Weight: xx-xx%
1. Explain the role of exposure bases in the
ratemaking process.
Definition of exposure base
Characteristics of exposure base
Impact of exposure change
Weight: xx-xx%
Coverage provisions
Reading(s): McClenahan, Finger, Principles, Bouska, Feldblum (WC), Graves & Castillo,
Jones
2. Use appropriate premium data to
Compilations of experience (CY/PY/AY)
estimate premium input into the overall
Written versus earned premium
rate level indication, adjusting for the
Rate changes
following:
Policy terms
 Coverage and benefit level
Distributional shifts/changes in volumes (tend
over time)
changes
 Rate level changes
Parallelogram method
 Premium trend
Extension of exposures
Definition of exposures
Impact of law changes
Weight: xx-xx%
Reading(s): McClenahan, Finger, Principles, Feldblum (Asset Share), Feldblum (WC), Jones,
ASOP #13
19
Learning Objectives
and the Syllabus
• Learning Objective Documents Provide
High Level Guidance
– Review of Current Syllabus Material
– Identification of Topics Requiring New
Syllabus Material
• Weights help Syllabus Committee Target
Specific Objectives
20
Future Changes to Learning
Objective Documents
• These are Living Documents
– Never Perfect
– Subject to Change
• Updates – When and How Often?
– Once a Year Per Exam Seems
Reasonable
– At Least Disruptive Time for Candidates
21
Future Changes to Learning
Objective Documents
• CAS Executive Council (VP-Admissions)
Performs Oversight and Final Approval of
Any Changes
– Just as it does with changes to the
Syllabus
– Just as it has with the current Learning
Objective Documents
22
Learning Objective Summary
• Transition to Published Learning
Objectives Should Help the CAS Achieve:
– Better Syllabus Content and Exam
Questions
– More Transparent Basic Education
Process
– Better Model for Evaluating Future
Changes to the Syllabus
– Better Model for Evaluating Future
Changes to the Desired Education of
Casualty Actuaries
23
Writing Exam Questions
Question 1 – According to Miller, “Writing Exam Questions”,
which of the following is true?
I.
Writing exam questions is the same now as it was
6 years ago
II. Writing exam questions is easier now than it was
6 years ago
III. Writing exam questions is harder now than it was
6 years ago
A. II only
B. I and II only
C. I and III only
D. II and III only
E. I, II and III
24
Writing Exam Questions
I.
Writing exam questions is the same now as
it was 6 years ago
True – Question writers have always wanted to
write good, fair, high quality questions.
1. Still takes the same time commitment
2. Still requires studying assigned readings
3. Still involves choosing the areas you want
to test
4. Still involves peer review by others
25
Writing Exam Questions
II. Writing exam questions is easier now than
it was 6 years ago
True – We have more tools to work with
1. We have identified objectives
2. They identify the readings tied to those
objectives
3. Question writing skills are taught – what to
do and what to avoid
4. We have a common language with which to
26
make constructive criticisms
Writing Exam Questions
III. Writing exam questions is harder now
than it was 6 years ago
True – We have been conditioned by years of
studying old questions
1. Triple True-False are often easier to write
than short answers for Multiple Choice
2. The easiest questions to write may not
always fit the objectives
3. The bar is higher and we don’t like to fail
27
Writing Exam Questions
Question 1 – Solution: E
A. Some candidates will think nothing has
changed
B. Some candidates will think the new
process could not possibly make it harder
to write questions
C. Some candidates always guess C when
they don’t know the answer
D. I used to guess D
E. Correct answer
28
What Have We Learned From
The Chauncey Initiatives?
• Questions should be focused on learning
objectives, rather than individual papers
• Triple True/False is not the only kind of
multiple choice question
• Art of selecting good “wrong” multiple
choice answers
29
What Changes Should The
Candidates See On The Exams?
• Better questions
• Questions with many possible full-credit
answers
• Less “according to” and “based on”
questions
• Heavy “list” papers have become openbook
30
Setting the Pass Mark
•
•
•
•
•
Identify Purpose of the Pass Mark
Convene Pass Mark Panel
Analyze Exam Statistics
Prepare Recommendation
Proceed through Approval Process
31
Purpose of the Pass Mark
• Pass Minimally Qualified (or better)
Candidates
– Those who have demonstrated a
sufficient grasp of the syllabus material
• Fail Others
• There is no predetermined pass ratio
32
Purpose of the Pass Mark
Failers
Passers
Minimally Qualified Candidate
33
Pass Mark Panel
• Panel includes:
New Fellows (1-3 years)
Fellows experienced in practice area
Officers of exam committee
• Recommends a pass mark independent of
the normal exam committee procedures
34
Pass Mark Panel
• Defines Minimally Qualified Candidate
What he or she should will know
What he or she will not know
What he or she will be able to
demonstrate on the exam
• Relates Criteria to Learning Objectives for
defining the minimally qualified candidate.
35
Pass Mark Panel
• Each panelist independently estimates how 100
minimally qualified candidates will score on each
question (and sub-part of each question).
• Scores are assembled and shared in a group
format.
• Group discusses ratings and may change
estimates
• Facilitator compiles ratings and shares results
with exam committee officers
36
Analyze Exam Statistics (back at
the Grading Session)
• Collect Initial Scores for All Candidates
• Review/Discuss Key Measures
 High, Low, Mean
 Percentiles, Percentile Relationships
 Pass Mark Panel Recommendation
 Prior statistics from previous exams
 CAS Board goal, “…that 40% or more of the
candidates should get a score of 70% or more on any
given exam; and all candidates that get such a score
should pass.”
• Pick an initial pass mark and re-grade candidates within
certain range of pass mark (+/- 3 points, for example)
37
Prepare Recommendation
• Recollect scores if any have changed and
review all relevant statistics again.
• Repeat process until only looking at the 5
exams above and the 5 exams below the
recommended pass mark.
• Justify Recommended Pass Score
38
Approval Process
• Part Chair
• General Officer (Spring / Fall)
• Exam Committee Chair (Arlie Proctor)
• VP-Admissions (Jim Christie) – The final
decision on the pass mark is the responsibility of
this position.
39
Appeal Process
• In the event of a candidate appeal, a grader may
be called upon to review the appeal and
reconcile the score with the grading key.
40
Join the Exam Committee
Fill out the annual CAS participation survey
or
Contact the exam committee recruiter
directly
Rhonda Walker
rpwalkerbhnj@comcast.net
41
Questions and Comments
Contact
Julie Stenberg at
julia.stenberg@travelers.com
Or
Arlie Proctor at
aproctor@munichreamerica.com
42