Job Analysis Study Summary Report
CIPRP (Certified Infrastructure Protection and Resiliency
Professional)
The American Board for
Certification in Homeland Security, CHS®
February 15, 2012
Prepared by:
Assessment Systems Corporation
Saint Paul, MN
INTRODUCTION
This report describes the job task analysis (JTA) study for the Certified in Critical Infrastructure
Protection, CCIPsm (CCIP) exam sponsored by the American Board for Certification in Infrastructure
Protection, (ABCIP). This examination is designed to identify candidates with minimal knowledge and
skills to earn the CCIP credential. ABCHS contracted Assessment Systems Corporation, a leading provider
of software and services for testing organizations, to provide psychometric consultation in the study.
A list of tasks was generated by a panel of experienced infrastructure protection professionals. This list
was used to create a survey regarding the importance and frequency of each task; 201 professionals
completed the survey, providing empirical information regarding which tasks are most important and
are completed most often. This report provides detail on the methodology and results of this survey.
METHOD
The methodology used in conducting the CCIP job task analysis study is consistent with the validation
processes recommended in the American National Standards Institute’s Accreditation Guidelines for
Certification Organizations (ANSI Standard 17024), the Accreditation Guidelines of the National
Commission for Certification Agencies (NCCA), and the Joint Technical Standards of the American
Psychological Association, American Educational Research Association, and the National Council on
Measurement in Education.
The design and implementation of this study consisted of a number of steps carried out between July
and November of 2011. These steps are described below.
The ABCHS Executive Committee
A committee of subject matter experts (SMEs) is necessary to oversee the job analysis process. ABCIP is
is responsible for the development and administration of the Certified in Critical Infrastructure
Protection, CCIPsm. The Board led the CCIP job analysis study.
The role of the ABCIP Board in this study was to participate in the development of the job analysis
survey instrument to ensure that the full range of critical infrastructure-related activities and knowledge
areas pertinent to performing these successfully were comprehensively and accurately represented.
Specifically, the board formally adopted task and knowledge area lists, reviewed a draft version of the
survey instrument, and assisted with the interpretation of the survey data. The committee participated
in a variety of conference calls and email discussions regarding the project and the job analysis process.
Drafting the Job Analysis Surveys
There are several designs available (Brannick & Levine, 2002) for a job analysis study; a model commonly
used for credentialing exams is a task inventory (Raymond & Neustel, 2006). The goal of this approach
is to produce a comprehensive list of professional tasks performed on the job, then have a wide range of
incumbents rate each task on aspects such as importance and frequency or time spent on the task in a
normal work week. This provides empirical evidence as to which tasks are more important or more
frequent in the job; those tasks should obviously have more weight on the final test than rare or
unimportant tasks. Documentation of this process substantially enhances the validity of score
interpretations from a certification exam.
The following presents an overview of the steps in the study methodology.
1.
2.
3.
4.
5.
Develop test definition and broad outline to provide the initial link in the validity chain
Generate exhaustive list of task elements and knowledge areas of the profession
Review task and knowledge lists
Develop rating scales for task and demographic questions to assess sampling
Publish and deliver survey with sampling plan
The survey development began with a conference call involving members of The American Board for
Certification in Infrastructure Protection (ABCIP) on July 14, 2011. In this meeting, Board members
began developing a list of critical infrastructure protection job tasks that a minimally competent
candidate could perform successfully.
Board members collaborated through email over the course of the next month to hone an appropriate
list of job task statements. The ABCHS Chief Association Officer oversaw this collaboration. Responses
were collected by ABCHS staff and collated with the assistance of subject matter experts. On August 15,
ABCIP formally adopted its list of job task statements.
The approved list of job tasks contained 123 statements categorized into the following task areas:
Incident Management (Planning and Response), Physical Plant Assessment and Consultation, Policy
Assessment and Consultation, and Personnel Management. It was agreed that the frequency scale for
the tasks would be Regularly, Frequently, Occasionally, and Never. The importance scale would be
Extremely, Moderately, Slightly, and Not.
The final survey consisted of demographic questions and task statements. On Friday, August 26, 2011,
ABCHS staff created the survey instrument in Survey Monkey and forwarded a link to all ABCIP Board
members. This link allowed them to preview the survey before distribution. Board members had two
weeks to preview the survey and provide Staff with feedback.
Administration of the Job Analysis Survey
On September 13, 2011, the survey was ready and was emailed to approximately 4,679 specialists with
expertise in Critical Infrastructure Protection (including ABCIP Board members). They were given an
initial deadline of September 30 to respond. The survey was redistributed in early November to those
4,087 who had not responded. They were given another two weeks, with final responses totaling 201.
The respondents appear to be a representative sample of those in the field of Critical Infrastructure
Protection.
Survey Results
The responses to job analysis surveys were collected by SurveyMonkey.com. ABCHS Staff tabulated and
analyzed the data. A report on the survey results was prepared for a meeting of the ABCIP Board where
the results would be reviewed and test specifications for the CCIP examination would be finalized.
Development of the Weighted Content Outline
The mean and standard deviation of both time and importance ratings was calculated for each task. In
addition, mean time and importance were combined with both an additive model (T + I) and
multiplicative model (T × I). Each of these provides an index of the significance of the task in the role of
a professional critical infrastructure protection job.
Although only one index can be used to make decisions regarding exam content weightings, the use of
these calculations provides a logical, empirical link between survey results and test specifications. More
important or more frequently performed tasks in the job should obviously have more weight on the final
test than rare or unimportant tasks.
In the case of the job analysis data for CCIP, both indices were nearly perfectly correlated. The
multiplicative method was chosen.
ABCIP Board Review and Approval of Test Specifications
Board members of the ABCIP began reviewing the Job Analysis and Test Specifications reports on
February 17, 2012. They were given until March 7 to review and discuss the results in these reports and
to finalize the related proposed test content outline for CCIP. Following discussion of the documents,
the ABCIP Board finalized the CCIP Job Analysis Report, Test Specifications Report, and exam content
outline as it was proposed and without revision. The board finalized this discussion on March 5, 2012.
SUMMARY
This report describes a job analysis study for the Certified in Critical Infrastructure Protection, CCIPsm
certification program. The goal of the study was to produce a comprehensive list of professional tasks
performed on the job, with empirical data regarding the importance and frequency of those tasks to
garner detailed data on the current structure of the jobs performed by professionals in this role and to
better understand the field of critical infrastructure protection. A job analysis study is an essential
foundation for the validity of a certification program.
The first step was the development of the list of tasks and knowledge areas by a committee of highly
experienced subject matter experts. Once this was completed, a survey was constructed utilizing the list
as well as a number of demographic variables. The data set was analyzed to provide a depiction of both
the demographic qualities of the sample as well as the structure of the profession and a proposed test
content outline.
While the foundation of validity, the completion of a job analysis survey is only one step in the test
development cycle. The next step is to develop a test that strictly adheres to the finalized content
outline. Each item must be specifically mapped to a content area, and the percentage of items from
each content area must match the finalized percentages.