LONGSCAN
CPS Case Narrative Record Review
Procedures for Establishing and Evaluating Inter-rater Reliability
(Adopted November 1997)
(Revised May 2001)
(Updated August 2007)
A. Initial training: Establishing Reliability
1. Coordinating Center (CC) Staff will be responsible for providing training. A lead
abstractor from each site (and other site abstractors, when possible) will ideally
receive cross-site training. If all local abstractors cannot participate in cross-site
training, then the lead abstractor will be responsible for training other local
abstractors using the same standards outlined below. Training that includes multiple
sites is preferable, when possible, because it is both more effective and efficient.
2. Each site will maintain CPS Case Narrative Record Review Guidelines and
Codebook with current updates. The Codebook will be used in training and referred
to during fieldwork.
3. Training will consist of two phases. In Phase 1, trainees will learn how to use the
form, where to find the needed pieces of information in the record (each site’s records
will vary somewhat), and how to classify maltreatment type and severity according to
both MMCS and NIS-2 classification systems. Phase 1 Training Vignettes will be
coded and discussed with trainer’s instruction and supervision. When, following
work with Phase I Training Vignettes, trainee is thought to be proficient in coding
maltreatment; s/he will be invited to independently score Phase 2 Practice Sets.
When coding is 90% accurate with the designated “gold standard” from the CC, the
abstractor is authorized to begin coding records in the field. If not, the trainer(s) will
work with trainee on problem areas. When ready, trainee will score additional Phase
2 Practice Sets. No abstractor should enter the field until a score of 90% accuracy is
achieved on a complete Phase 2 Practice Set.
(Note: Accuracy and reliability will be evaluated separately for MMCS and NIS-2
codes. Raters must be 90% accurate and reliable for both maltreatment
classification systems. Accuracy and reliability will also be determined separately
for “type” and “severity” ratings.)
B. Field activities: Questions and Follow-up
4. Coding questions that arise in the field should be taken to the lead abstractor at each
site. If the lead abstractor is unsure how to respond to the question, s/he should
consult with the Coordinating Center. The Coordinating Center will document all
coding issues that arise during training and telephone consults, and include new
information (when warranted) in future training sessions and periodic training
updates.
5. Once in the field, after having coded the first 50 records, a third set of 10 gold
standard vignettes should be scored by the abstractor. Again, 90% accuracy with the
gold standard is required. If the abstractor is not 90% accurate, s/he will be re-trained
and re-tested until 90% accuracy is reached. (Also the 50 records just coded should
be reviewed and recoded.) This process will be repeated again after another 100
records have been coded. (In this case, if accuracy is below 90%, only the 100
records just coded should be reviewed and recoded, following retraining.)
C. Field activities: Ongoing inter-rater reliability
6. In addition to initially establishing reliability during training and in the field, each
abstractor (including lead abstractors) is to be periodically assessed in terms of interrater reliability of their coding and abstraction work. This process will use actual
reports from CPS records – narratives of allegations and of substantiations - rather
than “gold standard vignettes.”
7. Periodically, a 10% sample of allegation and substantiation narratives will be
randomly selected by the CC. These will be sent to each abstractor for coding of
maltreatment type and severity according to both MMCS and NIS-2 maltreatment
classification systems. Inter-rater reliability kappa statistics will be calculated by
abstractor, site and study overall.
8. If an abstractor/coder is not 90% reliable, s/he will be re-trained and re-tested until
90% reliability is achieved.