Development of Coding Protocol


Coding protocol: essential feature of meta-analysis
Goal: transparent and replicable


Wilson
description of studies
extraction of findings
Coding Protocol
Page 1
Topics for Coding






Eligibility criteria and screening form
Development of coding protocol
Hierarchical nature of data
Assessing reliability of coding
Training of coders
Common mistakes
Wilson
Coding Protocol
Page 2
Study Eligibility Criteria


Flow from research question
Identify specifics of:








Defining features of the program/policy/intervention
Eligible designs; required methods
Key sample features
Required outcomes
Required statistical data
Geographical/linguistic restrictions, if any
Time frame, if any
Also explicitly states what is excluded
Wilson
Coding Protocol
Page 3
Study Eligibility Screening Form






Develop a screening form with criteria
Complete form for all studies retrieved as potentially
eligible
Modify criteria after examining sample of studies
(controversial)
Double-code eligibility
Maintain database on results for each study screened
Example
Wilson
Coding Protocol
Page 4
Development of Coding Protocol

Goal of protocol




Describe studies
Differentiate studies
Extract findings (effect sizes if possible)
Coding forms and manual

Wilson
Both important
Coding Protocol
Page 5
Development of Coding Protocol


Iterative nature of development
Structuring data





Wilson
Data hierarchical (findings within studies)
Coding protocol needs to allow for this complexity
Analysis of effect sizes needs to respect this structure
Flat-file (example)
Relational hierarchical file (example)
Coding Protocol
Page 6
Example of a Flat File
Multiple ESs handled by having multiple
variables, one for each potential ES.
ID
22
23
31
36
40
82
185
186
204
229
246
274
295
626
1366
Paradigm
2
2
1
2
1
1
1
1
2
2
2
2
2
1
2
ES1
0.77
0.77
-0.1
0.94
0.96
0.29
0.65
DV1
3
3
5
3
11
11
5
0.97
3
0.86
7.03
0.87
3
3
3
ES2
DV2
-0.05
5
0.58
0.83
0.88
5
5
3
0.91
-0.31
6.46
-0.04
0.5
3
3
3.
3
3
ES3
DV3
0.48
5
0.79
3
3
3
0.1
ES4
DV4
-0.2
11
0.068
5
1.17
0.57 .
0.9
3
3
Note that there is only one record (row) per study
Wilson
Coding Protocol
Page 7
Example of a Hierarchical Structure
Study Level Data File
ID
100
7049
PubYear
92
82
MeanAge
15.5
14.5
TxStyle
2
1
Effect Size Level Data File
Note that a single record in
the file above is “related” to
five records in the file to the
right
Wilson
ID
100
100
100
100
100
7049
7049
7049
Coding Protocol
ESNum
1
2
3
4
5
1
2
3
Outcome
Type
1
1
1
1
1
2
4
1
TxN
24
24
24
24
24
30
30
30
CgN
24
24
24
24
24
30
30
30
ES
-0.39
0
0.09
-1.05
-0.44
0.34
0.78
0
Page 8
Example of a More Complex Multiple
File Data Structure
Study Level Data File
ID
100
7049
PubYear
92
82
MeanAge
15.5
14.5
Outcome Level Data File
ID
100
100
100
7049
7049
TxStyle
2
1
OutNum Constrct
1
2
2
6
3
4
1
2
2
6
Scale
1
1
2
4
3
Effect Size Level Data File
ID
100
100
100
100
100
100
7049
7049
7049
Wilson
OutNum
1
1
2
2
3
3
1
1
2
ESNum
1
2
3
4
5
6
2
6
2
Months
0
6
0
6
0
6
0
12
0
TxN
24
22
24
22
24
22
30
29
30
CgN
ES
24 -0.39
22
0
24 0.09
22 -1.05
24 -0.44
21 0.34
30 0.78
28 0.78
30
0
Note that study 100 has 2 records
in the outcomes data file and 6
outcomes in the effect size data
file, 2 for each outcome measured
at different points in time (Months)
Coding Protocol
Page 9
Advantages & Disadvantages of
Multiple Flat Files Data Structure

Advantages





Disadvantages




Can “grow” to any number of ESs
Reduces coding task (faster coding)
Simplifies data cleanup
Smaller data files to manipulate
Complex to implement
Data must be manipulated prior to analysis
Must be able to select a single ES per study for any analysis
When to use

Wilson
Large number of ESs per study are possible
Coding Protocol
Page 10
Concept of “Working” Analysis Files
Permanent Data Files
Study Data File
select subset of ESs of
interest to current analysis,
e.g., a specific outcome at
posttest
Outcome Data File
ES Data File
create
composite
data file
verify that there is only a
single ES per study
yes
no
Average ESs, further select
based explicit criteria, or
select randomly
Composite Data File
Working Analysis File
Wilson
Coding Protocol
Page 11
Example: SPSS ES Data File
Wilson
Coding Protocol
Page 12
Example: SPSS ES+Outcome Data File
Wilson
Coding Protocol
Page 13
Example: SPSS ES+Outcome+Study Data File
Wilson
Coding Protocol
Page 14
Example: Creating Subset for Analysis
Wilson
Coding Protocol
Page 15
Example: Final Working File for
a Single Analysis
Wilson
Coding Protocol
Page 16
Concept of “Working” Analysis Files
Permanent Data Files
Study Data File
select subset of ESs of
interest to current analysis,
e.g., a specific outcome at
posttest
Outcome Data File
ES Data File
create
composite
data file
verify that there is only a
single ES per study
yes
no
Average ESs, further select
based on explicit criteria, or
select randomly
Composite Data File
Working Analysis File
Wilson
Coding Protocol
Page 17
What about Sub-Samples?

What if you are interested in coding ESs separately for
different sub-samples, such as, boys and girls, or highrisk and low-risk youth, etc?


Wilson
Just say “no”!
 Often not enough of such data for meaningful analysis
 Complicates coding and data structure
Well, if you must, plan your data structure carefully
 Include a full sample effect size for each dependent measure
of interest
 Place sub-sample in a separate data file or use some other
method to reliable determine ESs that are statistically
dependent
Coding Protocol
Page 18
Coding Mechanics

Paper Coding (see Appendix E)




include data file variable names on coding form
all data along left or right margin eases data entry
Coding into a spreadsheet
Coding directly into a computer database
Wilson
Coding Protocol
Page 19
Coding Directly into a Computer Database

Advantages





Avoids additional step of transferring data from paper to computer
Easy access to data for data cleanup
Data base can perform calculations during coding process (e.g.,
calculation of effect sizes)
Faster coding
Disadvantages


Wilson
Can be time consuming to set up
 the bigger the meta-analysis the bigger the payoff
Requires a higher level of computer skill
Coding Protocol
Page 20
Example of Database with Forms
Figure 5.11: Example FileMaker Pro Screen for Data Entry from the Challenge
Meta-Analysis
Wilson
Coding Protocol
Page 21
Assessing Reliability of Coding


Inter-rater reliability and double coding
Intra-rater reliability
Wilson
Coding Protocol
Page 22
Training Coders



Regular meetings (develops normative understandings)
Annotate coding manual
“Specialist” coders
Wilson
Coding Protocol
Page 23
Common Mistakes





Not understanding or planning the analysis prior to
coding
Underestimating time, effort, and technical/statistical
demands
Using a spreadsheet for managing a large review
Variable names not on coding forms
Not breaking apart difficult judgments
Wilson
Coding Protocol
Page 24
Common Mistakes

Over-coding—Trying to extract more detail than routinely
reported
Wilson
Coding Protocol
Page 25
Comments on Managing the Bibliography


Major activity
Information you need to track







source of reference (e.g., PsychLit, Dissertation Abs.)
retrieval status
 retrieved, requested from ILL, etc.
eligibility status
 eligible
 not eligible
 relevant review article
coded status
Word processor not up to the task
Spreadsheets are cumbersome
Use a database of some form
Wilson
Coding Protocol
Page 26