Effects of Korean-AGREE Ⅱ Scoring Guide on
Improving the Reliability of the Scores
Moo-Kyung Oh 1, Heuisug Jo1,2,3, Youkyoung Lee3,4,5
1Department
of Preventive Medicine, Kangwon National University Hospital, Chuncheon, Korea
2 Department of Health Management and Policy, Kangwon National University School of Medicine, Chuncheon, Korea
3Executive Committee for clinical practice guideline, The Korean Academy of Medical Sciences
4Department of Laboratory Medicine and Genetics, Soonchunhyang University Bucheon Hospital, Bucheon, Korea
5Department of Laboratory Medicine and Genetics, Soonchunhyang University College of Medicine, Cheonan, Korea
I. Background and Purpose
• Current status of Clinical Practice Guidelines in Korea
Reliability
• Higher ICC in SG
 Observed in almost domain and stastically significant in overall assessment
• Korean medical guideline information center (KoMGI)
• A couple of CPGs development handbook
• More than 100 of guidelines of the last decade
Development
Table 1. Inter-rater reliability of K-AGREE Ⅱ domain scores in CPG A
• Korean AGREE II instrument
• Web-based evaluation systems
Approval
Appraisal
Differences of
healthcare environments
• KoMGI
Dissemination
A lack of experiences in
evaluation of CPGs
Scoring
Guide
Implementation
Inter-rater disagreement
Domain 1
Scoring Guide group
ICC (95% C.I.)
p-value
0.815 (-0.344~0.995)
0.046
Domain 2
0.430 (-3.137~0.986)
0.251
-0.762 (-11.791~0.955)
0.595
Domain 3
0.722 (0.175~0.938)
0.011
0.473 (-0.565~0.882)
0.121
Domain 4
0.718 (-1.048~0.993)
0.096
-0.296 (-8.411~0.967)
0.503
Domain 5
0.424 (-1.925~0.960)
0.229
0.273 (-2.693~0.950)
0.312
Domain 6
0.000 (-16.443~0.999)
0.391
0.000 (-16.443~0.999)
0.391
0.826 (0.671~0.918)
<0.001
0.680 (0.395~0.850)
<0.001
Domain
Overall
Non-Scoring Guide group
ICC (95% C.I.)
p-value
0.682 (-1.307~0.992)
0.116
Table 2. Inter-rater reliability of K-AGREE Ⅱ domain scores in CPG B
This study aim to examine the effects of the K-AGREE II scoring guide
to reduce inter-rater differences.
II. Methods
Study design
Random assignment
Appraisal
Scoring Guide Group
14
Appraisers
Non-Scoring Guide Group
2
CPG
Reliability
Statistics
• To identify the distribution
• To evaluate the reliability
• To evaluate the consistency
Domain 2
0.769 (-0.675~0.994)
0.068
0.769 (-0.679~0.994)
0.069
Domain 3
0.796 (0.394~0.954)
0.002
0.424 (-0.710~0.871)
0.155
Domain 4
-1.333 (-15.940~0.941)
0.670
0.000 (-6.260~0.975)
0.422
Domain 5
0.888 (0.431~0.992)
0.005
0.272 (-2.696~0.950)
0.312
Domain 6
0.667 (-4.814~1.000)
0.182
0.792 (-2634~1.000)
0.116
Overall
0.869 (0.753~0.939)
<0.001
0.662 (0.362~0.841)
<0.001
Consistency
• Higher association in SG
 Association improve in SG, especially distinctive in CPG B
Consistency
(Korean AGREE II only)
Non-Scoring Guide group
ICC (95% C.I.)
p-value
-0.333 (-0.806~0.966)
0.512
Analysis
Distribution
(Korean AGREE II + Scoring Guide)
Domain 1
Scoring Guide group
ICC (95% C.I.)
p-value
0.821 (-0.303~0.995)
0.043
Domain
Descriptive analysis (Domain specific)
Intra-class correlation (Domain specific)
Association among appraisers (Each items)
III. Results
Table 3. Association of SG in CPG A
Table 4. Association of Non-SG in CPG A
Appraiser
1
2
3
4
Appraiser
1
2
3
4
1
1
.622
.348
.481
2
.622
1
.393
.596
3
.348
.393
1
-.052
4
.481
.596
-.052
1
1
1
-.225
-.434
-.459
2
-.225
1
.373
.502
3
-.434
.373
1
.833
4
-.459
.502
.833
1
Table 5. Association of SG in CPG B
Table 6. Association of Non-SG in CPG B
Appraiser
1
2
3
4
Appraiser
1
2
3
4
1
1
.853
.453
.491
2
.853
1
.651
.749
3
.453
.651
1
.641
4
.491
.749
.641
1
1
1
.556
.441
.127
2
.556
1
.479
.128
3
.441
.479
1
.372
4
.127
.128
.372
1
Distribution
• Higher Scores and higher variability in Non-SG
 Distinctive in domain 2, 3, and 5
IV. Discussion
• Scoring guide reduce the inter-rater disagreemet and improve the overall
reliability of the K-AGREE II instrument.
Domain Scores
 Those effects remarkable in low level CPGs development
 Inter-rater disagreement reflects the healthcare environment characteristics
Domain
Stakeholder
involvement
Scope
Stake Rigour Clarity Applica- Editorial
and
bility Indepen
involve- develop and
-dence
Purpose ment
-ment Presentation
Scope Stake Rigour Clarity Applica- Editorial
and
bility Indepen
involve- develop and
-dence
Purpose ment
-ment Presentation
Fig. 1. Distribution of K-AGREE Ⅱ
Fig. 2. Distribution of K-AGREE Ⅱ
domain scores according Scoring
domain scores according Scoring
Guide users in CPG A
Guide users in CPG B
Limitation
Lack of experience in
stakeholder involvement
Different understanding of
stakeholder
Low level of
CPGs implementation
Applicability
Confuse implementation
with dissemination
Scoring Guide
Providing clear standards
regarding the stakeholder
and the level of participation
Providing clear definition
of implementation
Providing the methodologies
and resources in detail