Architecture Design of Generic
Outcome Adjudication in CTMS
Wenle Zhao, PhD
Medical University of South Carolina, Charleston, SC, 29425, USA
Society for Clinical Trials 36th Annual Meeting
Arlington, VA, USA - May 17-20, 2015
Contents
1. The Motivation
2. The Challenges
3. The Generic Database Model
4. The Automated Coordination
5. Summary
1.The Motivation
2. The Challenges
3. The Generic Database Model
4. The Automated Coordination
5. Summary
Accurate Outcome ( Endpoint) Assessment is Important
Patients
Randomization
Treatment A
Treatment B
Outcome A
Outcome B
Analysis
The validity of the trial results relies on
the accuracy of outcome assessment.
Most Outcome Assessments are Subjective
Outcome
Safety Outcome
Efficacy Outcome
AE is Serious?
Quality of Life
AE is Expected?
Modified Ranking Scale
AE is Related to
Study Treatment?
NIH Stroke Scale
Hamilton Rating Scale for Depression
Glasgow Outcome Scale Extended
Hamilton Rating Scale for Anxiety
The Problem:
The Solution:
Uneven expertise and potential bias
Independent central adjudication
1. The Motivation
2.The Challenges
3. The Generic Database Model
4. The Automated Coordination
5. Summary
Example 1: A Simple SAE Adjudication
CRF
Completed?
May be not.
When it is
expected?
No schedule.
Is serious?
CRF Updated
after submission?
Could happen.
SAE CRF
Submitted
Is expected?
Done
How to notify the
adjudicator?
Auto Email.
Need more
information?
May be yes.
Is related?
Four challenges for coordinating outcome adjudication
 Adjudication procedures vary among projects
No fixed model for every procedure.
 Timing of outcome event unpredictable
No schedule.
 Narrow time window for completion of adjudication
“Time is of the essence.”
 Data changing after adjudication procedure started
The target is moving.
1. The Motivation
2. The Challenges
3. The Generic Database Model
4. The Automated Coordination
5. Summary
The Generic DB Structure for Adjudication Procedures
1
Data (CRF) Form
∞ ID
Form Name
Table Name
1
Adjudication Procedure
1
ID
Form ID
Procedure Name
1
Form Table
ID
Subject ID
Visit ID
Q01
Q02
……
Updated By
Updated On
Adjudication Step
The Model
Allow multiple steps.
Allow multiple updates.
Limit ≤ 5 close-ended Qs.
ID
∞ Procedure ID
Adjudication Step Name
Enable-logic condition
Disable-logic condition
Question 1
Answer 1 Code Group ID
……
Question 5
Answer 5 Code Group ID
Edit User Group ID
View User Group ID
1
1
Form Freeze Table
Freeze Record ID
∞ ID
Subject ID
Visit ID
Q01
Q02
……
Updated By
Updated On
∞
Adjudication Result
ID
∞ Adjudication Step ID
Record ID
Freeze Record ID
Answer 1
……
Answer 5
General Comments
Locked
1
Two Tables are included in the Database Template
One table specifies definitions of
each adjudication step.
The other table stores adjudication
results.
A freeze table is created for each data form table with adjudication
Example 1: Definition of Three-step SAE Adjudication
Adjudication
Step
Step 1
PM Review
(S1)
Enable logic
Disable logic
{S2 submitted after S1}
{New SAE CRF submission}
OR
{S3 submitted after S1}
{S1Q1=1} AND {S1Q2=1}
Step 2
AND
{S2 submitted after S1}
MSM
{no S2 submitted after S1}
OR
Adjudication
AND
{S3 submitted after S2}
(S2)
{no S3 submitted after S1}
Step 3
PM Review
(S3)
Question 1
Question 2
(Q1)
(Q2)
CRF properly Requires MSM
completed?
review?
0 = No
0 = No
1 = Yes
1 = Yes
Serious?
0 = No
1 = Yes
MedWatch
{S2 submitted}
Form needed?
AND
{S3 submitted after S1}
0 = No
{no S3 submitted after S2}
1 = Yes
Question 3
(Q3)
Related to
study drug?
Unexpected? 1 = Unrelated
0 = No
2 = Unlikely
1 = Yes
3 = Possible
4 = Probably
5 = Definitely
Example 2: Definition of Six-step SAE Adjudication
Adjudication Step
Enable logic
Disable logic
Question 1
(Q1)
Question 2
(Q2)
Step 1
Project Manager {New SAE CRF submission}
(PM) Review
{S2 submitted after S1}
CRF properly
completed?
0 = No
1 = Yes
Step 2
Clinical Event
{S1Q1=1} AND {S1Q2=1} AND
Coordinator (CEC) {no S2 submitted after S1}
Review
{S1 submitted after S2} OR
{S3 submitted after S2} OR
{S4 submitted after S2} OR
Event type?
Even packet complete?
1 = Neurological
0 = No
2 = Cardiac
1 = Yes
3 = Systemic
Step 3
Adjudicator 1
Review
(S2Q1=1) AND
{no S5 submitted after S3}
Step 4
Adjudicator 2
(ADJ2) Review
(S2Q1=1) AND
{no S5 submitted after S4}
Step 5
Adjudicator 3
(ADJ3) Review
{S3 submitted after S1} AND
{S4 submitted after S1} AND
{S1 submitted after S5} OR
{S3Q1≠S4Q1 or S3Q2≠S4Q2} AND {S6 submitted after S5}
{no S6 submitted after S5} AND
Step 6
Adjudication
Committee Chair
(ACC) Review
Requires CEC review?
0 = No
1 = Yes
{S1 submitted after S3} OR
{S5 submitted after S3}
{S1 submitted after S4} OR
{S5 submitted after S4}
{S5 submitted after S1} AND
{S5Q1≠S3Q1 or S5Q2≠S3Q2} AND
{S6 submitted after S1}
{S5Q1≠S4Q1 or S5Q2≠S4Q2} AND
{no S1 submitted after S6} AND
Outcome category=?
1 = ischemic stroke
2 = symptomatic
hemorrhagic
transformation of an
ischemic stroke
13 = transient ischemic
attack
Death type =?
1 = ischemic
2 = hemorrhagic
vascular
3 = nonvascular
The Generic Table for Adjudication Results
1. The Motivation
2. The Challenges
3. The Generic Database Model
4.The Automated Coordination
5. Summary
Automated Coordination of Adjudication Activities
Site submits a Case Report Form
N
Form has adjudication
procedure defined?
Y
System evaluates enable logics for
each adjudication steps
N
Enable logic = true &
disable logic = false?
Y
System sends email to [Edit] user
group members
User has adjudication
step [Edit] permission
& Enable logic = true &
disable logic = false?
Y
System displays [Adjudication Edit]
button for the user
User adds/edits
adjudication result and submits
adjudication record
The last freeze record is
the same as the current
CRF record?
N
User views a Case Report Form
N
System creates new freeze record,
links it to the adjudication record.
Y
Example 1: Three-step SAE Adjudication Procedure
Site adds/edits SAE CRF
N
S2Q1 = Yes & S2Q2 = Yes &
S2Q3 = Possible/ Probably / Definitely?
Site submits SAE CRF
Y
Auto email #5 to PM
Auto email #1 to PM
Step S1 = PM Reviews
S1Q1: CRF properly completed?
S1Q2: Requires MSM adjudication?
S1Q1 = Yes?
Y
N
Step S3 = PM Review
S3Q1: MedWatch Form required?
N
S3Q1 = Yes?
Y
System composes MedWatch Form draft
Auto email #2 to Site
N
Auto email #6 to Site
S1Q2 = Yes?
Y
Auto email #3 to MSM
Site edits MedWatch
Auto email #7 to PM
Step S2 = MSM Adjudication
S2Q1: Serious?
S2Q2: Unexpected?
S2Q3: Related to study treatment?
Auto email #4 to PM
PM finalizes MedWatch and submits to FDA
Auto email #8 to All Sites
PM Closes the Adjudication Procedure
Example 2: Six-step Outcome Adjudication Procedure
System determines adjudicators 1, 2, and 3 based on S2Q2
and the Random number R.
Site adds/edits SAE CRF and
upload the event packet
Site submits SAE CRF
Auto email #5 to ADJ1
Auto email #6 to ADJ2
Auto email #1 to PM
Step S3 = ADJ1 Review
S3Q1: Outcome category?
S3Q2: Death type?
Step S4 = ADJ2 Review
S4Q1: Outcome category?
S4Q2: Death type?
Step S1 = PM Reviews
S1Q1: CRF properly completed?
S1Q2: Requires CEC review?
Y
S3 = S4?
N
Auto email #7 to ADJ3
S1Q1 = Yes?
N
Auto email #2 to Site
S1Q2 = Yes?
Y
Auto email #3 to CEC
S2Q1 = Yes?
N
Auto email #4 to Site
Step S5 = ADJ3 Review
S5Q1: Outcome category?
S5Q2: Death type?
N
Step S2 = CEC Adjudication
S2Q1: Event packet complete?
S2Q2: Event type?
Y
Y
PM Closes
this
Adjudication
Procedure
S3= S5 or S4 = S5?
N
Auto email #8 to ACC
Step S6 = Chair Review
S6Q1: Outcome category?
S6Q2: Death type?
Auto email #9 to PM
Y
In this case, the Clinical Event Coordinator (CEC) completed the
adjudication within few hours after the Study Manager (SM) ‘s
completeness review passed.
In this case, the procedure has not been completed yet after two
months, due to CRF data change.
1. The Motivation
2. The Challenges
3. The Generic Database Model
4. The Automated Coordination
5.Summary
Our Experiences
Adjudication
Procedure
Three-step
SAE
adjudication
Six-step
outcome
event
adjudication
From step
To step
Event
count
1st quartile
(25%)
2nd quartile 3rd quartile
(50%)
(75 %)
hrs
hrs
hrs
CRF submit
PM review
1253
21
51
126
PM review
MSM review
1226
5
13
35
MSM review
PM close
1226
27
56
112
CRF submit
PM review
609
26
120
477
PM review
CEC review
599
96
197
364
CEC review
ADJ1/2 review
468
47
204
892
ADJ1/2 review
ADJ3 review
76
56
209
383
ADJ3 review
ACC review
10
23
23
25
PM: project manager
MSM: medical safety monitor
CEC: clinical event coordinator
ADJ1/2: adjudicators 1 and 2
ADJ3: adjudicator 3
ACC: adjudication committee chair
The Strategy to Success
 Allows multiple adjudication procedures per CRF.
 Allows multiple steps for each adjudication procedure .
 Allows different questions be asked for each adjudication step.
 Allows multiple CRF submissions after adjudication started.
 Only close-ended questions are allowed.
 No more than 5 questions are allowed.
 Using automated emails to help adjudication coordination.
Acknowledgement
This research is partly supported by following NIH/NINDS grants:
• U01NS059041 (NETT Palesch, Y. & Durkalski, V.)
• U01NS062778 (ProTECT, Palesch Y.)
• U01NS062835 (POINT, Palesch, Y.)
• U01NS087748 (StrokeNet, Palesch, Y. & Zhao, W.)
 The followings are contributed to the design, development,
validation and user feedback collection to this work:
• Keith Pauls
(DCU senior programmer)
• Jaemyung Kim
(DCU senior programmer)
• Catherine Dillon
(DCU senior data manager)
• The Clinical Coordination Center of the NINDS NETT at
University of Michigan
 Thousands investigators in hundreds institutions and thousands
patients participating our clinical trials.
Thank You!
Contact me at:
zhaow@musc.edu