The Australian Treatment
Outcome Profile (ATOP): Clinical
Monitoring for the Drug & Alcohol
Sector
Anni Ryan, Adrian Dunlop, Jennifer Holmes, Vi Hunt, Kristie Mammen, Raka Tierney, Nicholas Lintzeris.
Presented by Jennifer Holmes
Drug & Alcohol Service
Langton Centre
May 2012
Acknowledgements
 Patients and staff of Jacaranda House, the Langton Centre
and Newcastle OPT Clinics especially: Yvonne Sutton,
Rohan Holland, Betty Jago, Skye O’Donnell, Gonzalo
Rivas, Terry Schofield and Doungkamol Sindhusake.
 MHDAO, Ministry of Health, D&A Research Program Grant
2009/2010
 MHDAO, Ministry of Health, Research Grant 2011/2012
 UK National Treatment Agency, NHS
Overview
 Background to the ATOP
 Stage 1: Validating the ATOP in Australian populations
 Stage 2: Expanding across services in NSW
 Stage 3a: Australian dissemination / uptake
 ?Stage 3b: Developing Composite Measure of Treatment
Outcome (ATOP Treatment Episode Global Outcome)
What is the ATOP?
 A clinician administered 1-page validated ‘instrument’
 Self-report measures across 2 key domains in the preceding 4
weeks
– Substance use
– Health & well being
– Global ratings (0-10) of physical & mental health,
quality of life
– Housing, employment & study, violence, child
protection
Background: Clinical Outcome Monitoring
 Benefits of routine clinical outcome monitoring in
health services are well documented
– clinician administered e.g. HoNOS
– repeat blood tests (CD4 counts in HIV)
 Few examples of successful implementation of in
D&A services
Why is implementation so difficult?
length of the instruments (e.g. >6−10 pages long, taking >20 minutes to
complete)
+
inadequate attention to training
+
excessive data entry requirements
+
poor feedback for patients, clinicians and administrators
=
limited utility and resistance
Developing the ATOP
 Treatment Outcome Profile (TOP)
– a validated tool introduced by the National Treatment
Agency in the UK in 2007
– introduced across all NHS funded D&A services in
England
 Modified for Australian conditions = ATOP
– Reflecting substances commonly used in Australia
– Made more ‘intuitive’ to complete
– Validated in Australian populations & against common
Australian instruments
Why use the ATOP?
Different purposes for different people
 to assist in systematic, documented and ongoing client assessment
 to provide feedback to clients about ‘progress’ over time, and assist
in treatment care planning and motivation
 to assist in clinical handover when transferring/referring clients to
other services
 to assist in service evaluation – helping us to answer the broader
questions of “do our clients get better?”
 as a quality and potential research tool with services collecting
standardised data
Stage One: Project Aims
 Validate the Treatment Outcome Profile (TOP) under
Australian conditions
 Examine implementation and feasibility issues in 3 NSW
public OPT clinics
What do we mean by ‘validated scale’
 Face validity: “does it look right”
 Does it measure things as well as other instruments
– K-10, WHO-QOL, SF-12, PHQ-15
– OTI (substance use, injecting practices)
 Inter-rater reliability:
– do 2 different raters administering the instrument on
the same client get the same score?

Methods
Concurrent validity and inter-rater reliability:
 ATOP administered as part of routine care by clinic staff at 3
month intervals
 Research interview within 72 hrs of last ATOP: gold standard
instruments + repeat ATOP
Implementation and feasibility issues (incl. data management):
 clinician, service manager and patient perspectives
 satisfaction surveys and focus groups
Results - client demographics (n=104)
Gender
Male
Female
62 (59.6%)
42 (40.4%)
Age
0-30
31-40
41-50
51+
32 (30.8%)
37 (35.6%)
22 (21.2%)
13 (12.5%)
Results – concurrent validity
Concurrent validity
Kappa
Spearman’s
0.81
(p<0.001)
0.83 9p<0.001)
Section 2: Injecting Risk Behaviour
Injected with a needle/syringe used by
someone else
Section 3: Crime
Drug Selling
1 (p<0.001)
1 (p<0.001)
Committing Assault or violence
1 (p<0.001)
1 (p<.0001)
Results - concurrent validity
Section 4: Health and Social Functioning
Spearman’s
Psychological Health
vs WHOQuOL Q26
0.62 (p<0.001)
vs K10 total
-0.60 (p<0.001)
Physical Health
vs PHQ15 Total
-0.58 (p<0.001)
SF12 Physical Component Score total
0.65 (p<0.001)
Quality of Life
vs WHOQuOL Q1
0.69 (p<0.001)
vs WHOQuOL Social Relationship Domain total
0.55 (p<0.001)
vs WHOQuOL Environment Domain Total
0.66 (p<0.001)
vs SF12 Physical Component Score total
0.35 (p<0.001)
vs SF12 Mental Component Score total
0.62 (p<0.001)
Results - inter-rater reliability (n=103)
No significant difference between researcher mean and
clinician mean on all continuous items
All dichotomous items correlated (p<0.001)
Clinician feedback (n=20)
easy to administer: 80% agreed
appropriate for my client population: 85% agreed
format & style easy to understand: 85% agreed
length appropriate for routine practice: 70% agreed
appropriate for my setting: 65% agreed, 30% unsure
useful in developing a case plan: 65% agreed, 25% unsure
useful for identifying important problems 45% agreed, 35% ambivalent, 20% disagree
happy to use as part of regular client reviews 45% agreed, 30% ambivalent, 25% disagree
Dislike… crime questions
“too many, too difficult, all clients report nil…”
“Not sure about the crime questions, although they are
important.”
At first ATOP clients say “no” to crime questions but by 3 to 6
months clients start to answer “yes”
Clients express concerns over privacy/confidentiality/how this
information may be used esp. where there is legal/child
protection issues
Client feedback (n=123)
ATOP questions were easy to understand 93%
agreed
Helpful way of looking at how well treatment is
working for me 85% agreed
Helpful to have this same review every few months
85% agreed
Length of the ATOP was “about right” 90% agreed
Other comments…
“ I don’t think everyone will be honest about the crime
questions…”
“asks the right questions: interesting to do it again…”
“some questions feel like they are a stereotype of drug
users…”
“I think there should be more questions about how much
the client feels the program is helping them…”
Conclusions Stage One
 ATOP is a valid instrument for measuring treatment
outcomes in an Australian opioid maintenance treatment
population
 ATOP is compatible with routine clinical practice
 ATOP can feasibly be implemented as part of routine
clinical practice in public OPT Clinics
 Investigate feasibility of ATOP in other treatment settings

Stage 2 Pilot across NSW
Services
 Beyond OTP
 Training
 Protocol & Business Rules
 Data Systems
Stage 2
 Currently trialled across 10-15 sites
– Counselling
– OTP
– Withdrawal Management
 Feasilbility and clinician feedback
 Establishing protocol and business rules for services
 Training Package
– Train the trainer workshops
– Presentations and training manual
Putting the ATOP into Practice: a
crash course for clinicians and
service managers
How to complete the ATOP

Enter
–
–
–
–

Client MRN, date of birth and sex
Your name
Date of ATOP
The stage at which the ATOP is being done
Enter client responses
–
–
–
–
–
–
–
Use the Conversion Table to calculate standard drinks
Nil drug/alcohol use – enter “00” in the total box
Timeline – invite the client to recall the number of days in each of the past four weeks on
which they did something
Before asking Section 2: Items (e) to (h) remind the client about confidentiality (see box)
Yes and no – a simple tick for yes or no
Rating scale – a 10-point scale from poor to good. Together with the client, CIRCLE a
number.
Refused/can’t recall – write “NA” (short for Not Answered) next to the total box, tick box
or rating scale.
ATOP across Australia
 ATOP being explored/used in
– South Australia (DASSA)
– Tasmania (Tas govt sector)
– WA
– Victoria (Turning Point)
– NADA data set (NSW NGO sector)
Designing a Composite Measure of
clinical outcomes in D&A treatment
 The ATOP Research Team
 March 2012
Composite performance measures
 The combination of 2 or more indicators into a single number to
summarize multiple dimensions of performance and to facilitate
comparisons.
– integrate a large amount of information in a format that is
easily understood.
– increasing use by governments to assess performance.
– examples: Dow Jones Index, IQ ratings, NAPLAN, SF-36
In substance abuse treatment
 Challenge: To establish a measure that summarises
multiple clinical outcomes commonly associated with D&A
treatment
– Substance use (primary & secondary drugs)
– General health & well-being (physical health, mental
health, overall QOL)
– High-risk behaviours: injecting, homelessness, child
protection, violence, crime
Why a composite outcome index
 To be able to broadly describe whether or not a treatment
episode is associated with changes in patient well-being
 To be able to state for a particular program:
– 56% patients significantly improved
– 27% had no significant change
– 17% deteriorated
 Allows identification of benchmarks by:
– Drug type (alcohol, opiates etc…)
– Treatment type (e.g. counselling, OTP, withdrawal etc…)
– Patient factors (e.g. ATSI, age, rurality)
Examples to date: opioid treatment
 Dutch Heroin Trial (van den Brink et al. 2003 BMJ 327)
 Used a pre-specified dichotomous, multidomain outcome index
as the primary outcome parameter. Patients considered
‘responders’ if they showed
– at least 40% improvement in at least one of the three
domains (physical, mental, social) at end of treatment
compared with baseline;
– if this improvement was not at the expense of a serious ( ≥
40%) deterioration in functioning in any of the other outcome
domains; and
– if the improvement was not accompanied by a substantial ( ≥
20%) increase in use of cocaine or amphetamines.
Examples to date: cocaine treatment

Paul Crits-Christoph et al. Psychosocial Treatments for Cocaine
Dependence: National Institute on Drug Abuse Collaborative Cocaine
Treatment Study. Arch Gen Psychiatry. 1999;56:493-502

Composite Cocaine Use Measure
– pool information from multiple measures (urine drug tests, ASIs, and
weekly cocaine use inventory) to code each month of treatment as
abstinent vs not abstinent. Any indication of cocaine use from the 3
measures would lead to a “not abstinent” month.

Composite Psychiatric Measure
– To test hypothesis: degree of psychiatric symptoms interacting with the
treatment condition, a composite measure of 4 scales—the Hamilton
Rating Scale for Depression, the Beck Anxiety Inventory, the Brief
Symptom Inventory, and the ASI–Psychiatric Severity Composite score—
was created by converting each scale to a standard score and then
averaging the scores.
Examples to date: Alcohol treatment
Zweben & Cisler. 2003. Clinical and Methodological Utility of a Composite Outcome
Measure for Alcohol Treatment Research. Alcoholism: Clinical and Experimental
Research 27 (10), 1680–1685
 Project MATCH data: composite outcome measure to capture
multiple treatment outcomes among diverse client populations.
– self-reported alcohol consumption, alcohol problems, biological
markers, other areas of functioning (psychiatric dysfunction,
QOL).
 Findings on composite measure:
– 30% clients sustained a ‘remitted’ status (i.e., abstinent or
moderate drinking without problems) over 1-year follow-up;
– 70% of the clients had reached a nonremitted status (i.e.,
heavy drinking and/or problems).
 “The composite outcome index could be used usefully along with
singular measures of consumption to obtain a more complete
picture of what has occurred among clients”.
Methods for combining items into a Composite
Measure






Linear combinations
– Composite = [indicator1×weight1] + [indicator2×weight2] + . . . + [indicatorN×weightN]
– Expert Panel determine which indicators & which weightings
Regression-Based Composite Measures
– If a certain outcome is a ‘gold standard’, the weighting of individual items may be
determined empirically by optimizing the predictability of the gold standard end point.
Latent Trait Composite Measures
– Identify clusters of correlated items & latent trait modeling may be used to combine items
within clusters but not across clusters.
– e.g. substance use outcomes; general health/well-being; high risk / harm behaviours
Any-or-none Scoring of outcome measures
– In this method, a patient is counted as failing if he or she experiences at least 1 adverse
outcome from a list of 2 or more adverse outcomes.
Opportunity Scoring
– Opportunity scoring counts the number of times a given care process was actually
performed (numerator), divided by the number of chances a provider had to give this care
correctly (denominator).
– e.g. individual treatment care plan completed; MH-COPES completed
All-or-None Scoring of Process Measures
– Only those patients who receive all indicated processes of care are counted as
successes.
Recommendations re: Developing
Composite Measures (Peterson et al 2010)
1.
2.
3.
4.
5.
6.
7.
8.
9.
The intended audience & purpose of a composite measure should be explicitly
stated.
Decisions about which measures should be based on clinical importance of
patient outcomes and the reliability of individual performance measures.
Each individual component should be precisely defined to ensure consistent
application in different settings.
The description of the methods used for weighting and combining individual
measures into a composite performance measure should be transparent.
Developers should explore a variety of alternative methods for combining
measures and document whether conclusions about provider performance
differ with use of alternative methods.
Empirical testing needed to assess the properties of a composite measure
score.
Reporting of composite performance measures should be accompanied by
detailed reporting of individual domains and components.
Reporting of composite performance measures should include a measure of
the degree of uncertainty surrounding composite estimates for providers.
Composite performance measures must be reevaluated as that evidence
changes.
Could the ATOP be used to develop a
composite measure of D&A treatment
outcome / success?
 Instrument needs to be used by the sector
 Instrument shown to be robust across diverse populations,
drug types, treatment types
 Measures the main domains that we are most interested
– in over time (e.g. beginning & end of treatment)
– substance use, general health, high-risk behaviours
 Linked to other aspects of NMDS
– identifying primary/secondary drug use, demographics,
treatment types delivered )
ATOP Treatment Episode Global Outcome
Aim
To develop & validate the ‘ATOP Treatment Episode
Global Outcome’, as a means of assigning a global
outcome for each D&A treatment episode that broadly
reflects whether each treatment episode was associated
with a significant improvement, no change, or significant
deterioration in the main clinical domains.
Proposed methodology
ATOP Treatment Episode Global Outcome

National, multisite, project, 2-3 year duration
– Reference group of expert clinicians, consumer reps, researchers, data experts

Recruit large number of clients (100’s) entering a variety of treatment types
(counselling, OTP, withdrawal, rehab) and using different primary drug types
(alcohol, opioids, cannabis, other)
– ATOP at beginning & end of treatment episode (or intervals for OTP, rehab), &
– “Gold standard”: client & clinician global outcome ratings for each treatment
episode (confirmed by independent clinician & client panel, with broad criteria
identified by Reference group for different treatment types).

Identify algorithms for attributing ATOP Treatment Episode Global Outcome for
different treatment modalities & different primary drug types. Statistical methods
(e.g. RUC) to identify algorithms against ‘gold standard’ measure of client &
clinician global ratings.
– How much change in (a) substance use, (b) general health & (c) high-risk
practices is required for the ATOP scores to match client & clinician global
ratings.
The holy grail?
 Data systems that enable us to triangulate data re:
– client characteristics
– services provided (treatment episode data such as type of
service, number of contacts, by whom)
– client outcome
 Would enable benchmarking of services
 NMDS possibly only mechanism to ensure this
occurs.
For Further Information Contact
 ATOP Project Coordinator
– Kristie Mammen
– Kristie.Mammen@SESIAHS.health.nsw.gov.au
– Langton Centre 02 93328777
 Composite Measure Project
– Nicholas Lintzeris
 Validation of ATOP
– Anni Ryan
 Data Collection Systems
– Jennifer Holmes
– Jennifer.Holmes@SydneyMSIC.com
– 02 9360 1191