Evaluation of substitution
maintenance programmes
in Slovenia
• Bob Keizer, Franz Trautmann
• Trimbos Institute, Utrecht (Nl)
• Nino Rode
• Faculty of Social Work, Ljubljana (Si)
Evaluation of SMT in Slovenia:
Original request of MoH
Evaluation of
• quality of services delivered (including patient
satisfaction)
• cost-effectiveness of the programme, assessing
the input (human resource, financial) – output
(client numbers, interventions / services)
relation
• impact of the programme on patients, assessing
the development of a selection of clients in a
specified time interval starting at the time they
enter the programme
Realisation / Adaptations
• Focus of quality interviews (using internationally
agreed standards) on:
–
–
–
–
–
–
–
–
–
–
Staff (quantity and quality: professions, skills, knowledge,
attitude)
Team meetings
Treatment (intake, treatment services offered)
Client approach
Client registration
Accessibility of programme
Monitoring
Guidelines and protocols
Process management
Cooperation with other services
Realisation / Adaptations
• From cost-effectiveness to an analysis of the system
–
–
–
Lack of (standards for relationship) hard input and output data
lack of comparability with other forms of treatment (reference
standards)
Focus on management and organisational structure, financing
system, client numbers, staff number and structure, etc.
• Impact on patients:
–
–
–
–
Lack of ‘hard’ patient data at intake – no RCT
Option of retrospective analysis
Self-report (‘soft’ data, but perception of well-being important
notion in health concept)
Recommendation to develop tool based on EuropASI for the future
Realisation / Adaptations
• Presenting rationale, aims and evidence
for effectiveness of SMT
• Collecting context information
• Including force field analysis to reflect
highly politicised debate / surroundings in
Slovenia
• Comparing Slovenian practice against a
selection of reference countries
– 2 old (D, NL) and 2 new EU Member States (CZ, LT)
Context Input Process Product model
• Context: What needs to be done?
–
–
–
Background information
Force field analysis
Comparing with reference countries
• Input: How should it be done?
–
Quality intreview:
–
System analysis questionnaire:
–
Staff availability, expertise and training, motivation,
–
Efficiency of a system
–
Management organisation and cost of SMT
–
Comparison of the SMT with other services
–
–
–
Clients data (number, retention),
Staff structure
Financing
Context Iinput Process Product model
• Process: Is it being done?
– quality intreview:
– services: patient intake and treatment, procedures,
availability of services, accessibility of the centre
– organization: team-meetings, client registration
system, monitoring, guidelines, process management,
cooperation with other organizations
• Product: Did it succeed?
– Client satisfaction intreview:
– Satisfaction with the services
– Impact of the services
Data sources
• Rationale, aims and evidence:
–
Desk research
• Context information:
–
–
Interviews with key stakeholders
Desk research
• Force field analysis:
–
Interviews with key stakeholders (political parties, involved
Ministries, NGOs, media)
• Reference countries:
–
–
Interviews with 3 key experts p.c. (Focal Point, research,
harm reduction experience)
Desk research
Data sources
• Quality
–
–
–
Interviews with directors + staff (18 SMTCs)
Interviews with directors PHCCs (6 SMTCs: Kočevje, Koper,
Ljubljana, Logatec, Nova Gorica, Velenje)
Patient satisfaction interviews with 75 patients (6 SMTCs:
Kočevje, Koper, Ljubljana, Logatec, Nova Gorica, Velenje)
• Efficiency of system
–
–
–
Interviews: directors + staff (18 SMTCs)
Interviews with directors PHCCs
Desk research
Data analysis
• Mixed methods (qualitative and
quantitative) to get the full picture:
– Largely qualitative analysis
– force field analysis
– analysis of open questions and remarks
– Additional probing for clarification of inconsistencies
– Quantitative analysis
– analysis of frequencies (SPSS)
– comparison with the qualitative data
Relevance of this type of
evaluation
• Serving a mix of quantitative and
qualitative information
• Functioning as thorough SWOT analysis
– Strong points - weak points: serving information for
improving SMT
– Opportunities – threats: how to deal with threats
• Direct indications for adaptations
• Offering the possibility of discussion /
supporting to find a consensus
Findings: Weak Points
• Treatment data collection in use does not allow
formulating an individual treatment plan and
monitoring SMT on individual, centre and national
level;
• Management data collection in use does not serve
reliable and accurate data for thorough auditing;
• There are general guidelines but no (basic) protocols
resulting in substantial differences in treatment
policy and practice between the SMTCs;
• The regular funding is limited regarding staff
capacity for regular SMT services and insufficient for
offering additional services like counselling or social
work (in the SMTC);
Findings: Weak Points
• SMTCs are rather medically oriented, psychosocial aspects of problem drug use do not get
sufficient attention (at least partly to be
explained by insufficient staff capacity);
• SMTCs lack autonomy with regards to staff and
financial management;
• Working in SMTCs (like in other drug services)
has a relatively low status which makes it
difficult to find appropriate staff.
Findings: Strong Points
• The practise of the prescription of substitution
maintenance treatment in Slovenia is of a relative
high standard if compared to other countries. It
scores well on issues like different approaches for
specific target groups, etc.;
• SMT has a high coverage; it covers nearly the whole
country and around one third of the estimated total
of problem heroin users.
• Access to SMT is good (no waiting lists, appropriate
opening hours, no exceptional criteria for entering).
Also in this respect SMT is scoring well compared to
the reference countries;
Findings: Strong Points
• Monthly meetings of SMTC staff facilitates
cooperation between SMTCs and expert
exchange on among others SMT developments,
treatment issues and individual cases;
• Diversity of substitution substances prescribed
(taking into account individual needs);
• Following and picking up latest developments in
SMT quickly (e.g. introduction of Suboxone in
Slovene SMT);
• Overall consensus between staff and clients on
being treated with respect or ‘normally’ by the
other party is an important indicator for quality.
Recommendations
• Standardising work of SMTCs
–
–
Internal communication
Treatment policy through guidelines and checklists (e.g.
take-home medication)
• Standardised and computerised treatment data
collection
–
–
–
Facilitating on centre level treatment planning, monitoring
and evaluation
Facilitating in anonymised, aggregated form monitoring and
evaluation of SMTC work
Standardised recording of treatment services provided
reflecting caseload and case severity
Recommendations
• Management data collection
– Standardised and computerised (central) medication
registration
– Standardised and computerised financial reporting
system
• Shift management responsibility from
PHCC to SMTC:
• Stronger focus on psycho-social aspects
– Through additional staff capacity or
– Cooperation with other services
Recommendations
• Political responsibility:
– Creating conditions for standardising treatment
(guidelines and protocols, etc.) (not only SMTC)
– Creating conditions for a financing system (taking into
account the factual caseload and case severity)
– Auditing (monitoring and evaluation)
– Inter-ministerial cooperation to develop a consistent
and coherent demand reduction policy
– Inter-ministerial guidance to facilitate cooperation
between the different demand reduction services