Cross-National Perspectives on the Practical Application of
Structured Violence Risk Assessment Tools
Abstract
Mental health professionals are routinely called upon to assess the violence risk presented by their
adult patients, frequently aided by structured risk assessment tools. Though surveys of instrument
use and perceived utility have been conducted, these efforts have been largely circumscribed to
individual countries and have not compared the viewpoints of psychologists, psychiatrists, and
nurses. The current proposal aims to address these gaps by developing a Web-based survey that
will be translated into 7 languages and distributed via e-mail to members of professional
organizations in 16 countries. The survey will compare the use of violence risk assessment tools
across nations and compare the perceived role of such instruments in risk assessment and
management across professional groups. The project represents the first multilingual survey of
international violence risk assessment practices and perspectives, and will provide the first data on
the prevalence of specific tool use in a number of countries.
Singh Grant Proposal 1
Statement of the Problem
Prior surveys of risk assessment tool use and perceived utility have been circumscribed to a
small set of countries, namely the United States and the United Kingdom, and have not
compared findings across different groups of mental health professionals. Consequently, many
questions remain regarding the application of these instruments in practice. Specifically, what
structured violence risk assessment tools are most frequently used, how they are being used, and
what are their perceived roles in the assessment and management of future risk for violence? The
proposed project seeks to answer these questions by conducting a multilingual Web-based survey
to compare violence risk assessment practices and attitudes of psychologists, psychiatrists, and
nurses in 16 countries regarding the application of structured violence risk assessment tools.
Relation of the Problem to the State of the Field
Numerous violence risk assessment instruments, composed of static, dynamic, risk and/or
protective factors combined using either actuarial formulae or professional judgment, have been
introduced in recent years (Heilbrun, 2009; Otto & Douglas, 2010). Such structured instruments
have been implemented in psychiatric and correctional settings worldwide, where they are used
to inform medico-legal decisions that have significant implications for individual liberty and
public protection (e.g., involuntary hospitalization, length of hospital detention for treatment,
discharge from psychiatric hospitals and release from prisons).
As there are a number of structured violence risk assessment tools currently available and
recent meta-analytic evidence suggests that they are relatively interchangeable in terms of
predictive validity (Yang, Wong, & Coid, 2011), clinicians are faced with the challenge of
selecting the instrument that they perceive to be the best fit for their population and that will best
guide treatment planning. Knowledge of which tools are currently being used in practice and
which tools colleagues working in similar settings believe have the greatest utility in risk
assessment and management may inform this decision. Survey methodology represents one
approach that can be used to provide clinicians with such information. Further, surveys can
compare clinical practice to best practice recommendations and clarify reasons for any
discrepancies. For example, surveying clinicians could elucidate why, despite a large evidence
base concerning the superior predictive validity of structured approaches (Dvoskin, 2002;
Nicholls, Ogloff, & Ledwidge, 2007; Quinsey, Harris, Rice, & Cormier, 2006), unstructured
assessments of violence risk remain common in practice (Viljoen, McLachlan, & Vincent, 2010).
According to a search of the PsycINFO, EMBASE, and MEDLINE databases, five surveys
have been published investigating violence risk assessment practices. First, Tolman and
Mullendore (2003) conducted a postal survey to identify instruments used by psychologists in
violence risk evaluations of adults for United States courts. Respondents were 93 clinicians
licensed in Michigan and 71 diplomates of the American Board of Forensic Psychology (ABFP).
The researchers found significant differences in specific tool use, with the Violence Risk
Appraisal Guide (VRAG; Harris, Rice, & Quinsey, 1993) being the most commonly used risk
instrument amongst the clinicians and the Psychopathy Checklist-Revised (PCL-R; Hare, 2003)
amongst the diplomates. No measures of perceived utility were included.
Second, Higgins and colleagues (2005) surveyed violence risk assessment practices in mental
health trusts across England. General adult consultants from 45 randomly-selected trusts
responded to a postal survey and reported that although some form of structured instrument was
used in 67% of trusts, there was wide variation in the specific instruments that had been
implemented (details not reported). Though perceived utility was not examined, the authors
suggested that future research should investigate the perspectives of administering clinicians on
structured tools’ roles in assessment and management.
Singh Grant Proposal 2
Third, Archer and colleagues (2006) used a Web-based survey to examine the use of
psychological tests (including those used to assess violence risk in adults) among 152 clinical
psychologists recruited through the American Psychology-Law Society (AP-LS) LISTSERV and
the online ABFP diplomate directory. The researchers concluded that violence risk assessment
instruments such as the VRAG, Historical, Clinical, Risk Management-20 (HCR-20; Webster,
Douglas, Eaves, & Hart, 1997), and the Level of Service Inventory-Revised (LSI-R; Andrews &
Bonta, 1995) were used less frequently than were psychopathic personality scales such as the
PCL-R and the Psychopathy Checklist: Screening version (PCL:SV; Hart, Cox, & Hare, 1995).
Consistent with previous surveys, no measures of perceived utility were included.
Fourth, Khiroya, Weaver, and Maden (2009) surveyed violence risk assessment tool use in
adult medium secure units throughout the United Kingdom. The 29 clinical service directors who
responded to a brief postal questionnaire reported that a variety of violence risk assessment tools
had been implemented (most commonly the PCL-R/PCL:SV and the HCR-20) and that most
units routinely used more than one tool. Though they did examine perceived utility, the authors
measured this using a single item that did not specify utility in assessment and/or management.
Finally, Viljoen and colleagues (2010) conducted a Web-based survey of psychologists to
examine use of violence risk assessment tools in forensic evaluations of adults and juveniles. The
survey was distributed to members of several North American professional organizations as well
as the International Association of Forensic Mental Health Services (IAFMHS). Based on data
from 199 respondents, the authors found that clinicians routinely used structured instruments
(most commonly a PCL measure or the HCR-20 for adults and intelligence scales for juveniles),
though differences in tool use between countries were not examined. In terms of perceived
utility, respondents were asked whether they preferred actuarial tools, SPJ tools, or both;
however, they were not asked the reason(s) for their preference.
These prior studies have advanced our understanding of the use of violence risk assessment
tools in practice, but also share important limitations. First, no surveys have been published
comparing what instruments are used in routine practice in different countries. Second, previous
surveys have not compared patterns of tool use and perceived utility across professional groups.
Third, previous surveys have not attempted to disentangle risk assessment and management
practices. To address these limitations, the proposed project will survey the frequency of use and
perceived role of violence risk assessment tools in assessing and managing risk among
psychologists, psychiatrists, and nurses in 16 countries.
Project Method
Participants
Participants in the proposed study will include psychologists, psychiatrists, and nurses in 16
countries: the United States, Argentina, Australia, Belgium, Canada, Chile, Denmark, Germany,
Hong Kong, Mexico, The Netherlands, New Zealand, Spain, Sweden, Switzerland, and the UK.
For each country, an expert in risk assessment has been recruited to assist in the translation and
distribution of the survey. Participants will be included if they are between 18 to 65 years old and
have conducted at least one assessment of violence risk for an adult. Participants will be
excluded if they are not between 18 to 65 years old and/or have not conducted at least one
assessment of violence risk for an adult.
Materials
Survey. A survey including closed-ended questions based on a review of the violence risk
assessment literature and previous surveys concerning forensic assessment (e.g,. Borum &
Grisso, 1995; Jackson & Hess, 2007; Ryba, Cooper, & Zapf, 2003) has been developed. The
Singh Grant Proposal 3
survey has been designed to be between 20-25 minutes in length (cf. Viljoen et al., 2010). Items
are organized into seven blocks (with a general introduction and debriefing), including: (1)
respondent characteristics, (2) lifetime risk assessment procedures, (3) current risk assessment
procedures, (4) perceptions of actuarial and structured professional judgment instruments, (5)
legal defensiveness, (6) openness to evidence-based practice, and (7) the measurement of
predictive validity. In terms of benefits, respondents who complete the survey will have the
opportunity to provide their e-mail addresses to be entered into a drawing for eight cash prizes,
each valued at $50 USD. Respondents who wish to enter this drawing will do so by selecting a
hyperlink that will redirect them to a Qualtrics page that will allow them to enter this information
and have it stored independently of their survey responses, thus preserving confidentiality. There
are no foreseen risks or other benefits for survey respondents.
Participation letters. Three letters to invite participation have also been developed. These
letters explain the nature and purpose of the survey, apprise respondents that by electing to
participate they are giving their informed consent, and delineate the drawing incentive (see
Procedure section for details on these letters).
Procedures
The proposed study will be conducted in four phases: (1) survey development, (2)
translation, (3) distribution, and (4) data analysis and manuscript preparation.
Phase 1: Survey development. In Phase 1, the Web-based survey has been developed to
query the use of specific violence risk assessment tools in practice and the perceived utility of
those instruments in predicting future violence and making risk management decisions. Webbased surveys are an appropriate alternative to paper-based approaches when sampling mental
health professionals because of their generally high levels of Internet access (Solomon, 2001).
The survey has been constructed and will be administered using Qualtrics, an electronic survey
software. Qualtrics has been used in recent surveys of forensic mental health workers (e.g.,
Kimonis Fanniff, Borum, & Elliott, 2011; Williams, 2011) and has a number of benefits,
including data collection through a secure server, use of logic statements to guide respondents
through the survey, libraries of customizable question templates, and a continuous file saving
function to minimize data loss due to browser crashes. Qualtrics is licensed and supported at the
University of South Florida.
A review of the risk assessment literature and previous surveys of tool use was used to
compile a list of items for the survey. The items as well as the participation letters detailing the
purpose and nature of the survey were screened by my study team members in the Department of
Mental Health and Law: Drs. Sarah Desmarais, Randy Otto, Paul Stiles, and Richard Van Dorn.
In addition to this role, Drs. Desmarais and Van Dorn will assist in addressing regulatory issues
and in conducting analyses. Drs. Stiles and Otto will take on supervisory roles and assist with
data interpretation and ultimate publication. The primary investigator (Jay Singh) will be
responsible for coordinating these roles, communicating with the IRB, and data management.
Phase 2: Translation. In Phase 2, the survey and participation letters will be professionally
translated into seven languages, including Danish, Dutch, French, German, Spanish (Latin
American), Spanish (Spain), and Swedish. Translated materials will be sent to the international
collaborators for proofreading to ensure accuracy of the translations in terms of both content and
meaning. The collaborators also will make country-specific modifications to survey content, as
necessary. For example, changes to response options for items concerning professional degrees
and types of clinical evaluations conducted will be made where appropriate.
Phase 3: Distribution. In Phase 3, the participation letters and survey will be distributed
either electronically using e-mail addresses contained in either LISTSERVs or online
Singh Grant Proposal 4
membership directories of national and international professional organizations (with the
permission of the relevant list or website administrators) or through links on the websites of
professional organizations. Alternatively, professional organizations may wish to notify their
membership via a newsletter advertisement. Invitations letter and surveys will be distributed by
each participating country’s collaborator in the native language to the membership of three
classes of national organization: (1) a national organization of forensic psychologists, (2) a
national organization of forensic psychiatrists, and (3) a national organization of forensic
psychiatric nurses. Organizations from which permissions will be requested include: the
Argentinean Association of Behavioral Science, the American Academy of Psychiatry and the
Law, the American Board of Forensic Psychology, the American Psychology-Law Society, the
Association of German Professional Psychologists, the American Psychiatric Nurses
Association, Forensic Behavioral Services Incorporated, the Australian Psychological Society,
the Belgian Federation of Psychologists, the British Psychological Society, the Canadian
Psychological Association, the Dutch Association of Psychiatrists, Dutch Institute of
Psychologists (Forensic Section), Verpleegkundigen and Verzorgenden Nederland, the Forensic
Psychiatric Nurses Council, the German Psychological Association, the Mexican Psychological
Association, the Netherlands Institute of Psychologists, the New Zealand Psychological Society,
the Royal College of Forensic Psychiatrists, the Spanish Federation of Psychological
Associations, the Spanish Psychological Association, the Swedish Psychological Association,
and the UK Nursing and Midwifery Council. In addition, the English-language version of the
invitation letter with links to all translations of the survey will be distributed to the membership
of the following international organizations related to forensic mental health: International
Association of Forensic Mental Health Services, International Association for the Treatment of
Sexual Offenders, International Association of Forensic Nurses. The primary investigator (JPS)
will have responsibility for recruitment in the US and to international organizations. A list of all
organizations contacted will be maintained and provided to collaborators to prevent crosspostings.
To the extent possible, survey distribution will follow the widely-accepted Dillman method
(Dillman, Smyth, & Christian, 2009). In concordance with this approach, initial participation
letters will be sent out via e-mail on a Friday and will contain direct and active links to the
survey. Two reminder e-mails will be sent in seven day increments after the initial distribution to
remind potential research participants about the study. A fourth and final e-mail will include the
participation letter and relevant hyperlink, indicating that it is the final opportunity to participate.
Based on previous Web-based surveys of violence risk assessment practices that have used
similar methodology (Archer et al., 2006; Viljoen et al., 2010), a response rate between 45% and
65% is expected, comparable to electronic surveys in other fields (Baruch, 1999). Data will be
monitored as it is entered into Qualtrics by respondents.
Phase 4: Data analysis and manuscript preparation. In Phase 4, respondent data will be
exported from Qualtrics to STATA 10.1 (StataCorp, 2007) for analysis. Data will be kept on a
password protected desktop computer in a locked office in the Department of Mental Health Law
and Policy. Frequency distributions will be examined and measures of central tendency and
dispersion will be calculated for all variables. Differences in ratings of perceived utility between
individual countries will be explored descriptively, and, for omnibus testing, countries will be
collapsed into three groups (North America, UK, and other).1 For survey items with dichotomous
responses, differences between psychologists, psychiatrists, and nurses across countries will be
assessed using Kruskal-Wallis ANOVAs with post-hoc Pearson χ2 tests. For survey items with
1
These groups were chosen to allow me to examine whether the findings of previous surveys that have focused on tool use in
North America and the UK are generalizable to other countries.
Singh Grant Proposal 5
continuous responses, differences between professional groups and across countries will be
analyzed using parametric ANOVAs with post-hoc Bonferonni tests.
It is anticipated given the number of organizations being surveyed in the 16 countries that the
sample size will be approximately 10,000 participants. Power analyses using G*Power 3 (Faul et
al., 2007) suggest that 159 respondents (n = 53 per subgroup) need to be recruited to detect
moderate effect size (F = 0.25) differences across professions and geographic location. To have
adequate power to examine profession by geographic location interactions, 42 psychologists,
psychiatrists, and nurses need to be recruited in North America, the UK, and in the other
participating countries. Thus, we anticipate that there will be more than enough participants to
conduct the intended analyses. In all analyses, covariates (e.g., sex, age, race/ethnicity) will be
assessed using standard procedures and controlled for as necessary.
When analysis has been completed, a manuscript will be prepared for submission to a peerreviewed journal. Results also will be used to support a federal grant application (NIMH
K99/R00) focused on improving the efficiency of violence risk assessment through the
application of innovative technologies. Five years after the final report on the data has been
submitted for publication, the data will be destroyed by deleting the survey on Qualtrics
(removing all associated data) and all associated STATA files will be placed into the Microsoft
Windows electronic “Recycle Bin” and the “Empty the Recycle Bin” function will be used.
Anticipated Contribution
Despite the proliferation of violence risk assessment tools in criminal justice and mental
health systems, research on how these tools are actually being used in practice has remained rare
(Elbogen, Huss, Tomkins, & Scalora, 2005). Work comparing risk assessment practices in
different countries and professional groups has been particularly scarce. Therefore, the proposed
study, with its focus on the use and perceived utility of violence risk assessment tools in practice
by psychologists, psychiatrists, and nurses in 16 countries, represents an original contribution
that may inform research, clinical practice, and public policy (and, therefore, has the potential to
benefit society). Specifically, the survey will speak to whether recent research findings such as
the interchangeability of violence risk assessment tools in terms of predictive validity (Yang et
al., 2010), the instability of risk estimates produced by actuarial instruments when applied to
individuals (Hart, Cooke, & Michie, 2007), and the importance of clinical training in the
administration of structured assessment instruments (McNiel et al., 2008) are reflected in the
practice and attitudes of psychologists, psychiatrists, and nurses. The survey also may influence
practitioners’ selection of which risk assessment tools to implement, as it will identify which
instruments are perceived to be the most useful in assessing the likelihood of future violence and
which in identifying potential treatment targets to reduce violence risk. Further, survey results
will clarify which risk assessment tools are being used to meet current guidelines by professional
organizations of psychologists (e.g., American Psychological Association, 2006), psychiatrists
(e.g., American Psychiatric Association, 2004; National Institute for Clinical Excellence, 2009),
and nurses (e.g., Nursing and Midwifery Council, 2004) which recommend that violence risk be
assessed using evidence-based methods. Finally, the survey may inform public policy by
identifying countries where the use of empirically-validated violence risk assessment tools is not
common despite the large evidence base demonstrating the superiority of structured methods
over unstructured clinical judgment.
Singh Grant Proposal 6
References
American Psychiatric Association. (2004). Practice guidelines for the treatment of patients with
schizophrenia. Arlington, VA: American Psychiatric Association.
American Psychological Association Presidential Task Force on Evidence-Based Practice.
(2006). Evidence-based practice in psychology. American Psychologist, 61, 271-285.
Andrews, D. A., & Bonta, J. (1995). LSI-R: The Level of Service Inventory-Revised. Toronto,
ON: Multi-Health Systems.
Archer, R. P., Buffington-Vollum, J. K., Stredny, R. V., & Handel, R. W. (2006). A survey of
psychological test use patterns among forensic psychologists. Journal of Personality
Assessment, 87, 84-94.
Baruch, Y. (1999). Response rates in academic studies: A comparative analysis. Human
Relations, 52, 421-434.
Borum, R., & Grisso, T. (1995). Psychological test use in criminal forensic evaluations.
Professional Psychology: Research & Practice, 26, 465-473.
Dillman, D., Smyth, J., & Christian, L. (2009). Internet, mail, and mixed-mode surveys: The
tailored design method (3rd ed.). Hoboken, NJ: John Wiley & Sons, Inc.
Dvoskin, J. (2002). Knowledge is not power – Knowledge is obligation. Journal of the American
Academy of Psychiatry & the Law, 30, 533-540.
Elbogen, E., Huss, M. T., Tomkins, A. J., & Scalora, M. J. (2005). Clinical decision-making
about psychopathy and violence risk assessment in public sector mental health settings.
Psychological Services, 2, 133-141.
Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G*Power 3: A flexible statistical
power analysis for the social, behavioral, and biomedical sciences. Behavior Research
Methods, 39, 175-191.
Hare, R. D. (2003). The Hare Psychopathy Checklist-Revised (2nd ed.). Toronto, ON: MultiHealth Systems.
Harris, G. T., Rice, M. E., & Quinsey, V. L. (1993). Violent recidivism of mentally disordered
offenders: The development of a statistical prediction instrument. Criminal Justice &
Behavior, 20, 315-335.
Hart, S. D., Cox, D. N., & Hare, R. D. (1995). Manual for the Hare Psychopathy ChecklistRevised: Screening Version (PCL:SV). Toronto, ON: Multi-Health Systems.
Hart, S. D., Michie, C., & Cooke, D. J. (2007). Precision of actuarial risk assessment
instruments: Evaluating the ‘margins of error’ of group v. individual predictions of
violence. British Journal of Psychiatry, 190, s60-s65.
Heilbrun, K. (2009). Evaluation for risk in violence in adults. New York: Oxford University
Press.
Higgins, N., Watts, D., Bindman, J., Slade, M., & Thornicroft, G. (2005). Assessing violence
risk in general adult psychiatry. Psychiatric Bulletin, 29, 131-133.
Jackson, R., & Hess, D. T. (2007). Evaluation for civil commitment of sex offenders: A survey
of experts. Sexual Abuse: A Journal of Research & Treatment, 19, 425-448.
Khiroya, R., Weaver, T., & Maden, T. (2009). Use and perceived utility of structured violence
risk assessments in English medium secure forensic units. Psychiatrist, 33, 129-132.
Kimonis, E. R., Fanniff, A., Borum, R., & Elliott, K. (2011). Clinician’s perceptions of
indicators of amenability to sex offender-specific treatment in juveniles. Sexual Abuse: A
Journal of Research & Treatment, 23, 193-211.
McNiel, D. E., Chamerblain, J. R., Weaver, C. M., Hall, S. E., Fordwood, S. R., & Binder, R. L.
(2008). Impact of clinical training on violence risk assessment. American Journal of
Psychiatry, 165, 195-200.
Singh Grant Proposal 7
National Institute for Health and Clinical Excellence. (2009). Core interventions in the treatment
and management of schizophrenia in primary and secondary care. London: National
Institute for Health and Clinical Excellence.
Nicholls, T. L., Ogloff, J. R. P., & Ledwidge, B. (2007). Is the profound distrust of unbridled
clinical opinion in the violence risk assessment field unfounded? Poster presented at the
4th Annual Forensic Psychiatry Conference, Victoria, BC.
Nursing and Midwifery Council. (2004). Standards of proficiency for specialist community
public health nurses. London: Nursing and Midwifery Council.
Otto, R. K., & Douglas, K. S. (Eds.) (2010). Handbook of violence risk assessment. New York,
NY: Taylor & Francis Group.
Quinsey, V. L., Harris, G. T., Rice, M. E., & Cormier, C. A. (2006). Violent offenders:
Appraising and managing risk (2nd ed.). Washington, DC: American Psychological
Association.
Ryba, N., Cooper, V., & Zapf, P. (2003). Juvenile competence to stand trial evaluations: A
survey of current practices and test usage among psychologists. Professional Psychology:
Research & Practice, 34, 499-507.
Solomon, D. J. (2001). Conducting web-based surveys. Practical Assessment, Research, & Evaluation,
7. Retrieved November 30, 2011, from http://PAREonline.net/getvn.asp?v=7&n=19
StataCorp. (2007). Stata statistical software: Release 10.1. College Station, TX: StataCorp LP.
Tolman, A., & Mullendore, K. (2003). Risk evaluations for the courts: Is service quality a
function of specialization? Professional Psychology: Research & Practice, 34, 225-232.
Viljoen, J. L., McLachlan, K., & Vincent, G. M. (2010). Assessing violence risk and
psychopathy in juvenile and adult offenders: A survey of clinical practices. Assessment,
17, 377-395.
Webster, C. D., Douglas, K. S., Eaves, D., & Hart, S. D. (1997). HCR-20: Assessing risk for
violence (version 2). Burnaby, BC: Simon Fraser University, Mental Health, Law, and
Policy Institute.
Williams, A. M. (2011). Changes in community behavioral healthcare organizations subsequent
to the implementation of Medicaid managed care. Unpublished doctoral dissertation,
Indiana University of Pennsylvania, Indiana, PA.
Yang, M., Wong, S. C. P., & Coid, J. (2010). The efficacy of violent prediction: A meta-analytic
comparison of nine risk assessment tools. Psychological Bulletin, 136, 740-767.