McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying
relevant research: Case study on child protection social workers’ resilience. Research on
Social Work Practice, 22(6), 626-636. doi: 10.1177/1049731512453209.
Systematically Identifying Relevant Research: Case
Study on Child Protection Social Workers’ Resilience
Paula McFadden1, Brian J Taylor2, Anne Campbell3 and Janice McQuilkin4
1 Doctoral Candidate
2 Professor of Social Work
3 Senior Lecturer in Social Work
4 Subject Assistant Librarian
all at:
University of Ulster, Northern Ireland
Corresponding Author:
Professor Brian Taylor, University of Ulster, Social Work Room 21C13 Dalriada, Shore
Road, Newtownabbey, Northern Ireland BT37 0QB. Email: bj.taylor@ulster.ac.uk
Abstract
Context: The development of a consolidated knowledge base for social work requires
rigorous approaches to identifying relevant research.
Method: The quality of ten databases and a web search engine were appraised by
systematically searching for research articles on resilience and burnout in child protection
social workers.
Results: ASSIA, Social Services Abstracts and Social Sciences Citation Index (SSCI) had
greatest sensitivity, each retrieving more than double any other database. PsycINFO and
CINAHL had highest precision. Google Scholar had modest sensitivity, and good precision
in relation to the first 100 items. SSCI, Google Scholar, Medline and CINAHL retrieved the
highest number of hits not retrieved by any other database.
Conclusion: A range of databases is required for even modestly-comprehensive searching.
Advanced database searching methods are being developed but the profession requires
greater standardisation of terminology to assist in information retrieval.
Key Words: Databases, bibliographic; evidence based practice; information storage and
retrieval; review literature as topic; systematic literature searching.
Introduction
The need to base practice and policy in the social work profession on a foundation
of robust evidence and research is stronger than ever (Lindsey and Kirk, 1992; Darragh
and Taylor, 2008). Not only do tax-payers, charity-donors, politicians and managers seek
evidence of effective outcomes and management of social work services (Taylor and
page 1
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
Campbell, 2011) but also health service professionals seek evidence of effective
psychosocial interventions as alternatives to medication (Layard et al., 2007). A systematic
approach is required to the identification, appraisal, synthesis, dissemination and
application of robust research to support decision making within the profession (Cook et
al., 1997; Dempster, 2003; Taylor, 2010). The focus of this article is on the first of these
elements: the systematic identification of relevant research.
Systematic approaches to identifying relevant research are required to ensure that
social work practice is based on the best available evidence. The computer age has
transformed the processes of accessing research of interest to social work. The everincreasing volume of published material requires the development of methods to identify
material on the computerised data management facilities now available. The busy social
worker in any setting requires methods that are efficient as well as effective to identify
relevant research (Beall, 2007). For this purpose effectiveness is defined as the capability
of a database to identify all the relevant published items on a topic. Efficiency is defined as
ensuring that the searcher’s time and efforts are well used for the task of retrieving
relevant items, particularly by avoiding the retrieval of irrelevant items (Hoogendam et al,
2009). These qualities are represented here, respectively, by measures of sensitivity (the
ability to retrieve as many as possible of the items that you want) and precision (the ability
to avoid in the process giving you irrelevant items which you do not want) as defined
below. There is limited study on database searching in relation to identifying social work
research, where the complexities of practice and the nuances of language across
jurisdictions, states and cultures create particular challenges (Bender et al., 2011, Coren et
al., 2011; Golder et al., 2008; Taylor et al., 2003 and 2007; Taylor, 2009; Tripodi et al.,
2011).
Ten databases and a popular web search engine that provide abstracts of articles
of interest to the profession were evaluated by carrying out a search in relation to
resilience and burnout amongst child protection social workers. This topic lends itself to
empirical research using a wide variety of methods as is common in research relevant to
social work (Matthews et al., 1999). Hence the results of this study on database searching
are of interest beyond this particular topic, as a key issue is being able to retrieve material
using diverse research methods (Killick and Taylor, 2009). By contrast systematic reviews
that focus on effectiveness (Taylor, 2012) are able to capitalise on the search facilities that
have been developed specifically to add precision to the retrieval of experimental studies
on particular databases. This topic also encapsulated a range of terminology across
cultures, organisations and jurisdictions, as is common in other aspects of social work.
Methodology
Judging the availability of articles on a database is not as simple as knowing which
journals are abstracted (Crumley et al., 2005; Papaioannou et al., 2009). There may be
inconsistencies in coverage of intended journals (Holden et al., 2008 and 2009; Shek,
2008) and variations in indexing against the database thesaurus (a standard set of search
terms, such as MeSH, against which each article is indexed). In addition the structure of
the thesaurus may itself be more or less useful for particular topics of interest to the
profession. The method used here has been designed to enable an appraisal of database
quality that takes into account all three of these factors by focusing on what can be
retrieved with a relatively sophisticated search. The method is in effect asking a clinical
question and analysing the yield rather than assessing the retrieval of each database
against a given pool of articles (Moseley et al., 2009). The premise is that the searcher will
use the database thesaurus and whatever facilities are available on that database so as to
conduct as efficient a search as possible.
page 2
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
Social workers wishing to search for studies using a particular research design are
hindered by the fact that most databases of interest have very limited filters regarding
methodology. It is not feasible to use database facilities to filter by research method except
on one or two databases with more sophisticated filters (such as Medline and PsycINFO)
and then only in terms of types of experimental methods. There has been some
development in creating search terms to locate studies using qualitative research designs
(Fisher et al., 2006; Flemming and Briggs, 2006) and the development of established
search strings for such concepts is a great asset (Jenkins, 2004; Lee et al., 2012;
Shlonsky et al, 2011; White et al., 2001). Generally, however, searching must be done by
topic with subsequent filtering by hand in relation to the more precise question of interest.
This study was designed therefore to focus on this type of searching.
The overall study design contained the following stages, the main components of which
are described in more detail below.
1. The search question (study topic) was defined.
2. Inclusion and exclusion criteria were specified.
3. Relevant electronic bibliographic databases for searching were identified.
4. A general search structure was constructed with clear concept groups.
5. A search formula for each database was constructed to most effectively operationalize
the general search structure.
6. Available facilities on each database (such as the thesaurus of index terms on most
databases, and filters where provided) were identified and used where appropriate.
7. The search formulae were run on each database and the total hits retrieved by each
search were saved.
8. The first two authors independently appraised the hits retrieved from each database
against the inclusion criteria, subsequently comparing notes to seek agreement on
relevant items, with any disagreements being resolved by the third author.
9. A composite list of all relevant items was created by eliminating duplications by hand.
10. The ability of each database to retrieve relevant items (sensitivity) was calculated using
this total of number of relevant items retrieved across all databases as a denominator
(see definition below).
11. The ability of each database to avoid retrieving irrelevant items was calculated in terms
of its precision (see definition below).
12. To give an indication of overlap between databases, and the importance of using each
particular one, the number of unique relevant items identified on each database was
calculated.
The search question
The study topic was defined as “an examination of the effectiveness and efficiency
of electronic databases for retrieving research on the topic of resilience and burnout
amongst child protection social workers”. Turnover and staff retention are key issues at
present as undesired turnover has huge human and financial implications as well as the
loss of expertise and relationships with clients and families (Mor Barak et al., 2001).
Inexperience in teams and the need to retain competent and committed staff (Ellett et al.,
2009) are a high priority in this critical area of social work practice.
Inclusion and exclusion criteria
To be included papers had to focus on social workers in child protection or child
welfare work. Papers on resilience amongst other professions and in other areas of social
work practice were excluded. Papers had to report empirical research or be reviews of
relevant research. Government papers, policy documents, theoretical material and
descriptive case study articles were excluded. It became clear during scoping that a
page 3
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
number of relevant empirical studies were available through dissertation publications by
research students at Master’s and Doctoral levels so grey literature was included.
Extending the search to this type of grey literature should reduce the potential for
publication bias (Burdett et al., 2003) although it is recognised that this is likely to be less
of an issue for social work than for some other areas of study (Egger et al., 2003; Taylor,
Dempster and Donnelly, 2007). The studies selected were limited to English language due
to the high cost of translation. The study focused on a ten year time frame: 01 st January
2000 to 31st December 2009.
Selection of databases
Ten academic and professional databases and a web search engine were used for
this case study. The selection was influenced by accessibility within our work environment,
a scoping exercise and recent guidance from the Social Care Institute for Excellence in
London (Clapton, 2010). All of these databases include abstracts for articles. The
databases included were: Applied Social Sciences Index and Abstracts (ASSIA),
Cumulative Index to Nursing and Allied Health (CINAHL), Cochrane Library, Index to
Theses, Medline, PsycINFO, Social Care Online (SCO), Social Services Abstracts (SSA),
Social Sciences Citation Index (SSCI) and ZETOC. Google Scholar, a commercial web
search engine was also used.
Development of search formulae
The structure for developing search formulae suggested by Taylor (2003; see also
Taylor et al., 2003 and 2006) as developed in guidance by the Social Care Institute
Guidance (Coren and Fisher, 2006) was used to turn the review question into concept
groups. The four main concepts used were:
 (stress/burnout/workload/fatigue/turnover/retention/exhaustion/depersonalisation/)
AND
 (resilience/coping/adjustment/thriving/satisfaction/hardiness/) AND
 (child protection/child care/child welfare/) AND
 (organisation/workplace/supervision/job engagement/work attitudes/).
As an example, Figure 1 illustrates this general search structure as operationalized on
PsycINFO database. It was not possible to use methodological search filters (Shlonsky et
al., 2011) as we were interested in papers on a wide variety of aspects of the topic.
_______________________________________________________________
1. Stress, Psychological/ or Occupational Diseases/ or Burnout, Professional/ or
burnout.mp.
2. burned out.mp.
3. burnt out.mp.
4. Workload/ or work stress.mp.
5. occupational stress.mp.
6. Fatigue/ or vicarious trauma.mp.
7. compassion fatigue.mp.
8. vicarious experiences.mp.
9. psychological stress.mp. or Stress, Psychological/
10. work place stress.mp.
11. job related stress.mp.
12. Stress Disorders, Post-Traumatic/ or emotional trauma.mp.
13. emotional trauma.mp.
14. Personnel Turnover/ or intention to leave.mp.
15. Personnel Turnover/ or staff turnover.mp.
16. employee turnover.mp.
page 4
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
17. retention issues.mp.
18. low retention.mp.
19. emotional exhaustion.mp.
20. Depersonalization/ or depersonalisation.mp.
21. inefficacy.mp.
22. (stress disorder$ or diminished interest or professional burn*out or poor staff
retention).mp. [mp=title, original title, abstract, name of substance word, subject
heading word, unique identifier]
23. 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11 or 12 or 13 or 14 or 15 or 16
or 17 or 18 or 19 or 20 or 21 or 22
24. resilience.mp. or Adaptation, Psychological/ or Resilience, Psychological/
25. coping behaviour.mp.
26. psychological adjustment.mp.
27. psychological endurance.mp.
28. emotional adjustment.mp.
29. emotional resilience.mp.
30. occupational adjustment.mp.
31. intention to remain employed.mp.
32. Job Satisfaction/ or Personnel Loyalty/ or staff retention.mp.
33. thriving.mp.
34. vicarious resilience.mp.
35. compassion satisfaction.mp.
36. psychological hardiness.mp.
37. emotional hardiness.mp.
38. emotional resilience.mp.
39. self efficacy.mp. or Self Efficacy/
40. professional efficacy.mp.
41. sense of coherence.mp.
42. job engagement.mp.
43. job involvement.mp.
44. emotional intelligence.mp. or Emotional Intelligence/
45. strengths based practice.mp.
46. job satisfaction.mp. or Job Satisfaction/
47. role satisfaction.mp.
48. occupational resilience.mp.
49. coping strategies.mp.
50. protective factors.mp.
51. (coping behaviour$ or coping behavi* or emotional endurance or psychological
adaptation).mp. [mp=title, original title, abstract, name of substance word, subject
heading word, unique identifier]
52. 24 or 25 or 26 or 27 or 28 or 29 or 30 or 31 or 32 or 33 or 34 or 35 or 36 or 37
or 38 or 39 or 40 or 41 or 42 or 43 or 44 or 45 or 46 or 47 or 48 or 49 or 50 or 51
53. Child Abuse/ or Child Abuse, Sexual/ or child protection social work.mp. or Child
Welfare/
54. Personnel Turnover/ or Social Work/ or Personnel Selection/ or child welfare
worker.mp.
55. child protection officer.mp.
56. Professional-Family Relations/ or social casework.mp.
57. child protective services.mp.
58. child care social workers.mp.
59. child neglect.mp.
page 5
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
60. Parent-Child Relations/ or child maltreatment.mp.
61. child homicide.mp. or Infanticide/
62. social workers.mp.
63. social services.mp.
64. child abuse reporting.mp. or Mandatory Reporting/
65. child care workers.mp.
66. (child protection social worker$ or child welfare worker$ or social casework* or
prot* services or child* protect* or social work or child protect* teams or child abuse
or child neglect or child maltreatment or child homicide or child protection social
work).mp. [mp=title, original title, abstract, name of substance word, subject
heading word, unique identifier]
67. 53 or 54 or 55 or 56 or 57 or 58 or 59 or 60 or 61 or 62 or 63 or 64 or 65 or 66
68. organizational characteristics.mp.
69. Organizational Culture/ or organizational factors.mp.
70. job characteristics.mp.
71. Workplace/ or job involvement.mp.
72. job engagement.mp.
73. organizational structures.mp.
74. work load.mp. or Workload/
75. case load.mp.
76. working conditions.mp.
77. professional supervision.mp.
78. adequate supervision.mp.
79. vocational occupation.mp.
80. work attitudes.mp.
81. (organi?ational factors or organi?ational structures or organi?ational
characteristics or job characteristics$ or work load$ or case load$ or work*
conditions or professional satisfaction or professional stab* or job requirement$ or
job engagement or job pressure$ or vocational occupation$ or area of work life or
social support or work*load).mp. [mp=title, original title, abstract, name of substance
word, subject heading word, unique identifier]
82. 68 or 69 or 70 or 71 or 72 or 73 or 74 or 75 or 76 or 77 or 78 or 79 or 80 or 81
83. 23 and 52 and 67 and 82
84. limit 83 to (abstracts and english language and yr="1998 -Current" and english)
85. (empirical studies or research).mp. [mp=title, original title, abstract, name of
substance word, subject heading word, unique identifier]
86. 84 and 85
Figure 1 Search as operationalized on PsycINFO database
Terms used to express these concepts (that is, the range of terms within the
brackets in the general search structure above) were developed so as to include
international usage during the defined time period and variations in spelling. For example,
in USA and Canada studies often refer to “child welfare”, whereas UK and Australian
studies tend to refer to “child protection”. Index terms (MeSH/thesaurus/database key
words) were used where appropriate terms existed, as well as text term searching.
Available database facilities were used to improve precision. Search filters were
used where provided by the database to improve precision (White et al., 2001) such as to
eliminate articles not indexed as “research” on some databases.
Sensitivity and precision
page 6
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
The first quality required of a database is its ability to retrieve as many as possible
of the published relevant articles on the topic. Sensitivity is a measure of how many of the
relevant articles are retrieved by a database. The sensitivity of the search on a particular
database is calculated by dividing the number of relevant studies identified on that
database by the total number of relevant studies retrieved. The sensitivity might be
regarded as the ‘completeness’ of the database in terms of the fraction of the total number
of relevant items potentially available that are retrieved by the search on that database.
Some authors in information science use the term recall for this measure (Stokes et al.,
2009; Vaughan, 2004; Walters, 2009). The term sensitivity is used here as it is widely used
in disciplines such as health, social care and psychology with the same meaning as in
relation to diagnostic and screening tests.
The second quality required of a database is its ability to avoid retrieving irrelevant
items, so that the relevant items are more easily retrieved from among the ‘hits’ produced
by the database. This is most appropriately measured in terms of precision. Precision may
be defined as the number of relevant hits identified by a database divided by the total
number of hits (relevant and irrelevant) retrieved by that same database search. The
precision might also be termed the positive predictive value (Haynes & Wilczynski, 2004),
a term used to express this concept in some fields of clinical practice.
It is not possible to use the concept of selectivity in relation to database searching
with this research design as in practice the total number of items on the database from
which to calculate the total number of irrelevant items is not known. This study uses the
terminology and definitions of sensitivity and precision as used widely in the literature on
information retrieval in disciplines such as health care, social work and psychology (Adams
et al., 1994; Boynton et al., 1998; Dickerson et al., 1994; Golder et al., 2008; Hay et al.,
1996; Haynes & Wilczynski, 2004; Stokes et al., 2009; Taylor et al., 2003; Taylor, Wylie et
al., 2007; Watson and Richardson, 1999).
For the purposes of a Cochrane Review of the effectiveness of health and social
care interventions, one might seek high sensitivity and ignore precision so as to attain as
comprehensive inclusion as possible. However that is only possible because of the limited
number of research designs of interest, and because of (1) the availability of search filters
designed for this purpose and provided on relevant databases such as Medline (Minozzi et
al., 2000) and (2) the availability of databases – such as the Cochrane Library – dedicated
to studies using these research designs (Hoogendam et al, 2009). When the focus
involves research using other than experimental methods, database facilities are rather
more limited. Therefore searches in social work generally need to seek high precision as
well as high sensitivity so that the searcher is not inundated with irrelevant material (Allison
et al, 1999; Boynton et al., 1998).
Sensitivity and precision were calculated for each search. In the case of Google
Scholar items deemed relevant by the computer algorithm are rank ordered beyond the
control of the user. Sensitivity and precision were calculated for the first 100, 200 and 300
hits on this search engine so as to appraise the value of extending the length and rigour of
the search process to include each of these numbers of hits.
Overlap and unique relevant items identified
If the relevant items retrieved on a particular database were all available on other
databases, then that database might be regarded as redundant for the purposes of a
search on this topic. The issue of overlap between databases has been studied in relation
to various academic disciplines (Gluck, 1990; McDonald et al., 1999), more recently using
fuzzy set theory (Hood and Wilson, 2003). For the present purpose the main point of
interest is whether a database is worth using because it uniquely identifies relevant articles
not retrieved by any other database (Topfer et al., 1999). The number of unique relevant
page 7
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
items identified was calculated for each database as a measure of how many items would
be missed if that database were not used.
Results
The total number of relevant articles retrieved across all the databases was 69 (see
Table 1). This figure was used to calculate the sensitivity of the searches. Column two of
the Table gives the number of relevant items (of this 69) retrieved on each database.
In relation to Google Scholar, sensitivity and precision were calculated in relation to
the first 100, 200 and 300 hits retrieved and these are labelled GS100, GS200 and GS300
respectively in the Table. It needs to be acknowledged that databases are designed for
different purposes and the data presented here must be viewed with this in mind. In
particular the Cochrane Library is designed only to contain systematic reviews of the
highest quality experimental studies of the effectiveness of interventions.
Although this study did not explicitly compare database facilities, it was apparent
that there were few filters available for identifying research on the databases. Whilst
Medline had admirable filters for identifying various experimental designs it did not
facilitate a search for research using qualitative designs or ‘research’ in general. A search
filter, where available, for ‘research’ was perhaps the single most useful filter.
Table 1: Results of searching the databases
Database
Unique
Hits
1
Inter-rater
Agreement
84%
Sensitivity
Precision
ASSIA
Total Hits Relevant
Retrieved Hits
493
32
46%
7%
CINAHL
144
14
4
76%
20%
10%
Cochrane
Library
0
0
0
100%
0
0%
GS300
300
11
7
73%
16%
4%
GS200*
200
11
7
73%
16%
6%
GS100*
100
10
7
75%
14%
10%
0
0
100%
0%
0%
Index
Theses
to 30
Medline
563
8
4
100%
12%
2%
PsycINFO
87
9
2
88%
13%
10%
Social Care 14
Online
1
0
100%
1%
7%
page 8
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
Social
Services
Abstracts
SSCI
685
29
0
84%
42%
4%
834
31
13
91%
45%
4%
ZETOC
5
0
0
100%
0
0
Note 1: Total number of relevant hits (removing duplicates) 69.
*Note 2: The three searches on Google Scholar (GS100, GS200 and GS300) each have
their own calculation of precision, but only GS300 was used to compute sensitivity and other
comparisons between databases.
Sensitivity
ASSIA, SSA and SSCI had the highest sensitivity retrieving about 30 relevant hits
each, and each locating about 40-45% of the relevant items. CINAHL was next most
sensitive retrieving about half as many hits (20% sensitivity). Each of the three Google
Scholar searches were next most sensitive, each retrieving about a third of the number of
hits as the most sensitive databases, followed closely by PsycINFO and Medline. SCO
produced just one relevant hit. The Cochrane Library, Index to Theses and ZETOC
produced no relevant hits. Studies reported in theses that are included here as relevant
hits were found through searching other databases, not Index to Theses. Extending the
Google Scholar search from 100 to 200 items gave only one additional item on top of the
10 already retrieved, and extending it from 200 to 300 items gave no additional sensitivity.
Precision
CINAHL and PsycINFO had the highest precision, equalled by the search of the first
100 Google Scholar items. Generally precision was low.
Unique relevant items identified
SSCI produced the highest number (13) of unique relevant items not retrieved by
any other database, almost double the seven retrieved by the Google Scholar searches.
This in turn was nearly double the next nearest databases in terms of unique items
retrieved: Medline and CINAHL at four each. A searcher could omit using SSA and SCO
without missing any relevant hits. A searcher would miss only one by omitting ASSIA or
two by omitting PsycINFO if all other searches were conducted in each case.
Methodology of Retrieved Papers
Survey research methods were the most frequently used (50%), followed by
longitudinal studies (13%), qualitative studies (12%), experimental studies (6%) and action
research (3%). Ten percent of studies used mixed methods and six percent were literature
reviews.
Discussion
This study of database performance embodied a wide range of databases and a
commercial web search engine. The relevance of social work to the intended scope of the
databases varied and the databases have different journal coverage. What this study has
achieved is to view each of these databases in terms of its practical usefulness for
retrieving relevant material on the topic used for this case study. It cannot be assumed that
page 9
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
a database designed for social work will be more useful than one designed for a broader
range of social sciences or for other health and social care professions. This has been
tested through this study.
Due to resource constraints it was not possible to include Social Work Abstracts or
ProQuest Digital Dissertations as these were not available in our work environment. These
purchasing decisions may reflect perceived utility in relation to cost (Flatley et al., 2007;
Holden et al., 2008 and 2009; Shek, 2008; Taylor, 2009) although we were not privy to
such decision making. Searching EMBASE was not regarded as cost effective (Topfer et
al., 1999). The searches produced a number of relevant unpublished dissertations.
Dissertation results are included in the ASSIA and Social Services Abstracts hits which are
both ProQuest products, and it may be that subscription to these services provided
unofficial access to Dissertation Abstracts International. PubMed was not included as our
advice was that this basically would have given us Medline in a different format, but with
substantially the same content for our purposes.
It must be recognised that the databases have different target audiences, and
purposes. Whilst performance in this case study might give an indication of utility in
relation to some social work topics, performance of a database in relation to its intended
audience and purpose might be considerably different. In particular the Cochrane Library
is designed only for systematic reviews of the effectiveness of health and social care
interventions. Social Care Online and Social Services Abstracts are the only two of those
used designed specifically for social work and social care. As social science databases,
Applied Social Sciences Index and Abstracts and Social Sciences Citation Index might be
regarded as designed for social work as an academic discipline. Medline, PsycINFO and
Cumulative Index to Nursing and Allied Health are targeted towards allied professions, and
it is an asset for the profession that these abstract so much relevant social work material.
Google Scholar and ZETOC are more generic data search engines, and search
mechanisms such as these are perhaps likely to improve in quality in future.
Sensitivity was greatest on ASSIA, SSA and SSCI. These databases were the most
fruitful in retrieving articles relevant to this aspect of practice. British researchers may be
disappointed at the low number of relevant items retrieved on SCO, a service provided by
the Social Care Institute for Excellence which is a government body charged with
disseminating knowledge and developing guidance to promote high standards of practice.
PsycINFO and CINAHL had the highest precision on this topic out of the conventional
databases, but generally precision was disappointingly low. Index to Theses did not prove
worth using, even though we planned to include grey literature.
A complex search across multiple databases was required in order to achieve
comprehensive identification of relevant research. SSCI, Google Scholar, Medline and
CINAHL produced the highest number of unique hits (those not retrieved by any other
database). Although SSCI also had high sensitivity, these other databases did not,
reinforcing the importance of using a wide range of databases for comprehensive
searching on social work topics (Mendelsohn, 1984; Kemp and Bustman, 1997; Stevinson
and Lawlor, 2004). We did not use any more complex measure of overlap between
databases (Egghe and Goovaerts, 2007; Gluck, 1990; Hood and Wilson, 2001) although
this should be considered for future studies.
The lack of a large, international database with anything approaching a high level of
sensitivity is a major challenge for the profession. Efficient retrieval of a reasonably
comprehensive range of research is an essential underpinning for any systematic review
of best evidence to inform practice and policy. The number of relevant items retrieved on
only one database (unique hits) confirms the need to use a number of databases for
comprehensive searching, confirming studies on other social work topics and in related
professional disciplines (Brettle and Long, 2001; Mcdonald et al., 1999; Stevinson and
page 10
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
Lawlor, 2005; Subirana et al., 2005; Taylor, et al., 2007). These types of searches across
multiple databases will require social work professionals to engage with librarians and
information scientists increasingly if we are to be able to avail of the ever-increasing
volume of published material.
One measure that would help to curb the complexity of searching required would be
if the profession could develop a more restricted and standardised vocabulary for key
concepts, particularly those used in research (Brown and Sönksen, 2000; Curran et al.,
2007). This will become increasingly urgent if the generic web search engines mentioned
above come to have a more prominent place in knowledge retrieval for the profession. At
the point in time of conducting the search, it is not possible for the searcher to maximise
both sensitivity and precision. This can be illustrated graphically using Receiver Operator
Characteristic (ROC) curves which are used in various fields to conceptualise the ability of
a system to discriminate between the “wanted items” and the “noise” from which the
signals must be differentiated (Allison et al., 1999; Centor, 1991; Taylor, in press).
It was a worthwhile exercise to run the three searches on Google Scholar so as to
appraise the merit of using this search engine for this purpose. The sensitivity was modest,
being about a third of that achieved by the three best databases, but slightly better than
PsycINFO and Medline. It was noteworthy that using the second and third hundred hits on
Google Scholar gave very little more return than the first hundred. Using the first 100 items
retrieved on Google Scholar gave good precision, making this a good search facility for
more limited, rather than comprehensive, searching (Wong et al., 2003; Haynes et al.,
2005). Given the capacity to search the web pages of all journals on the World Wide Web,
this suggests that the Google Scholar search algorithm, of which the user has limited
knowledge and control, has sacrificed sensitivity for precision. This finding might be
viewed in the context of the literature on rapid versus extensive searching (Royle and
Milne, 2003). Google Scholar was commendable in retrieving the third highest number of
unique hits, highlighting its importance as a search tool on this topic. This finding of the
reasonable sensitivity and high precision of Google Scholar confirms studies in other topic
areas (Walters, 2009). It may be that other search engines and meta-search engines (such
as AltaVista, Ask.com, Dogpile, Metacrawler and Yahoo) should be included in future
studies of research retrieval (Vaughan, 2004; Robinson and Wusteman, 2007; Kumar and
Pavithra, 2010).
The range of research methods used in the papers retrieved confirms the validity of
this approach to constructing searches to identify social work research and the validity of
appraising the quality of databases by the methods used in this case study. The
methodology used here in effect appraises both the article coverage of the database, the
quality of indexing (where this is provided) and the search facilities available. For practical
purposes all these dimensions are important for effective and efficient retrieval. However it
must be borne in mind that the complexity of the searches employed here are likely to
require the assistance of a librarian or information scientist. The busy professional is
unlikely to have the time or skills to create such sophisticated searches.
The findings of this case study can be compared with two previous database
searching case studies (Taylor et al., 2003 and Taylor, et al., 2007) as detailed in Table 2,
both of which were on the topic of professional decision making on the admission of older
people to institutional care (out of home placement).
Table 2; Sensitivity and precision compared with other studies of database searching
Database
Taylor et al., 2003
Taylor et al., 2007
This study
Sensitivity Precision Sensitivity Precision Sensitivity Precision
page 11
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
%
%
%
%
%
%
AgeInfo
n/a
n/a
13
76
n/a
n/a
ASSIA
n/a
n/a
n/a
n/a
46
7
CINAHL
19
84
30
32
20
10
Cochrane
n/a
n/a
n/a
n/a
0
0
GS 300
n/a
n/a
n/a
n/a
16
4
Index
Thesis
n/a
n/a
n/a
n/a
0
0
Medline
41
48
52
29
12
2
PsycINFO
n/a
n/a
12
51
13
10
SC
Online*
1
94
4
30
2
7
SSA
n/a
n/a
9
41
42
5
SSCI
55
54
46
20
45
4
Zetoc
n/a
n/a
n/a
n/a
0
0
* Social Care Online was previously named CareData.
As databases are designed for a particular readership, one might expect sensitivity
to vary across databases according to the topic. The highest sensitivity on this topic of
resilience amongst child protection social workers was achieved by ASSIA, SSA and
SSCI; in the case studies cited above on decision making about the long term care of older
people Medline and SSCI had highest sensitivity (ASSIA was not included in those
studies). The tentative conclusion might be that SSA is more suited to an organisational
topic related to social work services, whilst Medline is more suited to a clinical topic,
particularly where there is an obvious interface with health services. SSCI proved itself on
both types of question, although ASSIA has not yet been tested on a more clinical topic
using this method. CINAHL achieved reasonable sensitivity on both types of topic.
In general this study did not achieve the levels of precision achieved on the earlier
cited studies by Taylor et al. This suggests that the concepts in this study were harder to
define or that the terminology is less clearly defined in the literature. Otherwise no
consistent picture emerged regarding precision of searching.
Google Scholar is freely available through the World Wide Web and therefore
accessible to all practitioners and researchers even if they have limited access to other
electronic databases. The findings of this study shed some positive light on the utility of
using this resource.
Conclusion
The best evidence to inform social work practice is not a luxury; it should be a
foundation for professional and management decisions. On this particular topic, we need
to know how the workforce is best managed and supported to do a critical job in child
protection social work. Systematic reviews of best evidence need to become a much more
regular part of the reading of social workers, their managers and policy makers.
page 12
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
This study illustrated the need to search a number of databases for comprehensive
searching on this topic, echoing the results of earlier studies. The availability on a
database of a search filter for ‘empirical research’ is an asset when searching for research
relevant to social work. The gradual development of established strings of search terms to
capture common concepts (such as “qualitative research”) helps to improve the quality of
searching. The performance of the publicly-available web search engine Google Scholar
suggests that such web search engines and meta-search engines should be included in
future studies of professional information retrieval. It seems likely that web search engines
will become increasingly important for this task in the future.
Finding a few relevant articles for an undergraduate essay is straightforward
provided the database has reasonable sensitivity and the searcher modest patience. The
challenge, however, is in any context where a reasonably comprehensive search is
required to develop a sound knowledge base such as PhD theses, preparation for
research projects, service improvement plans, policy documents and development of
professional guidelines and assessment tools (Taylor, advance access).
To underpin systematic reviews of best evidence the social work profession needs
to develop knowledge and skills in identification of relevant research. This does not relate
only to systematic reviews addressing questions of effectiveness, such as those published
by the Cochrane Collaboration and the Campbell Collaboration, crucial as these are. The
profession also needs the capability to know the best evidence for questions of
prevalence, change, perceptions and processes which might be addressed by research
designs other than experimental ones. Database searching is generally more complex in
social work than in health care disciplines as the terminology is not so clearly defined; is
more likely to change with time; and is less likely to be agreed internationally due to
variations in language use across states and cultures. This increases the need for
knowledge and skills in database searching to support the profession. The ability to
identify relevant research must become a priority if we wish to ensure that the best
available knowledge is accessible to inform practitioners, researchers, educators and
policy makers, for the ultimate benefit of the clients and families whom we serve.
Acknowledgements
We would like to acknowledge the PhD Studentship received by Paula McFadden from the
Department for Employment and Learning, Northern Ireland which supported this project
and the subsequent review and empirical work.
References
Adams. C.E., Power, K.F. and Lefebvre, C. (1994). An investigation of the adequacy of
MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental
health care. Psychological Medicine, 24(3), 741-748.
Allison, J.J., Kiefe, C.I., Weissman, N.W., Carter, J. and Centor, R.M. (1999). The art and
science of searching Medline to answer clinical questions. International Journal of
Technology Assessment in Health Care, 15(2), 281-296.
Beall, J. (2007). Search fatigue: Finding a cure for the database blues. American Libraries.
38, 46-50.
Bender, K., Tripodi, S. J., Sarteschi, C. and Vaughn, M. G. (2011). A meta-analysis of
interventions to reduce adolescent cannabis use. Research on Social Work Practice,
21(2), 153-164.
Boynton, J., Glanville, J., McDaid, D. and Lefebvre, C. (1998). Identifying systematic
reviews in Medline: Developing an objective approach to search strategy design.
Journal of Information Science, 24(3), 137-157.
page 13
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
Brettle, A.J. and Long, A.F. (2001). Comparison of bibliographic databases for information
on the rehabilitation of people with severe mental illness. Bulletin of the Medical
Library Association, 89(4), 353-362.
Brown, P.J.B. and Sönksen, P. (2000). Evaluation of the quality of information retrieval of
clinical findings from a computerized patient database using a semantic terminological
model. Journal of the American Medical Informatics Association, 7(4), 392-403.
Burdett, S., Stewart, L.A. and Tierney, J.F. (2003). Publication bias and meta-analyses (a
practical example). International Journal of Technology Assessment in Health Care,
19(1), 129-134.
Centor, R.M. (1991). Signal detectability: The use of ROC curves and their analysis.
Medical Decision Making, 11(2), 102-106.
Clapton, J. (2010). Bibliographic databases for social care searching, London: Social Care
Institute for Excellence.
Cook, D.J., Mulrow, C.D. and Haynes, R.B. (1997). Systematic reviews: Synthesis of best
evidence for clinical decisions. Annals of Internal Medicine, 126, 376-380.
Coren, E. and Fisher, M. (2006). Research resource 1: The conduct of systematic
research reviews for SCIE knowledge reviews. London: Social Care Institute of
Excellence.
Coren, E., Thomae, M. and Hutchfield, J. (2011). Parenting training for intellectually
disabled parents: A Cochrane systematic review. Research on Social Work Practice,
21(4), 432-441.
Crumley, E.T., Wiebe, N., Cramer, K., Klassen, T.P. and Hartling, L. (2005). Which
resources should be used to identify RCT/CCTs for systematic reviews?: A systematic
review. BMC Medical Research Methodology, 5 – 24.
Curran, C., Burchardt, T., Knapp, M., McDaid, D. and Li, B. (2007). Challenges in
multidisciplinary systematic reviewing: A study on social exclusion and mental health
policy. Social Policy and Administration, 41(3), 289-312.
Darragh, E. and Taylor, B.J. (2008). ‘Research and Reflective Practice’ in Higham P (ed)
Post Qualifying Social Work Practice, London: Sage.
Dempster, M. (2003). ‘Systematic review’. In R. Miller, and J. Brewer (Eds.), The A to Z of
social research. London: Sage.
Dickerson, K., Scherer, R. and Lefebvre, C. (1994). Identifying relevant studies for
systematic reviews. British Medical Journal, 309, 1286-1291.
Egger, M., Juni, P., Bartlett, C., Holenstein, F. and Sterne, J. (2003). How important are
comprehensive literature searches and the assessment of trial quality in systematic
reviews?: Empirical study. Health Technology Assessment, 7(1), 1-76.
Egghe, L. and Goovaerts, M. (2007). A note on measuring overlap. Journal of Information
Science, 33(2), 189–195.
Ellett, A.J., Ellett, C.D., Ellis, J. and Lerner, B. (2009). A research-based child welfare
employee selection protocol: Strengthening retention of the workforce. Child Welfare,
88(5), 49-68.
Fisher, M., Qureshi, H., Hardyman, W. and Homewood, J. (2006). Using qualitative
research in systematic reviews: Older people’s views of hospital discharge, London:
Social Care Institute for Excellence.
Flatley, R.K., Lilla, R. and Widner, J. (2007). Choosing a database for social work: A
comparison of Social Work Abstracts and Social Services Abstracts. Journal of
Academic Librarianship, 33(1), 47-55.
Flemming, K. and Briggs, M. (2006). Electronic searching to locate qualitative research:
Evaluation of three strategies. Journal of Advanced Nursing: Research Methodology,
57(1), 95-100.
page 14
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
Gluck, M. (1990). A review of journal coverage overlap with an extension to the definition
of overlap. Journal of the American Society for Information Science, 41(1), 43–60.
Golder, S., Mason, A. and Spilsbury, K. (2008). Systematic searches for the effectiveness
of respite care. Journal of the Medical Library Association, 96(2), 147–152.
Hay, P.J., Adams, C.E. and Lefebvre, C. (1996). The efficiency of searches for
randomized controlled trials in the International Journal of Eating Disorders: A
comparison of handsearching, EMBASE and PsycLIT. Health Libraries Review, 13,
91-96.
Haynes, B.R., McKibbon, A.K., Wilczynski, N.L., Walter, S.D. and Werre, S.R. (2005).
Optimal search strategies for retrieving scientifically strong studies of treatment from
Medline: analytical survey. British Medical Journal, 330: 1179.
Haynes, B.R. and Wilczynski, N.L. (2004). Optimal search strategies for retrieving
scientifically strong studies of diagnosis from Medline: Analytical survey. British
Medical Journal, 328, 1040-1042
Holden, G., Barker, K., Covert-Vail, L., Rosenberg, G. and A. Cohen, S. (2008). Does
Social Work Abstracts work? Research on Social Work Practice, 18(5), 487-499.
Holden, G., Barker, K., Covert-Vail, L., Rosenberg, G. and A. Cohen, S. (2009). Social
Work Abstracts fails again: A replication and extension. Research on Social Work
Practice, 19(6), 715-721.
Hood, W.W. and Wilson, C.S. (2001). The scatter of documents over databases in
different subject domains: How many databases are needed? Journal of the American
Society for Information Science and Technology, 52(14), 1242–1254.
Hood, W.W. and Wilson, C.S. (2003). Overlap in bibliographic databases. Journal of the
American Society for Information Science and Technology, 54(12), 1091–1103.
Hoogendam, A., de Vries Robbé, P.F., Stalenhoef, A.F.H. and Overbeke, A.J.P.M. (2009).
Evaluation of PubMed filters used for evidence-based searching: Validation using
relative recall. Journal of the Medical Library Association, 97(3), 186-193.
Jenkins, M. (2004). Evaluation of methodological search filters: A review. Health
Information and Libraries Journal, 21(3), 148-163.
Kemp, B.E. and Bustman, M.J. (1997) Social policy research: Comparison and analysis of
CD-ROM resources. Social Work Research, 21(2), 111-120.
Killick, C. and Taylor, B.J. (2009). Professional decision making on elder abuse:
Systematic narrative review. Journal of Elder Abuse and Neglect, 21(3), 211-238.
Kumar, B.T.S. and Pavithra, S.M. (2010). Evaluating the searching capabilities of search
engines and metasearch engines: a comparative study. Annals of Library and
Information Studies, 57(2), 87-97.
Layard, R., Clark, D., Knapp, M. and Mayraz, G. (2007). Cost-Benefit Analysis of
Psychological Therapy, Discussion Paper No 829. London: Centre for Economic
Performance, London School of Economics and Political Science .
Lee, E., Dobbins, M., DeCorby, K., McRae, L., Tirilis, D. and Husson, H. (2012). An
optimal search filter for retrieving systematic reviews and meta-analyses. Medical
Research Methodology, 12(1), 51.
Lindsey, D. and Kirk, S.A. (1992). The role of social work journals in the development of a
knowledge base for the profession. Social Service Review, 66(2), 295-310.
Matthews, E.J., Edwards, A.G.K., Barker, J., Bloor, M., Covey, J., Hood, K., Pill, R.,
Russell, I., Stott, N. and Wilkinson, C. (1999). Efficient literature searching in diffuse
topics: Lessons from a systematic review of research on communicating risk to
patients in primary care. Health Libraries Review, 16(2), 112-20.
McDonald, S., Taylor, L. and Adams, C. (1999). Searching the right database. A
comparison of four databases for psychiatry journals. Health Libraries Review, 16(3),
151–156.
page 15
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
Mendelsohn, H.N. (1984). Social work online. Database, 7(3), 36-49.
Mor Barak, M.E., Nissly, J.A. and Levin, A. (2001). Antecedents to retention and turnover
among child welfare, social work, and other human service employees: What can we
learn from past research?: A review and meta-analysis. Social Service Review, 75(4),
625–661.
Minozzi, S., Pistotti, V. and Forni, M. (2000). Searching for rehabilitation articles on
MEDLINE and EMBASE: An example with cross-over design. Archives of Physical
Medicine & Rehabilitation, 81(6), 720–722.
Moseley, A.M., Sherrington, C., Elkins, M.R., Herbert, R.D. and Maher, C.G. (2009).
Review: Indexing of randomised controlled trials of physiotherapy interventions: A
comparison of AMED, CENTRAL, CINAHL, EMBASE, Hooked on Evidence, PEDro,
PsycINFO and PubMed. Physiotherapy, 95(3), 151–156.
Papaioannou, D., Sutton, A., Carroll, C., Booth, A. and Wong, R. (2009). Literature
searching for social science systematic reviews: Consideration of a range of search
techniques. Health Information and Libraries Journal, 27(2), 114–122.
Robinson, M.L. and Wusteman, J. (2007). Putting Google Scholar to the test: A preliminary
study. Electronic Library and Information Systems, 41(1), 71-80.
Royle, P. and Milne, R. (2003). Literature searching for randomized controlled trials used
in cochrane review: Rapid versus exhaustive searches. International Journal of
Technology Assessment in Health Care, 19(4), 591-603.
Shek, D.T.L. (2008). Comprehensiveness of Social Work Abstracts as a database for
researchers and practitioners. Research on Social Work Practice, 18(5), 500-506.
Shlonsky, A., Baker, T.B. and Fuller-Thomson, E. (2011). Using methodological search
filters to facilitate evidence-based social work practice. Clinical Social Work Journal,
39(4), 390-399.
Stevinson, C. and Lawlor, D.A. (2004). ‘Searching multiple databases for systematic
reviews: added values or diminishing returns?’. Complementary Therapies in
Medicine, 12(4), 228-32.
Stokes, P., Foster, A. and Urquhart, C. (2009). Beyond relevance and recall: Testing new
user-centred measures of database performance. Health Information and Libraries
Journal, 26(3), 220–231.
Subirana, M., Solá, I., Garcia, J.M., Gich, I. and Urrútia, G. (2005). A nursing qualitative
systematic review required Medline and CINAHL for study identification. Journal of
Clinical Epidemiology, 58(1), 20-25.
Taylor BJ (2003) ‘Literature Searching’ in Miller R & Brewer J (eds) The A to Z of social
research, London: Sage.
Taylor, B.J. (2009). Invited commentary on papers by Holden et al. and Shek on the
quality of Social Work Abstracts. Research on Social Work Practice, 19(3), 366-369.
Taylor, B.J. (2010). Professional decision making in social work, Exeter: Learning Matters.
Taylor, B.J. (2012). ‘Intervention Research’ in Gray, M., Midgley, J. and Webb, S. (Eds.)
Social work handbook, New York: Sage.
Taylor, B.J. (advance access). Developing an integrated assessment tool for the health
and social care of older people. British Journal of Social Work, doi:
10.1093/bjsw/bcr133.
Taylor, B.J. (in press). Models for professional judgement in social work. European Journal
of Social Work,
Taylor, B.J. and Campbell, B. (2011). Quality, risk and governance: Social Workers’
perspectives. International Journal of Leadership in Public Services, 7(4), 256-272.
Taylor, B.J., Dempster, M. and Donnelly, M. (2003). Hidden gems: Systematically
searching electronic databases for research publications for social work and social
care. British Journal of Social Work, 33(4), 423-439.
page 16
McFadden P, Taylor BJ, Campbell A and McQuilkin J (2012) Systematically identifying relevant research:
Case study on child protection social workers’ resilience. Research on Social Work Practice, 22(6), 626-636.
Taylor, B. J., Wylie, E., Dempster, M. and Donnelly, M. (2006). An evaluation of the
AgeInfo database. London: Social Care Institute of Excellence.
Taylor, B. J., Dempster, M. and Donnelly, M. (2007). Grading gems: Appraising the quality
of research for social work and social care. British Journal of Social Work, 37(2), 335354.
Taylor, B., Wylie, E., Dempster, M. and Donnelly, M. (2007). Systematically retrieving
research: A case study evaluating seven databases. Research on Social Work
Practice, 17(6), 697-706.
Topfer, L.A., Parada, A., Menon, D., Noorani, H., Perras, C. and Serra-Prat, M. (1999).
Comparison of literature searches on quality and costs for health technology
assessment using the MEDLINE and EMBASE databases. International Journal of
Technology Assessment in Health Care, 15(2), 297–303.
Tripodi, S. J., Bledsoe, S. E., Kim, J. S. and Bender, K. (2011). Effects of correctionalbased programs for female inmates: A systematic review. Research on Social Work
Practice, 21(1), 15-31.
Vaughan, L. (2004). New measurements for search engine evaluation proposed and
tested. Information Processing and Management, 40(4), 677-691.
Walters, W. H. (2009). Google Scholar search performance: Comparative recall and
precision. Libraries and the Academy, 9(1), 5-24.
Watson, R.J.D. and Richardson, P.H. (1999). Accessing the literature on outcome studies
in group psychotherapy: The sensitivity and precision of Medline and PsycINFO
bibliographic database searching. British Journal of Medical Psychology, 72(1), 127134.
White, V.J., Glanville, J.M., Lefebvre, C. and Sheldon, T.A. (2001). A statistical approach
to designing search filters to find systematic reviews: Objectivity enhances accuracy.
Journal of Information Science, 27(6), 357-370.
Wong, S.S.L., Wilczynski, N.L., Haynes, R.B. and Ramkissoonsingh, R. (2003).
Developing optimal search strategies for detecting sound clinical prediction studies in
Medline. American Medical Informatics Association: Annual Symposium Proceedings,
2003: 728-732.
page 17