The NIHR Research Design Service
for Yorkshire & the Humber
How to Search and
Critically Evaluate
Research Literature
Author
Michael Hewitt
Revised by Christine Keen
This Resource Pack is one of a series produced by The NIHR RDS for the East
Midlands / The NIHR RDS for Yorkshire and the Humber. This series has been funded
by The NIHR RDS EM / YH.
This Resource Pack may be freely photocopied and distributed for the benefit of
researchers. However it is the copyright of The NIHR RDS EM / YH and the authors
and as such, no part of the content may be altered without the prior permission in
writing, of the Copyright owner.
Reference as:
Hewitt M., How to Search and Critically Evaluate Research Literature. The NIHR
RDS for the East Midlands / Yorkshire & the Humber, 2007
Michael Hewitt
Evaluation, Audit and Research Manager, Sherwood Forest Hospitals NHS Trust
Last updated: 2009
The NIHR RDS for the
East Midlands
Division of Primary Care,
14th Floor, Tower building
University of Nottingham
University Park
Nottingham
NG7 2RD
Tel: 0115 823 0500
www.rds-eastmidlands.nihr.ac.uk
The NIHR RDS for
Yorkshire & the Humber
ScHARR
The University of Sheffield
Regent Court
30 Regent Street
Sheffield
S1 4DA
Tel: 0114 222 0828
www.rds-yh.nihr.ac.uk
Leicester:
enquiries-LNR@rds-eastmidlands.org.uk
Nottingham: enquiries-NDL@rds-eastmidlands.org.uk
Sheffield:
Leeds:
York:
rds-yh@sheffield.ac.uk
rds-yh@leeds.ac.uk
rds-yh@york.ac.uk
© Copyright of The NIHR RDS EM / YH
(2009)
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
1
Table of Contents
Page
1.
Introduction……………………………………………………… 4
2.
The Research Process.........................................................
6
3.
Research literature……........................................................
11
4.
Using databases to locate published research…………...
14
5.
Planning and conducting a literature search……………...
18
6.
Examining the results and writing a review……………….
23
7.
Answers to exercises………………………………………….
34
8.
References………………………………………………………
37
9.
Further reading and resources……....................................
37
10.
Glossary………………………………………………………….
38
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
1
1. Introduction
The amount of health care research undertaken has grown enormously over the
past two decades. More research is being done by more people in almost every
professional group. This is due to a number of factors. They include:
• more health care staff undertaking research as part of academic studies
• increased funding for research
• pressure on academics in university health care faculties to undertake original
research that can be written for publication.
At the same time, health care professionals are being encouraged to base their clinical
practice on research based knowledge. Every year, hundreds of professional journals
are published containing thousands of reports of research studies. The health care
professional searching the literature for information on, for example, factors affecting
patient uptake of screening services, is likely to reveal a bewildering number of research
reports from a growing array of sources. The reports will vary in terms of quality,
comprehensibility and relevance to practice. Surprising as it may seem, it is an
unfortunate fact that not all published research is good quality. Many more studies,
while being basically well researched studies, will have limitations due to aspects of
research design. The technique of the literature review is to try to make all of the above
more manageable.
The process of carrying out a literature review can be described by two main activities.
These are searching and then critically evaluating research literature. In addition, the
process and findings should be written-up as a complete record. These activities are
outlined below and covered in later sections in more detail.
Searching the literature is a research method in its own right. The method uses specific
technical terms and there are guidelines to follow that are designed to seek out the
literature that is relevant to the defined area of study. The literature identified from the
search should then be critically evaluated. The purpose of critical evaluation is to
enable the practitioner to read research studies objectively: to identify the good points
and bad points, the strengths and weaknesses, the usefulness of a report and the
limitations. Additionally, critical reading of research reports increases the practitioner's
understanding of the research process. Lastly, practitioners undertaking research of
their own will benefit from developing their critical evaluation skills. It will assist them to
appraise their own research projects and enhance their skills at all stages of the
research process from project design to writing the final report.
This resource pack provides guidelines on how to both search for and then critically
appraise research reports. It is intended for use by health care professionals with little
or no previous research knowledge. Readers will be guided through the processes and
provided with information on the factors that can affect the quality of the literature search
and the research articles found. This will be supported by examples from health services
research. Exercises will be used to reinforce the reader's understanding of the text.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
2
LEARNING OBJECTIVES
The aim of this pack is to provide an introduction to searching and evaluating research
literature. After working through the pack you will be able to:
•
Appreciate the role of the literature review in the research process
•
Conduct a literature search
•
Follow the sequence of a written research report.
•
Systematically consider each section of a research report identifying strengths
and weaknesses that may be present.
•
Critically appraise the overall value of a research report and consider whether or
not to apply future research findings to practice.
•
Write a brief and well-structured literature review.
This pack defines some key terminology and presents an overview of the process of
literature searching and critical evaluation.
1.2 Summary
This section has presented an overview of searching and evaluating the literature. The
following sections look in more detail at the process. Sections 2 and 3 examine the
publication of health care research. They describe in more detail what is meant by the
literature and how a piece of research becomes a part of this body of knowledge.
Section 4 describes the methods that have been devised for selecting out particular
subsets of the literature, and section 5 includes a step-by-step approach to planning and
conducting a literature search. Section 6 provides guidance on how to process the
results of your literature search, critically evaluate the research articles found and
presents brief guidelines on how to write a literature review. To assist the learning
process practical exercises are provided at the key stages of the pack. These take you
through the essential steps of conducting your own review.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
3
2. The Research Process
Thousands of books and articles have been written about research. It is an enormous
topic and no one knows everything there is to know about it. But in order to search and
read research literature critically, it is essential that the reader has a basic
understanding.
Research is conducted in a systematic way. The process of research is made up of a
number of stages, which the researcher must proceed through, for the research project
to be completed satisfactorily. The main stages are presented in Figure 2.1 and are
described in brief in this section. If you have no previous awareness of research at all
you may find it useful to read The NIHR RDS EM / YH Resource Pack An Introduction to
the Research Process, which provides more information about each stage in the
process.
Identify research
problem
Literature review
Methodology
Plan
research
Access and ethical
considerations
Figure 2.1 The Research Process
Carry out
research
Pilot study
Data collection
Data analysis
Conclusions
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
4
When deciding to carry out a research study the researcher starts out with an idea for
which a plan of investigation is developed. In developing the plan the researcher takes
into consideration things such as:
•
•
•
•
•
•
what am I trying to find out?
what sort of information do I need?
what is the best way to collect the information?
where can I get the information?
how many people will I need to ask?
how will I analyse and make sense of the information I collect?
Once the information has been collected and analysed the researcher has a
responsibility to disseminate a report of the project. This includes:
•
•
•
•
•
explaining how the research was conducted
reporting the results
highlighting the limitations of the research
drawing out main conclusions
identifying possible recommendations for practice and for further research.
The research process is the sequence of events that the researcher goes through in
developing and disseminating the research project. Different textbooks list the
sequence of events in slightly varying ways but they are essentially similar to the list
shown in Figure 2.1. The remainder of this section provides a concise summary of each
stage.
2.1 Identification of the Research Problem
Every research project starts with an idea; something that the researcher is interested in
knowing more about or is worried about; something that is perceived as a problem or as
a knowledge gap that needs to be filled. At the outset, the research idea is often vague
or too broad to be covered in one research project. It needs to be refined to make it
manageable and researchable. The researcher needs to think the idea through and the
thinking process is informed by the next stage.
2.2 Review of the Literature
Before embarking on a research project the researcher should search and review
existing literature. The process of searching and evaluating research literature is
detailed in subsequent sections of this resource pack. However, it is useful at this stage
to provide descriptions of some key concepts.
What is a literature review?
A literature review is a self-contained piece of written work that gives a concise
summary of previous findings in an area of the research literature. It reflects an author’s
knowledge and interpretation of the area of interest. It has a reference section that lists
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
5
the individual pieces of work referred to in the review. Like all pieces of written research
output it should include a description of the methods used to create the work.
Literature reviews vary considerably in their depth and breadth, as well as style of
presentation, depending on the purpose intended by the author. This may range from a
superficial search of the literature to give a researcher an insight to an area of potential
research through to a scientifically rigorous ‘systematic review’.
Researchers committed to writing a review, however, should be encouraged to go
beyond superficial searches and simply listing research works; they will inevitably get a
biased or incomplete view of the research area under investigation. Instead, they
should develop skills that will enable them to systematically search for literature and
critically review the research uncovered by the search. This pack aims to help health
care professionals achieve these skills.
Why do a literature review?
A preliminary review of the literature will help in further identifying and clarifying your
research problem. A little further down the line it may provide the theoretical input to
your research idea and help in the formulation of the research question. More
specifically, a literature review will:
1. provide an up-to-date picture of the research area of interest and show which areas
have been investigated and the results obtained
2. identify methods of investigation that could be used in further research
3. give indications of problems that might be encountered and possible solutions
4. reveal common findings among studies
5. reveal inconsistencies between studies
6. identify factors not previously considered
7. provide suggestions for further research.
In summary, completion of the literature review enables the researcher to revisit the
original research idea and define the exact focus of the research problem.
Published literature reviews
Examples of published literature reviews include: Chalmers and Pearson (2005) on oral
hygiene care for residents with dementia; Richardson et al. (2005) on indoor
environment and asthma; and Lui et al. (2005) on effective problem-solving
interventions for caregivers of patients with stroke.
2.3 Research Methodology
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
6
Methodology
Methodology refers to matters regarding the structure and design of the research study.
It deals with such issues as: the type of information required; the research design; the
method of collecting data; and the source of information, which is known as the sample.
Type of information
This depends on the original research idea. If the researcher wants to collect
measurable information about a topic this is referred to as quantitative research. It looks
at how big a problem is, how many people are affected by it, how often something
occurs, whether one thing is more or less important than another. If the researcher is
trying to understand something in more detail or to describe a situation so that people
can understand it better, this is usually qualitative research. In qualitative research,
attention is focused on answering questions such as why? in what way? what are the
implications? Rather than, how many? how often? how much? as occurs in quantitative
research.
The research design
This refers to the type of research. There are many different types of research design.
Research can be carried out using experiments, surveys, case studies, action research,
ethnography, grounded theory, and phenomenology. The glossary in Section 2
contains a concise definition of each of these research designs.
The method of collecting data
The most commonly used methods of collecting information are interviews,
questionnaires and observation. Interviews are usually conducted on a one-to-one
basis but some studies may use group interviews or focus groups. Interviews can be
highly structured, semi-structured or unstructured. The degree of structure affects the
flexibility of the interview. Questionnaires comprise a written set of questions, which are
answered by all respondents in a study. Observation is a technique for collecting data
through visual observation of events. It requires the nature of the data to be observable.
The sample
In any research the researcher has to identify the population under study. If a School
Nurse wants to carry out research about school children she needs to decide if this
means any school child or is she interested in a particular section of the school
population. Furthermore, if she eventually decided that the research was actually about,
for example, "poor attendees" a criterion standard for poor attendance should be
identified. If the target population is large it may not be feasible to include everyone in
the research so a sample has to be selected. There are different ways of sampling such
as random sampling, convenience sampling, purposive and cluster sampling. Brief
definitions of these are included in the glossary at the back of the pack. There are
reasons to use different approaches to sampling and each approach has strengths and
weaknesses.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
7
2.4 Access and Ethical Considerations
In trying to access the sample the researcher has to consider how this can practically be
achieved and whether any permission must be sought. Access to patients normally
requires permission from the responsible medical officer or the NHS organisation.
Access to staff requires permission from their manager. When seeking permission the
researcher can usually secure practical help and advice in accessing subjects.
The researcher must also consider any ethical implications of the research. At the very
least this involves issues of confidentiality and anonymity but there may be other factors
to consider which unless resolved could potentially have negative implications for the
subjects directly or indirectly. The researcher should seek formal permission to
undertake the study from the local research ethics committee.
2.5 Pilot Study
Once the researcher is ready to undertake the study he should carry out a small pilot
study to check that the methodology has been correctly thought through. The pilot is the
study in miniature and gives the researcher an opportunity to identify any problems and
to modify the research method before embarking on the main study.
2.6 Data Collection
At last the researcher reaches the stage of conducting the interviews, sending out the
questionnaires or recording observations.
2.7 Data Analysis
The researcher reviews the data that has been collected and systematically analyses
the responses of subjects. Data are described, summarised and, if quantitative in
nature, statistically analysed in order to produce the results of the study.
2.8 Conclusions
The researcher reviews the results and identifies the main conclusions. Crucially it is
important to return to the original research question to see whether it has been
answered. Ways in which the results could be considered for application to practice are
recommended, as are areas where further research is needed. An important part of this
stage is recognition of the limitations of the study so that readers who may consider
acting on the results of the study can take them into consideration.
2.9 Summary
This section presents an overview of the research process and places the literature
review within this context. The next section looks at how the research literature is
developed.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
8
3. Research Literature
This section takes the reader through the notion of the ‘research literature’ in further
detail; it describes how research is published and how it is made accessible to
researchers and practitioners.
3.1 What is meant by the ‘Literature’?
In professional and academic disciplines the term ‘literature’ is used to describe all the
published work on a particular subject. Within this definition, no judgement is made
regarding the quality of any single piece of work.
The main body of the literature lies within academic and professional journals. It has
been estimated that over 20,000 journals published each year carry articles that are
relevant to the disciplines of the medical and health sciences. In addition to journal
articles, research is also published in books, reports, conference proceedings, theses
and dissertations. The different elements of the research literature are described below.
Journals
The result of the research process (Figure 2.1, Page 6) is a document that needs to be
made accessible to the relevant research community. This is necessary for two main
reasons. First, so that the findings are open to critical examination by others, and
second, that they are accessible to all who might benefit from them. The traditional
vehicle for publishing research findings is journals; traditionally these are magazinesized publications containing articles but increasingly they are being published in
electronic form for access via the world wide web.
Journals are published in issues at regular intervals usually weekly, monthly or
quarterly. Because of the regularity of publication they are also known as periodicals or
serials. This regularity means that each new issue contains articles that describe the
latest research findings; this is a distinct advantage over other publication media such
as books that take longer to produce and update.
The publication of an article in a journal involves a number of steps:
1. Writing the article: this requires the author or authors to present their research
findings in a broadly scientific style. The layout of the article may also need to
conform to a particular style laid down by the editorial board of the journal; this may
require the authors to include a summary (or abstract) of their work, or to keep within
a specified word limit for example.
2. Submission of the article to a journal: usually via the journal’s editor.
3. Refereeing: some journals require articles to be critically reviewed by experts in the
field prior to publication. This process is also known as ‘peer-review’. Referees may
suggest amendments to the original text before publication can proceed, or may
reject the article outright if they argue the work is fatally flawed in some way.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
9
4. Changes to the original text: if indicated by the referees.
5. Publication: if, or when, the editor accepts the article.
There are basically two main types of journal:
• Research journals
• Professional journals.
Research journals publish peer-reviewed articles. A few examples of the many
thousands of research journals available include the British Medical Journal, the Lancet
and the British Journal of General Practice. In contrast, professional journals publish
articles on professional issues, service developments, the use of research findings in
practice and some short research articles. They are primarily written for practising
health care professionals rather than researchers. Examples include the Health Visitor
Journal, the Journal of Community Nursing and Practice Nurse.
Reports
Research reports appear in many different shapes and sizes. In general, however, they
will give a more detailed account of a piece of research than that found in a journal
article. Reports of original research may arise from many different sources including
health authorities, professional organisations and pharmaceutical companies. The
publicity and distribution of some reports may be very limited making it difficult to know
of them or obtain copies.
Theses and dissertations
Theses and dissertations are very detailed and comprehensive accounts of research
work. They are usually submitted for a higher degree at a university. Like reports their
publicity and distribution may be very limited.
Conference proceedings
Conference proceedings comprise brief summaries of research work presented at
conferences. A more detailed and complete account of the work may appear at a later
date in a journal article, report or thesis. Researchers often use conferences to present
preliminary findings of their work.
Books
Textbooks generally provide comprehensive overviews of a particular subject. In doing
so they may refer to, sometimes extensively, the research literature found in journal
articles, reports, conference proceedings or theses. They are not usually used to
present new research findings. There are, however, a few exceptions to this and some
very important and influential research findings have been published in book format.
These are sometimes known as research monographs.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
10
3.2 Summary
Health care research is published in a variety of formats including journal articles,
books, reports, conference proceedings, theses and dissertations. The first stage of the
literature review is to locate all the research findings on a particular subject from the
literature, regardless of the publication format. The next section shows how this can be
achieved.
Exercise 1
The publication process
Locate one or two journals that present articles written by health care professionals or
researchers. Look for information on the publication of articles within each journal (this
may be presented as instructions for authors or may be found with the general
information about the journal or editors located on the front inside cover).
Use the information to answer the following questions:
•
Is there a limit to the number of words each article must not exceed?
•
Are the articles peer-reviewed?
•
Should each article include an abstract?
•
How should references be presented?
•
Can the article be submitted to the journal’s editor electronically?
Relevant literature
List any journals you know of that are likely to contain articles relevant to your
profession. Add to the list any relevant reports, theses, conference proceedings and
books that you know of.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
11
4. Using Databases to Locate Published Research
A key stage in the literature review is to search out all the research literature on a
particular subject. This may at first seem an impossible task given the huge volume of
research literature published worldwide. This problem, however, has long been
recognised and considerable effort has been made to simplify and speed up the
process. The results of this effort are available through on-line, web-based databases,
also known as bibliographic databases. These offer facilities for searching for published
research. In this respect they differ from other web-based search engines like Google
that offer access to huge amounts of information but do not necessarily find research
articles that have had some degree of quality control. This section describes the main
bibliographic databases used to locate published research and how to use them. The
databases give references to journal articles and conference papers, giving you the
bibliographic details of each published paper e.g. author(s), article title, journal title,
volume and year, and often include an abstract or summary of contents.
4.1 Bibliographic Databases
At the time of writing the NHS has an agreement with the suppliers of a number of
databases such that these resources are available to all NHS staff. Similar
arrangements are made for staff employed by universities. The databases are available
either through the service provider ‘Athens’ (www.athens.nhs.uk) or through the National
Library for Health (www.library.nhs.uk). A username name and a password are required
and your local NHS or university library should be able to provide these.
NHS staff can access the following databases:
•
Medline
•
Cinahl
•
Psycinfo
•
Embase
•
AMED (Allied and Complementary Medicine)
•
British Nursing Index
•
DH-Data
•
King’s Fund.
A brief description of these databases is provided in Table 4.1 below.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
12
Table 4.1
Key features of the Main Bibliographic Databases
Database
Name
MEDLINE
CINAHL
(Cumulative Index
to Nursing and
Allied Health
Literature)
PsycINFO
(Psychological
Abstracts)
EMBASE
AMED
(Allied and
Complementary
Medicine)
BNI
(British Nursing
Index)
DH-DATA Health
Administration and
Medical Toxicology
Subject coverage
Types of
sources
covered
Years
covered
Medline covers the whole field of medicine including dentistry, veterinary
medicine and medical psychology, clinical medicine, anatomy,
pharmacology, toxicology, genetics, microbiology, pathology, environmental
health, occupational medicine, psychology, biomedical technology, health
planning and administration, space life science, and many other related
subject areas.
Cinahl covers all aspects of nursing and allied health disciplines. Seventeen
allied health disciplines are covered: cardiopulmonary technology, dental
hygiene, emergency services, medical/laboratory technology, medical
assisting, athletic training, occupational therapy, optometry, physical therapy
and rehabilitation, the physician's assistant, radiological technology,
respiratory therapy, social service in health care, speech-language
pathology, nutrition and dietetics, audiology and surgical technology.
Journals from biomedicine, alternative therapy, health sciences,
librarianship, health promotion/education, and consumer health are also
scanned for relevant articles.
Psychology practice and research and psychological aspects of related
clinical disciplines including medicine, psychiatry, nursing and
pharmacology, drug and behavioural therapy, treatment of disease, drug
addiction, developmental psychology, and educational psychology, as well
as the psychological aspects of such areas as linguistics, social processes,
pharmacology, physiology, nursing, education, anthropology, business and
law.
EMBASE covers the whole world's biomedical literature whilst concentrating
in particular on European sources. The emphasis of the database is on the
pharmacological effects of drugs and chemicals. Over 40% of current data
is drug-related. Other subjects: human medicine and biological sciences
relevant to human medicine, health affairs (occupational and environmental
health, health economics, policy and management), drug and alcohol
dependence, psychiatry, forensic science, pollution control, biotechnology,
medical devices and alternative medicine.
Complementary or alternative medicine: acupuncture, herbalism,
reflexology, homeopathy, holistic treatment, iridology, hypnosis, traditional
Chinese medicine, moxibustion, chiropractic, occupational therapy,
meditation, osteopathy, physiotherapy, yoga, psychotherapy, rehabilitation,
healing, research, diet therapy, ayurvedic medicine, Alexander technique.
Journal articles
1950 -
Journal articles,
pamphlets, pamphlet
chapters, books,
book chapters,
masters theses &
dissertations.
1982 -
Journal articles, book
chapters,
dissertations
1806 -
Journal articles
1974 -
Journal articles,
newspapers and
books
1985 -
Nursing and midwifery plus medical, health management and allied health
fields relevant to UK nurses and midwives. Covers mostly UK nursing and
midwifery journals although a representative number of non-UK specialist
journals are also covered. Examples of subject areas covered: accident &
emergency nursing, breast cancer, evidence based practice, learning
disabilities, midwifery, nurse practitioner, orthopaedic nursing, perinatal &
neonatal mortality, psychiatric nursing, reflective practice, student nurses,
theatre nursing, wounds.
Core subjects: health service and hospital administration, with an emphasis
on the British National Health Service, including planning, design,
construction and maintenance of health service buildings; medical
equipment and supplies; public health, nursing and primary care;
occupational diseases; social policy; and social services for children,
families, people with disabilities and elderly people. Other topics: medical
toxicology and environmental health including chemicals in food and other
consumer products, and the environment; pesticides; industrial chemicals;
health consequences of smoking, radiation and noise; air and water
pollution; and radiation biology.
Journal articles
1994 -
Journal articles,
books, reports,
pamphlets,
administrative
circulars and other
official
Publications.
1983 -
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
13
Database
Name
King’s Fund
database
Subject coverage
The database reflects the King's Fund's focus on improving health and
health care, covering policy and management of health and social care
services in the UK rather than clinical issues and treatments. Core subjects
include NHS management, social care, health inequalities, urban health and
regeneration, race and health, partnership working, primary care, mental
health, public involvement, and workforce development in the NHS.
SOCIAL SCI
SEARCH
Social Sci Search is a multidisciplinary index to the international literature of
the social sciences providing complete bibliographic data, author abstracts
and dated references. It corresponds to the social sciences citation index.
Subjects covered include Anthropology, Archaeology, Economics, Hygiene
and Public Health and many other areas.
Types of
sources
covered
Journal articles from
English-language
journals, books,
reports, pamphlets,
web publications,
strategic planning
documents and
government circulars.
Official UK health
policy documents and
'grey' literature from
health and social care
organisations.
Journal articles, book
reviews, discussions,
editorial, meetings.
Years
covered
1979 -
1972-
The databases noted in Table 4.1 are referred to as indexing or abstracting databases.
This is because they provide the basic index information about an article such as
author(s), article title etc, or in the case of an abstracting database the abstract or
summary of each article is provided where available.
Other useful resources to be found at www.library.nhs.uk include:
•
BioMed Central
140+ open access, peer-reviewed ejournals covering the whole of biology and
medicine.
•
E-books
On-line access to the full text of over 400 eBooks including titles from World
Health Organisation and major publishers. The national collection focuses on
mental health.
•
Proquest
Search over 1500 full text journals online.
•
Pubmed
PubMed is a service of the U.S. National Library of Medicine that includes over
16 million citations from MEDLINE and other life science journals for biomedical
articles back to the 1950s.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
14
4.2 On-line Searching
On-line searching of databases is an interactive process. The searcher types in a
keyword on the keyboard, and the computer displays the result as the number of
references retrieved. The references themselves can be seen when the user issues a
‘display’ or ‘print’ command. Usually the commands are available via an on-screen link.
The OVID online interface links it search across a number of the bibliographic
databases noted in Table 4.1. An example of a basic search is shown below
Subject: The management of asthma in general practice
The following keywords were entered in turn and then combined:
• asthma
• general practice
The following is displayed on the screen:
Ovid MEDLINE (R)
2003 to February Week 1 2007
# Search History
Results
Display
1
asthma.mp.
16867
DISPLAY
2
general practice.mp.
3630
DISPLAY
3
1 and 2
118
DISPLAY
For each keyword entered the computer generates a result that indicates the number of
articles found. The final line requests that the first two keywords are combined in one
search. That is, find only those articles that appear under both subject headings. If the
result is still very large it may be necessary to select subheadings to the keywords or
add additional keywords to refine and narrow-down the search.
Each search set can be displayed which shows the reference to each article found
together with the abstract. References and abstracts can be selected and printed out,
or transferred electronically to other computer software such as databases and word
processing packages.
Searching these databases can be a daunting task for the beginner. There are a large
number of techniques available to optimise the search e.g. combining as shown above,
but these are not always obvious to the novice searcher. If you are unfamiliar with these
it is recommended that you work with a librarian who is trained in on-line searching.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
15
4.3 Summary
This section has introduced the main tools for finding research literature. These are
the bibliographic databases that are designed to facilitate effective and efficient
searching through the vast amounts of literature in order to find the articles that are
of interest to you. However, before sitting down in front of a computer it is advisable
to have a plan of the literature search you wish to conduct. Guidelines to help
construct and implement a search plan are presented in the next section.
5. Planning and Conducting a Literature Search
The objective of a literature search is to retrieve as much accurate information on a
given subject as is possible from suitable sources. It is clear from this and the preceding
sections that the novice reviewer is faced with a difficult task. The pool of available
research literature is huge and only a tiny fraction of it, the particular subject of interest,
needs to be located. Help is at hand of course in the form of bibliographic databases,
but which ones should you use?
This section presents some basic guidelines to follow when tackling a literature search.
Like all types of investigation, an effective search requires careful planning. A badly
organised search is likely to yield little relevant information and waste time.
5.1 General Points
A search takes time to complete, even for the experienced reviewer. A reasonable
continuous period of time should be set aside for task; an hour here or there is not
sufficient. It is important to be systematic and record each step of your search. While
there are no hard and fast rules for the period of the search it would be unwise to restrict
it to recent years e.g. publications from the last 2 years. Finally, decide on your subject
of interest and stick to it during the search. When searching on-line databases many
interesting looking titles will appear - don’t get side tracked.
5.2 Bibliographic Databases
The choice of bibliographic databases may well depend on the subject of interest and
potential coverage by the different databases on offer. The main databases cover much
of the mainstream medical and health literature, but at the same time do not cover all of
the journals in all disciplines.
5.3 Keywords
To increase the chances of retrieving relevant information from a search you need to
create a description of the subject of interest. This takes the form of a set of words or
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
16
phrases, which are known as ‘search terms’ or ‘keywords’. Each search term or
keyword identifies a part of the subject and provides a focus for the search.
The process of creating keywords involves three stages:
1. Identify the key concepts in your research area
2. Analyse the concepts; extend their scope to find broader terms; define them with
increasing precision to produce narrower terms; produce a list of synonyms;
produce a list of related terms.
3. Map the list of key words or terms to the subject headings of each index to be
used in the search.
5.4 Creating Key Terms: an example
The area of interest is health visiting and the prevention of accidents to children in the
home. The key concepts are health visitor, accident prevention, children, and home. To
analyse the concepts it can help to create ‘spider diagrams’ as shown below (Figure
5.1).
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
17
Broader terms
Community nursing
Community health services
Public health nursing
Broader term
Accidents
ACCIDENT
PREVENTION
HEALTH
VISITOR
Related terms
Health education
Preventive health services
Wounds and injuries
Related terms
Family support
Social support
Home care services
Broader term
Environment
Public health
Broader term
Child welfare
CHILD
ACCIDENTS,
HOME
Narrower terms
Pre-school
Infants
Narrower term
House calls
Figure 5.1. Key concepts and related terms
Start the diagram by writing down the key concepts on a blank piece of paper. Carry out
a brainstorming activity and make a note of related, broader and narrower terms as well
as synonyms for each one.
5.5 Recording the Results
The results of a search are references to articles. It is advisable to keep a copy of the
articles it has retrieved. Computerised searches will generate reference lists that can be
transferred electronically on to a computer and loaded into a database or word
processor. Alternatively they can be printed out on to paper or e-mailed.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
18
5.6 Satisfactory Searches
When conducting your search it is always a good idea to periodically examine the
references and decide if the search is ‘on target’. It is difficult, often impossible, to know
how many references to expect on a particular subject. However, if you already know of
some key references in the area and your search does not find these, you should
suspect a problem.
There are a number of problems that can occur at each stage of search process. For
example you may have:
1. defined your subject too narrowly and ended up with no references
2. defined your subject too broadly and ended up with thousands of references
3. used inappropriate keywords
4. chosen the wrong (or not the best) databases for you search.
Finding too little: be especially careful if you find nothing. Often this is not the case.
Proving that there is nothing is not easy. You should try several different approaches
before coming to this conclusion. It is easier to begin a search with a broad strategy
and narrow appropriately than to construct a narrow search and try to broaden it if you
find nothing.
Finding too much: this is a common problem particularly with computerised searching.
Try limiting the search by selecting relevant sub-headings for the search terms you are
using.
If you decide your search is unsatisfactory then you need to revisit keyword and index
selection and iterate through the process again.
5.7 Summary
This section has provided an insight into conducting a literature search. To maximise
the success of a search careful planning is required. Decisions need to be made for
instance concerning the bibliographic databases to be used and the time period to
search. One important step is the construction of keywords to be used in searching;
care here may save you time looking through irrelevant material later. Finally, it is
important to record the searches you perform and the results.
The output from this the literature search is a list of references to articles of the subject
of your interest, you may have saved the full text of some articles offered on-line. The
next section looks at how to analyse and review this literature.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
19
Exercise 2
1. Plan and conduct a literature search
a) Decide on an area of interest for investigation. Try to think of a specific
area/problem/feature/question of your work rather than a general area. For
example, ‘Are Practice Nurses effective in delivering health care?’ is too general;
whereas, ‘Do Practice Nurses give effective quit-smoking advice?’ is more specific
and a more appropriate starting place.
b) Collect together any articles you may already have on the area of interest. Use the
articles and your clinical experience in the area to make a list of key
terms/phrases/concepts.
c) Use the list of key terms and generate broader, narrower, and related terms and
synonyms (use spider diagrams as shown in Figure 5.1 if this helps).
d) Conduct the search, be systematic and print out or save a copy of the results and
search terms.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
20
6. Examining the Results and Writing a Review
Examining the results of a literature search can be performed at different levels of
complexity. This may range from a superficial scan of the references to decide on
relevance, to a detailed analysis of the quality of each article. The aim of this section is
to present an overview of these levels of analysis. A brief introduction on how to write
up the findings of a literature review is also provided.
6.1 Relevance
The first level of analysis is to scan the list of references, remove duplicates and assess
the relevance of the material your search produced. However, assessing relevance on
the strength of an abstract is not always possible and in some instances the full paper
will be required. In addition, the quality of the material can only be assessed by looking
at the original work. Collecting copies of original articles is, therefore, the next task and
a visit to the nearest medical library is one place where a search can begin.
Medical libraries will publish a list of journals they subscribe to and some of your
references may be photocopied from stock. However, libraries have finite budgets and
shelf space and will only hold a limited number of journals. It is not likely, therefore, that
all of your references will be available from a single library. This problem is resolved by
the Inter-library Loans (ILL) system whereby items from other libraries are lent or
photocopied on request. Two important points must be made with regard to the ILL
system. First, it takes time to process each request and provide the material; this delay
can be as long as four weeks. Second, the ILL system is not free and a charge may be
made for each item requested. Consult with your local library staff for details.
6.2 Quality
It is important to note that the contents of an article do not necessarily represent facts
just because they are in print. An article may instead represent the views or opinions of
an author that are not based on research evidence. Reading literature critically and
assessing quality can be a complex and involved activity. It is a skill that needs to be
learned, practised and developed. There are, however, a number of guidelines that can
help you get started with this process.
Peer review
The first section described the steps involved in the publication process. One important
stage in the process is peer-review where each article is critically reviewed prior to
publication. This acts as a filtering stage with the idea that only works of sufficient
quality are published.
To some extent, therefore, you can judge an article by the journal in which it appears.
Journals that use peer review will have an editorial committee and/or an editorial
advisory board. They will outline their arrangements for peer-review in each issue of the
journal. This can usually be found on the inside front or back cover of the journal.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
21
Reading articles
Before you consider a detailed analysis of each article it is worth previewing each one
by scanning the abstract, the introduction, headings and subheadings, tables and
figures, discussion and conclusions, and the reference list. This will provide you with an
initial impression of the article; whether there are any obvious omissions, lack of detail,
errors in presentation of figures. A reference list, for example, that includes articles from
a range of journals and years, with books and other formats included, will indicate that a
proper search of the literature has been made.
6.3 Critical Evaluation
Critical evaluation (also known as critical appraisal) of research articles involves asking
the question ‘Am I persuaded by this study’s results?’ Answering this question involves
asking a series of other questions:
•
•
•
•
•
Is the purpose of the study clear and well defined?
How was the study done? Are the methods clearly described and appropriate?
Are the results presented in a clear and understandable format?
Does the interpretation of the results seem consistent with the results presented?
Are there other explanations that could account for the results?
In essence the method of evaluation requires consideration of the individual sections of
the research article and this in turn enables an overall impression of the quality of the
research as presented. Below the reader is shown how to critically read all
components of a research report. Typical questions that the reader should ask when
reading a research article will be discussed.
6.4 Introduction
Every article starts with some form of introduction although it may not be explicitly
identified as an introduction. The purpose of the introduction is to quickly inform the
reader what the research is about and why it is important. The introduction may be very
short or it can be of considerable length. In the latter case this is usually because it
contains the literature review.
6.5 The Research Problem
At an early stage, usually in the introduction, the article should clearly indicate the
purpose of the research. The research problem can be stated as an aim, a question or
as a hypothesis. The research study may have several aims, questions or hypotheses.
It is not the case that one type of research statement is better than another but that
some types of research are better described by using one rather than another.
Quantitative research designs such as experimental designs are concerned with testing
out an idea so a hypothesis is frequently used. Qualitative studies are often exploratory
in their nature so it may be appropriate to use an aim and/or questions to identify the
purpose of the study.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
22
The important thing common to any research problem is that it should be researchable.
For example, the question "how much illness is there?" is not answerable because
different people and different groups define illness differently and it would be impossible
to measure. It’s rather like the old question "how long is a piece of string?" Answer: it
depends on the piece of string. However, if the study sets out to investigate "how many
working days are lost each year due to reported sickness?" this is researchable
because we know exactly what we want to measure. The following rules should be
observed in stating the research problem.
• An aim must be achievable.
• A question must be answerable.
• A hypothesis must be testable.
Exercise 3
Research Problems:
Look at the research problems listed below. Identify whether each one is an aim, a
question or a hypothesis and consider their researchability.
•
The purpose of this study is to identify the main reasons for GP referral of patients.
•
The number of staff employed by a primary care trust affects the number and type of
patients treated each year.
•
What is a good nurse?
The second thing we need to know about a research problem is the researcher's
definition of the key concepts or key variables. For example, if the study is about
"stress" how has the researcher defined stress? To some people, stress is a feeling or
response to unpleasant events while others think of stress in terms of the actual events.
A third group might take a broad definition of stress and describe stress as both a cause
and effect.
District nursing is another example. Some people might define "a district nurse" as any
nurse who works in the community, possibly including nursing assistants. Other people
restrict the definition to nurses employed at a certain grade or in possession of a
specialist qualification. Try out the example in Exercise 6.2 to see how people's
definitions can vary.
Exercise 4
Key concepts in a research problem:
If you were to read an article about "elderly people" what age group would you expect
the article to be about? You can either do the exercise alone by listing the different
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
23
ways you would define elderly or you could ask a group of friends to define the word
"elderly".
You may be surprised at the wide range of responses.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
24
6.6 The Literature Review
The literature review should put the present research problem into context. It should
contain a summary of the current state of knowledge about the topic and identify gaps in
the literature thereby making a case for carrying out research in this field. If the topic
has been extensively researched previously the proposed research may not be
necessary so it is up to the researcher to make the case. For example, the problem may
not have been investigated in relation to the proposed patient or population group. Or
previous research findings may provide conflicting results and this warrants further
study.
You should consider the following questions when evaluating the literature review:
• Is the literature review up to date?
• Is the literature review sufficiently comprehensive? Has the researcher provided you
with enough information to convince you that the research needs to be done?
• Are there any key pieces of literature that have been overlooked? Answering this
would depend on the reader's knowledge of the field.
• Is there a balanced review of differing viewpoints or findings? Or is the literature
review one sided ignoring alternative evidence?
• Is the cited literature appraised or is the author trying to impress you with a list of
references and no information about their contents?
• Are the contents relevant to the proposed research or have irrelevant references
been included?
• Does the writer present a logical coherent case for pursing the study?
6.7 Methodology
When the researcher describes the methodology there are two conditions that should
ideally be met. Firstly the researcher should provide sufficient information that the
reader could repeat the study if desired. Secondly the chosen methodology should be
justified. More specifically, the reader should consider the following points.
The type of information
Is qualitative or quantitative information required to deal with the research problem and
is the chosen methodology appropriate? This depends on the nature of the research
problem. If objective and measurable information is required then a quantitative
approach is appropriate but if the research sets out to explore a phenomenon in detail a
qualitative approach is necessary.
The research design
The design should be outlined and reasons given for using this approach. LesserThe NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
25
known designs would be enhanced by a brief definition: not everyone is familiar with
ethnographic studies or the Delphi technique!
The method of data collection
This should be appropriate for the type of information required. It would be time wasting
to use unstructured interviews for essentially quantitative studies where information
could be more efficiently collected through structured interviews or questionnaires.
Conversely, self completed questionnaires are unsuited to qualitative research: even
when there is space for comments or for respondents to express ideas the space is
limited and requires respondents to have skills in articulation and literacy.
The data collection tool
This should be described so that the reader knows what questions or observations were
included. Ideally, a copy of the interview schedule, questionnaire or observation
framework will be included but this is not always possible due to constraints of space
and journal style. The reader should at least have an idea of how many items were
included in the schedule, the topics covered and whether respondents replies were
recorded verbatim or whether their answers were fitted into a set of pre existing codes
as is the case with yes/no, fixed choice or multiple choice questions.
A second point to note regarding the data collection tool is whether the researcher has
developed an original tool or used an existing tool. It is acceptable to use data
collection tools that have been developed by previous researchers. It may even be
preferable since the tool is more likely to have been subjected to tests for reliability and
validity.
You should consider the following questions when evaluating the methodology
• Are all aspects of the methodology appropriate?
• Is the choice of method justified?
• What information are you given about the research design and the data collection
tool?
• What does the researcher say about the reliability and validity of the methodology?
• Could the study be replicated on the basis of available information?
The sample
Three factors should be considered: the criteria for selection, the method of sampling
and the size of the sample. The population under investigation should be defined and
the criteria used for selection clearly stated. For example, if the study is about "school
age children" what is the age range and does it include any or a specific type of school.
The method of selecting the sample should also be described so that the potential for
bias can be assessed. The representativeness of the sample is a factor in considering
whether the results of the study could be generalisable. The size of the sample is the
third factor for consideration. In qualitative studies it is usual to collect information from
a small sample because the methods of data collection and analysis are so time
consuming. It is not unusual to find qualitative studies using samples of less than twenty
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
26
subjects. Quantitative studies use much larger samples, which should have been
calculated as capable of providing statistically significant results.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
27
You should consider the following questions when evaluating the sample:
• How was the sample selected?
• Is the sample representative of the population?
• Is the sample large enough for the findings to be significant?
Exercise 5
Methodology:
Look at the extract below. It is taken from a research study that investigated nurses’
beliefs about smoking.
A questionnaire was sent to 20 nurses working in one hospital trust. The nurses were
asked to answer 20 questions about smoking.
A.
B.
C.
From this extract, what do you know about the methodology?
What don’t you know about the methodology?
On the basis of the information you have, what would you say critically about the
methodology?
Exercise 6
Sampling:
The following extract is taken from a study exploring the impact of introducing a
Saturday morning Family Planning Clinic in one health centre.
A survey was
undertaken using self-completed questionnaires as the method of data collection.
A sample of 50 women were invited to participate in the study. The sample consisted of
patients who visited the health centre for an appointment with their GP. The 50 women
were recruited during morning surgeries over a one-week period.
Using the information you have on sampling, comment on the sample. Include the
following:
•
The size of the sample.
•
What kind of sampling technique was used?
•
Potential problems with the sample.
6.8 Ethical Considerations
The researcher should identify any ethical issues. If none are identified the reader
should not automatically assume that there were none. You should consider the
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
28
following ethical issues:
• Would permission be required to undertake the study and, if so, did the researcher
seek permission?
• Is it likely that the researcher would have needed to seek approval from an ethics
committee and, if so, did he?
• What information was given to potential participants to enable them to decide
whether or not to participate and was sufficient information given?
• Were there any potential risks to participants and did the researcher take sufficient
steps to minimise the risks?
6.9 Data Collection
The methodology section should already have described the data collection tool. The
report should also identify any factors that could affect the outcome of data collection.
For example, if several people were involved in collecting the data this can affect the
reliability and validity. If several interviewers use a loosely structured interview
schedule, each interviewer is likely to use the schedule in a different way affecting
consistency. In a highly structured interview this would be less of a problem. Secondly,
when the researcher conducts the data collection personally, respondents are
sometimes reluctant to say exactly what they think. The phenomenon of the interviewer
affecting the outcome of an interview is known as interviewer bias.
The time of day, day of the week, time of the year or total period of time taken for data
collection can affect the outcomes. For example, asking people about their intended
actions may not be accurate if they are asked to forecast too far in advance.
Conversely, recall about past events can be poor if the event was too far in the past.
Asking people about their behaviour at a certain time of the year may be affected by
seasonal changes. You should consider the following questions when evaluating the
data collection:
• Who collected the data?
• When was data collected?
• Is there anything about the time scale that could potentially influence the results?
6.10 Results
The first result that should be included in the report is the response rate: the percentage
of the subjects who were invited to take part in the study who actually participated. Low
response rates should be viewed with some suspicion, as this can be indicative of some
fault in the research design. It can suggest that subjects were inappropriately selected
or that the method of data collection discouraged some potential respondents from
participating: perhaps the data collection tool was long-winded, difficult to complete or
caused offence.
The second set of results to look for are the characteristics of the sample. This provides
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
29
further information about some of the key characteristics of those taking part in the
study. It can include age, gender, grade of staff, number of years qualified, proportion of
staff who are qualified versus those who are unqualified, clinical classification of
patients, and so on. If the reader is considering implementing the research findings in
practice it is important to know how the research sample compares with the population
with which the reader is concerned.
The main results come next. In a journal article the word limit for papers may prevent
the researcher from presenting all the findings but it is his responsibility to point out that
only selected results will be presented and why. If the results are otherwise incomplete
this can lead the reader to suspect that the researcher is hiding something even if this is
not the case. For example, if a questionnaire contains ten items and only eight are
reported then the reader should wonder why this is so.
Results should be presented in a way that makes them easy to understand. If the
results are difficult to follow the potential value of the research may be lost. Tables and
figures can be helpful in presenting results but they should be clearly labelled so that
they are easy to follow. The results should be accurate and the text should match
information contained in tables and figures. Check the totals. It is not unusual to find
mistakes in the adding up so that the total number of respondents presented in a result
is greater or less than the sample group!
Results can be presented using words, numbers or percentages. It is misleading when
researchers cite percentages when dealing with small samples. For example, 10% of a
population of 10,000 is 1,000 people but 10% of a sample of ten accounts for only one
person. Beware of phrases like "the majority of the sample (60%) agreed with the
statement while only 40% disagreed". In a sample of ten this means than 6 people
agreed and only two fewer disagreed. While this is technically a majority it is only a
majority of two and a larger sample may have produced completely different results.
In summary, when reading the results ask the following:
•
•
•
•
•
•
What are you told about the respondents?
Are the results complete?
Are the results accurate?
Do the contents of figures and tables match the main text?
Have percentages been used properly in a way that does not mislead?
Are claims made about the results when these are inappropriate due to sample size
or narrow majorities?
6.11 Discussion
Most research articles contain a discussion of the results before finalising the paper with
conclusions. The researcher highlights the most interesting or important findings and
discusses any notable features. For example, if the sample was divided into sub groups
any differences or similarities between them would be pointed out. Conflicting
responses would be highlighted such as respondents claiming they have knowledge or
experience of something but then contradicting this later.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
30
The results should also be discussed in the context of previous research. The
researcher should explain how the findings add to previous knowledge pointing out
whether the findings of this study concur or dispute earlier evidence.
6.12 Conclusions
Towards the end of the article, the researcher summarises the research by identifying
the main conclusions that can be drawn from the findings. The researcher should return
to the original research problem and state whether the purpose of the research was met
through the study. As a consequence of the conclusions, the researcher should make
recommendations. These may be recommendations for practice that can be made as a
result of the research. Or, there may be recommendations for further research if the
researcher believes this to be necessary. Further research is usually recommended
because the original research study is flawed in some way or limited due to resources.
A good researcher will identify the limitations of the study for you. If not, it is part of
critical evaluation to identify the limitations. The limitations of a study are a major factor
in determining whether or not to implement the findings.
You should consider the following questions when evaluating the discussion and
conclusions:
• Was the aim of the research achieved, the question answered or the hypothesis
accepted or rejected?
• Are the results discussed in the light of previous research i.e. is the “new” knowledge
put into context?
• Can the main conclusions be traced back to the results?
• Are the recommendations for practice realistic in view of the results?
• Are the recommendations for research specific? Would the recommended research
really add further to the knowledge base?
• Are the limitations to the study fully acknowledged?
• At the end of it all, was the research worthwhile?
Exercise 7
A lecturer leading a health care research course for health care professionals decided
to investigate whether the teaching methods she used were the methods most preferred
by students. By asking them to complete a questionnaire, she discovered that the
teaching methods used most often with students were also the ones they a) found most
helpful and b) found most enjoyable. She concluded that the teaching was achieving a
quality standard by giving the students what they wanted and that the range of teaching
methods she employed would be suitable for teaching the same topic in other colleges.
•
Could any other conclusions be drawn from these results?
•
What are the main limitations of this study?
•
What do you think of her recommendations for practice?
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
31
6.13 Using checklists
There are a number of checklists that can be used to aid critical evaluation of research
articles. They have been specifically developed to critically assess different types of
research designs and articles e.g. randomised controlled trials, qualitative studies,
economic evaluations (for instance see Crombie 1996; Greenhalgh, 1997). Also visit the
Centre for Reviews and Dissemination (CRD) website to download checklists for various
types of study: http://www.phru.nhs.uk/casp/critical_appraisal_tools.htm
6.14 Writing a review
The structure of a review generally follows that of an original research article. It will
have sections for methods, results, discussion and conclusions as well as an
introduction.
The introduction should state the purpose of the review and give brief background
information on the subject of the review. The methods section should describe in detail
the methods used to compile the review. This will include details of which databases
were used, the period covered by the search and keywords used in searching. Details
of articles found in the search but excluded later should be given. This may include
reasons associated with irrelevance of work to the subject of the review or low quality.
The results section summarises the main findings reported in the articles reviewed. One
useful way of presenting results from many studies is in table form, for example:
Table 6.1. Compliance of insulin use by diabetics (fictitious reports and data)
Article
No. of
patients
Smith (2006)
Jones (2004)
Brown (2005)
50
56
55
Age range of
patients
(mean)
50-65 (55)
55-75 (62)
57-70 (62)
Method of
measuring
compliance
Self report
Self report
Blood-sugar level
No. (%) of
patients
compliant
42 (84)
48 (86)
28 (51)
The table should be accompanied by text highlighting key points you wish to make
about the data. This may include areas of agreement or disagreement between studies,
and comments on any of the methods used by the researchers in obtaining their results.
The review ends with a discussion of your findings and any conclusions you wish to
make. This may highlight important gaps in the field or how research in the area could
be taken forward, including implications for health care practice.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
32
6.15 Summary
Compiling a literature review is a crucial part of the research process. The first task is to
search the literature for articles on your particular subject of interest. This is aided by
the use of a range of bibliographic databases. The collection of articles must then be
assessed for relevance and quality. Finally, the whole process should be documented
in the form of a structured report. The whole endeavour requires certain skills, planning,
time and access to good library facilities. However, a good review will provide an
overview of the research already conducted, identify gaps or limitations in the research
and act as a sounding board for future research ideas.
Exercise 8
Summarising your results:
•
Scan the list of references from exercise 5.1. Use title or abstract information to
assess the relevance of each one. Make a note of relevant articles.
•
Collect together copies of the relevant articles from your library.
•
Extract key information from each relevant article and enter in a table (see Table
6.1 for an example).
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
33
7. Answers to Exercises
Exercise 1
This exercise is based on self-selected examples and hence it is not feasible to provide
a specific answer. You should, however, draw on the guidelines provided in the text
when approaching it.
Exercise 2
This exercise is based on self-selected examples and hence it is not feasible to provide
a specific answer. You should, however, draw on the guidelines provided in the text
when approaching it.
Exercise 3
Research Problems:
•
The purpose of this study is to identify the main reasons for GP referral of
patients.
This is an aim. In its present form it seems that the study is concerned with
identifying reasons for all types of GP referrals. This is possible but it is more
likely that researchers would be more interested in focusing on particular types of
referral e.g. orthopaedic, gynaecology etc.
•
The number of staff employed by a primary care trust affects the number and
type of patients treated each year.
This is a hypothesis as it predicts a relationship between the number of staff
employed by a NHS Trust and the number and type of patients treated. It seems
a strange study as one would assume that the more staff employed, the more
patients would be treated. Similarly, the larger the trust the greater scope for
developing specialised services so it would also be expected that there would be
a greater range of patients treated.
•
What is a good nurse?
This is a research question. But it is not possible to answer because there is no
one definition of a good nurse. Although there may be some commonly held
beliefs of what makes a good nurse, the detail may vary according to who is
asked. It would be better to focus on the perceptions of a particular group e.g.
what do children perceive as being “a good nurse”?
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
34
Exercise 4
Key concepts in a research problem:
“Elderly people" may be variously defined as those belonging to a particular age group
(for example, over 60 years of age, over 75, over 80) or because they share a
characteristic (pensioners). I have also known individuals to answer this question by
identifying health characteristics.
Exercise 5
Methodology:
•
The research design appears to be a survey, the method of data collection is a
postal questionnaire and we know the sample is 20 nurses.
•
We don’t know whether the data being collected is qualitative or quantitative in
nature. Nor do we know the definition of a nurse - is it any nurse working in a
hospital or is it restricted to nurses working within a particular specialty like
respiratory medicine?
•
It is a small scale localised study, probably a convenience sample. This will
affect the generalisability of the results. Insufficient information is provided about
the questionnaire and there is no mention of reliability or validity. It would not be
possible to replicate the study on the basis of the information provided.
Exercise 6
Sampling:
The sample is small. Even in a single-handed practice it would represent a small
proportion of the female population although not all women in the practice would have
need of a family planning clinic due to their age. It is likely that this would apply to at
least some of the 50 women selected so the results would actually be based on the
views of less than 50 women. The sample is a convenience sample as only those who
attended the health centre could be selected. We do not know when the sampling took
place and there could be something about the time of year that affects the sample. For
example, if the research took place during the school holidays, many women of
childbearing age might be on holiday or if the survey took place during the flu
vaccination period many of the women attending might be older women who do not
need a family planning clinic.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
35
Exercise 7
Drawing conclusions
•
It is possible that the students identified these teaching methods as their
preferred methods because they are the only methods with which they are
familiar: they did not identify other methods because the methods were never
used and were not understood by the students! It is also possible that the
students were biased for some reason and gave positive answers to the lecturer,
perhaps because they liked her or they may have been worried about her
response if she had received more negative feedback.
•
The main limitation of this study is that the sample came from one college and
the situation is not necessarily representative of the situation in other colleges.
Furthermore she only sampled her students, not all students in the college.
•
Following on from A and B, her recommendations are inappropriate.
Exercise 8
This exercise is based on self-selected examples and hence it is not feasible to provide
a specific answer. You should, however, draw on the guidelines provided in the text
when approaching it.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
36
8. References
Centre for Reviews and Dissemination, critical appraisal checklists:
http://www.phru.nhs.uk/casp/critical_appraisal_tools.htm (Accessed 15 February 2007)
Chalmers, J. and Pearson, A. (2005) Oral hygiene care for residents with dementia: a
literature review. Journal of Advanced Nursing, 52 (4):410-419.
Crombie, I.K. (1996). The Pocket Guide to Critical Appraisal. BMJ Publishing Group,
London.
Greenhalgh, T. (1997). How to read a paper : getting your bearings (deciding what
the paper is about). British Medical Journal, 315:243-246.
Lui, M., Ross, F. and Thompson, D. R. (2005). Supporting Family Caregivers in Stroke
Care: A Review of the Evidence for Problem Solving. Stroke. 36 (11):2514-2522.
Richardson, G., Eick, S. and Jones, R. (2005). How is the indoor environment related to
asthma?: literature review. Journal of Advanced Nursing. 52 (3):328-339.
9. Further Reading and Resources
Hart, C. (2001). Doing a Literature Search: A Comprehensive Guide for the Social
Sciences. Sage Publications Ltd, London.
TRIAGE – The NIHR RDS EM / YH Research Information Access Gateway at
http://www.trentrdsu.org.uk/resources.html (accessed 15 February 2007)
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
37
10. Glossary
Abstract
a very concise overview of an article.
Action research
small-scale interventions in real life situations involving practitioners.
E.g. introducing a specialist practitioner role in the practice setting.
Article
a self-contained report on a particular subject.
Case studies
an in depth study of a single or small number of units. The unit may
be individual people, patients, groups or organisations.
e.g.
evaluating a new service.
Cluster sampling
used when the population is diversely spread over a geographical
area and for various reasons it is preferable to have groups of
subjects from several sites rather than randomly selecting the whole
sample from the whole population. e.g. to investigate the grades of
community nurses employed nationally the sample could select a
sample of community nurses in one NHS Trust in each of the health
regions.
Convenience
also called incidental sampling. Utilises readily available subjects
and sampling and is often used in small-scale localised research
projects. The sample may not be representative of the population as
a whole and the results may not be generalisable. e.g. patients
selected from one geographical area such as an electoral ward may
have particularly high or low levels of deprivation.
Ethnography
qualitative investigation of cultures and population subgroups that
seeks to explore, describe and explain cultural behaviour. E.g.
understanding of mental illness within a particular Asian sub group.
Experimental
controlled investigations, which try to establish to establish cause
and designs effect between two or more variables with the purpose
of predicting outcomes. E.g. whether one type of medication is more
effective than another in treating a particular illness.
Grounded theory
research that involves the formulation, testing and development of
new theories. E.g. identification of the grief process following
bereavement.
Hypothesis
a testable idea that is written as a statement describing the research
problem.
Incidental
sampling
see convenience sampling
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
38
Interview
schedule
a list of questions used in an interview.
Journals
magazine-sized publications containing articles.
Literature
all published written work.
Literature review
a concise summary of published work on a specified subject.
On-line
connection via a computer network to a large remote computer that
holds references to the literature.
Phenomenology
descriptions of individuals' lived experience of events. E.g. the
experience of caring for someone with Pre senile dementia.
Population
a collective name for the group of people that the researcher is
interested in studying.
Purposive
sampling
also called quota sampling. Subjects are selected because they
have special knowledge of the topic under investigation. e.g. key
stakeholders in an organisation.
Quota sampling
see purposive sampling
Random
sampling
a method that gives every member of the population an equal
chance of being selected. Frequently described as the method of
sampling least likely to produce a bias but sampling errors can occur
which can unintentionally produce bias.
Reliability
a term that refers to the ability of the data collection tool to give
consistent results. E.g. a 12" ruler would consistently measure an
item as being the same length irrespective of how many times it was
used or who used it.
Respondent
the people who agree to take part in a research project and from
whom data is collected.
Sampling
the selection of subjects from the population under study.
Sequential
sampling
the size of the sample is not pre-set. The researcher collects data
from each subject in turn until he is satisfied that there is no new
information for collection - the topic is saturated. Used mostly in
qualitative research e.g. In setting up a new service potential users of
the service are asked what they would want until no new ideas
emerge.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
39
Stratified
sampling
used when the population contains sub groups and it is necessary to
ensure that representatives of all groups are included e.g. patients in
different age bands, nurses employed on different professional
grades, health care staff from a range of professions. Randomisation
within each sub group can be applied.
Surveys
investigations into the beliefs, attitudes or behaviours of a section of
the population which share certain characteristics e.g. patients'
expectations of their health centre.
Systematic
review
an unbiased and scientifically rigorous approach to reviewing the
literature.
Systematic
sampling
involves taking the nth name on a list such as every third person or
every tenth. Sometimes used by researchers who claim to have
used random sampling but factually this approach automatically
eliminates certain members of the population who may have a
perspective which is useful to the study but goes unnoticed. e.g.
taking the first named member of the household from an electoral
poll will almost certainly eliminate the younger members of the
population.
Validity
a term that refers to the accuracy with which a data collection tool
collects the information it intends to collect.
Variables
the features of particular interest in a research study. They include
characteristics of the subjects in the sample group, any interventions,
and outcomes.
If you have difficulty in understanding any of the definitions you may prefer to consult
other sources. Introductory books on research often contain a glossary. Specialist
dictionaries will also be helpful. If you would like more information on any of the terms,
ideas or processes you could look at the appropriate The NIHR RDS EM / YH resource
packs. There are a number of titles in the series and each pack concentrates on a
particular aspect of research.
The NIHR RDS for the East Midlands / Yorkshire & the Humber 2009
Literature Searching
40