Gathering & Evaluating Information from Secondary
Sources
Published by the Universities of Edinburgh, Glasgow and Strathclyde
Acknowledgements
The material from this booklet has been developed from discussion groups and
interviews with the research staff of Glasgow and Strathclyde Universities
The advice and contributions of Mr P.P. Benassi, Dr Avril Davidson, Mr Keri Davies,
Prof George Gordon, Prof J.R. Morris, Mr Gerry McBride, Mrs Janice Reid, Dr Alan
Taylor and Mrs Sheila Thompson are acknowledged. With special thanks to Miss
Alison Faichney of the University of Glasgow library.
The advice of the project Steering Group: Prof Michael Anderson, University of
Edinburgh; Dr Nuala Booth, University of Aberdeen; Dr Ian Carter, University of
Glasgow; Ms Jean Chandler, University of Glasgow; Dr Avril Davidson, University of
Glasgow; Prof George Gordon, University of Strathclyde; Prof Caroline MacDonald,
University of Paisley; Prof James McGoldrick, University of Dundee; Dr Alan Runcie,
University of Strathclyde; Prof Susan Shaw, University of Strathclyde; Dr Alan Taylor,
University of Edinburgh; Prof Rick Trainor, University of Glasgow is also
acknowledged.
The project was funded by the Scottish Higher Education Funding Council.
Other titles in Series
Gaining Funding for Research
Interpreting and Documenting Research and Findings
Preparing the Research Brief
© Universities of Edinburgh, Glasgow and Strathclyde 1999
Cartoons D. Brown & W. L. Wilson
ISBN 0 85261 687 2
Printed by Universities Design and Print
Introduction
This booklet is one of a series of booklets aimed at researchers in the early stage of
the research career cycle. In this instance the booklet is primarily targeted at postgraduate students beginning their doctoral studies.
The booklet is not intended to be read in one fell swoop, but rather to be dipped into
as and when the occasion arises. It is divided into three sections. The first two
sections comprise general comments on library and Internet use. These comments
are either adapted from library literature, or personal comments from specialists. The
third section provides practical advice on the specialised techniques and skill required
in the gathering and evaluation of secondary information. This section is subdivided
into points of advice, and examples from experienced researchers to highlight these
points. At intervals within any section the main points are highlighted by bullet
points, the bullet points always refer to the points and examples preceding them.
Information for the second section was collected through a series of interviews and
discussion groups, which were formed from experienced staff, PhD students, and
Contract Research Staff (CRS).
The booklet is not intended to be exhaustive or definitive. The issues raised are
those which most exercised the minds of the researchers providing the comments for
its preparation. These comments do offer interesting contrasts of opinion, either
because commentators disagreed about the way to approach a certain issue, or
because researchers from different subjects took different approaches in their
methodology. The nature of the examples provided in the booklet are a reflection of
the interests of those taking part in the discussions and interviews, and possess no
greater significance than that.
There are two appendices, the first gives a list of useful Internet sites, the second a
list of useful reference databases.
Whether it is planning future research, putting in grant applications, or placing your
results and conclusions in context, the gathering and evaluation of secondary
information is very much a basic and essential skill. It is the aspect of research that
is first encountered. Be it looking up a word in a dictionary at primary school, or
gathering information for an essay at college, we all have some experience upon
which we can build.
Three ages in gathering and evaluating ‘by Mr. Shakespeare’
"At first the infant,
Mewling and puking in the nurse’s arms.
And then the whining school-boy, with his satchel
And shining morning face, creeping like snail
Unwillingly to school.
And then the lover,
Sighing like furnace, with a woeful ballad"
Contents
SECTION ONE: USING THE LIBRARY ...........................................................5
INTRODUCTION.................................................................................................. 5
PHOTOCOPYING ................................................................................................. 5
ELECTRONIC JOURNALS ........................................................................................ 5
IDENTIFYING USEFUL REFERENCES ......................................................................... 6
Physical Reference Sources .......................................................................... 6
Electronic medium ....................................................................................... 7
DATABASE SEARCHING ......................................................................................... 8
Keywords.................................................................................................... 8
Using Truncation ......................................................................................... 8
Using Boolean Operators .............................................................................. 9
Adjacency ................................................................................................... 9
Field Searching ............................................................................................ 9
Using Parenthesis or Nesting ........................................................................ 9
Controlled Vocabulary ................................................................................ 10
Re-using Saved Queries ............................................................................. 10
Refining Results ........................................................................................ 10
JOURNAL EVALUATION ....................................................................................... 11
Impact Factor (from ISI database information) ............................................ 11
Immediacy Index....................................................................................... 11
Cited Half-life ............................................................................................ 12
Location.................................................................................................... 12
SECTION TWO: USING THE INTERNET ......................................................13
INTRODUCTION................................................................................................ 13
BOOLEAN BASICS ............................................................................................. 13
SAVING DOCUMENTS ......................................................................................... 14
CAPTURE A GRAPHIC ......................................................................................... 14
BOOKMARKING ................................................................................................ 14
SECTION THREE: PRACTICAL APPROACHES TO THE GATHERING AND
EVALUATION OF SECONDARY INFORMATION ...........................................15
INTRODUCTION................................................................................................ 15
SCOPE OF THE INITIAL INFORMATION GATHERING PROCESS ......................................... 15
Introduction .............................................................................................. 15
Points to Consider ..................................................................................... 15
ENSURING THAT NOTHING IMPORTANT IS MISSED ..................................................... 17
Introduction .............................................................................................. 17
Points to Consider ..................................................................................... 17
AWARENESS OF UNPUBLISHED RESEARCH ................................................................ 18
Introduction .............................................................................................. 18
Points to Consider ..................................................................................... 18
PROCESSES TO RUN CONCURRENTLY WITH INITIAL INFORMATION GATHERING .................. 19
Introduction .............................................................................................. 19
Points to Consider ..................................................................................... 19
DIFFICULTIES IN GATHERING INFORMATION ............................................................ 20
Introduction .............................................................................................. 20
Points to Consider ..................................................................................... 20
BEING UNDERSTOOD WHEN REQUESTING INFORMATION ............................................. 22
Introduction .............................................................................................. 22
Points to Consider ..................................................................................... 22
ETHICAL CONSTRAINTS ON THE USE OF SECONDARY DATA ........................................... 23
Introduction .............................................................................................. 23
Points to Consider ..................................................................................... 23
EVALUATING PUBLICATIONS ................................................................................ 25
Introduction .............................................................................................. 25
Points to Consider ..................................................................................... 25
IDENTIFYING THE RISK OF INACCURATE INFORMATION ............................................... 27
Introduction .............................................................................................. 27
Points to Consider ..................................................................................... 27
BIAS, AND HOW TO AVOID IT ............................................................................... 31
Introduction .............................................................................................. 31
Points to Consider ..................................................................................... 31
REFERENCES ................................................................................................... 32
Books and other publications ...................................................................... 32
APPENDIX A - SOME ABSTRACTING AND INDEXING JOURNALS ...............34
INTRODUCTION................................................................................................ 34
Arts/History/Languages/International .......................................................... 34
Economics/Politics/Sociology/Anthropology .................................................. 34
Education ................................................................................................. 34
Engineering/Mathematics ........................................................................... 35
Law .......................................................................................................... 35
Management/Business ............................................................................... 35
Music........................................................................................................ 35
Theology/Philosophy .................................................................................. 35
Science/Health (Human and otherwise) ....................................................... 36
APPENDIX B - SEARCH ENGINES AND WEB SITES ....................................37
BRITISH ACADEMIC RELATED WEBSITES NATIONAL INFORMATION SERVICES AND SYSTEMS . 37
BUBL INFORMATION SERVICE ............................................................................. 37
GENERAL WWW SEARCH ENGINES ....................................................................... 37
Dogpile ..................................................................................................... 37
Searching for Softwareg2057 ...................................................................... 38
Subject Gateways ...................................................................................... 38
Evaluation of Information Websites - A Selection .......................................... 38
Section One: Using The Library
Introduction
Meticulous planning and attention to detail are essential in conducting a literature
search. The first step is to ensure that you understand fully the services that are
offered by the library staff, and what materials the library holds - on the shelves, in
reserve collections, on microfiche or microfilm. Bibliographies, abstracts, indices and
encyclopaedias can save hours of searching for sources, but they are not always
easy to find or to understand (Bell, 1993). However, since 1993 methods of
accessing information have changed significantly, and no doubt by 2000 will have
shifted substantially again. Details on available information can just as easily be
accessed electronically as through the medium of the printed page. Use of library
catalogues on the institution’s web site will identify what journals or books are
available, and their loan status. BIDS and other databases providing indexing and
abstracting facilities are readily available . Support your local librarians- they will help
a lot!
The three main sub-sections of Section One provide information on:



Methods of evaluating journals for their impact within their field.
Techniques for database use,
Methods of identifying useful references,
Photocopying
It is not necessary to photocopy everything. Photocopying is time consuming and
expensive. In the long run it is easier to be decisive and restrict your photocopying
to articles which are relevant. Not photocopying an article will not lead to future
problems as long as you keep your records to a standard which will allow you to
return to the article at a later date. If you cannot decide on the importance of an
article, try photocopying just the abstract and introduction. Bear in mind that you and your institution - must make sure copies taken are within the limits of the
Copyright Act,1988.
Remember to read the literature you photocopy. It is all too easy to succumb to the
temptation of thinking that photocopying is an end in itself!
Electronic Journals
Many journals are now available online, as well as, or instead of, in print. This allows
you to consult, print or download (subject to copyright), articles in your own office or
laboratory, and can give you faster access and links to, for example, other articles, or
databases e.g. Medline citations. There are over 1,000 such journals - and the
number grows every day. In most universities, these journals can be accessed from
the institution’s networked PCs, and from elsewhere via dial-in services. You should
contact the Library Enquiry Service to obtain further information.
Identifying Useful References
Most libraries have literature describing what the library has on offer. This is well
worth checking as the variety and detail of this literature can be substantial. Library
Internet sites often allow offsite access to catalogues of library contents and loan
status of books/journals. One way to start building a good understanding of your
local library services is a library induction course, particularly if the institution runs
specialist induction courses.
The following section is split into two subsections, identifying references through
physical sources, and identifying references through electronic media. At the end of
the booklet Appendix A lists a range of electronic and physical reference sources.
Physical Reference Sources
Reference Lists
Reference lists at the end of books, journal and review articles are an important
method of identifying essential literature. Bear in mind that references in books tend
to be about three years out of date by the time they come into print. For more
recent references journals are generally better.
Indices and Abstracts
There are a wide range of indices and abstracts which allow you to scan many
journals at a time, and give you access to resources far greater than that of any
single library. These fall broadly into two types. Abstracting journals, organised by
subject, which provide, along with the basic information, an abstract giving some
details of the paper’s content. Citation indices begin searches with older papers and
allow the identification of those contemporary papers which cite the older paper in
their reference lists. Abstracting services are only as good as the journals covered, so
check that the key publications in your field are included. (Appendix A details some
abstracts, databases and citation indexes).
Bibliographies
Bibliographies provide a record of what has been published in a particular area. They
can be general, or subject specific. International bibliographies (e.g. Répertoire
International, for musical sources) can prove valuable sources of reference. It is
important to remember that bibliographies can become dated very quickly. If you
want to find out what is being published currently, consult major publishers’
catalogues (e.g. Blackwell, Routledge), or, for the most complete record of material
published in the U.K., the British National Bibliography (Moore, 1996).
Encyclopaedias and Dictionaries
Apart from general basic sources, subject encyclopaedias and dictionaries may be
the major reference tools of a discipline. At their best, they can provide authoritative
summaries of fundamental principles and current research trends. However, such
material can go out of date quickly, and it is a good idea to look at the date of
publication, the dates of references it contains, and the general scope of coverage of
any reference tool (Moore 1998).
Browsing
Consider simply browsing the shelves. Find a book that interests you from a
reference list, a verbal recommendation, whatever, and then scan the titles of
the books situated on the nearby shelves. Alternately browse the journals. Go
to the recent journal section in your library and look through the contents
pages of the journals which have some connection to your subject.
Electronic medium
Computer Search Service
Most universities offer a Computer Search Service. Among the most comprehensive
services, offering access to over 200 databases world-wide via international online
hosts, are provided by search analysts on your behalf. Some databases are based on
printed sources, e.g. INSPEC, others are available only on computer. They cover a
wide range of subjects from Astronomy to Marketing to Zen. They are generally
more up-to-date, and offer more sophisticated searching techniques than are
available on CD-ROM discs. Make an appointment to discuss your research topic with
your specialist librarian. They will advise you as to the most suitable database, and
on an appropriate search strategy. (Ashworth, Faichney, 1998)
Reference Manipulation
It is possible to create a personal information store, to search by keyword, or to
produce lists of references on your subject. It is possible to store your search results
in electronic form. They can then be downloaded to your own databases using
‘personal bibliographic database software’, enabling you to keep and search articles
based on your own research area. When purchasing ‘personal bibliographic database
software’ make certain that it provides software import facilities for the databases
which you use regularly.
Updating Services
Libraries may offer updating services, which use keywords linked to a personal
profile constructed to match your research interests. This profile is regularly run
against international databases and the results then sent to you (Faichney, 1998).
Libraries may charge for this service, though the cost is usually modest. However,
independently of your institution’s library some data sources, e.g. NISS Biomedical,
will now send regular updates of references in the subjects of your choosing.
References can be identified by a variety of routes which include:






reference lists at the end of books/ journal articles
bibliographies
indexes and abstracts
encyclopaedias and dictionaries
browsing
computer search and updating services
Database searching
Précis of Marlborough (1998).
The interface (what you see on the computer screen) to databases can vary widely.
However, there are particular skills which are common to most, and which should be
mastered in order to increase efficient retrieval of information. It can make the
difference between finding two papers or 20 papers, or alternatively, reducing a
result of 200 papers to a manageable number.
The three basic principles considered in the following section are: choosing
keywords; using truncation; using Boolean operators. This section then further
examines how these techniques may be used in the location of references.
Keywords
It is important when carrying out an electronic search to use the correct terminology.
Failure to do so is likely to result either in missing the data sought, or in acquiring far
more data than bargained for.
Example: Thus "woodmouse" may bring in references to both A. sylvaticus (found in
the British Isles) and P. maniculatus (found in North America). Americans spell labour
‘labor’, so there may be rather limited results if ‘standard’ English spelling is used for
searching a North American English language database.
Your librarian should be able to advise you on where guides on terminology are
located (e.g. the ‘Synopsis of Animal Classification’, or for anatomical terminology,
‘Nomina Anatomica’) Consider some lateral thinking if searches are producing less
than desired: synonyms, animals in a similar ecological niche.
Using Truncation
If you enter a keyword in the singular form, you will retrieve references only in that
form. As well as singular and plural, many keywords have variant word endings - all
of which may be relevant.
e.g. computer, computers, computing, crystal, crystals, crystallisation.
In order to reduce typing (never mind hours ‘thinking up’ all possible combinations)
most databases allow truncation using a wildcard. This allows the user to type the
relevant part of the word stem plus the wildcard symbol.
Examples are comput* , legal*
The * symbol represents the wildcard symbol, which will vary from database to
database.
NB. However, if used without due care, wildcards can retrieve rather more than a
lifetime’s worth of reading. If this happens, refine your search by adding extra
conditions.
Using Boolean Operators
These operators are used when you wish to specify combinations of keywords during
an electronic database search.
AND - "Byzantium" AND "Religion", both words must be present in the reference for
retrieval.
OR - "woodmouse" or "long-tailed field mouse" (same animal), if either word is
present the reference will be retrieved. This is useful for synonyms, or alternative
spellings/topics.
NOT - "weevil not pine", references containing the word pine will not be retrieved.
Thus having found data on the pine weevil you may now wish to find data on other
weevil species.
Adjacency
It is possible to search for a phrase. In some databases you simply enter the phrase:
e.g. lung cancer, heart disease, Russian history.
Alternately the database may need you to indicate that you wish to find one keyword
adjacent to another; History (w) Russian, heart (w) disease or, remembering to use
truncation, Histor* (w) Russian.
The precise command used to indicate adjacency varies from database to database.
Field Searching
Most references in databases are divided into fields. Some examples are ‘author’,
‘title’, ‘document type’.
Example: A record in the BIDS database looks like.
TI: Fly Fishing
AU: Hartley_JR
JN: Yellow Pages, 1981, Vol. 12, No. 10, pp. 806-808
Where TI is title, AU is author, JN is journal details.
In many databases you can look for information in these fields, giving you much
more control over your search. Thus it is possible to search for papers written by a
particular author, for keywords appearing only in the title field, for a particular
journal, for papers published in a particular year or years, written in a particular
language, or as a publication type e.g. review.
Using Parenthesis or Nesting
It is possible to construct complex search statements. However, to ensure that the
search is carried out in the manner desired you must use parenthesis to clarify the
logic (in much the same manner as is used in mathematical equations). When in
doubt keep it simple and combine it later.
Example: You are looking for recent information on water pollution.
A possible search statement may look like this.
(pollut* or toxic*) and (water or river* or stream* or lake* or reservoir*) and
PY³1993
(Again the nature of the commands, * and Publication Year are database specific).
Controlled Vocabulary
Many databases use a thesaurus to index records within the database. This is a way
of ensuring that all records on a particular topic have the relevant thesaurus term
assigned to them. Using thesaurus terms can be a way of ensuring that you get
highly relevant information, and reduce the number of ‘false hits’. The thesaurus may
refer you to related terms.
Example: The Medline database has a highly structured thesaurus. This would tell
you not to use the term ‘cancer’ but to use ‘neoplasm’ and consider using ‘carcinoma’
and ‘tumour’. ‘Cancer’ will produce results, but more relevant papers will be selected
using the recommended terms.
Re-using Saved Queries
In some databases (e.g. BIOSIS) you can re-use saved queries, showing all of the
previous searches you have made with the number of matching entries
e.g. Search 1 Author = Auntie Mary articles=1
Search 2 Keyword = canary articles=5000
You can then combine any number of these search results using standard Boolean
operators i.e. type in ‘1 AND 2’ will identify articles by Auntie Mary with keyword
canary. This is very similar to using operators in the standard search menu but
allows you to try different combinations more easily and see how many articles each
throws up.
Refining Results
One does not always get what one wants, but occasionally what one deserves usually at the wrong time.
If you get too many papers there are a number of options:





Check to see if you can use thesaurus keywords
Add additional keywords, or Boolean connectors e.g. NOT
Use more specific keywords
Check to see if you can use thesaurus keywords
Restrict your results by limiting to particular fields e.g. publication years
If you get too few papers, or no papers:





Check that you have no typing or spelling errors within your command
Experiment with truncations
Have you used all possible synonyms or alternative spellings/keywords?
Use the OR command for new combinations
Are there subject or thesaurus terms you could use?
Journal evaluation
It is important to remember when using the journal evaluation measurements listed
below that they are not the final and ultimate truth. Journals such as ‘Nature’ are
highly cited, but have a definite emphasis in terms of what they will publish. Thus for
an ethologist more specialist magazines such as ‘Animal Behaviour’, or ‘Behaviour’,
are far more likely to be of interest. Indeed in small specialisms, or in most arts
disciplines, there may be few journals which are covered by such measurements. In
these circumstances seeking out journals of high impact factor may result in missing
valuable information from smaller, less well cited, sources.
Impact Factor (from ISI database information)
The Journal impact factor is a measure of the frequency with which the ‘average
article’ in a journal has been cited in a particular year. The impact factor will help you
evaluate a journal’s relative importance, especially when you compare it to others in
the same field. It is calculated by dividing the total number of citations to articles
published in the previous two years by the total number of articles published in the
two previous years. Obviously there are limitations to the value of impact factor
assessments: where delays in publication exceed two years, by definition no paper
can be cited within a two year period, also impact factors tend to be dominated by
English language, American-cited journals in certain areas of science.
The impact factor is useful in understanding the significance of absolute citation
frequencies. It tends to discount the advantage of large journals over small ones, of
the frequently issued over the less frequently issued, and older over newer. In each
example the first is likely to produce/have produced a larger body of literature than
the second. All things being equal, the larger the body of literature, the more the
journal will be cited. By providing some qualification of the quantitative data in the
Journal Citation Reports (JCR), the impact factor is an important tool for journal
evaluation.
Immediacy Index
The immediacy index is useful in comparing how quickly journals are cited. Because
it is a per article average, the immediacy index tends to discount the advantage of
large journals over small ones. However, frequently issued journals may have an
advantage here. An article published early in the year has a better chance of being
cited than one published later in the year. There is the issue of the length of the
waiting list to get published. For comparing journals specialising in cutting edge
research, the immediacy index can provide a useful perspective.
The index is calculated by dividing the number of current citations to articles
published in the same year by the number of articles published in the current year
(i.e. 1979 citations to 1979 articles in journal XXXX, divided by total 1979 articles in
journal XXXX).
Cited Half-life
The cited half-life is the number of publication years from the current year which
account for 50% of current citations received (e.g. in 1988 if the score is 10 then ½
the articles cited are 10 years or older). This figure helps you evaluate the age of the
majority of cited articles published in a journal. On the CD-ROM Science and Social
Science Citation Index only journals with more than 100 citations are listed with a
citation half-life.
Location
"Impact factors" etc. are produced by ISI in two forms. Your library may have the
information, updated annually, in microfiche, or as a database, called "Journal
Citation Reports". This information is rarely available free via the Web, as it is
copyright data. (Faichney,1998)
Section Two: Using The Internet
Introduction
The Internet is vast in terms of resources and information, finding what you want
can be compared to looking for a very small needle in a very large haystack.
Thankfully there are search tools available that are just made for the purpose of
searching the Internet, these are known as Search Engines (see Appendix B).
There are a number of these search engines, and they are all free to use. To use
these engines you need simply to go to their Web Site and enter the text for which
you wish to search. Doing this will bring back thousands of results containing one or
all the words you entered, many of which may not be relevant to the subject you
were looking for. To narrow down your searches you must do what is known as a
Boolean Search.
Aside from this brief section there are tutorial sites on how to search the Internet to
be found on the Internet, one example of which is
http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/
You should also look at subject-related gateways, collections of evaluated sites
(Appendix B).
Boolean Basics
To get just what you want you must use the Search Engine’s own Language.
The terms, which are part of a century old system called Boolean Logic, work as
effective filters for finding just the information you are looking for. Most of the
Search Engines use some form of Boolean query. Check the help sections of the
engine you are using to see if it supports Boolean searches.
NEAR - This term will find words located within a certain number of characters of
each other. Not every service uses NEAR in the same way, and some do not offer it
at all. For example, AltaVista uses it to find words within ten characters of each
other, whilst WebCrawler lets you specify the number of characters.
Example for Alta Vista: NEAR finds documents containing both specified words or
phrases within 10 words of each other. Mary NEAR lamb would find the nursery
rhyme, but probably not religious or Christmas-related documents.
PARENTHESES - Organise your searches even further by using parentheses. NOT
cats AND dogs will return pages with dogs in them; NOT (cats AND dogs) will avoid
pages with both names.
QUOTES - Putting a string of words into quotes will turn it into a phrase. For
example typing "The World Trade Centre" will bring back results with this phrase in
them; without the quotes you would have to sift through probably thousands of
results containing one or two of the words only.
Saving Documents
When you have found something useful you may wish to save the document. You
can do one of the following, the actual choice will depend upon the software you are
using.
1. COPY (click and drag) and PASTE into a document in your own space
2. You can get a copy of the whole document as follows:- In Internet
Explorer go to the Navigation icon - to Benchmarks - FILE - then SAVE AS;
you should now be able to save this as a document in your own folder
3. In Netscape go to FILE - SAVE AS.
Capture a Graphic
1. In Netscape Navigator point to picture and click right mouse button, you
will then be asked for a file name under ‘save image as’.
2. In Internet Explorer the procedure is the same although you now enter a
file name under ‘save image as’.
Bookmarking
Having found a useful site, bookmarking makes this quickly obtainable for future
reference.
1. In Netscape Navigator click on menu heading BOOKMARKS,
2. Choose ADD BOOKMARK.
It is possible to organise bookmarks for ease of reference.
1.
2.
3.
4.
In Netscape Navigator click on menu heading BOOKMARKS,
Choose GO TO BOOKMARKS.
Choose ITEM, then INSERT FOLDER
Name the folder, copy the files to it.
Section Three: Practical Approaches To The Gathering
And Evaluation Of Secondary Information
Introduction
The purpose of this section is to provide practical answers to the competencies
identified in the Gealy, Westlake & Clarke, (1997) report. The information was
gathered from a series of interviews and discussion groups held with university
researchers. Participants ranged in experience from postgraduate students to senior
members of the research staff. The practical approaches are presented in two forms,
a general statement of principles and, where possible, actual examples to highlight
the approach suggested.
It may appear at times that some of the advice is contradictory, and indeed it may
be so. There are two possible reasons for this. Firstly, the commentators may have
been from different academic disciplines, e.g. arts, science, social science. That is
not to say that neither could learn from the other, but at times their approaches
were divergent. A second possibility was that the two commentators quite simply did
not agree and, as there is rarely an absolute right or wrong in these matters, both
comments were noted.
On the principle that divergent disciplines can learn from each other, the comments
have been grouped by the nature of their aims, rather than by discipline. That some
sections are dominated by contributions from a particular field simply reflects the
specialisations of those taking part in the project.
Scope of the initial information gathering process
Introduction
Faced with a new project there are certain opening moves which should be gone
through before committing yourself to a specific line of enquiry. Ensuring that you
are thoroughly familiar with the background reading, keeping an open mind at the
early stages, talking to and taking advice from your colleagues and supervisor are all
vital in building a sound foundation for your future research.
Points to Consider
Good supervision is a major factor in assisting a new researcher. New PhD
students should make good use of their supervisor to direct their initial
reading. Supervisors should help you not only in identifying which are the
best opening books, articles to read, but also the electronic aids that are
available to help you locate further information.
Make sure that you understand the basics of the subject before you tackle the
serious reading. Familiarise yourself with the terminology, techniques and
methodologies employed in your new field. Although you may be impatient to delve
to the heart of the matter it is important not to skimp on the general background
reading. It is not only frustrating to come across words, comments, or references to
techniques and theories with which you are unfamiliar, there is the risk that
unfamiliarity with key words will result in missing important information during
literature searches.
Talk to those familiar with the field, they can help you avoid going down the wrong
track and let you know what the current trends are. Similarly, the advice of other
researchers can be immensely valuable. They have had recent experience of
beginning in a new subject, and can provide helpful pointers to problems for new
initiates.



If you are a PhD student you have a right to expect competent help and
support from your supervisor.
Familiarise yourself with the background to the subject before attempting its
complexities.
Communicate with your colleagues.
In some subjects it may not be possible to determine the precise scope of the
information gathering process at the outset. Indeed the initial step in the information
gathering process may be the gathering of information to determine the areas of
interest. Start on a broad field, then home in to specific areas of interest. Look at
previous work to see if it has concentrated on one aspect of the subject more than
another. Ask yourself why they have concentrated on these areas. Is it because they
have not had access to the primary sources, is there a genuine gap in previous
studies in this field?
In the arts it may be advantageous to begin reading the secondary
material, rather than the source material itself. Collected essays, when
they are from a range of authors, are good for identifying where the major
current debates lie in relation to the interpretation of primary material.
Use of collected essays not only allows identification of where the debates
are, but also where the gaps in the information are. Ask yourself the
question, are the topics I think important being discussed already, if not,
why not?
Example: An American edition of ‘Wuthering Heights’ was produced which
contained five essays on the book, each by a radically different critic. This style of
book has the advantage of allowing an immediate comparison of a range of
interpretations. Another useful aspect of this book was the placing of the footnotes
relating to the different approaches in the same volume as the primary text.
On the principle that recent secondary material both refers to and covers the earlier
comments, the initial reading process may, in some subjects, be considered complete
after most of the material published in the last 20 to 25 years had been gathered.
However, although the material of the last 20-25 years may cover all the major
contributions to that field, there is a risk that any idea unfashionable over that period
will be missed if an examination of earlier texts is completely excluded.




Keep an open mind at the beginning of a new project.
Try to use your initial reading to identify interesting areas of study.
Consider reading the secondary material before source material.
Collected essays (subject reviews) are better than monographs for identifying
areas of debate and gaps in knowledge.

Finally, remain flexible in your data gathering, projects can change in
emphasis as they proceed, avoid allowing your thinking and data collection to
sink into a rut.
Ensuring that nothing important is missed
Introduction
Communication with colleagues, and a thorough use of review papers and electronic
databases are vital to ensure that no major papers are missed. But researchers
should take care to ensure that information gathering is an ongoing process to avoid
finding that their publications have omitted a major paper published shortly before
the end of their project.
Points to Consider
It is essential to check reference lists at the end of recent publications.
Reviews can be particularly useful in locating hitherto unnoticed
references.
Talk to people, not only those within your own field, but also those outwith. Fields of
research tend to overlap, historians may have interesting comments on art, a
geneticist may have a good reference for a population ecologist. Conferences are a
good source, and not just the presentations themselves. One of the most valuable
aspects of conferencing is the opportunity to chat informally to others in your field.
For new researchers it is worth remembering that other researchers and staff can
benefit from your reading lists to help keep themselves up to date with the literature,
do not imagine that you have nothing to contribute.
Ideas go in and out of fashion, do not rely entirely on recent publications. Check a
few of the classical texts, these may provide references to ideas not currently in
vogue.
Check online databases to ensure that no references have been missed. These
checks should involve at least two separate databases (e.g. for medics Embase and
MEDLINE), no one database will possess all the references. Local experts, including
library staff, can provide considerable help in literature searches, particularly for the
inexperienced.
Try to identify key authors, e.g. those that most of the major texts cite, their use
when searching citation indexes can significantly reduce the chance of missing
relevant material.
As you proceed take the opportunity to present your work in
departmental/professional seminars and conferences. The feedback gained from
these ‘adventures’ is usually instructive, not only during the question period following
your presentation but in informal discussion afterwards. At the same time take the
opportunity to collect the e-mail addresses of people in your field. The regular
exchange of information and ideas is another good way of ensuring you miss
nothing. Though some care might be taken to ensure that you do not give away new
ideas before you have had a chance to develop and seek funding for them.
Perhaps one of the most likely times for gaps to appear is before the final completion
of the final project. It may have been some years since the initial literature checks, a
final run through the literature is vital at this point.






Present your work at seminars and conferences, the feedback can be
invaluable.
Use at least two databases, no database will have all the references.
Check old references, ideas go in and out of vogue.
Talk to people, and not just those immediately involved in your own field.
Remember that last literature review before completion.
Make full use of the reference lists in journals and books.
Awareness of unpublished research
Introduction
Communication with your colleagues in the field is a recurring theme throughout this
booklet, and an awareness of ongoing work is absolutely vital. Potentially a lot of the
information on who is funding what, and who is doing what, can be located on the
Internet.
Points to Consider
Talk to others in the field. It may be possible to use the Internet to keep
track of new work. However, with increasing competition between
research groups, and the inevitable parallel increase in secrecy, it is
unlikely that anything on the Net will be genuinely leading edge. Generally
the web tends to detail papers near completion or completed and awaiting
publication.
The majority of grant funding agencies publicise the grants that they have awarded.
It is possible to write to the grant funding bodies (especially the charities), and they
will provide information on the areas which they are funding. The agencies will often
supply a brief abstract that refers to novel findings, or a new hypothesis. The
funding agencies supply (at least annually) reports of their funding patterns which
delineate areas of funding, and include reports from laboratories on research grants
awarded and work undertaken (the latter may include some data). These reports can
be useful in allowing an awareness of developments in other research laboratories
with which you are not in direct contact. This is helpful because it allows you to
target resources towards areas that are currently being funded, or to stay away
because the area is ‘over-subscribed’. More importantly, some of the grants may be
very speculative, and the abstract could allow you to understand where a body of
work may be heading over the next three to five years, rather than where it was six
months ago as with published work.
Conferences, or the informal contacts made at them, are essential in this
matter. But make sure that you use new contacts to the full, keep in touch
after the event, write to them to find out what they are doing. Informal
coffee room chats can often let you know if you are not up to date.
Example: The postgraduates of one university were described as musk oxen. They
would come into the social gatherings at conference, then stand in a circle speaking
only to those that they already knew (basically themselves), making no new contacts
whatsoever. It may take a lot of courage, especially if your companions are musk
oxen, to make new contacts, but the rewards for making the effort can be very
significant.
At conference making a poster or oral presentation enhances the likelihood of
meeting other workers in the field and obtaining information from them.



Use the Internet, but do not rely on it for information on unpublished work,
especially work/ideas in early stages.
Keep an eye on to whom, and for what, the funding bodies are providing
cash.
Network (relax), do not be a musk ox.
Processes to run concurrently with initial information
gathering
Introduction
For new researchers this is a vital aspect of information gathering, and one which
must be got to grips with as soon as possible. It is easy for inexperienced
researchers, who are as yet unaware of how ridiculously long some processes take,
to fall into the trap of concentrating on early reading to the neglect of all other
activities. The advice below list some actions which you would be well advised to
begin whilst your reading progresses.
Points to Consider
The most significant processes to begin early are those reliant upon a response from
others. For a field biologist or social scientist this may be the gathering of volunteers
to help in surveys, for a laboratory based study the availability of essential materials
should be confirmed long before experimentation is due to begin. Ask yourself a few
questions. Is a reagent available commercially, can it be produced in-house? Write to
potential donors of materials (plasmids, wooden billets for weevil traps, copies of
letters). Remember to check what the price is; financial, acknowledgement, coauthorship. You need to expect some problems regardless - so always check your
source of supply. Be aware that there may be contractual obligations in receiving
material, which will usually need to be cleared through your institution’s Research or
Industrial Liaison Officer.
Similarly it is important to check methodologies during the initial reading
period, what is and what is not possible. Although methodologies may be
referred to in the literature, a precise description can be hard to come by,
authors often omitting steps which appear obvious to themselves, but
which can be less than obvious to the uninitiated.
Example: This case, from a PhD study radio-tracking woodmice, highlights some of
the above points. Whilst there were several descriptions of techniques, securing a
radio-collar onto a mouse proved more difficult than anticipated. The woodmice
proved generally unenthusiastic. Holding the mouse, tightening the butterfly antenna
using small pliers, attempting to heat shrink a rubber band to secure the radio-collar,
whilst taking care not to burn mouse or self with the miniature soldering iron, proved
near impossible. Eventually, on speaking to the authors of the earlier publications the
researcher discovered that the papers described two very different methodologies.
The method the researcher was using had a superior lightweight collar, but required
a lightly sedated mouse. The second paper used a heavier collar, but did not require
sedation of the mouse.
Are you a competent statistician, or are you going to make use of a statistician’s
services? If the latter is the case it is worth while speaking to your advisor before
beginning the work. Statisticians very definitely like to advise people on data analysis
before the data have been collected.
Essential data gathering from other bodies often requires an early start.
Not all government agencies and voluntary bodies have their data on
computer files; they might just as easily be in a series of boxes in the attic.
If this is the case, and the data are needed early in the project, it is
essential to begin making the arrangements as soon as possible.
Example: When a PhD student first sought information from the Government
agency on the distribution of mayfly in streams in North East Scotland they proved
uncooperative. Eventually, having obtained the agreement necessary to be allowed
access to the Agency database, he found that the data were not computer based
records but instead on thousands of data sheets stored in box files.








Reading can become tedious and its value decline; avoid making it your sole
occupation for months on end.
Do not wait until the reading is completed before contacting others for
help/information.
A good way to ensure a problem is to take the word of others that there is no
problem - check it out.
Contact statisticians in the early stages for advice!
Start to master the essential methodologies in parallel with the initial reading.
Not everybody has discovered computers, or has the funds to make full use
of them, you may need time to collect data.
Not all data will be of expected value, whatever people lead you to believe
they have, do not rely on it to replace your own data collection process.
Finally, if the information needed is identified as difficult to gather in the
section below - start gathering now!
Difficulties in gathering information
Introduction
The difficulties which may arise range across a wide spectrum. The mobility of a
large section of the academic community can make tracing a specific individual very
difficult, some individuals and organisations may simply refuse to help. Imagination,
flexibility and forward planning will greatly reduce the number and level of problems
encountered.
Points to Consider
The response to requests for information and help varies from individual to
individual and organisation to organisation. If you need assistance from
others, ask colleagues in the department if they have collaborated with
the organisation in the past. It may be that success/speed of response is
heavily dependent upon the person within the agency that you contact.
It is not always easy to track down authors of papers, they can prove a
mobile collection of individuals with a habit of moving between research
institutions. Personal contacts can be useful in helping you to track down
individuals. The contacts need not necessarily be people you know, new
contacts can be made as the occasion requires. Perseverance is necessary.
Example: A recent publication by the researcher will provide the postal address of
the host institution. International directory enquiries should be able to provide the
telephone or fax number for that institution. An Internet search on the institution
may allow access to the staff directory, and from that the current e-mail address for
the person in question or alternately, the head of department who may know where
the person has gone.
Do not assume that the individual will be willing to communicate in English. In order
to avoid giving offence, and in the hope that due consideration might increase the
probability of co-operation, some researchers append a comment to the effect that,
"I am writing in English because I do not speak your language, but hope that as your
publication was in English, this will be acceptable".
After tracking the person down there is no guarantee of co-operation, even if
assistance has been given to other researchers and groups in the past (see ‘ethical
constraints’).
Projects in new fields can have particular difficulties in locating
information. You may know, or at least hope, that the information is there,
but if previously tried and tested methods are non-existent, or of only
limited availability, then locating the information requires persistence and
flexibility.
Example: One of the current research programmes of a contributor considered
appetite regulation in juvenile salmon. The project was examining the newlydiscovered appetite/fat reserve-controlling hormone, leptin, in fish. In order to do
this a salmon "cDNA library" (a complete set of DNA sequences) would have been
very helpful. However, the researcher was unfamiliar with this branch of the science
and was unaware of whether such a library existed. Several personal contacts were
e-mailed to ask if they knew of the existence of such a library, or of any other
researchers who may have some information on the subject. Of the four or five
people contacted initially, none had any first-hand information, but three suggested
other contacts. One of these suggested a researcher in Newfoundland, who had
some salmon cDNA libraries, and who expressed a keen interest in the proposed
work - a potential future collaborator.






Think laterally.
Use personal contacts to build links to other research groups and sources of
information.
Persistence pays off.
Regard the collection and interpretation of data as mutually complementary
processes.
Explore a range of sources simultaneously, they may interact to give new
insight.
Be imaginative and flexible.





Some government agencies are not renowned for their speed of response:
identify your point of contact carefully.
Co-operation is not guaranteed, try to have more than one iron in the fire.
Start now, it may take some time to find the people you want.
Consider acknowledging your failure to write in the native language of the
person from whom you are seeking assistance.
There are a range of techniques available for tracking down potential sources
of advice, these include; old publications, e-mail/letter enquiries to
colleagues, Internet, International directory enquiries.
Being understood when requesting information
Introduction
Being understood is about getting the information you want, and not what somebody
else thought you wanted. Generally verbal communication is not ideal without a
written back up, in which, to avoid confusion, jargon should be kept to a minimum.
Points to Consider
The best way of being certain that a request for information is being
understood is by ensuring that your terms of reference are common. This
is most likely when the person with whom you are speaking has a good
theoretical background in your field, and when you limit the jargon used.
Example: For a student of English literature studying the role of women in 17th
century novels/plays it may be important to set your comments within the historical
context of the role of women in the 17th century. Whilst it is likely that a 17th
century historian specialising in feminist issues will appreciate your terms of
reference, it is unlikely that an historian of the same period, but working in another
discipline, will not.
Do not make a single request for the information: if there is any likelihood of being
misunderstood repeat the question, discuss it verbally, but also send the question in
written form. It may be that you feel the verbal request was clear, and that you both
understood it. But if the person on the other side of the line is not dealing with your
request for several days, or in their native language, then memory and clarity may
both fade.
One of the PhD students in the discussion group drew a line between approaching
familiar contacts, with whom students may be comfortable, and unfamiliar and/or
senior members of staff, with whom conversations may be intimidating. In the latter
circumstances a written, rather than a verbal request, would help to ensure that
there is no confusion. A written request will also reduce the risk of the researcher
being given the information that the ‘wiser’ head thought would be useful, as against
what was actually wanted.
If you need a constant supply of information/assistance from a source,
keep them informed of your activities. This will help those providing the
information to understand what you want from them. In the
circumstances described below the researcher benefited considerably from
regularly updating collaborators on his progress. In this case by reassuring
them that their concerns were unjustified. (Before deciding to forward
information, take advice from colleagues).
Example: One project, which formed part of a PhD study, required two fixed sample
lines to be located on a private reserve. Water level range gauges and dipwells were
located along the length of the sample line to provide essential hydrological and
hydrochemical data. The reserve warden was concerned as to the possible visual
impact of the equipment on the reserve. After completing a first year report, which
included a detailed description of both the work in progress, and planned future
work, a copy was sent to the site warden. Being given the opportunity to read the
report allowed the warden greater opportunity to understand the relevance to
himself of the project’s aims, which in turn made negotiating site access easier.




Make certain you are using the same terms of reference.
Do not rely upon verbal communication, send a written request - especially if
you are uncomfortable/nervous dealing with the person providing the
information.
Make certain the information you get is the information you want.
If you need assistance from an organisation/individuals, keep them informed
of your progress and future plans. They may not read the information you
send them, but they will appreciate the consideration.
Ethical constraints on the use of secondary data
Introduction
Most institutions and professional bodies will possess a code of ethics; researchers
should ensure that they are fully aware of this code. The following section considers
the rights of the providers of information, the original subjects from which the data
were obtained and the relationship between yourself and your colleagues.
Points to Consider
If information from another source has been published then it must be
acknowledged. But not only acknowledged. It is important to ensure that
the intended use of the data is acceptable to the provider. Failure to
ensure that the use of the data is acceptable to the provider may not only
result in their refusal to collaborate with you in the future, but may also
damage the research projects of others.
Example: On requesting information on the distribution of lochs containing a
particular freshwater fish from the foremost authority on this subject, the researcher
was denied this information. This was despite both the researcher and the person
holding the information having the same conservation objectives. The reason given
was that this information had previously been provided to an outside body which had
then used the information in a manner contrary to the wishes of the provider.
Not only should the proposed use of the data be acceptable to the provider, any
guarantees given to the original subjects must be borne in mind.
The need to check for permission to cite information may be partly gauged
by the environment in which the information was provided. In a formal
presentation - at conference. - it is unlikely that the author will have any
concerns with comments being cited. A casual remark is, however, a very
different affair. Correspondents will definitely be aggrieved if you publish
their work before they do.
Example: Whilst working in Ireland an experienced researcher showed a colleague
some early drafts of a paper on which she was working. A few months after moving
to Scotland she received a copy of a research proposal from her former colleague.
This proposal had taken her work and presented it as an original idea, with no
acknowledgement of her contribution. The proposal plagiarised, word for word,
entire paragraphs of the researcher’s draft manuscript. Further confirmation that the
researcher’s draft manuscript was the source of the proposal came from references
to Greek publications, a language which the researcher, but not the colleague, could
read.
Plagiarism as blatant as this is generally unusual, but is a clear example of unethical
behaviour. The best way of protecting yourself from such behaviour is to listen to the
advice of your colleagues as to who can, or cannot, be trusted. However, do not
allow the presence of a few rogues to discourage you from testing ideas out on
colleagues.





Check that the information is being used in a manner acceptable to the
provider.
Check that your use of the information does not breach the original
conditions of collection.
Ask yourself, would I provide this information, if not, perhaps you are asking
too much.
Acknowledge sources of information.
Confirm that your use of the information is acceptable to your source.
Medical consent forms are not always rigorous, and are not infrequently written to
ensure the widest possible legal interpretation. Researchers must bear in mind that
legal and ethical are not one and the same. What is legal, is not necessarily ethical
and vice versa.
Where volunteers are being used to collect data, the researcher has a moral duty to
ensure that they are acting according to the correct ethical standards. It is not an
excuse to claim that you did not know of unethical behaviour in people working for
you. You should know, and should have made it clear beforehand what was/was not
acceptable.
The following example from the experiences of Stephen Waters (Bell,
1993) provides interesting insights into the difficulties of carrying out
research on one’s own colleagues. Although the example is predominately
one of gathering the basic data, nonetheless, there are points to be learnt
regarding what happens to that data after it become secondary published
data.
Example: Stephen Waters was a teacher who decided, as part of an Open
University course, to investigate the role of his own head of department. He went
through a fairly prolonged negotiation period to reassure colleagues as to his
trustworthiness before embarking on his programme of research. Of interest here
are some of the comments he later made about the guarantees he had given prior to
undertaking the research. He had promised all participants an opportunity to verify
statements prior to production of the final report. This proved almost impossible,
most participants not having time to read the entire manuscript. Therefore, lacking
time to identify all their comments within the manuscript. He had further promised
all participants a copy of the final report, which ultimately cost rather more than
anticipated. The ethical agreement he reached with contributors was only made
verbally. This created problems at a later stage, when it transpired that none of the
contributors could precisely recall the conditions agreed upon. In retrospect he
regretted not providing them with a written copy of the agreement. However, it was
in seeking to publish the data that the greatest problems arose. All contributors had
been promised anonymity. A promise which could be met externally, nobody outwith
the school could identify the contributors. However, as all the contributors came
from the same school, it proved impossible to provide anonymity internally.






Remember that legal and ethical are not interchangeable terms.
If you use others to collect data, you have a responsibility to ensure their
ethical behaviour.
Even with your colleagues, you must earn their trust.
Consider costs (e.g. time and/or money) before making promises to
contributors.
If guarantees are required, and they are of any length, record the agreement
in writing.
Where guarantees of anonymity are provided they must be valid under all
circumstances, unless otherwise agreed.
Your sins will find you out, a cautionary tale from the world of biology.
Example: A colleague of the researcher was having trouble cutting a plasmid given
to her by a visiting worker; eventually she contacted the researcher who had
generated the original plasmid only to discover that he was not aware that the
plasmid had left his laboratory. The visiting worker had ‘acquired’ some of it whilst
working in the other researcher’s institute! This sort of thing is totally unnecessary
and entirely dishonest - most scientists are very generous and will happily give out
their plasmids/vectors.
Evaluating publications
Introduction
Do you agree with the argument, if not why not? Even if you can find no point of
agreement with the publication, does it advance your argument? How valid are the
conclusions or parallels drawn, is the basis well founded, or are the claims of the
authors specious? This section considers where less reliable information is most likely
to be found, and suggests various methods by which publications may be evaluated.
Points to Consider
Ask yourself if you agree with the conclusions in the paper, and if not, why
not?
Having decided on agreement, or disagreement, what is the value of the paper to
your own work, how will using the material help advance your argument? Even if you
do not agree with the overall thrust of the author’s argument this does not
necessarily reduce the value of the publication to your research. The material can
still be useful in providing historical or other information, regardless of interpretation.
One method of estimating the possible importance of a paper to your work is to
break the paper down into a series of fields/categories. For example: species used,
branch of subject, period of art, location of study (laboratory, field, continent). These
categories are often similar to the key words used in the paper. Each can then be
further sub-divided into increasingly specialised areas. The further down this
hierarchy the subject matter of the paper proceeds towards one’s own interests, the
more useful it is likely to be.
One topic which can be particularly difficult when starting out is the
evaluation of schools of thought. You might be attracted to the work of a
certain group. How do you evaluate their material in a way that allows you
to gain a fuller understanding of their research criteria? One way to do so
is to use the suggestions made in the example below to check the claims
being made by the group whose material is being analysed. Other
possibilities include examining collections of essays by the same group;
that way you can usually find out what their primary concerns are,
especially if the collection includes a coherent introduction. Tertiary
material is important here, i.e. reviews of the work of such groups: writing
about the writing, rather than the primary material. The final method is
conversation with others working in the same area as yourself: your
supervisor, other research students, people from other institutions. As
your confidence and experience grows it will become easier to reach a
definite conclusion about how your work relates to other research groups.
Example: Several influential collections of essays were published on Shakespeare in
the mid-1980s. One of which took, as its starting point, the structures of power in
the English American colonies, then applied the conclusions drawn from the
American situation to Shakespeare’s London, especially in the performance of
politically sensitive plays. The dramas used are Shakespeare’s historical plays
covering the career of Henry V, both before and after his succession to the throne.
The essay is famous both for its stylistic brilliance, and the issues of historiography
that it raised. This essay is usually quoted as primary text for the group of critics
known as the American New Historicists. However, its conclusions can easily be
challenged. Firstly, the fledgling American colonies were ruled by military governors,
and were subject to a form of martial law. To translate the implications of this form
of government wholesale to contemporary London is, at best, misleading. Secondly,
there are only three paragraphs in a 27-page essay on the Henry V plays, yet these
plays were, ostensibly, the main concern of the essay. Thirdly, the essay gives no
indication of how the Henry plays rework historical material that was already two
centuries old by the time Shakespeare took up his quill. The mature critique of this
essay would question its historical veracity: does the New Historicist historicise at all?
Similarly do not confuse interesting with useful or informative. One researcher
admitted that she had recently published the most boring material she had ever
read. But argued that although the paper, defining genetic code, was of a singularly
monotonous nature, nonetheless, the paper presented considerable potential as the
basis for future work.
There are times when you will feel it is necessary to reject the conclusions drawn in
a publication. However, this is not necessarily a reason to reject the paper outright.
The spin that people put on the discussion may not fit in with your idea, but at least
the data are there. It may be possible to use the data/results by commenting upon
alternate interpretations.









Do you agree with the argument, if not, why not?
How does the publication advance your own argument?
Consider how close the material lies to your own field of research.
Use reviews of the work in question to help yourself generate ideas and
evaluate the publication.
Find other work by the same school to identify their main areas of concern.
Consider the validity of the parallels drawn - realistic, or merely convenient.
How much attention is paid to the primary source, this can be lost in a welter
of secondary argument.
That the paper is a cure for insomnia does not mean it contains nothing of
intrinsic value.
Rejecting the conclusions of an author does not mean that the publication is
of no value.
Identifying the risk of inaccurate information
Introduction
It is important to consider how the information was presented, non-refereed material
inevitably carries greater risk of inaccuracy. The contributors considered inaccuracy
to be more likely when the work of an individual crossed interdisciplinary boundaries.
The final point is a warning against accepting material at face value.
Points to Consider
Do not assume that information presented informally, or at conference,
was necessarily accurate. Enthusiasm can carry researchers along,
encouraging them to leave checks and balances to a later stage. This can
necessitate a substantial revision of conclusions, or results, after the
checks have been introduced. Similarly anecdotal information, from
amateur or other local sources, can be immensely useful - or immensely
misleading. It is often difficult to judge which. Generally it is wise to check
this information out with colleagues or collaborative sources.
Example: A PhD student found it necessary to conduct an experiment examining
the potential of the ammonium ion as an activator of the enzyme sodium-potassium
ATPase in the fish gill. The first time the experiment was performed the result was
positive. An exciting result, which the student quickly passed on to the supervisor.
However, the necessary sample number for reliable statistical analysis required the
experiment to be run several times. On each, and every subsequent occasion, the
results were negative, the initial positive result had to be discounted. Unfortunately,
by this time the supervisor had told a (fortunately small) number of colleagues about
the false positive result, which they then had to retract.
Be especially careful of conclusions drawn in publications which cross
inter-disciplinary boundaries. If in doubt check with colleagues in the
relevant fields, or expand your reading to give yourself a greater insight
into those fields with which you are relatively unacquainted.
Example: A few years ago a very well known and respected writer published a
major book on Renaissance culture. The author investigated a whole host of
interdisciplinary concerns, which ranged through history, literature, and art.
However, in one example she was wrong - at least, according to specialist art
historians. She took a line on Renaissance iconography that borrowed heavily from
the classics, especially in relation to the exploitation of women. Her example was the
painting of the Rape of the Sabine Women. Despite its title, the painting in fact
depicts the Sabine women reconciling the Sabines and the Romans. The painting
examined only one incident in the story. The critic was wrong: rather than showing
women being raped, the painting shows them acting as mediators. This example is
especially useful, since the critic was trying to integrate art history, literary studies,
and the classics; her project fails because of an error in an area outside her own
immediate specialism.



Work which crosses inter-disciplinary boundaries can be especially prone to
errors.
Conference or seminar presentations should not be regarded as the new
gospel.
Consider the reputation of the source, but do not be put off by controversy.
An understanding of basic statistical principles is vital to checking the
validity of analysis. Errors in statistical analysis are surprisingly
commonplace, and new errors can go unspotted for considerable lengths
of time.
Example: Population studies that examined trends in populations over time
frequently repeat the error of temporal or spatial pseudo-replication. These studies
take the population sizes measured at the same site over a period of months as
independent data, whereas they are repeated measures.
A fair reliance but not absolute can be put on referreed papers. Even in referreed
papers there are a few pointers of which to be aware. Look for a lack of proper
explanation in the data, e.g. a band in the photograph of the gel which has not been
properly explained. Cross-reference data with earlier publications, or look for
inappropriate phrasing. It was widely agreed that one of the best indications of a
dubious paper was an over-interpretation of the results, often indicated by highly
speculative discussions.
It is important to examine the results. What are the technical limitations
of the study? Check that the claims made in the conclusion do not exceed
these limitations.
Example: A simple example of inadequate methodology invalidating conclusions
may be found in a study using a fluorescent calcium indicator to follow changes in
intracellular calcium. All fluorescent probes have a limited range over which they can
follow the intracellular changes in calcium accurately. This study used an indicator
that did not accurately follow relatively low calcium levels, and then suggested that a
significant alteration in low levels of calcium had been caused by the experimental
manipulation. The data were flawed, in this instance because of the inadequacy of
the probe to reproduce events at low calcium levels.
A parallel between the examination of results and their statistical analysis,
may be drawn with the suggestion that, in the arts, a flaw in the ‘data’
may be used to indicate dubiety in the conclusions. Sometimes even slight
inaccuracies can cause real trouble, especially if they are related to the
major research claims.
Example: An influential book on the inter-relationship between poetry and politics in
the 17th century was published in the early Eighties. However, the book was marred
by a basic historical error. The writer notes in passing that John Donne was Bishop of
London. He was in fact Dean of St Paul’s. This error may appear unimportant, but
does indicate that further checks may be wise.
The extent to which the argument stands up without the reader having to refer to
the footnotes or to other sources of supporting evidence can be a helpful initial
pointer to the strength of a secondary source. However, art history which does not
show evidence in its footnotes of having taken into account relevant original works of
art, circumstantial evidence, and written primary sources (e.g. period letters, reviews
or other texts), or which depends on citing a few which are readily available in
common secondary sources, may be suspect if it claims to offer a ‘new’ reading or
interpretation of work(s) of art/ideas. If the footnotes show that an author has made
use of archive, collaboration or other primary sources not hitherto or commonly used
in relation to the subject, this may lend credibility to claimed ‘new’ conclusions.
Though, if the footnotes reveal too exclusive or selective a range of primary sources,
however ‘original’ these may be, this indicates a need for caution.
There are exceptions to the ‘primary sources’ test, e.g. a work of mediaeval art for
which little comparable art may survive, and there may be no relevant period texts.
In this case the discussion should be able to draw on circumstantial evidence.
Circumstantial evidence may include the location where the object was found, and its
possible purpose as indicated by size, shape, ornamentation.
Evidence of reference to relevant primary, or circumstantial sources, is not a
sufficient test if the argument constructed has gaps, or contradicts itself, and makes
no effort to account for these.
Close attention to primary material is absolutely necessary when
considering major claims to originality. When dealing with secondary
material which makes such claims it is always wise to assume a posture of
extreme scepticism.
Two Examples: (i) In the 19th century the Scottish painter William Dyce painted
the ‘Highland Ferryman’. This painting was assumed, in the standard secondary
literature, to have been painted from nature when Dyce was in Scotland in 1857 and
living on Deeside. However, the ferryman and surrounding scenery were out of scale,
indicating that the painting may have been a composite of a photograph and a
model in the artist’s studio. The researcher then sought evidence of the photograph.
But the evidence uncovered contradicted both earlier hypotheses, indicating that the
painting was neither a real life scene, nor a composite. George Washington Wilson (a
photographer of the period) described a photograph he had taken which exactly
matched the painting, and which indeed, Wilson claimed was the basis for the
painting. The conundrum was eventually solved by looking at the problem in reverse.
In this case examining the source not of the painting, but the photograph. This was
the origin of the composite, a photograph created from two or more negatives.
(Willsdon, 1990)
(ii) One student once wrote in considerable detail about the violet colour scheme in
a Monet painting, not realising that the violet was an artefact of a second rate
photograph.





Check that all available evidence was properly considered, that the argument
presented was not overly selective of the data.
Check the data upon which the publication was based were valid.
Check that the statistical analysis upon which the publication was based was
valid.
Confirm that the methodology and data could provide the results claimed.
Check secondary material against the primary source.
Current ideas and knowledge are not the only useful sources of
information In the example below another researcher used the knowledge
base and views of an earlier period because they offered a more relevant
guide than a modern source to the interpretation of painting completed in
1900. In this instance the researcher sought out, not what secondary
material was now available, but what secondary material had been
available at the turn of the century.
Example: Although referring to recent published history may give a summation of
research to date on the signing of the Magna Carta, it may not indicate how much of
this information would have been available to the artist who portrayed the subject at
the Royal Exchange in 1900. The researcher consulted history textbooks in use at
the turn of the century to identify the views and knowledge which were current at
that period with regard to the signing of the Magna Carta. These revealed that there
had been, at that time, considerable interest in this event as a key foundation, not
just for the modern parliamentary system, but in particular for London’s historic
system of government. In combination with other contextual evidence, this finding
led to the conclusion that the subject composition and imagery, of this, and other
Exchange murals, were intended as ‘propaganda’ for the preservation of London’s
traditional government in the face of threats to its survival from the new London
County Council (Willsdon, 1997).
Consider reversing the usual conclusions and looking at the material ‘back
to front’.
Example: In this case the researcher wanted to consider the perspective of the
people for whom the mural, Gustav Klimt’s Beethoven Frieze (1902) had been
produced, rather than her own late 20th century perspective. This arose from a
feeling that the ‘standard’ interpretation of Klimt’s frieze, as an illustration of words
from Schiller’s ‘Ode to Joy’, was too simplistic. The standard explanation did not
allow for the frieze’s wider cultural context, turn of the century Vienna. The
researcher therefore explored the extreme (null) hypothesis, that the frieze had little,
or only limited, connection with Schiller’s ‘Ode’. The researcher looked at the frieze
not from a literal standpoint, which had assumed that because part of it had been
exhibited by Klimt with a label quoting Schiller’s ‘Ode’, it must illustrate the ‘Ode’.
Rather the researcher considered the frieze from the much larger perspective of
Klimt and his contemporaries, who were familiar with a complex framework of
literary, musical and philosophical reference no longer shared by modern viewers
(Willsdon, 1996).
All contributors were unanimous in their distrust of unrefereed material on the
Internet. Do not be put off an idea merely because it is controversial, but exercise
extra caution when dealing with unrefereed papers.
Another way of ensuring that conclusions are within a reasonable margin
of error is to utilise a recognised standard, or set of standards. In artsbased subjects this can be difficult, but there are possibilities. Use of an
internationally accepted standard, such as the ‘Oxford English Dictionary’,
can be an invaluable tool in deciding how acceptable the conclusions
might be. This kind of approach can lead to a whole series of analyses
which are verifiable, and which may even challenge accepted thinking.
Example: There is a famous line in Shakespeare’s Sonnet 18, which includes the
phrase "the darling buds of May". Conventionally, this has been taken to be a nice,
sweet reference to the kinds of things that Shakespeare is assumed to mean: the
nice English countryside, romantic love (e.g. A Midsummer Night’s Dream). However,
a simple look at the Oxford English Dictionary shows that "bud" is Renaissance slang
for breast, and that "darling" was a kind of apple sold on the streets of London in
1598 - a Renaissance Granny Smith’s. Breasts have obvious sexual connotations, as
does the apple, the ultimate sign of female sexuality and duplicity in western
patriarchal religious discourse: Eve in the Garden of Eden. "May" refers
simultaneously to the Virgin Mary and the month of May, conventionally associated in
Renaissance iconography with country matters, especially sexuality. The inescapable
conclusion is that the meanings of modern sweetness are unacceptable as the only
possibility if one wishes to place these poems in their historical context.




Check that interpretations are relevant to the context of the times.
Scepticism should be adopted at all times.
Received wisdom is no guarantee of accuracy, consider alternative
interpretations of the data.
Check statements against internationally recognised standards.
Bias, and how to avoid it
Introduction
There can be no question that we all bring some degree of bias to whatever we
read. The issue is less can we avoid bias as how to recognise it and ameliorate its
effects. The following section provides a few suggestions.
Points to Consider
It is difficult to avoid this problem, perhaps impossible, was the comment of several
optimists on the discussion groups. It was suggested that the cultivation of extreme
scepticism, even towards publications with which you agree, is a step toward the
goal of eradicating bias. This is difficult to do, but once you begin to question the
literature that once you agreed with, then you stand upon the threshold.
Try turning your research idea around. If you believe that your methodology
produces the correct results take the opposite point of view. Consider the world from
the new perspective. My methodology cannot produce valid results, only this
alternative technique can.
Try to be inclined towards work which does not fit your favoured world view, attempt
to counterbalance by trying to adopt a positive approach to papers which run
contrary to your expectations.
If data is contrary to your own, approach the laboratory in question and ask to swap
antibodies or material with them to re-run your/their experiments, but not all
institutes will agree to this procedure.
Finally give the publication some thought at a later date. There are few of us who
can listen to an argument or read a paper then alter our ‘world view’. Most of us can
alter our views on reflection. If there is an author with whom you strongly disagree,
try going back to him/her in a few weeks time.
World views change, that which is acceptable today may well be ridiculed
tomorrow, try not to let convention determine your outlook.
Example: In botany, circa 1850, the general morphology of the root was described
as having three layers to the vertical structure. This was repeated in text books until
1950, when someone announced that he could not actually see this structure down
the microscope. This picture had been drawn, redrawn and photographed for around
a hundred years.
Remember Darwin/Wallace, Mendel or Heinrich Schliemann: in each case their work
was controversial, forgotten, or considered somewhat eccentric.





Develop healthy scepticism, try to get into the habit of questioning even that
material with which you agree.
Consider the opposite view: my way cannot work, but theirs can.
Can you swap material with those you disagree with?
Give it time, think it over.
Remember yesterday’s eccentric/heretic may be tomorrow’s hero.
References
Books and other publications
Bell, J. (1993) Doing Your Research Project: A Guide for First-Time Researchers in
Education and Social Science. Open University Press, Buckingham. 176 pp.
Brown, S., McDowell, E., Race, P. (1995) 500 Tips for Research Students. Kogan
Page Ltd, London. 127 pp.
Gealy, N. and Clarke, D. (1998) Development of an Interim Workplan for the
Researcher’s Lead Body. Maloney and Gealy, 24-26 Mossbury Rd. London. 30 pp.
Gealy, N., Westlake, D., & Clarke, D. (1997) Draft Occupational Standards In
Research. Maloney and Gealy, 24-26 Mossbury Rd. London. 59 pp.
Skelton, F. and Walker, L. (1995) Pilot Study to Assess the Benefits of Gathering
Evidence of Research Competencies for PhD Students to Improve Their Subsequent
Employability. Glasgow University. 21pp.
Willsdon, C.A.P. (1990) ‘Dyce in camera’: new evidence of his working methods.
The Burlington Magazine. Vol. 132, No. 1052, Nov. 1990, p 759-765..
Willsdon, C.A.P. (1996) Klimt’s Beethoven Frieze: Goethe, Temelkunst and the
fulfilment of wishes. In Image: Music: Text. Eds. M. Pointon and P. Binski. Art
History book issue. Vol. 19, March, p 44-73.
Willsdon, C.A.P. (1997) The Mural Decoration at the Royal Exchange in The Royal
Exchange. Ed. A. Sunders. London Topographical Society. p 311-335.
Glasgow University Library Publications
Ashworth, S.(1998) Starting Research in Engineering. Glasgow University Library
Guide.
http://www.lib.gla.ac.uk/Docs/Research/engineering.html
Faichney,A.(1998) Starting Research in Life Sciences. Glasgow University Library
Guide. http://www.lib.gla.ac.uk/Docs/Research/lifesci.html
Faichney, A. (1998) How to use JCR (Journal Citation Reports). Glasgow University
Library Guide. http://www.lib.gla.ac.uk/Resources/Databases/jcrs.html
Marlborough, H.S. (1998) Introduction to Database Searching. Glasgow University
Library Guide. http://www.lib.gla.ac.uk/Docs/Guides/searching.html
Moore, J.N. (1998) Starting Research in Psychology. Glasgow University Library
Guide. http://www.lib.gla.ac.uk/Docs/Research/psychology.html
Worlledge-Andrew,H.(1998)Starting Research in UK Law. Glasgow University
Library Guide. http://www.lib.gla.ac.uk/Docs/Research/uklaw.html
Appendix A - Some Abstracting And Indexing Journals
Introduction
As a start, connect to BIDS at http://www.bids.ac.uk, which contains a selection of
databases, including the multidisciplinary group of ISI Citation Indexes. Click on the
name of the subject database you wish to search. Other databases are available via
sources such as EDINA or OCLC . A password supplied by library staff will usually be
required. Once the password is issued you may be able to access information from
outside the library or off-campus. Note that the popular BIDS-ISI service will be
superseded from 2000 by the same data as ISI Web of Science.
The following selected list is by no means exhaustive:
Arts/History/Languages/International








ABELL:Annual Bibliography of English Language and Literature. Print from
1920.
ABREES: Abstracts Russian and East European economic affairs.
Arts and Humanities Citation Index from BIDS-ISI Citation Indexes,1981
British Humanities Index- print or database.
Hispanic American Periodicals Index, in print and database (from EUREKA)
Historical abstracts-print from 1955, and database.
MLA International Bibliography- print or database, covers literature,
languages, linguistics
Periodicals Contents Index- database as PCI from EDINA. Indexes range of
arts journals from nineteenth century onwards.
Economics/Politics/Sociology/Anthropology








Applied Social Sciences Index and Abstracts, printed version, (database
available as ASSIA Plus).
International Bibliography of the Social Sciences-covers economics, sociology,
politics and anthropology.-in print- or from BIDS as IBSS.
International Development Abstracts, in print , and online in Geobase CDROM.
International Political Science Abstracts, available since 1951 in printed
version.
Journal of Economic Literature, available in paper copy since 1969 and on
CD-ROM as Econlit.
Social Sciences Citation Index from BIDS-ISI
Sociological Abstracts (Good international coverage), in print or database.
Sage Urban Studies Abstracts, available since 1987, printed version only.
Education


ERIC- Educational database, U.S. oriented.
British Education Index, available since 1954, printed version is solely an
index, citations give no summary of content. As database, also available
within International ERIC.
Engineering/Mathematics







Applied Mechanics Reviews, available since 1948 in printed version. (Review
articles with extensive bibliographies of the literature).
Aqualine Abstracts. Print or database
(World’s scientific literature on water {but excludes aqua vitae and uisga
beatha})s
Engineering Index in print = Compendex database available through
BIDS/EDINA
Index to Scientific and Technical Proceedings available via BIDS-ISI Citation
Indexes
INSPEC. Database via BIDS/EDINA or CD-ROM. Combination of print indexes
Computer and Control Abstracts/Electrical and Electronics Engineering
Abstracts.
Mathematical reviews, available on paper, or online as MathSci(Net)
MathDatabase/ Zentralblatt fur Mathematik/ Mathematics Abstracts.
1931onwards. See http://www.emis.de/cgi-bin/MATH/
Law






Current Law:UK up-to-date printed digest of cases and legislation
Current Legal Information, electronic version of Current Law, Daily Law
Reports, UK Legal Journals Index.
European Current Law: monthly digest (Both E.U. and national levels, digest
of legislation and cases, tables of competitive decisions).
Index to Foreign Legal Periodicals database available via EUREKA.
Index to Legal Periodicals available via OCLC. (Mainly U.S. and Canada).
Lawtel -UK collection of legal databases including cases and legislation.
Management/Business






Business Periodicals On Disk (BPO), on CD-ROM since 1987.
ABI/Inform database. ABI is database of first choices for
management/financial administration.
Institute of Management International Databases Plus (IMID) available on
CD-ROM
International Directory of Company Histories -print-available since 1988.
OneSource UK Private+. 10 years data on 110,000 companies - CD-ROM.
database.
Wilson Business Abstracts (Mainly U.S. business magazines and trade
publications).
Music
(Helpful titles in tracing music)
 The New Grove Dictionary of Music and musicians.
 WorldCat, a catalogue of records submitted by libraries worldwide from OCLC
databases.
 Popular Titles and Subtitles of Musical Compositions
 Music Index, print and database
Theology/Philosophy

ATLA Religion Database- in print as Religion Index

Philosophers Index-print and database.
Science/Health (Human and otherwise)










Astronomy and Astrophysics Abstracts, available since 1969, print version.
Biological Abstracts , printed, since 1926. On-line via EDINA or CD-ROM as
BIOSIS.
CAB Abstracts. In print - 29 individual abstract journals in agriculture, animal
Chemical Abstracts available since 1907. Print and database. Covers
biochemistry, microbiology, pharmacology, chemistry.
Geobase is recommended database for Geography, Georef for Geosciences .
Index Medicus, varying print titles from 1879. On-line as MEDLINE since 1966
, covers medical dental, nursing and veterinary material - U.S. bias.
Physics Abstracts, in paper form only from 1964, thereafter available as part
of INSPEC database.
Science Citation Index available on BIDS-ISI
Psychological Abstracts as printed version. Database as PsycLIT for coverage
of books (1987-) and journal articles(1887 onwards).
Zoological Record, available since 1864 in print , or CD-ROM database.
Appendix B - Search Engines And Web Sites
Kenny Forte, Paisley University. (June 1998)
British Academic Related Websites
National Information Services and Systems
http://www.niss.ac.uk/
Comment: Good for keeping up to date with current HE events.
 HE institutions & conferences
 Museums
 Bookshops
 Newspapers
 magazines
BUBL Information Service
http://bubl.ac.uk/
Comment: Good starting point when beginning a new search for information.
 journal abstracts
 catalogue of Internet resources arranged in subject order
 banks, government departments, charities, hospitals, churches
 UK registered companies
 UK web directories (G.O.D. UK, Hensa, Lifestyle UK, UK Directory, UK Index
Ltd, UK Plus, Yahoo UK, Yell (Electronic Yellow Pages)).
General WWW Search Engines
Information about the major search engines can be found here:
www.searchenginewatch.internet.com/facts/major.html

Currently the most popular search engine, as rated by professional
researchers, is AltaVista (used for 45% of searches). HotBot came in at 20%,
and Excite and Infoseek both rated 14%. The source for this information is:
http://searchenginewatch.internet.com/resources/reviews.html
 Search engine tutorials (articles and resources to help you better use search
engines to find what you are looking for):
http://searchenginewatch.internet.com/resources/tutorials.html
Dogpile http://www.dogpile.com/
Comment: Very good site since it searches other search engines for you. Can be very
slow, works best in the morning, as users in the U.S.A. are not yet on-line.
Internet resources searched by Dogpile:
 Usenet: Reference, Dejanews, AltaVista and Dejanews’ old Database.
 FTP: Filez and FTP Search. (Only the first word will be passed on to FTP
Search.)
 Stock Quotes: Enter Tickers or Company Name
 Business News: Search for Business News. Africa News, Agence France ,
M2 Airlines, Asiainfo, Business Wire , Canadian Corp , Content Factory,
Fednet, Infolatina, Inter Press, Interactive Sports, Itar-Tass, M2, Phillips, PR
News, PIO, Resource News, SABI, UPI, UPI, US Newswire, Washington Tech,
WENN, Xinhua
 Other News Wires: Yahoo News Headlines, Excite News and Infoseek
NewsWires.
Excite
Alta Vista
HotBot
InfoSeek
Yahoo
Eureka
FTP Search
Global Online Directory
Lycos
Searching for Softwareg2057
www.tucows.cableinet.net
Comment: Good fast site for freeware and shareware software
Subject Gateways
These are UK-produced collections of quality evaluated websites aimed at the
academic community.
ADAM- Art, Design, Architecture & Media Information Gateway
AHDS- Arts and Humanities Data Service
Biz/ed -Business and economics on the Internet
EEVL-Edinburgh Engineering Virtual Library
OMNI- Organising Medical Networked Information
SOSIG- Social Science Information Gateway
Evaluation of Information Websites - A Selection
Resources on Evaluating Information on the Internet
Evaluation of information sources(VirtualLibrary)
Evaluating Library Resources