Handling the Literature
Prof Carole Goble
carole.goble@manchester.ac.uk
COMP80122
28 January 2015
You will read a lot of papers
Most will be irrelevant
Many will be poor
Some will be important
A few will change your life
You will find lots of papers or none
Find, Manage, Read, Synthesize ….
Purpose: Announce and Convince
Defend results are plausible or
correct and method convincing
and repeatable.
Review & Learn Verify the results
empirically. Trust. Understand.
Convince, comfort, credibility.
Reuse Use the explained and
trusted results (data, method) for
new / my science on demand.
Compare. Extend.
Is it “true”?
Can I repeat it?
Am I convinced?
Is it plausible?
Can I reproduce it?
Can I use it?
Is it a useful contribution?
Scholarly Communication
Forms
• Making an impact
– Demo, Magazine articles: reviewed
– Blogs, twitter, forums: unreviewed
– Technical reports
• Proposing an idea or view
– Position statement, Commentary,
Perspectives, Magazine Department,
Doctoral Consortiums
– Highly cited, editorialised, low rigour,
established figures
• Presenting a preliminary research
finding, on-going work, ideas, small
extensions to existing work,
• Short paper, workshop paper, poster
• Medium rigour, peer review
Scholarly Communication
Forms
• Presenting a research finding
• National Conferences:
– new ideas/applications/tools,
medium extensions, more serious
reviewing
• International Conferences:
– mature work, serious reviewing, but
time-constrained, check track
• Journal article:
– lots of mature work (e.g., 2
conference papers into 1 journal
paper), serious reviewing, not timeconstrained
– High rigour, peer-review
Additional material
•
•
•
•
•
•
Conference papers means a presentation
Slides, videos
Web pages
Blogs
Technical reports
Other?
More forms
•
•
•
•
•
•
•
•
Position paper
Systems paper
Theory paper
Vision paper
End-to-End paper
Surveys papers
Summary papers
…
• Deep and narrow
• Broad and shallow
Salami paper writing
Challenge: rebuilding a body of work
Finding and choosing papers
• Key players and key
papers everyone
cites
Finding and choosing papers
• Special repositories / libraries
Lab Note books
• Use one. A book. Or electronic.
• A wiki? A Blog?
http://www.atriumresearch.com/html/eln.htm
Finding and choosing papers
• Citation management
Managing your references
• Use a reference management system
http://en.wikipedia.org/wiki/Comparison_of_reference_management_software
Places to find tools
https://www.force11.org/catalog
Utopia Documents: http://getutopia.com
Kristian Garzi
a PhD student speaks about
tools
I can’t find any papers!
• Adjacent fields?
• Different terminology?
• Related topics?
Synthesize: beyond Shopping Lists
• Annotated Bibliography
• Literature review framework
– Categories
– Clusters
– Timelines
• Comparisons on aspects
– Cross cutting the papers
Mind-mapping tools can help
• http://cmap.ihmc.us/
• http://www.mindjet.com/uk
/mindmanager/
• http://freemind.sourceforge
.net/wiki/index.php/Main_P
age
• http://www.biggerplate.co
m/
http://www.biggerplate.com/mindmaps/NwUuYpZx/critical-literature-review-template
Smart Reading
•
•
•
•
•
•
•
•
•
•
•
Understand the context of the paper.
Beginnings and endings.
Survey the structure.
Use figures and tables. Generate them if absent.
Decide when to read every word.
Summarise the paper.
Read out loud.
Explain the paper to your cat.
Multiple reading passes. Over time.
Set aside time to think about it and digest it.
Get help. Talk to people. Set up a reading group.
Where does it fit with my work
•
•
•
•
Is it relevant? If not why not?
How does it fit with your framework?
Yes – you will need a framework!
Can you relate the terminology and notation
to yours?
• Keeping a record of the contribution.
When will the paper become
relevant?
• Over time
• An ongoing framework
• Revisiting
Hints for Reviewing Papers
•
•
The answer to each question tells you something about the technical content of the paper
The ease of extracting the answer to each question tells you something about the quality of the
writing.
Questions
• Is this a vision/position/direction paper, or a measurement/implementation paper?
• If you know the area well, can you mentally slot this paper somewhere in the taxonomy? ("Differs
from X as follows; has the following in common with Y;" etc.) If the paper is radically brilliant, new,
or iconoclastic work, this question may not apply.
• Can you summarize the single most important contribution in one or two sentences?
Issues
• Will this advance the state of the art?
• Did you learn anything new?
• Does it provide evidence which supports/contradicts hypotheses?
• Experimental validation?
• How readable is the paper?
• Is the paper relevant to a broader community?
Goals of Review
• Guide the program committee in selection process
• Help authors (to revise paper for acceptance, to understand rejection, to improve further research
and future projects)
John Ousterhout's Hints for Reviewing Papers
Make yourself a template
•
•
•
•
•
•
•
•
•
Author housekeeping stuff
Paper genre
Problem statement/motivation
Key ideas
Technical contribution
Technical flaws
Evaluation
Presentation
Comparison
– To authors’ other work
– To third party’s work
– To your work
•
•
•
•
•
How would I extend this paper?
What questions does it raise?
Future work of author.
What else?
Author log
Make yourself a template
•
•
•
•
•
•
•
•
•
Author housekeeping stuff
Paper genre
Problem statement/motivation
Key ideas
Technical contribution
Technical flaws
Evaluation
Presentation
Comparison
– To authors’ other work
– To third party’s work
– To your work
•
•
•
•
•
Well-established class of problems,
e.g., FO theorem proving, image
retrieval etc.
Novel class of problems (is it really
new?)
Single problem or many problems
How would I extend this paper? Implicit or explicit new way of thinking
about a problem?
What questions does it raise?
Future work of author.
What else?
Author log
Make yourself a template
•
•
•
•
•
•
•
•
•
Author housekeeping stuff
Paper genre
Problem statement/motivation
Key ideas
Technical contribution
Technical flaws
Evaluation
Presentation
Comparison
– To authors’ other work
– To third party’s work
– To your work
•
•
•
•
•
How would I extend this paper?
What questions does it raise?
Future work of author.
What else?
Author log
Implicit or explicit new way of doing
things?
New, i.e., developed by authors?
existing?
Developed by authors or others?
New combination of existing techniques?
Good or better/worse than X and why
Make yourself a template
•
•
•
•
•
•
•
•
•
Author housekeeping stuff
Paper genre
Problem statement/motivation
Key ideas
Technical contribution
Technical flaws
Evaluation
Presentation
Comparison
– To authors’ other work
– To third party’s work
– To your work
•
•
•
•
•
How would I extend this paper?
What questions does it raise?
Future work of author.
What else?
Author log
What is the author's thesis? What are they
trying to convince you of?
Summarize the author's argument.
How does the author go about trying to
convince you of the thesis?
Make yourself a template
•
•
•
•
•
•
•
•
•
Author housekeeping stuff
Paper genre
Problem statement/motivation
Key ideas
Technical contribution
Technical flaws
Evaluation
Presentation
Comparison
– To authors’ other work
– To third party’s work
– To your work
•
•
•
•
•
How would I extend this paper?
What questions does it raise?
Future work of author.
What else?
Author log
Does the author describe other work in
the field?
If so, how does the research described
in the paper differ from the other work?
Make yourself a template
•
•
•
•
•
•
•
•
•
Author housekeeping stuff
Paper genre
Problem statement/motivation
Key ideas
Technical contribution
Technical flaws
Evaluation
Presentation
Comparison
– To authors’ other work
– To third party’s work
– To your work
•
•
•
•
•
How would I extend this paper?
What questions does it raise?
Future work of author.
What else?
Author log
Empirical (run tests): test suite and testing
must match problem targeted
Theoretical: correct and
understandable/convincing and relevant
Does the paper succeed?
Are you convinced of the thesis by the
time that you have finished reading the
paper?
Make yourself a template
•
•
•
•
•
•
•
•
•
Author housekeeping stuff
Paper genre
Problem statement/motivation
Key ideas
Technical contribution
Technical flaws
Evaluation
Presentation
Comparison
– To authors’ other work
– To third party’s work
– To your work
•
•
•
•
•
How would I extend this paper?
What questions does it raise?
Future work of author.
What else?
Author log
Does the author indicate how the work
should be followed up on?
Does the paper generate new ideas?
What are your tips?
Reading groups
Printing and physical markup
Not printing and electronic markup
Read everything 3 times