medium.com /a-academic-librarians-thoughts-on-open-access/4-ways-to-find-review-papers-systematic…
4 Ways to find review papers, systematic reviews,
meta-analysis, and other rich sources of references
— 2D Search templates, Connected Papers & more.
Aaron Tay ⋮ 12-15 minutes ⋮ 03/06/2021
Starting your research in a totally new area and unsure where to start? Wouldn’t it be
nice if someone had done the work for you already, surveying the research landscape of
papers of importance and interest in the area you are interested in and even commenting
and evaluating on the results?
Sounds like a pipe dream? Not really. Find one of the following types of research and
things becomes tons easier as they bring together citation rich resources on a specific
area as assessed by an expert in the area. Say the concept you are studying is “Servant
leadership”. What type of items would help you quickly gain an understanding of the
literature written on that concept?
Review articles (sometimes known as survey articles, they are more narrative
reviews and have less formal inclusion critera) e.g. Servant leadership: A review and
synthesis
Systematic reviews (similar to review articles but tend to be more methodical in
setting out how studies are found, screened and selected) e.g. A systematic literature
review of servant leadership theory in organizational contexts
Meta-analysis (Systematic reviews + pooling together results/data from multiple
studies for additional analysis) e.g. Do ethical, authentic, and servant leadership
explain variance above and beyond transformational leadership? A meta-analysis
Phd and Masters Theses (Theses tend to have very comprehensive reviews) —
Some examples
Bibliographies (References lists often annotated with comments on a certain topic)
— e.g. The Hine Bibliography of Resources on Servant Leadership
But these citation rich items don’t exactly grow on trees. So how can you find them?
Here are the automated methods I know of (in order of most known to lesser known
methods). There are also various more manual ways like hand searching journal titles
(some fields will have particular outlets that publish such content e.g. Annual Reviews of
… ), conference proceedings that aren't covered here of course.
1. Keyword Search in Google Scholar
2. Search within a database with a suitable Subject or Publication type
3. Structured search template in 2D search (NEW!)
4. Citation mapping tools like Connected Papers, CoCites. (NEW!)
In my examples below I am going to use “Creativity” as a sample topic.
The accuracy (or rather precision and recall) of these different techniques to
identify reviews etc differ depending on the technique used. Simple minded
boolean string matching is likely to have higher recall (less false negatives) but
lower precision (more false positives), while with machine learning techniques
which are increasingly used , depending on how the training is conducted it may
lead to higher precision (less false positives) at the cost of lower recall (more false
negatives) if trained that way. Lastly of course there is simple human curation or
hybrid human+ML may get the best of both worlds, but human input is expensive
and is unclear how much this is used today.
1. Keyword search in Google Scholar
There isn’t much to say about this. To find a systematic review, just go to Google Scholar
and enter your keyword <topic keyword> systematic review.
This generally works because most reviews and systematic reviews have those words in
the title, and Google Scholar generally prioritizes matches with titles. e.g. “Determinants
of organizational creativity: a literature review”
Searching for reviews, meta-analysis and bibliography in Google Scholar
Still you can try a more refined search like intitle:review <topic keyword> to see if it gives
you better results. e.g. intitle:review creativity.
Or even combine terms with boolean like this.
creativity intitle:review OR intitle:meta-analysis OR intitle:bibliography
Sidenote: Interestingly, China’s Baidu Xueshu (百度学术) / Baidu Academic or their
answer to Google Scholar, does have a automated function to try to find review papers
under their analysis function
Analysis function in Baidu Academic that surfaces review papers
It also tries to find “classic papers” (seminal papers), recent works and theses. I’m still
trying it out, while Baidu Academic seems to have a smaller index than Google Scholar, it
does seem to be in the same ballpark as Microsoft Academic, Lens.org and acts as a big
web scale cross disciplinary database that should be sufficient for most needs. And yes,
it covers the usual international journals.
2. Search within a database with a suitable document
type or subject type filter
Of course the limits of using keyword searching in Google Scholar is that you are going to
get a lot of irrelevant results past the first few results even with the refined search syntax
above.
A database which has a filter for reviews, meta-analysis etc would take the guess work
out of it wouldn’t it? Indeed some databases do have such a filter, typically as a
publication type filter.
TIP : In fact, almost all the methods below do not give 100% accuracy because
the ‘reviews’ are often identified using complicated search filter patterns (‘Hedges’
in Pubmed speak) or Machine learning (Microsoft Academic, Semantic Scholar)
using methods akin to #3 and to to some extent #4 below.
Take PubMed, where there is a filter for “Reviews”, “Systematic reviews” and “Metaanalysis”.
Creativity in Pubmed filtered to Review, Systematic Review and Meta-analysis
Unfortunately not many databases have such a filter, most are on the life sciences side
(e.g. Psycinfo).
Sometimes, a database might not have a outright filter for such items but a somewhat
hidden way to do such searches is available if they support controlled vocabulary and the
controlled terms includes a subject for reviews or bibliographies.
This means someone has helpfully classified these items and all you need to do is to
search with these subjects.
TIP : In some vocabularies, a controlled term for say “Systematic reviews” might
be use for articles/content written on the theory and practice of doing systematic
reviews, rather than for labelling the item as a systematic review (i.e. as a
publication type), while others may include both. You can check by running the
search and looking at what appears.
One example is ERIC the education database, and in their thesaurus you can see
Literature Reviews, combine that with your topic and you can do a search for literature
reviews!
Creativity + Literature reviews in ERIC
Again while this is useful, a cross-disciplinary database that has such controlled terms
would be useful.
Of the Cross-disciplinary databases Scopus and Web of Science do indeed have filters
for this. For example, you can use the Document type “Review” for Scopus and Web of
Science.
Document type — Review in Scopus
REVIEW in Web of Science
However both databases are not only pay to access but also have selective coverage.
Another big Cross Disciplinary index that has a review filter is Semantic Scholar, though it
shares a lot of the same underlying data with the next entry below via a data partnership.
https://www.semanticscholar.org/
Is there one with as broad a coverage as possible (competitive with Google Scholar)
AND some way to filter this way?
And indeed one exists — Microsoft Academic .
As noted in this article, Microsoft Academic is one of the largest sources of academic
content out there and they use NLP and Machine learning to auto-classify over 200
million pieces of content into subjects including “Systematic review”, “Review article”,
“Meta-analysis” and “Bibliography”
So do searches like
Creativity Systematic Review
Creativity Review Article
Creativity Meta-analysis
Creativity Bibliography
A slightly more advanced technique would be to combine them all together with Boolean
but as Microsoft academic doesn’t support Boolean, you can do this in Lens.org which
has Microsoft Academic data and does support Boolean. However, I recommend you
look at the next method first before trying it out.
3. Structured search template in 2Dsearch
As noted above, the PubMed filter for reviews is actually a complicated search pattern to
try to identify reviews.
Search Strategy Used to Create the PubMed Systematic Reviews Filter
Can we do the equalvant for a Cross disciplinary database like Microsoft Academic or
Lens.org?
Indeed we can. Using both methods employed in #1 and #2 we can create a complicated
Boolean search in Lens.org that maximizes the recall of the search (so we can get as
many correct items) while also keeping the precision of the search high (so we can avoid
wrong results).
Why Lens.org? It is one of the largest open sources of academic records out
there (>200 million as of 2020), including data from Crossref, Pubmed, PMC,
JISC CORE, Microsoft Academic and more AND unlike other large search
indexes has a very powerful Boolean and search function that allows you to craft
a powerful search strategy to maximise recall and precision. For example the
search template is powerful enough to include a section to search within specific
journal like Annual reviews or Cochrane Database of Systematic Reviews
Due to the complexity of the search, I employ the use of 2Dsearch to create a search
template for your ease of use.
Saved search template
I detail the method in — Finding reviews on any topic using Lens.org and 2d search — a
new efficient method. You should really read it, but the cliff notes is as follows.
1. Go to the saved 2D search template here (no sign in required)
2. Scroll to the bottom and locate the block in the search strategy that is on “creativity”
and change the terms to your topic (e.g. “Servant Leadership”)
Edit the part of the box that says Creativity with your topic keyword
3. Generate the search results in Lens.org
Some results generated by Lens.org
4. Enjoy!
That’s it.
4. Citation mapping tools like Connected Papers,
CoCites.
I’ve been tracking the rise of tools that are designed to help literature review by mapping
the literature. However while tools like Citation Gecko and CoCites are useful and have
the potential to identify review or seminal papers, they generally do not outright try to
identify them.
The only exception to this is the new Connected Papers tools. I’ve done a long review
here but in this piece I will focus on the relevant portions.
Connected Papers, allows you to put in one paper and it will generate a map of similar
papers using a similarity function based on a combination of co-citations & bibliometric
couplings.
For example take this 2008 paper of mine — Improving Wikipedia Accuracy: Is edit age
the answer? and throw it into Connected Papers to see what it generates.
Connected Papers generated network for the paper
The twist here is that you can use the papers found in this generated network to try to
identify not just seminal works by clicking on “Prior works” button but also try to “find
surveys of the field or recent relevant works which were inspired by many papers
in the graph” by clicking on the Derivative works.
Derivative Works generated
While this method doesn’t always work, it works a lot more than I expected and you get
the added bonus of finding similar pages as well as seminal papers on top of review
papers!
Methods compared roughly
At this point you may be bewildered by the methods available and wondering which to
use. There is no one sized fit all answer, it depends on how comprehensive you want the
search to be and how long you want to do the search.
In general, the fastest and quickest way would be to use keyword searching in Google
Scholar, it won’t be the most complete, but it tends to give you good results on the first
page. If you are in a discipline where there is a dominant database like PsycInfo, then
you should definitely use that as well.
If for some reason you only want to find such items in a restricted set of “high impact”
journals, you can use the filter functionality in Web of Science and Scopus, though there
is no reason I see why you might do that.
Semantic Scholar’s review publication type seems to give me more mixed results, and
while Baidu Academic analysis function can help surface some relevant China paper
missed by other methods, I find like Semantic Scholar it can be a bit hit or miss in
classifying review papers.
The saved 2D search template here is something I recommend and use when I want to
be comprehensive, it tends to give me the best balance between precision and recall and
works well in my experience for most disciplines.
Connected Papers isn’t particularly geared for this purpose , though it’s other functionality
like mapping out related papers and seminal papers is helpful.
If you want to do a comprehensive overview for a thesis say, I would try all of these
methods! Some might find additional relevant items missed by others.
Conclusion
I hope the techniques shown here are of use to you. Once you have found all these
review papers, systematic reviews, meta-analysis, bibliographies what do you do next?
Stay tuned for my next post!