EROS (Early Review Organizing Software): a web-based software to facilitate
the initial phases of a systematic review
The workload of the initial phases of the process of developing a systematic review (SR) is
often underestimated. A typical process during the screening phase of studies for inclusion
in a review includes the management of search results using reference management
software, and the deletion of duplicates. The next step is a sensitive screening of the
studies, preferably done by pairs of independent reviewers. This screening is done initially
by title and abstract and in a second phase by full-text1, 2. Probably the most challenging
step is the risk of bias assessment by each relevant domain1. This process is not only timeconsuming, but it is also complicated, tiresome, and prone to mistakes, especially if we
have to deal with a great number of references retrieved from the search strategy. A
computer-software designed to cope with the overwork of the initial phases of a SR would
improve substantially the amount of time consumed in these tasks. There is a generalized
lack of development in this area, and the available options are not very accessible or
affordable by the majority of groups, especially small teams from developing countries.
The Argentine Cochrane Center IECS developed a new web-based software that helps in
performing the first following stages of a SR: importation of citations from a reference
manager software (like Endnote, Reference Manager or Procite) or directly after a search in
several medical electronic databases (such as Pubmed, Embase, Lilacs and Cochrane
library), screening by title/abstract and first agreement between co-reviewers at this stage,
uploading of full-text of those studies selected to be screened by full-text, screening in the
full text phase (tracking exclusion reasons), second agreement at this stage between coreviewers, and distribution of full-text for data collection and quality assessment.
The software has 3 types of potential users: administrator, reviewer and librarian. Each of
them has different levels of access to edit and view the project. The administrator has full
access to all the functions, is the one in charge of setting up the project, can perform a
follow up of every stage (with several friendly tables and lists), he allocates the studies to
the reviewers and can obtain the final flowchart and list of included studies. The librarian
imports the citations in the title and abstract stage and once finished uploads the full-text
files. Finally, the reviewers classify the studies (including or excluding them) and mark an
exclusion criterion (in those that are excluded).
Every type of user can track his/her own process through the whole systematic review.
EROS allows a multilingual interface to facilitate collaborative work teams around the
world.
EROS is currently being used in the simultaneous conduction of 20 systematic reviews
requested to our institution, and one Cochrane systematic review, with a range of 3-11
reviewers and one librarian per SR. The amount of citations included in each review range
between 110 and 5200. The main characteristics of EROS are detailed in Box 1
Box 1. Main characteristics of EROS
a)
b)
c)
d)
e)
f)
g)
h)
i)
j)
k)
l)
m)
n)
o)
p)
q)
r)
s)
t)
Easy concurrently management of different systematic reviews;
Differentiation of roles assigned to reviewers, administrators and librarians;
Real-time tracking of the whole process for each role;
Allows to work simultaneously in a multi-language environment;
Adequate, equitable and timely allocation of full-texts for evaluation and data
abstraction;
Notes, internal mailing to facilitate consensus and methodological decisions
Easy agreement resolution (simple, with colors, showing the other reviewer
comments to take quick decisions)
Estimation of reviewers agreement (kappa coefficient)
Quality assessment (by pairs of reviewers)
Automatic building of the study flowchart;
Possibility to work simultaneously in different SR’s stages;
Easy customizable inclusion/exclusion criteria and other relevant features.
Easy customizable labels
E-mail alerts for new tasks
Search of allocated and total studies retrieved (to check the reference lists and
duplicates)
Duplicates elimination
Filters by classification status and by reviewer (to solve disagreements)
Sort by ID number, title or classification status
Compatible with RevMan, reference manager software, medical electronic
databases (not only importing files, but also exporting files from EROS)
Repository of most important files
Administrator functions
The administrator uses a step-by-step manager to set up and manage a review (see Figure
1). After each step is finished, it automatically goes to the next one, although the
administrator can also go directly to any other different step.
Figure 1 Step by step EROS Manager
Administrator Display
Librarian Display
STEP 1. Select a project: the administrator chooses an old project to be edited, or set up a
new one.
STEP 2. Project Data: Complete or edit systematic review characteristics. Some of the
review attributes are (i.e. name, dates of preliminary reports, dates of paper submission,
other deadlines).
STEP 3. Users and roles: the administrator adds users to a specific SR, and assigns them a
role (reviewer or librarian).
STEP 4. Define screening classification by Title/Abstract: the administrator decides which
classifications will be used for this stage of the review. There are seven classifications by
default which can be customized (adding or deleting some of them). Each classification
finally belongs to one of two categories: a) studies that should be evaluated by full text (i.e.
high probability of inclusion / intermediate or low probability of inclusion / systematic
review / narrative review); b) studies that should be excluded (i.e. excluded / reference fulltext / reference non-full-text). As there are some articles that may contain important
information to be used for the introduction or discussion (but not primary data extraction
for the SR analysis), administrator could use the classifications named “Reference” to mark
those articles that could be used in the future but should not be analyzed by full text for
primary data. The administrator can include a description of the classifications which could
be viewed by the users posing the cursor on the specific help icon working as a reminder of
the meaning of the category.
STEP 5. Define eligibility classification by full-text: the administrator decides which
classifications will be used for this stage of the review. There are three classifications by
default which can be customized (adding or deleting some of them): “included”,
“excluded” or “contact the author for more information”.
STEP 6. Define exclusion criteria: the administrator adds the criteria that wants to be used
during the screening stages. These criteria will be included in the flowchart at the end of the
review.
STEP 7: Labels. This function allows including customizable labels or tags to identify
subsets of references for different purposes (i.e. using a subset of citations in a future
systematic review).
STEP 8. Quality criteria: the administrator decides which quality criteria will be used for
each study design. By default, there are several types of studies, and several study designs
for each type of study. Each design has some predefined quality criteria that can be
customized. More study designs and type of studies can be added.
STEP 9. Import citations: the administrator or the librarian imports the citations after the
search has been performed. Citations can be imported in several different file formats (i.e.
RIS Pubmed, RIS Cochrane, XML Endnote, etc.). This is performed in just two simple
steps: a) export the file from the database or reference manager software; b) import to
EROS.
STEP 10. Allocate articles for Screening by Title/Abstract: the administrator distributes
each study to a pair of reviewers, in an equitable manner. This is customizable too,
allowing the administrator to distribute more studies to some reviewers and less to others.
STEP 11. Upload full text files: the administrator or the librarian upload the full text files
attaching them to each citation of included ones in this phase.
STEP 12. Allocate articles for eligibility by Full-text: the administrator distributes each
study to a new pair of reviewers in order to be re-screened and eventually for data
abstraction, in an equitable manner. This is customizable too, allowing the administrator to
distribute more citations to some reviewers and less to others.
The administrator can calculate a kappa coefficient for the inter-reviewers agreement, and a
flowchart that shows the process of the review.
Reviewers’ functions
Screening by title/abstract phase: each reviewer can view the list of the studies that has to
classify. This list can be ordered in several ways and can be filtered by different
parameters. The reviewer has to assign a classification criterion and, in case that the study
is excluded, an exclusion criterion should also be stated. A box for comments is available.
Since both reviewers have to classify the study, two scenarios are possible: a) agreement
(both think that it should be evaluated by full text, or both think that it does not have
primary data); b) discrepancy (one thinks that it should be evaluated by full text and the
other one thinks that it does not have primary data). Studies with agreement turn to green
color in the screen display and studies with discrepancy turn to red. After discrepancies are
solved by the reviewers, they become green too. (see Figure 2).
Eligibility by full-text: each reviewer has a list of the studies that has to classify, with a link
to the full text. As the files are on-line, reviewers don´t need to download the files to the
computer, overloading their e-mails and storing discs with a big amount of information.
Also, this characteristic of EROS allows reviewers to use different computers at this stage
too. Since both reviewers have to classify the study, the same scenarios (agreement and
discrepancy) are present here. After reviewers agree which studies should be finally
included, they have a list of them to begin the data extraction.
Figure 2 Review Display: Examples of different citations’ stages
Figure 3 Review Display: control panel
A computerized SR tool in the initial phases like EROS saves time, reduces workload for
each involved role, and probably enhances SR’s methodological quality.
References
1.
2.
Higgins J, Green S, (editors). Cochrane Handbook for Systematic Reviews of Interventions
Version 5.1.0 [updated March 2011]. The Cochrane Collaboration2011: Available from
www.cochrane-handbook.org.
Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of
randomized controlled trials in systematic reviews: accuracy and reliability of screening
records. Stat Med. Jun 15 2002;21(11):1635-1640.