Minutes of the Methods Board Meeting
2014
HEADLINES AND ACTIONS –UPDATE 2015 – methods.cochrane.org for further
information













Methods need to communicate and encourage a better understanding of the principles
underlying the practice of Evidence Based Methods (EBMethods). This will be the topic of
the Methods Symposium in 2015. Symposium planned, editorial published and further post
colloquium editorials planned.
Cochrane in good financial health.
Need to consider revenue generation as move towards an open access model.
For Methods Groups to function effectively and meet Cochrane’s objectives requires
greater financial and centralised support, in particular, the CRG networks and co-ordination
of recruitment for research projects. This concern is covered in the Structure and Function
Review for Methods.
Article on updates for systematic reviews with key recommendations is underway. This is in
final stages reviewed twice by whole Hamilton Group. Development of an Updating strategy
is proposed as a target for 2016 within the Strategy to 2020.
Cochrane needs to be realistic about the level of voluntary capacity available to commit to
Cochrane work. This concern is covered in the Structure and Function Review for Methods.
Cochrane needs to value and support development opportunities provided by Methods
Group. This concern is covered in the Structure and Function Review for Methods. covered
in the survey ‘Being valued’.
Cochrane Rebrand launch 31st January 2015. All Methods websites and rebrand have
completed.
All Methods Group websites to link into central Methods site. This is in process moving to
the individual templates first step. Methods.cochrane.org requires further development to
integrate information across all individual websites. Please contact Louisa Dunn on any
matters regarding websites. ldunn@cochrane.org
Cochrane major initiatives underway: Learning and development strategy, CAST – author
support tool. The colloquium will update on these tools and strategies.
Training (Learning) and professional development strategy will provide limited funds for
development of training materials. Training Innovation Fund underway
http://training.cochrane.org/TIF closing date 30th October.
Further work continues on the Methods Research and Review Development Framework
Further progress on how to manage and implement the Framework needs to intersect with
the current Structure and Function Review.
Review quality issues from screening project: poor implementation of protocol methods,
poorly described interventions and the need to improve interpretation and the application
of GRADE through the protocol and subsequent review. Board view reported to the Editor in
Chief and his team. There will be a wider more encompassing development of a quality
strategy, please contact Toby Lasserson for further information. tlasserson@cochrane.org
1





Action: Jane will send Mike a link to Andrew Booth’s Wiki that looks at the application of
theory and the use of Frameworks in reviews. Jane and Mike need to confirm.
Action: To review further the purpose, function and resourcing of the networks in
collaboration with other ongoing developments within CRGs. This is now taking up into the
wider scope of both the Structure and Function Review for Methods and the CEU quality
strategy.
Action: Paolo to obtain and share information on user/access statistics pre and post transfer
of websites to methods.cochrane.org. Websites have only just moved over Jackie needs to
follow this through with Paolo.
Action: To advise David Tovey and the CEU that the Methods Board supports a shift to
protocol screening. Done.
Action: Holger will ensure he conveys these points to the CSG in their future meetings.
Holger to report.
OFFICIAL MINUTES
Monday 22nd September from 10.00 am to 15.00pm
22nd Colloquium in Hyderabad 2014
Chair: Holger Schünemann, Applicability and Recommendations Methods Group
In attendance
Methods Groups:
AEMG: All convenors unable to attend this year
APSMG: Not able to attend apologies given
ARMG: Holger Schünemann, Gordon Guyatt
BMG: Isabelle Boutron, Jonathan Sterne
EconMG: Ian Shemilt
EquitMG: Peter Tugwell
CMIMG: Lorne Becker,
IPDMG: Mike Clarke
IRMG: Carol Lefebvre
NRSMG: Barney Reeves,
PROMG: Gordon Guyatt
PMG: Katrina Williams, Carl Moons
PMAMG: Lisa Askie,
QIMG: Jane Noyes,
SDTMG: Petra Macaskill, Mariska Leeflang
SMG: Joanne McKenzie, Joseph Beyene
Interventions Handbook: Jackie Chandler, Miranda Cumpston
DTA Handbook: Petra Macaskill, Mariska Leeflang
Methods rep on CCSG: Holger Schünemann
Others present or attended: Toby Lasserson (CEU), Jackie Chandler (Methods Co-ordinator), Maria
Burgess (minutes), Julie Wood (Head of Communications and External Affairs), Miranda Cumpston
(Training co-ordinator), Clare Davenport.
Apologies: Delays with visa prevented Mona Nasser from attending.
1. Welcome
2
Holger welcomed everyone to the meeting noting 15 members present. Agenda agreed no
items for AOB.
2. Scientific Discussion
Mike Clarke addressed, ‘Why do we need evidence based methods?’ When normally arguing
‘Why we need evidence based health care? We can swap ‘health care’ for ‘methods’.
Therefore, the rationale for evidence-based methods is equally to recognise that there is
uncertainty and potential for ‘harm’ in the application of methods as in interventions.
Key presentation points were:
 In order for practitioners to interpret reviews correctly, they need to understand the
conduct of this research.
 There are potential risks and harms (shown by a study on student midwives) with poor
interpretation of review information.
 A review that compares methodologies is a 'study' not a 'review'.
 Mike introduces the concept of ‘Studies Within A Review (SWAR)’. Several SWARs are
required for results to be applicable across reviews.
Main Discussion points:

A finding in one study cannot generalise, so how do we arrive at the right method? We need
to acknowledge the uncertainty in the methods and that each methodological study
provides a part of the information, as with other evidence based syntheses. Applying a
GRADE methodology could assist in determining validity of compared methods to establish a
way to summarise the ‘harms’ or ‘uncertainties’ of different methods to reviewers.
 A key objective of Methods Groups is the development of Evidence Based Methods, with
some Groups supporting specialist registers.
 Cochrane supports methodological work through the Methods Innovation Fund, however,
this work probably requires better communication to the wider Cochrane community. Mike
provided examples of his work (SWAR 2) with three PhD’s working in this area looking at
different summaries of reviews. Such work underpins the utility of the CMR.
 Our colleagues in Cochrane might view this as ‘navel gazing’ and therefore we would need to
be clear as well as pragmatic about when and where to apply these approaches. The impact
of some developments such as network meta-analysis and the inclusion of non-randomized
studies might influence the future conduct of trials or prevent head-to-head comparisons?
 In learning from the past with the development of systematic reviews, we need to establish
the practice of EBMethods. In summary, how are we going to move the agenda for
EBMethods forward and address a culture change in Cochrane as we have done with
‘Summary of findings’ tables?
Action: Jane will send Mike a link to Andrew Booth’s Wiki that looks at the application of theory
and the use of Frameworks in reviews. This work is a result of one of the Methods Innovation
Fund projects. This provides a resource to reviewers to consider what approach to use in their
review.
Decision: The Methods Board agreed that this could be a topic for next year’s symposium
At this point Peter Tugwell and Mike Clarke leave to join the Co-eds meeting and Joseph Beyene
joins.
3. Minutes of last meeting held in Quebec for approval
Minutes approved with the removal of Katrina Williams from the attendance list.
3
4. Matters Arising and Actions
Madrid Minutes (2011)
No further action required on the request to consider archiving material. The Cochrane
Operations Unit and the Cochrane Editorial Unit will be moving to a single office in London.
Archiving particularly important historical material will be considered.
Auckland (2012)
4.1 An evaluation of the MIF process was conducted with a short survey to all executives. MARS
AC has overseen the process. Board report provides details of current MIF allocation.
Action: The evaluation of the MIF process is attached to these minutes and available on the
Cochrane Methods website.
Quebec Minutes (2013)
7.4 CMR proposal was not accepted by the CSG and they have requested that any future
submission consider sustainability. Board report provides details and the CSG minute. A meeting
with interested parties needs to be arranged to consider and progress a further proposal.
7.5 The Training Strategy consultation received Author forum feedback. Ian Shemilt, Yemisi
Takwoingi, Jill Hayden and Andrew Booth represented Methods interests. Miranda Cumpston,
Training Co-ordinator will present at this meeting.
7.5 Aetiology Reviews Methods Group has not progressed, although Jos Verbeek remains
interested in setting up a Group.
5. Report to the Methods Board
Holger requested any additional agenda items (none). Report presented by Jackie.
5.1 CRG Networks
A meeting with Network leads raised a number of issues. These covered when to formulise the
networks and increase proactive recruitment, engagement of volunteers and the development
of network member credentials, development of other networks and the need to consider
whether other networks can or should be developed, and whether this is the right model for the
CRGs. The CRG Structure and Function review led by David Tovey, Editor in Chief, will review the
current approach, which may modify these networks. The networks will need to be adaptable,
workable and able to respond to the needs of CRG’s and therefore the networks require
resources to ensure they can function effectively.
Ian reported that the Economics Methods Group was also building up a network and mentoring
people to act as health economic advisers or editors based in CRGs, with some success with 15
groups (10 coming on in that last year). This is a Methods Group elective function. Greater clarity
is required on understanding the purpose of this function within Methods Groups as to whether
it should focus only on a key set of methods and ensure that the networks function
appropriately. Furthermore, whether there should be central co-ordination of these networks.
Action: To review further the purpose, function and resourcing of the networks in collaboration
with other ongoing developments within CRGs.
5.2 Updating Workshop McMaster University
Holger summarised the meeting objectives and presented in the Board report. Guidance for
Review Groups has progressed. A journal article of the key recommendations is in preparation. A
4
strategy is planned for the mid-year meeting. The joint Executives meeting will discuss the
output of the meeting later today.
5.3 Handbook updates
There is a Handbook meeting for authors and editors at the Colloquium.
6. Report from the CSG Representative
Holger's report key points:









CSG had a 'Board Development Governance Day' to review the function and legal
requirements of the Steering Group. Well received by CSG members and will probably
change the way the Group functions.
Cochrane remains financially healthy.
CSG decided that the Mid-Year business meetings should be held in easily accessible
locations (eg. in Europe, Dubai or Toronto). This is to promote attendance. The venue for the
2015 Mid-Year meeting is unclear since Bahrain withdrew their application.
The Cochrane Author Support Tool (CAST) is an authoring tool to help automate the work of
authors currently not conducted RevMan. A collaborative effort, subject to negotiations,
chosen to develop this tool is Covidence, Metaxis and the EPPI centre. Early versions of CAST
will be available at the beginning of next year.
Decisions are yet to be made on Gamechanger submissions.
The CSG has approved the Training and Professional Development Strategy.
CSG have approved support for the TSC support team.
South Africa is the location for the 2017 Colloquium.
CSG have agreed that Cochrane should evaluate the role and work of Colloquia. This will
happen over the next year.
Board discussion of report
The focus of this discussion centred on the transparency of Cochrane’s financial decisions
emphasizing greater financial support for Methods Groups is a continuing requirement.





Although the accounts are currently healthy projected spends (moving to an open-access
model) need to be taken into account. Current committed spending includes high spend
projects such as the Gamechanger, CAST and training. Cochrane is trying to take a strategic
approach to its financial management.
The Methods Board request better communication on where and how money is invested.
Individuals can apply for national funding, as some do, supported by, but not directly
involving Cochrane. However, national funding structures and availability vary.
If funders are aware of Cochrane’s financial accounts, they may argue that they do not need
to support activities (DTA example). Cochrane should communicate clearly about its
strategic funding plans to funders. Clarity on funding plans may ensure greater co-operation
between funders and Cochrane and their similar respective objectives.
Cochrane assumes people will provide support under extraordinarily favourable financial
arrangements benefitting Cochrane, such as the Methods Innovation Fund. However, a
counter viewpoint suggests the Strategy to 2020 provides greater transparency. Board
members were responsible for reading the financial report. Ian and Isabelle attempted to
generate more funds for Methods Groups e.g. EU funding. Board members could investigate
funding options open to them. This requires raising the profile of Cochrane methodological
research to funders. Ian provided a structure and template to credit Cochrane work and use
5

of Cochrane resources, Methods Groups affiliations etc. in publications, published in
Cochrane Methods last year.
These points about finance can be raised at the AGM.
Action: Holger will ensure he conveys these points to the CSG in their future meetings.
7. Methods Groups Feedback
Brief presentations were invited on methods dissemination and implementation issues including
recruitment issues from CRGs for pilot projects and research studies.
7.1 Jon Deeks - DTA Methods Roll Out
Seven years ago, Cochrane introduced DTA reviews, with funding provision. Members of the DTA
Review Support Group visited every CRG in Europe and provided them with initial training. This was
labour intensive and expensive, but valuable. The Group have also provided regional training in USA
and Australia. There have been three main tensions: limits with in CRGs to complete DTA reviews,
how much central support is required, and how can we build capacity? Three specialist-training
events were run over the past 5 years for CRG statisticians, methodologists and other review
specialists.
The Group now have 200 active DTA protocols and reviews in progress. An expert group of editors
(DTA Editorial Boards) manage the project. Thisw high workload has required two boards, and every
month there are two teleconferences. This provides a dual editorial system that runs alongside the
CRGs own clinical review editorial system. Turnaround time is 6-8 weeks to get comments get back
CRGs. Two CRGs have complained this is too slow, however overall quality is maintained.
In the last two years, invited CRG representatives who joined the board received training. Training is
‘on the job’ to develop DTA editor competency. Originally, these boards were set up to enable CRGs
to conduct DTA reviews independently. However, CRGs are resistant to losing this support structure
so some DTA expertise will continue. Financial provision is available to fund one person part-time.
Online distance support learning material in progress should be complete by Christmas. The DTA
editors are unfunded. Funds were used for training events, travelling to CRGs etc. Central process
costs are small (administrative support – Cochrane picks up the conference call costs, and Wiley
provide a system of manuscript central). The DTA Handbook continues to develop but is slow.
Key point from discussion:

This successful network was an advance on the model of the CRG networks previously
discussed. Prognosis will consider this model for roll out. However, funds are required to
ensure the success of the model. This may be more for new review types, but is also
required for all networks. Jon had a budget of 1.6 million over 7 years from NIHR UK.
Cochrane has supported the Dutch Centre to support the European Groups for two years
with funds for training.
7.2 Barney Reeves – MIF Project ACROBAT-NRS
A Cochrane Risk of Bias Assessment Tool for Non-Randomized Studies (ACROBAT-NRS)
Version 1 of the tool will launch at the Colloquium. Issues of feasibility, reliability, and validity
remain. The tool requires review authors to do more preparation at the protocol stage. The tool is
based on first principles of epidemiology rather than Mike’s EBMethods approach. Tool requires full
6
pilot. A guidance document accompanies the tool. Learning materials and worked examples are
required.
Key points from discussion:

The tool will accommodate different study designs including signalling questions for health
systems research designs. Study design will drive customisation of signalling questions. The
current RoB tool will also have signalling questions developed for randomized trials. The
domains remain consistent across study designs. The tool will accommodate designs such as
interrupted time series, case control and controlled before and after studies. Tools for
Quality appraisal/assessment for DTA and Prognostic studies are developed. This tool
therefore should specify that it is for intervention studies only.
Post meeting note: The tool is amended ACROBAT-NRSI I-Interventions.

The tool is difficult, time consuming to apply, and requires a degree of epidemiological
expertise to get to grips with working with the tool through the study. Therefore, this could
create a barrier to its implementation for the appropriate inclusion of NRS for either adverse
effects or benefits. Barney believed that there should be some way to create some economy
in managing the tool protecting the author from unnecessary details. The guidance focussed
heavily on confounding (20 pages) and this was not important for many reviews as it
addresses specific issues such as time varying covariates. In addition, selection bias issues
such as lead-time bias were not relevant for new treatments.
7.3 Holger - MIF project Enhanced ‘Summary findings’ table
Holger presented the MIF project on the implementation and comparison of different formats of
‘Summary of findings’ table. No major changes to SoFs were expected. Accepted standards of
presenting quantitative results are not being followed and often do not include risk difference, or
absolute difference or reduction. The project evaluated alternative formats that review authors
could use to use risk difference as one way to present absolute effects. The project planned to
conduct two non-inferiority trials to evaluate the alternative formats of SoFs. There is a presentation
of the first trial results in an oral at the Colloquium this week.
Research issues were:

The project at the first stage (protocol) engaged with 16 groups, of which 12 were CRGs and
Methods Groups. This was an internet-based trial to obtain views on presentation of
findings, taking 20 minutes. Groups were requested to use their email lists to access review
authors. Problems identified were:
o Reluctance to use Group email databases to contact potential study participants.
o Individuals in Methods Groups contacted were not able to locate someone who
could co-ordinate research activity.
o Groups were primed with the study protocol, however, they took 1-4 months to
approve access. This was compounded by changes to editorial personnel.
o Eventually recruitment of participants was drawn from outside Cochrane. This was
problematic as this was a Cochrane funded research project. Cochrane has access to
an extensive range of contacts through its networks. However, accessing these
networks was an obstacle.
o It is difficult to rely upon very busy people with competing priorities. This requires
central support to work with Methods Groups to co-ordinate and engage people in
research activity.
7
o
o
Cochrane’s expectation that people (Universities) should provide activity for free
with limited incentives is a challenge. Drug companies are more effective in this
respect with financial incentives.
The trainer’s network is another contact resource for recruitment. Workshops can
be used as base for conducting the sort of research suggested. The key message is
one of identifying the incentives.
Holger will explore further the ideas and feedback for the second trial proposed.
7.4 Carl Moons – Exemplar Prognosis Reviews
The three exemplar reviews are currently at the stage of two protocols published and third on its
way. These reviews are examples of the three types of prognosis studies undertaken:
1. Prognostic factors: Low back pain
2. Baseline prognosis (average outcome for particular populations – course of health
state): Autism disorders
Not all health states are diseases e.g. pregnancy, new born health
3. Predictive modelling: Post-operative nausea and vomiting
These reviews use observational studies, and RCT data is analysed as a cohort not for treatment
effects. Predictive effects rather than causal effects. These reviews, previously descriptive, narrative
syntheses have moved on in the last 2 years with methods for meta-analysis and quantitative
synthesis. Tools have been developed for RoB (Quality in Prognosis Studies (QUIPS) and (Prediction
model risk of bias assessment tool (PROBAST)), data extraction and search strategies. Five
workshops (4 new) will be conducted during the Colloquium. A workshop on RoB of prognostic
factors will be conducted next year. “Methods Groups are the backbone of Cochrane in providing
the necessary tools to conduct reviews”, Carl Moons, PMG.
PMG is a young group with six convenors: Richard riley, Sue Woofleden, Jill Hayden, Doug Altman,
Katrina Williams and Carl.
Development issues are:






Many reviews of prognosis studies are conducted outside Cochrane. The current restraint to
three exemplar reviews might be hampering progress in Cochrane as author teams seek to
use PMG to support non Cochrane reviews. These reviews are being published in high end
journals e.g. BMJ and the Lancet with the additional issue that Cochrane may not be
properly credited.
PMG believe like DTA that they should start with at least 10 exemplar reviews. This would
prevent loss of suitable titles to external publications rather than the CDSR.
The DTA Handbook is seven years plus in the making and it is clear the proposed PMG
Handbook for these reviews will not be possible in the very near future and that it should be
a process of ‘learn as we go’. Further reflection suggested that is it not realistic or feasible to
have a Handbook for every type of review. There are peer-reviewed materials available.
PMG is willing to follow as best it can with current resources, the DTA model, and partner a
PMG contact with a CRG member.
There is a problem with some of these complex reviews (PMG, DTA, QIMG) to identify a CRG
home.
Publication of different types of review can hold up implementation.
8


Some reviews pushed by funders slip under the radar when Cochrane is seen as reluctant to
publish certain methods. However, this has resulted in inadequate and inappropriate use of
methods.
In looking forward the Methods Research and Review Development Framework should begin
to facilitate progress along the process steps. Current accepted review types such as PMA
and IPD also struggle due to the various complex barriers in Cochrane and may find
publication outside easier. Therefore, further progress is required to address the
complexities of research and review development systems, processes and capacity. This
should take account of changes and developments in Cochrane such as the possible
clustering of CRGs, which could afford a better contact point between Review and Methods
Groups such as PMG.
8. Main Item for Discussion
Joined by IKMD to discuss website, and by Communications and External affairs Department to
discuss branding.
8.1 Website
Methods Groups websites will be brought into a more coherent structure directly linking from
the main Cochrane Methods website. This will include the new branding. The
methods.cochrane.org beta website needs a complete revamp as the umbrella website. This is
an opportunity to create a ‘front end’ Cochrane Methods that links and connects Methods
Groups sharing important information. The ‘front end’ intends to look similar to the current
UKCC site. Users will become aware of the existence of all Methods Groups when accessing an
individual group's website.
Key points were:





Groups will retain ownership and editing rights to their own web pages in their current
location but with a new web address. Currently xxmethodsgroup.cochrane.org will change
to methods.cochrane.org/xxmethodsgroup. From a back end of view, nothing will change
and access to the website will be as normal, there will be no need to transfer content. Redirection will be automatic, although advertising of the new address is advised.
Two are on their own servers and will need to consider moving to the Cochrane Drupal
system. For C-CEMG this needs to be done quite quickly as they need to move from their
current host site.
Greater integration of shared information can be conducted iteratively overtime.
Communicating changes that informs all groups should be easier.
User’s will go to one place, a single entry point, for all Groups and can move around this
information more easily. User/access statistics for the websites would be helpful to do a
comparative evaluation of before and after the change. Paolo confirms that they have this
information and can share it.
Action: Paolo to obtain and share information on user/access statistics pre and post transfer of
websites.
8.2 Cochrane Rebrand
Presentation by Julie Wood incoming Head of Communications and External Affairs Department:
The goal of the rebrand is to unify the Cochrane Groups and gain greater impact on the outside
world. The new logo retains much of the old logo, but introduces a secondary colour, purple. The
9
strapline will remain. Our organisation’s values are “For Knowledge, For Change, For You”. SourceSans-Pro is the new (and free!) font. Main brand launch will take place on January 31st 2015, and the
Cochrane Library (with Wiley) should launch around the same time. All Groups roll out their rebrand
between January 31st and March 31st in English, with multiple language rollouts happening later.
Key points were:

Sub-brands: All groups will be able to choose a colour to complement their logo to
differentiate themselves from other groups. Brand design examples for reports, websites
etc. were shown. The yellow is slightly harder to see on websites. Colour choice is limited
to five colours.
There was a proposal to shortened long names of some Methods Groups. Fields, for
instance, are dropping the word 'fields' from the logo to shorten the name and improve
impact. For example, the Child Health Field will read 'Cochrane Child Health'.
The strap line was about 'healthcare professionals'. The introductory paragraph in Julie's
presentations mentions 'medical professionals', she was requested to amend this to just
'professionals'.


Action: Julie to consider change



The Methods Groups convenors wanted to ensure their individuality and have their own
sub-brands as well as the sub brand 'Cochrane Methods'. There will be appropriate
occasions to refer to this logo for the collective of MGs.
Groups co-registered with Campbell will require a discussion with Campbell about
reflecting the joint status of the Group.
Cochrane Exchange at the Colloquium will have informative fliers on the re-brand and
CEAD are available for people to have more in-depth discussions about the rebrand.
Jon Deeks and colleagues joined the meeting.
8.3 The proposed Cochrane Training Strategy from Miranda Cumpston
The Training and Professional Development Strategy is developed. This involved an extensive
consultation process involving Methods Groups representatives. CSG has approved the strategy.
Jackie will circulate the paper.
Action: Strategy attached to these minutes.
The strategy shifts the language from ‘training’ to 'Learning and Development'. The strategy focuses
on infrastructure and learning pathways, supporting good training and the learning needs of all
Cochrane collaborators. Overarching principles are trying to improve the quality of the reviews
where there are some critical gaps, provide co-ordinated training for editors, and smaller projects
addressing particular training needs. Many projects will be collaborative including working with
Methods Groups. The methods training event will remain ongoing. The DTA support structure has
been UK/Europe focused, therefore there is a gap in training to implement DTA reviews elsewhere.
There will be increased capacity in the central office for training staff that will provide greater
resources to support Methods Groups training. Key components of strategy are:

Planning to provide better infrastructure for online training, including software to convert
presentations into something more interactive, learning managing systems that can provide
pathways for those who are working through distance training courses.
10



There will be a source of money through the training innovations fund (£30,000 a year). This
is a very flexible fund. Methods Groups can apply for funds next year.
A new focus on training evaluation will target learning specifically to authors and editors
etc.
Co-ordinated mentor scheme, centrally supported, will facilitate mentoring relationships.
Discussion points were:



Generating revenue from training activities, such as online courses and Centre workshops
will develop.
Cochrane will consider its current partnerships with universities e.g. Portsmouth and will
broaden formal links with other universities linked to Cochrane to develop online courses
and provide training in university courses. Universities will be approached via the key
individuals known to Cochrane rather than a blanket approach allowing for benefits to be
shared.
Funding of £30,000 is a starting point but quite limited, although linked with other support.
8.4 Implication for Methods from the screening project –Toby Lasserson

This project started in September 2013 and has screened 415 reviews. Between 10-12%
(n=50) have no problems or errors. However, 10% of reviews have quite serious errors with
some reporting major inconsistencies that are hard to correct. Issues fall into four key areas:
1. Implementation of protocol methods, the most challenging to solve, 2. Interpretation, 3.
Inconsistency, and 4. Investigating 'funny looking' results. Screening emerged from 'critical
instances' (recalled reviews). To conduct an audit of published reviews will require
relinquishing the requirement of the top 12% of Groups providing consistently good reviews
to be screened. A sample from the middle group and all those in the bottom 10% will
continue full screening.
Issues identified were:




Reviews do not elaborate on interventions well.
Some reviews conduct too many comparisons. This causes a drift toward describing results,
but not making the link between the findings, objectives and question of the review.
Best reviews are more compact and tidily link all sections of the review together.
GRADE is left to the last minute and seen as a pit stop on the way to completing a ‘Summary
of findings’ table. Sixty-two per cent of reviews have a ‘Summary of findings’ table and this is
increasing. The best reviews use GRADE as a way to interpret the findings and integrate this
process throughout the review.
Discussion key points were:



Question formulation and protocol development are key to ensuring the integrity of the
review. They should be fit for purpose. Removing or improving reviews at this point makes
sense. Also important to ensure specialist advice is provided at protocol development.
Screening is a quality control measure and involves a lot of work.
The Methods Board supports a shift in screening reviews to protocols.
Action: To advise David Tovey and the CEU that the Methods Board supports a shift to protocol
screening.
11
8.5 Methods Strategy: Direction and Challenges
This item focussed on the Methods Research and Development Framework. This work started in
Paris in 2012 now the 2014 Strategic target 1.4: Nonstandard Reviews Framework. The scope of this
target broadened to include a range of methodological developments. There needs to be some
process of approval for new methods. However, criteria that trigger process steps in the Framework
need identification. Furthermore, procedures that involve the Methods Board, Methods Executive
and the MARS AC in the process steps (decision points) requires further clarification. For example
ensuring a degree of checking the development prior to presentation at the MARS AC. Draft
Framework model was circulated.
Discussion points were:







Cochrane Handbook provides endorsed methods and therefore endorsement cannot be
retrospective. Everything in the current Handbook is a baseline going forward. Some
developments will not need to go through the Framework process steps. Therefore, clarity
was required on the trigger criteria for developments to ensure a level playing field.
Any development should be clear about implementation expectations as some tools are to
be widely implemented, whereas others are on as required basis.
External peer reviews might be required as not all Board members can comment on
particular methods. Peer review could be a shared activity across the Cochrane and
Campbell collaborations with the identification of other high priority areas of activity. As
part of the Framework, a process of responsibility and accountability that involves the Board
and Executive providing highly developed decisions, which proceed to MARS AC for
ratification.
Implementation of methods gains greater weight with an approval process.
Change management requires consideration when addressing implementation of methods
and is not solely for the Methods Board.
The Board agreed timescales for the integration of trigger criteria into the Methods Research
and Development Framework, with work proceeding on the development of the Manual
detailing the processes discussed. This will go through a number of drafts and with a final
Framework for approval at next year’s Board meeting. Board members were encouraged to
be involved with requests to contribute comment etc.
Holger suggests final approval of policy to take place at the next face-to-face meeting in a
year, with drafts circulated during the year. The Board agreed this.
Close and thanks.
12