Quality Control of IEEE Conferences:
Ensuring Excellence and Integrity
IEEE China Mini Panel of Conference Organizers (POCO)
19 September 2015
Bin Zhao
IEEE Technical Program Integrity Committee (TPIC)
Agenda
IEEE Conferences
Issues in Conference Quality
Quality Control and Assurance
Recommended Practice
Q&A
•
•
2
R10/China is very important to IEEE from many perspectives
IEEE wants to be the favorite place for R10/China’s
Electronic and Electrical Engineering Community:
• to hold conferences or conventions
• to publish technical papers
9/19/2015
IEEE Conferences
Bright minds share the latest research at IEEE sponsored
and co-sponsored conferences around the world.
1,600 Annual Conferences
Research
3
9/19/2015
Collaboration
Publications
IEEE Global Conference Overview
In 2014 IEEE….
Sponsored
805
1635 conferences
financially
sponsored
830
Partnered with over
in
90+ countries
technically
co-sponsored
1000 non-IEEE entities
With Universities and other not-for-profit organizations
Over
480,000 attendees
266K
Acquired
4
9/19/2015
Financially sponsored
conferences
215K
Technically co-sponsored
conferences
1510 conference proceedings
Conference Quality Issues
“Pay-to-Publish” behavior
– most authors are no-shows
Paper Stuffing
– 1000 papers from conference
w/100 attendees
Proceedings Hijacking
– unauthorized people
substituting content
Citation Stacking
– numerous self-references in a
single paper
Sponsorship Shopping
– mainly in TCS
5
9/19/2015
Machine-generated
papers
Content outside of
conference scope or
Organization technical
scope
Plagiarized papers
Poor English /
grammar
Potential Consequences
Loss of IEEE’s Reputation
Loss of IEEE Conference
Publications’ Ability to be
Indexed
6
9/19/2015
The IEEE Brand Promise
Advancing Technology for Humanity
Maintain IEEE leadership as
the trusted source of
advancements and applications
of technology
Create standards, innovative
research venues, and a global
technology forum
Provide opportunities to conquer technical
challenges with imagination
Enable today’s innovators to improve tomorrow’s
quality of life, as our tagline promises
7
9/19/2015
The IEEE – Who We Serve
Global Reach
Academia, government
and corporate
organizations around
the world rely on IEEE
publications via our
online database: the
IEEE Xplore Digital
Library.
– Search >3,650,000 items
– IEEE is the most cited
publisher in new
technology patents
8
9/19/2015
Root Causes and Driving Forces
Need for Authors to Publish
– Major force in parts of Asia
Monetary Profit
– $200k revenue per pay-to-publish conference
Different Cultural Viewpoints
Inexperienced Conference Organizers or IEEE OUs
Decentralized, Trust-Based Model in IEEE
Problems mostly associated with “hotspots”
– Confluence of needy authors, predatory organizers, and
naïve IEEE organizational units (OUs)
– Mostly occurs in areas of the world experiencing rapid
growth and in TCS conferences
9
9/19/2015
Corrective Actions
New conference approval policies
Joint Volunteer / Staff Conference Quality Team
– TPIC: Technical Program Integrity Committee
Dedicated support staff
– Tracking, Analyzing and Following Up on Complaints
– Screening 10% of Incoming Xplore® Content
Communications to change behavior
Education program developed and launched
10
9/19/2015
Result of Cross-IEEE Partnership
Operation Screening
Issues and shortcomings in conference organization
– Insufficient engagement by co-sponsoring IEEE OUs
– Inadequate resources dedicated to peer review
– Implication of a pay-to-publish/no-show business model
– False advertising and misrepresentation
– Partnering with co-sponsors whose non-profit status is
questionable
– Very short time span between conference initialization
request and the actual event
 sometimes four to six months or less
 normal practice is one to two years
11
9/19/2015
Content Screening
Conference content scope and quality issues
– Papers well outside of the stated subject matter scope of the
conference
– Papers well outside of IEEE’s engineering/technology related
subject areas
– Papers that are poorly or unintelligibly written
– Fraudulent content, e.g., machine-generated papers
– Papers that lack some or all of the basic elements commonly
found in scientific research papers:
 Introductory question or hypothesis -> brief summary of related
previous scholarly work by others -> research methods ->
results -> conclusions
12
9/19/2015
Decision Process
Third-Party Company Evaluates Every Paper from
Conference, Searching for Suspect Papers
– Out of scope
– Poor writing quality
Volunteers Screen Subset of Papers, Then Decide
– Check third-party evaluation
– Examine other factors (review process, program, etc.)
Accept / Reject Entire Conference
– Evaluating the Conference, not Individual Papers
13
9/19/2015
Scope Definition
Aerospace
General Topics for Engineers
(Math, Science & Engineering)
Bioengineering
Communication, Networking &
Broadcasting
Components, Circuits,
Devices & Systems
Computing & Processing
(hardware/software)
Engineered Materials,
Dielectrics & Plasma
Engineering Profession
Fields, Waves &
Electromagnetics
14
9/19/2015
Geoscience
Nuclear Engineering
Photonics & Electo-Optics
Power, Energy Industry
Applications
Robotics & Control Systems
Signal Processing & Analysis
Transportation
Would This Paper Be Presented at an Seminar
in an EECS Department at a Major University?
IEEE Covers Many Areas of Technology
More areas than just
Electrical Engineering & Computer Science
OPTICS
SEMICONDUCTORS
IMAGING
COMMUNICATIONS
RENEWABLE ENERGY
SMART GRID
INFORMATION TECHNOLOGY
AEROSPACE CIRCUITS
BIOMEDICAL ENGINEERING
ELECTRONICS
LTE WIRELESS BROADBAND
NANOTECHNOLOGY
CLOUD COMPUTING
15
9/19/2015
Writing and Content Quality
Main Criteria – Does the writing quality interfere with the
ability to understand the technical content?
– We do not expect or require perfect English!
Fraudulent content
– Plagiarism
– Machine-generated papers
Also Look for Basic Scientific Paper Content
– Hypothesis
– Previous work in the field
– Methods / Apparatus
– Results / Data
– Discussion – Does the Data support the Hypothesis?
– Conclusions
16
9/19/2015
Current TPIC/Quality Focus
Plagiarism Checking
– Being required for all conference papers
– Software will be free if IEEE holds copyright
– Conference organizers must evaluate output
Continue Refine Scope Definition
– Engineering Management
– Life Sciences / Bio-Engineering
– Materials Science
Online Training Modules
– Make training easy
– Modules in Chinese
Closely Engage with Sponsoring OUs
– Well-defined approval & oversight processes
17
9/19/2015
Key Factors Influencing
Conference Quality
TPIC/Conference Quality team are no longer solely in
reactive/enforcement mode; we are in proactive/outreach
mode
Sponsorship
Technical Program
Paper Quality
18
9/19/2015
Sponsorship & Engagement
Goal: Run High Quality Conferences!
Vet Conferences Carefully Before Granting TCS
Don’t Grant TCS and Then Ignore
Require and Maintain
“Direct and Substantial Involvement”
Pay Attention to Inquiries from MCE Staff
Communicate with Your Chapters
Don’t Promise that Content will be in Xplore®
Report Suspicious Activity to tpii@ieee.org
19
9/19/2015
High Quality Technical Program
The portion of the conference that contains the presentation and
discussion of research, generally in the form of:
– Oral Presentations, Poster Sessions, Workshops and Tutorials
Technical papers are collected and published as the proceedings of the
event, and are often accessed in a digital repository, such as IEEE Xplore
– Peer review required for publication of conference proceedings in IEEE Xplore
Effective Technical Program Development – see a reference:
https://www.ieee.org/conferences_events/conferences/organizers/planning_techni
cal_program.html
20
9/19/2015
Characteristics in Technical
Program of High-Quality
Technical program flow and content
– Technical scope determined tracks and sessions
– Allocating appropriate time for presentation and interaction
– Assigning appropriate space
Supported by high-quality peer review
– Identifying and organizing reviewers
– Establishing review guidelines
– Indicating review process in Call for papers
– Integrated approach in managing
 Paper submissions
 Reviews
 Final program development
21
9/19/2015
Conference Organization Timeline
Establish TP
Committee
10-12
Months
Recruit
Reviewers
Plan Key
Dates
9-10
Months
Graphic by John Tracy
22
9/19/2015
Call for
Papers
Abstract or
full paper
submission
deadline
Peer Review
Process
Notice of
acceptances
sent out
(final paper)
to authors
6-8 Months
4-6 Months
3-4 months
10-12 weeks
Peer Review
– Number of reviews per paper, and papers per reviewer
 We suggest 3 reviews per paper (2 is an absolute minimum)
 A maximum of 8 to 10 papers per reviewer
– Abstract or extended abstract vs. full papers (or twostep approach)
 We recommend a deep-dive review of an extended abstract,
followed by a later, shorter review of the full paper
– Open, single-blind, and double-blind review
 We recommend single-blind review (the reviewers are
unknown to the authors)
23
9/19/2015
Review Criteria
Scope
– Determine whether the paper is in the conference’s
stated field(s) of interest (and IEEE’s)
Quality
– Is the paper well-written?
– If it is an original research paper, did the author follow
the traditional IMRAD model (introduction, methods,
results, and discussion)?
Originality
– Is the author proposing a new idea or a novel approach?
24
9/19/2015
Acceptance and Rejection
You can require that authors make certain changes to papers
before they are accepted
Communicate to authors the results of the review process
and any next steps
– Provide link to PDF eXpress
– Requirements for presentation at the conference
– Final paper submission dates
Consideration to use pre-screening
– Improved possibility to pass quality check although it is
not guaranteed
– Understand the issues and root causes more specifically
for future events and taking corrective actions
25
9/19/2015
Screening for Plagiarism:
An Introduction to CrossCheck
CrossCheck compares submitted manuscripts to a database
of published articles (including content behind publishers’
paywalls)
Crosscheck identifies papers that surpass a predetermined
threshold of similarity to previously-published papers
IEEE recommends a 30% similarity threshold (i.e., any
submitted paper with 30% or more similarity to previously
published content will be flagged for a visual, side-by-side
review with similar, previously-published papers)
26
9/19/2015
Using CrossCheck
Conference Technical Program Chairs can register
for CrossCheck through the CrossCheck Portal
o https://crosscheck.ieee.org/crosscheck/
Several peer review tools used widely within the
IEEE conferences community are integrated with
CrossCheck
27
9/19/2015
Challenges in Peer Review
Almost every conference faces these challenges:
– Gathering enough high-quality papers by the submission
deadline
– Finding enough experts to complete their reviews by the
deadline
Good conferences attract good papers and good
reviewers!
28
9/19/2015
Non-Presented Papers
“No-shows”
IEEE empowers conference organizers to limit the
distribution of any paper that was not presented in person
at the conference
– Authors unable to attend the conference should arrange for a
qualified substitute
– Organizers must notify authors in the conference call for papers
if they choose to implement IEEE’s non-presented papers policy
If a paper is included in the proceedings distributed on site
and is later identified as a non-presented paper, the paper
should still be included in the proceedings delivered to IEEE
– Flag the paper when generating the Packing List so the paper
can be archived in our internal content management system
29
9/19/2015
Conference Quality Opportunities
– Main quality risks:
 Papers outside the stated subject matter scope of the conference
and IEEE’s fields of interest
 Papers that are poorly or unintelligibly written
 Content that indicates author(s) wrongdoing, e.g., machinegenerated papers, plagiarism, excessive self-citations
– Serious engagement needed by co-sponsoring IEEE OUs
– Appropriate resources need to be dedicated to peer review
– Beware pay-to-publish business models
The IEEE Technical Program Integrity Committee (TPIC)
and the IEEE Conference Quality team co-manage an
ongoing quality assurance process
30
9/19/2015
Working Together with A Common
Goal
– Working closely with Conference Organizer Engagement and
Education team to develop online and in-person training
materials (e.g., peer review best practices, stringent criteria
for IEEE co-sponsorship)
– IEEE volunteers and staff continue to help conference
organizers and leaders of IEEE organizational units whose
conferences have been compromised by poor quality or
inappropriate content
– Communication is the key to our success:
we are engaging with our community to identify, develop and
refine best practices
TPIC and Conference Quality team want to work with
conference organizers / IEEE OUs to improve quality,
maintain the health of Xplore, boost technical activities
31
9/19/2015
Q&A
32
9/19/2015