March 2013
doc.: IEEE 802.11-13/0230r2
802.11 Comment Resolution – a Tutorial
Date: 2016-01-19
Authors:
Name
Company
Adrian Stephens
Intel
Corporation
Submission
Address
Phone
email
adrian.p.stephens@intel.com
Slide 1
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Introduction
• This tutorial is intended to help 802.11 members to
approve adequate comment resolutions to working
group and sponsor letter ballots
• The audience is anybody in 802.11 involved in writing
or voting on comment resolutions
• Red text is reserved for examples of poor practice
Submission
Slide 2
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Why do we need a tutorial?
• We haven’t always don’t an adequate job in the past,
which has cost us time/hot-air/embarrassment/quality
• We have specific rules that we need to follow
• New people are coming in all the time, which dilutes
“corporate memory” of what to do and what to avoid
• We should, as much as we can, avoid “learning by
making mistakes” because the cost of making mistakes
in this process can be significant
Submission
Slide 3
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
The purpose of comment resolution
• To increase the level of support by balloters for a draft
• To improve the quality of a draft standard
• To engage with voters and gain from their
viewpoint/experience
• To complete a ballot with a draft of high quality that
meets the approval of most (>75%, typically 95%) of its
voters in a timely fashion.
• Note the tension – frequently cannot satisfy all the
commenters and complete in a timely fashion
Submission
Slide 4
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
What can happen if we do this badly?
• We do not gain sufficiently from the possible benefits of
engaging with voters
• We produce a poor quality draft document
• We get inadequate involvement from WG members
who might give up trying to the process
• We risk delay by not dealing with real issues in a
responsive or timely fashion
• We risk delay in the EC through appeals to process
• We risk delay in the IEEE-SA standards board
Submission
Slide 5
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Valid Comments
• A comment needs to identify where the issue is in the draft
• It needs to identify what the issue is
• It needs to identify a proposed change in sufficient detail that the
CRC can readily identify changes that they would reasonably
expect to satisfy the commenter.
• The wording from the IEEE-SA Standards Board Operations
Manual (2010 p24) is: “This vote must be accompanied by one or
more specific objections with proposed resolution in sufficient
detail so that the specific wording of the changes that will cause the
Do Not Approve voter to change his or her vote to Approve can
readily be determined .”
• Needs to be in scope (next slide)
Submission
Slide 6
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Valid comment scope
A comment is in scope if:
• It is in the initial ballot
• It is in a recirculation, and relates to material in the
balloted draft changed since the previous draft, or
relates to material affected by that change
• Is in a recirculation, and relates to text that is the target
of an unsatisfied “must be satisfied” comment from a
“no” voter.
Submission
Slide 7
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Examples of invalid comments - 1
There’s a bug
somewhere in the
document
fix it
This is an invalid comment.
It fails to locate and identify
the issue. Fails to identify
changes in sufficient detail
so that the specific wording
of the changes … can be
determined.
This is a wholly inadequate
comment.
Submission
Slide 8
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Examples of invalid comments - 2
Clarify
On p123 line 45 in
equation 8.12 why is
the lower limit of the
summation a zero?
This is an invalid comment.
The comment is asking a
question. It is not proposing a
change that can in any sense be
interpreted as “specific
wording”
The CRC can attempt to answer
the question in its comment
resolution response, but is under
no obligation so to do.
Submission
Slide 9
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Examples of invalid comments - 3
In clause 10, coexistence
with mode X devices has
not been considered.
Submission
Add a
This is an invalid comment.
coexistence
with mode X
The comment identifies a potentially
device solution. “big issue”, but doesn’t provide specific
changes – it is essentially giving the
CRC permission to do more work. In
some cases comments of this type make
the unstated assumption that the “big
issue” is a requirement for the draft,
even if no such requirement appears in
the PAR or has otherwise been agreed:
effectively the comment says “I think
that coexistence with mode X devices
should be a mandatory requirement.”
Slide 10
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Examples of invalid comments – 4
The 802.11 style manual
indicates that all Clause 10
parameter names should
be in MixedCase.
Submission
Review all
parameter lists
in Clause 10
and adjust to
conform to the
MixedCase
style.
Slide 11
This is an invalid comment.
The commenter is giving the CRC (or
probably the TG’s editor) permission
to do more work.
A volunteer (probably the editor) may
agree to perform this task given an
“accept” resolution. But if no
volunteer to do the work is found, a
rejection on the basis of lack of
“sufficient detail” is also appropriate.
This resolution may or may not cause
the commenter to come back with an
improved set of proposed changes.
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Recommendations for comment resolution
• Address all comments – be responsive to the technical issue
wherever this is possible
– Answer the comment made, not the comment you wish they’d made.
Answer all the points in the comment.
– The proposed change should relate to the comment directly, not be an
excuse to fix something else.
• Minimize procedural resolutions, except near the “end”
– Address the technical issues in preference to using a “scope” or “invalid
comment” resolution.
• Use neutral language – address the issue, not the commenter
Submission
Slide 12
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Technical aspects of comment resolution - 1
• Use the correct resolution
– Accepted means “we agree with the comment and the proposed
change”. It cannot be used if the proposed change is not
sufficiently detailed that the editor and TG members know what
the intended change is, e.g. if the commenter offers alternative
solutions.
– Revised means “we agree with the comment, but we approved a
change different from the commenter’s proposed change”
– Rejected means “we disagree with the comment, and made no
changes”. Don’t use “Rejected” if the comment is addressed in
another comment. Instead, repeat part of the resolution of that
comment as appropriate. Don’t use “Rejected - Fixed in Draft
x.y” as a resolution because it doesn’t say what you changed to
resolve the comment.
Submission
Slide 13
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Technical aspects of comment resolution - 2
• Avoid over-use of references to submissions
– This forces the commenter to do work that you could have done.
– Copy and paste textual resolutions from submissions.
– Reserve references for material that cannot be represented in plain
text, e.g. graphics and heavy use of insert/delete.
– e.g. – bad usage: “Rejected – see 11-12/1234r0”.
• Use URLS
– Any references to external documents should be URLs, not 802.11
document numbers. This is particularly important in sponsor
ballot, because the ballot group member don’t necessarily know
where to look for 802.11 documents.
Submission
Slide 14
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Technical aspects of comment resolution - 3
• Avoid references to other comment resolutions
– When we present unsatisfied comments to the EC (or RevCom
read them), they might see only a subset of comments, e.g. the
“Must be Satisfied” ones. When RevCom look at a comment
resolution, they do not necessarily see the same comment ID as
the TG works with.
– These both mean that references to comment IDs in resolutions are
useless.
– Instead of referencing a comment resolution – copy it.
• For example:
– Revised. At page 1234 line 56, change ‘nurgle’ to ‘flange’.
– Note to editor (this is the same as comment resolution for CID
321).
Submission
Slide 15
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Technical aspects of comment resolution –
4 – promises of future action
• Invalid Resolution: “Rejected. We will consider this comment in a
later ballot, but we ran out of time to discuss it now.”
• Avoid promises of future action. Any such promise is an
immediate red flag to the EC / RevCom. Two exceptions to this
rule:
– Editorial comments can be deferred to the publication editor
– For comments out of scope: “Such comments need not be addressed in the
current standards balloting process and may be considered for a future
revision of the standard” (IEEE-SA SB OM)
• A member of the TG who is in the sponsor ballot pool might offer
to submit certain comments from the end of WG ballot into
sponsor ballot; but if he does so, it is an individual act, not an
official one.
Submission
Slide 16
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Technical aspects of comment resolution –
5 structure of resolution
• Resolution should include:
– “Accepted”, “Revised” or “Rejected” classification.
– A response to each issue raised in the comment.
– Clearly distinguish between any response directed to address a
question raised in the comment and instructions to the editor.
• e.g.:
– “Revised.
– At p 1234 line 56 change “shall” to “should”.
– In reply to the commenter, this allows legacy STAs to continue this
behaviour, but allows new STAs to choose what to do.
Submission
Slide 17
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Technical / General / Editorial comments
• Comments should be resolved without regard to how
the commenter classifies them, except:
– Usually the editor gets to propose resolutions to most of the
editorial comments
– In WG ballot, editorial comments that are not part of a no vote
might be resolved “This comment has been passed to the TG
technical editor for consideration during preparation of a
subsequent draft”
– Near the end of the sponsor ballot, editorial comments might be
resolved with “This comment will be passed to the IEEE-SA editor
for consideration during publication editing.”
• The last two are permissible exceptions to the “don’t
promise future action” rule.
Submission
Slide 18
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Examples of Unresponsive Comment
Resolutions
Submission
Slide 19
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Responsive / unresponsive - 1
Comment
In clause 10, coexistence with
mode X devices has not been
considered.
Proposed resolution
Add mandatory coexistence
mechanism Y as described in
document Z.
• Resolution: Reject. The TG considered the comment
and did not agree with the proposed resolution.
• Analysis: Unresponsive. The response is generic and
does not include the subject matter of the comment.
Submission
Slide 20
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Responsive / unresponsive - 2
Comment
In clause 10, coexistence with
mode X devices has not been
considered.
Proposed resolution
Add mandatory coexistence
mechanism Y as described in
document Z.
• Resolution: The TG wasn’t sure whether to accept the
comment or not and couldn’t agree on a change
• Analysis: Unresponsive. The response is wishy-washy
and leaves the commenter with inadequate information
on the TG’s position.
Submission
Slide 21
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Responsive / unresponsive - 3
Comment
In clause 10, coexistence with
mode X devices has not been
considered.
Proposed resolution
Add mandatory coexistence
mechanism Y as described in
document Z.
• Resolution: Reject. Coexistence with mode X devices is not a
requirement of the PAR. The TG considered the comment and did
not agree with the proposed resolution on the basis that it added
significant complexity with insufficient benefit.
• Analysis: Responsive. The response includes the subject matter of
the comment, and provides concise reasons why the comment was
rejected. It is not necessary for the resolution to provide extra
detail on precisely how much complexity would be tolerable and
how the TG members traded off complexity versus benefit.
Submission
Slide 22
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Types of Procedural Rejection
Submission
Slide 23
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Procedural Rejections – 1 (insufficient
detail)
Context:
Example resolution:
The comment fails to locate and
identify the issue. Fails to
identify changes in sufficient
detail so that “the specific
wording of the changes … can be
determined”.
Rejected. The comment fails to
identify a specific issue to be
addressed. It fails to identify
changes in sufficient detail so
that the specific wording of the
changes that will satisfy the
commenter can be determined.
Submission
Slide 24
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Procedural Rejections – 2 (asking a
question)
The comment is asking a question
Rejected. The comment fails to identify
a specific issue to be addressed. It fails
to identify changes in sufficient detail so
that the specific wording of the changes
that will satisfy the commenter can be
determined.
In reply to the commenter, the lower
summation limit is zero because this is
the starting index for variable I
representing iteration over subcarriers.
Submission
Slide 25
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Procedural Rejections – 3 (big issue)
• Rejected. The comment fails to identify changes in sufficient detail
so that the specific wording of the changes that will satisfy the
commenter can be determined.
• Note, the CRC can (but is not required to) admit or accept the
validity of any reported “big issue”. Note that the CRC doesn’t
need a valid comment to make any changes it wishes, and if it
agrees that there is a big issue and some ballot cycles later
determines how to resolve that issue, it is at liberty to make those
changes.
Submission
Slide 26
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Procedural Rejections – 4 (out of scope)
• Rejected. The comment is out of scope: i.e., it is not on
changed text, text affected by changed text or text that
is the target of an existing valid unsatisfied comment.
Submission
Slide 27
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Procedural Rejections – 5 (can’t agree on a change)
• Context: The TG cannot reach a consensus on any other
resolution.
– The CRC has considered one or more other resolutions that attempt to
satisfy the comment, but no such resolution has received 75% approval.
– It should be recorded in the minutes what other resolution(s) have been
considered, together with the results of any straw polls or motions on
those alternative solutions so that the commenter can understand that this
solution was used after due diligence to satisfy the comment.
• Rejected. The CRC could not reach consensus the changes
necessary to address the comment this comment.
Submission
Slide 28
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Procedural Rejections – 6 (pile on)
• Context: A commenter repeats (or is substantively similar to)
either their own or another commenter’s valid unsatisfied
comment from a previous ballot.
• The resolution should include a statement that the CRC has
previously considered the comment (or a substantively similar
comment), along with identification (by reference or copy) of the
original comment and its disposition detail and status.
• Example: Rejected. The CRC has previously considered this
comment in the Initial Sponsor Ballot. The comment considered
was “The flange needs to be nurgled” and the resolution of that
comment was “Rejected. The comment resolution committee
believe the cost of nurgling the flange outweighs any benefit from
doing so.”
Submission
Slide 29
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Typical Ballot Lifecycle
• Approval cycle:
–
–
–
–
–
•
•
•
•
•
WG Ballot
EC approval to proceed to sponsor ballot
Sponsor Ballot
EC approval to proceed to RevCom
IEEE-SA RevCom and Standards Board approvals
Permissive vs restrictive
Solicit input from “no” voters
Recirculate unchanged
EC approval
RevCom / IEEE-SASB approval
Submission
Slide 30
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Ballot Lifecycle - Permissive vs restrictive
• In the first ballot, all comments are in scope (although
not all comments are necessary valid).
• Early in the process, the TG will probably attempt to
satisfy comments regardless of whether they are valid
• As the draft nears the end of a sequence of
recirculation ballots, the TG wants to proceed to the
next stage in the process.
• At this point they are likely to become restrictive,
rejecting comments on the basis of validity or other
procedural grounds.
Submission
Slide 31
Adrian Stephens, Intel Corporation
March 2013
Ballot Lifecycle - Solicit
doc.: IEEE 802.11-13/0230r2
input from “no” voters
• Comment resolution provides an opportunity for 2-way
communication between the commenter and the TG.
• There is no reason to limit this to communicating via
the official tools. The TG are encouraged to engage
directly with commenters to understand better what
they are saying in a comment, and to seek feedback on
its possible resolution.
• The TG chair should seek feedback from unsatisfied
comments from previous rounds so that the actual
unsatisfied comments are identified for presentation to
the EC.
Submission
Slide 32
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Ballot Lifecycle - Recirculate
unchanged
• At the end of a recirculation ballot, unless the ballot received
no comments, the group will determine that there are
reasons to validly reject all the comments received.
• In this case an unchanged draft is sent to the next (final)
recirculation along with the reasons for these rejections
• Comments might be received from this final recirculation,
but they are either a reiteration of an earlier comment (i.e.,
not new) or they are out of scope.
• Thus the process is guaranteed to terminate, should the TG
so wish
– i.e., no commenter can force the process to continue indefinitely
Submission
Slide 33
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Ballot Lifecycle - EC
approval
• The approval of the EC is required to progress between
WG ballot and Sponsor ballot, and to go to RevCom
• Approval is granted (either unconditional or
conditional) at plenary sessions of 802 (March, July,
Nov), and can also be sought in EC telecons, or by EC
ballot
• The TG produces and approves a report on the status
of the ballot and unsatisfied voters/comments to solicit
this approval.
• In the case of conditional approval, an email on
meeting the terms of the conditional approval is sent by
the WG chair to the EC when the terms are met.
Submission
Slide 34
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Ballot Lifecycle - RevCom
/ IEEE-SASB
approval
• RevCom reviews material it retrieves from the
MyBallot system in order to determine whether the
process for sponsor ballot has been properly followed.
• They have visibility of all comment resolutions.
• RevCom members might enter into Q&A with the TG
chair about issues they have with comment resolutions.
• RevCom makes a recommendation to the standards
board (IEEE-SASB), which is responsible for the
actual approval.
Submission
Slide 35
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Difference between WG and Sponsor Ballot
• During WG comment resolution, the TG resolves
comments
– Motions can be made only during 802.11 sessions, except
– Once conditional approval has been obtained from the EC, a TG
telecon can approve resolutions by motion (802.11 OM)
• During Sponsor ballot comment resolution, the WG
chair delegates responsibility to a comment resolution
committee (CRC) to resolve comments
– Any 802.11 voter can join this CRC, which is practically
indistinguishable from a TG, except that it can vote during
telecons.
Submission
Slide 36
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
Requirements on TG officers
• Post timely spread sheets. All 802.11 members should
have essentially equal access to the comments and their
proposed resolutions.
• Ensure all comments have a valid resolution
• Ensure resolutions are actioned in the draft. This is
mainly the responsibility of the editor.
– Nothing discombobulates commenters more than the group
approving a change and then not making that change.
• Chase unresponsive “no” voters
• Prepare reports to the EC
• (Tools, support and training are available from WG
officers)
Submission
Slide 37
Adrian Stephens, Intel Corporation
March 2013
doc.: IEEE 802.11-13/0230r2
The end?
• If this tutorial informed you, but has no effect on your
future behaviour, your attendance was a waste of time.
• What are you going to do to improve the quality of
your comment resolutions?
• Call to action: Be critical of your own and others
comment resolutions.
– Spending time getting this right now should save you time in the
future.
Submission
Slide 38
Adrian Stephens, Intel Corporation