2004-01-04
Project
IEEE C802.20-04/004
IEEE 802.20 Working Group on Mobile Broadband Wireless Access
<http://grouper.ieee.org/groups/802/20/>
Title
Evaluation Methodology: Phased Approach
Date
Submitted
2004-01-04
Source(s)
Mike Youssefmir
ArrayComm Inc
2480 N. First Street, Suite 200
San Jose, CA 95131-1014
Reza Arefi
Todd Chauvin
ArrayComm Inc
2480 N. First Street, Suite 200
San Jose, CA 95131-1014
Arnab Das
Flarion Technologies
Bedminster One
135 Route 202/206 South
Bedminster One, NJ 07921
Shigeru Kimura
Kyocera Corp
Kazuhiro Murakami
Kyocera Corp.
YUZA Masaaki,
NECinfrontia
2-6-1 Kitamikata Takatsu Kawasaki
2138511 Japan
Voice: N/A
Fax: 408-428-9083
Email: mike@arraycomm.com
Voice: 571 275 7312
Email: reza.arefi@ieee.org
Voice: 408-952-1829
Fax: 408-428-9083
Email: chauvin@arraycomm.com
Voice: 908-997-2003
Fax: 908-947-7090
Email: arnab@flarion.com
office phone: +81-45-943-8019
fax phone : +81-45-943-6175
Email: shigeru_kimura@csg.kyocera.co.jp
Voice: +81-45-943-6102
Fax: +81-45-943-6123
Email: kazuhiro_murakami@csg.kyocera.co.jp
E-mail:yuzam@pb.jp.nec.com
tel:+81-44-820-4682(office)
fax:+81-44-820-4555
Re:
Evaluation Methodology: Phased Approach
Abstract
We propose a structured approach to the evaluation of proposals in which evaluation is conducted in
multiple phases with each phase progressively adding more complexity.
Purpose
Release
This document has been prepared to assist the IEEE 802.20 Working Group. It is offered as a basis
for discussion and is not binding on the contributing individual(s) or organization(s). The material in
this document is subject to change in form and content after further study. The contributor(s)
reserve(s) the right to add, amend or withdraw material contained herein.
The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this
contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to
copyright in the IEEE’s name any IEEE Standards publication even though it may include portions
of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in
part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that
this contribution may be made public by IEEE 802.20.
Patent
Policy
The contributor is familiar with IEEE patent policy, as outlined in Section 6.3 of the IEEE-SA
1
Standards Board Operations Manual <http://standards.ieee.org/guides/opman/sect6.html#6.3>
and in
Understanding Patent Issues During IEEE Standards Development
<http://standards.ieee.org/board/pat/guide.html>.
Notice
2004-01-04
1.
IEEE C802.20-04/004
Introduction
The evaluation criteria working group is chartered to evaluate different candidate air interface proposals for
the IEEE 802.20 standard, permitting an apples-to-apples comparison of proposals. The IEEE 802.20
standard defines operations at the Physical and Link layer of the ISO Model. Per the 802.20 PAR, the
physical and link layer must be optimized for the delivery of broadband IP services. Many of the
characteristics and aspects of the higher ISO layers are important for the evaluation of proposals, but
nevertheless, complicate the evaluation and simulation process. The many parameters and models
introduced in the simulation process will thus make it difficult to checkpoint different proposals against one
another and otherwise obfuscate the fair comparison of proposals.
We therefore propose a structured approach to the evaluation of proposals in which evaluation is conducted
in multiple phases with each phase progressively adding more complexity. The evaluation work for each
proposal may then be compared at each phase to ensure a progressive “apples to apples” comparison of
proposals. The group’s overall goal is also furthered with this structured approach by providing
performance metrics for the physical and link layer performance early rather than later in the evaluation
process.
2.
Phased Evaluation of Technology Proposals
The evaluation methodology should follow a 3-phase approach.
In the first phase, key system level performance metrics are gathered with simplified traffic, channel, and
network models. The emphasis of this phase is to build relatively simple simulation tools and develop air
interface performance results that can be easily understood and interpreted. The results from this phase
ensure the integrity of air interface simulations across proposals and provide the group with a good
understanding of each technology proposal, and the benefits of the key components of the air interface
techniques employed by each proposal. Outputs of this phase shall consist of many of the key system
capacity metrics such as user data rate CDFs and minimum service levels, aggregate throughput, number of
active users, and spectral efficiency at different load/coverage operating points. In this phase a limited set
of channel models should be used and higher layer considerations should be kept simple by for example
using a “hungry user” infinite queue user traffic model.
In the second phase, the impact of air interface behavior on the performance of higher layers is
incorporated into the evaluation. For example in the second phase, the impact of TCP could be modeled,
but network or backhaul delays would be ignored. The traffic models used to evaluate proposals could be
extended beyond the hungry user model to include HTTP or FTP. In addition, multiple channel models
could be considered and simulated in this phase.
In the final phase, the impact of air interface behavior on end-to-end performance is incorporated into the
evaluation. For example, the effect of network impairments could be modeled in this phase or the traffic
models could be made more sophisticated.
The details of the precise evaluations to be undertaken in each phase shall be determined by the evaluation
group, and then specified in the table below:
Phase
Phase-1
Traffic Model
Simplified e.g.. Hungry
user, full buffers.
Channel models
Network models
Simplified
Simplified
Simplified.
.
Extended
Phase-2
Extended
Extended
Phase-3
Extended
Extended
2
2004-01-04
IEEE C802.20-04/004
3.
References
[1]
802.20 Evaluation Criteria – Ver 06, IEEE C802.20-03/94
3