China Mobile: Setting the pace in IP network O&M
TextStart
Boasting more than 580 million users as of the end of 2010, China Mobile is one of the
largest mobile operators in the world. Always a pace-setter, the operator had built an
IP-based mobile network, taking the lead and preparing well for the data service growth.
Yet, as it entered this new stage, it still faced a number of challenges. For example, the
operator lacked sufficient measures to follow up or trace user complaints, while the
mechanism lagged behind in ensuring consistent QoE. In addition, good network KPIs
had not corresponded to good user experience, and an efficient and accurate system was
badly needed to optimize the complex and huge network architecture. Huawei’s
VisualIP has helped China Mobile efficiently manage its IP network, ensure QoS,
guarantee QoE, and optimize data traffic.
VisualIP: Efficient IP network assistant
On August 16, 2010, the network monitoring center of China Mobile Guangdong was
flooded with alarms with the ID number 3008. The QoS had been badly degraded, with
packet loss ranging 1.06 to 3.26%, which was far beyond the normal value. After
checking the network topology, the engineers found a number of routers had been
marked out by the system, indicating possible faults. With simple clicks, the router that
lost large amount of packets had been identified, and the whole troubleshooting process
took less than 10 minutes.
This was a typical case of packet loss resulting in disrupted communication between
core network equipment. For faults of this kind, China Mobile Guangdong now needs
only a few minutes for troubleshooting using a simple operation. However, the process
was more complex in the legacy system.
Difficult fault location
The O&M difficulties in a legacy system were obvious. When a fault arose, problems
like lowered connection rate and degraded voice quality caused user complaints. Based
on the time and location of the complaints, engineers needed to manually check
relevant equipment to narrow down the fault location, and then deliver the information
to specific network personnel for troubleshooting.
All the processes involved multiple departments and efficiency could be low
considering the large amount of coordination efforts. In addition, engineers needed to
be skillful in understanding multiple network technologies. More difficulties in fault
location would occur when the IP network produced random faults, and sometimes the
faults could be solved automatically.
Due to the advancing IP transformation, the traditional O&M mechanism could hardly
keep up with the requirements of an IP network. Analyzing the new O&M challenges,
China Mobile Guangdong actively explored a practical solution, instead of waiting for
the technology to mature. After in-depth research with vendors and other partners, the
operator raised three necessary features for the IP O&M system.
The first requirement was quick fault location, which would enable engineers to
quickly locate faults after receiving user complaints or detecting network/service faults.
The second was an enhancement of the user satisfaction level.
By monitoring network performance and service quality in real time, the operator
wanted to be able to locate the faults and analyze the trend to prevent future faults. The
third was efficient daily network management. By establishing a health evaluation
system for the All-IP network, the operator wanted to learn of the network performance
in real time.
According to Gartner, the top three requirements driving operator’s O&M investment
are active prevention of network faults, quick network troubleshooting, and service
level assurance (SLA). Forming more than 54% of the total investment, these
requirements also testified the correctness of China Mobile Guangdong’s O&M
strategy.
VisualIP becomes an ideal choice
To enhance its O&M capability, China Mobile Guangdong had trialed and deployed a
number of O&M systems for its All-IP network. As part of the effort, the Huawei
VisualIP solution greatly supported the O&M efforts and enhanced the efficiency.
An industry first, the VisualIP solution enables interaction between core network and
carrier network. By simulating real service routes and detecting the All-IP network, the
solution can enhance network performance and QoE. Gaining a clearer picture of the IP
carrier network, the operator can monitor both the core network and carrier network
and perform troubleshooting accordingly. Ideally matching the All-IP transformation,
the VisualIP O&M solution enhances efficiency and network quality.
Fault location time cut by 90%
Like a time bomb, random faults occur in traditional IP networks and are hard to locate,
with the fault either reappearing after a long time or never recurring. The VisualIP
solution can store the QoS information in real time, and engineers at China Mobile
Guangdong can view the history information at will, enhancing the O&M efficiency
greatly.
In a legacy network, network O&M always involved multi-department coordination
and it was difficult to define the responsibility. Through network status modeling and
intelligent diagnosis, VisualIP can help pinpoint the location and perform
troubleshooting. Thanks to the interactive management of both core network and
carrier network, the O&M responsibility is clearly defined and cross-departmental
coordination is minimized.
O&M changed from passive to active
Unlike TDM networks, All-IP networks may gradually accumulate problems, leading
to abnormal performance, or even breakdowns. It also takes time until a fault finally
shows up. The VisualIP solution monitors the network in real time and records statistics
of the IP network status. By checking and analyzing the health of the network, the
solution helps engineers prevent network faults, changing the O&M model from
passive to active.
Visualized network monitoring
In a TDM network, user experience is the standard to measure the success or failure of
a service. Yet the industry still lacks a unified standard to measure IP network quality.
The QoS concept, though helping solve part of the problem, does not adapt well in the
telecom field. Based on its rich IP network deployment and O&M experience, Huawei
launched the VisualIP solution, helping China Mobile Guangdong evaluate and
enhance its IP network performance.
Instead of showing complex and dull data in the daily network O&M, the VisualIP
solution automatically generates the network topology, in which equipment status is
displayed based on the level. In this context, network monitoring is visualized to the
specific interface of a network element.
QoE: A new benchmark to evaluate quality
QoE evaluation lags behind
Sceanario I: VIP customer Peter complained: “The signal is weak on my balcony and
in the corner of garage, while my calls are of poor quality and lose connection from
time to time.” After checking the network KPIs, the operator found that the call drop
rate was less than 0.3%. A further local DT test showed that the coverage was more
than 99%, and the mean opinion score (MOS) for voice was less than 3.6.
Generally reflecting the overall network performance instead of the QoE for an
individual or a hotspot area, traditional KPIs are lagging behind in this context.
Scenario II: After trying several times and waiting for a long time, Ana still failed to
connect to the video she loved through her mobile. Yet the operator still had good KPIs;
for example, the success rate for SMS push, forward, mobile originated (MO) SMS,
and mobile terminated (MT) SMS was over 99%, while the success rates for SDCCH
allocation, PDCH allocation, PDP activation, and page return were higher than 98.9%.
Good KPIs have not corresponded to equally good QoE, as traditional O&M systems
are designed for network elements (NEs) instead of the service quality for end users.
Various O&M units are run daily to ensure the sound performance of their respective
network sections, while ignoring end-to-end service quality and user experience.
Operators need a system to evaluate user experience, ensuring that the QoE of end users
can be measured based on quantitative data. This can help evaluate the overall network
performance, and maintain and optimize the network based on the QoE. Supplementing
the legacy KPIs, this system is key to enhancing service quality.
China Mobile is taking the lead in ensuring QoE. According to Liu Aili, Vice President
of China Mobile, the network KPIs should reflect not only the running status and
efficiency of equipment, but also the service quality.
According to Liu, China Mobile has gained rich experience in this field. With growing
services and the launch of its TD-SCDMA network, the operator found that network
quality had been increasingly affected by terminals, client software, networks, service
platforms, and their collaboration. The legacy O&M system focuses on the network
elements instead of the QoE of end users, and this leads to a substantial gap between
network KPIs statistics and real user experience.
Liu adds that NE-based KPIs are the basis for network O&M, and more importantly,
the operator needs a QoE-based O&M system that boasts comprehensive and
systematic performance indexes. In addition, it should move beyond traditional KPIs,
and work harder to improve QoE and enhance network performance in a
comprehensive way.
Creating a new benchmark
In February 2010, China Mobile and Huawei teamed up for QoE enhancement. By
analyzing user complaints, existing O&M status and the market environment, the two
parties identified the main reasons for the gap between the KPI statistics and real user
experience.
First, legacy measurement tools were limited and the KPI system was incomplete,
making it hard to evaluate the network coverage and user experience like voice quality
and one-way connection.
Second, legacy KPIs focused more on the network elements, instead of the end-to-end
service flow that closely impacted user experience. Though DT/CQT indexes could
reflect the network condition of areas like main trunk lines and hotspots, they could
hardly provide real-time monitoring and reflect the conditions of the whole network.
Based on in-depth research into QoS and QoE, China Mobile Sichuan established a
customer experience evaluation system as a supplement to the existing KPI system.
Referring to the methodology proposed by TMF, the system can evaluate whether the
SLA is satisfactory to users through the established KPI and key quality indicator
(KQI).
The system consists of two key parts: QoE indicator model and QoE evaluation model.
QoE indicator model: It uses a top-down and layer-by-layer approach to break down
and categorize the indicators. According to ETSI TS 102 250-1, QoE can be accessed
through accessibility, retainability and integrity. Then a further classification can be
done to identify the KQIs.
For the voice service, the three major KQIs are E2E accessibility, call drop rate
(retainability), and voice quality (integrity), ensuring that calls are well connected with
good quality.
To enhance voice quality, China Mobile added the VQI indicators and created a GSM
voice quality evaluation standard together with Huawei in November 2010. Defining
how to collect, calculate, compile statistics from, and report the VQI data in GSM voice
service, the standard lays the basis for collecting VQI statistics for the whole network.
For data services like WAP, there are two major KQIs to ensure service quality, which
include: 1) WAP access success rate, or refresh success rate (accessibility), and; 2)
refresh duration, which relates to the page size and download rate (integrity).
QoE evaluation model: QoE is subjective in nature, so a fuzzy evaluation is adopted
for measurement. When evaluating a complex subject, the indicators are layered based
on their priorities and evaluation is performed on each layer.
In June 2010, the joint project team reported the progress and achievements to the
China Mobile Group, including the QoE-KQI-KPI model, mapping relations and cases
in real network monitoring.
Based on an E2E signal monitoring system, the QoE evaluation system has been highly
recognized by the Group, which provides theoretical support for future development.
Enhanced O&M mechanism
In the second half of 2010, China Mobile Zhejiang began piloting its QoE system. To
enhance QoE, the operator combined methodology, O&M workflow and tools; it also
updated the KQI-KPI-PI evaluation system, including the voice and data services in the
GSM and TD-SCDMA networks.
Referring to the joint project team’s research, China Mobile Zhejiang established the
QoE indicator model and QoE evaluation model; other measures included: 1)
integrating seven indicators closely related to the voice service QoE into the KQI
system; 2) restoring and streamlining the KPIs/PIs that impact QoE; 3) delivering
definition, statistic methods for the NMS and signal monitoring, as well as the
measurement points.
The updated system has indicators that are streamlined and more closely related to QoE,
which helps O&M engineers focus on the key issues.
Adopting the SmartCare solution for E2E signal collection, China Mobile Zhejiang has
realized network-wide monitoring across networks and vendors.
By collecting statistics from CS and PS cores, wireless interface signaling, QoS data of
the carrier network, voice quality, and more, the operator has abundant data for its
KQI-KPI-PI evaluation, while enhancing the network performance based on QoE.
Besides establishing the VIP assurance system to improve complaint processing in real
time, China Mobile Zhejiang has also set up a mechanism to take care of the very
annoyed persons (VAPs). In addition, the system can analyze the variation and
development trend of the QoE indexes, helping to evaluate and manage the QoE in a
visualized manner.
The year 2010 saw China Mobile’s continuous effort to enhance QoE, which included
establishing the QoE evaluation system, promoting relevant research efforts, and
piloting in its provincial branches. The operator is also shifting its focus from NEs to
end users’ terminals, experience and evaluation. This helps deliver customer care
quickly before any complaint arises.
The shift of the O&M mindset, starting from the TD-SCDMA network, has been
implemented to the overall network and services of China Mobile. This further helped
the operator establish a QoE-based evaluation system with comprehensive and
systematic performance indexes, and enhance the network performance continually.
SmartCare elevates QoE
Challenges are growing in today’s network O&M. For example, there are insufficient
tools to follow up on growing user complaints; it is hard to assure the service quality of
VIP customers; good network KPIs do not correspond to equally good user experience;
the network optimization system lacks the tools to ensure an increasingly complex
network architecture.
After adopting the SmartCare platform, the China Mobile branch in Shaoxing, a city in
China’s Zhejiang Province, has greatly enhanced its complaint handling efficiency and
accuracy. By fully integrating network optimization and the O&M service, the operator
has also substantially enhanced QoE.
Challenges in five aspects
Network KPIs needed improvement
KPI monitoring is an essential element of network O&M. Facilitating the evaluation of
network performance, KPIs can also help detect potential problems and guide network
optimization. After years of development, China Mobile had developed mature KPI
systems, yet more indicators could be added to enhance QoE, including the indicators
for end users, specific services, and E2E service provisioning.
Complaint handling incurred high cost
Efficient complaint handling can improve user satisfaction and loyalty. After receiving
a customer complaint through the 10086 hotline, the service personnel transferred the
complaint to relevant engineers to identify where the fault occurred, and to which cell it
belonged. If the complaint was a new one, then field tests were arranged. Generally,
more than 95% complaints required field tests, which incurred high costs.
Difficult QoS assurance for VIP users
VIP users like enterprise users are prioritized by China Mobile Shaoxing. Traditional
O&M focused on the cells where VIP users were active, and if a VIP user went beyond
that cell coverage or experienced any fault, the operator found it hard to locate that user.
If the service quality of an individual user could be assured, the operator could discover
the potential problems and do troubleshooting in advance. Even if the customer
complained, China Mobile Shaoxing could deliver enhanced service by solving or
clarifying the issue in a timely manner.
Daily O&M demanded more
The daily O&M tasks of China Mobile Shaoxing mainly include optimizing the
least-satisfactory cells and solving the problems detected in road tests, which help to
ensure a healthy network. Legacy daily maintenance was based on the statistics from
NEs, showing only the overall performance and was not specific to individual users. To
improve service quality, the operator needed a system that reflected comprehensive
data from both NEs and users.
Complex network optimization
A special optimization of GSM network mainly included frequency and neighborhood
optimization. Traditional network optimization needed highly skillful engineers and the
efficiency was low, as engineers were required to have rich experience and be familiar
with the wireless network.
China Mobile Shaoxing did a calculation: A neighborhood optimization containing five
BSCs required two skilled engineers with 10 working days, while frequency
optimization demanded more. In this context, an efficient and accurate system was
needed for frequency and neighborhood optimization.
Enhancing QoE through SmartCare
In November 2010, China Mobile Shaoxing decided to integrate its O&M process with
SmartCare, laying a sound basis to enhance QoE.
KQI is introduced
China Mobile Shaoxing divided and managed the network performance indicators by
three layers: key quality indicator (KQI), key performance indicator (KPI) and
performance indicator (PI). The QoE was measured through accessibility, integrity and
retainability, ensuring the objectiveness and E2E feature. Besides adding overall
monitoring to the voice service, the operator also leveraged the SmartCare solution to
realize KQI-based monitoring and enhance O&M efficiency.
Easy handling of user complaints
The efficiency of handling complaints can partly reflect an operator’s service and
technology competency. Efficient and low-cost complaint handling helps enhance
brand image and competitiveness. China Mobile Shaoxing integrated its complaint
handling process with SmartCare. After receiving a complaint through the 10086
hotline, the service personnel could trace the communications history, and identify the
time, cell and signaling processes related to the complaint. This could help handle
complaints and enhance efficiency, while reducing OPEX and the number of field tests.
Timely VIP assurance
VIP assurance is to monitor in real-time the indicators of VIP users, perform fast
troubleshooting, and clarify complaints in a timely manner. Using SmartCare, China
Mobile Shaoxing assured the services for nearly 5000 VIP users. Dividing the users
into 21 groups, the operator could handle the abnormalities in a timely manner,
reducing user complaints. In addition, problems could be clarified in time if any user
complained.
Identifying VAP users and cells
Traditionally, cells with poor KPIs were identified through NEs. SmartCare can
monitor the network quality based on user-related indicators and identify the VAP cells
that deliver poor user experience.
China Mobile Shaoxing and Huawei established the specifications to identify the VAP
users, and focus on and solve relevant issues. The SmartCare also helped to identify and
solve some potential problems, reducing complaints and the number of possible VAP
users.
Efficient network optimization
SmartCare provides MR-based neighborhood and frequency optimization; advanced
and accurate, this function is efficient and has lowered the requirement on engineers.
Thanks to SmartCare, China Mobile Shaoxing enhanced its O&M efficiency greatly
during daily operations.
Fruitful results
The QoE system would inevitably impact the China Mobile Shaoxing’s legacy O&M
process. For example, the O&M engineers would need to learn the new process and
tools. The operator joined hands with Huawei to enhance the training effort and
properly implemented the new process. After one-and-a-half months of work, starting
from December 1, 2010, the QoE system was well integrated and had been running
smoothly.
China Mobile Shaoxing optimized the KQIs based on the routine monitoring results.
Compared with the last month, January 2011 saw a steady growth in network quality.
For example, the E2E call completion rate was up 1.4 percentage points to 95.1%, VQI
score increased 0.03 to 3.98, and the downlink HQI was enhanced by 1.15%.
The operator received over 300 complaints about voice service through the 10086
hotlines. The number of field tests had been reduced from 95% to 75%, and the
complaint processing efficiency had been enhanced, as some had been handled directly.
Based on accessibility, integrity and retainability, the daily network optimization
identified the users with less-satisfactory QoE. As a result, 108 VAP users in 22 cells
had been indentified, among which the problems of 86 VAP users in 15 cells had been
solved.
China Mobile Shaoxing optimized the neighborhood and adjusted the local frequency
twice a week, enhancing the network quality steadily. For example, the downlink HQI
had been enhanced from 96.50% to 97.65%, the success rate of switching had been
improved from 98.8% to 99.1%, and the work efficiency had been enhanced 500%.
TextEnd