Enterprise Network Services Overview
Online Services for the Enterprise
Published: January 2010
For the latest information, see www.microsoft.com/online.
The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of
publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of
Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.
This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO
THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document
may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical,
photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this
document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any
license to these patents, trademarks, copyrights, or other intellectual property.
©2009 Microsoft Corporation. All rights reserved.
Microsoft, Bing, Hotmail, MSN, and Windows Live are trademarks of the Microsoft group of companies. All other trademarks are property of their
respective owners.
Microsoft Online Services | Network Service Description
2
Contents
Introduction ................................................................................................................................................. 4
Network Architecture .................................................................................................................................. 5
Inter-DC Network (LAN) ............................................................................................................................ 6
Microsoft Backbone Network .................................................................................................................... 7
Edge Network ........................................................................................................................................... 8
Packet Flow .............................................................................................................................................. 8
Connectivity Design Principles ............................................................................................................... 10
Network Security ....................................................................................................................................... 11
Internet Security ...................................................................................................................................... 11
Separation (Compartmentalization) ........................................................................................................ 11
Appendix A: Read More About Microsoft Online Services Standard Offerings .............. 14
Microsoft Online Services | Network Service Description
3
Introduction
This document describes the Microsoft Online Services networking infrastructure components and
security features that support delivery of all Online Services for the enterprise that use the Internet for
transport. These include all of the offerings that are part of the Microsoft Business Productivity Online
Standard Suite (Microsoft Exchange Online, Office Live Meeting, Office Communications Online, and
SharePoint Online), Microsoft Dynamics CRM Online, and many others. The document is intended for
network engineers and system integrators who work with Microsoft Online Services customers.
(Dedicated offerings are covered in a separate downloadable document, Microsoft Online Dedicated
Service Descriptions and Service Level Agreements).
The components and features that are described include:

Enterprise network architecture for Microsoft Online Services

Microsoft Internet connectivity

Network security
Microsoft is constantly investing in new technologies, expansion, and innovation of its network
infrastructure. This document is updated regularly to reflect changes that are deployed to the network to
support Microsoft Online Services.
Microsoft Online Services | Network Service Description
4
Network Architecture
The network architecture for Microsoft Online Services was designed specifically to support enterpriselevel services and applications. It was built around the pillars of performance, stability, security,
redundancy, and scale. The network can be separated into three distinct functional sections: the intraData Center (intra-DC) local area network (LAN) environment, the global backbone, and the edge
network for Internet connectivity. Some of the highlights of this architecture are noted here:

Based on a layered LAN architecture that allows for best-in-class technologies and equipment to
be deployed in an independent (of layer) manner.

Provides the necessary functionality to enable virtualization for each Online Service within a data
center.

Enables the sharing of multiple services on the same physical hardware, thus optimizing costs
and utilization.

Uses the Microsoft global backbone to enable connectivity to thousands of Internet service
providers (ISPs). This means that packets exit closer to the customer and in an optimized
manner.

Utilizes multiple means of abstraction and checkpoints to help ensure that only desired traffic is
allowed.

Uses redundant, very high-capacity links throughout, to help ensure stability and performance.
The overall network is illustrated in Figure 1.
Data Center
Internet
Edge
Edge
Router
Edge
Router
Anchor Site
Anchor Site
Internet
Core
Core
Router A
Core
Router B
Data Center
Data Center
Data Center
Router A
Access
Layer 3
Data Center
Router B
Access
Router B
Access
Router A
Layer 2
Aggregation
Anchor Site
Switch A
Switch B
Internet
Load Balancer A
Load Balancer B
Firewall A
Firewall B
TOR Switches
TOR Switches
S
E
R
V
E
R
S
S
E
R
V
E
R
S
TOR Switches
... .
Data Center
S
E
R
V
E
R
S
Anchor Site
Top of Rack/ Servers
Figure 1: Microsoft Online Services network architecture
Microsoft Online Services | Network Service Description
5
The primary components of the Microsoft Online Services network architecture are discussed in detail in
the following sections.
Intra-DC Network (LAN)
The intra-DC network provides connectivity to the servers that host the applications that make up a given
Online Service. This is illustrated in Figure 2 below.
The first layer consists of servers that host the Online Service applications, which are located in racks that
contain two top-of-rack (TOR) switches. Each server is connected via separate network interface
controllers (NICs) to each switch. Every switch has two connections into the Layer 2 Aggregation.
Layer 2 Aggregation exists to consolidate many racks, and also to host shared services such as load
balancing and firewalls. The advantage of hosting these devices in this layer is that they can be shared
across racks and services with no change in physical topology.
The next layer, Layer 3 Aggregation, is the main routing layer for all virtual LANs (VLANs); it is where IP
address blocks are configured. Each service terminates within a dedicated Virtual Routing and
Forwarding (VRF) instance, and the routers are virtualized using VRF Lite. The routers also connect to
the backbone network to provide connectivity to internal Microsoft administrative networks and to the
Internet. Note that each Online Service resides on dedicated LANs and is physically separated from all
other Microsoft services such as Windows Live Hotmail Web-based e-mail service. Security checkpoints
to these LANs treat internal Microsoft traffic in the same manner as Internet traffic; in other words, internal
traffic is not trusted any more than external traffic. This provides an additional level of security and
abstraction.
Microsoft Online Services | Network Service Description
6
Admin/
Mgmt
Internet
L3 Aggregation
Gateway Multi-Service
Router w/ VRF A
Gateway Multi-Service
Router w/ VRF B
L2 Aggregation
Firewall A
Load
Balancer A
Firewall B
L2 Switch A
L2 Switch B
Top-of-Rack
Switch 1
Load
Balancer B
Top-of-Rack
Switch 2
Servers
Servers/Host
Figure 2: Intra-DC network architecture
As can be seen in Figure 2, redundancy and high availability are central themes. Two devices are used
for routing and switching functions, and all connections are on a redundant basis. Firewall and loadbalancer deployments use duplicate systems with automatic failover. Each service rack has two separate
network connections and two individual power feeds to help ensure availability. Each data center network
stamp has redundant, high-capacity (n x 10GE) links into the Microsoft backbone. These links provide
protected connectivity to the Internet edge network and to other Microsoft locations.
Microsoft Backbone Network
The Microsoft backbone network, also known as the core network, provides high-bandwidth, low-latency
connectivity to other Microsoft data centers and to the edge of the Microsoft network. Microsoft has a
global network that takes advantage of modern technologies such as multi-protocol label switching
(MPLS) and dark fiber with dense wavelength division multiplexing (DWDM) to deliver this level of
connectivity. The backbone also has connectivity to many major "carrier hotels," data centers that are
used to connect to ISPs, carriers, and other enterprises. These are called anchor sites in Figure 1. The
backbone network is designed to handle the massive amount of traffic that is generated, getting it to its
destination as fast as possible. The backbone network also carries inter-DC traffic, in addition to Internet
and other external traffic.
Microsoft Online Services | Network Service Description
7
Edge Network
The final component of the network is the edge network, which is used for Internet connectivity. Microsoft
is one of the largest traffic destinations on the Internet due to the broad range of Microsoft hosted
services such as the MSN network of Internet services, Microsoft.com, and Bing. Given the enterprise's
desire for high performance and redundancy, Microsoft has an aggressive and open policy to solicit and
connect with as many ISPs and enterprises as possible. This has been done on a global basis using
direct, private connections and via membership in public exchange points such as LINX, PAIX, and
Equinix. All of these efforts have gained Microsoft a position as one of the "top five best connected
networks in the world," according to FixedOrbit.com. The advantage this brings to our customers is being
close (in hops) to the Microsoft services that they are using. In addition, service quality is continuously
being improved by provisioning multiple links to ISPs in different geographies and implementing optimal
routing policies. Finally, link utilization is constantly monitored and capacity upgraded as needed. Figure 3
illustrates the connectivity strategy for a given data center.
Route Collection
Internet
Internet
Internet
Data Center
Backbone
Metro
Internet
Data Center
Route Collection/Internet
Anchor Site
Data Center
Microsoft links
Transport
Figure 3: Edge network architecture
As mentioned before, anchor sites are carrier hotels that are used to connect to ISPs and exchanges.
Because the main data center may not be in a favorable location for connectivity to a broad range of
ISPs, a metro network is used to transport traffic between the anchor site and the DC. This is provisioned
using multiple redundant high-capacity links.
Packet Flow
Figure 4 presents a logical view of the Microsoft Online Services network architecture, and depicts how
packets flow through it.
Microsoft Online Services | Network Service Description
8
Internet
Customer
Backbone &
Edge Routing
Network Security
Policy Enforcement
Point
Load
Balancing
Multi-Service
Access
Router (MAR)
Management
Network Security
Policy Enforcement
Point
Enterprise
Service
Figure 4: Logical network architecture of Microsoft Online Services
A Microsoft Online Service is provided out of the Enterprise Service cloud, which is made up of the racks
of servers, TORs, and aggregation switches. Flows coming into this cloud can be load-balanced if
needed. Note that the load balancer can also provide additional functionality such as network address
translation (NAT) and Secure Sockets Layer (SSL) offload. The Management cloud contains the servers
and applications that are used by Microsoft to administer and manage servers in the Enterprise Service
cloud. Security features help ensure that only trusted flows are allowed.
For packets bound for the customer, the first ones arrive in the multi-service access router (MAR) inside
of the VRF instance for the service. Customer-bound packets are sent to a firewall that provides for deep
packet inspection of flows. If allowed, the packets are sent to the customer via the backbone and edge
networks, which provide connectivity to the Internet. Routing optimizations are implemented to help
ensure that the best path is used to reach the customer.
Microsoft Online Services | Network Service Description
9
Connectivity Design Principles
Microsoft Online Services customers should keep in mind the design factors of reliability, capacity, and
latency when planning network connectivity to Microsoft data centers. Note that all services (including
those that are not specifically Online Services) are accessed over the Internet with no specific transport
requirements such as dedicated circuits or virtual private networks (VPNs).

Reliability: Microsoft has very robust and broad connectivity to most Tier 1 and Tier 2 ISPs
globally. This means that multiple paths are available to reach a given destination network. In
addition, redundancy is implemented at all levels of the network, including equipment and links.
We strongly recommend that the customer connect to at least two separate ISPs for access to
Microsoft Online Services. Multiple ISP connections provide the redundancy required to help
ensure that users have uninterrupted access to critical services at all times.

Capacity: Regardless of transport method, it is critical that the customer perform initial planning
and ongoing capacity analysis to help ensure that adequate bandwidth is available for reaching
Microsoft Online Services at all times. These processes require accurate prediction of bandwidth
demand and ensuring that proper measuring tools are in place to monitor usage. Access to
Microsoft Online Services may be impacted if the same link is used for access to Microsoft as
well as for general Internet traffic. For example, flash traffic may overwhelm traffic that is destined
for Microsoft Online Services, which can cause degraded network service or lack of access. We
recommend that the customer provision separate links for Internet access and Online Services
access.

Latency: Latency is a critical network factor that directly affects perceived and actual
performance of a given Microsoft Online Service. Each Online Service provides general guidance
for acceptable round-trip time (RTT) between the customer and Microsoft Online Services. When
provisioning, tests must be conducted ahead of time to help ensure that RTT is within acceptable
tolerances.
Microsoft Online Services | Network Service Description
10
Network Security
Because Microsoft Online Services manages multiple customer environments from a single management
space, network infrastructure controls are specifically designed to help ensure the confidentiality and
integrity of customer data through strict compartmentalization. Under no circumstances is access
permitted between one environment and another. Any traffic that is not part of the Online Service is
treated in a non-trusted manner, including internal Microsoft traffic. The Online Services network also
enables reliable data availability through equipment redundancy, resiliency, and industry-standard highavailability design practices.
Internet Security
Microsoft Internet connections are used to transport traffic for various Microsoft Online Services. Microsoft
applies a rich set of security controls and optimizes routing to help ensure the desired level of
performance. In particular, three levels of security are implemented to prevent unwanted traffic from
entering the Microsoft network or the VLAN of the service.
1. As traffic heads toward the VLAN, two sets of network filters allow only authorized networks on
given ports and protocols to reach the servers for a given Online Service.
2. At the router, security by abstraction obscures the routes and allows only authorized traffic to
pass through. Because virtualization is used on the router level, only the needed routes are
present in the routing table of the Online Service. Thus, routing is not available to any other
destination and must pass through the firewall for validation.
3. All unrecognized traffic is routed to the firewall, where specific rules govern the type of traffic that
is allowed to pass through on a stateful basis. Any traffic that does not meet the firewall rule list is
simply dropped.
In addition to this three-tiered security, there’s a final checkpoint in data centers: only servers that are
managed by Microsoft and configured for Internet access can receive Internet traffic.
Separation (Compartmentalization)
One key strategy that Microsoft Online Services uses to maintain the confidentiality and integrity of
customer data is compartmentalization. Multiple techniques are used to control information flows between
the various clouds shown in Figure 5:

Network separation: Network segments are physically separated by virtualized routers that are
configured to prevent communications between Online Services unless otherwise desired.
Routing for all Online Services is, in effect, on a "need-to-know" basis. Further, all networks
outside of a given Online Service, including other Microsoft Online Services, are treated as an
external environment, just as the Internet is.

Logical separation: VLAN technology is used to further separate communications between
Customer Network and Managed Network segments.

Firewalls: Firewalls and other network security enforcement points are used to limit data
exchanges with servers that are exposed to the Internet, and to isolate systems from the backend systems that are managed by Microsoft.

Protocol restrictions: Only known and required services and applications can be used to access
servers on an Online Service network from the Management network. The access is restricted by
strict policy filters.
Figure 5 illustrates these connections and associated restrictions.
Microsoft Online Services | Network Service Description
11
Network Security Policy Communication Flows
Internet
Customer
Management
Enterprise
Optional
Never allowed
Service
Controlled by policy
Allowed – No network policy (customer policy only)
Figure 5: Online Services network communication flows
Figure 6 illustrates the separation of Microsoft Online Services from other networks and enforcement
points.
Microsoft Online Services | Network Service Description
12
Network Security
Enforcement Point
Internet
e
Int
rn
ffic
Tra
l
a
Network Security
Enforcement Point
Microsoft
Backbone
- Includes
services such as
Bing and Hotmail
All Other
Services
Network
Security
Enforcement
Point
- Global reach
- Very high performance
- Redundant, scalable, highcapacity links
- Purpose-built, enterprise
network architecture
- All services exist inside separate
virtualized instances
- Redundant, high-capacity uplinks
- All external traffic (including
Microsoft) is treated as non-trusted
Microsoft
Online
Services
- Imaging
- Backup
- Patching
- Server management
Network Security
Enforcement Point
Admin/
Management
Figure 6: Separation of the Microsoft Online Services network
Microsoft Online Services | Network Service Description
13
Appendix A: Read More About Microsoft Online Services
Standard Offerings
The following links provide additional detail about Microsoft Online Services Standard offerings.
Main Web site detailing the Standard offerings [Link]
Contains useful information about the Microsoft Business Productivity Online Standard Suite.
Get Started with Business Productivity Online Standard Suite [Link]
Contains useful information about the Microsoft Business Productivity Online Standard Suite.
Microsoft Office Live Meeting Service Description
Office Live Meeting is an enterprise-class Web conferencing service. With Office Live Meeting,
organizations can engage customers through real-time meetings, training sessions, and events presented
over the Internet. Office Live Meeting operates in an infrastructure separate from Microsoft Online
Services. However, Microsoft Online Services provides consulting services to help organizations
efficiently adopt and begin using the Office Live Meeting service.
Security in the Business Productivity Online Standard Suite from Microsoft Online Services [Link]
This white paper describes the security and reliability features of the Business Productivity Online
Standard Suite from Microsoft Online Services. It details the capabilities, technologies, and processes
that are used, and examines how the experience of Microsoft in building and operating enterprise
software has led to the demonstrated reliability of its Microsoft Online Services offerings.
Guidance for Microsoft Online Services Multinational Customers [Link]
This white paper is targeted at IT professionals in a multinational company who are interested in
evaluating Microsoft Online Services. It provides the tools and guidance for using Microsoft Online
Services from multiple locations worldwide, both during an evaluation stage and after purchase, when
Microsoft Online Services are introduced for the first time to the multinational company’s branch offices in
different regions.
Microsoft Online Services | Network Service Description
14