Paul Collinge
Senior Premier Field Engineer
Stage 1 Performance troubleshooting engineer not required
1.
•
•
•
•
•
2.
•
•
3.
•
4.
•
Stage 2 Performance troubleshooting engineer is required
5.
6.
7.
8.
•
•
•
•
•
•
•
•
•
•
•
• Latency/Round Trip Time
• TCP Window Scaling
• GEO DNS issues
•
•
•
•
•
•
•
Routing and Peering
TCP Idle time settings
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Measuring RTT from the Proxy to Office 365
•
•
•
•
•
•
Client
Proxy
Office 365 Datacentre
54.88ms
0.346ms
Internal RTT (ms)
External RTT (ms)
Total RTT to O365
54.88
346
400.88
Here we can see clearly, the poor RTT is outside the customer’s environment, on the ISP link to
Office 365. If this RTT is unexpected, the customer can engage their ISP to investigate.
•
•
•
•
•
•
•
•
http://www.verizonenterprise.com/about/network/latency/
TCP Window Scaling
enabled?
Maximum TCP receive buffer
(Bytes)
No
65535 (64k)
Yes
1073725440 (1gb)
Impact of TCP Window Scaling
Presuming a 1000 Mbps link here is the maximum throughput we can get with TCP window scaling disabled
and then with it enabled
Round Trip Time (ms)
Maximum Throughput
(Mbit/sec) without scaling
300
1.71
447.36
200
2.56
655.32
100
5.12
1310.64
50
10.24
2684.16
25
20.48
5368.32
10
51.20
13420.80
5
102.40
26841.60
1
512.00
134208.00
Maximum Throughput
(Mbit/sec) with scaling
Impact of enabling this setting
Before & After correctly enabling TCP Window Scaling- Download a 14mb PDF
600.0
500.0
507.0
Seconds
400.0
300.0
200.0
100.0
21.0
0.0
Australia Proxy (Incorrect Windows Scaling settings)
Australia Proxy TCP Window Scaling enabled
Office 365
Office 365
Australia PC
Australia PC
Australian PC downloading a large PDF before and after correctly setting the TCP Window Scaling on the Proxy
Internet
egress point
• This trace is from the UK
• Look for hosts called NTWK.MSN.NET
which is the Microsoft global network
• In the example we hit the MSFT
network in 10ms
• Then traverse the USA via
• ASH-Virginia
• ATB-Georgia
• LAX-Los Angeles
• Then to APAC via
• TYA - Tokyo
• SIN-Singapore
Tracing route to OUTLOOK-APACNORTH.OFFICE365.COM [132.245.65.146]
over a maximum of 30 hops:
1
1 ms
1 ms
1 ms SkyRouter.Home [192.168.0.1]
3
11 ms
11 ms
11 ms ip-89-200-132-100.ov.easynet.net [89.200.132.100]
4
10 ms
10 ms
10 ms igbtmdistc7503.msft.net [195.66.236.140]
5
84 ms
84 ms
84 ms xe-0-3-2-0.ash-96cbe-1a.ntwk.msn.net [207.46.45.227]
6
96 ms
95 ms
95 ms ae2-0.atb-96cbe-1a.ntwk.msn.net [207.46.33.228]
9 140 ms 142 ms 140 ms 191.234.83.150
10 142 ms 138 ms 139 ms ae11-0.lax-96cbe-1b.ntwk.msn.net [207.46.47.11]
11 256 ms 256 ms 256 ms ae2-0.tya-96cbe-1a.ntwk.msn.net [207.46.46.149]
12 265 ms 265 ms 265 ms ae0-0.tya-96cbe-1b.ntwk.msn.net [204.152.140.181]
13 288 ms 290 ms 292 ms xe-7-0-1-0.sin-96cbe-1a.ntwk.msn.net [207.46.38.252]
14 290 ms 288 ms 287 ms xe-5-3-1-0.sin-96cbe-1b.ntwk.msn.net [207.46.41.39]
15 279 ms 279 ms 279 ms ae1-0.sg2-96cbe-1a.ntwk.msn.net [191.234.80.90]
18 280 ms 280 ms 279 ms 132.245.65.146
•
•
•
•
•
TCP Idle time settings
•
•
•
•
•
•
•
•
•
•
•
Initial connect:
14:12:24.6483418 19.0046514 0.0003578 iexplore.exe 10.200.30.40 btssig-msl.bcp-01.Contoso.sig HTTP:Request, CONNECT
Contosoemeamicrosoftonlinecom-3.sharepoint.emea.microsoftonline.com:443 , Using NTLM Authorization NTLM NEGOTIATE MESSAGE
Proxy Response:
14:12:24.6876389 19.0439485 0.0283000 iexplore.exe btssig-msl.bcp-01.Contoso.sig 10.200.30.40
HTTP:Response, HTTP/1.1, Status: Proxy authentication required, URL: Contosoemeamicrosoftonlinecom-3.sharepoint.emea.microsoftonline.com:443
NTLM CHALLENGE MESSAGE
We then send the request again, this time with NTLM authentication for the proxy: Contoso Office 365 Network Review
Second request with NTLM Auth:
14:12:24.6883198 19.0446294 0.0004838 iexplore.exe 10.200.30.40 btssig-msl.bcp-01.Contoso.sig HTTP HTTP:Request, CONNECT
Contosoemeamicrosoftonlinecom-3.sharepoint.emea.microsoftonline.com:443 , Using NTLM Authorization NTLM AUTHENTICATE MESSAGE
Version:NTLM v2, Domain: headoffdom, User: paul.collinge, Workstation: W7TEST20
200 OK response from proxy but this takes 3 seconds.
14:12:27.7859643 22.1422739 3.0878394 iexplore.exe btssig-msl.bcp-01.Contoso.sig 10.200.30.40 HTTP HTTP:Response, HTTP/1.1,
•
http://support.microsoft.com/kb/2637629
•
•
•
•
•
•
•
•
•
•
•
DNS Performance
•
•
•
•
•
•
•
13:52:52 16/04/2013
31.2765664
0.0000000
10.200.30.40
10.214.2.129
DNS:QueryId = 0xE41, QUERY (Standard query), Query for Contosoemeamicrosoftonlinecom3.sharepoint.emea.microsoftonline.com of type A on class Internet
13:52:56 16/04/2013
35.0579179
3.7813515
10.214.2.129
10.200.30.40
DNS:QueryId = 0xE41, QUERY (Standard query), Response - Success, 157.55.232.50, 2.22.230.131 ...
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Multiple things in the 3 way handshake to the proxy/Office 365 should be checked to ensure they are
optimal
Check the TCP options in both the syn and the syn ack
TCPOptions:
- MaxSegmentSize: 1
type: Maximum Segment Size. 2(0x2)
OptionLength: 4 (0x4)
MaxSegmentSize: 1460 (0x5B4) //Max is 1460 bytes on Ethernet, shouldn’t be much lower than this. If it is, it will
cause poor performance
+ NoOption:
+ WindowsScaleFactor: ShiftCount: 8
+ NoOption:
+ NoOption:
+ SACKPermitted: //This should be enabled as It allows us to better handle dropped packets
http://blogs.technet.com/b/onthewire/
Network Analysis tool
http://msdn.microsoft.com/en-us/library/dn850362.aspx
http://channel9.msdn.com/Events/TechEd
www.microsoft.com/learning
http://microsoft.com/technet
http://developer.microsoft.com