® IBM Software Group TCP/IP Configuration and Diagnosis with WebSphere MQ: Part I Justin Fries WebSphere® Support Technical Exchange IBM Software Group Agenda Introduction Internet Protocol Transmission Control Protocol WebSphere MQ Channels The TCP KeepAlive Option Questions and Answers WebSphere® Support Technical Exchange 2 IBM Software Group Network Introduction WebSphere® Support Technical Exchange 3 IBM Software Group Hardware & Ethernet Hardware: Physical Layer Data is transmitted as bits Cables, hubs, connectors, pinouts, voltages Ethernet: Data-Link Layer Data is transmitted as frames of limited size MTU: Maximum Transmission Unit is usually 1500 bytes NIC: Network Interface Cards have a hardware address MAC: Media Access Control addresses are (mostly) unique Frames are routed between NICs by MAC address Frames contain data from the upper layers WebSphere® Support Technical Exchange 4 IBM Software Group Internet Protocol IP: Network Layer Data is transmitted as packets Two protocol versions: IPv4 and IPv6 IP Packets Maximum size is limited by the MTU May be fragmented if they are too large May be lost or dropped in the network May get delayed and arrive out of order Routed between machines by IP address Contain data from the upper layers WebSphere® Support Technical Exchange 5 IBM Software Group IPv4 and ICMPv4 IPv4 Defined by RFC791 (1981) Addresses are 32 bits long •Written as: •Loopback: 192.168.1.100 127.0.0.1 Maximum packet size is 64KiB Header is variable: 20+ bytes ICMPv4 Internet Control Message Protocol Defined by RFC792 (1981) Information and error packets WebSphere® Support Technical Exchange 6 IBM Software Group IPv6 and ICMPv6 IPv6 Defined by RFC2460 (1998) Addresses are 128 bits long: 2001:0db8:0000:0000:0000:0000:c980:00b4 •Shortened: 2001:db8::c980:b4 •Loopback: ::1 Maximum packet size is 64KiB Optionally packets may go up to 4GiB Header is a fixed size: 40 bytes ICMPv6 Defined by RFC4443 (2006) Information and error packets WebSphere® Support Technical Exchange 7 IBM Software Group Useful IP Tools ping All systems Uses ICMP echo request/echo reply packets traceroute, traceroute6 Most systems tracert Windows only Manipulates TTL/Hop Limit to map the network route Sends out packets with TTL values of 1, 2, 3, 4… Watches for ICMP “Time exceeded” responses tracepath Linux only Simplified traceroute that adds MTU discovery pathping Windows only Creates combined ping+traceroute reports WebSphere® Support Technical Exchange 8 IBM Software Group Sample Output: pathping C:\> pathping bach.raleigh.ibm.com Tracing route to bach.raleigh.ibm.com [192.168.11.97] over a maximum of 30 hops: 0 paterno.austin.ibm.com [192.168.248.159] 1 bryant.austin.ibm.com [192.168.248.3] 3 192.168.4.14 4 franck.raleigh.ibm.com [192.168.11.42] 5 bach.raleigh.ibm.com [192.168.11.97] Computing statistics for 300 seconds... Source to Here This Node/Link Hop RTT Lost/Sent = Pct Lost/Sent = Pct 0 0/ 100 = 0% 1 1ms 0/ 100 = 0% 0/ 100 = 0% 0/ 100 = 0% 2 19ms 0/ 100 = 0% 0/ 100 = 0% 7/ 100 = 7% 3 104ms 57/ 100 = 57% 50/ 100 = 50% 3/ 100 = 3% 4 22ms 10/ 100 = 10% 0/ 100 = 0% Address paterno.austin.ibm.com [192.168.248.159] | bryant.austin.ibm.com [192.168.248.3] | 192.168.4.14 | franck.raleigh.ibm.com [192.168.11.42] | bach.raleigh.ibm.com [192.168.11.97] Trace complete. WebSphere® Support Technical Exchange 9 IBM Software Group Transmission Control Protocol TCP: Session Layer Data is transmitted as segments Each segment fits within an IP packet: “TCP/IP packet” TCP Segments MSS: Maximum segment size is limited by the MTU Retransmitted by the system if they are lost or dropped Rearranged to the right order if received out of sequence Any duplicate or nonsense segments are dropped Contain data from the upper layers WebSphere® Support Technical Exchange 10 IBM Software Group Transmission Control Protocol TCP Defined by RFC793 (1981) Connection oriented: Two endpoints exchanging data Full duplex: Data flows both ways Header is variable: 20+ bytes Options: Timestamps, MSS, Window scaling, SACK, etc. Adds reliability on top of IP Highly stateful and complex WebSphere® Support Technical Exchange 11 IBM Software Group User Datagram Protocol UDP: A partner to TCP Defined by RFC768 (1980) Packets are called datagrams Header is a fixed size: 8 bytes Connectionless and stateless No reply, no acknowledgement Flinging packets into the void Briefly supported by WebSphere® MQ on AIX only Added by MQSeries 5.1 and removed by WebSphere MQ 6.0 Used by client devices running Windows 3.1/95/98/ME/NT WebSphere® Support Technical Exchange 12 IBM Software Group Source & Destination Ports Port numbers identify a partner endpoint on a machine Every TCP/IP connection is identified by its two endpoints Client IP address Client port number Server IP address Server port number Rather like making a phone call between two hotels IP address: The main hotel phone number Port number: The room you want to reach Some port numbers are restricted The well-known ports 1…1023 need super user access WebSphere® Support Technical Exchange 13 IBM Software Group Sequence Number Startup of a TCP Connection Each side chooses an Initial Sequence Number (ISN) For security reasons, ISNs are randomly chosen • Client initial sequence number: • Server initial sequence number: 58501256 4190910085 Established TCP Connections The sequence number counts how many bytes are sent Sequence numbers do wrap around (32-bit values) • After 10MiB, client sequence is: • After 100MiB, server sequence is: WebSphere® Support Technical Exchange 68987107 800390 14 IBM Software Group Acknowledgement Number The sequence number of the next expected byte During normal operation this number increases over time Server sends sequence number 800390 + 100 bytes Client sends acknowledgment number 800490 If a packet goes missing, repeat the last acknowledgment Server sends 800490 + 100 bytes …which goes missing... Server sends 800590 + 100 bytes Client acks 800490 Server sends 800690 + 100 bytes Client acks 800490 Server sends 800490 + 300 bytes Client acks 800790 WebSphere® Support Technical Exchange 15 IBM Software Group Selective Acknowledgment (SACK) Option which allows non-contiguous data to be acknowledged Allows just the holes to be filled in by retransmission SACK support is negotiated at connection startup SACKs indicate missing or delayed packets In the last scenario, the client had acknowledged 800490 Server sends 800490 + 100 bytes …which goes missing… Server sends 800590 + 100 bytes Client acks 800490, SACKs 800590-800690 Server sends 800690 + 100 bytes Client acks 800490, SACKs 800590-800790 Server sends 800490 + 100 bytes Client acks 800790 WebSphere® Support Technical Exchange 16 IBM Software Group Window Size Indicates how much data the partner endpoint can accept Prevents a fast machine from overloading a small one For example, an IBM z10 mainframe vs. a PDA Window behavior The window shrinks as data is sent to the partner The window grows as data is read by the partner The window is “closed” when it reaches zero • This may indicate a performance problem! Window scaling option Allows values larger than 64KiB for fast networks Window value can be multiplied x2, x4, … x16384 WebSphere® Support Technical Exchange 17 IBM Software Group Flags SYN Synchronize sequence numbers at startup Counts as one byte for acknowledgement FIN This endpoint is finished sending data Counts as one byte for acknowledgement ACK This endpoint is acknowledging data Almost all packets have this flag set RST The connection is being forcefully reset Hopefully rare in a working environment Other flags you may see: PSH, URG, ECE, CWR WebSphere® Support Technical Exchange 18 IBM Software Group Establishing a TCP Connection Three-Way Handshake Client sends a SYN packet with its ISN (and options) Server responds with a SYN+ACK and its own ISN (and options) Client ends the handshake with an ACK Establishment Example Client SYN 58501256: Options: SACK, Timestamps Server SYN+ACK 4190910085:58501257 Options: Timestamps Client ACK 58501257:4190910086 WebSphere® Support Technical Exchange 19 IBM Software Group Terminating a TCP Connection Three- or Four-Way Handshake Each side sends a FIN and expects an ACK Either side can close first, or both at the same time One side can finish while the other remains active Termination Example One side Other side FIN (+ACK) 59549834:4191958663 + e.g. 100 bytes of data ACK 4191958663:59549935 ………… …………………………………………………………………………………………………… Other side FIN (+ACK) 4191958663:59549935 One side ACK 59549935:4191958664 WebSphere® Support Technical Exchange 20 IBM Software Group After Termination… The same TCP connection cannot be reestablished right away Client IP address Client port number Server IP address Server port number The system puts the connection into a TIME_WAIT state Old packets are discarded or expire in the network The total time waited is twice the MSL value Maximum Segment Life (MSL) Arbitrarily defined as 2 minutes by RFC791 (1981) However, some systems go as low as 15 seconds WebSphere® Support Technical Exchange 21 IBM Software Group Resetting a TCP Connection The RST Flag Immediately aborts a TCP connection No acknowledgement is required for a reset Resets usually indicate abnormal conditions Reasons for a Reset A program tried to connect to a port with no listener A program closed a connection with unsent data A program crashed with data still waiting to be sent An intermediary firewall or router killed the connection WebSphere® Support Technical Exchange 22 IBM Software Group TCP State Diagram Normal client path is shown in red Normal server path is shown in blue Common paths are shown with darker arrows These states are listed by netstat WebSphere® Support Technical Exchange 23 IBM Software Group WebSphere MQ WebSphere MQ uses sockets for Internet communication Supports only TCP sockets As of WebSphere MQ V6.0 Supports both IPv4 and IPv6 As of WebSphere MQ V6.0 Non-Internet protocols: DECnet, SNA LU6.2, NetBIOS, SPX The Berkeley sockets programming interface (1983) Almost universally used today by network programs A socket behaves very much like a file descriptor A socket represents a communications endpoint All sockets programs must first follow two steps 1. Create a socket for communications: 2. Assign an address to the socket: WebSphere® Support Technical Exchange socket() bind() 24 IBM Software Group WebSphere MQ Listener The WebSphere MQ listener (runmqlsr) is a TCP/IP server program Normally listens for any connections on port 1414 It can listen on a specific network address instead The port number is also configurable The listener uses its socket to listen() for connections The system will now accept three-way handshakes Handshakes in progress are tracked by the system The “backlog” limits the number in progress When a handshake completes... The listener receives the connection using accept() The system replies with a new, connected socket The original socket continues listening WebSphere® Support Technical Exchange 25 IBM Software Group WebSphere MQ Channels WebSphere MQ channels are TCP/IP client applications The listener must be running for them to succeed The listener address and port must be known Channels start by creating and binding a TCP/IP socket By default channels use “any address” and “port 0” This lets the system choose a free ephemeral port Once the channel has bound its endpoint it calls connect() Initiates the three-way handshake with the CONNAME If successful, the connection is now established WebSphere® Support Technical Exchange 26 IBM Software Group An Established Connection Ethernet Both sockets have a send buffer and a receive buffer for data write() adds data to the send buffer for transfer read() removes data arrived on the receive buffer poll() and select() wait for a buffer to be ready WebSphere® Support Technical Exchange 27 IBM Software Group Socket Options Applications have some control over TCP behavior The operating system is responsible for most of the work Some of its behavior can be modified with setsockopt() Socket options can control: How large should the send buffer be? How large should the receive buffer be? Should outgoing data be sent with no delay? Should close() wait until all data has been sent? The KeepAlive option, which tests for dead connections WebSphere® Support Technical Exchange 28 IBM Software Group Transmission Segment Header (TSH) Channel Startup MQ channels negotiate attributes by exchanging TSHes Allow queue managers at different levels to interoperate Channels Between Queue Managers TSHes flow at predictable intervals between the systems They carry MQ messages, heartbeats, and control data Client Channels TSHes flow only when the client program makes a request They carry the API request to MQ and the API reply back They can carry heartbeats only when MQGET is waiting WebSphere® Support Technical Exchange 29 IBM Software Group Idle Connections Established TCP connections are not obligated to exchange data A program may have a good reason for staying silent For example, consider the amqsputc sample How will the other end of the connection react to silence? Send some data to see whether the program responds Wait for a while, then close() the connection Wait forever and ever and ever… How will firewalls and other network devices in between react? Wait for a while, then reset the connection Wait forever and ever and ever… WebSphere® Support Technical Exchange 30 IBM Software Group The KeepAlive Option A socket option that applications may choose to enable WebSphere MQ will enable KeepAlive if requested TCP: KeepAlive=Yes When KeepAlive is enabled for a connection… 1. The system starts a timer for the connection 2. The system resets this timer whenever packets arrive 3. The system tests the connection when the timer expires 4. The system aborts the connection if there is no response WebSphere® Support Technical Exchange 31 IBM Software Group The KeepAlive Timer The KeepAlive timer value The default is two hours of inactivity (RFC1122) This timeout is usually a global operating system value MQ channels on z/OS can use the KAINT parameter When the timer expires the system sends test packets The packets contain no application data The packets have the wrong sequence number The other side should immediately ACK the right number WebSphere® Support Technical Exchange 32 IBM Software Group KeepAlive Recommendations Enable KeepAlive for your queue managers Particularly those which host MQ client applications This includes WebSphere MQ JMS applications Reduce the initial KeepAlive timeout from two hours A value of two to five minutes is reasonable If your firewall resets idle connections, reduce the initial KeepAlive timeout below the firewall timeout AIX administrators should install these patches: AIX 5.2: IY89387 AIX 5.3: IY89429 WebSphere® Support Technical Exchange 33 IBM Software Group W. Richard Stevens Excellent reading if you want detailed information: Stevens, W. Richard. UNIX Network Programming: Networking APIs: Sockets and XTI, Vol 1, 2nd ed. New Jersey: Prentice Hall, 1998. Stevens, W. Richard. TCP/IP Illustrated: The Protocols, Vol 1. Massachusetts: Addison-Wesley, 1994. Stevens, W. Richard. Advanced Programming in the UNIX Environment. Massachusetts: Addison-Wesley, 1990. Or visit his archived web site: www.kohala.com/start WebSphere® Support Technical Exchange 34 IBM Software Group Summary IP: Unreliable packet routing between machines TCP: Reliable data transfer between programs WebSphere MQ: Uses sockets to transfer messages TCP/IP KeepAlive: Is the connection still OK? Next Time: Putting theory to practice WebSphere® Support Technical Exchange 35 IBM Software Group Additional WebSphere Product Resources Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: http://www.ibm.com/developerworks/websphere/community/ Learn about other upcoming webcasts, conferences and events: http://www.ibm.com/software/websphere/events_1.html Join the Global WebSphere User Group Community: http://www.websphere.org Access key product show-me demos and tutorials by visiting IBM Education Assistant: http://www.ibm.com/software/info/education/assistant View a Flash replay with step-by-step instructions for using the Electronic Service Request (ESR) tool for submitting problems electronically: http://www.ibm.com/software/websphere/support/d2w.html Sign up to receive weekly technical My support emails: http://www.ibm.com/software/support/einfo.html WebSphere® Support Technical Exchange 36 IBM Software Group Questions and Answers WebSphere® Support Technical Exchange 37
