Selecting Sequence Numbers R a y m o n d S. T o m l i n s o n Bolt B e r a n e k Cambridge, and N e w m a n Inc. Massachusetts Introduction A c h a r a c t e r i s t i c of almost all communication protocols is the use of u n i q u e n u m b e r s to i d e n t i f y i n d i v i d u a l pieces of data. These i d e n t i f i e r s permit error control through acknowledgement and r e t r a n s m i s s i o n techniques. U s u a l l y s u c c e s s i v e pieces of data are identified with s e q u e n t i a l n u m b e r s and the i d e n t i f i e r s are thus c a l l e d s e q u e n c e numbers. This paper discusses techniques for selecting and synchronizing s e q u e n c e n u m b e r s such that no errors will occur if c e r t a i n n e t w o r k c h a r a c t e r i s t i c s can be b o u n d e d and if adequate data error detection measures are taken. The discussion s p e c i f i c a l l y focuses on the p r o t o c o l d e s c r i b e d by Cerf and Kahn, (I) but the ideas are a p p l i c a b l e to other s i m i l a r protocols. The P r o b l e m One of the p r o b l e m s with the p r o t o c o l d e s c r i b e d by Cerf and Kahn w h i c h was brought out by our e x p e r i m e n t s at BBN is "How do you identify duplicate packets from a previous use of a particular connection?". What is n e c e s s a r y is a means for either the r e c e i v e r of the late packet to i d e n t i f y the packet as late or a means for the o r i g i n a t o r of the packet to tell the receiver that the packet was late. The protocol as described provides a fine m e c h a n i s m for i d e n t i f y i n g late packets while data are a c t i v e l y f l o w i n g between two ports. It w o u l d also work fine if hosts never went down and had large a m o u n t s of storage. But hosts do go down and have to be restarted and hosts don't have an i n e x h a u s t i b l e supply of storage. The S o l u t i o n The e s s e n c e of the s o l u t i o n is that s e q u e n c e numbers must be chosen such that a p a r t i c u l a r s e q u e n c e number never refers to more than one byte at any one time ~nd the valid range of sequence numbers must be positively synchronized whenever a c o n n e c t i o n is used. The former r e q u i r e s c a r e f u l a t t e n t i o n to the method used for s e l e c t i n g initial s e q u e n c e numbers. The latter r e q u i r e s a more i n v o l v e d h a n d s h a k e than that provided by the protocol. (I) Cerf, V . G . and Kahn, R. E., "A P r o t o c o l for Packet Network Intercommunication," IEEE Transactions on C o m m u n i c a t i o n s , Vol. C O M - 2 2 #5, May 1974, pp. 637-648. - l l - Positive Synchronization Achieving positive synchronization requires a three way h a n d s h a k e for SYN and a two-way handshake for REL. This is necessary because the passive side of the c o n n e c t i o n must have p o s i t i v e a s s u r a n c e from the active side that the packet received is current. Simply receiving a packet does not p r o v i d e this assurance. A s s u m i n g one end as the i n i t i a t o r and the other end as responder, the normal p r o c e d u r e for s y n c h r o n i z i n g s e q u e n c e n u m b e r s is: I) I n i t i a t o r (one that sends a SYN with a unique sequence cannot be o u t s t a n d i n g in the net). number 2) R e s p o n d e r r e c e i v e s this packet but does not p r o c e s s the data (if any) because it does not k n o w w h e t h e r or not the packet is a late duplicate. 3) R e s p o n d e r r e t u r n s a packet w h i c h ACK's number and SYN's the responder's number. the initiator's unique sequence 4) The i n i t i a t o r r e c e i v e s the r e s p o n d e r ' s SYN and b e l i e v e s it b e c a u s e it ACK's an a p p r o p r i a t e s e q u e n c e n u m b e r and SYN's a p r e v i o u s l y u n s y n c h r o n i z e d (in that d i r e c t i o n ) connection. The i n i t i a t o r now knows that the r e s p o n d e r is w i l l i n g to go ahead and knows w h e r e the r e s p o n d e r is going to start but the r e s p o n d e r does not yet know if the initiator was r e a l l y trying to s y n c h r o n i z e or if the packet r e c e i v e d was a late duplicate. 5) The i n i t i a t o r sends responder's sequence packet back a number. which ACK's the 6) The r e s p o n d e r r e c e i v e s this packet and believes it b e c a u s e it ACK's an a p p r o p r i a t e s e q u e n c e n u m b e r and has a sequence number of its own which is in the a p p r o p r i a t e range. The o r i g i n a l packet and this one may now be processed further, data delivered, REL's p r o c e s s e d etc. The h a n d s h a k e for REL needs to be only t w o - w a y because valid sequence numbers have been established and may be used to acknowledge the REL. Since the need to send a REL may occur at times when there are no data bytes to transmit, a dummy byte which is not delivered to the user p r o c e s s is sent to p r o v i d e s o m e t h i n g to be a c k n o w l e d g e d . - 12 - A c o n s e q u e n c e of this need for p o s i t i v e synchronization is that any data in the initial packet with the SYN may not be d e l i v e r e d to the user process until the v a l i d i t y of the packet is verified. It is u s e f u l to include data in the initial packet, however, since that data may be acknowledged in the first r e s p o n s e packet e l i m i n a t i n g the need for a s u b s e q u e n t packet for acknowledging the data. A m i n i m u m e x c h a n g e for data flow in one d i r e c t i o n is four packets. This is i l l u s t r a t e d by example below. The fourth packet is r e q u i r e d to i n f o r m the i n i t i a t o r that it may forget the c o n n e c t i o n i n f o r m a t i o n . It is also n e c e s s a r y to provide a m e c h a n i s m to negatively acknowledge r e s p o n s e s to s p u r i o u s packets. A REJ bit or command s h o u l d be p r o v i d e d for this purpose. Selecting Unique Sequence Numbers The a s s u m p t i o n that n e t w o r k s may g e n e r a t e d u p l i c a t e packets with possibly long l i f e t i m e s renders the task of p r o v i d i n g for u n i q u e s e q u e n c e number n o n - t r i v i a l . The period of time over w h i c h a p a r t i c u l a r s e q u e n c e n u m b e r may refer to a p a r t i c u l a r byte requires knowing an upper bound on the packet l i f e t i m e in the network. This bound must be m o d e r a t e l y tight because, as this bound becomes looser, either packet o v e r h e a d must i n c r e a s e due to the need for more bits of s e q u e n c e n u m b e r or t r a n s m i s s i o n rate must be r e s t r i c t e d so that s e q u e n c e n u m b e r are not reused before packets die out in the network. The p e r i o d during which a p a r t i c u l a r s e q u e n c e number refers to a p a r t i c u l a r byte is then some sma~l multiple of this packet l i f e t i m e ( d e p e n d i n g on how many r e v e r b e r a t i o n s the protocol will support). I will use T as a parameter of the protocol design w h i c h d e s i g n a t e s this small m u l t i p l e of the packet lifetime. The size of the w i n d o w plays a part in d e t e r m i n i n g w h e t h e r a late packet might be c o n f u s e d with current packets. A large window increases the m a x i m u m s e q u e n c e n u m b e r the r e c e i v e r will accept as valid. The m a x i m u m w i n d o w size is a design parameter of the protocol and must be s p e c i f i e d b e f o r e the protocol is complete. For our purposes, the w i n d o w size can be conveniently t h o u g h t of as e x t e n d i n g the a p p a r e n t packet lifetime (T) and will not be d i s c u s s e d f u r t h e r here. Also subsumed into the parameter T is the m a x i m u m host s e r v i c e i n t e r r u p t i o n time. If a host stops executing for a period of time and then resumes with no loss, any packets held in that host during that period, w h e t h e r they are data packets or a c k n o w l e d g e m e n t packets, are e f f e c t i v e l y d e l a y e d by the duration of the service interruption. This delay is i n d i s t i n g u i s h a b l e from any delay c a u s e d by the n e t w o r k itself. - 13 - A n o t h e r p a r a m e t e r of the p r o t o c o l d e s i g n is the m a x i m u m data rate. The m a x i m u m data rate design p a r a m e t e r may be less than or greater than the data rate achievable by the network and associated hardware. The design parameter will probably ultimately be less than the hardware/network limitation as technology improves. If this is the case, s o f t w a r e c o n t r o l will have to be used to limit the a c t u a l data rate. Given the m a x i m u m data rate (R) and the maximum lifetime (T) and the m a x i m u m s e q u e n c e n u m b e r (M), we can the m i n i m u m s e q u e n c e n u m b e r cycle time (C) as C : packet define M/R and state lifetime that this must be larger than the maximum packet C > T This inequality must same s e q u e n c e numbers, in the network. be g u a r a n t e e d o t h e r w i s e p a c k e t s with the but from an e a r l i e r cycle, may still exist The s e l e c t i o n of values for some of these parameters is arbitrary. One could select a field w i d t h for s e q u e n c e n u m b e r s thus d e t e r m i n i n g M and compute R from that. T is determined by the network characteristics. The a m o u n t by w h i c h C e x c e e d s T must also be selected. The amount by which C exceeds T d e t e r m i n e s how f r e q u e n t l y s e q u e n c e n u m b e r s must be r e s y n c h r o n i z e d when activity is low (see below). Therefore, C should be s u b s t a n t i a l l y g r e a t e r than T. Since the c h o i c e of sequence numbers is directly under c o n t r o l of the sender, it is best to place the r e s p o n s i b i l i t y for selecting unique sequence n u m b e r s on the sender. The r e c e i v e r then a c c e p t s a packet on the basis of w h e t h e r the s e q u e n c e n u m b e r falls w i t h i n the c u r r e n t w i n d o w or not. If the r e c e i v e r has no current window then the handshake d e s c r i b e d above is used to e s t a b l i s h one. To p r e v e n t r e u s i n g a p a r t i c u l a r s e q u e n c e n u m b e r too soon it is necessary for the sender to have k n o w l e d g e about when that sequence number was last used. Figure I illustrates the s i t u a t i o n when c o m p l e t e k n o w l e d g e of this sort is a v a i l a b l e . The curve of actual sequence numbers used as a f u n c t i o n of time permits a region of f o r b i d d e n s e q u e n c e n u m b e r vs. time points to be defined. The curve of actual sequence numbers is not p e r m i t t e d to r e e n t e r this region. Complete knowledge of this kind r e q u i r e s a p r o h i b i t i v e amount of storage. - 14 - l l | ~r 0 g l l l l I l 0 ,,.4 0 S~3g~nN 30N3n03S 0 Another p o s s i b i l i t y is to retain a few n u m b e r s w h i c h permit the s e q u e n c e n u m b e r curve to be bounded. This scheme and the previous one r e q u i r e a m e m o r y w h i c h p e r s i s t s at least as long as T even in the event of s y s t e m c r a s h e s or other m e m o r y destroying events. The method I will describe uses a t i m e - o f - d a y clock to govern the s e l e c t i o n of sequence numbers. The choice of a time-of-day clock for this p u r p o s e is n a t u r a l since time is the factor w h i c h d e t e r m i n e s w h e t h e r d u p l i c a t e s might still exist. If the t i m e - o f - d a y clock has its own n a t u r a l period such as seconds since midnight, then it is n e c e s s a r y to c h o o s e the s e q u e n c e cycle time (C) such that the clock period is an i n t e g r a l m u l t i p l e of C. There are several mappings of clock to s e q u e n c e number. The mapping which maintains maximum resolution is given by the expression (V mod C)*M/C w h e r e V is the clock value, C is the s e q u e n c e cycle time in units of V, and M is the m a x i m u m s e q u e n c e number. The s e q u e n c e n u m b e r e q u i v a l e n t given by this f o r m u l a is c a l l e d ISN (initial sequence number) in the f o l l o w i n g d i s c u s s i o n . ISN is used for the i n i t i a l sequence n u m b e r for e s t a b l i s h i n g or r e - e s t a b l i s h i n g a c o n n e c t i o n in the a b s e n c e of other i n f o r m a t i o n . A plot of ISN as a f u n c t i o n of time is shown in figure 2. The step size d e p e n d s on the r e s o l u t i o n of the t i m e - o f - d a y clock and the range of s e q u e n c e n u m b e r s required. The step size is exaggerated in the figure. Also on figure 2 are drawn a n o t h e r s t a i r c a s e p r e c e e d i n g the ISN curve by one step, a staircase delayed from that by C-T, and a h o r i z o n t a l line l a b e l l e d "last used seq no." These lines delimit a r e g i o n of a l l o w e d sequence numbers. If p a c k e t s are never given s e q u e n c e n u m b e r s o u t s i d e of this region, there will never be any p r o b l e m w i t h c o n f u s i n g late arriving packets with current packets. This is b e c a u s e the s e q u e n c e n u m b e r of a late a r r i v i n g packet will be outside the allowed region e i t h e r b e c a u s e the c u r r e n t s e q u e n c e n u m b e r s have gone beyond it thus r a i s i n g the lower b o u n d a r y or b e c a u s e it is impossible for enough time to have p a s s e d to place it in the a l l o w e d r e g i o n of the next cycle. R e m e m b e r that the next cycle will not occur until an amount of time g r e a t e r than T has passed and, by definition, the packet cannot p e r s i s t for that l e n g t h of time. The follows: sequence number constraints - 16 - may be summarized as d I I.L -,-I .IJ 0 q~ ,ml o~ Z W o 0 I o I-a W gc~ I-- oH 0 $~381~InN 3ON3n03s - ITT - o a. The c u r r e n t s e q u e n c e n u m b e r m u s t not "get a h e a d " of the ISN c l o c k by m o r e than one step because packets with those sequence numbers from the previous sequence n u m b e r c y c l e m a y be left in the n e t w o r k . b. The c u r r e n t s e q u e n c e n u m b e r m u s t not "fall b e h i n d " the ISN clock by more than C - T - < o n e c l o c k s t e p > b e c a u s e d u p l i c a t e s of packets generated with such sequence n u m b e r s m a y a p p e a r l a t e r and be c o n f u s e d w i t h the (then c u r r e n t ) ISN. c. Potential sequence numbers must increase monotonically e v e n w h e n the c o n n e c t i o n is i n a c t i v e b e c a u s e t h e r e w i l l a l m o s t c e r t a i n l y be a c o n f l i c t if t h e r e are any p a c k e t s r e m a i n i n g in the n e t w o r k . Referring to f i g u r e 2 again, note that one of the b o u n d s of the a l l o w e d r e g i o n is the l a r g e s t (last) used sequence number. This would seem to require that that n u m b e r be r e m e m b e r e d at least as long as its r e u s e m i g h t c a u s e c o n f u s i o n (T). This w o u l d r e q u i r e e x c e s s i v e a m o u n t s of s t o r a g e in all e x c e p t a few special cases. It is n e c e s s a r y to r e m e m b e r it for a s h o r t time b e c a u s e it may e x c e e d the ISN curve. This c o u l d be a v o i d e d by m o v i n g the ISN c u r v e to the e x t r e m e left edge of the a l l o w e d r e g i o n . Doing this, however, makes it immediately impossible to use the s e q u e n c e n u m b e r s e l e c t e d from the ISN curve for communication until the next t i c k of the clock. This is, in fact, the r e a s o n w h y the ISN c u r v e in f i g u r e 2 is s h o w n d i s p l a c e d from the left edge of the a l l o w e d region. The solution to this p o s s i b l e d i l e m m a is to r e m e m b e r the last u s e d s e q u e n c e n u m b e r if it is g r e a t e r than ISN until the next clock tick. At this p o i n t the ISN c u r v e w i l l n e c e s s a r i l y e x c e e d the last u s e d s e q u e n c e n u m b e r since, w h e n the connection was a c t i v e , the s e q u e n c e n u m b e r s w o u l d not h a v e b e e n p e r m i t t e d to exceed the ISN c u r v e by m o r e than one t i c k s i n c e d o i n g so w o u l d have p l a c e d the s e q u e n c e n u m b e r s o u t s i d e of the allowed region. In the event of a memory d e s t r o y i n g event, such as a s y s t e m r e s t a r t , it is n e c e s s a r y to wait for one t i c k of the ISN because that insures that an i n i t i a l s e q u e n c e n u m b e r s e l e c t i o n w i l l be g r e a t e r than the s e q u e n c e n u m b e r s in use just p r i o r to the m e m o r y lo~s. The r u l e s for s e l e c t i n g i n i t i a l s e q u e n c e numbers are as follows: I. R e m e m b e r the next 2. the last u s e d tick of ISN. sequence number at least until Inhibit initial sequence number selection until at least one ISN tick has o c c u r r e d f o l l o w i n g a m e m o r y loss (system restart). - 18 - 3. If the last used (incremented by sequence number is known, I) for a new i n i t i a l s e q u e n c e use it number. 4. If no last used s e q u e n c e n u m b e r is being r e m e m b e r e d and rule 2 is satisfied, use ISN for a new i n i t i a l s e q u e n c e number. Once an i n i t i a l s e q u e n c e number is selected, packets are given sequence n u m b e r s as s p e c i f i e d by the protocol. S e q u e n c e numbers so s e l e c t e d must not equal or exceed ISN plus one step. If data flows at less than m a x i m u m rate for a long enough time, the c u r r e n t s e q u e n c e n u m b e r w o u l d cross into the f o r b i d d e n r e g i o n on the right. Packets e m i t t e d w i t h these s e q u e n c e n u m b e r s and d e l a y e d by time T w o u l d conflict w i t h a later value of ISN. This must be p r e v e n t e d by r e s y n c h r o n i z i n g the s e q u e n c e n u m b e r s to a larger value. In order to r e s y n c h r o n i z e s e q u e n c e numbers, the current s e q u e n c e n u m b e r s must be r e l e a s e d and then a new s e q u e n c e (using s e q u e n c e n u m b e r ISN) e s t a b l i s h e d . Examples of C o n n e c t i o n Synchronization The dialogs below i l l u s t r a t e the i n t e r c h a n g e of letters that occurs in v a r i o u s situations. Each line of the dialog consists of a packet label in p a r e n t h e s e s f o l l o w e d by the a c t i v i t y at process A w h e r e "<--" s i g n i f i e s the packet being r e c e i v e d by A, "-->" signifies the packet being transmitted by A and "..." s i g n i f i e s that A is u n a w a r e of the packet at that time. Next appears a description of the packet in the form <SEQ n><control><data>. Next appears the a c t i v i t y at process B where "-->" s i g n i f i e s the packet is r e c e i v e d by B, "<--" s i g n i f i e s the of packet is t r a n s m i t t e d by B, " ... " s i g n i f i e s that B is u n a w a r e the packet. The label is a handle on the packet for i l l u s t r a t i o n purposes. When a packet is d u p l i c a t e d or r e t r a n s m i t t e d it will be given the same label. When the delay b e t w e e n t r a n s m i s s i o n and r e c e p t i o n of a packet is u n i m p o r t a n t , the packet will be shown to be t r a n s m i t t e d and r e c e i v e d on the same line. Unidirectional Connection with Lbl A Packet (I) --> <SEQ 0 > < S Y N > < > A synchronizes (2) <-- <SEQ O><SYN, Sender Initiating B --> to O. ACK 0><> - 19 - <-- B syn@hronizes 0. (3) --> (4) __ (5) --> to 0 and acknowledges <SEQ 0><ACK 0><10 Data Bytes> A acknowledges B's 0 and sends data. --> some __ <SEQ 0><ACK 10><> B a c k n o w l e d g e s A's data. ,. (6) <SEQ 10><ACK <SEQ 0><ACK --> 0><6 Data Bytes> 16><> ----> (7) ----> <SEQ 16><ACK 0, REL><20 A sends last data bytes (8) <-- <SEQ 0><ACK 36, REL, C T L > < D u m m y Byte> B a c k n o w l e d g e s all A's data (and the REL) and sends its own REL. (9) --> <SEQ 36><ACK I><> A a c k n o w l e d g e s B's REL. Unidirectional (1) (2) --> <-- with Receiver Data Bytes> and REL. --> Initiating <SEQ 0><SYN><> A synchronizes --> to 0. <SEQ 0><SYN, ACK 0><14 Data Bytes> B synchronizes to 0, a c k n o w l e d g e s and sends data. (3) --> <SEQ 0><ACK 14><> A a c k n o w l e d g e s B's data. (4) <-- <SEQ (5) --> <SEQ 0><ACK (6) <-- <SEQ 24><ACK (7) --> <SEQ 0><ACK (8) <-- <SEQ 29><ACK 14><ACK __ 0><10 __ A, --> Data Bytes> ----> 24><> 0, REL><5 29, CTL, I><> - 20 - Data Bytes> REL><Dummy Byte> ----> Minimal (I) (2) Late --> <-- Sender Initiating <SEQ 0><SYN><21 Data B y t e s > A s y n c h r o n i z e s to 0 and sends <SEQ 0><SYN, ACK 21><> B a c k n o w l e d g e s A's data and own sequence number. Data d e l i v e r e d to user process. --> data. __ SYN's its is not yet (3) --> <SEQ 21><REL, ACK 0><> A a c k n o w l e d g e s B's s e q u e n c e n u m b e r and REL's the connection. B d e l i v e r s data to user process. __> (4) --> <SEQ 2 2 > < A C K I> A a c k n o w l e d g e s B's REL and drops the connection. Note that if this final ACK gets lost, B will r e s e n d its REL and A will respond with an error reply. B must interpret this as e q u i v a l e n t to the ACK. _-> Duplicate SYN (I) <SEQ x > < S Y N > < 2 3 Data Bytes> B receives a late d u p l i c a t e (2) (3) Late Exchange __ --> <SEQ 0><SYN, ACK x+23><> B responds synchronizing s e q u e n c e numbers. A is not the connection. --> packet. __ its aware <SEQ x+23><REJ, ACK 0><> A rejects the response. B throws away the data and resets the c o n n e c t i o n . Duplicate SYN Plus (I) <SEQ x><SYN><22~ Data Bytes> B receives the late d u p l i c a t e c o n t a i n i n g SYN. (2) __ own of Late <SEQ 0><SYN, B responds. --> Response ACK x+22><> - 2 1 - --> packet - _ (3) (4) --> Simultaneous (I) --> (2) <SEQ x + 2 2 > < A C K y><31 Data Bytes> B r e c e i v e s a d u p l i c a t e of the o r i g i n a l r e s p o n s e to the response to packet (I). Note that it ACK's y w h i c h is o u t s i d e B's current s e q u e n c e numbers. The packet is rejected since A's sequence numbers have not been verified and the ACK is out of range. --> <SEQ x+22><REJ, ACK 0><> A rejects the response p r e v i o u s example. --> as in the Initiation <SEQ 0 > < S Y N > < I 5 Data B y t e s > A s y n c h r o n i z e s to 0. <SEQ 0 > < S Y N > < > Meanwhile, B s y n c h r o n i z e s __ too. --> (I) <SEQ 0 > < S Y N > < I 5 Data Bytes> A's packet arrives. (3) <SEQ 0><SYN, ACK 0><> B a c k n o w l e d g e s A's s e q u e n c e n u m b e r and resynchronizes its own since it has not yet r e c e i v e d v e r i f i c a t i o n from A. (2) <-- (4) --> (3) __ <SEQ O > < S Y N > < > B's o r i g i n a l packet a r r i v e s at A. <SEQ 0><SYN, ACK 0><15 Data B y t e s > A sends its first packet again but this time acknowledges B's sequence number. B is now completely synchronized. <SEQ 0><SYN, ACK 0><> B's first r e s p o n s e a r r i v e s at A. now completely synchronized. c o n t i n u e s as in the above cases. - 22 - -- A is This --> Conclusion The methods described above for m a n a g i n g use of s e q u e n c e numbers on a c o n n e c t i o n assure that each byte of information is u n i q u e l y i d e n t i f i e d in spite of packet d u p l i c a t i o n and long delay in the network, and in spite of service d i s r u p t i o n s or c o m p l e t e m e m o r y loss by either or both c o m m u n i c a t i n g hosts. It is our belief that any lesser schemes will lead to u n r e l i a b l e n e t w o r k p e r f o r m a n c e and are, therefore, u n s a t i s f a c t o r y . i 23 -