Modeling and Analysis of Multipath Video Transport over Lossy Networks Using Packet-Level FEC Xunqi Yu1 , James W. Modestino1 and, Ivan V. Bajic2 Electrical and Computer Engineering Department, University of Miami 2 School of Engineering Science, Simon Fraser University Abstract The use of forward error correction (FEC) coding is often proposed to combat the effects of network packet losses for error-resilient video transmission on packet-switched networks. On the other hand, path diversity has recently been proposed to improve network transport for both singledescription (SD) and multiple-description (MD) coded video. In this work we model and analyze an SD coded video transmission system employing packet-level FEC in combination with path diversity. In particular, we provide a precise analytical approach to evaluating the efficacy of path diversity in reducing the burstiness of network packet-loss processes. We use this approach to quantitatively demonstrate the advantages of path diversity in improving end-to-end video transport performance using packet-level FEC. 1. Introduction To transmit packet video over lossy packet-switched networks, packet-level forward error correction (FEC) is often proposed to combat packet losses typically due to network congestion, link failures, and timeouts. The efficacy of packet-level FEC is often limited by the bursty nature of typical network packet-loss processes. The use of path diversity, where packets are routed over multiple paths, has recently been proposed to improve network transport for both single-description (SD) and multiple-description (MD) coded video [3, 4]. Path diversity can reduce the effects of extended packet bursts as seen at the FEC decoder, thereby improving the FEC performance. In this work, we model and analyze a SD-coded video transmission system using a combination of both packet-level FEC and path diversity, and demonstrate the efficacy of path diversity in improving joint source-channel coding (JSCC) performance for packet video transmission. We consider two specific multipath transport scenarios: In the first scenario, we simply assume the paths share no joint links and are totally independent of each other. We provide a precise quantitative analysis of the resulting effective loss-burst-length distribution and residual decoded packetloss rate using path diversity. Using these results, we demonstrate the efficacy of path diversity in improving end-to-end video transmission performance using packet-level FEC. In the second scenario, we assume different paths may share some joint links and, therefore, the packet-loss processes on different paths may be correlated. We investigate the effect of the resulting path correlation on FEC performance. The paper is organized as follows: In Section 2 we describe a general system model for packet-video transmission over networks using packet-level FEC and path diversity. In Section 3 we investigate the multipath video transport system for the case of disjoint paths using packet-level FEC. In Section 4 we consider the multipath video transport system for the case of joint paths. Finally, in Section 5 we provide a summary and conclusions. 2. Multipath Video Transport System Figure 1 illustrates the general SD video multipath transmission system model considered in this paper. Several important system model parameters are also indicated. As shown in this figure, a video transmission system has the following components: a video encoder and decoder, a packetlevel FEC encoder and decoder, and a multipath transport network. ( R s ,ß ) ( n,k ) Video Encoder Packet-Level FEC Encoder R Path #1 Path #2 ... 1 Multipath Transport Network Path #N PL dec Video Decoder FEC Decoder Information Packets Parity Packets Figure 1. Video transmission system model. 265 L1 2.1. Video Encoder/Decoder Assume the source generates a space-time video signal which is used as input to the video encoder. We assume the encoder is a typical block-based hybrid motion-compensated video encoder 1, which encodes the video signal at rate R s bits/sec with INTRA refresh rate β (as a fraction of macroblocks coded in the intra mode). The latter parameter is indicative of the error resilience capability of the encoded video. We assume the compressed video data are packetized with M macroblocks (MBs) per transport packet. At the video decoder, the received video packets are depacketized and the video signal is reconstructed. For lost video MBs, passive error concealment will be used to mitigate the distortion due to unrecovered packets. 2.2. Packet-Level FEC Encoder/Decoder In this paper we use an interlaced Reed-Solomon RS(n, k) coding scheme [6] to provide FEC. For every block of k information packets an additional p = n − k redundant packets are transmitted. The channel-coding rate is then given by bits/channel use (1) Rc = k/n ; Assume the total available network bandwidth is fixed at R bps. As a result of the overhead introduced by channel coding, the source coding rate has to be throttled to Rs = R ∗ Rc bps. (2) At the FEC decoder, packets received from different paths will be reordered as necessary. Some packets may be lost due to network congestion, link failures, and timeouts. Let P (j, n) denote the block error distribution seen by the FEC decoder after packet reordering, i.e., the probability that j packet losses occur within a block of n consecutive packets, n ≥ 1, 0 ≤ j ≤ n. With Np denoting the number of lost packets within this block, if N p > n − k we assume the lost packets within this block cannot be recovered by the FEC decoder. Then the residual packet-loss rate of the original video packets after channel decoding can be shown to be given by n j ∗ P (j, n) /n . (3) P Ldec = j=n−k+1 2.3. Multipath Transport Network As illustrated in Fig. 1, the encoded video packets are transported over the multipath transport network composed of N paths. To simplify the analysis, we assume a simple cyclic or round-robin multipath transport scheme: the 1-st packet is transported on path #1, the 2-nd packet is transported on path #2, etc., until the (N + 1)-th packet which will then be transported on path #1. 1 In Section 3.3 we will make specific use of the ITU-JVT JM 6.1 codec for the newly developed H.264 video coding standard to provide some numerical examples. L2 … LK Receiver Sender Figure 2. A transport path consisting of K links. As illustrated in Fig. 2, each path consists of several transmission links connected by routers. Due to temporary buffer overflow and link outages, the packet losses typically occur in bursts with possibly varying burst lengths. We use a twostate discrete-time Markov-chain model, called the Gilbert model [7], to capture the bursty nature of each link. The Gilbert model for link i has two states, “Reception” and i i and P10 , rep“Loss”, and two independent parameters: P 01 resenting the associated state transition probabilities. In the literature, two alternative parameters are often used to characterize the Gilbert model: the steady-state packetloss rate P Li and the average loss burst length LB i . The relationships between these parameters are given by: Pi 1 (4) P Li = π i (1) = i 01 i ; LB i = i , P10 + P01 P10 where π i (1) is the steady-state probability of being in the loss state. We assume the packet-loss processes of the links along a given path are independent. Then, the end-to-end packet loss process of the entire path consisting of K links can be modeled as an aggregate Gilbert channel, with average packet loss rate (P L) and average burst length (LB) given by PL = 1 − K π i (0) , (5) i=1 and K 1 − i=1 π i (0) LB = K , i ( i=1 π i (0))(1 − K i=1 (1 − P01 )) (6) where π i (0) = 1 − π i (1) is the steady-state probability of i being in the reception state for intermediate link i and P 01 is the corresponding transition probability from the reception state to the loss state [7]. 2.4. Video Distortion Models The end-to-end distortion of a reconstructed video sequence, denoted by D, results from two components: the distortion induced by source compression, denoted by D s , and the channel distortion due to packet losses, denoted by Dc . We make use of the additivity assumption in [1], which states that the end-to-end distortion is the sum of D s and Dc , i.e., D = Ds + Dc . (7) As shown in [1], the compression distortion D s can be expressed as: θ + D0 , (8) Ds = Rs − R0 where Rs is the source coding rate and θ, R 0 and D0 depend on the INTRA rate β and other model parameters, which are 266 0 Path 1 0.06 Residual Packet−Loss Rate (PLdec) 10 N=1 Path 2 N=2 −2 10 ... Receiver Prob {BL ≥ x} Sender Path N Path 1 ( PL 1 , LB 1 ) Receiver Sender N=3 −4 N=4 10 N=∞ PL = 0.15 LB = 8 −6 10 ... Path 2 ( PL 2 , LB 2 ) −8 10 Path N ( PL N , LB N ) 1 2 3 4 5 6 7 Loss Burst Length (x) specific to the encoded video sequence and can be obtained by fitting the model to experimental data. The channel distortion (due to packet losses) D c can be expressed as a function of the decoded packet-loss rate P Ldec and the INTRA coding rate β as [1] T −1 1 − βt , (9) Dc = αP Ldec 1 + γt t=0 where T = 1/β and the parameter γ describes the efficiency of loop filtering to remove the effects of errors due to packet losses. Likewise, the model parameters γ and α can be obtained by fitting the model to experimental data. 3. Disjoint Paths First, consider the case where the different paths share no joint links so that packet-loss processes on different paths are independent. Therefore, as illustrated in Fig. 3, each end-toend path can be modeled as an independent Gilbert channel. The channel parameters associated with the aggregate Gilbert channel for the i-th path, P L i and LBi , can be obtained from (5) and (6), respectively 2. The FEC performance is dependent on the burstiness of the underlying packet-loss processes. Generally, the less bursty the packet losses are, the better performance FEC can achieve. In this subsection, we quantitatively investigate the effectiveness of multipath transport in reducing the burstiness of packet losses. The results are used to explain the FEC performance improvement using path diversity described in the next subsection. Suppose the random sequence {Y n } represents the packet loss process perceived by an end node, with 1 denoting loss and 0 denoting reception. The effective average loss burst length perceived by an end node can be expressed as k=1 2 For length. k ∗ P r{LBef f = k} = ∞ (k + 1) ∗ P {1k 0|01} , k=0 (10) simplicity, we assume each of the disjoint paths are of the same 0.03 0.02 0.01 0 1 10 PL = 0.07 LB = 4 0.04 Bernoulli Losses 2 3 4 5 Number of Paths (N) Figure 5. Residual packet-loss rates P Ldec vs. number of paths N . where the random variable LB ef f denotes the loss-burstlength seen by the receiver after packet reordering. Here we model each of the N independent paths as an aggregate Gilbert model. To simplify the analysis, we assume these Gilbert models are homogeneous, each with average packet-loss probability and average burst length P L and LB, respectively. Furthermore, we assume the packets are transmitted over the network using the cyclic multipath transport scheme described in Section 2.3. Then the loss-burst-length distribution, P r{LBef f = k + 1}, k ≥ 0, can be expressed as k P L (1 − P L) k P {1 0|01} = P LN−2 P00 k−N+1 P LN−2 P01 P11 P10 ; ; ; 0≤k ≤N −3 k =N −2 k ≥N −1, (11) where P01 , P10 are the state transition probabilities of each of the aggregate Gilbert models and can be computed from P L and LB according to (4) with P 00 = 1 − P01 , P11 = 1 − P10 . Therefore, from (10) and (11), the effective average loss burst length is ALBef f 3.1. Analysis of Loss-Burst-Length Statistics ∞ 9 Figure 4. Complementary cdf of loss-burst-length for different N . Figure 3. Multipath transport network with disjoint paths. ALBef f = 8 RS (7,5) RS (15,9) 0.05 = = + + = ∞ (k + 1)P {1k 0|01} k=0 N−3 (k + 1)P Lk (1 − P L) k=0 N−2 (k + 1)P LN−2 P00 k=N−2 ∞ k−N+1 N−2 P01 P11 P10 N−1 P L 1/(1 − P L) . (12) This last expression indicates that, somewhat surprisingly, when the number of paths N ≥ 2, the effective average loss-burst-length ALB ef f is independent of N and LB, but only depends on P L. Figure 4 provides a numerical example of the complementary cdf of the loss-burst-length, P r{LBef f ≥ x}, with different orders of path diversity N . For comparison, we indicate the corresponding complementary cdf for the Bernoulli channel (N = ∞), where the losses are totally independent 3. It indicates that, although the effective average loss-burst-length ALB ef f is the same for N ≥ 2, the loss-burst-length distributions are quite different. More specifically, it shows that, with an increase of path 3 When N → ∞, each packet will be transmitted on a different path. In this case, the channel reduces to a Bernoulli channel, where the packet losses are totally independent. 267 N RS(n,9) N=∞ 32 30 N=5 Path 1 Path 2 24 Receiver ... Sender N=1 26 Independent paths Path N 22 Dj. Path 1 ( PL 1 , LB 1 ) No FEC 20 18 0 Receiver Sender 28 J. Path (PL J , LB J) 0.05 0.1 0.15 0.2 0.25 0.3 0.35 Sender Dj. Path 2 ( PL 2 , LB 2) ... End−to−end Performance (in PSNR) 34 Packet−Loss Rate (PL) Figure 6. The end-to-end PSNR perfor- 3.2. Analysis of Residual Packet-Loss Rates We need to obtain the block error distribution P (j, n) to evaluate P Ldec according to (3). Consider an arbitrary block of n packets. Without loss of generality, we assume the 1-st packet within this block is transmitted on path #1. Assume there are N paths in total. Then, out of n packets, the number of packets transmitted on the i-th path is n + hi , 1≤i≤N, (13) li = N where hi = 1, 0, if i ≤ n mod N , otherwise . (14) Out of these n packets, let b i denote the number of lost packets transmitted on the i-th path. The total number of lost packets is then N bi = j . (15) i=1 Since the packet-loss processes of these N path are independent of each other, we have N Pi (bi , li ) , (16) P (j, n) = S i=1 where the sum is over the set S= (b1 , b2 , . . . , bN ) : N bi = j Correlated paths with 2 joint links Figure 7. Multipath transport network with joint paths. diversity order N , the probability P r{LB ef f ≥ x} will decrease for relatively large x. This means that with increasing path diversity N , the residual packet-loss probability with FEC, P Ldec , will be reduced. We will further demonstrate this in the next subsection. , (17) i=1 and Pi (bi , li ) denotes the block error distribution on the i-th path. If we model the packet-loss process over each path as Receiver Receiver Dj. Path N ( PL N , LB N) mance vs. packet-loss rate P L for the Susie sequence with RS(n, 9) codes and path diversity N , where the FEC coding rates Rc are optimally selected; Other model parameters are set as follows: R = 80 Kbps, β = 0.02, LB = 4. Sender Figure 8. Path correlation model. a Gilbert model, as expressed by (5) and (6), then the block error distribution P i (bi , li ) on each path can be obtained by a recursive algorithm first proposed in [8]. Figure 5 provides a numerical example of the efficacy of path diversity in reducing the residual packet-loss rates for the RS(7, 5) and RS(15, 9) codes, where each path is modeled as a homogeneous and independent Gilbert model with P L = 0.07 and LB = 4. It demonstrates that, as expected, packet transport with path diversity can improve the efficacy of FEC coding significantly. We have also indicated in Fig. 5 the limiting residual packet loss performance if the losses are independent (N → ∞) 4 , i.e., a Bernoulli channel. Observe the rapid approach of the residual packet-loss rates to their limiting values with increasing N . Also note that for use of a fixed code there is little advantage to path diversity orders N > 4. Additional results described in [9] also demonstrate the effectiveness of multipath transport in reducing the probability of large bursts and their effect on end-to-end performance. 3.3. Video Performance Using Multipath Transport Figure 6 demonstrates a comparison of the end-to-end video performance with different network packet-loss rates (P L) for the following cases: 1) without coding (R c = 1) or path diversity (N = 1); 2) with an RS(n, 9) code, but no path diversity (N = 1), where R c is optimally selected5 ; 3) combined RS(n, 9) code and path diversity (N ≥ 2), where Rc is again optimally selected, for the QCIF Susie test sequence. For comparison, we have also indicated the performance achieved on a Bernoulli channel (N = ∞). The QCIF Susie sequence (176×144) consists of 150 frames at f r = 30 4 The limiting value for the RS(15, 9) code is extremely small and indistinguishable from zero in Fig. 5. 5 Optimally selecting R to maximize end-to-end performance for a c given overall transmission rate R represents a joint source-channel coding (JSCC) approach. 268 frames/sec. We assume every M = 11 MBs are packetized into one packet. Therefore, each QCIF frame is packetized into 99/M = 9 packets. Other system parameters are indicated in the figure caption. This figure indicates that, compared to the case of JSCC without path diversity, the combination of JSCC and path diversity can provide significantly improved end-to-end video performance. For example, for P L = 15%, there is a performance advantage of 2 dB in going from N = 1 to N = 4. However, unlike the fixed code case illustrated in Fig. 5, observe that when the code rate is optimally chosen as part of a JSCC approach there is a considerable performance advantage to path diversity orders N > 4. 4. Correlated Paths In the previous section, we assumed that the different paths share no joint links so that the packet-loss processes on different paths are independent. However, in actual multipath transport networks there may be some shared or joint links between different transport paths. In this case, the packetloss processes on different paths may be correlated. In actual networks the connection topologies (joint/disjoint links) between the sender and the receiver may be quite varied so that precise modeling of video transport using path diversity can be fairly complex. However, it has been shown in [4, 5] that the important end-to-end properties of a 2-path network can be captured using a simplified three-subpath topology, where subpaths 1 and 2 are formed by the disjoint links along the two paths and subpath 3 is formed by the joint links along both paths. In this section we consider a similar approach as in [4, 5], except that we will consider a more general N -path network. We will specifically concentrate on the effect of path correlation on the end-to-end video performance using packet-level FEC. To model the effect of path correlation on FEC performance, we consider a simplified scenario: different paths share common joint links, as shown in the upper subfigure of Fig. 7, i.e., a number of disjoint links followed by a series of joint links. 4.1. Analysis of Residual Packet-loss Rates We assume each subpath can be described by a Gilbert model, as illustrated in the bottom subfigure of Fig. 7, where the associated model parameters can be derived from the corresponding portion of the original path using (5) and (6). To evaluate the residual packet-loss rates P L dec after FEC decoding, again we need to determine the block error distribution P (j, n) as seen by the FEC decoder after packet reordering. Assume there are j packets lost out of n consecutive packets. If the total number of packets lost over disjoint links is bD = d then the number of packets lost over the joint links is bJ = j − d. Therefore, we have j P r{bD = d} ∗ P r{bJ = j − d} . (18) P (j, n) = d=0 Since out of these n packets d packets have already been lost over the disjoint links, only the remaining n − d packets will be transmitted over the joint links. Therefore, we have P r{bJ = j − d} = P J (j − d, n − d) , (19) where P J (j − d, n − d) denotes the block error distribution over the joint path. Let bD i denote the number of packets lost over disjoint path i. Therefore, we have N bD (20) bD = i . i=0 Following a similar approach as in going from (15) to (17), it can be shown that [9] P (j, n) = j N d=0 Sd PiD (bD i , li ) ∗ P J (j − d, n − d) , i=1 (21) where li is given by (13) and N D D D D Sd = b 1 , b 2 , . . . , bN : bi = d . (22) i=1 If we model the packet-loss process over each joint/disjoint subpath as a Gilbert model, then the block J error distributions P iD (bD i , li ) and P (j − d, n − d) on each of the joint/disjoint paths can be obtained by the recursive algorithm originally proposed in [8]. 4.2. Effect of Path Correlation on Packet-Level FEC Performance In order to investigate the effect of path correlation on FEC performance, we consider a simplified 2-path transport scheme and assume there are 5 intermediate links on each of the 2 paths, as illustrated in Fig. 8. We further assume that each intermediate link is independently modeled by a Gilbert channel with parameters P L i and LB i . Therefore, the corresponding end-to-end P L and LB can be computed from (5) and (6), respectively, for each path. However, the two paths may share some joint links. Let J denote the number of joint links. The larger J is, the more correlated the two paths are. Figure 8 shows the case of J = 0 and J = 2. The block packet-loss distribution P (j, n) and the residual packet-loss rate after channel decoding P L dec can be obtained from (21) and (3), respectively. Table 1 shows the effect of path correlation on the FEC performance with two different RS codes and for two channel conditions. More specifically, it shows the residual packetloss rates P Ldec using a relatively weak RS(22, 18) code and a stronger RS(31, 18) code under two different channel conditions: a relatively bad channel with P L = 10% and LB = 8 and a relatively good channel with P L = 5% and LB = 4. Generally, the results indicate that increased correlation among paths results in a higher residual packet-loss rate P Ldec after channel decoding. However, it should be 269 32 J 0 1 2 3 4 5 P L = 10%, LB = 8 RS(22, 18) RS(31, 18) 8.05 % 3.03 % 8.12 % 3.09 % 8.20 % 3.34 % 8.28 % 3.58 % 8.37 % 3.80 % 8.46 % 4.01 % End−to−end Performance (in PSNR) noted that, when the channel conditions are relatively good, or when a weak code is used, the path correlation level (J) does not make much difference in the FEC performance. But when the channel conditions are relatively bad (P L = 10% and LB = 8) and a stronger code is used, the path correlation level J can have a significant effect on the FEC performance. This means that path correlation has a significant impact on FEC performance only when channel conditions are severe and strong FEC codes are used. These analytical results are consistent with the conclusions in [2], where the results there were obtained by simulations. 30 N=∞ N N=5 RS(n,9) 29 28 27 N = 1 26 25 No FEC 24 23 0 P L = 5%, LB = 4 RS(22, 18) RS(31, 18) 2.84 % 0.32 % 2.90 % 0.34 % 2.95 % 0.39 % 3.00 % 0.43 % 3.05 % 0.46 % 3.11 % 0.49 % Table 1. Decoded packet-loss rates (P L dec) of a 2path transport network with different path correlations and FEC code schemes. 31 1 2 3 4 5 Number of Joint Links (J) Figure 9. The end-to-end PSNR performance vs. the number of joint links J for the Susie sequence with RS(n, 9) codes and path diversity N , where the FEC coding rates Rc are optimally selected; Other model parameters are set as follows: R = 80 Kbps, β = 0.02, P L = 15%, LB = 4. strated the effect of path correlation on the packet-level FEC performance. References 4.3. Effect of Path Correlation on Video Performance Figure 9 demonstrates the effect of path correlation on video performance transmitted over different multipath transport networks. Specifically, it demonstrates a comparison of the end-to-end video performance achieved over multipath transport networks with different numbers of joint links (J) for different path diversity orders (N ) and coding strategies (with or without coding) as shown in Fig. 6. The path correlation model is the same as described in Section 4.2, but with different path diversity N . For the coded systems, the code rate Rc is chosen optimally. Other model parameters are indicated in the caption. This figure indicates that, for all path correlation levels (J), increased path diversity combined with JSCC generally can provide improved end-to-end video performance. However, with increased path correlation, the advantage achieved with the use of path diversity will decrease, and with J ≥ 4 there is little difference in performance independent of the value of N . 5. Summary and Conclusions We have modeled and analyzed an SD coded video transmission system employing packet-level FEC in combination with path diversity. We provided a precise analytical approach to evaluating the efficacy of path diversity in reducing the burstiness of packet-loss processes. Using this approach we have quantitatively demonstrated the advantages of path diversity in improving end-to-end video transport using packet-level FEC. Finally, we have quantitatively demon- [1] K. Stuhlmuller, et al., “Analysis of video transmission over lossy channels,” IEEE J. Select. Areas in Comm., vol.18, pp. 1012-1032, June 2000. [2] Q. Qu et al., “On the effects of path correlation in multipath video communications using FEC,” IEEE Globecom, pp. 977-981, Nov. 2004. [3] J. G. Apostolopoulos, “Reliable video communication over lossy packet networks using multiple state encoding and path diversity,” VCIP, pp. 392-409, San Jose, CA, Jan. 2001. [4] J. G. Apostolopoulos and M. D. Trott, “Path diversity for enhanced media streaming,” IEEE Communications Magazine, pp. 80-87, Aug. 2004. [5] J. G. Apostolopoulos et al. “Modeling path diversity for multiple description video communication,” ICASSP, pp. 2161-64, May 2002. [6] V. Parthasarathy, J.W. Modestino, and K.S. Vastola, “Reliable transmission of high-quality video over ATM networks,” IEEE Trans. on Image Proc., vol.8, pp. 361-374, Mar. 1999. [7] E. N. Gilbert, “Capacity of a burst-noise channel,” Bell Sys. Tech. Journal, vol.39, pp. 1253-1266, Sept. 1960. [8] E. O. Elliott, “A model of the switched telephone network for data communications,” Bell Sys. Tech. Journal, vol.44, pp. 89-109, Jan. 1963. [9] Xunqi Yu, “Video transmission on packet-switched networks,” Ph.D. dissertation, Department of Electrical and Computer Engineering, University of Miami, Coral Gables, FL 33124, in progress. 270