Telecommunications Industry Association (TIA) TR-30.3/99-12-115 Clearwater FL 01Dec 1999 COMMITTEE CONTRIBUTION Technical Committee TR-30 Meetings SOURCE: Satchell Evaluations CONTRIBUTOR: Stephen Satchell Phone: (775) 832-7157 Fax: (775) 831-2011 E-mail: firstname.lastname@example.org TITLE: Why Move PN 3509 To Timed Transfer. PROJECT: PN-3509 DISTRIBUTION: Members of TR-30.3 and meeting attendees _______________________________ Copyright Statement ." The contributor grants a free, irrevocable license to the Telecommunications Industry Association (TIA) to incorporate text contained in this contribution and any modifications thereof in the creation of a TIA Standards publication; to copyright in the TIA's name any TIA Standards publication even though it may include portions of this contribution; and at the TIA's sole discretion to permit others to reproduce in whole or in part the resulting TIA Standards publication. Intellectual Property Statement The individual preparing this contribution does not know of patents, the use of which may be essential to a Standard resulting in whole or in part from this contribution. Prior Publication: This paper has been published on the Web since October 20, 1999. Holders of this paper are hereby given license to reproduce this paper for use within their company only. Anyone wishing to republish or distribute copies of this paper outside their own company should contact the author for licensing arrangements. Summary: The author examines the characteristics of testing as specified in the Telecommunications Industry Association (TIA) Telecommunications System Bulletin (TSB) 38, and based on that examination makes specification recommendations for TIA Project Number (PN) 3509. In the conclusion, the author contends that the standard throughput test should take place in an interval of 150 seconds, and that an optional 315-second transfer interval be specifically specified in the Standard. Further, the author's analysis serves as rebuttal for the idea of maintaining the TSB 38 testing technique for V.32 bis and V.34 modems, showing that particularly for V.34 the tests could be considered marginal. Purpose This white paper describes the reasoning behind moving to time-based transfers in throughput testing. This discussion pays particular attention to issues regarding V.90 (PCM) modem testing. Finally, this paper tries to arrive at a rational time interval to use for testing, based on specific goals and requirements. The intent is to provide the basis of a rationale for the change in the procedure for measuring modem data throughput between TIA TSB 38 and PN 3509. Throughput and the Error Model for Modem Testing The usual method for determining the "goodness" or robustness of a signal converter in a modem is to pass blocks of random (or pseudo-random) bits through a modem connection over a telephone channel with well-known characteristics, and measure the number of blocks that were received correctly versus the number of blocks transmitted. The result is called the error ratio measurement; a number of these measurements can be made to estimate the error rate of the data converters over the particular connection. (See TIA TSB 38 and PN 3509 for definitions of error rate and error ratio. There is a difference.) Some number of errors are normal. The typical design goal is to have at most one block error in one thousand blocks. In order to accurately measure the error ratio for a signal converter, and thus accurately characterize the error rate for the modem, there needs to be enough data transferred during each test such that at least ten (10) error events occur during each test. This means that to verify that the modem meets the design specification of one error in one thousand, the tester runs ten times that number of blocks, or ten thousand (10,000) blocks of data, so that if the number of blocks lost or corrupted is less than eleven (11) then the modem meets the criteria. In order to perform a block-error ratio test the modem needs to have the capability of transferring data in a bit-synchronous manner. Fewer and fewer modems include this capability as a standard part of the modem product, because the market need for synchronous modems continues to drop. Also, critics have made a case that users transfer characters , not bits or blocks of bits, and that those users typically take advantage of error control schemes such as the ITU Recommendation V.42 error-control scheme for doing so. This suggests that using a throughput measurement can provide accurate, indirect measurements of block-error ratio. The key to using throughput for accurate error ratio measurement is ensuring that you have enough opportunities for errors in the datastream. Unlike the block-error ratio test, where a fixedsize block of 1000 bits is used as the "test cookie," the V.42 packetizing scheme uses varying size blocks of data. To make matters worse, typical V.42 error control implementations will change the block size to maximize throughput in the face of what V.42 perceives the current error ratio to be. The block size can range from 512 data bits (before headers, flags, and zero-bit insertion) to 8192 data bits. One final note. There is a tradition in the computing industry that the real test of a system is to offer a million opportunities to fail, and measuring the number of successes. In modems, this tradition was upheld by requiring any test of robustness to pass one million bits of data. This was back when one baud (symbol per second) held one bit. TSB 38 Throughput Measurement In TSB 38, the method developed to test modem throughput was to take files of fixed length and content, determine the compressibility of those files, and determine the number of times the files needed to be transmitted in particular conditions. In TSB 38 the basic file size was fixed at 32 kilobytes; there was no technical reason at the time to require larger files, and the size accommodated BERT equipment available at the time so that testing could be performed without waiting for equipment updates. For throughput tests where V.42 bis compression is turned on, a software implementation of V.42 bis was utilized to determine the optimum source file size in order to obtain one million bits of compressed data (before packetizing and zero-bit insertion) through the modem's signal converter. The compression characteristics were determined for a 2048-element dictionary with up to 32 character strings. It was felt that for testing modems conforming to ITU Recommendation V.32 bis that this would result in a sufficiently long test to adequately test the modem. V.32 bis Using files of this length, the tests ran via V.32 bis at least sixty-nine (69) seconds, and resulted in the processing of at least 165,600 symbols of data, each symbol carrying six bits of data. At slower speeds, the number of symbols blossom to over a quarter million, which is getting into the correct magnitude to adequately test the signal converter without extending the tests overmuch. There was time for problems with equalizer convergence to emerge as well. The problem was that errors happen to symbols, not to bits, and so even in V.32 bis there weren't a million opportunities for error; there were about one-sixth the opportunities, worst case. The other effect is that the effect of a single symbol error to the throughput measurement is magnified by the larger blocks used by the V.42 protocol; the retransmission size is larger. (The effect is mitigated by the fact that this happens in real life.) In the worst case, any one of 1,365 symbols can cause a maximum-sized block to be discarded. Smaller V.42 blocks, at 173 symbols each, correlate better with the 166 symbols that cover the 1000-bit block used in BLER testing. V.34 The fastest rate, 33,600 bits/s, uses 3,429 symbols per second with almost 10 bits encoded per symbol. This means that the minimum transfer time for a test is around 29.8 seconds, transferring only 102,184 symbols during a test. At the slower rate of 28,800 bits/s, test time extends to 34.7 seconds, and a maximum of just under 119,000 symbols during the test. With this modulation scheme, the number of opportunities for errors drops close to an order of magnitude under the target value of one million. Interestingly, the transfer time comes close to the time required for the two modems to complete the call establishment sequence, about 20-30 seconds. In short, half of the call time is spent setting up the call, and the other half is spent sending a million bits of data. The data-transfer time is roughly half of the time taken by V.32 bis (as you would expect) so any effects of equalizer drifting or hunting may be masked by the short transfer time. V.90 With V.90, we add an interesting wrinkle: we have two different modulation methods, two different symbol rates, and two different data rates. In the downstream direction, the PCM "modulation method" transfers 8000 symbols per second, encoding just under 7 bits per symbol for a top rate of 53,333 bits/s. In the downstream direction, the V.34 modulation transfers 3200 symbols per second for a top practical rate of 28,800 bit/s. Assuming a one-way transfer downstream, the transfer of the test file would require at least 183/4 seconds. In that time, only 150,000 samples would occur in the downstream channel during the test. This is far too short a time to measure reliably and repeatably with anything other than millisecond-granularity clocks in the testing DTE. Moreover, any stability problem with equalization, clock recovery (a major issue with PCM modems), or drift are not tested by so short a transfer. In a one-way upstream test, the test would extend out to a minimum of 34.7 seconds, with 111,000 symbols transferred during the test. This is to be expected, because the upstream direction uses V.34 modulation techinques, and so all the comments regarding V.34 testing apply here. When we start talking about two-way tests (transmitting data in both directions at the same time) we are no longer measuring what we think we are measuring. The original intent of the two-way test was to measure how well the modem handles CPU resource allocation within the controller portion of the modem. With the asymetric character of the modem, though, some of the resource sharing is masked by the early completion of data transfer in the downstream channel, leaving the entire CPU to catch up in the upstream channel if that is a problem. The PN 3509 Proposed Method The method of testing throughput changes when you move to the PN 3509 method of performing throughput tests. In that document, the testing is done based on time instead of on volume; that is, you transfer data for a set amount of time and determine how many characters you sent in that time. This is exactly the opposite of the method used in TSB 38, where you transmit a fixed number of characters and see how long it takes. Both methods yield the same measurement, characters per second. The difference is in the transmit-stop conditions. Picking a transfer interval Setting the length of the interval for the data transfer is an interesting exercise. This paper suggests that the appropriate transfer time is that which balances tradition -- allow for one million opportunities for error -- with total testing time. The current practice of using around 160,000 opportunities to fail is, as tradition holds, too small. Further, the time selected should be long enough that equalizer settling, jitter, and wandering would be properly exposed by the testing. The current minimum of 19 seconds is entirely too short for this purpose. If we were to strictly follow tradition, and assume that virtually all modem products we were testing would run at symbol rates of 3200 or higher, then the time that allows for one million symbols is 5- 1/4 minutes. This would cause the TIA 3700 network model coverage test, or the PN 3857 integrated network model coverage test, to require just under 19 hours to complete. (This allows 45 seconds for call establishment, handshake, and call teardown and 45 seconds to set up the test equipment for each test channel.) The PN 3857 universal network model coverage test would add another 13.5 hours, for a total test time of 32.5 hours for network model coverage testing. 32.5 hours is just too long for most purposes, although this paper suggests that the 5-1/4-minute (315 seconds) test time be provided as a guideline for "thorough testing." From a statistical standpoint, cutting each test's time doesn't reduce the confidence of the testing overmuch, but it does reduce the amount of time it takes to perform the testing. To make it simple, we would specify the test time as 150 seconds. This cuts the TIA 3700 testing time to 11.2 hours, and full PN 3857 testing time to just over 19 hours. The results from such a test represent roughly a threefold improvement over existing TSB 38 testing for V.32 bis and v.34 testing, and provides a million-screwup-opportunity for V.90 downstream tests, a bonus. Conclusion PN 3509 testing for V.32, V.32 bis, and V.34 testing should use the time-based testing technique so that the resulting tests better measure the effects of equalizer settling and drift, as well as increasing the number of symbols transferred during a test. The benefits of keeping the old technique of using data-volume-based throughput testing for V.32 bis and V.34 modems, specifically for the ability to compare tests to previously performed tests, is outweighted by the improved test coverage of the time-based technique. The argument pales when you realize that much V.90 testing has been published using TSB 38 procedures. Grandfathering one without grandfathering all just doesn't make sense. Additionally, the standard should specify 150 seconds for "standard" testing, and specify the optional value of 315 seconds for performing a million-opportunity test in V.32, V.32 bis, and V.34 testing. We would also need to caution users that any comparison between TSB 38 data and PN 3509 data would have tobe done carefully if at all. It would be better if testing of V.34 and V.90 products was re-done under PN 3509.