Chapter 22 How to Generate a Truly Random Number What is a Random Number? The traditional mathematical definition of randomness is based on Kolmogorov-Chaitin complexity, which defines a random string as one, which has no shorter description than the string itself. Now, is “7” random? Well, can you find a shorter description? Is Archimedes’ constant (our old friend π from Chapter 19) a random number? The description in Chapter 19 is certainly shorter than the infinitude of seemingly random digits of π. Many statistical tests exist for the purpose of seeing if a pattern exists in the purported randomness. “7” may fail some of these tests, while π seems to pass. I guess that part of the answer depends on for what purpose you require randomness. We’ll offer the following definition of a random number: it is a number that I can compute but that you cannot. Such a number has surprisingly practical uses. Use of Randomness Sequences that seem random (and generally in reality are pseudo random) are needed for a wide variety of purposes. They are used for unbiased sampling in the Monte Carlo method, and to imitate stochastic natural processes. They are used in implementing randomized algorithms that require arbitrary choices. And their perceived unpredictability is used in games of chance, and in data encryption and secure communications. To generate a random sequence on a digital computer, one starts with a certain seed, then iteratively applies some transformation to it, progressively extracting as long as possible a random sequence. In general, one considers a sequence “random” if no patterns can be recognized in it, no predictions can be made about it, and no simple description of it can be found by an adversary. But, purely by nature of the fact that the sequence can be generated by iteration of a definite transformation, then a simple description of it certainly does exist. Most current practical random sequence generation computer programs are based on linear congruence relations (like the one we used in Chapter 10), or linear feedback shift registers (analogous to linear cellular automata). The linearity and simplicity of these systems lead to efficient algebraic algorithms for predicting the sequences (or deducing their seeds), and limits their degree of randomness unless they are re-seeded often. Their usefulness comes down to the generation of good random seeds. Bad Seeds As the World Wide Web was gaining broad public appeal, the need for secure transmittal of payment information (such as credit-card numbers) became evident. Netscape’s browser began to use the Secure Sockets Layer (SSL) for such transactions. Basically, SSL protects communications by encrypting messages with a secret key - a large, random number. The security of SSL depends crucially on the unpredictability of this number. In 1995, Goldberg and Wagner showed that the algorithm that Netscape used to compute this “unknowable” seed was deeply flawed. Because Netscape would not release detailed information about how the seed was derived (an important point that we shall return to), Goldberg and Wagner resorted to the (honorable) task of reverse-engineering Netscape’s algorithm by manually decompiling the executable program. Here is what they found: global variable seed; RNG_CreateContext() { (seconds, microseconds) = time of day’ /* time elapsed since 1970 */ pid = process ID; ppid = parent process ID; a = mklcpr (microseconds); b = mklcpr (pid + seconds + (ppid << 12)); seed = MD5 (a, b); } mklcpr (x) { return ((0XDEECE66D * x + 0x2BBB62DC) >> 1); } 22-1 The mklcpr function just scrambles the input a bit, and MD5 is the well-known hashing function. The seed generated depends only on the values of a and b, which in turn depend on just the time of day, the process ID, and the parent process ID. If an adversary can predict these three values, the seed can be computed and the whole scheme is compromised. The time of day can be known to within a certain precision, often better than a second, which means that there are only one million possible choices for the microsecond part (and often a lot less because the clocks on most computer systems do not have true microsecond resolution). If you are running on the same machine that is generating the seed, the process IDs will be known, and, if you are not, it is usually possible to guess their values. Firstly, ppid is often 1, or, if not, then it is usually just a bit smaller than the pid. The pid s are normally not considered secret and are often present in message packets from the system. Goldberg and Wagner showed that one could deduce the seed in less than a minute. Shortly afterwards, it was discovered that Kerberos V4 (another secure protocol) suffered from the same problem, maybe even worse than Netscape since it used the standard Unix random() function instead of the much better MD5. At about the same time, it was announced that although the MIT-MAGIC-COOKIE-1 key had 56 bits, only 256 seed values were possible because of poor use of the rand() function: key = rand() % 256; /* take the remainder after dividing by 256 */ Similar examples of flawed thinking and sloppy implementation abound, e.g. Sun derived NFS file handles (which serve as tokens to control access to a file and therefore need to be unpredictable) from the traditional pid and time-of-day, but forgot to load the time-of-day variable, and on and on. The conclusion is that generating good random numbers is hard and that in spite of that (or perhaps, because of that!) it is often done poorly. As Donald Knuth once quipped: “A random number generator should not be chosen at random”. A good random number process is not just a complicated or (as many would think sufficient) an obscure (“secret”) procedure. Netscape, IBM and many, many others have fallen into this trap that the process must be kept secret. But we must heed Kerckhoffs’ second principle: “compromise of the system should not inconvenience the correspondents”. That is: the system should remain secure even if all the details about its inner workings, down to the actual source code, are known to all, and even if the adversaries have access to the very system producing the randomness. The Entropy Pool One solution to the randomness problem is to maintain a pool of information pertaining to physical parameters, properties, and activity of the system. Anything that is determined by external factors can be – and should be - used as input to the pool, such as the time between keystrokes, the timing of disk interrupts, number of network packages arrived, and the like. On a multi-user system, such as the AS/400, the number of page faults, the number of disk read/writes, the time to wait until eligible to get the next time slice, and other such information depends on the overall activity in a manner that is hard to predict or precisely control. These things can go into the pool too. The word entropy is often used as synonymous with randomness and the pool of randomness information is often called the entropy pool. One could have several entropy pools: a fast pool that contains “distilled” values from the other, slower, pools, and a number of ever slower pools containing further data about usage statistics, event occurrences, etc, being continuously collected and percolating up to the faster pools. When you need a random number or a seed for a random number generator, you “consult” the fast pool. Typically a cryptographically sound “message digest”, such as MD5 or SHA (the Secure Hash Algorithm) is computed over the pool and used as your next random number or seed. In this chapter we shall describe the construction of an entropy pool for the AS/400. Sources of Entropy We shall utilize the following sources of randomness (or entropy): ● User input (e.g. pass phrases) ● Timing information, such as time of day and processor time used ● Resource Management Data collected by the system ● Variation of when the pool collector runs 22-2 Some of this information is also available to an adversary, but not at the same time, so the data will be somewhat different, that is: an adversary will have to try a range of values instead of knowing exactly what the value is. By collecting a wide variety of data, all of which can only be known to the adversary as a range rather than as definite values, we enlarge the search space from which our values can be found. If this process goes on for long enough time, it begins to become unfeasible to guess which values within the compounded sets of ranges we are actually using. This “shotgun” approach tends to make life miserable for a would-be adversary. Timing Information The Materialize Process Attributes instruction, MATPRATR, has an option to return the amount of processor time (“CPU-time”) used until now by the current thread. An adversary could be monitoring my invocation stack and ask for the same information at the time the entropy collector runs. Because of the dynamic nature of the situation he is likely to get only an approximation to the result I would get. This is all we need: to force him to consider a range of values. All those ranges multiply together to form a large space to search. Here is how to retrieve the processor time: DCL SPCPTR .P24-ATTR INIT(P24-ATTR); DCL DD P24-ATTR CHAR(128) BDRY(16); DCL DD P24-MAX-SIZE BIN(4) DCL DD P24-ACT-SIZE BIN(4) DCL DD P24-CPU-TIME-USED CHAR(8) DEF(P24-ATTR) POS( 1); DEF(P24-ATTR) POS( 5); DEF(P24-ATTR) POS(19); ENTRY GET-THREAD-ACTIVITY INT; CPYNV P24-MAX-SIZE, 128; MATPRATR .P24-ATTR, *, X'24'; /* THREAD ACTIVITY */ CPYBLA CALLI TIME-VALUE, P24-CPU-TIME-USED; ADD-TIME-TO-ENTROPY, *, .ADD-TIME-TO-ENTROPY; The TIME-VALUE is in the standard 64-bit timestamp format that we explored in Chapter 4: DCL DD TIME-VALUE CHAR(8); DCL DD TIME-VALUE-HI BIN(4) UNSGND DEF(TIME-VALUE) POS(1); DCL DD TIME-VALUE-LO BIN(4) UNSGND DEF(TIME-VALUE) POS(5); We convert the timestamp to a number as we did in Chapter 4: DCL DD TIMESTAMP DCL DD TWO**32 DCL DD TWO**15 PKD(21,0); /* CAN HOLD UNSIGNED 64-BIT INTEGER */ PKD(11,0) INIT(P'4294967296'); PKD(11,0) INIT(P'32768'); DCL INSPTR .ADD-TIME-TO-ENTROPY; ENTRY ADD-TIME-TO-ENTROPY INT; MULT TIMESTAMP, TIME-VALUE-HI, TWO**32; ADDN(S) TIMESTAMP, TIME-VALUE-LO; DIV ENTROPY-VALUE, TIMESTAMP, TWO**15; We divide by 215 because the lower 15 bits are all zeroes anyway. The result is what I call the ENTROPYVALUE for that piece of information. We finally divide by 2 31 and use the remainder (stored as an unsigned 4-byte value) as the bits to add to the entropy pool: DCL DD ENTROPY-BITS DCL DD ENTROPY-VALUE BIN(4) UNSGND; PKD(31,0); DCL INSPTR .ADD-TO-ENTROPY-POOL; ENTRY ADD-TO-ENTROPY-POOL INT; REM ENTROPY-BITS, ENTROPY-VALUE, TWO**31; CPYBLA ENTROPY-POOL(POOL-POSITION:4), ENTROPY-BITS; Entropy Pool Format The entropy pool holds a number (currently eight) of entropy pool entries. Each entry consists of seven pairs of 4-byte values. The first value of the pair is a particular measurement and the second value is the real-time duration between measurements. 22-3 Here is a (pseudo) declaration of the structure: DCL DD ENTROPY-POOL; DCL DD ENTROPY-POOL-ENTRY (8); DCL DD ENTROPY-POOL-ENTRY-PAIR (7); DCL DD ENTROPY-POOL-ENTRY-PAIR-MEASUREMENT DCL DD ENTROPY-POOL-ENTRY-PAIR-DURATION BIN(4) UNSGND; BIN(4) UNSGND; To make it harder for the adversary to replicate the entropy pool, the seven measurements in an entry are not made at the same time. They are staggered in time by a variable amount (the yield time) as explained below. Each time the pool is consulted, a new entry is added to the pool with a new set of measurements. When the pool overflows (it only holds eight entries, so that happens quickly), the oldest entry is discarded to make room for the new entry. In this way the pool is constantly renewed. We include an option to flush the pool and start with a clean slate. The Yield Time The YIELD MI-instruction allows us to introduce considerable variability in when we execute a particular piece of code. It works like this: upon execution of YIELD, the dispatching algorithm is run. If another thread of equal or higher priority is eligible to run, then one of these threads is chosen and dispatched to run, and our thread will wait until it’s its turn again. Otherwise, the current thread resumes execution. We could execute YIELD several times in a row, hoping that some other thread will run so that it will take some time before we are given control again. We could even compute a hash of the current contents of the entropy pool, form an integer from some of the middle bits of the hash-value, divide that integer by a parameter MAX-YIELDS and use the remainder to determine how many times to execute the YIELD instruction. We measure the total amount of real time that this whole process takes (the yield time) and use that as the duration part of the entropy pair. The yield time will depend on other activities going on at the same time, and is hard to predict on a busy system. In all fairness, we should note that the yield time could be manipulated somewhat by an adversary because he could cause some of that “other activity”. The code reads: REM MATMATR CPYBLA : CPYNV YIELD; MATMATR CPYBLA SUBLC MULT ADDN(S) DIV TIMES-TO-YIELD, SHA-HASH-MIDDLE, CUR-MAX-YIELDS; .MACHINE-ATTR, X'0100'; TIME-BEFORE, MAT-TIMESTAMP; /* clock value before the loop */ LAST-YIELDS, TIMES-TO-YIELD; SUBN(SB) TIMES-TO-YIELD, 1/HI(=-1); .MACHINE-ATTR, X'0100'; TIME-AFTER, MAT-TIMESTAMP; /* clock value after the loop */ TIME-VALUE, TIME-AFTER, TIME-BEFORE; TIMESTAMP, TIME-VALUE-HI, TWO**32; TIMESTAMP, TIME-VALUE-LO; ENTROPY-VALUE, TIMESTAMP, TWO**15; Finally, we add that value to the pool and return for the next measurement: REM ADDN(S) CPYBLA ADDN(SB) ENTROPY-BITS, ENTROPY-VALUE, TWO**31; POOL-POSITION, 4; ENTROPY-POOL(POOL-POSITION:4), ENTROPY-BITS; POOL-POSITION, 4/POS(.ADD-TO-ENTROPY-POOL); Resource Management Data The Materialize Resource Management Data MI-instruction, MATRMD, is used to retrieve activity data collected by the system. MATRMD .Receiver, Control; The first operand is a space pointer to a receiver area. The second operand is an 8-byte character value. The first byte identifies the generic type of information requested, and the remaining 7 bytes must be binary zeroes. Operand 1 points to the result, which has the following format: DCL DD RESOURCES CHAR(nnnn) BDRY(16); DCL DD RMD-MAX-SIZE BIN(4) DEF(RESOURCES) POS( 1) INIT(nnnn); 22-4 DCL DD DCL DD DCL DD RMD-ACT-SIZE BIN(4) DEF(RESOURCES) POS( 5); RMD-TIMESTAMP CHAR(8) DEF(RESOURCES) POS( 9); RMD-DATA CHAR(nnnn) DEF(RESOURCES) POS(17); As usual, the receiver contains the number of bytes provided ( MAX-SIZE) and actually available for materialization (ACT-SIZE). One could allocate the receiver dynamically depending on the number of bytes available, but we’ll keep it simple here and allocate a fixed sized receiver data area (5000 bytes). The TIMESTAMP variable contains the standard 64-bit timestamp for the time at which the data is valid (usually just the current clock value). The detailed format of the data depends on the type of information requested. We’ll only document the format of data that we have considered is carrying a reasonable amount of entropy. Processor Utilization Selection option x‘01’ returns the processor utilization, which is the total amount of processor time used, both by threads and by internal machine functions, since the last IPL: DCL DD RMD-PROCESSOR-TIME CHAR(8) DEF(RMD-DATA) POS( 1); ENTRY GET-PROCESSOR-UTILIZATION INT; CPYBLA WHICH-RESOURCE, X'01'; /* PROCESSOR UTIL MATRMD .RESOURCES, WHICH-RESOURCE; CPYBLA CALLI B */ TIME-VALUE, RMD-PROCESSOR-TIME; ADD-TIME-TO-ENTROPY, *, .ADD-TIME-TO-ENTROPY; .RETURN; Storage Management Counters Selection option x‘03’ returns various counters related to storage management: ● Access Pending is a count of the number of times that a paging request must wait for the completion of a different request for the same page ● Storage Pool Delays is a count of the number of times that threads have been momentarily delayed by the unavailability of a main storage frame in the proper pool ● Directory Look-up operations is a count of the number of times that auxiliary storage directories were interrogated Directory Page Faults is a count of the number of times that a page of an auxiliary storage directory was fetched ● ● ● ● ● DCL DCL DCL DCL DCL DCL DCL DCL DCL DD DD DD DD DD DD DD DD DD Access Group Member Page Faults is a count of the number of times that a page of an object contained in an access group was fetched Microcode Page Faults is a count of the number of times that a page of LIC was fetched Microtask read operations is a count of the number of reading one or more pages on behalf of a task rather than a thread. Microtask write operations is a count of the number of writing one or more pages on behalf of a task rather than a thread. RMD-ACCESS-PENDING RMD-STG-POOL-DELAYS RMD-DIR-LOOKUPS RMD-DIR-PFAULTS RMD-AG-MBR-PFAULTS RMD-LIC-PFAULTS RMD-TASK-READS RMD-TASK-WRITES RMD-RESERVED1 BIN(2) BIN(2) BIN(4) BIN(4) BIN(4) BIN(4) BIN(4) BIN(4) BIN(4) UNSGND UNSGND UNSGND UNSGND UNSGND UNSGND UNSGND UNSGND UNSGND DEF(RMD-DATA) DEF(RMD-DATA) DEF(RMD-DATA) DEF(RMD-DATA) DEF(RMD-DATA) DEF(RMD-DATA) DEF(RMD-DATA) DEF(RMD-DATA) DEF(RMD-DATA) ENTRY GET-STORAGE-USAGE-COUNTERS INT; CPYBLA WHICH-RESOURCE, X'03'; MATRMD .RESOURCES, WHICH-RESOURCE; CPYNV ACCUMULATOR, 0; ADDN(S) ADDN(S) ADDN(S) MULT(S) ACCUMULATOR, ACCUMULATOR, ACCUMULATOR, ACCUMULATOR, /* STORAGE COUNTERS */ RMD-RESERVED1; RMD-ACCESS-PENDING; RMD-STG-POOL-DELAYS; 10; 22-5 POS( 1); POS( 3); POS( 5); POS( 9); POS(13); POS(17); POS(21); POS(25); POS(29); ADDN(S) ADDN(S) ADDN(S) MULT(S) ACCUMULATOR, ACCUMULATOR, ACCUMULATOR, ACCUMULATOR, RMD-DIR-PFAULTS; RMD-AG-MBR-PFAULTS; RMD-LIC-PFAULTS; 100; ADDN(S) ADDN(S) MULT(S) ADDN(S) ACCUMULATOR, ACCUMULATOR, ACCUMULATOR, ACCUMULATOR, RMD-TASK-READS; RMD-TASK-WRITES; 1000; RMD-DIR-LOOKUPS; CPYNV CALLI B ENTROPY-VALUE, ACCUMULATOR; ADD-TO-ENTROPY-POOL, *, .ADD-TO-ENTROPY-POOL; .RETURN; Why didn’t we just add up the various counters? Because adding several numbers destroys a lot of the entropy contained in them. If the sum of 5 non-negative numbers is 10, there are 1001 different ways they could sum to 101. All that information is lost by just summing the numbers. If the numbers were correlated, e.g. all had the same high-order digits, e.g. 5000, 5008, 5004, and the like, then the entropy would only derive from the changing low-order digits, so it is only important to not add up the low-order digits. A somewhat crude way of achieving this would be to multiply the running sum by ten before adding in the next number. The above code does something like this, multiplying the ACCUMULATOR by various amounts now and then. What we are trying to achieve is to but 10 pounds in a 5-pound bag. We’ll use the same technique for some of the other measurements. If there is a lot of variation in the numbers we are accumulating, this technique is too crude and we lose some entropy, but then we have a lot to lose from, so we come out with a reasonable amount after all. After all this hand waving, it is time to get back to the code. Main Storage Pool Information Selection option x‘09’ returns various counters related to storage pool management: ● Data Written is the amount of data written from a storage pool to disk to satisfy demand for resources from the pool (i.e. the overhead incurred to service threads) ● Thread Interruptions (Database and Non-DB) is the total number of interruptions to threads. Data transferred (Database and Non-DB) is the amount of data transferred from disk to the pool ● There are a several pools, and we must iterate through all pools collecting data as we go. Although we lose entropy by adding the numbers for all pools, we retain the entropy associated with the fact that that is activity on the pools: DCL DD RMD-NBR-OF-POOLS DCL DD RMD-POOLS(100) BIN(2) UNSGND DEF(RMD-DATA) POS( 5); CHAR(32) DEF(RMD-DATA) POS(17); DCL DD FOR-THIS-POOL PKD(21,0); DCL DD POOL-NBR BIN(2) UNSGND; DCL DD THE-POOL CHAR(32); DCL DD POOL-DATA-WRITTEN BIN(4) DCL DD POOL-THREAD-INT-DB BIN(4) DCL DD POOL-THREAD-INT-NON BIN(4) DCL DD POOL-DATA-POOL-DB BIN(4) DCL DD POOL-DATA-POOL-NON BIN(4) UNSGND UNSGND UNSGND UNSGND UNSGND DEF(THE-POOL) DEF(THE-POOL) DEF(THE-POOL) DEF(THE-POOL) DEF(THE-POOL) ENTRY GET-STORAGE-POOL-ACTIVITY INT; CPYBLA WHICH-RESOURCE, X'09'; MATRMD .RESOURCES, WHICH-RESOURCE; CPYNV ACCUMULATOR, 0; CPYNV FOR-NEXT-POOL: CPYNV CMPNV(B) CPYBLA CMPBLAP(B) MULT(RS) 1 POS( 5); POS( 9); POS(13); POS(17); POS(21); /* STORAGE POOLS */ POOL-NBR, 1; FOR-THIS-POOL, 0; POOL-NBR, RMD-NBR-OF-POOLS/HI(DONE-WITH-POOL); THE-POOL, RMD-POOLS(POOL-NBR); THE-POOL, X'00', X'00'/EQ(=+2); ACCUMULATOR, P'1.2345';: N (non-negative) integers can sum to S in: W = (S + N - 1)!/S!/(N - 1)! ways 22-6 ADDN(S) MULT(S) FOR-THIS-POOL, POOL-DATA-WRITTEN; FOR-THIS-POOL, 10; ADDN(S) ADDN(S) MULT(S) FOR-THIS-POOL, POOL-THREAD-INT-DB; FOR-THIS-POOL, POOL-THREAD-INT-NON; FOR-THIS-POOL, 1000; ADDN(S) ADDN(S) ADDN(S) ADDN(SB) DONE-WITH-POOL: FOR-THIS-POOL, POOL-DATA-POOL-DB; FOR-THIS-POOL, POOL-DATA-POOL-NON; ACCUMULATOR, FOR-THIS-POOL; POOL-NBR, 1/POS(FOR-NEXT-POOL); CPYNV CALLI B ENTROPY-VALUE, ACCUMULATOR; ADD-TO-ENTROPY-POOL, *, .ADD-TO-ENTROPY-POOL; .RETURN; Disk Storage Information Selection option x‘12’ returns various counters related to disk storage, or ASPs (Auxiliary Storage Pools) as they are called on the AS/400. For each storage unit (i.e. disk) we extract the numbers of the following: ● ● ● ● ● ● ● ● Blocks transferred to main storage Blocks transferred from main storage Requests for transfer to main storage Requests for transfer from main storage Permanent Blocks transferred from main storage Requests for Permanent data transfer from main storage Times the disk queue was checked to see if it was empty Times the disk queue as actually empty There are generally several disk units, and we must iterate through all units collecting data as we go. Although we lose entropy by adding the numbers for all units, we retain the entropy associated with the fact that that is activity on the units: DCL DCL DCL DCL DD DD DD DD FOR-THIS-DISK PKD(21,0); DISK-NBR BIN(2) UNSGND; RMD-NBR-OF-DISKS BIN(2) UNSGND DEF(RMD-DATA) POS( 3); RMD-OFFSET-TO-DISKS BIN(4) UNSGND DEF(RMD-DATA) POS(29); DCL SPCPTR .THE-DISK; DCL DD THE-DISK CHAR(208) BAS(.THE-DISK); DCL DD DISK-UNIT-ID CHAR(22) DCL DD DISK-BLOCKS-IN BIN(4) UNSGND DCL DD DISK-BLOCKS-OUT BIN(4) UNSGND DCL DD DISK-REQIO-IN BIN(4) UNSGND DCL DD DISK-REQIO-OUT BIN(4) UNSGND DCL DD DISK-PERM-BLKS-OUT BIN(4) UNSGND DCL DD DISK-REQIO-PERMS BIN(4) UNSGND DCL DD * CHAR(8) DCL DD DISK-QUEUE-SAMPLED BIN(4) UNSGND DCL DD DISK-QUEUE-EMPTY BIN(4) UNSGND DEF(THE-DISK) DEF(THE-DISK) DEF(THE-DISK) DEF(THE-DISK) DEF(THE-DISK) DEF(THE-DISK) DEF(THE-DISK) DEF(THE-DISK) DEF(THE-DISK) DEF(THE-DISK) POS(123); POS(145); POS(149); POS(153); POS(157); POS(161); POS(165); POS(169); POS(177); POS(181); ENTRY GET-DISK-ACTIVITY-COUNTERS INT; MULT(R) ACCUMULATOR, LAST-YIELDS, PRV-HASH-LO; CPYBLA TIME-VALUE, RMD-TIMESTAMP; DIV(SB) TIME-VALUE-LO, TWO**15/EQ(=+2); MULT(S) ACCUMULATOR, TIME-VALUE-LO;: REM(SB) LAST-YIELDS, 4/NZER(DONE-WITH-DISKS); CPYBLA MATRMD CPYNV SETSPP ADDSPP CPYNV FOR-NEXT-DISK: CPYNV CMPNV(B) MULT(RS) ADDN(S) WHICH-RESOURCE, X'12'; .RESOURCES, WHICH-RESOURCE; ACCUMULATOR, 0; /* DISK UTILIZATION */ .THE-DISK, RESOURCES; .THE-DISK, .THE-DISK, RMD-OFFSET-TO-DISKS; DISK-NBR, 1; FOR-THIS-DISK, 0; DISK-NBR, RMD-NBR-OF-DISKS/HI(DONE-WITH-DISKS); ACCUMULATOR, P'1.2345'; FOR-THIS-DISK, DISK-QUEUE-SAMPLED; 22-7 ADDN(S) MULT(S) FOR-THIS-DISK, DISK-QUEUE-EMPTY; FOR-THIS-DISK, 100; ADDN(S) ADDN(S) ADDN(S) ADDN(S) MULT(S) FOR-THIS-DISK, FOR-THIS-DISK, FOR-THIS-DISK, FOR-THIS-DISK, FOR-THIS-DISK, ADDN(S) ADDN(S) FOR-THIS-DISK, DISK-BLOCKS-IN; FOR-THIS-DISK, DISK-REQIO-IN; DISK-PERM-BLKS-OUT; DISK-REQIO-PERMS; DISK-BLOCKS-OUT; DISK-REQIO-OUT; 1000; ADDN(S) ACCUMULATOR, FOR-THIS-DISK; ADDSPP .THE-DISK, .THE-DISK, 208; ADDN(SB) DISK-NBR, 1/POS(FOR-NEXT-DISK); DONE-WITH-DISKS: CPYNV CALLI B ENTROPY-VALUE, ACCUMULATOR; ADD-TO-ENTROPY-POOL, *, .ADD-TO-ENTROPY-POOL; .RETURN; Because it is fairly expensive in CPU-cycles to get this information, we only refresh the disk information on the average every fourth time, filling in with a timestamp and some function of the previous yield-time the other 75% of the time. Activity Level Control Data Selection option x‘16’ returns the Activity Level control information. At the MI, activity levels are called “multiprogramming levels” or MPLs. MPLs are grouped into classes. A class determines how many concurrent threads you can have at that level. We aggregate the following information from all classes: ● ● DCL DCL DCL DCL DD DD DD DD The number of threads assigned to the class The number of active to wait transitions FOR-THIS-MPL PKD(21,0); MPL-NBR BIN(2) UNSGND; OFFSET-TO-MPLS BIN(4) INIT(20); RMD-NBR-OF-MPLS BIN(2) UNSGND DEF(RMD-DATA) POS(3); DCL SPCPTR .THE-MPL; DCL DD THE-MPL CHAR(32) BAS(.THE-MPL); DCL DD MPL-CURRENT-MPL BIN(4) UNSGND DEF(THE-MPL) POS( 9); DCL DD MPL-NBR-OF-THREADS BIN(4) UNSGND DEF(THE-MPL) POS(17); DCL DD MPL-NBR-ACT-WAIT-TRANS BIN(4) UNSGND DEF(THE-MPL) POS(25); ENTRY GET-MULTIPROGRAMMING-LEVELS INT; CPYBLA WHICH-RESOURCE, X'16'; MATRMD .RESOURCES, WHICH-RESOURCE; CPYNV ACCUMULATOR, 0; SETSPP ADDSPP CPYNV FOR-NEXT-MPL: CPYNV CMPNV(B) CMPNV(B) MULT(RS) ADDN(S) MULT(S) ADDN(S) /* MPL CONTROL INFO */ .THE-MPL, RMD-DATA; .THE-MPL, .THE-MPL, OFFSET-TO-MPLS; MPL-NBR, 1; FOR-THIS-MPL, 0; MPL-NBR, RMD-NBR-OF-MPLS/HI(DONE-WITH-MPLS); MPL-CURRENT-MPL, 0/EQ(=+2); ACCUMULATOR, P'1.2345';: FOR-THIS-MPL, MPL-NBR-OF-THREADS; FOR-THIS-MPL, 10; FOR-THIS-MPL, MPL-NBR-ACT-WAIT-TRANS; ADDN(S) ACCUMULATOR, FOR-THIS-MPL; ADDSPP .THE-MPL, .THE-MPL, 32; ADDN(SB) MPL-NBR, 1/POS(FOR-NEXT-MPL); DONE-WITH-MPLS: CPYNV CALLI B ENTROPY-VALUE, ACCUMULATOR; ADD-TO-ENTROPY-POOL, *, .ADD-TO-ENTROPY-POOL; .RETURN; 22-8 Can We Measure the Amount of Entropy in the Pool? This is really a difficult task. The entropy is a measure of the “unexpectedness” of the data in the pool. Consider the following little anecdote. It is about a ship’s Captain and his First Officer. One day the First Officer learned about some personal problem and got drunk. The Captain wrote in the logbook (that is supposed to record significant events): “Today the First Officer was drunk”. When the First Officer finished his next watch, he wrote in the logbook: “Today the Captain was sober”. The more unexpected the data is, the harder it is for an adversary to guess of predict it A crude, but effective, measure of the entropy in the pool is the compressibility of the pool. Completely random data cannot be compressed to a shorter string (The US patent office no longer grants patents on perpetual motion machines, but has recently granted a patent on the mathematically impossible process of compression of truly random data: 5,533,051 "Method for Data Compression". But then, they grant patents on things like “one-click shopping” and US patent 5,443,036 - look it up!). If you compress the pool, the length of the compressed result is usually shorter than the length of the data in the pool, because the pool data is not perfectly random. Now, add another entry to the pool (causing the oldest entry to be discarded). Compress the pool again. If no new information was added the length of the compressed result stays about the same, because the “new” data is already in the compressor’s dictionary. If, on the other hand the new entry is significantly different from the previous entry, the compression is going to be less good and the length of the compressed result increases. We learned in Chapter 12 how to use the CPRDATA MI-instruction to compress a string of data. Apply that to the entire pool: DCL SPCPTR .COMPRESS-CONTROL INIT(COMPRESS-CONTROL); DCL DD COMPRESS-CONTROL CHAR(64) BDRY(16); DCL DD CMPR-SOURCE-LENGTH BIN(4) DEF(COMPRESS-CONTROL) POS( 1); DCL DD CMPR-BYTES-AVAILABLE BIN(4) DEF(COMPRESS-CONTROL) POS( 5); DCL DD CMPR-LENGTH BIN(4) DEF(COMPRESS-CONTROL) POS( 9); DCL DD CMPR-ALGORITHM BIN(2) DEF(COMPRESS-CONTROL) POS(13); DCL DD CMPR-MBZ CHAR(18) DEF(COMPRESS-CONTROL) POS(15); DCL SPCPTR .CMPR-SOURCE DEF(COMPRESS-CONTROL) POS(33) INIT(ENTROPY-POOL); DCL SPCPTR .CMPR-RESULT DEF(COMPRESS-CONTROL) POS(49) INIT(ENTROPY-COMPRESSED); DCL DD FP BIN(2); DCL DD SIZE BIN(2); DCL DD ENTROPY-COMPRESSED CHAR(600); ENTRY ESTIMATE-ENTROPY-ADDED INT; CPYNV CMPR-SOURCE-LENGTH, POOL-POSITION; CPYNV CMPR-BYTES-AVAILABLE, 600; CPYNV CMPR-ALGORITHM, 2; /* LZ1 */ CPYBREP CMPR-MBZ, X'00'; CPRDATA .COMPRESS-CONTROL; CPYBREP SUBN(S) MULT(S) DIV(R) ENTROPY-COMPRESSED, X'00'; CMPR-LENGTH, 12; /* - FIXED OVERHEAD */ CMPR-LENGTH, 100; /* to get percentage */ CUR-CONFIDENCE, CMPR-LENGTH, CMPR-SOURCE-LENGTH; After subtracting a 12-byte fixed overhead, we computed the resulting length as a percentage of the input length. The result, CONFIDENCE, will serve as a “confidence” indicator. Experiments show that if the compressed length is above 80% of the original length, there is a decent amount of entropy in the pool. We can now monitor the entropy of the pool. If the CONFIDENCE is too low, we can increase the yield-time thus forcing the entropy collector to run for a longer time, increasing the chances of changes in the Resource Management Data as well as giving us more variation of the duration (more entropy right there) of the collections steps. Aging the Entropy Pool Having calculated the “confidence” number one more task awaits us: if the pool is now filled, we must discard the oldest entry, to make room for the next entry: SUBN CMPNV(B) FP, ENTROPY-POOL-SIZE, POOL-STEP; POOL-POSITION, FP/LO(.RETURN); 22-9 POOL-IS-FILLED: SUBN(S) POOL-POSITION, POOL-STEP; SUBN SIZE, POOL-POSITION, 1; ADDN FP, POOL-STEP, 1; CPYBOLAP ENTROPY-POOL (1:SIZE), ENTROPY-POOL (FP:SIZE), X'00'; B .RETURN; Note the CPYBOLAP instruction that works like CPYBLAP, but allows Overlapping operands. Having identified the contents and outlined the management of the entropy pool, we can put it all together in the MIGETETP program to be discussed next. The Get Entropy Program (MIGETETP) Let’s first look at the parameters to MIGETETP. The first parameter is the SHA hash value of the entropy pool, or more precisely of the entire internal state of MIGETETP, which includes the entropy pool. The hash value is 160 bits (20 bytes) long: DCL SPCPTR .PARM1 PARM; DCL DD PARM1 CHAR(20) BAS(.PARM1); DCL DD PARM-HASH CHAR(20) DEF(PARM1) POS(1); The second parameter is a control parameter. It has three input fields: ● The value of MAX-YIELDS that determines the maximum number of times the program will yield its timeslice ● The RESTART parameter where a value of “Y” will flush the pool. A value of “Y” is automatically reset to “N” by MIGETETP ● A pass PHRASE that will included in calculation of the hash value. And two output fields: ● ● The CONFIDENCE indicator The number of CALLS that have been made to the program. DCL SPCPTR .PARM2 PARM; DCL DD PARM2 CHAR(90) BAS(.PARM2); DCL DD PARM-MAX-YIELDS BIN(4) DEF(PARM2) DCL DD PARM-CONFIDENCE BIN(4) DEF(PARM2) DCL DD PARM-RESTART CHAR(1) DEF(PARM2) DCL DD PARM-PHRASE CHAR(64) DEF(PARM2) DCL DD PARM-CALLS PKD(21,0) DEF(PARM2) DCL DD * CHAR(6) DEF(PARM2) POS( 1); POS( 5); POS( 9); POS(10); POS(74); POS(85); /* /* /* /* /* /* DFLT = 1000 */ > 80%: GOOD */ Y OR N */ USER SUPPL. */ NBR OF CALLS*/ UNUSED */ The optional third parameter exposes the entire internal state of the program. It is used for debugging only and should be removed from the production version: DCL SPCPTR .PARM3 PARM; DCL DD PARM3 CHAR(600) BAS(.PARM3); DCL DD PARM-STATE CHAR(640) DEF(PARM3) POS(1); DCL OL PARMS(.PARM1, .PARM2, .PARM3) PARM EXT MIN(2); The Internal State The CURRENT field is a copy of the control parameter: DCL SPCPTR .STATE INIT(STATE); DCL DD STATE CHAR(640) BDRY(16); DCL DD CURRENT CHAR(90) DEF(STATE) POS( DCL DD CUR-MAX-YIELDS BIN(4) DEF(CURRENT) DCL DD CUR-CONFIDENCE BIN(4) DEF(CURRENT) DCL DD CUR-RESTART CHAR(1) DEF(CURRENT) DCL DD CUR-PHRASE CHAR(64) DEF(CURRENT) DCL DD * CHAR(17) DEF(CURRENT) DCL DCL DCL DCL DD DD DD DD NBR-OF-CALLS NBR-OF-PARMS POOL-POSITION LAST-YIELDS PKD(21,0) BIN(2) BIN(4) BIN(4) DEF(STATE) DEF(STATE) DEF(STATE) DEF(STATE) 1); POS( 1) INIT(1000); POS( 5); POS( 9) INIT("N"); POS(10); POS(74); POS( 91) INIT(P'0'); POS(103); POS(105) INIT(1); POS(109) INIT(0); DCL DD ENTROPY-POOL-SIZE BIN(2) DEF(STATE) POS(113) INIT(500); DCL DD ENTROPY-POOL CHAR(500) DEF(STATE) POS(115); 22-10 DCL DD POOL-STEP BIN(2) DEF(STATE) POS(615) INIT(0); DCL DD PREVIOUS-HASH CHAR(20) DEF(STATE) POS(621); DCL DD PRV-HASH-LO BIN(2) UNSGND DEF(PREVIOUS-HASH) POS(19); DCL INSPTR .RETURN; The NBR-OF-CALLS variable is increased by 1 every time the program is called. Its current value is returned to the caller, who can then check that no other program (e.g. a rogue program that is part of the application and runs in the same process as the caller) has accessed the entropy pool. It is trivial to change the program to have its state information belong to the caller: DCL SPCPTR .PARM3 PARM; DCL DD PARM3 CHAR(640) BAS(.PARM3); DCL OL PARMS(.PARM1, .PARM2, .PARM3) PARM EXT MIN(3); DCL DD STATE CHAR(640) DEF(PARM3) POS(1); DCL DD CURRENT CHAR(90) DEF(STATE) POS( … 1); Generation of Entropy After some initial house-keeping: ENTRY * (PARMS) EXT; STPLLEN NBR-OF-PARMS; CPYBLA CURRENT, PARM2; ADDN(S) NBR-OF-CALLS, 1; CPYNV PARM-CALLS, NBR-OF-CALLS; CMPBLA(B) CPYBREP CPYBREP CPYNV CPYNV /* get control values */ /* increment call nbr */ /* return new value */ CUR-RESTART, "Y"/NEQ(GET-MACHINE-ACTIVITY); PREVIOUS-HASH, X'00'; /* wipe the pool if */ ENTROPY-POOL, X'00'; /* RESTART = ‘Y’ */ POOL-STEP, 0; POOL-POSITION, 1; We get a new set of measurements: GET-MACHINE-ACTIVITY: CALLI GET-THREAD-ACTIVITY, *, CALLI GET-PROCESSOR-UTILIZATION, *, CALLI GET-STORAGE-USAGE-COUNTERS, *, CALLI GET-STORAGE-POOL-ACTIVITY, *, CALLI GET-DISK-ACTIVITY-COUNTERS, *, CALLI GET-MULTIPROGRAMMING-LEVELS,*, CALLI .RETURN; .RETURN; .RETURN; .RETURN; .RETURN; .RETURN; ESTIMATE-ENTROPY-ADDED, *, .RETURN; With an updated entropy pool we can compute a new hash value, but first we return updated controlvariables: CMPNV(B) CMPBLA(B) CMPBLA(B) CPYNV NBR-OF-PARMS, 3/NEQ(=+4); /* this version only exposes the state */ CUR-RESTART, "Y"/EQ(=+3); /* if the caller supplies the previous */ PARM-HASH, PREVIOUS-HASH/EQ(=+2); /* value of the entropy hash */ NBR-OF-PARMS, 2;: CPYBLA CPYBLA CPYNV CUR-RESTART, "N"; PARM-RESTART, CUR-RESTART; PARM-CONFIDENCE, CUR-CONFIDENCE; MAKE-FINAL-HASH: CPYBREP CIPHER-WORKAREA, X'00'; CPYNV CIPHER-LENGTH, 640; /* hash is computed over */ CIPHER .PARM1, CIPHER-OPTS, .STATE; /* the entire internal state */ CPYBLA PREVIOUS-HASH, PARM-HASH; CMPNV(B) CPYBLA NBR-OF-PARMS, 3/NEQ(=+2); PARM3, STATE;: RTX *; 22-11 The Entropy Monitor Program (MIETPMON) Programs that generate random output are notoriously difficult to debug; how do you know that the output is correct when its main characteristic is that it be unknowable? The only way I know of is peer review (one of the main purposes of this chapter). It is, however, comforting to have some kind of feeling for what the output looks like and how the program behaves. To this end, we wrote a simple testprogram, MIETPMON, using our screen handler and automatic refresh mechanism to show the entropy pool in real time. A description of the screen is provided in the include file MIETPSCR. A small problem of possible interest is how to describe the four columns of data to show. Here is the description of the row where the columns start: DCL DCL DCL DCL DCL DD DD DD DD DD * * * * * CHAR(10) CHAR(10) CHAR(10) CHAR(10) CHAR(10) INIT("L050010030"); INIT("L050310010"); INIT("L050430010"); INIT("L050550010"); INIT("L050670010"); DCL DCL DCL DCL DCL DD DD DD DD DD * CHAR(30) INIT("CPU time by Thread . . S-VALUE-1ST CHAR(10) S-CURCHG-1ST CHAR(10) S-AVGCHG-1ST CHAR(10) S-MAXCHG-1ST CHAR(10) . . . INIT(" INIT(" INIT(" INIT(" "); "); "); "); "); In the program we define four arrays defined on the four entries named. Since the size of the description block for the row is 120 bytes, the arrays are defined with an “Array Element Offset” (AEO) of 120: DCL DCL DCL DCL DD DD DD DD S-VALUE (14) S-CURCHG(14) S-AVGCHG(14) S-MAXCHG(14) CHAR(10) CHAR(10) CHAR(10) CHAR(10) DEF(S-VALUE-1ST) DEF(S-CURCHG-1ST) DEF(S-AVGCHG-1ST) DEF(S-MAXCHG-1ST) POS(1) POS(1) POS(1) POS(1) AEO(120); AEO(120); AEO(120); AEO(120); The screen looks like this: Entropy Pool Monitor CPU time by Thread . . . Yield time . . . . . . . Internal call count . . Yield time . . . . . . . Total Processor time . . Yield time . . . . . . . System Storage counters Yield time . . . . . . . Storage Pool activity . Yield time . . . . . . . Disk activity . . . . . Yield time . . . . . . . MPL activity . . . . . . Yield time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Confidence level . . . . . . SHA Value . . . . . . . . . F3=Exit F5=Refresh Value 41020835 619 1409 2641 1453283625 354 1726376466 3027 1377222174 936 1119798982 3422 166500272 3733 Cur. Change 13967 471 1 1065 81106 4579 100000 2853 1884 4132 948748708 212 214 2041 2001/01/28 Avg. Change 28257 3004 1 1920 119362 1936 296463 2092 18564 2016 677840823 2251 1640922 2186 23:23:39 Max. Change 108852 401076 1 79762 678963 33892 19207000 41257 590529 29825 2133236063 24077 39156265 49763 82 > 80 is good 2000 8503E6BA3F58FDDBA4C510DF3A018C90FA57C9B3 F10=Restart F11=Keep good F12=Cancel F19=Start Auto And shows in the “value” column the current value of the newest entropy pool entry. The other columns show the magnitude of the latest change, the average change, and the maximum change. Press Enter of F5 to refresh the values (i.e. get a new entry in the pool). F10 flushes the pool and starts over. F11 puts the monitor into a mode where it will try to adjust the yield time so that the confidence value stays above (and actually just above) the 80% level. Finally F19 starts an automatic 3-second refresh cycle. The program is straightforward and will not be shown in detail here (but you can get it and its screen description from the “programs” download area). We’ll just touch upon one little detail. To generate a new screenful of values we call MIGETETP, then test the CONFIDENCE level. If it is not above 80% for five consecutive cycles, we double the MAX-YIELDS parameter. If it is not below 82% for five consecutive cycles, we halve the MAX-YIELDS parameter. We are thus striving to keep the CONFIDENCE level between 80 and 82% which gives us a high level of randomness, but also keeps control on the length of time it takes to reach that level: GENERATE-SCREEN: CPYNV START-POS, CUR-POSITION; 22-12 CALLX .MIGETETP, MIGETETP, *; TEST-FOR-LOW: CMPNV(B) SUBN(SB) MULT(SB) CONFIDENCE, 80/HI(TEST-FOR-HIGH); MEASURING-INTERVAL, 1/POS(TEST-FOR-HIGH); MAX-YIELDS, 2/NNAN(SET-MEASURING-INTERVAL); TEST-FOR-HIGH: CMPNV(B) SUBN(SB) DIV(S) CMPNV(B) CPYNV CONFIDENCE, 82/LO(SAVE-NEW-VALUES); MEASURING-INTERVAL, 1/POS(SAVE-NEW-VALUES); MAX-YIELDS, 2; MAX-YIELDS, ORIG-YIELDS/HI(=+2); MAX-YIELDS, ORIG-YIELDS;: SET-MEASURING-INTERVAL: CPYNV MEASURING-INTERVAL, 5; Conclusion This Chapter attempts to solve the following problem: Assume that someone has full access to your source code and to the source of any and all of your other code, can you still produce a number that he cannot guess? It is also assumed that he knows when your program is executing (he may be executing it himself), so you cannot simply use the clock. He could guess at every microsecond in a 1-second window around the time of the run. If you used the time, chances are that one of his guesses would be correct. It doesn’t matter that he does not know which one. TCP/IP Flaw Because of Lack of Randomness Just to show how important random numbers are we offer the following quote from the trade press: by Dennis Fisher, eWEEK, March 12, 2001 For the second time in as many months, researchers have found a serious flaw in one of the key pieces of the Internet's software backbone. Security vendor Guardent Inc. on Monday announced it has identified a potentially huge problem in the inner workings of TCP (Transmission Control Protocol), one half of the TCP/IP standard that enables Internet traffic to flow across heterogeneous networks. In January, researchers identified several holes in the BIND (Berkeley Internet Name Domain) software that runs most of the Internet's name servers. The latest problem, which is nearly identical to one found in Cisco Systems Inc.'s IOS software two weeks ago and first reported by eWEEK, involves the manner in which machines running TCP select the ISN (Initial Sequence Number). The ISN, a random value known only to the two machines at either end of a TCP session, is used to help identify legitimate packets and prevent extraneous data from muddying a transmission. ISN values are exchanged by the sending and receiving hosts and are supposed to be chosen randomly. Each successive packet then contains a sequence number that is based on the ISN plus the number of bytes transferred to the receiving host. But if the ISN is not chosen at random or if it is increased by a nonrandom increment in subsequent TCP sessions, an attacker could guess the ISN, thereby enabling him or her to hijack the session's traffic, inject false packets into the stream or even launch a denial of service attack against individual Web servers. Despite today's advisory, the INS flaw is hardly a new problem. The architects of the early Internet knew that the lack of randomness in the ISN would be a problem as far back as the mid-1980s and warned of the potential consequences. Indeed, AT&T Corp. researchers submitted a paper to the Internet Engineering Task Force in 1996 proposing a fix for the problem. While they acknowledge that it takes a very knowledgeable cracker to exploit the TCP flaw, Guardent officials say it's only a matter of time before someone develops a set of tools to do the job and posts them on the Internet. "The hard part was the reduction of this from theory to practice," said Jerry Brady, vice president of research and development at Guardent, in Waltham, Mass. "But if someone makes a tool for this available, it wouldn't take a very experienced person to [launch an attack]." Guardent officials alerted CERT and affected vendors to the problem before making it public -- still, they are likely to take some heat 22-13 for publicizing the flaw before a fix is ready. "We're trying to break new ground here," Brady said. "We were intentionally vague about the details of the problem. We want to work with the vendors to fix this." 22-14