INFORMATION LOSSLESS MACHINES
INFORMATION LOSSLESSNESS
We try to reconstruct the input sequence from the corresponding output sequence of a machine. Whenever a
machine is used as an encoding device, that is whenever a machine is provided with an input sequence and its output is
the coded message, its information losslessness guarantees that the coded message can always be deciphered.
DEFINITION A machine is information lossless (or, lossless) (IL) if the knowledge of the initial state, the
output sequence, and the final state is sufficient to determine uniquely the input sequence. A machine that is not lossless
is said to be lossy.
M1
x=0
x=1
CONDITIONS FOR LOSSINESS
A
B,0
B,0
Example 1: Consider M1. We are given the following
B
A,1
B,0
information: the initial state is A, the final state is B, and the output sequence is 0.
What is the input sequence? It may be 0 or 1. Hence M1 is lossy.
Hence, in general, if for some state Si and two distinct input symbols ip and iq,
the ip- and iq- successors of Si and the corresponding outputs are identical,
the machine is lossy.
Example 2: Consider M2. This machine does not possess
M2
x=0
x=1
the property mentioned in the above example. But M2 is lossy. One can
A
A,0
B,0
see this by noting that if the initial state is A, the final state is B, and the
B
B,0
A,1
output sequence is 00, corresponding input sequence may be 01 or 10.
01
10
In other words, we have: A ----- B and A ----- B.
00
00
Loss of information occurs whenever two states, Si and Sj, which can be reached from a common state Sc by
means of two distinct input sequences while producing identical output sequences, merge into a final state, Sf, and
produce the same output. Clearly, once the machine has reached state Sf, no future experiment will make possible the
retrieval of the input sequence which transferred M from Sc to Sf. This case which is necessary and sufficient for a
machine to be lossy, is illustrated below.
x2/z2  ----- Si





x1/z1
xn/zn





Sc

x1’/z1





Condition for information loss
Sf

xn’/zn

(xi’ denotes any input symbol that
is not identical to xi)
x2'/z2  -----  Sj
From the foregoing discussion it is evident that in order to test a machine for losslessness, it is first necessary
to determine if, for some state, two or more successors and their sorresponding outputs are identical or if any merger of
the type illustrated above exists.
INFORMATION LOSSLESSNESS OF FINITE ORDER
Suppose that a system of lossness machines is used for encoding and decoding purposes. The encoder receives
an input sequence and in turn produces an output sequence, which is transmitted to a decoder. Clearly if the encoder is
lossless, its output can be reconstructed from its output, together with the informtion regarding its initial and final states.
The major drawback in such a decoding process lies in the fact that the information regarding final state is transmitted
by the encoder only after the entire message has been transmitted. Consequently, the entire message must be stored
before the deciphering process can begin. And since the output squence may be arbitrarily long, the lossless machine
can not serve as a practical tool for encoding and decoding purposes. In view of this limitation, it becomes desireable to
look for machines for which it is not necessary to store the entire message, but where the deciphering process can start
when only the initial state and a finite length of the output sequence are available.
1
DEFINITION: A machine is said to be (information) lossless of order  (ILF-) if the knowledge of the
initial state and the first  output symbols is sufficient to determine uniquely the first input symbol.
The knowledge of the initial state and the first input symbol is sufficient to determine the next state, and thus
the second input symbol can be computed from the ( + 1)st output symbol, and so on. The integer , which is a
measure of the delay in the deciphering process, is said to be the order of losslessness if  is the least integer satisfying
the above condition, that is, if for some initial state and a sequence of  – 1 output symbols, there exist at least two
possible input sequences differing in their initial input symbols (we’ll see that the deciphering delay is  1).
Example 3: Consider M3.This machine is information lossless
M3
x=0
x=1
of first order, where the first input symbol can be determined from the
A
C,1
D,0
knowledge of the initial state and first output symbol. Hence there is
B
D,0
A,1
no delay in deciphering the inputs for this class of machines. Since
C
D,1
B,0
for every state of M3 the output associated with the 0-successor is
D
C,0
B,1
different from the output associated with the 1-successor, knowing
the initial state and the first output symbol is sufficient for the
identification of the first input symbol. For example, if M3 is initially in state A, and if the response to as yet unknown
input symbol an output of 1 is produced, we can unambiguously identify the input symbol as a 0.
TEST FOR INFORMATION LOSSLESSNESS
DEFINITION: Two states Si and Sj, are said to be (output) compatible if
1) there exists some state Sp such that both Si and Sj are its zk-successors, or,
2) there exists a compatible pair of states (Sr, St) such that (Si, Sj) are their zk-successors [in such a case we
say that the compatible pair (Si, Sj) is implied by (Sr, St)]. These concepts are schematized below.
Si and Sj are compatible
“of the first kind”
(These show themselves in the upper table):
Sp
If Sr and St are compatible,
then Si and Sj are compatible “of the second kind”
(These show themselves in the lower table):

/zk

Si
Sr
 /zk
Si

/zk

Sj
St
 /zk
Sj
Our goal is to trace the chains of compatibles to see whether any compatible pair consisting of identical states
is present, if it is, this “merger” indicates lossiness.
TESTING TABLE AND TESTING GRAPH FOR LOSSLESSNESS
Testing table is made up of two tables.
The upper table is the singleton output successor table, this table shows the compatibles of the first kind.
If any repeated entry in the upper table is present, stop: M is lossy.
Otherwise proceed constructing the lower table.
The lower table is constructed in the following manner. Any compatible pair appearing in the upper table is
made a row heading in the lower table. The successors of these pairs consist of all implied compatible pairs. Any
implied pair which has not used as a row heading is now made a row heading, its successors are found, and so on. The
process terminates when all compatible pairs have been used as row headings.
Suppose that the testing table contains a compatible pair consisting of repeated entries, e.g. (Sk, Sk); then there
exists either some compatible pair (Si, Sj) which implies (Sk, Sk) or some state Si which has identical successors for
two or more inputs. But since these cases have been shown to be necessary and sufficient for lossiness, the machine in
question must be lossy. Hence result:
A machine is lossless if and only if its testing table does not contain any compatible pair consisting of repeated
entries.
Example 4: Consider M1. Upper table is as follows:
Since (BB) appears in the the upper table
M1 is lossy. There is no need to construct the
lower table.
M1
A
B
z=0
(BB)
B
Example 5: Consider M2. The testing table is shown.
Since (BB) appears in the table, M2 is lossy.
M2
A
B
AB
z=0
(AB)
B
(AB)(BB)
z=1
A
z=1
A
-
2
Testing graph G is a directed graph such that:
1) Corresponding to every compatible pair there is a vertex in G,
2) An arc labeled zk is drawn from vertex SiSj to vertex SpSq, where p  q, iff (Sp, Sq) is a compatible pair
implied by (Si, Sj).
THEOREM: A machine M is lossless of order  = L + 2 if and only if
1) its testing table does not contain any repeated entries, and
2) its testing graph is loop-free and the length of the longest path in the graph is L.
Proof: Assume that M is lossless.
a) Let’s prove “If M is ILF then G is loop-free”. Instead, we’ll prove “If G has loops then M is not ILF”.
G has a loop. Let SiSj be some vertex in the loop. Clearly every compatible pair is accessible from some state
of M by a pair of distinct input sequences which yield identical output sequences. Thus we can find a pair of different
input sequences which take M to SiSj, while producing identical output sequences. If we now observe the output
symbols that the machine produces, while going through all the compatible pairs in the loop, we find that M is back in
SiSj, without supplying any additional information to make possible the identification of the first input symbol. And
since this loop may be repeated as many times as we wish, we may construct a pair f arbitrarily long input sequences
that start in the same state of M and differ in the first symbol but produce identical output sequences. Thus M is not ILF.
b) If G is loop-free then M is ILF. Proof is similar to the proof of the theorem “M is FM iff its G is loop-free”.
To determine the order of losslessness (), consider the longest path in G. It takes one input symbol to get from
a state of M into the first compatible, and it takes L inputs to go through the longest path. Since the compatible that has
been reached after L + 1 inputs does not imply any other compatible (a sink), one more input will yield different
outputs, depending on which of the states of the compatible the machine is in. Thus L + 2 output symbols (together
with the knowledge of the initial state) are sufficient to determine the first input symbol.
It is evident that if M is ILF-then   [n(n – 1)/2] + 1.
The case of  = 1 is detected by the absence of compatible pairs (example: M3).
The case of  = 2 is detected by the absence of arcs in the graph.
Example 6. Consider M4
M4
A
B
x=0
B,0
A,1
M4 z=0
z=1
A AB
B AB
AB No repeated entries: IL
x=1
A,0
B,1
M4 Graph:
AB
No loops: ILF, no arcs:  = 2
Example 7. Consider M5. No repeated entries in the testing table.  M5 is lossless (IL).
Proceed: G has loops.  M4 is not lossless of finite order (not ILF).
M5
A
B
C
x=0
A,1
E,0
D,0
x=1
C,1
B,1
A,0
M5
A
B
C
z=0
E
(AD)
z=1
(AC)
B
-
M5
D
E
C,0
B,1
B,0
A,0
D
E
AC
AD
BC
AE
DE
AB
(BC)
A
(AE)(DE)
(AB)(AC)
-
B
(AB)(BC)
(AB)(BC)
AD
AE
z=10
BC
1
1
AB
 1
0 0
DE
0
AC
No repeated entries.M5 is lossless.
Loops: AE-BC-AE, BC-AE-AB-BC,
AB-AB, AB-BC-DE-AB.
 M5 is not lossless of finite order.
As an example of finding an output sequence which may result from the application of two input sequences
which differ at least in the first symbol starting from the same state; consider the loop BC-AE-AB-BC in the graph,
and consider the compatible BC. BC is accessible from D by 0 and 1 inputs, yielding identical outputs, namely 0. Thus,
one can find the following input sequences, X1 and X2, which yield the same output sequence Z (underlined portions
can be repeated). No matter how long we observe the output, the ambiguity as to the first input symbol is not resolved:
1/0 B 0/0
E 0/1 B 1/1
B 0/0 E ...
0/0 C 1/0
A 0/1
C 1/0 A ...
D
A 1/1
X1:
X2:
Z:
1 0 0 1 0 0 1 0 0 1 ...
0 1 0 1 1 0 1 1 0 1 ...
0 0 1 1 0 1 1 0 1 1 ...
3
It is interesting to note that M5 is lossless, although state A can be reached by a 1 input from both states C and
E, and the output produced is 0. This situation does not imply lossiness, since the pair CE is not compatible, i.e. C and E
cannot be reached from any initial state by means of two distinct input sequences while producing identical output
sequences.
M6
A
B
C
D
Example 8: Consider M6
x=0
x=1
A,0
B,0
C,0
D,0
D,1
C,1
B,1
A,1
M6
A
B
C
D
AB
z=0
(AB)
(CD)
(AC)(AD)
(BC)(BD)
-
z=1
(CD)
(AB)
AB
0000
AC
AD
BC BD
CD
(AC)(AD)
1111
(BC)(BD)
AC
CD
AD
BC
No Loops: ILF
BD
L = 1,  = L + 2 = 3
No repeated entries: M6 is IL
A longest output which does not resolve the ambiguity about the first input symbol:
Since  =3, there must be at least one output sequence of length  – 1 = 2 which can not resolve the
ambiguity. These can be found from the graph. Consider (CD) [or (AB)]. We can reach (CD) from C with output 1.
Hence appending to this 1 just another 1 in order to reach a sink, we obtain 11.
input:
1
C
output:
0
C
1
0
D
1
C
1
D
1
Hence, if we know that the initial state is C
and the output is 11, we can not determine
whether the first input is 0 or 1.
A
1
But, if we know that the inital state is C and the output is 110, we can determine the first input: It is 0.
And, if we know that the inital state is C and the output is 111, we can determine the first input: It is 1.
RETRIEVAL OF THE INPUT SEQUENCE
M is IL. The knowledge of (the initial state + output sequence + final state) of M] The input sequence.
Example 9: Consider M5 again. M5 is IL (although it is not ILF). Hence, given the initial and the final states,
and the output sequence, we must be able to find the corresponding input sequence. Starting with the initial state and
using the output successor table, we work in the forward direction to find the possible states of M at successive time
instants. Then, starting with the final state and using the output predecessor table we work in the backward direction.
The intersection of the two sets of possible states at a given time instant will be a singleton, since M is IL. This
procedure is shown below for M5, where the initial state is given as A, final state B, and Z = 110001100101.

M5
x=0
x=1
M5
z=0
z=1
z=0
z=1
M5
A
A,1
C,1
A
(AC)
CE
A
A
B
E,0
B,1
B
E
B
D
BE
B
C
D,0
A,0
C
(AD)
D
A
C
D
C,0
B,0
D
(BC)
C
D
E
B,1
A,0
E
A
B
B
E
State Table
Singleton output successor table
Singleton output
predecessor table

Possible successors 
A
A
A
A
B
A
A
A
A
A
A
A
A
to initial 
C
C
D
C
D
B
B
D
B
B
D
C

state 
E
C
C
E
C
C
E
B
1
Output sequence:
1
0
0
0
1
1
0
0
1
0
1
Possible predecessors  A

to final state 
A
C
D
B
D
C
E
A
A
C
D
B
D
B
E
B
D
B
E
A
A
C
D
C
A
A
C
D
B
B
E
State sequence:

Input sequence:
0
1
0
0
1
0
1
0
1
1
0

B
B
0
4

Whenever a given output sequence has been generated by a lossless machine, the state transitions and input
sequence can be determined uniquely. If, however, at some point the intersection of the sets containing the possible
successors and predecessors consists of two or more states, then there exist at least two distinct input sequences which
produce identical output sequences. Therefore the machine in question is not lossless. If at some point the intersection is
empty, the corresponding output sequence could not have been produced by the given machine subject to the specific
initial and final states.
INVERSE MACHINES:
An inverse Mi is a machine which , when excited by the output sequence of a machine M, produces (as its
output) the input sequence to M, after at most a finite delay. Evidently a deterministic inverse can be constructed only if
M is lossless, and it can be constructed so that it produces M’s input sequence after just a finite delay if and only if M is
lossless of finite order.
Example 10: Consider M3. M3 is lossless of first order. For any possible initial state and output sequence, the
knowledge of the initial state of M3 and the first output symbol (y) is sufficient to determine uniquely the first input
symbol (x) to the machine. Hence there is no delay in deciphering the inputs to the machine (deciphering delay =  – 1).
The state transitions of the inverse machine M3i are therefore given by the singleton output successor table, as shown
below. The outputs associated with these state transitions are found by means of the state table of M3. If machine M3 i is
placed in cascade with M3, it will provide as its output (z) an exact replica of the input sequence to M3.
M3
NS,z
.
x=0
x=1
A
C,1
D,0
B
D,0
A,1
C
D,1
B,0
D
C,0
B,1
----------------------------
M3
i
NS,x
.
z=0
z=1
A
D,1
C,0
B
D,0
A,1
C
B,1
D,0
D
C,0
B,1
--------------------------------
x:
M3:
y:
M3i:
z:
0 0 1 1 0 0 1
B D C B A C D B
0 0 1 1 1 1 1
B D C B A C D B
0 0 1 1 0 0 1
x  M3  y  M3i z
For every lossless machine of order , the knowledge of the state at time t –  + 1 and the last  output
symbols, i.e. z(t  + 1), z(t –  + 2), ... , z(t), is sufficient to determine uniquely the input symbol x(t –  + 1).
Consequently, if we send the output sequence produced by a lossless machine M of order  into a register which
consists of  1 delay units, we can design a combinational circuit that has as inputs the contents of that register, as
well as the state of M at time t –  + 1, and in turn produces the value of x(t –  +1). The combinational circuit can
be specified by a truth table in which the value of x(t –  + 1) is specified for every possible combination of
S(t –  + 1) and z((t –  + 1), z(t –  + 2), ..., z(t). The information regarding the state of M can be supplied to the
combinational circuit by a copy of the original machine M, which is set to be at t =  – 1 in the same state that M was
in at t = 0, and receives as its inputs a delayed (by  – 1 units) version of the inputs to M. the schematic diagram of
such a deciphering system is shown below.
5
The foregoing deciphering system does not yield an economical realization, since it requires a copy of the
original machine as well as a ( – 1)-delay register. In fact, if we were to construct a composite state table for the
inverse machine (i.e. a composite table for both the register and the copy of M), we would find that in many cases it can
be considerably simplified. The question now arises is whether we can find directly from M’s description a minimal
inverse, without going through the above procedure. Indeed this can be accomplished, as shown subsequently.
MINIMAL INVERSE MACHINE
M7
We shall demonstrate the procedure in the following example.
PS
NS, z .
Example11. Consider M7. My is lossless of third order (verify).
x=0
x=1
Therefore if we know the initial state and the values of three successive
A
C,0
D,1
outputs produced by transitions from this state, we can determine the first input to
B
D,0
C,1
the machine. T
C
A,0
B,0
Let us now define a set of triples, denoted (S(t), z(t+1), z(t+2)).
D
C,1
D,1
The first member of each triple is a possible initial state of M7;
ILF - 3
the second member is one of the output symbols that can be produced by a single
transition from that state; and the third member is another output symbol that can follow this initial state and the first
output symbol. A triple is defined for each possible initial state and for all possible sequences of outputs of length 2.
For M7 we obtain the following triples:
(A,0,0)
(B,0,1)
(C,0,0)
(D,1,0)
(A,1,1)
(B,1,0)
(C,0,1)
(D,1,1).
The triple (A,0,1), for example, is not defined, because the output sequence 01 cannot be generated by M7 when
started initially in state A. All of the eight cases where the first member is A or B are shown below (construct the
remaining eight cases). In these examples the sign “]” points to an impossibility.
--------------------------------------------------------------------------x:
0
1
A
C
A
C
A
C]
A
D]
A
D
B
D
z:
0
0
0
1
1
0
1
1
--------------------------------------------------------------------------x:
0
1
C
A
B
D]
B
D
B
C
B
C]
D
B
z:
0
0
0
1
1 0
1
1
--------------------------------------------------------------------------The inverse of M7, denoted M7i, has eight states, corresponding to the eight triples given above. We shall
often refer to a state of the inverse as an inverse state. For every state of M7i, the next inverse state is a triple whose
members are determined in the following manner:
1. The first member is the state to which M7 goes (underlined in the above examples) when it is initially in the
state that is the frst member of the present inverse state, and when it is supplied with the first input symbol.
2. The second member is the third member of the corresponding present inverse state (double underlined).
3. The third member is the present output of M7.
The state table of M7i is given below. Renaming the states and applying minimization , we obtain a 6-state machine.
M7
i
PS
E
F
G
H
I
J
K
L
(A,0,0)
(A,1,1)
(B,0,1)
(B,1,0)
(C,0,0)
(C,0,1)
(D,1,0)
(D,1,1)
NS, x
z=0 .
(C,0,0), 0
(D,1,0), 1
(D,1,0), 0
(C,0,0), 1
(A,0,0), 0
(B,1,0), 1
(C,0,0), 0
(D,1,0), 1
.
z=1 .
(C,0,1), 0
(D,1,1), 1
(D,1,1), 0
(C,0,1), 1
(B,0,1), 1
(A,1,1), 0
(C,0,1), 0
(D,1,1), 1
PS
E
F
G
H
I
J
K
L
z =0
I, 0
K, 1
K, 0
I, 1
E, 0
H, 1
I, 0
K, 1
NS, x
.
z=1
J, 0
L, 1
L, 0
J, 1
G, 1
F, 0
J, 0
L, 1
Renaming the states
E  K and
FL
6
An example of input- , output-, and state-sequences:
i
M7 (minimal)
PS
NS, x
.
z =0
z=1
E
I, 0
J, 0
F
E, 1
F, 1
G
E, 0
F, 0
H
I, 1
J, 1
I
E, 0
G, 1
J
H, 1
F, 0
x1  M7  z  M7i  x2

x1:
0
1
0
0
0
1
1
0
1
M7:
A
C
B
D
C
A
D
D
C
B
z:
0
0
0
1
0
1
1
1
0
M7i:
H
I
E
I
G
E
J
F
F
E
x2:
1
0
0
1
0
0
0
1
1

x2 is a replica of x1 with a delay of 2 (delay =  – 1).
Ignore the first two outputs (underlined).
The initial state of the inverse machine:
Singleton input predecessor
table for M7i
z=0
z=1
.
FGI
E
FGJ
F
I
G
J
H
EH
I
EH
J .
Suppose that M7 starts in state A. Then the first
two outputs of M7 will be either 00 or 11. If the output of M7 is 00,
then the inverse machine must be in the inverse state (A,0,0), which
we have renamed E (in the above example this case is pictured,
state A of M7 and state E of M7i are shown bold). Hence, M7 must
start from a starting state from which M7i goes to E after receiving 00.
This can easily be found from the singleton input predecessor table of M7 i,
which is shown on the right.
M7:
z:
M7i:
A
A
0
E
H
0
[F
[G
I
1
E
F
G
J
I
E
H
1
F
F
G
J
The inverse state (A,0,0) and (A,1,1) cases are shown above. We see that if M7 starts at A and if its first two
outputs will be 00, M7i should be in state E or H initially. But if M7’s first two outputs will be 11, M7 should start in F,
G, J, I, E, or H. Therefore, if M7s tarts in state A, M7i should start in either E or H, since we have
{E, H}  {F, G, J, I, E, H} = {E, H}.
[Note that in Kohavi the computation of the required initial state of the inverse machine is in error. Find the error].
By analogous computations we may put up the following table (verify):
If the starting state of M7 is
the starting state of M7i should be
A
E or H
B
E or H
C
F, G, I, or J
D
F, G, I, or J
-------------------------------------------------
7