Information Theory

I , satisfying

b. h(p) is a monotonically decreasing and continuous function o f p, 0 < p I I . Show that the only function satisfying the given conditions is h(p) = - C logbp where C > 0, b > 1.

CHAPTER TWO

Noiseless Coding 2.1. Introduction Our first application of the notion of uncertainty introduced in Chapter 1 will be to the problem of efficient coding of messages to be sent over a "noiseless" channel, that is, a channel allowing perfect transmission from input to output. Thus we do not consider the problem of error correction; our only concern is to maximize the number of messages that can be sent over the channel in a given time. T o be specific, assume that the messages to be transmitted are generated by a random variable X whose values are x,, . . . ,x,. A noiseless channel may be thought of intuitively as a device that accepts an input from a specified set of "code characters" a,, . . . , a, and reproduces the input symbol a t the output with no possibility of error. (The formal definition of an information channel will be deferred until Chapter 3 ; it will not be needed here.) If the symbols xi are to be communicated properly, each xi must be represented by a sequence of symbols chosen from the set {a,, . . . , a,}. Thus, we assign a sequence of code characters to each x,; such a sequence is called a "code word." Since the problem of error correction does not arise, efficient communication would involve transmitting a given message in the shortest possible time. If the rate at which the symbols a j can be sent through the channel is fixed, the requirement of efficiency suggests that we make the code words as short as possible. In calculating the long-run efficiency of communication, the average length of a code word is of interest; it is this quantity which we choose to minimize. T o summarize, the ingredients of the noiseless coding problem are:

I . A random variable X, taking on the values x,, . . . ,xM with probabilities~,,. . . ,p, respectively. X is to be observed independently over and over again, thus generating a sequence whose components belong to the set {x,, . . . ,2,); such a sequence is called a message. 2. A set {a,, . . . ,a,) called the set of code characters or the code alphabet; each symbol xi is to be assigned a finite sequence of code characters called the code word associated with xi (for example, x, might correspond to a,a,, and x, to a3a,a3a,). The collection of all code words is called a code. The code words are assumed to be distinct.

28

INFORMATION THEORY

3. The objective of noiseless coding is to minimize the average code-word length. If the code word associated with xi is of length n,, i = I , 2, . . . , M, we will try to find codes that minimize pi n,.

z!zl

2.2.

The problem of unique decipherability

I t becomes clear very quickly that some restriction must be placed on the assignment of code words. For example, consider the following binary code : 21

x, x, x4

0 010 01

10 The binary sequence 010 could correspond t o any one of the three messages x,, x,x,, or xlx4. Thus the sequence 010 cannot be decoded accurately. We would like to rule out ambiguities of this type; hence the following definition. A code is uniquely decipherable if every finite sequence of code characters corresponds to at most one message. One way to insure unique decipherability is to require that no code word be a "prefix" of another code word. If A , and A, are finite (nonempty) sequences of code characters then the juxtaposition of A , and A,, written AIA,, is the sequence formed by writing A , followed by A,. We say that the sequence A is a p r e f x of the sequence B if B may be written as AC for some sequence C. A code having the property that no code word is a prefix of another code word is said to be instantaneous. The code below is an example of an instantaneous code. XI

x, x, x,

0 100 101 11

Notice that the sequence 11111 does not correspond to any message; such a sequence will never appear and thus can be disregarded. Before turning to the problem of characterizing uniquely decipherable codes, we note that every instantaneous code is uniquely decipherable, but not conversely. For given a finite sequence of code characters of an instantaneous code, proceed from the left until a code word W is formed. (If there is no such word, the unique decipherability condition is vacuously satisfied.) Since W is not the prefix of another code word, W must correspond to the first symbol of the message. Now continue until another code word is formed. The process may be repeated until the end of the message.

29

NOISELESS CODING

For example, in the instantaneous code (0, 100,101, 11) above, the sequence 1011 10100101 is decoded as x,x,x,x,x,. Now consider the code 21 0 x, 01 This code is not instantaneous since 0 is a prefix of 01. The code is uniquely decipherable, however, since any sequence of code characters may be decoded by noting the position of the ones in the sequence. For example, the sequence 0010000101001 is decoded as x,x, x,x,x, x,x,x,x,. The word "instantaneous" refers to the fact that a sequence of code characters may be decoded step by step. If, proceeding from the left, Wis the first word formed, we known immediately that W is the first word of the message. In a uniquely decipherable code which is not instantaneous, we may have to wait a long time before we know the identity of the first word. For example, if in the code

we received the sequence 0 0 . . -001 we would have to wait until the end tn+l--+ of the sequence to find out that the corresponding message starts with x,. We now present a testing procedure that can always be used to determine whether or not a code is uniquely decipherable. To see a how the procedure works, consider the code of Fig. 2.2.1, "1 C x2 which is not instantaneous but could conceivably be x3 ad uniquely decipherable. We construct a sequence of sets 2, abb So, S,, S,, . . . , as follows. Let So be the original set of bad x, code words. To form S,,we look at all pairs of code x6 deb words in So. If a code word W , is a prefix of another bbcde code word W,, that is, W , = W,A, we place the suffix A F i g 2.2.1. A in S,. In the above code, the word a is a prefix of the , ,,.,, word abb, so that bb is one of the members of S,. In general, to form S,, n > 1, we compare So and S,-,. If a code word W E SOis a prefix of a sequence A = W B r Sn-,, the suffix B is placed in S,, and if a sequence A' r Sn-, is a prefix of a code word W' = A'B' r So, we place the suffix B' r S,. The sets S,, n = 0, 1,. . . , for the code of Fig. 2.2.1 are shown in Fig. 2.2.2. We shall prove

,

Theorem 2.2.1. A code is uniquely deciperable if and only if none of the sets S , , S,, S,, . . . contains a code word, that is, a member of So.

30

INFORMATION THEORY

In Fig. 2.2.1, the code word ad belongs to the set S,; hence, according to Theorem 2.2.1, the code is not uniquely decipherable. In fact, the sequence abbcdebad is ambiguous, having the two possible interpretations a , bbcde, bad and abb, c, deb, ad. A systematic method of constructing ambiguous sequences will be given as part of the proof of Theorem 2.2.1.

s1

so a c ad abb

d bb

s 2

s 3

s 4

eb cde

de

b

S L i s, ad d bcde

s, eb

S, empty (n > 7)

bad deb bbcde

Fig. 2.2.2. Test for unique decipherability.

Proof. First suppose that the code is not uniquely decipherable, so that there is a sequence of code characters which is ambiguous, that is, corresponds to more than one possible message. Pick an ambiguous sequence G with the smallest possible number of symbols. Then G may be written in at least two distinct ways: where the Wi and W,' are code words (assume n 2 2, m 2 2; otherwise the conclusion is immediate). Now define the index of the word W,(respectively W,') in G as the number of letters in W, W, . Wi-, (respectively W,' . - W;,), i = 2, . . . , n, j = 2, . . . ,m. The minimality of the number of letters of G implies that the indices of W,, . . . , Wn, W,', . . . , W,' are distinct. If Wl has fewer letters than W,', define the index of W, to be -I and that of Wll to be 0; reverse this procedure if W,' has fewer letters than Wl. (Note that W,' cannot equal Wl for if so, W,' - - . W,' = W, . . - W,, contradicting the minimality of the number of letters of G.) Let Ul, U,, . . . , Un+, be the words of G, arranged in order of increasing index. If j < i and index Ui> index U,, but index U,+l < index Uj the number of letters in U,,we say that Ui is embedded in U,. We claim that for each i = 3, . . . , n m, either U, is embedded in some Uj, j < i, or the subsequence A, of G which begins with the first letter of U, and ends with the letter immediately preceding the first letter of U,,,,is in one of the sets S,, n 2 1. (The sequence A,+, is defined to be U,, itself.) The claim is true for i = 3 by inspection. The various possibilities are indicated in Fig. 2.2.3a and b. If the claim has been verified for i j r, consider U,,,. If U,+, is not embedded in some Uj,j < r 1, we have:

-

+

+

+

NOISELESS CODING

F-Ul+l+-U3 +u4 (a) U3 embedded in U2

( d ) Ur embedded in U,, i
Fig. 2.2.3. Proof of Theorem 2.2.1.

CASEI . U, is not embedded in any U,, i < r (see Fig. 2.2.3~). By mduction hypothesis, A, r some S,, n 2 1. But then A,+, r S,+,. CASE2. U, is embedded in some U,, i < r (see Fig. 2.2.3d). Then U , = A,U,+, . . . U,A,+,. By induction hypothesis, A, r some S,, n 2 1. By definition of the sets S,, U,,, . . . U,A,+, r S,,,, U,,, . U,A,+, r SntP, . . . UrAr+l E Sn+r-t, Ar+1 E Sn+r-t+lNow U, ,+ cannot be embedded in any Ui;hence A,+, = U,,, r some S,, n 2 1 , and the first half of the theorem is proved. 3

Conversely, suppose that one of the sets S,, k 2 1 , contains a code word. Let n be the smallest integer 2 1 such that S , contains a code word W. If we retrace the steps by which W arrived in S,, we obtain a sequence Ao, Wo, A19 Wl, - . . , An, Wn such that A,, W o , W,, . . . , W,, are code words, A,, . . . , A , are sequences of code characters such that A, E Si, i = 1 , . . . ,n, A , = W,, Wo = AoAl, and for each i = 1,2, . . . , n - 1 either A, = WiAi+, or Wi = A,A,+,. For example, for the code of Fig. 2.2.2, we obtain A,=a A,=bbtSl A2=cdetS2 A3=detS3 A 4 = b t S 4 AS=adcS6 W,, = abb Wl = bbcde W , = c W 3 = deb W 4 = bad W , = ad

32

INFORMATION THEORY

We now give a systematic way of constructing an ambiguous sequence. We construct two sequences, one starting with A, W, and the other with W,. The sequences are formed in accordance with the following rules. Having placed W, at the end of one of the sequences:

CASE1. A, = W,Ai+,. Add W,,, at the end of the sequence containing

wi.

CASE2. Wi = A,A,+,. Add Wi+, at the end of the sequence not containing W,. Continue until W , is reached. We shall illustrate this procedure for the code of Fig. 2.2.2.

W, = abb W, = A,A, so form WOW,= abbc. (Notice that the sequence A, W, is longer than the sequence W,, hence W, is added to the shorter sequence.) A, = W,A3 so form W, W, W , = abbcdeb. A, W , = abbcde,

(After the addition of W,, the sequence beginning with W, is still the shorter, so W3is added to that sequence.) W 3= A3A4so form A, W, W , = abbcdebad.

(After the addition of W,, the sequence beginning with W, exceeds the sequence beginning with A, in length, and thus W4 is added to the latter sequence.) W4 = A,A, so form W ,W , W3W , = abbcdebad. The sequence abbcdebad = A, Wl W4 = W, W, W3W , is ambiguous. We now show that the procedure outlined above always yields an ambiguous sequence. We may establish by induction that after the word Wi is assigned (i = 1,. . , ,n - I), one of the sequences is a prefix of the other. By inspection, this is true after W, is assigned. The inductive step is accomplished by a tedious case-by-case analysis. For example, suppose that W, is assigned to the first sequence and that after W, is written down, sequence 2 is a prefix of sequence 1. In addition, suppose that W,-, was assigned to sequence 2 (see Fig. 2.2.4). Then necessarily W,-, = A,-,A, and W , = A,A,+,, therefore Wi+, is assigned to sequence 2. If W,,, = A,+,A,+, then A,,, is a prefix of W,+l. If A,,, = W,,,Ai,, then W,,, is a prefix of A,+,. In either case, one of the sequences is still a prefix of the other. The other cases may be handled in a similar fashion. Now after Wn-, is assigned we have either W,,-, = An_,Wn or An_, = W n - ,Wn. Another case-by-case analysis shows that after W , is assigned, the two sequences are identical. Since W, # A,, we have produced an ambiguous sequence.

33

NOISELESS CODING

We remark that since the sequences in the sets Si, i ;l.0, cannot be longer than the longest code word, only finitely many of the Si can be distinct. Thus eventually the S , must exhibit a periodicity; that is, there must be integers Nand k such that Si = Si+, for i 2 N(it may happen that S, is empty for i 2 N; in fact a code is instantaneous if and only if S, is

Fig. 2.2.4. Proof of Theorem 2.2.1.

empty for i 2 1). Thus the testing procedure to determine unique decipherability must terminate in a finite number of steps, and an upper bound on the number of steps required may be readily calculated for a given code.

2.3. Necessary and suficient conditionsfor the existence of instantaneous codes

As we have seen in the previous section, an instantaneous code is quite easy to decode compared to the general uniquely decipherable code. Moreover, we shall prove in Section 2.6 that for the purpose of solving the noiseless coding problem, we may without loss of generality restrict our attention to instantaneous codes. Thus it would be desirable to examine the properties of such codes. We start by posing the following problem. Given a set of symbols x,, x,, . , x, a code alphabet a,, a,, . . , aD, and a set of positive integers n,, n,, . . , n,, is it possible to construct an instantaneous code such that n, is the length of the code word corresponding to s,? For example, if M = 3, D = 2, n, = 1 , n, = 2, n, = 3, a possible code is (0, 10, 110). If M = 3, D = 2, n, = n, = n, = 1, no uniquely decipherable code, instantaneous or otherwise, will meet the specifications. The complete solution to the problem is provided by the following theorem.

..

.

.

Theorem 2.3.1. An instantaneous code with word lengths n,, n,, . . . ,n, exists if and only if 5 1 (D = size of the code alphabet). ProoJ To prove the theorem we make use of the idea of a tree of order D and size k , which'is simply a system of points and lines such that

zz,

D-"1

INFORMATION THEORY

(a) Order = 2, slze = 3

Fig. 2.3.1. Trees.

each sequence s of length
Fig. 2.3.2. Tree representation of a binary code.

35

NOISELESS CODING

length n, excludes Dn~-"k"terminal points," that is, points corresponding to sequences of length n,. Thus the total number of terminal points excluded by the given code is 25, Dn~-"i. Since a tree of order D and ~ points, the total number of terminal points size n , has D n terminal excluded cannot exceed D n ~ Thus, . Dn~-"a 5 Dnx, or 25, D-"i < - 1. Conversely, suppose we are given positive integers n,, n,, . . . ,n~ satisfying D-"i 5 1. Again assume n, 5 n, 5 - . 5 n,. To construct the required instantaneous code, select any point on the tree of order D and size n M corresponding to a sequence of length n,. This excludes Dn~-"1 terminal points. Since D-"i 5 1 , or equivalently 22, Dn~-"1L; Dnx, we have Dn~-"1< D n 4 and a t least one terminal point remains. Hence a point corresponding to a sequence of length n, remains, and we may choose this point to correspond to the code word of length n,. The total number of points excluded is now Dnx-"1 DnM-"2, and again it follows from the hypothesis that this number is less than D n ~and , hence we may choose a point correspondjng to a sequence of length n,. The process may be continued until all code words are assigned. We remark that the condition 25, D-"i < - 1 is equivalent to

I$,

I:,

,:z

+

where w , is the number of code words of length j, and r is the maximum code-word length. This follows since the sum Z& DVni may be written as D-1 + D-1 + . . . + D-1 + 0 - 2 + . . . D-,

+

4-I

tlmes-

t

4 -

-2'"

2.4. Extension of the condition ZE, D-"i < I to uniquely decipherable codes We have shown that the condition 25, D-"i < 1 is necessary and sufficient for the existence of a base D instantaneous code with word In this section, we prove a stronger result, lengths n,, n,, . . . , n,. namely, that the same condition is necessary and sufficient for the existence of a uniquely decipherable code. The sufficiency part is immediate, since every instantaneous code is uniquely decipherable. Thus it remains to prove necessity. Theorem 2.4.1. If a uniquely decipherable code has word lengths n,, n,, . . . ,n,, then D-"< 5 1.

z&

36

INFORMATION THEORY

Proof. For convenience we use Now

z;=, o j D P i instead

of

z,:

D--"<.

Each term in the expansion of (2.4.1) is of the form w I ID-'lwl, D-'2 . * . w,,D-'n where 1 j i, j r for each k, and hence n j i, . . . i , j nr. Thus

+

($I

Tkzn

+

nr

W,D-'

=2N,D-~

(2.4.2)

where

We claim that Nk is the total number of messages using n (not necessarily distinct) symbols x, whose coded representation is of length k. To see this, we note that the number of sequences of code characters that can be formed using code words of length i , , . . . , in, in that order, where i, + . . . + i , = k (that is, the number of sequences that start with a code word of length i , , follow with a code word of length i,, . . . , and end with a code word of length i n ) is w,, . . . win. By unique decipherability, every sequence of code characters corresponds to at most one message. Thus Nk cannot exceed the total number of coded sequences of length k, that is, Nk 5 Dk. By (2.4.2),

Taking nth roots, we have

Since (2.4.4) holds for any n, we may let n + co to obtain

2.5.

2 ~ =O,J ~ Dj- I.~

The noiseless coding theorem

Let us return to the noiseless coding problem. A random variable X takes on values x,, . . . , x, with probabilities p,, . . . ,pM respectively. Code words W,, . . . , W,,,of length n,, . . . ,n , respectively, are assigned t o the symbols x,, . . . ,x ~ The . code words are combinations of characters taken from a code alphabet a,, . . . , a,. We wish to construct a uniquely decipherable code which minimizes the average code-word length A = x E l p , n i . We shall approach the problem in three steps. First we establish a n absolute lower bound on A ; then we find out how close we

37

NOISELESS CODING

can come to the lower bound; then we attempt to synthesize the "best" code. The lower bound is set by the following theorem.

zEl

pini is the Theorem 2.5.1. (Noiseless Coding Theorem). If A = average code-word length of a uniquely decipherable code for the random variable X,then fi 2 H(X)/log D with equality if and only if pi = D-"1 for i = 1, 2, . . . , M. Note that H(X)/log D is the uncertainty of X computed using logs to the base D, that is, M

M

z

Ho= - E log, p 2-- i pi log, pi. I o D ~

log2 D i=1 H(X)/log D is equivalent to

i=1

Proof. The condition A

2

Since pini log D = p i log Dni = -pi log D-"i, the above condition may be written as

-

M

M

pi log D-"' 2 - 2 pi log Pi.

i=l

i=l

This looks suspiciously like Lemma 1.4.1, except that the terms D-"i do not necessarily add to unity. This difficulty may be circumvented by defining q, = D - " S / ~ $D-"J. Then the q,'s add to unity, and Lemma 1.4.1 yields M M D-"' (2.5.1) - Z: PI log pi 5 - Z: pi log 7 4-1 i=l ED-"' 5=1

with equality if and only if pi = D-"z/zjM=, D-"1 for all i. Hence by (2.5. I),

H(X)

5 ii log D

+ log 2 D-"j

(2.5.2)

D-nt with equality

if and only if

p, =

for

all i.

2 0-"' j=l

But by unique decipherability (Theorem 2.4.1)

zEl D-"j 5 1; hence

38

INFORMATION THEORY

Now if p, = D-"c for all i, then M

M

i=l

i=l

H(X) = - 2 p, log pi = z p i n i log D = ii log D . It therefore remains to prove that if H(X) = A log D, then pi = D-"I for all i. By (2.5.2), H(X) = A log D implies that log

(2

D-"j)

2 0;

i=i

by unique decipherability log

(z (Il ) DPni)

5 0,

i=I

and thus log

zEl

D-"j

=0

or D-"j = 1. It now follows from (2.5.2) that pi = D-"i for all i. This completes the proof. We shall call a code that achieves the lower bound set by the noiseless coding theorem absolutely optimal. An example of an absolutely optimal binary code is the following.

X

Probabilities Code Words 112 0 x2 114 10 23 118 110 H ( X ) = A = P. 24 118 111 In general we cannot hope to construct an absolutely optimal code for a given set of probabilities pl,p2, . . . , p M , since if we choose ni to satisfy p, = D-"i, then ni = (-logp,)/(log D ) may not be an integer. However, we can do the next best thing and select the integer ni such that 21

-logpi 5 n, < -logpi+l (i=1,2, . M ) (2.5.3) log D log D We claim that an instantaneous code can be constructed with word lengths n,, n,, . . . , nM. To prove this, we must show that D-"1 1. From the left-hand inequality of (2.5.3), it follows that logp, 2 -n, log D, or pi 2 D-"i. Thus zElD-"1 z z l p i = I. TO estimate the average code-word length, we multiply (2.5.3) by pi and sum over i, to obtain

zzl

Thus we have proved the following theorem.

<

39

NOISELESS CODING

Theorem 2.5.2. Given a random variable X with uncertainty H ( X ) , there exists a base D instantaneous code for X whose average code-word length fi satisfies

H(X)
log D

log D

Consequently the average code-word length may be brought within one digit of the lower bound set by the noiseless coding theorem. I n fact, we can always approach the lower bound as closely as desired if we are allowed to use "block coding." Suppose that instead of assigning a code word to each symbol xi, we take a series of s independent observations of X and assign a code word to the resulting group of s symbols. In other words, we construct a code for the random vector Y = ( X I , X,, . . . , X,), where the X, are independent and each X, has the same distribution as X. If X (and hence each Xi) assumes M values, then Y assumes M S values. Block coding will in general decrease the average code-word length per value of X as illustrated by the following example. X

p

Code Word

Y

=

( X I ,X,)

p

+

+t

Code Word

ri = ,Be ,Se (2) (3) = code characters12 values of X =

i3 code characterslvalue of X

By Theorem 2.5.2, we may construct an instantaneous code for Y whose average code-word length ti, satisfies

H(Y) log D

H' )' + 1 code characters/value of Y . sfis
Since X I , . . . , X, are independent and identically distributed, H ( Y ) = H(X,, . . . , X,) = H ( X l ) . . . H ( X , ) = s H ( X ) . Thus

+

+

s H o < f i s < -s H ( X ) + 1, log D log D

H(X)<%< H(X) log D

s

log D

+

1. s

40

INFORMATION THEORY

But A,/s is the average number of code characters per value of X; it follows from (2.5.4) that A,/s may be made arbitrarily close to H(X)/log D by choosing s large enough. Thus H(X)/log D may be interpreted as the minimum average number of base D digits required to encode one observation of X. If we consider the case D = 2, we make the observation that any instantaneous binary code corresonds to a sequence of "yes or no" questions, as illustrated in Fig. 2.5.1. In the example, an answer "yes" corresponds to a zero in a code word; an answer "no" to a one. Thus, X

Sequence of "Yes or No" Quest~ons

Code Words

XI

00

xz

01

XI

X2

No x l or xz?

X4

100

xs

101

No -x,

x3

Fig. 2.5.1. A binary instantaneous code and its corresponding sequence of "yes or no" questions.

as we indicated in Section 1.3, the uncertainty H(X) may be interpreted as the minimum average number of "yes or no" questions required to determine the result of one observation of X. 2.6.

Construction of optimal codes

The only part of the noiseless coding problem that remains is the construction of the code which minimizes the average code-word length for a given set of probabilities p,,p,, . . . , p M . We first show that in the search for optimal codes, we may restrict our attention to instantaneous codes. More precisely, we prove: Lemma 2.6.1. Suppose a code C is optimal within the class of instantaneous codes for the given probabilities p,, p,, . . . ,p M ; in other words, no other instantaneous code for p,, . . . ,pM has a smaller average codeword length than C. Then C is optimal within the entire class of uniquely decipherable codes. Proof. Suppose a uniquely decipherable code C' has a smaller average code-word length than C. Let n,', n,', . . . , n', be the code-word lengths of C'. By Theorem 2.4.1, zEM=, D-"z' < 1. Rut by Theorem 2.3.1 there Hence exists an instantaneous code C" with word lengths n,', . . . ,n,'. the average word length of C" is the same as the average word length of C', contradicting the fact that C is the best instantaneous code.

NOISELESS CODING

41

Before turning to the synthesis of optimal instantaneous codes, we imagine that we have constructed an optimal code and determine what properties such a code must have. In other words, we shall derive necessary conditions on an optimal instantaneous code. To simplify the presentation we consider only the binary case. For an extension to nonbinary code alphabets see Problem 2.5. The results are summarized as follows. Lemma 2.6.2. Given a binary instantaneous code C with word lengths nl, n,, . . . ,nM associated with a set of symbols with probabilities-pl, p,, . . . ,p,. For convenience, assume that the symbols are arranged-in order of decreasing probability (p, 2 p, 2 . . 2 p,) and that a group of symbols with the same probability is arranged in order of increasing code-word length. (If pi = pi+, = . . =pi+,., then ni I ni+, _< . . _< n,,,.) Then if C is optimal within the class of instantaneous codes, C must have the following properties: a. Higher probability symbols have shorter code words, that is, p, > p, implies n, I n,. b. The two least probable symbols have code words of equal length, that is, nMPl = nnf. c. Among the code words of length n~ there must be at least two words that agree in all digits except the last. For example, the following code cannot be optimal since code words 4 and 5 do not agree in the first three places.

Proof. To prove (a), we note that ifp, > p, and nj > n,, it is possible to construct a better code C' by interchanging code words j and k. The difference between the average code-word length of C' and of C is

hence C' is better than C. To prove (b) we note first that nM-, n, for if pM-l > p M then n,-, j n, by (a); if pM-, = p,, then ,n, 5 n, by our assumption about arrangement of code words associated with equally probable symbols. Now, if nM > nM-,, we may drop the last digit of the Mth code word to obtain a code that is still instantaneous and better than the original code.

42

INFORMATION THEORY

Finally, (c) is proved by observing that if no two code words of maximal length agree in all places but the last, then we may drop the last digit of all such code words to obtain a better code. We now describe a method, due to Huffman, for the construction of optimal (instantaneous) codes. If we have an array of symbols x,, . . . , x , with probabilities p,, . . . , p , (again assume p, 2 p, 2 . . 2 p,), then we combine the last two symbols xM-, and x, into an equivalent symbol x,,,, with probability p,, + p M - , . Now suppose that somehow we could construct an optimal code C , for the new set of symbols (see Table 2.6.1). We now construct a code C , for the original set of symbols x,, . . . , x,. The code words associated with x,, x,, . . . , 2,-, are exactly the same as the corresponding code words of C,. The code words associated with x,-, and x , are formed by adding a zero and a one, respectively, to the code word W,,,,, associated with the symbol x , , , in C,. We now claim that C , is an optimal code for the set of probabilities p,, Pz,.. ~ P M . T o establish the claim, suppose that C , is not optimal. Let C,' be an optimal instantaneous code for x,,x,, . . . , x,. Then C,' will have code words W,', W,', . . . , W,' with word lengths n,', n,', . . . , n,w' By Lemma 2.6.2c, at respectively. By Lemma 2.6.2b, n&, = n,'. least two words of length nM' agree in all digits but the last. We may assume without loss of generality that W2;-, and W,' are two of these words. (If necessary we may interchange words of the same length without affecting the average word length.) At this point we again combine the symbols x,, and x , and construct a code C,' by taking as the code the word W,,' (or Wh-,) with the last digit removed (see word for x,,,, Table 2.6.2). We now establish that C,' has a smaller average word length than C,, contradicting the optimality of C,. The average word length of C,' is

,

+ . . . + p , - , n k , + (pnl-1 + - 1) = plnlf + - . . + p,-,nL-, +~~-~nh-.,

(A,)' = p1nlf

P M ) ( ~ ~ - I

+PM~M'-

(PM-1

+ PM).

Since C,' has a smaller average word length than C 1 , plnll

+

'

..

+ ~ ~ ~ - ~ n +; Mp -~e - , n L+ pnfn;M < plnl + . . .

Using the fact that n,-,

+ PM-2nM-2 + P M - ~ ~ M+- pawn,. ~

= n , (by construction of C , ) we obtain

43

NOISELESS CODING

Table 2.6.1. Construction of the Huffman code X

Probabil- Code words ities of C ,

Word lengths

X

Probabilities

Code words Word of C , lengths

Table 2.6.2. Proof of optimality of the Huffman procedure ProbabilX ities "1

P1

"a

Pz

Code words of C,'

Word lengths

X

Probabilities

Code words Word of C,' lengths

w1;

"1;

21

Pl

w1;

w2

"a

22

Pa

w2

"

1;

nz

U' = WM'with last digit removed.

Since n,-, - 1 = n ~ , , ~ the , , expression on the right is the average word length of C,, and thus the proof is complete. The above result tells us exactly how to construct optimal codes. An example is carried through in Fig. 2.6.1. The idea is simply to combine the two least probable symbols until only two symbols remain. An optimal binary code for two symbols must consist of the two code words 0 and 1 ; we can then work backwards, constructing optimal codes for each array until we arrive at the original set of symbols. 2.7.

Notes and remarks

The material of this chapter has become fairly standard, and is treated in Feinstein (1958), Abramson (1963), and Fano (1961). Theorem 2.2.1 is due to Sardinas and Patterson (1950, 1953), whose proof exploited the fact that the collection of finite sequences of code characters is a semigroup under juxtaposition. The somewhat cumbersome direct argument in the present text is the author's. The phrase "instantaneous code" was

44

INFORMATION THEORY

Optimal Codes x1.466° "2.3

x2.3

I

xl

4 . 5

O0 O1

x1

O0

"4,66

O1

2,

x2

lo 11

6

X3

X3

2,

O0 10 11 010 011

xl 2,

X3 X

x, 2,

O0 10 1I 011 0100 0101

Fig. 2.6.1. Huffman code.

coined by Abramson (1963). Theorem 2.3.1 is due t o Kraft (1949); Theorem 2.4.1 was proved by McMillan (1956); the proof in the text is due to Karush (1961). Theorem 2.5.1 is due to Shannon (1948); the proof follows Feinstein (1958). The construction of Section 2.6 is due to Huffman

(1 952). PROBLEMS 2.1 Determine whether or not each of the following codes is uniquely decipherable. If a code is not uniquely decipherable, construct an ambiguous sequence. abc abed e dba bace ceac ceab eabd

2.2 a. For the binary code below, let N ( k ) be the number of messages that can be formed using exactly k code characters. For example, N(l) = 1 (that is, X I ) , N(2) = ( " l x l , '29 ~ 3 ) N , ( 3 ) = ('lxlxl, x 1 x 2 , x 1 x 3 , xpl,~ 3 ~ 1 ) . Find a general expression for N(k) (k = 1, 2, . . .).

NOISELESS CODING

b. Repeat part (a) for the code below.

2.3 Construct a Huffman code for the symbols below. Compare the average code-word length with the uncertainty H(X).

Symbols

2.4

Probabilities

Symbols

Probabilities

Show that the average word length of a Huffman binary code satisfies

2.5 Explain how to construct a base D instantaneous code that minimizes the average code-word length and apply the results to construct an optimal ternary code for the set of symbols below.

Symbols

Probabilities

Symbols

Probabilities

CHAPTER THREE

The Discrete Memoryless Channel 3.1. Models for communication channels In this chapter we shall begin the analysis of communication channels whose inputs are subject to random disturbances in transmission. In the usual communication situation, an object of some kind (for example, a letter of the alphabet, a pulse or other waveform) is selected from a specified class of inputs. The channel is a device that acts on the input to produce an output belonging to another specified class. The random nature of the channel may in many cases be described by giving a probability distribution over the set of possible outputs. The distribution will in general depend on the particular input chosen for transmission and in addition may depend on the internal structure of the channel at the time the input is applied. We shall try to arrive at a definition of an information channel that reflects these considerations. We are going to specialize to the so-called "discrete case," that is, the situation in which the information to be transmitted consists of a sequence of symbols, each symbol belonging to a finite alphabet. More general models will be discussed in Chapter 8. If we apply a sequence a, . . . , a, at the input of a channel, then at the output, perhaps after an appropriate delay, we will receive a sequence B1, . . . ,/?,. It is reasonable to describe the action of the channel by.giving a probability distribution over the output sequences B,, . . . , ,6, for each input sequence a, . . . , a,; the family of distributions should also reflect the fact that the "internal state" of the channel at the time the input is applied will affect the transmission of information. Physically, we expect that many channels have "memory"; that is, the distribution of the output symbol B, may depend on previous inputs and outputs. We do not, however, expect "anticipatory" behavior; in our model the distribution of p, should not depend on future inputs or outputs. Thus in giving the distribution of /?,,. . . , #?,,we need not consider inputs beyond a, . . . , a., We are led to the following definition. Definition. Given finite sets r and I", to be called respectively the input alphabet and output alphabet, and an arbitrary set S called the set of states,

THE DISCRETE MEMORYLESS CHANNEL

a discrete channel is a system of probability functions a,, Dl,

. . . , an E

1'

. , B, r1 E

srS n = l,2, ...,

that is, a system of functions satisfying I . pn(B1,

PI, . . .

. . ,Bn I al, . . . , a,;

S ) 2 0 for all n, a,,

. . . , a,,

Bn, 8.

I

2. Ipl ,...,pnpn(B1,. . . , Bn a,, a,, . . . , a,, S .

. . . , a,; S ) = 1 for all n,

I

Physically we interpret p,(B1, . . . , B, a,, . . . , a,; s) as the probability that the sequence ,8,, . . . , @, will appear at the output if the input sequence a,, . . . , a , is applied and the initial state of the channel, that is, the state just prior to the appearance of a,, is s. The state of the channel may change as the components of the input sequence are applied; our model assumes that knowledge of the initial state and the input sequence determines the distribution of the output sequence. In this chapter we consider only a very special case of the discrete channel. However, the techniques of analysis which will be developed are basic to the analysis of more general channels. Specifically, we are going to consider the discrete channel without memory; such a channel is characterized by the requirements that successive symbols be acted on independently and that the functions p, do not depend on the state s. Formally we have the following: Definition. A discrete channel is memoryless if

I

1. the functions p,(B1,. . . , B, a,, . . . , an; s) do not depend on s, hence may be written p,(B,, . . . , B, a,, . . . , a,), and

I

I

I

2. ~ n ( B 1 ., . . Bn ~ 1 . ,. . an) = pl(B1 I ~ 1p1V2 ) ~ 2 )' ' pl(Sn 1 an) for all a,, . . . , a , E r, P I , . . . , Pn E I",n = 1, 2, . . . , 1

7

The second condition may be replaced by the conjunction of two other conditions as follows: Lemma 3.1.1.

define

Given probability functions satisfying condition (I),

48

INFORMATION THEORY

(This quantity is interpreted as the probability that the first n - k output symbols will be PI,.. . ,Bn-, when the input sequence a,, . . . , an is applied.) Also define

this expression is interpreted as the conditional probability that the nth output symbol will be Bn, given that the input sequence a,, . . . , an is applied and the first n - 1 output symbols are PI,. . . , /3n-1. (Note that the subscript n on the functions above corresponds to the length of the input sequence.) The functions satisfy condition (2) if and only if for all n = 1,2, . . . , both of the following conditions are satisfied.

Condition (a) puts into evidence the memoryless feature of the channel, and condition (b) indicates the nonanticipatory behavior. Proof. Suppose that condition (2) is satisfied. Then

THE DISCRETE MEMORYLESS CHANNEL

To prove (b), note that the above argument shows that ~n(Blr. . .

Bn-1

I a19 . . .

n-1

9

an) =

IT pl(Bk I

k=l

= ~n-l(B1, . . . , Bn-1

1 ~ 1. ,. . ,an-1).

An induction argument now establishes (b). Conversely, if (a) and (b) are satisfied then pn(B1, .

.

Bn

I MI, . .

an)

= pn(B1r . . . r Bn-1 I ~ 1. , ~ n ) ~ n ( BI nair . . . , an; 81, - - Bn-1). By (b), the first term on the right is pn-,(PI,. . . , Bn-, I a,, . . . , an-,); by (a) the second term ispl(Bn I an). Proceeding inductively, we establish (2). Note that we could have specified k = 1 in the statement of condition (b) and the lemma would still hold. Thus a discrete memoryless channel is characterized by a matrix whose elements are a,@ = pl(B I a), a I?, @ E I?'. (aa8 is the element in row a and column B.) The matrix [p,(B I a)] is called the channelmatrix; henceforth we shall drop the subscript and writep(p I a) forpl(B I a). In this chapter, the word "channel" will mean "discrete memoryless channel" unless otherwise specified. 3.2. The information processed by a channel; - 9

channel capacity; classification of channels

Consider a discrete memoryless channel with input alphabet* x,, . . . , x,, output alphabet y,, . . . ,y, and channel matrix [aij], a i j = p(yj xi), i = I , . . . , M , j = 1 , . . . , L. If a n input symbol is chosen at random, that is, if X is a random variable taking on the values x,, . . . , xM with probabilities p(xl), . . . ,p(xM) respectively, then the channel output also becomes a random variable. The joint distribution of the input X and the output Y is given by P{X = xi, Y = y,} = p(xi)p(yj xi), i = 1, 2, . . . , M , j = 1,2, . . . , L, and the distribution of Y is given by

I

I

M

z

P { Y = Y,) = i=l P(x,)P(Y, I xi),

J =

1 , 2 , . . . L. 3

...

* In Chapter 2 we distinguished between the symbols x,, x,, ,x,, which were the building blocks for a message, and the code alphabet a,,a,, . . ,a,. Coding is necessary since the inputs accepted by the channel may have no relation to the symbols x,, x,,. . . x,. In this chapter we assume for convenience that x,, x,, . . . ,x, is the code alphabet. We do this because we would like to reserve the symbol H ( X ) for the uncertainty at the input to a channel. If it becomes necessary to distinguish uncoded information from coded information, then we shall denote the original (uncoded) symbols by m,, m,, etc. ; thus the m, will be the components of a message. Each message will be assigned a code word, that is, a sequence of symbols from the alphabet xl, x*,. . . ,xw

.

50

INFORMATION THEORY

Thus the specification of an input distribution induces in a natural way an output distribution and a joint distribution on input and output. We may therefore calculate the input uncertainty H ( X ) , the output uncertainty H ( Y ) , and the joint uncertainty of input and output H ( X , Y ) as well as the conditional uncertainties H ( Y X)and H ( X Y). It is natural to define the information processed by the channel as

I

I

I(X ( Y ) = H ( X ) - H(X

( Y).

(3.2.1)

By the discussion in Section 1.5, we have I ( x ] Y ) = H(Y) - H(Y X ) = I ( Y X ) = H ( X ) H ( Y ) - H ( X , Y ) . It is important to notice that the information processed by a channel depends on the input distribution p(x). We may vary the input distribution until the information reaches a maximum; the maximum information is called the channel capacity. Specifically, we define the channel capacity as C = max 1(X/ Y ) . (3.2.2)

I

I

+

dz)

(There is a true maximum rather than just a least upper bound; see Problem 3.12.) The significance of the channel capacity is not a t all clear from what we have done so far. However, we shall prove later in this chapter that (loosely speaking) it is possible to transmit information through a channel at any rate less than the channel capacity with an arbitrarily small probability of error; completely reliable transmission is not possible if the information processed is greater than the channel capacity. The calculation of the capacity of a channel is a difficult problem in general; the solution in certain special cases will be discussed in the next section. It is convenient at this point to introduce certain classes of channels which are easy to analyze. a. A channel is lossless if H(X ) Y ) = 0 for all input distributions. In other words, a lossless channel is characterized by the fact that the input is determined by the output and hence that no transmission errors can occur. Equivalently, the values Fig. 3.2.1. Lossless channel. of Y may be partitioned into disjoint sets

THE DISCRETE MEMORYLESS CHANNEL

Deuce of Spades Ace of Hearts

:

\Hearts

Deuce of Hearts

Deuce of Diamonds

Deuce of Clubs / Fig. 3.2.2. Deterministic channel.

I

B,, B,, . . . , BM such that P { Y E Bi X = z,)= 1 (i = 1,. . . , M). The structure of a lossless channel is indicated in Fig. 3.2.1. b. A channel is deterministic ifp(y, xi) = 1 o r 0 for all i, j; that is, if Y is determined by X, or equivalently H( Y X ) = 0 for all input distributions. An example of a deterministic channel is one whose input X i s the identity of a playing card picked from a n ordinary 52-card pack, and whose output Y is the suit of the card (see Fig. 3.2.2). If the card is picked a t random so that all values of X ( a n d hence X1 Y1 of Y ) are equally likely, then the information is H ( Y ) - H( Y X ) = '2 Y2 H(Y) = log 4. c. A channel is noiseless if it is lossless and deterministic (see Fig. 3.2.3). Note that the YM X" definition agrees with the intuitive notion of a noiseless channel used in Chapter 2. Fig. 3.2.3. Noiseless channel.

I

I

I

0

52

INFORMATION THEORY

I

d. A channel is useless (or zero-capacity) if I(X Y) = 0 for all input distributions. Equivalently, a useless channel may be characterized by the condition that H ( X Y) = H(X) for allp(x), or alternately X and Yare independent for all p(x). Since independence of X and Y means that p(y, xi) = p(y,) for all i,j, a channel is useless if and only if its channel matrix has identical rows. A lossless channel and a useless channel represent extremes of possible channel behavior. The output symbol of a x lossless channel uniquely specifies the input symbol, o so that perfect transmission of information is possible. A useless channel completely scrambles all input messages. Since p(xi y,) = p(x,) for all i, j, the con1 ditional distribution of X after Y is received is the I-a same as the original distribution of X. Roughly speakFig' 3.2.4. ing, knowledge of the output tells us nothing about channel. the input; for the purpose of determining the input we may as well ignore the output completely. An example of a useless channel is shown in Fig. 3.2.4. e. A channel is symmetric if each row of the channel matrix [p(y, xi)] contains the same set of numbers pl1,pZ', . . . ,pL1and each column of For example, [p(yi xi)] contains the same set of numbers qll, q,', . . . , 9,'. the matrices Y1 Yz Y3 Y4 Y1 Yz Y3

I

I

I

I

I

represent symmetric channels. The rows of the channel matrix are identical except for permutations, and similarly for the columns. I t is an immediate consequence of the definition of a symmetric channel that for such a channel, H ( Y X) is independent of the input distribution p(x) and depends only on the channel probabilitiesp(y, I xi). T o show this, we note that if X = xi, the probabilities associated with the output symbols y,, . . . ,yL are (not necessarily in this order)pll,p,', . . . ,pL1. Hence

I

L

H(Y I X = xi) = - 2 pir log p,', i=1 'Therefore

L

=-

2 pjl log p,',

i=1

i = 1, 2 ,

. . . ,M

THE DISCRETE MEMORYLESS CHANNEL

Fig. 3.2.5. Binary symmetric channel. for any input distribution p(x). The most celebrated example of a symmetric channel is the "binary symmetric channel," shown in Fig. 3.2.5.

3.3. Calculation of channel capacity Before attacking the general problem of calculating the capacity of an arbitrary discrete memoryless channel, we consider a special case in which a closed-form expression for the capacity can be found, namely, the case of a symmetric channel. Consider a symmetric channel with input alphabet x,, . . . ,x, output alphabet y,, . . . ,y,, and channel matrix with row probabilities p,', . . . , pL' and column probabilities q,', . . . ,9,'. Since H(Y X) does not depend on the input distribution, the problem of maximizing the information I(X Y) = H(Y) - H( Y X)reduces to the problem of maximizing the output uncertainty H(Y). Now we know from Theorem 1.4.2 that H(Y) 5 log L with equality if and only if all values of Y are equally likely. Thus if we can find an input distribution under which all values of Y have the same probability, that input distribution would maximize I ( X ( Y). We claim that the uniform input distribution will do the trick; in other words, if all input symbols of a symmetric channel are equally likely, then all output symbols are equally likely as well. To prove this, we let p(xi) = 1/M for i = 1, 2, . . . , M and write

I

I

I

I

However, the term z,"=,p(yj xi) is the sum of the entries in the jth column of the channel matrix. Since the channel is symmetric, we have XElP(g, xi) = I E l qk), independent of j. Thus p(y,) does not depend on j, or equivalently all values of Y have the same probability. The maximum H(Y) = log L is therefore attainable, and we conclude, using (3.2.3), that the capacity of a symmetric channel is

I

L

Csym = log L

+ 2 pif log pi'. I=1

(3.3.1)

54

INFORMATION THEORY

The capacity of the binary symmetric channel of Fig. 3.2.5 is CRsC= log 2 B log B (1 - B) log (1 - B), or (3.3.2) Cssc = 1 - H(B, 1 - p) (see Fig. 3.3.1). Before trying to compute the capacity in the general case, we need some further properties of the information I ( X Y ) . Suppose that X is the input to a discrete memoryless channel and Y the corresponding

+

+

I

Fig. 3.3.1. Capacity of the binary symmetric channel.

output. Let Ik(X I Y ) (k = 1 , 2 , . . . ,r) be the information processed when the input distribution is given by P{X = x ) = pk(x)(x = x,, . . . , xM). In other words, we consider the operation of the channel under various input distributions pl(x), . . . ,p,(x). We have the following theorem. Theorem 3.3.1. The information processed by a channel is a convex function of the input probabilities. Specifically, if a,, . . . , a , are nonnegative numbers whose sum is unity, and we define an input distribution pO(x) = a g , ( x ) [a convex linear combination of the p,(x)], then the information Io(X Y ) corresponding to po(x) satisfies

z:=l

I

I

In particular if we take a lossless channel then Ik(X Y) = H k ( X ) = - I e p r ( x ) logp,(x), and we have I

Hence the uncertainty is also a convex function.

THE DISCRETE MEMORYLESS CHANNEL

55

Proof. The subscript k will indicate that the quantity in question is t o be evaluated when the input distribution is pk(x). Let A1 = Io(X I Y) k=l

Then

I

aklk(X Y).

A1 = Ho(Y) - Ho(Y I X) But

2 ak[Hk(Y) - Hk(Y I X)1.

k= 1

I

Sincepo(y, I xi) is just the channel probabilityp(y, xi) we have

=

2

k=l

I X).

Consequently

>

Since a, 2 0, Lemma 1.4.1 yields A1 0, completing the argument. Note that the hypothesis EL=, a, = 1 is needed to insure that po(x) is a legitimate probability function, that is, 2&po(xi) = 1. Theorem 3.3.1 yields valuable information about the problem of maximizing I(X Y ) . If P{X = xi) = pi, i = 1,2, . . . , M, the resulting information I(X I Y) may be regarded as a function I(pl, . . . ,pM) of the M variables pl, . . . ,pM. In the problem of evaluating channel capacity, we are trying to find the particularp,, . . . ,pMthat maximize I. Theorem 3.3.1 assures us that any solution found by the methods of the differential calculus yields an absolute maximum rather than a relative maximum or minimum or saddle point. In particular we have:

I

Lemma 3.3.2. Let g(p) be a real-valued function of the vector variable p = (pl, . . . ,pM), defined for pi 2 0, i = 1, . . . ,M. Suppose that g is convex on the set

56

INFORMATION THEORY

+

+

that is, g((1 - a)p up') 2 (1 - a)g(p) ag(pl) for all p, p' c S and all a E [0, I]. [Note that this implies by induction that

if a, ; lO,x~;=,a,= l , p k = (p,,, . . . , p k M ) c S ,k = l , 2 , . . . , r.1 Suppose also that g is continuously differentiable forp, > 0, i = 1, 2, . . . , M. Then ifpi* > 0, i = 1 , 2 , . . . , M, and ag(p1, . . . P,) = 0 ap, for all i at the point p* = (pl*, . . . ,pM*) c S, then an absolute maximum of g(p) on S, that is, an absolute maximum of the function g restricted to S, is attained at p = p*. Proof. Suppose g(pl) > g(p*) for some p' c S. The line segment joining p* to p' is the set of points (1 - a)p* up', 0 5 a 5 1. Note that the line segment lies entirely within S. If we examine the values of g along the segment we find by convexity that 7

+

g[(l

- a)p* + ap'l

as long as 0

< a 5 1.

- g(p*)

2

(1 - a)g(p*)

The hypothesis that

+ ag(pr) - g(p*)

-- 0 for all i at p = p*

a~i

implies that the gradient of g is zero at p*, and hence that the directional derivative of g is zero for all possible directions. In particular, the difference quotient (3.3.3) must approach zero as a -+ 0, a contradiction. We conclude that for all p in S, g(p) 5 g(p*). We can now exhibit a closed-form expression for channel capacity which is valid in certain cases.

Theorem 3.3.3. Suppose that the channel matrix Il of a discrete memoryless channel is square and nonsingular. Let q,, be the element in row i and column j of n-l, i, j = 1,. . . , M. Suppose that for each k = l , 2 , ... M,

The channel capacity is given by

THE DISCRETE MEMORYLESS CHANNEL

and a distribution that achieves capacity is given by

Proof. The function that we are trying to maximize is of the form

The expression (3.3.6) may be regarded as being defined for all nonnegative real values of the variables pi = p(x,). Of course, the expression is equal to the information processed by the channel only for 0 L; pi 5 1, i = 1, . . . , M, 2&pi = 1. Thus I(X Y) is a continuously differentiable function of the variablesp(x,), . . . ,p(xM) when allp(x,) are greater than 0. Note that the quantities p(y5 xi) are fixed for a given channel. Let us try to maximize (3.3.6) subject to the condition z z l p ( x i ) = 1, and let us hope that the solution to the maximization problem does not involve negative values for the p(x,). (The techniques of the differential calculus do not cover constraints of the form p(xi) 2 0.) Using the method of Lagrange multipliers, we try to maximize Z(X Y) l 2 z l p ( x i ) by setting the partial derivatives with respect to each of the p(x,) equal to zero. Since H(Y X) = 2Elp(xi)H(Y X = xi), we have

I

I

I +

I

I

For the remainder of the proof we write H(Y I xk) for H(Y I X = x,). We are also going to assume that the logarithms are to the base e for convenience in differentiation; we will switch back to base 2 at the end. Now 5=1

Thus

i=l

58

INFORMATION THEORY

Thus the equations

become

k = 1 , 2, . . . , M ,

where a=l

and

In matrix form, (3.3.7) becomes

[Note that 1 - A = z & p ( y j have

1 xk)(l - A).]

Since

n

is nonsingular we

[- ~ ~ , H ( Yx iI) ] M

p(y,) exp ( 1 - A) = erp

0.3.8)

i=l

Summing (3.3.8) overjand then taking logarithms, we have [observing that x & p ( x i ) = 1 implies z E l p ( y j ) = 1 and vice versa]

zfzl

I

To find the numbers p(x,) we note that p(y,) = p(x,)p(y, xi), or in matrix form, [p(yl) ' ' ' P(YM)] = [ p ( ~ l' )' ' p ( ~ M ) l n . Postmultiplying both sides of this equation by I7-I we have

THE DISCRETE MEMORYLESS CHANNEL

59

From (3.3.8) and (3.3.10) we obtain M

p(xk) = exp ( 1 - l ) z q , , e x p i=1

[- z q j i H ( Y Ixi)]. M

(3.3.11)

i=l

By hypothesis, the right side of (3.3.11) is strictly positive for each k. Thus (3.3.1 1) and (3.3.9) give us numbers p(xl), . . . ,p(xM), jl with all p(xk) > 0, zEl p(xk) = 1, such that

I

+

By Theorem 3.3.1, I(X Y) A z E , p ( x k ) is the sum of a convex and a linear (hence convex) function and is therefore convex on the set of nonnegative numbers whose sum is unity. By Lemma 3.3.2, we have found, for the given A, an absolute maximum of the function I(X I Y) 2 Z,M=,p(xk) over the domain p(x,) 2 0, p(x,) = 1. We claim that the solution yields an absolute maximum for the information processed. For if the channel capacity C i s greater than the number I(X I Y) found above, then let p*(xl), . . . ,p*(xM) be an input distribution which achieves capacity. Let p'(xl), . . . ,pl(xM) be the input distribution found in the Eagrange multiplier calculation, and let I ( X I Y) = I' be the resulting information. Then

zzl

+

M

I !

+n
i=l

But since pf(xi), i = 1, . . . , M, yields an absolute maximum,

a contradiction. It remains to find the channel capacity explicitly. If we multiply (3.3.7) byp(xk) and sum over k, we obtain Hence 1 - 1is the channel capacity C . From (3.3.9) we obtain

The theorem now follows from (3.3.12) and (3.3.11). (The transition from base e back to base 2 is easily accomplished if we observe that uncertainty in binary units is logz e times the corresponding uncertainty in natural units.)

60

INFORMATION THEORY

If the channel matrix rI is square and nonsingular, but the assumption (3.3.4) is not satisfied, then the Lagrange maximization will yield negative or zero values for at least one of thep(x,), indicating that the maximizing distribution is located on the boundary. It then becomes necessary to set some of the p(x,) equal to zero and try to maximize I(X I Y) as a function of the remaining variables. However, if we set one or more of the p(x,) equal to zero, we are essentially reducing the number of channel inputs without, in general, reducing the number of possible outputs. The reduced channel matrix is no longer square so that the formulas of Theorem 3.3.3 do not apply. The general problem of computing channel capacity is a problem in numerical analysis, best treated by convex programming methods. (See the notes and remarks at the end of the chapter.)

3.4. Decoding schemes; the ideal observer We now consider for the first time the problem of reliable transmission of messages through a noisy communication channel. In order to achieve reliability, we must be able to determine the input message with a high degree of accuracy after seeing the received sequence of symbols. We are looking for the "best" way of finding the correct input, in other words, for the best "decoding scheme." To formulate the problem more precisely, suppose that a channel has an input alphabet x,, x,, . . . ,x,, an output alphabet y,, y,, . . . ,y, and a channel matrix [p(y, 1 x,)]. For simplicity we consider first the special case in which a single symbol, chosen at random according to a known input distribution p(x), is transmitted through the channel. A decoder or decision scheme is an assignment to every output symbol y, of an input symbol x,* from the alphabet x,, . . . , .,x The interpretation is of course that if y, is received, it will be decoded as x,*. The decoder may be thought of as a deterministic channel with input alphabet y,, . . . ,y, and output alphabet x,, . . . ,.,x If Z is the output of the decoder, then we may express Z as a function of Y, say Z = g(Y). (See Fig. 3.4.1.) Equivalently, we may think of the decoder as partitioning the values of Y into disjoint subsets B,,. . ,B M , such that every y in B, is decoded as x,.

.

21

x

Y1

channel y,

x1

decoder

x,

x~ YL x~ Fig. 3.4.1. Representation of a decoder.

--

THE DISCRETE MEMORYLESS CHANNEL 112

'I

114

Overall probability of error = 1/4

1I 4

Fig. 3.4.2. Example of channel and decoder.

As an example, consider the channel and decoder combination of Fig. 3.4.2. The symbol y, is decoded as x, and the symbols y, and y, as x,. The probability of error in this case is just the probability that x, is chosen, since x, and x, are always decoded perfectly. We now propose the following problem: For a given input distribution p(x), construct the decision scheme that minimizes the overall probability of error. Such a decision scheme is called the ideal observer. To find the required decoder, assume that each output symbol y, is associated with an input symbol x,* ( j = 1,2, . . . , L) (see Fig. 3.4.3). Let p(e) be the overall probability of error and p(el) the overall probability of correct transmission. Given that y, is received, the probability of correct transmission is the probability that the actual input is x,*. Thus we may write the probability of correct transmission as

The probability p(yj) is the same for any decision scheme since p(y,) depends only on the input distribution and the channel matrix. For each symbol y, we are free to choose the corresponding xi*. It follows from (3.4.1) that if we choose xi* as that value of X which maximizesp(x yi), we have maximized p(el y,) for each j, and therefore we have maximized the probability of correct transmission. To summarize, the ideal-observer

I

1

-

decoder xi*

Fig. 3.4.3. Construction of the ideal observer.

62

INFORMATION THEORY

decision scheme associates with each output symbol y, the input symbol x that maximizesp(x yj). (If more than one input symbol yields a maximum, any one of the maximizing inputs may be chosen; the probability of error will not be affected.) In a similar fashion we may consider a situation in which a sequence x = (a,, . . . , a,) chosen in accordance with a distribution p(x) = p(a,, . . . , a,) is transmitted through the channel. The probability that an output sequence (PI, . . . , /I,) is produced is then given by

I

p(B1,

. . . , Bn) =

2. .

el,.

=

,an

~ ( a 1 ., . . , an)p(B1, . . . , 8, I air . . . , a,)

~ ( ~ .1. 7. , an)~(BlI ~&P(B,I a,) . . . p(Bn Ian). Z: . . . .an

al,

(Note that a,, . . . , a n need not be chosen independently.) We may define a decoder or decision scheme as a function that assigns to each output sequence (PI,. . . , Pn) an input sequence (a,, . . . , a,"). The ideal observer, that is, the decision scheme minimizing the overall probability of error for the given input distribution, is, just as before, the decoder that selects, for each (PI, . . . , /I,), the input sequence

An important special case occurs when all inputs are equally likely. If, say, x,, . . . ,x, have the same probability, then

Hence for a fixed y, maximizing p(x, I y) is equivalent to maximizing p(y 1 2,). Thus when all inputs are equally likely, the ideal observer selects the input xi for which p(y I xi) is a maximum. The resulting decoder is sometimes referred to as a maximum-likelihood decision scheme. The ideal observer suffers from several disadvantages. It is defined only for a particular input distribution; if the input probabilities change, the decision scheme will in general also change. In addition, it may happen that certain inputs are never received correctly. For example, in the channel of Fig. 3.4.2 an error is always made whenever x, is sent. It would be more desirable to have a decision scheme with a uniform error bound. A decoding scheme with uniform error bound E is a decoder for which the probability of error given that xi is sent is less than E for all i. For such a decision scheme the overall error probability is less than E for any input

THE DISCRETE MEMORYLESS CHANNEL

63

distribution. On the other hand, no general method is known for actually constructing decision schemes with uniform error bounds, while the ideal observer is quite easily obtained as soon as the input distribution is known. Also, as we shall see in Chapter 4, the ideal observer for the binary symmetric channel has a significant interpretation which gives considerable insight into the theory of error correcting codes. Finally, we shall see in the proof of the fundamental theorem (Section 3.5) that if we are studying the behavior of channels under long sequences of inputs, then in a sense (to be made precise in the next section) we may without loss of generality restrict attention to the ideal observer, calculated with all inputs equally likely. 3.5.

The fundamental theorem

We now try to formulate precisely the idea that it is possible to transmit information through a noisy channel at any rate less than channel capacity with an arbitrarily small probability of error. Suppose we are given a source of information that produces a sequence of binary digits at the fixed rate of R digits per second. Then in n seconds the source will produce nR digits; hence the total number of messages the source can produce in a period of n seconds is 2nR. In any n-second interval, the source will produce one of the 2nR possible sequences. (In the informal discussion we ignore the fact that nR may not be an integer; in general the number of possible messages will be a t most [2nR]= the largest integer _<2"". Note also that for the purpose of this discussion we need not consider nonbinary sources; if a source produces S base D digits per second, then we can find an equivalent binary rate R by setting DnS = 2"": R = S l o g D.) Suppose that the information produced by the source is to be transmitted through a discrete memoryless channel. Assume for the sake of definiteness that it is possible to send symbols through the channel a t any rate 5 1 symbol per second. The way we are going to achieve reliable transmission is the following. Suppose that we wait for a period of n seconds and observe the message produced by the source. We then assign a "code word," that is, a sequence of symbols from the input alphabet of the channel, to each message. Thus instead of coding individual digits produced by the source we are coding an entire block of digits. We transmit the code word and, after receiving the output sequence, make a hopefully accurate decision as to the identity of the input sequence. Now if we are to maintain the source transmission rate R, we can take no longer than n seconds to process the information produced by the source in an n-second interval. Since the channel rate is at most 1 symbol per second, the code word assigned to the source message must not contain more than

64

INFORMATION THEORY

n symbols. Thus in order to maintain the source rate R we need a vocabulary of 2nR sequences of length < n , whose components belong to the channel input alphabet. The basic idea of the fundamental theorem is that given E > 0, if we choose n sufficiently large we can find 2nR code words of length n along with a corresponding decision scheme with a probability of error uniformly < E , that is, <E no matter which code word is transmitted. Thus we can maintain the transmission rate R and at the same time reduce the probability of error toward zero. A price must be paid in order to achieve the results of the fundamental theorem. First of all, since we must wait n seconds before encoding the source message, there is a possibly long delay between production and transmission of a message. (Further delay may be introduced in the encoding and decoding operations.) In addition, the coding and decoding schemes needed to meet the specifications of the fundamental theorem are in general exceedingly complex and difficult to construct and to implement. An example may help to clarify the previous ideas. Suppose that R = and n = 5. Then in 5 seconds the source will produce one of 2nR = 4 possible messages, say m,, m,, m,, and m,. If we are using a binary symmetric channel, then we should assign a binary sequence of length 5;s to each message. For example, we might make the*following assignment. m000 m, m, 01101 m, 11010 m, 10111

+

Now we could assign distinct code words to the messages m,, m,, m,, m, by using binary sequences of length 2 : m, 00 m, 01 m, 10 m, 11 In using sequences of length 5, however, we gain error correcting ability. For example, in the second assignment above an error in a single digit In any code word will cause the word to be confused with another word. In the first assignment, there is a decision scheme to correct all single errors in transmission. If for a given received sequence v we select the code word w that differs from v in the fewest number of places, then noting that for all i # j, the words w, and w, disagree in at least 3 digits, we see that if the received sequence v differs from the transmitted sequence w in at most 1 digit, the decoder will make a correct decision.

THE DISCRETE MEMORYLESS CHANNEL

65

We shall have much more to say about the properties of codes in Chapter 4. We hope that the intuitive idea behind the fundamental theorem is clear; if further discussion is still needed, perhaps a rereading of Section 1.1 will help. We turn now to the statement and proof of the fundamental theorem. First, a few definitions. Given a discrete memoryless channel, an input n-sequence (respectively output n-sequence) for the channel is a sequence of n symbols, each belonging to the channel input (respectively output) alphabet. A code (s, n) is a set of s input n-sequences x'", . . . , X(8) , called code words, together with a corresponding decision scheme, that is, a function which assigns to each output n-sequence one of the sequences x"), . . . , x'"), or equivalently a partition of the set of all output n-sequences into disjoint sets B,,. . . , B,, called the decoding sets, with the interpretation that an output in B, will be decoded as x(". Note that we do not require that the x({)be distinct. However, if x ( ~=) x'j) for i # j, and the transmission of x ' ~ )results in an output sequence belonging to Bi,we shall regard this as a decoding error, since according to our previous discussion the indices i and j will correspond to distinct messages produced by the source. We shall use the symbol X = (XI,. . . , X,) to denote a randomly chosen input n-sequence, and Y = ( Y , , . . . , Y,) for the corresponding output n-sequence. Once the sequences in a code and the decoding sets are specified, various error probabilities can be computed. For example, the probability of error given that x'') is transmitted is

I

p(e\ x'~')= P{Y B, X = x'"} =

2 p(y (x'"'). r#Br

Suppose that a code word is chosen at random in accordance with a distribution 9 which assigns to the code word x ' ~ the ) probability p(xci)), i = 1,2, . . . , s. Suppose that the code word chosen is transmitted through the channel and a decision is made at the output in accordance with the given decision scheme. The resulting probability of error is given by 3

pD(e) = 2 p(xti')p(e ( x'"). i=l

The average probability of error of a code is defined by S

p(e) = s-'

2 p(e Ix'~'). i= 1

pG

Thus is the error probability p9(e) associated with the uniform distribution, that is, the distribution assigning the same probability to all code words.

66

INFORMATION THEORY

The maximum probability of error of a code is defined by p,(e) = max p(e 1 x ( ~ ) ) Thus . if p,(e) 5 E then, no matter which code word is ljigs transmitted, the probability of error is S E ; in other words, we have a decoding scheme with uniform error bound E. Finally, a code (s, n, 1) is a code consisting of s input n-sequences, whose maximum probability of error is 51.t We may now formally state the basic result. Theorem 3.5.1. (Fundamental Theorem of Information Theory). Given a discrete memoryless channel with capacity C > 0 and a positive number R < C, there exists a sequence of codes A,, A,, . . . such that A, is a code ([2nR],n, 1,) and 1, -+ 0 as n + co. Thus it is possible, by choosing n sufficiently large, to reduce the maximum probability of error to a figure as low as desired while at the same time maintaining the transmission rate R. We shall try to approach the proof gradually. Shannon's original argument was informal and went something like this. Suppose that an input n-sequence X = (XI, . . . , X,) is chosen at random, with the components Xi being chosen independently. Suppose further that p(u), u E I? = channel input alphabet, is an input distribution that achieves channel capacity and that each Xi is chosen in accordance with this distribution, that is, P{Xi = u) = p(u), u E r. Let X be transmitted through the channel and let Y be the corresponding received sequence. Write H(X) for the common uncertainty H(Xi), i = 1,2, . . . , n, and similarly define H(Y), H(X Y), H(Y X), and H(X, Y). Now from Section 1.3 we know that for large n there are approximately 2nH(") "typical" input n-sequences, each with probability roughly 2-nH(X); similarly there are 2nH(Y)typical output n-sequences and 2nH(X,Y) "typical pairs" of input and output n-sequences. A typical pair may be generated by first selecting a typical output n-sequence y, and then selecting a typical input n-sequence x such that (x, y) is a typical pair. Since the number of typical output sequences is 2nH(y)and the number of typical pairs is 2nH(XpY), we may conclude that for each typical output sequence y, there are 2n[H(X,Y)-H(Y)I = 2nH(XIY) input sequences x such that (x, y) is a typical pair. In other words, if we receive a typical sequence y and are faced with the problem of determining the corresponding input sequence, there are roughly 2nH(XIY)(not 2nH(X))possibilities, each with approximately the same probability. (See Fig. 3.5.1.) Now suppose that a code is formed by selecting 2nR (R < C ) typical input n-sequences successively, with the sequences chosen at random,

I

I

This terminology was introduced by Wolfowitz (1961).

THE DISCRETE MEMORYLESS CHANNEL

67

independently, and with the 2nH(X) possible input sequences being equally likely at each choice. If a code word x ( ~is) transmitted through the channel and the typical sequence y is received, an error in decoding is possible only if at least one code word x'j), j # i, belongs to the set S of inputs associated with y.t But znR P{at least one x'j', j # i, belongs to S ) 5 2 P{x'~'s. S )

The above discussion suggests the possibility of constructing codes that maintain any transmission rate R < C with an arbitrarily small probability

Fig. 3.5.1. Intuitive justification of the fundamental theorem.

of error. If a code is selected by choosing 2nRinput n-sequences independently and at random, with the components of each code word chosen independently according to the distribution that achieves capacity (equivalently, if each code word is selected by choosing a typical input n-sequence at random, with all typical sequences having the same probability), then "on the average," accurate decoding can be accomplished. ~urthermore,if (x, y) is aiypical pair, then

than C . It follows that for "most"

hence if a typical input sequence x is transmitted, it is very likely that the output sequence y will belong to the set

Thus if x is a code word and we impose the requirement that the decoding set associated with x be a subset of A,, the effect on the probability of ?We assume the ideal observer decision scheme with all code words equally likely.

68

INFORMATION THEORY

error will be small. This observation gives some information about the structure of the decoding sets for the desired codes, and will be useful in following the reasoning to be used in the proof of the fundamental theorem. Although the above discussion is not in the form of a mathematical proof, Shannon's remarkable intuitive argument is the ancestor of all known proofs of the fundamental theorem. We shall give two proofs of the fundamental theorem. The first proof will be based on an (unfortunately impractical) algorithm for constructing the desired codes. We shall require that the decoding set associated with a given code word x be a subset of the set A, described above; this device will allow us to estimate the relationship between the number of words in a code and the probability of error. The second proof will be a precise translation of the idea that a code selected randomly will under certain conditions exhibit, on the average, desirable properties. We first proceed to the "algorithmic" proof. The key part of the argument is the following result. Lemma 3.5.2. (Fundamental Lemma). Given an arbitrary discrete memoryless channel, let n be a fixed positive integer, andp(x) an arbitrary probability function defined on the set of all input n-sequences. If x is an input n-sequence and y an output n-sequence, let

For any real number a, let (x, y): log [Note A is a set of (2n)-tuples, with the first n components in the input alphabet and the last n components in the output alphabet; an intuitive justification for the appearance of such a set has been given above.] Then for each positive integer s there exists a code (s, n, A) such that

A 5 s - 2-"

+ P{(X, Y)

A)

(3.5.1)

where P{(X, Y) A ) is computed under the distribution p(x). The essential content of the fundamental lemma is that given any probability function p(x), then by means of an algorithm to be described below, an algorithm whose steps depend in a significant way on the particular function p(x), we can construct a code whose maximum probability of error is bounded by the expression (3.5.1). In order to use this result to prove the fundamental theorem, we will take s = [2nR],R < C,

THE DISCRETE MEMORYLESS CHANNEL

69

and then choose the number a and the distributionp(x) so that the bound on the maximum probability of error will approach 0 as n approaches infinity. Proof. We shall construct the code by means of an iterative procedure as diagrammed schematically in Fig. 3.5.2. For any input n-sequence x,

Fig. 3.5.2. Proof of the fundamental lemma.

let A, be the "section of A at x," that is, A, = { y : ( x ,y ) e A}. Let E = . 2--" P((X, Y) # A). We are attempting to construct a code (s, n, A) with A E. If E 2 1, there is nothing to prove since in that case any code (s, n, A) has A j E . Thus assume 0 < E < 1. First pick if possible any input n-sequence x") such that

s

+ <

take the decoding set B, to be A,(,,. (Thus if x ( l ) is transmitted, the probability of error cannot exceed E.) Then select any input n-sequence x ( ~such ) that

where Axla) - B1 is the set of elements that belong to A,m) but not to B,. Take B, = A X I S, B,. (Note that we cannot choose B, = A,w since

70

INFORMATION THEORY

the decoding sets must be disjoint.) In general, having chosen x"), xtk-l) and B,, . . . , Bk-,, pick any input n-sequence x ( ~such ) that

...,

Since there are only finitely many input sequences, the process must terminate, say, after t steps (conceivably t = 0). If we show that t is necessarily >s, the sequence x"), . . . , and corresponding decoding sets B,, . . . , B, form, by construction, a code (s, n, A) with

A 5 max P{Y Bi 1 X = xti') Il i s s

+ P{(X, Y) 4 A);

To show t 2 s, we will prove that E 5 t . 2-a t 2 s then follows from the definition of E . Let us examine P{(X, Y) E A). We have P{(x, Y) t A) =

2

P(X, Y) =

(x,Y)EA

T

5 E. the result

1

P ( X ) [ ~ ~ ~x)]. (Y

Let B = Uj=, B,. (If t = 0 take B to be the empty set 0. The argument below will go through essentially unchanged.) Now

(B denotes

the complement of B). Since B n A, c B, the first term of (3.5.2) cannot exceed

2 p(x)Y2~P(Y I x) = Po! r 8). B But since B, c Axti,,

1

NOWif y E Axti,, then ( x ( ~ )y), E A, and hence log [p(y ~ ( ~ ' ) / p ( y> ) ]a, or

Therefore the first term of (3.5.2) is St 2-".

71

THE DISCRETE MEMORYLESS CHANNEL

In order to estimate the second term of (3.5.2), we note first that

B n A,

=0

when x is a code word x(%)'.

(3.5.3)

To verify this, we will show that if a sequence y belongs to A,cr,, then y must also belong to B. Now we know that B, = A,,r, B , ; thus if y # Ur:: B, then y E Bk c B. On the other hand if y : E: tU B,, then since B is the union of all the decoding sets, y E B. This establishes (3.5.3). We now claim that

Ursl

2

1 <1 -

p(y x )

E

for all input n-sequences x .

(3.5.4)

YER~A,

By (3.5.3), the statement (3.5.4) holds for x = x"), . . . ,x ' ~ ) so , assume x not equal to any of the x ( ~ ) If . P { Y E B n A, X = x ) 2 1 - E then since B n A, = A, - UE=,B,, we could enlarge the code by adding the input n-sequence x(~+')= x and the decoding set B,,, = B n A,. This contradicts the assumption that the process terminates after t steps. This proves (3.5.4) and shows that the second term of (3.5.2) is 5 1 - E. Thus P { ( X , Y ) E A ) t ~ 2 - " 1 - E , or E j t -2-" P{(X,Y ) # A); the proof is complete.

I

+

+

Notice that the fact that the channel is memoryless is not used in the proof. We used only the existence of a system of probability functions p(y x), where x is an input n-sequence and y an output n-sequence. Thus the lemma applies to any discrete channel whose probability functions do not depend on the state, or equally well to an arbitrary discrete channel whose initial state is chosen at random in accordance with a known distribution. This observation will be useful in Chapter 7.

I

Proof o f Theorem 3.5.1. Let n be any positive integer. Let po(a) be an input distribution that achieves channel capacity. In Lemma 3.5.2, take s = [2nR],a = n[(R C ) / 2 ] < nC, and determine the distribution p(x) by choosing X I , . . . , X , independently, each Xi having the distribution po(a). In other words, if x = ( a , , . . . , a,), let

+

By Lemma 3.5.2, there is a code ([2nR],n, A,) with A,

5 [2nR]2-tn'R+C'+ P { ( X , Y ) 4 A ) .

(3.5.5)

Since R < C , the first term of (3.5.5) approaches zero as n approaches infinity. Now

72

INFORMATION THEORY

In order to estimate this expression, we first observe that since XI, . . . , Xn are independent, so are Y,, . . . , Yn. (The converse is not true; see Problem 3.13.) To see this, note that

(

X1

$A ={

,

. . . ,a

, . . . , Bn):los

[TI-

,(Pi

I"J]

P(@,)

- 2 log ---

2

n i-1

Now for each k = 1,2, . . . , n, define a random variable U(X,, Y,) as follows: If Xk = a,, Yk = B,, let a,) . -----U(Xk,Yk) = log p(Bk

I

~ @ k )

Then U(Xl, Y,), . . . , U(X,, Y,) are independent, identically distributed random variables with expected value

U(Xi, Y,),the arithmetic But by the weak law of large numbers, (lln) average of n independent, identically distributed random variables with expected value C, converges in probability to C. Since i ( R C ) < C, P{(X, Y ) A} -+ 0 as n + w , and the proof is complete. Note that in order for the above argument to succeed, we must choose the number a large enough so that the first term of (3.5.1) will approach zero, but small enough so that the second term of (3.5.1) will approach zero. We are now going to give the second proof of the fundamental theorem.

+

THE DISCRETE MEMORYLESS CHANNEL

73

As we have indicated previously, the idea is that if a code is selected by choosing the code words at random in a certain way, then with high probability a "good" code is obtained. Before we proceed, we note that for the purpose of proving the fundamental theorem, we may assume without loss of generality that we are using the ideal observer with all code words equally likely. More precisely we have the following: Lemma 3.5.3. Given a discrete memoryless channel and a positive number R,, suppose that for each R < R, we can find a sequence of codes ([2nR],n) whose average probability of error approaches zero as n -+ co. Then it follows that for each R' < R, we can find a sequence of codes ([2nR'],n, 1,) such that 1, a 0 as n + co. of a code with words [Recall that the average probability of error d l ) ,. . . , x'" and a given decision scheme S is the probability of error associated with a random experiment; this involves selecting a code word at random with all words equally likely, transmitting the word through the channel, and then making a decision at the output according to the scheme S. In particular we may take S to be the ideal-observer decision scheme (calculated for the uniform input distribution). Thus Lemma 3.5.3 implies that if we can construct codes to maintain any transmission rate below capacity and at the same time have an overall probability of error, calculated for the ideal-observer decision scheme with all words equally likely, approaching zero with increasing code-word length, then it is possible to construct a (possibly) different set of codes which maintain any rate below capacity and whose maximum probability of error approaches zero.] Proof. Let the code words of the code ([2nR], n) be x'l)', . . . , X ' 8 ) , s = [2nR]. If the average probability of error is p,(e), then

I

<

We claim that p,(e x")) 2p,(e) for at least half of the words d l ) , . . . , x'". For if p,(e x'") > 2 p,(e) for (say) i = 1,2, . . . ,r where r ;t s/2, then

I

contradicting (3.5.6). (All we are saying here is that no more than half of a set of nonnegative numbers can exceed twice the average of the set.) Thus we canfind code words dill', . . . , d i r ) , r 2 s/2, such that p,(e ( d i k ) ) 2pn(e) -+ 0 as n -+ a.Thus by using the words x("), . . . , x"*) and the corresponding decoding sets B,,, . . . , B,, to form a subcode

<

74

INFORMATION THEORY

of the original code, we can form a sequence of codes (r,, n, A,) where r, 2 $[2nR]and A, 5 2p,(e) -t 0 as n + co. Now 4[2?1R] 2

[g . 2nR] = [2n(R-117~) ]=[ p i ]

where R,' -+R as n -+co. Hence if we are given R' < R,, we need only choose R such that R' < R < R, and construct the corresponding sequence of subcodes (r,, n, An). For sufficiently large n, r, 2 [2nR']; this proves the lemma. We may now give the second proof of Theorem 3.5.1. Proof of the Fundamental Theorem by "Random Coding." As in the first proof of the theorem, let p,(a) be an input distribution that achieves channel capacity, and let p(x) be the distribution on input n-sequences determined by choosing X I , . . . , X, independently, each X, having the distribution p,,(a). We construct a random experiment as follows. 1. Choose code words (of length n) x"), . . . , x(") independently (s = [2nR]); for each i = 1,2, . . . , s, let P { x ( ~=) x) = p(x). In other words, choose the components of each x(" independently inaccordance with the capacity-achieving distribution. For example, if the channel is binary and the distribution p(0) = 2/3,p(l) = 113 achieves capacity, and s = 3, n = 4, then the probability of obtaining a code with the words x(l) = 1101, x ( ~=) 0010, x ( ~=) 0101, is

2. Calculate the average probability of error ~ ( x ' l )., . . , for the code selected in step 1, using the decision scheme determined by the ideal observer with all words equally likely. The notation emphasizes that E is a random variable which depends on x"), . . . , x'". We are going to show that E[E] approaches zero as n approaches infinity. Since there must be at least one code with E j E[E], it will then follow that we can construct a sequence of codes whose average probability of error goes to zero. Lemma 3.5.3 then yields the desired result. By definition of the average probability of error, E = (11s) E,, where E~ = E,(x(~), . . . , x(")) is the probability of error when x(Q is transmitted for the code chosen in step 1. Hence the expected value of E is 1" (3.5.7) E[E] = - 2 E[E,].

z:=,

S i,l

We may write E[EJ as follows: E [ E ~=]

2 P{x") X

1

= x)E[E, x ( ~= ) x].

(3.5.8)

THE DISCRETE MEMORYLESS CHANNEL

75

Now if Bi is the decoding set corresponding to the code word x ( ~ )then , Ei

=

Equivalently, ~i

where

2 P(Y1 x ' ~ ' ) .

y$Ri

=

2Y P(YI ~ ( ~ ) ) y i

yi = yi(x(l),. . . , x ' ~ ) y) ; = 0 if y E Bi = 1 if y # B i

(note that the B, = B,(x(l),. . . , x(") are "random sets" which depend on the code chosen in step 1). Consequently

I

I

) x ) may be interpreted as the where p*(e x ( ~ )x , y) = P { y # B, x ' ~ = probability of a decoding error given that

a. x(') is transmitted, c. y is received. From (3.5.8) and (3.5.9) we have

1

To estimate p*(e x(", x , y), we note that if x ( ~=) x is transmitted and y is received, then by definition of the ideal observer with equally likely inputs, an error in decoding will be made only if there is a n input nsequence x' such that p(y x') 2 p(y x), and a t least one code word x"), j # i, is chosen to be x' (see Fig. 3.5.3). PW 1 x') Now let G(x, y) be the set of all input n-sequences x' such that p(y x') 2 p(y x ) ; thus G(x, y) is the set of all input n-sequences > y'x that could conceivably appear at the output x p(yl~) of the decoder if x is transmitted and y is Fig. 3.5.3. Proof of the funreceived. If x'" = X is transmitted and y is damental theorem by random received, the occurrence of an error implies coding.

I

1

I

I

76

INFORMATION THEORY

that at least one x'j', j # i, belongs to G(x, y). Therefore ~ { # B~ y I x"' = x )

I ~ { x ' jr' G(x, y)

for at least one j # i ( x'" = x )

Since the x ( ~are ) independent, p*(e ( x'", x, y)

I 2 ~ { x " c' G(x, y)). i#i

(Observe that only pairwise independence is required here, that is, all we need is that x ' ~ 'and xfj' be independent for i # j.) But since each x ( * ) has the same distribution, P{x(" c G(x, y)) =

2

.j = 1, 2, . . . , s.

p(xf),

(3.5.1 1 )

x'rGfx,~)

If we define Q(x, y) to be the probability (3.5.1 l), we have

Now consider the set A of Lemma 3.5.2 with a = Jn(R

+ C ) , that is,

We are going to estimate Q ( x ,y) for ( x , y) c A. If ( x , y) E A then log [p(x y)/p(x)] > i n ( R C ) , or p(x) < 2-*n'R+C'p(x y). Hence ( x , y) c A implies [note (x,y) c A, x'r G(x, y ) => (x', y) 6 A ]

+

I

I

< 2-*"'

Q(x, y) = 2 p(xl) x'rff(x,y) It follows from (3.5.12) that

I

p*(e x'", x , y)

R+C'

2

p(xl ( y) j 2-'nfR+C'.

x'cU(x.y)

s 2-fn'R+C' fo r

(x, Y ) ~ A .

(3.5.13)

Now from (3.5.10) we have

I

By (3.5.13) and the fact thatp*(e x'j), x , y) is always 51, we have

Since s j 2nR, the first term of (3.5.14) approaches zero as n approaches infinity. The second term approaches zero by the argument used in the previous proof of the fundamental theorem [see text after (3.591. It

THE DISCRETE MEMORYLESS CHANNEL

77

follows that E[E,]5 b,, i = 1,2, . . . , s, where lim bn = 0. By (3.5.7), n- w E[E] 5 b, -+ 0, and the proof is complete. Note that the proofs of the fundamental theorem do not give us a convenient algorithm for constructing the required codes. In fact, all efforts to construct such an algorithm have so far failed. All that can be said is that if a code is selected at random, on the average it will be good. We can sharpen this statement as follows. Suppose we desire a code with average probability of error <S. In the random-coding proof, choose n large enough so that E[E] < d/K, where K is a fixed positive number. It then follows that the probability of obtaining a code whose average probability of error Z is 2S is less than 1/K. For suppose P{E 2 S ) 2 1/K. Then we would have

2 SP{S 2 6 ) 2 6 / K,

a contradiction.

3.6. Exponential error bounds The fundamental theorem asserts the existence of codes ([2nR],n, 1,) with lim 1, = 0. In this section we study the rate at which 1, approaches n-ao

zero and show that 1, + 0 exponentially with n. The estimation of 1, is based on the following lemma.

Lemma 3.6.1. If Y is any nonnegative (discrete) random variable and d is any positive real number, then P { Y 2 d ) 5 E(Y)/d. Proof.

Now taking Y = erx where X is any random variable and r is any positive real number, we obtain the following lemma.

Lemma 3.6.2. If X is an arbitrary (discrete) random variable, b any real number, and r any positive real number, then P{X

2 b) 5 e-rbE[erX].

For by Lemma 3.6.1, P{erx 2 erb) 5 E[erx]/erb.

78

INFORMATION THEORY

We digress for a moment to consider a property of characteristic functions.

Lemma 3.6.3. Let X be a discrete random variable having only finitely many possible values. Assume X is nondegenerate, that is, not identically constant. Let p(s) = E[ed], p(s) = log, &). Assume s is real; then a. pM(s)> 0 for all s. b. p(s) - spf(s)< 0 for s

< 0.

(The primes indicate differentiation.) Proof. Suppose X takes the values z l , z,, . . . ,z,. Then

and

Now the Schwarz inequality states that for arbitrary random variables Yl and Y,, [E(YlYZ)l2 5 E(Ylz)E(YZz)with equality if and only if a Yl b y z = 0 for some constants a and b, not both zero. (To prove the Schwarz inequality, note that

+

If Yl E 0, the inequality is immediate, so assume Yl

+ 0. Then

is a quadratic polynomial f ( l ) which is nonnegative for all A; hence the quadratic equation f ( A ) = 0 has either two complex roots or one real repeated root. Thus the discriminant [2E( Yl YZ)l2- 4E( Y:) E( YzZ)is SO and the result follows. Now if Yl r 0, equality holds and also

79

THE DISCRETE MEMORYLESS CHANNEL

+

aYl b Y, r 0 provided b = 0. Thus assume Yl $ 0. Then equality holds if and only if the discriminant off (1) is zero, that is, if and only if E [ ( 1 Yl YJ2] = 0 for some 1, or equivalently if and only if 1Y, Y, G 0 for some 1.) In the Schwarz inequality, take Yl = eat,, Y, = Xedt2; we then find bXe"12 E from (3.6.1) that p"(s) 2 0, with equality if and only if ae"t2 0, that is, a bX = 0, for some a and b, not both zero. Since X i s nondegenerate by hypothesis, equality cannot occur, and therefore p"(s) > 0 for all s, proving (a). To establish (b), let g(s) = p(s) - sp'(s). Then

+

+

+

+

by (a). Thus g(s) is montone increasing for s < 0. Since g(0) = p(0) = log p(0) = 0, (b) is proved. We now establish the exponential bound on the probability of error. Theorem 3.6.4. Under the hypothesis of Theorem 3.5.1, there exist positive constants K and a, depending on the channel and on R, and a sequence of codes ([2nR],n, A,), such that 1, 5 Ke-"". Proof. In view of (3.5.5) [or (3.5.14)] we need only show that in the proof of Theorem 3.5.1, P{(X,Y) # A ) -+0 exponentially as n a. As in the proof of Theorem 3.5.1, write

-

where the random variables U(X,, Yi) are independent and identically distributed with expected value C. Let Ui = U(Xi, Y,), and Z = ZZ, Ui. Then by Lemma 3.6.2,

P{Z I; d ) = P{-Z 2 - d ) I; eTdE[e-rZ],

r

> 0,

d any real number,

or

P{z 5 d ) j e-rdE[erZ], But E [eTZ]= E

I

E [eruz]

eTU* =

r

< 0.

by independence of the Ui

zl:

Let p(r) = E[erU1],p(r) = log, p(r) as in Lemma 3.6.3. Then

80

INFORMATION THEORY

In (3.6.2) we are free to choose d and r, subject to r < 0. To obtain the "tightest" bound for a given d we might try to differentiate the exponent with respect to r and set the result equal t o zero. If we set d = n,uf(r) in (3.6.2) we obtain

By Lemma 3.6.3b, ,u(r) - r,ul(r) < 0 for r < 0. In addition, ,ul(r) is a continuous function (which is also monotone increasing for all r by Lemma 3.6.3a) such that ,ul(0) = plf(0)/pl(O) = pf(0) = E(U,) = C. It follows that there is an r, < 0 such that ,u1(r0)2 (C R)/2. Using (3.6.3) we have

+

where ,u(ro) - ro,u'(ro) < 0. This establishes the exponential bound.

3.7. The weak converse to the fundamental theorem In this section we prove a result that may be regarded as a converse to the fundamental theorem. Intuitively, what we prove is that it is not possible to maintain a transmission rate R > C while at the same time reducing the probability of error to zero. First we relate the probability of error of a code to the uncertainty measure.

Theorem 3.7.1. (Fano's Inequality). Given an arbitrary code (s, n) consisting of words x'l), . . . , x'", let X = (X,, . . . , X,) be a random vector that equals x ( ~ with ) probability p(x(')), i = 1, 2, . . . ,s, where p(x(") = 1. [In other words, we are choosing a code word at random according to the distribution p(x(')).] Let Y = ( Y , , . . . , Y,) be the corresponding output sequence. If p(e) is the probability of error of the code, computed for the given input distribution, then

z:=,

Proof. Consider any output sequence y. Let g(y) be the input selected by the decoder; thus if y is received, an error will occur if and only if the transmitted sequence is not equal to g(y). Now divide the set of values of X into two groups, one group consisting of g(y) alone, and the other group consisting of all other code words. By the grouping axiom (Section 1.2, Axiom 3) we obtain

82

INFORMATION THEORY

.

I;,

Also (Corollary 1.4.3.1) H( Y,, . . , Yn) 5 H(Y,) with equality if and only if Y,, . . . , Yn are independent. The lemma follows. We are now ready for the main result. Theorem 3.7.3. (Weak Converse). The average probability of error of any code (s, n) must satisfy

where C is the channel capacity. Consequently if s 2 2n'C+*1where 6 > 0,then

Thus if R > C, no sequence of codes ([2nR],n) can have an average probability of error which -+0 as n -+ w, hence no sequence of codes ([2nR],n, 1,) can exist with lim A, = 0. n-m

Proof. Choose a code word at random with all words equally likely, that is, let X and Y be as in Theorem 3.7.1, with p(x(0) = l/s, i = 1, 2, . . . ,s. Then H(X) = log s so that I(X I Y) = log s - H(X I Y) (3.7.3) By Lemma 3.7.2,

I

Since I(Xi Y,) j C (by definition of capacity), (3.7.3) and (3.7.4) yield logs - H(XI Y) j nC.

(3.7.5)

By Theorem 3.7.1, Hence

I

H ( x y ) _< H ( m , 1 - P O )

I

H(X Y) j log 2

+ p q log (s - 1).

+ p(e)log s.

(3.7.6)

(3.7.7)

The result now follows from (3.7.5) and (3.7.7). Given a sequence of codes with at least 2n(C+d'words of length n and

-

average probability of error pn(e), Theorem 3.7.3 implies that p,(e) cannot be reduced to zero by allowing n to become arbitrarily large. However, p,(e) need not be a monotone function of n, and it is at least conceivable that there might be an integer n, such thatpns(e) = 0. In this case we could, by using the codes of word length no, achieve perfect reliability while maintaining a transmission rate above capacity. But this

83

THE DISCRETE MEMORYLESS CHANNEL

possibility is excluded, for an examination of (3.7.5) and (3.7.6) shows that the number of words in a code with zero average probability of error is at most 2nC. Thus if the rate of transmission exceeds capacity, the average (hence the maximum) probability of error is bounded away from zero. It is possible to prove a stronger result than Theorem 3.7.3, namely the strong converse to the fundamental theorem. The strong converse states that for any sequence of codes ([2nR],n, 1,) with average probability of errorp,(e), notonly is it true thatpn(e) cannot approach zero if R > C, but in fact lim p,(e) = 1 (hence lim 1, = 1). n-m

n-o~

Special cases of the strong converse will be considered in Chapters 4 and 8. For a general discussion of this subject, see Wolfowitz (1961).

3.8. Notes and remarks The material in Sections 3.1 and 3.2 is standard and is discussed by Feinstein (1958) and Wolfowitz (1961). The method of calculation of channel capacity described in Section 3.3 is due to Muroga (1953); see also Fano (1 961). Programming techniques and algorithms for the general problem of computing channel capacity are discussed by Eisenberg (1963). The fundamental lemma 3.5.2 and the corresponding proof of Theorem 3.5.1 are due originally to Feinstein (1954, 1958). The present form of the fundamental lemma is due to Blackwell, Breiman, and Thomasian (1958, 1959) and Thomasian (1961). The idea of "random coding" was suggested (1948) and later made precise (1957) by Shannon. Lemma 3.5.3 was pointed out by Wolfowitz (1961). The exponential bounding technique of Section 3.6 is due to Chernoff (1952); the technique was first applied to the calculation of error bounds by Shannon (1957). The essential content of Theorem 3.6.4 is that if XI,.. . , Xn are independent, identically distributed random variables, each having only finitely many possible values, and if p is the common expected value, then for every E > 0, P{I(X, . . . Xn)/n - pl E ) 0 exponentially as n co; in other words, we have exponential convergence in the weak law of large numbers. For more general results of this type, see Katz and Thomasian (1961). Theorem 3.7.1 was proved by Fano (1952, unpublished). The result was applied to prove the weak converse by Feinstein (1958). Another proof of the fundamental theorem has been given by Wolfowitz (1961). Wolfowitz's approach is combinatorial and is based on a precise and general formulation of the notion of a "typical" sequence (Section 1.3). The approach of Wolfowitz seems to be better suited than the one

+

-+

+

>

-+

84

INFORMATION THEORY

given here for proving the strong converse, and also for dealing with questions involving the computation of the capacity of channels with memory (Chapter 7). However, the approach of this chapter seems to generalize to a broader class of channel models. [See, for example, Dobrushin (1959).] A possible source of difficulty in understanding the physical interpretation of the fundamental theorem is the fact that if an error is made in decoding, an entire block of digits produced by the source may be reported incorrectly. However, if we have a code (2nR,n, A,), where each code word is assigned to a sequence of nR source digits, the average number of errors per block is nRA, (making the conservative assumption that a decoding error results in a complete scrambling of the associated block of source digits.) Since the number of source digits per block is nR, the average number of errors per digit produced by the source is A,. Thus by reducing A, toward zero we are in fact achieving reliable transmission. One final remark on terminology. The information processed by a channel is the information conveyed about a single input letter by a single output letter. Hence the units of Z(X Y) (and the units of C) are sometimes called "bits/symbol." We have seen that if the channel can transmit 1 symbol per second, then C is the least upper bound on the number of bits (that is, binary digits) per second which can be reliably handled by the channel. Similarly if the channel can transmit u symbols per second, then the corresponding upper bound on the source rate is uC bits per second.

I

PROBLEMS

' I:

3.1 A discrete memoryless channel is characterized by the matrix Y1

::[:

Y2

Ys

x s Q 4 If p(xl) = 112, p(x,) = p(x3) = 114, find the ideal-observer decision scheme and calculate the associated probability of error. 3.2 Given a discrete memoryless channel with input alphabet xl, . . . , z ~ , output alphabet yl, . . . ,y,, and channel matrix [p(y, I x,)], a randomized decision scheme may be constructed by assuming that if the channel output is y, ( j = 1, . . ,L), the decoder will select x, with probability q,, (i = 1, . . . , M). For a given input distribution show that no randomized decision scheme has a lower probability of error than the ideal observer. 3.3 (Feinstein 1958). The output of a discrete memoryless channel K, is connected to the input of another discrete memoryless channel K,. Show that

.

THE DISCRETE MEMORYLESS CHANNEL

85

the capacity of the cascade combination can never exceed the capacity of Ki, i = 1,2. ("Information cannot be increased by data processing.") 3.4 Let X and Y be discrete random variables, with X taking on the values s,, . . . ,xM and Y the values yl, . . . ,y ~ Let . g be an arbitrary function with domain {x,,. . . , xM}. Define Z = g(X). Show that H ( Y I Z) 2 H(Y I X ) . 3.5 A number of identical binary symmetric channels are connected in cascade, as shown below. Let Xk be the output of the kth channel ( = the input of the (k 1)th channel). Assume 0 < ,9 < 1, where ,9 is the probability that a single transmitted digit is in error.

+

Problem 3.5

-

Let p, = P{X, = O), n = 0, 1, . . . . Assuming p, known, find an explicit & as n + w regardless of p,, so that the expression for p,. Show that p, capacity of the cascade combination approaches zero as n -+ w. 3.6 Find the channel matrix of a cascade combination in terms of the individual channel matrices 3.7 Find the capacity of the general binary channel whose channel matrix is

n1

n,

3.8 (Shannon 1957). If and are channel matrices of discrete memoryless channels K , and K,, respectively, the sum of Kl and K2 is defined as the channel whose matrix is

Thus the sum channel may be operated by choosing one of the individual channels and transmitting a digit through it. The input or output symbol always identifies the particular channel used. X f Ci is the capacity of Ki, i = 1, 2, and C is the capacity of the sum, show that

3.9 (Shannon 1957). The product of two discrete memoryless channels Kl and K , is a channel whose inputs are ordered pairs (xi, xi') and whose outputs are ordered pairs (yk, y,') where the first coordinates belong to the alphabet of K , and the second coordinates to the alphabet of K,. The channel probabilities are given by

Thus the product channel is operated by sending a digit through K, and then independently a digit through K2. Show that the capacity C of the product channel is the sum C1 C, of the individual capacities.

+

86

INFORMATION THEORY

3.10 (Fano 1961). Evaluate (in an effortless manner) the capacities of the channels whose matrices are given below.

c.

0

erase

1

[Channel (c) is the "binary erasure" channel.] 3.11 It is desired to construct codes for a discrete memoryless channel with the property that all code words belong to a specified set. Prove the following. a. (Thomasian 1961). Let p(x) be an arbitrary probability function defined on the set of all input n-sequences. For any real number a, let A = {(x, y): log [p(y x)/p(y)] > a ) . Let F be any set of input n-sequences. Then for each positive integer s there is a code (s, n, A), with aN code words belonging to F, such that 1 < s - 2-' P{(X, Y) # A ) P{X # F).

I

+

+

b. Consider the binary symmetric channel. For each n, let F, be the set of all binary n-sequences whose weight (that is, number of ones) is 5 rn, where r is a fixed number E [O, 4). We wish to prove a theorem of the following form. Given R < C,, there exist codes ([2nR],n, I,), with all words belonging to Fn,such that 1, -+ 0 as n + a. How large can C, be? 3.12 Given a discrete memoryless channel with capacity C,show that there is an input distribution p(x) such that the corresponding information processed is equal to C. Hence Cis a true maximum rather than simply a least upper bound. 3.13 Let XI,. . . , X, be a sequence of inputs to a discrete memoryless channel, and let Y,,. . . , Ynbe the corresponding output sequence. Show that it is possible to find a channel and an input distribution such that Y,,. . . , Yn are independent without X,,. . . , Xn being independent.

CHAPTER FOUR

Error Correcting Codes 4.1.

Introduction; minimum-distance principle

In this chapter we attack the problem of finding "good" encoding schemes to combat errors in transmission. As the fundamental theorem shows, there are codes that allow transmission a t any rate below channel capacity with an arbitrarily small probability of error. However, the proofs we have given provide no clue as to how to construct the codes; in fact, there is at present n o systematic procedure for constructing codes that meet the specifications of the fundamental theorem. On the other hand we d o have considerable information about the structure of codes, especially for the binary symmetric channel. We will be able to give precise bounds on the error correcting ability of codes, and in addition we will exhibit encoding and decoding schemes which have desirable properties with respect to implementation. Until the final section of the chapter, our discussion is restricted to the binary symmetric channel. The inputs to the channel will be chosen from a set of binary code words of length n, that is, a set of n-digit sequences of zeros and ones. We assume in all cases that the code words occur with equal probability, and that the ideal-observer decision scheme is used. Since an error may be made in any digit, the set of possible output sequences is the set of all 2" binary sequences of length n. Our first problem is t o find the form of the ideal-observer decision scheme for such a code. If the input code words are w,,w,, . . . , w, and the output sequences are v,, v,, . . . , V z n , then we wish to associate a code word with each received sequence in such a way that the probability of error is minimized. We have found that the ideal observer associates with each received sequence v the code word w for which p(w v) is a maximum. Furthermore, since all code words are equally likely, the code word that maximizes p(w v) is the same as the code word that maximizesp(v w). We now define the distance? d(v,,v,) between two n-digit binary sequences v, and v, as the number of digits in which v, and v, disagree. For example, if

I

I

t Also called the "Hamming distance" after its inventor. 87

I

88

INFORMATION THEORY

then

d(vl, v,) = 3. If d(w, v) is the distance from a code word w to a received sequence v then in order that w be received as v there must be exactly d(w, v) errors in transmission, and these errors must occur in those digits where w and v disagree. Hence p(v W ) = pa(wJ)(1 - p ) n - d w . v ) , where p is the probability that a given digit is transmitted incorrectly. Let us now compare p(v w,) and p(v I w,). If we write di = d(w,, v) (i = 1,2) we have

I

I

< p < 3, so that (1 - p)/p > 1 . p(v I wl) > p(v I w,) if and only if dl < d,.

We shall always assume 0

Thus

Therefore the code word w that gives the largest conditional probability p(v W ) is that word whose distance from v is a minimum. We have thus established the following result.

1

Theorem 4.1.1. Given a code for the binary symmetric channel consisting of s equally likely code words of length n. The "ideal observer" decision scheme is a "minimum-distance" decoder, that is, the decoder selects for each received sequence v the code word w for which the distance d(w, v ) is a minimum. (If there is more than one minimum, any one of the minimizing words may be chosen without affecting the probability of error.) As an example, suppose the code words are w, = 00000 w, = 10011 w, = 11100 w, = 01111. If the sequence v = 0101 1 is received, it is decoded as w, since d(w,, v) = I and d(w,, v) > 1 for i # 4. Similarly, the sequence v' = 001 10 may be decoded as either w, or w, since d(w,, v') = d(w,, v') = 2 and d(w,, v') = d(w3,v') = 3. It is easily verified that the Hamming distance has the characteristic properties of a distance function, that is, d(vl,v,) = 0 if and only if v, = v,. a. d(v,, v,) 2 0, b. 4 ~ 1 v,) , = d@,, ~ 1 ) c. d(vl, v,) j d(vl, v,)

.

+ d(v,, v,)

(the triangle inequality).

ERROR CORRECTING CODES

4.2.

Relation between distance and error correcting properties of codes; the Hamming bound

Since each received sequence v is decoded as the code word "closest" to v in the sense of Hamming distance, it appears that a "good" code is one whose code words are "far apart." If the distance between code words is large, many transmission errors must be made before a code word w is transformed into a received sequence v which is closer to some other word w' than to w. As a start toward making this idea precise, we prove the following lemma. Lemma 4.2.1. Let w,, w,, . . . , w, be binary code words of length n having the property that for a given positive integer e,

d(w,, w,) 2 2e

+1

for i # j.

Then all single, double,. . . ,e-tuple errors in transmission can be corrected. If d(wi, w j ) 2 2e for i # j, then all single, double, . . . , (e - 1)-tuple errors can be corrected and e-tuple errors can be detected but not, in general, corrected. Conversely, any code for which all errors of magnitude j e are corrected must satisfy d(w,, w j ) 2 2e 1 for i # j, and any code for which all errors of magnitude <e - 1 are corrected and all errors of magnitude e are detected must satisfy d(w,, w j ) 2 2e for i # j. Proof. First suppose that the minimum distance between code words is at least 2e 1. Then in order that a code word w be decoded as another code word w' the received sequence must be at least as close to w' as t o w. In order for this to happen, at least e 1 digits of w must be in error. Hence all errors involving e or fewer digits are correctible. If the distance between two code words w and w' is 2e, then an e-tuple error in w will result in a received sequence v whose distance from w is e and whose distance from w' is at least e. [The last assertion follows from d(w, w') d(w, v) d(v, w').] If the distance between v and w' is exactly e, then an e-tuple error in w and an e-tuple error in w' may both result in v. Thus in general it is possible to detect the presence of an e-tuple error but not to decode with perfect accuracy. The converse follows by similar reasoning.

+

+

+

<

+

A geometric verification of Lemma 4.2.1 is shown in Fig. 4.2.1. If d(w,, w j ) = 9 , a change of four or fewer digits in wi will leave the received sequence closer to wi than to w j . If d(w,, w j ) = 8, errors in three or fewer

digits will be corrected, but a change of four digits in wi may leave the received sequence equally far from wi and w j . Another geometric approach to binary coding is to visualize binary sequences of length n as vertices of

90

INFORMATION THEORY

an n-dimensional cube. (See Fig. 4.2.2.) A change in one digit of a binary sequence corresponds to a movement from one vertex to an adjacent vertex. If dist (w,, wj) = d, then every path along the edges of the cube

(a) dist (w,, wj)= 2e

+I

(b) d~st(w,, w,)= 2e

Fig. 4.2.1. Relationship between distance and error correction.

from wi to wj passes through at least d vertices. T o form a code with minimum distance d, we select points on the cube which cannot be connected by paths of length less than d. In Fig. 4.2.2 the points 001, 010, and 111 form a code of minimum distance 2.

Fig. 4.2.2. Binary code words as points on an n-cube.

We saw in Chapter 3 that if a channel can transmit one symbol per second, a set of 2nRcode words is adequate to transmit the information produced by a source which emits R binary symbols per second. Thus a high transmission rate corresponds to a large number of code words. I t is of interest to determine how many code words there can be in a code that corrects e-tuple and all smaller errors. An upper bound is provided by the following theorem. Theorem 4.2.2. (Hamming Upper Bound on the Number of Code Words). If a code consisting of s binary sequences of length n corrects all single, double, . . . , e-tuple errors, then

91

ERROR CORRECTING CODES

Proof. Let the code words be w,, w,, . . . ,w,. For each code word wi, we can draw a "sphere" with "center" at w, and "radius" e; the "sphere" is the set of all binary sequences v such that d(w,, v) 5 e. Since the code is e-tuple-error correcting, the spheres are disjoint. Since the sphere associated with wi contains all sequences that differ from w, in 0, 1, 2, . . . , or e places, the number of sequences in each sphere is

Thus the total number of sequences included in the spheres is

However, the number of sequences belonging to the spheres cannot exceed the total number of n-digit binary sequences, namely 2". Thus

Note that if, for a fixed e and n, s is the largest integer satisfying (4.2.1), there may not exist an e-tuple-error correcting code containing s code words of length n. For example, if e = 1, n = 4, we have 2"/(1 n) = If, and the largest integer that satisfies the Hamming bound is s = 3. However, there is no single-error correcting code with more than two words. (Try it and see.) Thus the Hamming bound is a necessary but not suflcient condition for a code to correct e-tuple and all smaller errors. We remark again for emphasis that we associate each received sequence v with the closest code word wi. If v can arise from an e-tuple error in w, and an el-tuple error in w,, e' > e, then we associate v with w, rather than w,. In other words, we always choose to correct the error of smaller magnitude.

+

4.3.

Parity check coding

Let us summarize the results we have obtained thus far. We are considering a binary symmetric channel whose inputs are binary code words w,, w,, . . . , w,. To obtain a high transmission rate, we would like to have the number of code words as large as possible; at the same time we would like to correct as many transmission errors as possible. These objectives will in general conflict. For example, if we wish to correct single errors and the code-word length is 5, we can find a code with four words, 00000, 10011, 11100, 01 111. If we wish to correct double errors, we cannot use more than two words, for example, 00000, 11111. The

92

INFORMATION THEORY

Hamming bound suggests that as the error correction requirements increase, the maximum possible number of code words decreases. A consideration neglected so far is ease of decoding. The ideal-observer decision scheme is a "minimum-distance" decoder; that is, the received sequence vj is decoded as the code word wi that minimizes d(w,, v,). Thus the decoder must contain a "table" or "code book" whose entries are all binary sequences of the specified code-word length. Each possible received sequence is assigned to the closest code word. The storage requirements for such a decoder are quite severe. In this section we investigate a coding scheme that improves the efficiency of the decoding process, and at the same time gives considerable insight into the structure of error correcting codes. The earliest "parity check" codes were devised as a very simple and easily mechanized error detection scheme. Given a code consisting of all binary sequences of length n, an extra digit (the "parity digit") is added to each code word, the digit being chosen so as to make the total number of ones even (or odd). For example, if the original code is {00,01, 10, 1I}, and an "even parity" is used, that is, the parity digit is chosen so that the number of ones is even, the resulting code is {000,011, 101, 110). It is not difficult to see that if a single error (or in fact any e-tuple error where e is odd) is made in transmission, the received sequence will have an odd number of ones. Hence all errors involving an odd number of digits can be detected. The only operation the decoder must perform is to add the number of ones in the received sequence. This idea was generalized by Hamming and later by Slepian, who formulated the basic theory we present here. If r,, r,, . . . ,r , are the digits of a code word, the fact that the number - . . + r, = of ones is even may be expressed by the condition r, r, 0 modulo 2.t Thus it appears that the binary sequences of length n which have an even number of ones are precisely the sequences which satisfy a certain modulo 2 linear equation. To generalize this idea we consider codes whose words satisfy a set of simultaneous linear equations.

+ +

Definition. Given a set of simultaneous linear equations of the form allr, al,r2 . - . a d n = 0

+

+ +

(mod 2).

(4.3.1)

t Modulo 2 arithmetic is the arithmetic of the field consisting of two elements 0 and 1 , with the rules 0 + 0 = 1 + 1 = 0 , 0 + 1 = 1 + 0 = 1 , 1 . 1 = 1 , 0 . 1 = 1 . O = O.O=0,0/1 =0,1/1 = l , a - b = a + b f o r a , b = O o r l .

ERROR CORRECTING CODES

93

The set of solutions to (4.3.1) is called aparity check code (The terminology group code is also used, for reasons which will become clear in Section 4.4.) The m by n matrix A = [a,,]is called the parity check matrix. If the rank of A is t , and if columns j,, . . . ,j, of A are linearly independent, then the n - t = k digits ri, j # j,, . . . ,j # j,, may be specified arbitrarily in (4.3.1), and are called a set of information digits of the code. The digits r,,, . . . , rit are then determined from (4.3.1), and are called a set of check digits for the code. A parity check code must contain 2'"words, where k = n - t = n - rank A ; each assignment of values to a set of information digits determines a unique code word. If a parity check code is to be used to process the information produced by a source which emits R binary symbols per second, then in accordance with the discussion in Chapter 3 we would set 2nR= 2&,or R = kln. Thus a high percentage of information digits corresponds to a high transmission rate; on the other hand, in general the greater the percentage of check digits the greater the error correcting ability of the code. An example of a set of parity check equations is shown below for the case n=6,t=m=3,k=3.

We may choose r,, r,, and r, as check digits and r4, r,, and r, as information digits. To form a code word, we first choose values for r,, r,, and r,; for example, r4 = 0, r, = 1, r, = 1. We then solve the parity check equation toobtainr, = r 4 r 6 = 1 , r , = r, + r 4 r , = O , r , = r , + r 4 + r , = 1. Thus the code word is 110011. Since there are three information digits, there are 23 = 8 code words in the code; all eight code words are listed in Table 4.3.1. The fact that the code words of a parity check code are determined by a set of linear equations allows us to use algebraic techniques to study the

+

Table 4.3.1.

+

Example of a parity check code rl

r2

r3

r4

r5

r6

94

INFORMATION THEORY

properties of codes. However, there is a basic question concerning the adequacy of parity check codes. It would be desirable to know whether or not such codes meet the specifications of the fundamental theorem. In other words, given a binary symmetric channel with capacity C > 0 and a positive number R < C, we would hope to find a sequence of parity check codes which maintain the transmission rate R with a probability of error which approaches zero with increasing code-word length. In fact we can find such a sequence, as we shall prove in Section 4.6. Thus from the point of view of the fundamental theorem, there is no loss of generality in considering parity check codes. We now turn to the problem of relating the parity check equations to the error correcting ability of the code. If the digits r,, r,, . . . ,r , are transmitted over a binary symmetric channel, the received digits will be rl ,r, , . . . ,r,' where rip= ri if no error has been made in r,, and ril = ri 1 (mod 2) if an error in ri has occurred. Suppose that at the receiver we try to determine whether or not r,', r,', . . . , r,' satisfy the parity check equations. In other words, we calculate I

,

+

(mod 2).t

The binary sequence

.I

.cm

is called the corrector or syndrome associated with the sequence v (r,', . . . , r;). In matrix form,

=

(In general, code words and received sequences will be written as row vectors; vT is the transpose of v.) If no error has occurred, then rif = rt

t For the remainder of this chapter all arithmetic will be modulo 2 unless otherwise specified.

95

ERROR CORRECTING CODES

for all i and hence c is the zero vector 0 . We will see in the next section that the construction of the minimum-distance decoder involves the consideration of the correctors associated with various error combinations. Before looking at the construction of the minimum-distance decoder, we develop a few properties of correctors. Suppose that the sequence w = (r,, . . . ,r,) is transmitted and the sequence v = (r,', . . . , r,') is received. The sequence z = v - w = (rlf - r l , . . . , rn' - r,) is called the error pattern vector corresponding to w and v; the error pattern vector is thus a binary sequence with ones in the positions that are in error, and zeros elsewhere. The corrector associated with v is c = AvT = A ( Z ~ w T ) = AzT AwT = AzT (Wis a code word so that AwT = 0 ) . Consequently the corrector is determined by the error pattern vector. Now if z has ones in positions j,, j,, . . . ,j, and zeros elsewhere, that is, digits j,, . . . ,j, are in error, then the vector AzT is simply the (mod 2 ) sum of columns,j,, j,, . . . ,j, of A. For example, in the code described by the parity check equations (4.3.2),errors in positions 2 and 4 will yield the corrector

+

+

Observe that a single error in position 1, or a triple error in positions 2 , 3 , and 5, will yield the same corrector; in general many different error patterns will correspond to a given corrector. (In the next section we find out how many.) 4.4.

The application of group theory to parity check coding

The object of this section is to prove the following theorem. Theorem 4.4.1. The minimum-distance decoding scheme for a parity check code may be described as follows.

1. Given a received sequence v, calculate the corrector c associated with v. 2. Examine all error patterns whose corrector is c, and correct the error of smallest magnitude. In other words, if among all sequences z such that AzT = C, zO is a sequence of minimum weight, that is, minimum number of ones, v is decoded as v - 2,. Thus if v is received, we declare that the code word v - z, was transmitted (note that v - z, is a code word since A(v - z , ) ~= AvT - AzOT = c - c = 0 ) and that error pattern zo occurred.

%

INFORMATION THEORY

Theorem 4.4.1 implies that the minimum-distance decoder may be regarded as a function which assigns an error pattern to each corrector. Thus instead of storing a table of all 2" binary sequences of length n and their associated code words, we need only store a table of all possible correctors and their associated error patterns. Since the correctors are linear combinations of the columns of A and the rank of A is t = n - k , there are 2t = 2"-" distinct correctors, in general a much smaller number than 2". To prove Theorem 4.4.1, as well as to gain further insight into the structure of parity check codes, it is convenient to introduce some concepts from group theory. Definition. A group is a set G together with an operation,t which we refer to as "addition," satisfying

+ b c G (closure). 2. a + ( b + c) = ( a + b) + c for all a , b, c, c G (associativity). 3. ThereisanelementOrGsuchthata + 0 = 0 + a = a for a l l a r G 1. a E G, b r G implies a

(identity element).

4. For each a c G there is an element ( - a ) (-a) a = 0 (inverse).

+

A group is abelian or commutative if a

+

E

G such that a

+ (-a)

=

+

b = b a for all a , b r G. The integers under ordinary addition and the set of binary sequences of a fixed length n under modulo 2 addition are examples of abelian groups. As far as the application to parity check coding is concerned, the basic result needed is the following: Theorem 4.4.2. Let S be the set of code words in a parity check code. Then S is a group under modulo 2 addition. Proof. Let A be the parity check matrix of the code. Then a binary sequence w is a code word if and only if w satisfies the parity check equations, that is, if and only if AwT = 0. TO prove the closure property we note that if w1 c S, w , r S, then AwlT = 0, AwzT = 0. Hence A(wl w , ) ~ = A(wlT wZT)= AwlT AwaT = 0 ; hence w1 w, c S. The associative property follows from the definition of modulo 2 addition. We may take the zero vector as the identity element, since the zero vector always satisfies the parity check equations and is therefore a code word. We may take each element of S as its own inverse, again by definition of modulo 2 addition. This completes the proof.

+

+

+

+

t By an operation we mean a function that assigns to every pair a, b of elements in G, another element a b.

+

97

ERROR CORRECTING CODES

The set S, of course, is an abelian group, since modulo 2 addition is commutative. If the code words of S are of length n, S is said to be a subgroup of the group 33, of all binary sequences of length n, since S is contained in the larger group 33,. We now prove the converse of Theorem 4.4.2. Theorem 4.4.3. Let S be a set of binary sequences of length n that forms a group under modulo 2 addition. Then S is a parity check code, that is, there is a parity check matrix A such that the code determined by A is S. Proof. Let us arrange the code words in an s by n matrix M, where s is the total number of code words. An example is given in Table 4.4.1. It may be verified that the code does in fact form a group; for example, W1 W 2 = Wp,W, W7 = W2.

+

+

Matrix o f code words

Table 4.4.1.

Let k be the rank of M, and m = n - k. Then k is the maximum number of linearly independent rows of M, and also the maximum number of linearly independent columns of M. Assume for convenience that w,, w,, . . . ,w, are linearly independent. (If not, we may rearrange the rows.) Then every code word in S is a modulo 2 linear combination of w,, w,, . . . ,w,. But every linear combination of w,, w,, . . . ,w, is a member of S since S is a group. Thus S coincides with the set of all binary sequences w which can be expressed as

By the linear independence of w,, . . . ,w,, each distinct choice of (A,, . . . , A,) produces a different w. Thus the number of code words in S is 2,. Now assume for convenience that the last k columns of M (columns m 1, m 2, . . . , m k where m = n - k) are linearly independent. Then the remaining columns 1, 2, . . . , m may be expressed as linear combinations of columns m 1, m 2, . . . , m k. In other words, each digit r,, r,, . . . , r , of the code words of S is expressible as a linear

+

+

+

+

+

+

98

INFORMATION THEORY

combination of digits r,,,, r,,,, . . . ,r,,,. Hence we may write a set of linear equations relating r,, r,, . . . , r , to r,,,, r,+,, . . . , r,,,, that is,

The equations (4.4.2) are of the form

where

and I , is an identity matrix of order m. Thus we have shown that every code word w E S satisfies A w ' = 0, where A is the matrix (4.4.3). T o show that every solution of AwT = 0 is in S we note that in solving the equations (4.4.2) the variables r,,,, I,+,,. . . , rm+k may be specified arbitrarily. Thus A W ~= 0 has 2k solutions. But we have shown that S has 2k elements, each of which satisfies AwT = 0. Hence the set of row vectors w satisfying AwT = 0 is identical to the set S of code words, that is, the code determined by A is S. The proof is complete. T o illustrate the process of constructing the parity check equations (4.4.2), consider the code of Table 4.4.1. Since w,, w,, and w, form a maximal set of linearly independent code words, any parity check equations satisfied by w,, w,, and w , will necessarily be satisfied by all code words. [To see this, we observe that AwiT = 0 for i = 1 , 2 , . . . , k implies A(llwlT . . . IkwkT)= 0 for all possible choices of the I , . ] Thus

+

+

-

t Addition and subtraction are the same in modulo 2 arithmetic, but the use of the minus sign in certain arguments facilitates generalizations to nonbinary coding (see Section 4.9).

ERROR CORRECTING CODES

we restrict our attention to the submatrix

formed by w,,w,, and w,. Since the last three columns of Q are linearly independent, we should be able to express the remaining three columns in terms of these. For example, to express column 1 in terms of columns 4, 5, and 6, we have to find numbers a,,a,, and a, such that

+

Equation (4.4.4) yields a, = 1, a, = 1, a, a, = 0 or a, = 1 . Similarly, to express columns 2 and 3 in terms of columns 4, 5, and 6 we have to solve

Solving (4.4.5) and (4.4.6) we obtain b, = 1, b, = 1, b, = 0, dl = 1, d, = 0,da = 1 . From (4.4.4), (4.4.5), and (4.4.6) we may write

Hence the parity check matrix is

Because of Theorems 4.4.2 and 4.4.3 a parity check code is often called a "group code." The term "(n, k) code" is also used to refer to a parity check code with code words of length n and k information digits.

100

INFORMATION THEORY

We now turn to the problem of constructing the minimum-distance decoder for a parity check code. As an example, consider the code defined by the parity check equations

(4.4.7)

The four code words corresponding to (4.4.7) are w, = 0000, w, = 0101, w, = 1110, and w, = 1011. Table 4.4.2. Error pattern vectors and received sequences for a group code

Various error pattern vectors and their associated received sequences are shown in Table 4.4.2. The leftmost element in any row is an error pattern vector. The element v, ( i = 0, 1, 2, 3) in a given row represents the received sequence when code word w, is transmitted and the error pattern vector is v,. For example, if w, is transmitted and the error pattern vector is 0110, that is, errors are made in positions 2 and 3, the received sequence is 1000. The rows of Table 4.4.2 are examples of what are called cosets in group theory. Definition. Given a subgroup S of a group G, the coset associated with an element z E G is the set of all elements z w, where w ranges over S. The coset associated with z is written z S.

+

+

In the example we are considering, G is the group of all sixteen binary sequences of length 4 under modulo 2 addition, S is the set of code words {w,, w,, w,, w,). Various cosets are

The key property of cosets is the following.

101

ERROR CORRECTING CODES

+

+

Lemma 4.4.4. The cosets z, S and z, S are either disjoint or identical. (Thus in Table 4.4.2 no sequence appears in more than one row.) Proof. Suppose z, S and z, S are not disjoint. Then there is an element v' such that v' E z, S and v' E z2 S. Hence v' = z, w, for some w, c S a n d v' = z, w, for some w, E S. To show that z, S a n d z, S are identical, suppose v E z, S. Then v = z, wS for some w, S. Therefore v = v' - w, w, = z, (w, - w, w,). Hence v is of the form z, w where w = w, - w, w, r S. Thus v E z, S. A symmetrical argument shows that v E z, S implies v E z, S, completing the proof.

+

+

+

+ +

+

+

+

+

+

+ +

+

+ +

+ +

+

Each coset z + S has exactly as many elements as S since all the elements z + w (w E S)are distinct. (Ifz + w, = z + w,, then w, = w,.) Hence the number of cosets is the number of elements of G divided by the number of elements of S. If G is the group of 2" binary sequences of length n and S is a group code with 2k code words of length n, then the number of cosets is 2n-k.Thus the number of cosets and the number of possible corrector values coincide. This is not a coincidence, as we shall see shortly. We now establish a symmetry property of the minimum-distance decoder for a group code. Lemma 4.4.5. Let S be a group code. Suppose that for a particular code word w, and a particular error pattern vector z,

for all code words w,. Then

for all code words w,w'. The interpretation of Lemma 4.4.5is as follows. Suppose that the error pattern z is corrected by the minimum-distance decoder when the code word w, is transmitted, that is, the minimum-distance decoder assigns to the received sequence w, z the code word w,,so that condition (4.4.8) is satisfied. Then by (4.4.9)we may without loss of generality assume that the error pattern z is corrected no matter which code word is sent. In other words, without changing the probability of error we may adjust the minimum-distance decoder so that a given error pattern is either always corrected or never corrected. From now on, we assume that the minimum-distance decoder always has this property; thus the statement "the error pattern z is corrected" is unambiguous.

+

102

INFORMATION THEORY

Proof. First observe that if v,, v,, and v, are binary sequences of the same length, then d(vl v,, v, v,) = d(v,, v,). For example,

+

+

d(0111, 1010) = 3 and d(0111

+ 1100, 1010 + 1100) = d(1011,0110) = 3.

We therefore have d(z

+ W, w') = d(z + w + w, - w, w' + w, - w) = d(z + w,,w' + w, - w) 2 d(z + w,, w,)

(by4.4.8)

Another consequence of Lemma 4.4.5 is that if we wish to correct an error pattern z,, we cannot correct any other error pattern z, in the coset z, S. For if z, = z, w where w E S, w # 0, then the transmission of w together with the error pattern z, will yield the same received sequence as the transmission of the zero vector together with error pattern z,. Thus it is impossible to correct both z, and z, for all possible transmitted sequences. Hence exactly one sequence in each coset can serve as a correctible error pattern. Since the probability of an error pattern increases as the number of digits involved in the error decreases, it appears that among the sequences in a given coset, we should choose as the correctible error pattern that sequence with the fewest number of ones. This result is established by the following theorem.

+

+

Theorem 4.4.6. To construct the minimum-distance decoding scheme for a group code S, it is sufficient to choose as the correctible error pattern for each coset a sequence with minimum weight, that is, minimum number of ones. If z is such a sequence, then the received sequence z w (w 4 S) is decoded as w. Proof. If z is a sequence of minimum weight in the coset z S,then d(z, w) = d(z - w, w - w) = d(z - w, 0). Since z has minimum weight among all sequences z w, w r S, we have d(z - w, 0) 2 d(z, 0). Hence the sequence z is at least as close to zero as to any other code word, so that in the minimum-distance decoder we may correct the error pattern z when 0 is transmitted, and hence by Lemma 4.4.5 we may correct z for all possible transmitted sequences. The theorem is proved.

+ +

+

It follows from Theorem 4.4.6 that the minimum-distance decoder may be constructed in the following way. Write down a table of cosets (such as Table 4.4.2); choose a sequence z of minimum weight in the coset z S; rearrange the table so that the

+

103

ERROR CORRECTING CODES

+

+

.

+

S are z 0, z w,, . . , entries in the row corresponding to z z w,r-,, where the w, are the code words. (Thus z is the leftmost element of its coset, or the so-called "coset leader.") If a received sequence v is in the column of the coset table headed by w,, then v = w, z for some correctible error pattern z ; hence v is decoded as w,. The minimumdistance decoding scheme for the code defined by (4.4.7) and Table 4.4.2 is shown in Table 4.4.3. Notice that row 3 has two sequences of weight one, namely 0001 and 0100. Either sequence can be chosen as the coset leader.

+

+

Table 4.4.3. Minimum-distance decoding scheme for a group code

We now show that there is a one-to-one correspondence between the 2n-k correctible error patterns of a group code and the 2"-% possible values of the corrector. Theorem 4.4.7. All sequences in the same coset of a group code Shave the same corrector. Two sequences in different cosets have different correctors. Proof. If z , and z , are in the same coset, then z, = z, w for some w E S. Thus AzZT= AzlT SO that zl and z , have the same corrector. On the other hand, if z, and z, have the same corrector then AzIT = A z , ~ or A(z, - z , ) = ~ 0. But then z, - z, satisfies the parity check equations, hence z, - z , is a code word w. Therefore z, and z, = z , w are in the same coset.

+

+

*

' ,

>

Finally, we prove the main theorem of this section.

Proof of Theorem 4.4.1. Let c be the corrector associated with a given received sequence v; let v belong to the coset z, S where z, is a sequence of minimum weight in z, S. By Theorem 4.4.7, the sequences S are precisely the sequences whose corrector is c. It of the coset z, follows that among all sequences z such that AzT = C, zOis a sequence of minimum weight. If v = z, w , then, by Theorem 4.4.6, v is decoded as w = v - z,. This agrees exactly with the procedure given in Theorem 4.4.1.

+

+

+

+

104

INFORMATION THEORY

We conclude this section with an example. Consider the code whose parity check matrix is 1 0 0 0 1 1 -

A=

0 1 0 0 1 0 0 0 1 0 1 1 -0

0 0 1 0 1-

The minimum-distance decoding table is shown in Table 4.4.4. Notice that all single errors are corrected, along with some (but not all) double errors and some triple errors. Table 4.4.4. Minimum-distance decoding table for a parity check code

Code Words 000000 111010 101101 010111

0000

110111 OOO111 011111 010011 010101 010110

loo0 0100 0010 OOO1 1110 101 1

11OOOO 001010 011101 100111 101oo0 010010 OOO101 1 1 1 1 1 1 110011 ll0lOl 110110 m 1 1 OOO110

1 100 1010 1001 0110 001 1 0101 1111

110100 001110 011001 loo011 11OOO1 001011 011100 100110

1101 0111

1OOOOO 0 1 m Single OOlOOO errors 000100 m 1 0

011010 101010 110010 111110 111OOO OOOOO1 111011

100100 Double loo010 errors 1 m 1 010100 01oo01

Triple errors

(Transposed) Correctors

011110 011OOO 011011 101110 101011

001101 111101 100101 101001 101111 101100

001001 001111 001100 111001 111100

To calculate the probability of error for a group code, we note that correct transmission will occur if and only if the error pattern is a coset leader and hence correctible. Thus if Niis the number of coset leaders of weight i, or equivalently the number of i-tuple errors corrected by the code, then the probability of correct transmission is

105

ERROR CORRECTING CODES

For the code of Table 4.4.4,

+

+

~ ( e '= ) (1 - BIB 68(1 - B)5 7B2(1 - p), + 2 p ( 1 - 813. Note that the probability of correct transmission p(el I w) is the same for each transmitted code word w ; hence for a parity check code the average and maximum probability of error coincide. 4.5.

Upper and lower bounds on the error correcting ability of parity check codes

In this section we investigate the relation between the number of code words in a parity check code and its error correcting ability. As we have seen previously, an increase in the number of words in a code will in general be accompanied by a reduction in the number of errors corrected. We will try to obtain quantitative results to this effect. We first propose the following problem. We wish to construct a parity check code that has certain error correcting properties; for example, we might want a code that corrects e-tuple (and all smaller) errors. How do we choose the parity check matrix so that the resulting code has the desired properties? T o begin with, let us consider the construction of a single-error correcting code. Since the corrector associated with an error in a single digit r, is the jth column of the parity check matrix, we need only choose n distinct (nonzero) columns; the resulting code must be single-error correcting, since no two distinct single errors can result in the same corrector. (Also, since no column is zero, a single error cannot be confused with the "no error" condition.) For example, if n = 7, k = 4, the code defined by the following parity check equations will correct single errors.

PI-1

r,, r,, r, are check digits. r,, r,, r,, r, are information digits. Lr7-I

Note that if the parity check matrix A is m by n, we may as well assume that the rank of A is m. For if the rank is r < m, then m t rows are linearly dependent on the remaining t rows and may be eliminated. We may also assume for convenience that the m by m submatrix in the left-hand corner of A is nonsingular; in fact we may assume without loss of generality that the submatrix is an identity matrix. If the submatrix is not originally an identity matrix the equations may be diagonalized by elementary row

-

106 transformations. (4.3.2) :

INFORMATION THEORY

For example, consider the parity check equations

=O +r,=O rl r , = 0. Let us add equation 1 to equation 3, obtaining r1

+r4+r,

rz

+ r , + r, + r , + r4

r

+r +r

r1

+

+r,+r,

=O

+ r, = 0

+ +

r, r, = 0. r, Now let us add equation 3 to equation 2, obtaining

r,

+ r4 + r , + r4 4- r,

=0

=0 r, r, = 0. r, The submatrix in the lower left-hand corner is now an identity matrix, and the new equations have exactly the same solution as the original equations; in other words, both sets of equations define the same code. The diagonalization process will be successful as long as the original submatrix is nonsingular. The general relation between the parity check matrix and the error correcting ability of the code is contained in the following theorem.

r,

+ +

Theorem 4.5.1. The parity check code defined by the matrix A will correct e-tuple (and all smaller) errors if and only if every set of 2e columns of A is linearly independent. ProoJ By Theorem 4.4.1, e-tuple and all smaller errors will be corrected if and only if all error patterns of weight < e yield distinct correctors, that is, if and only if no linear combination of e or fewer columns of A equals another such linear combination. But this is precisely the condition that each set of 2e columns of A be linearly independent. As a n example, the following parity check matrix describes a doubleerror correcting code with 7 check digits and 3 information digits. 1 0 0 0 0 0 0 1 0 1 -

0 1 0 0 0 0 0 0 0 1

ERROR CORRECTING CODES

107

It may be verified that every set of 4 columns of the matrix (4.5.1) is linearly indewdent; However, we may find triple errors which are not correctible. For example, -

c(rl)

+ c(r& + c(rJ =

= c(r3)

+ c(r4) + c(rg)

-

[~(r,)=jth column of A]. Thus a triple error in r,, r,, and r, will be confused with a triple error in r8, rl, and r,. Next we shall consider the following problem. Suppose we wish to construct a parity check code that corrects e-tuple and all smaller errors, with a fixed code-word length n. We ask how many check digits are necessary to construct the code. We wish to use as few check digits as possible, since a small number of check digits corresponds to a large number of code words. In general, we cannot determine the minimum number of check digits exactly, but we can establish some useful lower and upper bounds, as follows. Theorem 4.5.2. (Hamming Lower Bound on the Number of Check Digits). The number of check digits in an e-tuple-error correcting parity check code must satisfy i--0 \

C

I

where n = code word length m = number of check digits = n - k. Proof. In order that a parity check code be capable of correcting e-tuple and all smaller errors, each error combination involving e or fewer digits must result in a distinct corrector. The number of possible i-tuple errors in an n-digit code word is equal to the number of ways of choosing i digits out of n, namely

(7) .

The number of possible corrector

values is 2"; hence in order that there be a unique corrector for each error combination, we must have

108

INFORMATION THEORY

which proves (4.5.2). (The term "1" appears on the right since if no error is made, the associated corrector must be the zero vector.) The Hamming lower bound on the number of check digits is in fact identical to the Hamming upper bound on the number of code words, which we proved for general binary codes in Section 4.2. To see this, note that the number of code words is 2k = 2n-m = 2n/2m. Hence (4.5.2) is equivalent to *9I

which is (4.2.1) for the special case of a parity check code. The Hamming lower bound is necessary but not suficient for the constructibility of an e-tuple-error correcting parity check code. In other words, if m, is the smallest positive integer satisfying (4.5.2) for a given n and e, there may not be a parity check code using mo check digits. For example, if n = 10, e = 2, we find from (4.5.2) that m, = 6. However, there is no double-error correcting parity check code with fewer than seven check digits, as the reader may verify by trial and error. An upper bound on the number of check digits required for the construction of a code is provided by the following theorem. Theorem 4.5.3. (Varsharmov-Gilbert-Sacks Condition). An e-tupleerror correcting parity check code with words of length n may be constructed if the number of check digits m satisfies

The condition (4.5.3) is suficient but not necessary for constructibility of a code. In other words, if for a fixed n and e, mo is the smallest positive integer satisfying (4.5.3), then it is guaranteed that an e-tuple-error correcting parity check code with mo check digits exists. However, there may be an e-tuple-error correcting code with fewer check digits. For example, if n = 10 and e = 2, then (4.5.3) yields m, = 8. However, it is possible to construct a double-error correcting code with seven check digits. [See the parity check matrix (4.5. I).] Proof. The proof is accomplished by giving a synthesis procedure for the required code. We shall choose the columns c(r,), c(r,), . . . , c(r,) of the parity check matrix successively. By Theorem 4.5.1 we must satisfy the condition that every set of 2e columns be linearly independent. First we choose c(r,) arbitrarily. [Of course we must have c(r,) # 0 since the zero vector must correspond to perfect transmission.] We now

109

ERROR CORRECTING CODES

choose c(r2) such that c(r,) # 0 and c(r,) # c(r,). provided 2" > 2. We now choose c(ra) such that

This can be done

+

c(r3 # 0 , c(rg)# ~ ( r l ) , c(rJ # c ( ~ z ) , c(r3 f ~ ( r l ) ~ ( r z ) , thus guaranteeing that c(rl), c(r2)and c ( r 3 will be linearly independent. A choice for c(r3) can be made if 2" > 4. Let us digress for a moment to consider the special case e = 2. Then every set of four columns must be linearly independent. Thus we choose c(r3 such that c(r$ is not equal to any of the following:

+

0 , c ( r A c(rZ), c(r3), c(r1) c ( ~ z ) , 4r1) c(ra), c(r2) c(r31, c(r1) c(r3 c(f-3). Such a choice can be made if 2" > 8. We now choose c(r,) such that

+

+

+

+

+

+

+

c(rJ c(r,) c(r$ since we do not Note that c(r,) can equal c(r,) require that every set of five columns be linearly independent. Therefore c(r,) can be chosen if

Now let us return to the general case. Suppose we have determined c(rl),c(r2),. . . , c(rnPl). We then choose c(rn) such that

~ ( r n# ) ~ ( r , )# c(rn) # ~ ( r , )#

0 ~ ( r , ) , i = 1 , 2, . . . ,n - l i , j = I , 2, . . . ,n - 1, i # j c(ri) c(r5), i,j = 1, 2, . . . ,n - 1, i,j, k distinct ~ ( r , ) c(r5) c(rk),

+ +

+

+

+

+

ck,) # ~ ( r i , ) ~ ( r i , ) ' ' ' ~ ( r i ~ , - ~ ) , i,, i2, . . . , iZc-, = 1, 2, . . . ,n - 1, i,, i,,

. . . ,i2,-, distinct

At worst, all of the above combinations are distinct.? The number of

t It is not necessary that all the combinations be distinct; for example, in the parity check matrix (4.5.1) we have 2e = 4 but several combinations involving three or fewer c(r,); c(r,) + columns are equal, for example c(r,) c(r,) c(r,) = c(rJ c(r,) c(rd = c(rd

+ c(rJ + c(r1d.

+

+

+

+

110

INFORMATION THEORY

combinations is

hence the theorem. In general the smallest value of m (say m,) satisfying the VarsharmovGilbert-Sacks condition exceeds the minimum value of m (say m,) satisfying the Hamming lower bound. However, it was proved by Sacks (1958) that m,/m, 5 2. It may also be seen from (4.5.2) and (4.5.3) that the upper and lower bounds coincide when e = 1. In other words, a single-error correcting code is constructible if and only if 2" ;1: 1 n. (This fact may be seen directly without reference t o Theorem 4.5.3, since for single-error correction we merely need n distinct nonzero columns.) The bounds of this section say nothing about the probability of error of a code, which is a more basic quantity than the number of errors corrected. However, the results will be useful in Section 4.7, where we give precise bounds on the probability of error for general binary codes.

+

4.6.

Parity check codes are adequate

In this section we show that parity check codes meet the specifications of the fundamental theorem. More precisely, we will prove the following result. Theorem 4.6.1. Given a binary symmetric channel with capacity C > 0 and a positive number R < C, there is a sequence of parity check codes (s,, n,, A,), i = 1 , 2 , . . . , where n, < n, < . . . , s, = 2", R 5 k a. / nI. < C, and lim li= 0. i-

co

Thus for each i we can construct a parity check code with at least 2"iR words and a (maximum) probability of error liwhich approaches zero as i -t oo. Hence, even if we are restricted to the class of parity check codes, we can still maintain the transmission rate R < C with an arbitrarily small probability of error. Proof. First observe that given a parity check code with 2k words, we may find a set of generators for the code, that is, a set consisting of k code words (say) w,, . . . , w,, such that each word of the code can be written as a modulo 2 linear combination of w,, . . . , w,. (See the proof of Theorem 4.4.3.) The idea of the proof is to form a parity check code by selecting generators at random, and then to show that the "randomcoding" proof of Section 3.5 goes through almost word for word. In particular we consider the following random experiment:

111

ERROR CORRECTING CODES

1. Fix R' such that R < R' < C. Let k and n be positive integers such that R 5 kln R'. Independently select binary sequences w,, w,, . . . , w, of length n, choosing the components of each sequence independently with 0 and 1 equally likely. 2. Form a code with s = 2'" wordst w,,j = 0, 1 , . . . ,2' - 1, by taking all possible modulo 2 linear combinations of w,, . . . , w,; in other words, set w, = b,,w, . . b,,wk (mod 2), where b,, = 0 o r 1, i = 1,2, . . . , k. (Let w, be the zero vector, corresponding to bi, = 0 for all i.)

<

+ -+

Now if a,, is the tth component of w,, then

Since the components of the words w,, . . . ,w, are chosen independently, the a,,, t = 1,2, . . . , n, are independent. Furthermore P{a,, = 0) = P{a,, = 1) = 112 for all t, provided j # 0. T o see this, we observe that there are as many sequences (a,,, . . . , a,,) that produce a,, = 0 in (4.6.1) as there are sequences that produce a,, = 1. For i f j # 0,at least b,,a,, = 0,then define a,,* = one coefficient b,, is not zero. If a,,, m # r ; a,,* = 1 a,,. This yields b,,a,,* = 1. In this way we establish a one-to-one correspondence between sequences that produce a,, = 0 in (4.6.1) and sequences that produce a,, = 1. Thus the components of the words w,, j # 0, are (statistically) independent and have the distribution that achieves channel capacity (that is, 0 and 1 are equally probable). By a similar argument, we may show that the words w, and w,., j, j' # 0, j # j', are independent. T o accomplish this, we will prove that

+

P{atj = y , atj. = y ' ) = 114 for

ym=,

Ym=,

y , y' = O , l ,

t = 1, 2,

. - . n.

(4.6.2)

t Although the words w,, . . . ,w, chosen in step 1 are statistically independent, a particular choice may result in w,,. . . , wk being linearly dependent. In this case, the words w,,j = 0,1, . . . , 2k - 1, will not be distinct, and we obtain a code which strictly speaking is not a parity check code as defined in Section 4.3. If, however, the number of distinct words of the code is 2', r < k, and w,', . . . ,w,' is a set of r linearly independent words, we may select sequences w:,,, . . . , wk' such that w,', . . . ,w,' are linearly independent, and in this way construct a parity check code with 2kdistinct words. It can be shown (see Problem 4.1 1) that the average probability of error of the new code cannot exceed the average probability of error of the original code. Hence if the random experiment described above yields a code with average probability of error E , we may always find a parity check code with the same number of words and an average probability of error 5 E . We note also that in this proof and elsewhere, "independent" will mean "statistically independent"; the term "linearly independent" will not be abbreviated.

112

INFORMATION THEORY

It then follows thatP{a,, = y, a,,, = Y ' ) = P{a,, = y ) P{a,,. = y'); hencc a,, and a,,, are independent for each t , which implies the independence of w , and w,,. To prove (4.6.2),we note that sincej # j', we have b,, # b,,, for some r. Say b,, = 1 and b,,, = 0 . Then given any sequence (a,,, . . . ,a,,) such that

we define a,,* = a,,, m # r ; a,,* = 1

+ a,,.

It follows that

This establishes a one-to-one correspondence between sequences that produce a,, = a,,, = 0 in (4.6.1) and sequences that produce a,, = 1, a,,. = 0. Thus

P(a,, = 0 , a,,, = 0 ) = P{a,, = 1 , a,,. = 0 ) .

(4.6.3)

But

so that both terms of (4.6.3) are 114. In a similar fashion, we establish the other relations of (4.6.2). Note that although the w, arepairwise independent, it is not true that the entire set (w,, w,, . . . , w,-,) is independent. However, w,, . . . , w, do form an independent set. Now let us try to reproduce the "random-coding" proof of Section 3.5. The role of the code words x"), . . . , x(" in that proof is played by w,, . . . ,w,-,. The constant a in the definition of the set A is taken as +n(Rf C). If x is any binary sequence of length n, we have shown that P{wj = x) = (1/2)",which is the probability determined by selecting independently the components of the input sequence, each component chosen with the distribution that achieves capacity. The random-coding proof may be taken over word for word from Section 3.5, except for two difficulties. In the original proof, the code words were independent. However, as we observed at the time [see the discussion preceding (3.5.1I ) ] , only pairwise independence is required. A slight difficulty is also caused by our leaving out the zero code word. Let S, be the code consisting of the words w,, . . . , w,-,, with the decoding sets B,, 1 5 i 5 s - 1 , determined by the minimum-distance decoding scheme. Let El be the average probability of error of S,. Let S, be the code formed by adding the zero word to S,. However, instead of using the minimum-distance decoder for S,, take the decoding set of w, to be B,

+

ERROR CORRECTING CODES

113

when i # 0, and take the decoding set of w, to be the empty set (so that an error is always made when wois transmitted). The average probability of error of S2 is then

Finally, let S2* be the code S , with the minimum-distance decoder, and let e * be the average probability of error of S,*. Since the minimumdistance decoder corresponds to the ideal observer with all words equally likely, E * j E,. The random-coding proof shows that EIEl] -0 as n+ oo. Since s = 2k 2 2nR-+ co as n -+ CO, E[E*]+ 0 as n -+ oo. Since R < C, we can find integers ki and ni, i = 1,2, . . . , such that n, < n, < . . . and R j kilni < R' < C . Thus the above procedure yields codes that maintain the transmission rate R and have an arbitrarily small probability of error. The theorem is proved. 4.7.

Precise error bounds for general binary codes

In Chapter 3 we showed that we could construct a sequence of codes ([2nR], n, A,) with An 40 at an exponential rate. In this section we obtain more precise information about the rate of approach of An to zero. We shall need a few combinatorial results. Lemma 4.7.1.

I f 0
< 1 , q = 1 - p, and np is an integer, then

Proof. We use Stirling'sformula:

If no terms of the exponential series are taken, n ! is underestimated; if one term is taken, n! is overestimated, etc. To establish the left-hand inequality, we write

(2

n!

= (np)! (nq)!

2

J2?rn

nne-" exp (-1/12np - 1/12nq) J2nnp (np)"ePUJ2nnq (nq)nae-na '

-

[We have underestimated n ! and overestimated (np)!and (nq)!.] Assume that np 2 1 and nq 2 3. [By the symmetry in p and q of the expression (4.7. l), the only remaining cases to consider are np = 1, nq = 1 ; np = 1 ,

114

INFORMATION THEORY

nq = 2 ; np = 2, nq = 2 ; for these cases, the inequality may be verified by direct substitution.] Then

and

Hence

which proves the left-hand inequality. To establish the right-hand side, we overestimate n ! and underestimate (np)! and (nq)!,as follows:

Assume p 2 q ; then 1 / 3 6 0 ( n ~5 ) ~ 1/360np since np 1/360(nq)3 F; 1/360nq. Thus

2

1;

similarly

1 1 1 < (since p 2 q ) - ++-360nq 5 0. 12nq 180nq Thus

Since the upper bound is symmetrical in p and q, the restriction p may be dropped; this completes the proof.

2q

Lemma 4.7.2. Let Y , ( n , A) be the "tail" of a binomial distribution (with parameters n and p) from In to n, that is,

ERROR CORRECTING CODES

where An is an integer and 0 < p

< 1 < 1.

Then

-

[8n1(1 - A)]-% 2-nB(L,")5 Y J n , 1) I;2-nB(1pu) where B(1, p) = -H(1, 1 - 1 ) - 1 log p - (1 - 1 ) log q 1 1-1 = 1 log ( 1 - 1 ) log P 4 It follows that

(4.7.2)

+

lim n+m

- -1 log Y u ( n ,1,)

= B(1, p)

il

provided n l n is an integer and 1, -1. obtain

In particular if p = q = $ we

so that

(

A L$.(;)])

lim - - log n-r w

= H(1, 1

- A)

provided

1.

-r

1. (4.7.5)

Proof. Let X be a binomially distributed random variable with parameters n and p. Since X may be regarded as the number of successes in a sequence of n Bernoulli trials, X may be represented as Xl X, . Xn where X I , . . . , Xn are independent. Xi, interpreted as the number of successes on trial i, is 1 with probability p and 0 with probability q. Thus

+ +. +

In (4.7.6), r is any positive real number. We may try to find an appropriate value of r in (4.7.6) by differentiating the exponent:

Expression (4.7.7) becomes zero if r is chosen so that

116

INFORMATION THEORY

If A > p, then Aq > ( 1 - A)p so that a positive r satisfying (4.7.8) can be found. With this value of r, (4.7.6) becomes

Y y ( n ,A) = P { X

2 nil}

19 < e r p [ - n ~loge (1 + n log. -

+

(9

+

A)] 9

Y,(n, A) I exp, { [ - A log 1 - A logq jl log ( 1 - A) A log p logq - log (1 - A)]n}. Thus lp (,, jl) I 2nR(A,-"2n"og v2n(l-" 108 a

+

+

9

which establishes the right side of (4.7.2). To prove the left side, note that

The result then follows immediately from Lemma 4.7.1; thus (4.7.2) is proved. Since (4.7.3), (4.7.4), and (4.7.5) follow from (4.7.2), the proof is finished. It may be verified from the definition of B(A, p) that

B(1, P ) = H(P, 1 - p)

+ (1 - p) dP-d H(p, 1 - p) -H(A,

1 - A).

(4.7.9)

Thus B(A,p,) is the difference between the tangent

to H@,l - p ) at the point po and the value of H at the point A. (See Fig. 4.7.1.) Now consider a binary symmetric channel with probability of error p < 112. Given R < C = 1 - H(B, 1 - B), we know that there exist codes ([2nR], n, E,) with E , -+ 0 as n -+ oo. We will try to obtain precise information about the rate of approach of E , to zero. Let us define a parameter A ( p 5 A 5 112) by setting

R=1

- H(A, 1 - A) = 1 + A log A + (1 - A) log (1 - A). (4.7.10)

117

ERROR CORRECTING CODES

Then as R increases from 0 to C, 1 decreases from 112 to B. Define the critical rate R, as 1 - H(I,, 1 - A,), where

or equivalently

We have the following theorem.

0

PO

Fig. 4.7.1.

x

%

P

Interpretation of B(&p).

Theorem 4.7.3. Let E,* be the lowest average probability of error among all codes ([2nR],n) for the binary symmetric channel (where R is a fixed positive number). If R, 5 R < C, then a = lim [-(lln) log &,*I

-

n-

m

exists and equals B(1, B) where 1 is determined from R by (4.7.10). Thus for large n, E,* 2-"" where a = B(1, 8). Proof. We first establish an upper bound on E,*. As in the "randomcoding" proof of Section 3.5, we choose code words w,, . . . , w,, s = [2""], independently and with P{w = x} = (112)" for all code words w and all binary n-sequences x. We then calculate the average probability of error E for the code selected, using the ideal observer with all words equally likely, that is, minimum-distance decoding.

118

INFORMATION THEORY

Suppose that the word wi of the code chosen above is transmitted through the channel. Let us calculate the probability p*(e wi, x, y ; t) I of a decoding error given that

I

a. wi is transmitted b. w, = x c. y is received, where dist ( x , y) = t. An error in decoding implies that at least one code word w,, j # i, is at least as close to the received sequence as wi. Thus

el wi, x, y ; t ) 5 P{dist (w,, y) 5 t for at least one j # i ) 5 j2 P{dist (wi, Y ) 5 t } . #i Now the probability that w, and y agree in any specific digit is 112. If t = np wherep < 112, then P{dist (w,, y)

< t } = 2 (") ($ k k=O

(by Lemma 4.7.2). Thus p*(e)wi, x , y; t ) 5 2nR2-n[1-H('J-u)1 if

t = np, p

< 112. (4.7.1 1)

I

Clearly p*(e wi, x , y ; t ) 5 1 under any circumstances, in particular if t = np where p 2 112. As in the random-coding proof of Section 3.5 we are looking for E [ E , wi = X I , where ci is the probability of error when wi is transmitted. We have [see (3.5.9)]

I

where N is the number of errors in transmission, and

=1

if t = np, p 2 112.

We break (4.7.12) into three parts, first summing from 0 to [np], then from [np 11 to [nil]and finally from [nA I ] to n. [Recall that il and R

+

+

119

ERROR CORRECTING CODES

are related by (4.7.10)]. First we have, using (4.7.12) and the fact that H(P, 1 -P) I H(B, 1 - B ) f o r O I p I B < 112,

j 2-n[H(U-"-H(B.l-P)1

(by 4.7.10).

(4.7.13)

For the second contribution to (4.7.12) we have

By Lemma 4.7.2, P{N

2 t)

= Yp(n, p) j

2-nR(p,p) if

t = np, p

> B.

(4.7.15)

Also, by (4.7.12), h(t) I 2nR2-n[l-H(v.l-")1

if t = np, p I 1 < 3 .

(4.7.16)

Using (4.7.10), (4.7.1 9, and (4.7.16), we find that (4.7.14) becomes

Finally we estimate the third contribution to (4.7.12): n

2

z n

P { N = t)h(t) j

P { N = t)

t-[nl+ll

l-[nA+l]

= P{N

2

[nl

+ 11) j 2-nB("p)

(by 4.7.15)

From (4.7.12), (4.7.13), (4.7.17), and (4.7.18) we obtain

Consider the term

(4.7.18)

120

INFORMATION THEORY

Differentiating with respect t o p , we obtain

-(

'

g f ( p )= 0

when -- -) l i o r p = ~ , 1 - p 1-p

<0 g'(p) > 0

for p

g'(p)

for p

< 1, > A,

We have the following two possibilities. CASE1. R 2 R,, hence 1 5 A,. Then g ( p ) is a decreasing function ofp for 0 5 p _< A. Consequently the largest term in (4.7.19) is the last. CASE2 . R < R,, hence il > A,. Since g(p) is a minimum whenp = A,, we may overbound (4.7.19) by replacing p by A, in all terms. Since the n/2 for large enough n, we have: number of terms is 5 [nil] 1 n ~f R 2 R,, then E [ E ,( W ,= X ] I - 2-nB(rl'B'. 2 ~f R < R,, then

+

We need only the first case in the present proof. Since the bound of (4.7.20) does not depend on i or x,

We now establish a lower bound on E,*. The average probability of correct transmission of a code S with words w,, . . . , w , is

where z is the error pattern vector and the z i i , j = 1 , 2 , . . . , k,, are the distinct error patterns that are corrected when wi is transmitted. The number k , of error patterns corrected when w , is transmitted is the number of elements in the decoding set associated with wi. Since the decoding sets are disjoint and together exhaust the entire space of 2"' possible received sequences, it follows that 2;=, ki = 2". (Note that although zi,, . . . , zikt are distinct, z,,, may equal z , , ~for i # j.) Since

121

ERROR CORRECTING CODES

the probability associated with a given error pattern increases as the weight of the pattern decreases, it follows that E cannot be smaller than the number obtained when the z,, have minimum weight in (4.7.22). T o minimize the weights of the z,, subject to the requirements that k, = 2" and that z,,, . . . , z,,, be distinct, we can do no better than to correct, for each word wi,all error patterns of weight 0, 1, . . . , t - 2, where t - 2 is the largest

z;=,

integer satisfying the Hamming bound

2::;

(n) q 5

heor or em

4.2.2),

and to complete the decoding sets by correcting (t - 1)-tuple errors for some (but not all) of the code words. The average probability of error of this (possibly hypothetical) code S' cannot exceed the original figure 2; this provides a lower bound on E,*. If N is the number of errors made in transmission, then N = t implies a decoding error in S ' ; hence

Now by Lemma 4.7.1, with np, = t - 2,

+

+

Thus H(p,, 1 - p,) j H(1, 1 - A) (2n)-l log 2n n-l. It follows that for large enough n and any fixed 6 > 0,we have pl < A 6, hence t

p =- jA n

+6

for

n sufficiently large.

+

(4.7.24)

By (4.7.23) and Lemma 4.7.1, En

*2

[gnp(l - p)l-M2n~(r+-p)p"p(1 - B)"('-V) = [8np(l - p)]- %2-nB(zl.#) 2 (2n)-x2-nB(~.B). (4.7.25)

Thus by (4.7.24) and (4.7.25),

where 6 > 0 is arbitrary and n is sufficiently large. The theorem now follows from (4.7.21) and (4.7.26). Theorem 4.7.3 gives very precise information about the asymptotic behavior of binary codes with [2nR]words of length n as long as R is not

122

INFORMATION THEORY

below the critical rate R,. For values of R below R,, the asymptotic behavior of -n-l log E,* is not known exactly, although it may be estimated from above and below. We define a * = lim sup (-n-'

log E,*)

n+m

a* = lim inf (- n-' log E , * ) . n'm

From (4.7.26) we have

a* and from (4.7.21) we have, for R a,

I B(1, B)

< R,,

2 H(I, 1 - 1) + B(A,, #I -)H(A,.

-

1 - A,).

(4.7.28)

2-an where a lies between the limits (4.7.27) and Thus for large n, E,* (4.7.28). Both the upper and lower bounds above can be improved. Wc will prove one result concerning the lower bound. Theorem 4.7.4. (4.7.10.), by

a, 2 1B(4, 8) - R, where 1and R are related, as i n

R

=1

- H(1, 1 - A).

Proof. We make use of the Varsharmov-Gilbert-Sacks condition (Theorem 4.5.3). We know that there exists a code with 2k words of length n correcting e-tuple and all smaller errors, provided

Now by Lemma 4.7.2,

+

[Since (2e - l)/(n - 1) < (2e l)/n, the second requirement implies the first.] Hence, setting k = nR in the Varsharmov-Gilbert-Sacks condition, we find that there is an e-tuple-error correcting code with 2nH words of length n, if

+

1 = In In particular, since 1 - R = H(1, 1 - 1) we may take 2e (thus (2e l)/n and the application of Lemma 4.7.2 above is legitimate).

+

<+

123

ERROR CORRECTING CODES

We may bound the average probability of error% of such a code as follows. If w, is transmitted then the occurrence of an error implies that the received sequence is at least as close to one of the words w,,j # i, as to w,. Hence received sequence is at least as close to w, as to wi

I

I

he

B

I

I

%

A:

Fig. 4.7.2. Error bounds for the binary symmetric channel; en* in the shaded area.

-

2-anwhere a lies

Suppose that w iand w, differ in u positions. By Lemma 4.2.1, for a code to correct e-tuple and all smaller errors, the minimum distance between code words must be at least 2e 1; therefore u 2 2e 1 = In. Now

+

+

I

P{received sequence is at least as close to w jas to w i w itransmitted) = P{at least half of the u digits in which w, and w, differ are in error) =

Consequently p(e)j 2-n["(xJ)-R1

and the theorem is proved.

124

INFORMATION THEORY

Certain quantities (such as i u ) in the above argument have been treated as integers in order to avoid laborious details; a completely precise analysis would require the use of the greatest integer function. The bound of Theorem 4.7.4 may be improved to a , 2 1B(&,@)as was shown by Gallager (1963). The proof in the text is due to Wyner (1964), who also proved that a* 1 21(1 - 1)B(&,B). The bounds described in this section are shown in Fig. 4.7.2. 4.8.

The strong converse for the binary symmetric channel

In this section we prove, for the special case of the binary symmetric channel, that if the transmission rate is maintained above channel capacity, not only is the probability of error bounded away from zero, as we have established in Section 3.7, but in fact the probability of error must approach one with increasing code-word length. Specifically, we will prove the following result. Theorem 4.8.1. Consider a binary symmetric channel with capacity C. If E is any fixed positive number, and 1is any fixed nonnegative number less than one, then for sufficiently large n, any code (s, n, 1) must satisfy

< 2n(C+e) In particular, given any sequence of codes ([2nR],n, 1,) with R sufficiently large n we must have 1, > 1 ; in other words,

> C, for

Proof. The idea of the proof is to estimate the number of sequences in the decoding sets of the code. There must be enough sequences in each decoding set to make the probability of correct transmission at least 1 - 1for each possible transmitted code word; on the other hand, the number of sequences is limited by the fact that there are only 2" binary sequences of length n. Given a code (s, n, 1) with 0 1 1< 1, let r be the smallest integer such that

It follows that each decoding set has more than

ill

1;::

sequences.

For suppose that the decoding set B associated with the code word w has fewer sequences. Since the probability of an error pattern increases as its weight decreases, the largest possible probability of correct transmission corresponds to the correction of errors of the smallest possible

125

ERROR CORRECTING CODES

weight. Now if B in fact had

2::;

(2

sequences, p(e ( w) would be

minimized if the correctible error corresponding to the points of B were the zero pattern together with all single, double, . . . , (r - 1)tuple errors. In this case the probability of correct transmission when w is transmitted would be

Since the number of sequences of B is at most

I

p(e' w) < 1 - 1 in any event, contradicting the fact that we have assumed a code with maximum probability of error I. Thus we have a lower bound on the size of each decoding set. However, since there are s decoding sets and 2" sequences altogether, we have

Let N be the number of errors in transmission. Then

by definition of r. Now N / n converges in probability to p as n -+ co by the weak law of large numbers, so for any fixed 6 > 0,P { N 5 n(p - 6 ) ) 4 0 as n co. In particular, P { N 5 n(P - 6 ) ) < 1 - I for sufficiently large n (remember that A < 1). Thus for n sufficiently large we have

From (4.8.2) and (4.8.3) we have

We may write [n(p - 6 ) - 1 1 = n(p - 6') where 6' Lemma 4.7.2 then yields

+

6 as n -+ co.

Since C = 1 - H(p, 1 - p) and 6' may be taken arbitrarily small, the strong converse is proved.

126

INFORMATION THEORY

We have shown that if the transmission rate is maintained above capacity, the maximum probability of error approaches 1. In fact, the average probability of error also approaches 1. (See Problem 4.10.) 4.9. Nonbinary coding

Most of the results in this chapter can be extended to more general situations. In this section we present some of the extensions without proof. In many cases, the proofs are very close to the arguments given in the binary case; where this is not so, we give a reference. For a given positive integer q, the q-ary symmetric channel is a discrete memoryless channel with input and output alphabet (0, 1, . . . , q - 1) and channel probabilities p(i i) = 1 - /3, p ( j ( i) = /3/(q - 1) for i # j. Thus the probability of receiving digit j given that digit i is transmitted is the same for all j # i. As in the binary case, define the distance between two n-sequences as the number of digits in which the two sequences disagree. Then the ideal observer with equally likely inputs is a minimumdistance decoder, as in Theorem 4.1.1. Lemma 4.2.1 goes through word for word, but Theorem 4.2.2 becomes

I

If q is a prime or a power of a prime, then the elements 0, 1, . . . , q - 1 form a finite field under appropriate definitions of addition and multiplication. Finite fields will be discussed in Chapter 5 (see page 142); for a more complete treatment see Peterson (1961) or Albert (1956). A parity check code for the q-ary symmetric channel may be defined as a set of solutions to a system of homogeneous linear equations over the field with q elements. The discussion of Section 4.3 applies to the more general situation. Theorem 4.4.1 carries over word for word. The discussion of Section 4.4 must be modified. A parity check code forms a vector space? over the field with q elements; conversely, any set of

t A vector space V over a field F is an abelian group together with an operation of "scalar multiplication," that is, a function which assigns to each pair ( I , v), I E F, v c V, an element I v E V, such that for all elements I , p c F and all elements v, w E V we have I(v w) = I v + I w , ( I p)v = I v pv, (Ip)v = I(,ILv), and 1 . v = v, where 1 is the multiplicative identity of F. Now if F i s a finite field withq elements, where q is a prime, then the elements of F a r e the integers modulo q. If V is any abelian group consisting of sequences of length n whose components belong to F, where the group operation is componentwise modulo q addition, we may define I v = v v ... v (A times). With this definition, V becomes a vector space over F.

+

+

+

+ +

+

127

ERROR CORRECTING CODES

q-ary sequences that forms a vector space is a parity check code. If q is a prime, any abelian group of q-ary sequences is also a vector space over the field modulo q; thus only the group structure is needed when q = 2. With this modification, Section 4.4 survives essentially intact. Theorem 4.5.1 is unchanged, but Theorems 4.5.2 and 4.5.3 become, respectively : qm

i=O

q - 1 ) and q m >

(

2e-1

2

i=o

("

' ) ( q - 1)'

The proof of Theorem 4.6.1 parallels that of the binary case. Error bounds for nonbinary channels are significantly harder to find than those for binary channels. Blackwell, Breiman, and Thomasian (1959) proved that given any discrete memoryless channel with capacity C > 0 and given any R < C such that C - R 5 log, e, there is a code ([2""], n, E,) with n

< 2e-[(~-~)2/1eab(lo~2e)21n

where a is the number of input symbols and b the number of output symbols. Sharper bounds may be found in Fano (1961). 4.10. Notes and remarks

A general reference for error correcting codes is Peterson (1961). Single-error correcting parity check codes were introduced by Hamming (1950), who also pointed out the geometric interpretation of the binary coding problem. The general notion of a parity check code and the results of Section 4.4 are due to Slepian (1956). Theorem 4.5.3 is due to Varsharmov (1957) and Gilbert (1952); the argument in the text follows Sacks (1958). Theorem 4.6.1 was stated by Elias (1955). Lemma 4.7.2 is an application of the exponential bounding technique of Chernoff (1952). Theorem 4.7.3 and the bounds (4.7.21)aredue t o Elias (1955,1956); see also Wozencraft and Reiffen (1961). For a further discussion of error bounds on codes with low transmission rates, see Gallager (1963) and Wyner (1965). In particular, Wyner proves the following results, which provide a strengthening of the Hamming bound. Lemma 4.10.1. Let wi = (xil, xi2, . . ., xin), i = 1, 2, . . ., S, be code words of an arbitrary binary code. Let d be the minimum distance of the code, that is, d = min dist (wi, w,). Let y be any binary n-sequence and i#5

d, the distance between wi and y ( i

=

1, 2,

. . . ,s).

Then

128

INFORMATION THEORY

Proof. Without loss of generality we may assume y = 0. [If y # 0, consider a new code with words wi y (mod 2).] Since the minimum distance is d,

+

Now

with ordinary (not modulo 2) arithmetic. Thus

In the expansion of the inner summation, we have terms of the form xTk2 xjk2,j > r, and terms of the form xik2 xrk2,i < r ; thus each term xTk2appears s - 1 times. Hence

+

xik2; thus Since y = 0, we have di = 2,"=,

Now by the Schwarz inequality for sums,

Thus

Since xtk2 = xik, we have d, = 2;=:=, xi,, so that

+

129

ERROR CORRECTING CODES

On dividing through by n, the lemma follows. Note that w,, need not be the entire code.

. . . , w,

Now let M(n, d) be the maximum possible number of code words in a binary code with words of length n and minimum distance d. We have the following theorem.

Theorem4.10.2. I f d l n

< 4,then

41

- 4p), K(p) = p / J ~- 4p. where p = d/2n, t = (1/2p)(l Proof. Consider a code consisting of M(n, d ) words of length n, with minimum distance d. In the space of all 2" n-sequences, consider, for each code word wi, the sphere S, consisting of all sequences y such that d(w,, y) 5 [td/2]. Now within the sphere, assign to each sequence y whose distance from w, is r the mass p,(y) = td/2 - r. The total mass of each sphere is therefore

If a sequence y belongs to each of the spheres S,, i = 1 , 2 , . . . , s, then we assign a mass p(y) to y given by the sum of the masses assigned to y in each sphere; that is,

Consequently the total mass assigned to all sequences is

We shall bound M(n, d ) by finding a bound on p(y)/n; then (4.10.4) becomes

I * z d i = - std -b= 2n n i-1

-

1,p(y).

Let b = b(y) =

s t p - b where d, = d(wi, y).

(4.10.6)

Substituting (4.10.6) into (4.10.1) we obtain s2t2p2- 2stpb

+ b2 - s2tp + sb + s2p - sp < 0

or 0

5 b2 j S [ P - sp(t2p - t

+ 1) - b(l - 2tp)l.

(4.10.7)

130

INFORMATION THEORY

+

Since t2p - t 1 = 0 and 1 - 2tp satisfied only when

>0

for p

< 114,

(4.10.7) can be

Thus By (4.10.3) and (4.10.5) we have

as asserted. It is unfortunate that the synthesis procedure given in the proof of the Varsharmov-Gilbert-Sacks condition is not practical, for codes that satisfy the condition have desirable asymptotic properties. In particular, we may construct codes that correct e-tuple and all smaller errors, provided

where R = kln is the transmission rate, and n is the code-word length (see (4.7.29)). By the weak law of large numbers, the number of errors in transmission will be of the order of n/3 for large n, where ,i3 is the probability of a single digit being received incorrectly. Thus, roughly speaking, by setting e = n@ we can achieve arbitrarily high reliability provided 1 - R 2 H(2@, 1 - 2/3), or R 5 1 - H(28, 1 - 28). [Note that ,i3 must be less than 4 for this to be valid; see the discussion preceding (4.7.29).] Hence if we could synthesize codes meeting the Varsharmov-Gilbert-Sacks condition, such codes could be used to maintain any transmission rate up to 1 - H(2/3, 1 - 28) (which is close to capacity for small /I) with a n arbitrarily small probability of error. Theorem 4.8.1 is a weakened version of a result proved by Weiss (1960); see also Wolfowitz (1961). PROBLEMS 4.1 A Hamming code may be defined as a group code with the property that the jth column of the parity check matrix is the binary number j. For example, if A is 3 by 6 then p

O

1

0

1

0

~

ERROR CORRECTING CODES

where, for example, the column

rll

represents the binary number 01 1 = 3. Digits 1, 2, 4, 8, . . . may be chosen as check digits (to avoid degeneracy, assume that there are no zero rows). If a Hamming code has m check digits, what is the maximum code-word length n ? Give the parity check matrix for m = 4 and n as large as possible. 4.2 a. Show that a parity check code will correct e-tuple (and all smaller) errors and detect (but not necessarily correct) (e 1)-tuple errors if and only if 1 columns of the parity check matrix is linearly independent. every set of 2e b. Given an e-tuple-error correcting parity check code with parity check matrix A = [I, I A , ] , where I, is an identity matrix of order m. Show that the code defined by the matrix

+

+

+

will correct e-tuple errors and detect (e 1)-tuple errors. (This corresponds to adding one check digit which performs a parity check on all the digits of a code word.) c. Find a parity check matrix of a single-error correcting, double-error detecting code with 5 check digits and 8 information digits. d. What is the analog of the Varsharmov-Gilbert-Sacks condition for codes that correct e-tuple and detect (e + 1)-tuple errors? 4.3

A group code has the following parity check matrix.

a. If a,,

=

a,, = a,, = a,, = 1 , list the code words in the resulting code.

b. Show that a,,, a,,, a,,, a,, can be chosen so that the code will correct single errors and detect double errors. Show also that not all double errors can be corrected.

132

INFORMATION THEORY

4.4 Show that the minimum distance between code words of a parity check code is d if and only if every nonzero code word has at least d ones and at least one code word has exactly d ones. (In other words, the minimum distance equals the minimum weight over all nonzero code words.) 4.5 It is desired to construct a parity check code to detect e-tuple and all smaller errors. If an error of magnitude <e is made, the decoder must decide that an error has occurred; it is not required that the decoder provide any information about the nature of the error.

a. Find the necessary and sufficient conditions on the parity check matrix for this type of error detection. b. Find a convenient condition on the code words which is-equivalent to the condition of part (a). c. Given a parity check code with 2k words of length n, find the number of error patterns that can be detected by the code. [Note that this number is the same for all (n, k) codes.] 4.6 Agenerator matrix for an (n, k) binary code S is a matrix G whose rows consist of k linearly independent code words of S. Every code word of S may be obtained as a linear combination of the rows of G. Assuming that the last k columns of G are linearly independent (if not we can interchange columns), we may reduce G by elementary row transformations to a new generator matrix

where B is k by n7 (m = rank of the parity check matrix identity matrix of order k. a. Show that the matrix A

=

=

n

- k) and Ik is an

[I, I BT] is a parity check matrix for S.

b. Find parity check matrices for the codes whose generator matrices are

ii.

1.

G = [I

G =

1

1

I

11.

4.7 Given a single-error correcting, double-error detecting parity check code characterized by the parity check matrix A. Show that it is possible to delete a row and a column of A such that the resulting matrix is the parity check matrix of a single-error correcting code. 4.8 A binary code is said to be close-packed if for some e, all single, 1)-tuple errors, but no errors of double, . . . , e-tuple errors, possibly some (e greater weight are corrected by the minimum-distance decoder, no matter which code word is transmitted. The code is said to be lossless if, for each transmitted word, the code corrects all single, double, . . . , e-tuple errors, but nothing else. If a code is lossless we may surround each word w i by a sphere of radius

+

ERROR CORRECTING CODES

133

e consisting of all sequences whose distance from wi does not exceed e, such that the spheres are disjoint and cover the entire space of 2" binary sequences.

Equivalently, a lossless code meets the Hamming bound. Note that by the argument after (4.7.22), a close-packed or lossless code has the lowest average probability of error among all codes having the same number of code words and the same code-word length. a. Find a single-error correcting lossless parity check code with sixteen code words. b. Find a double-error correcting lossless parity check code with four check digits. c. Show that the Hamming code (Problem 4.1) is close-packed.

+

4.9 Show that the lower bound H ( I , 1 - I ) B(I,, ,8) - H(A,, 1 - I,) of (4.7.28), plotted as a function of R [where R = 1 - H ( 1 , 1 - I ) ; see Fig. 4.7.21 is tangent to B ( I , ,8) at I = I,. 4.10 Consider a binary symmetric channel with capacity C. If R > C and 0 5 I < 1 , show that for sufficiently large n, no code ( [ 2 n R ] n) , can have an average probability of error less than I . Hence given a sequence of codes ([2""],n), R > C, with average probability of error p,(e),we have 1 a s n - m. 4.11 Consider an arbitrary code (s, n) with not all words distinct. Assume that w, = w,, and form a new code by replacing w, by any sequence w,' # w,. Assume that in each case the decision scheme is determined by the ideal observer with all words equally likely. Show that the average probability of error of the new code cannotexceed the average probability oferror of the original code.

-

CHAPTER FIVE

Further Theory of Error Correcting Codes 5.1. Feedback shift registers and cyclic codes

In Chapter 4 we talked about general properties of codes but we never came to grips with the synthesis problem. If we wish to construct an e-tuple-error correcting code for a specified word length n, the only

Fig. 5.1.1.

Feedback shift register.

general method we have available at this point is the very laborious procedure described in the proof of the Varsharmov-Gilbert-Sacks bound. (In the special case e = 1 there is no problem, since all we need to do is to take a parity check matrix with distinct nonzero columns.) It would be desirable to be able to write down a parity check matrix which we knew t o be the matrix of an e-tuple-error correcting code. In this chapter we attack the synthesis problem and in addition study some of the algebraic properties of codes. The class of codes (the "cyclic codes") that we shall examine may be thought of intuitively as being generated by a physical device called a feedback shift register. A feedback shift register (see Fig. 5.1.1) is a system of binary storage devices F,, F,, . . . , Fm-,; switches; and a modulo 2 adder. The boxes labeled a,, a,, . . . ,a,-, represent switches. If a, = 1, the corresponding switch is closed and the contents of F, are fed back to the input of Fm-,.

135

FURTHER THEORY OF ERROR CORRECTING CODES

If a, = 0, the switch is open. The system is controlled by a clock pulse; at each pulse the contents of Fi are shifted to Fj-l ( i = 1, 2, . . . , m - I), and the new number stored in Fm-, is aoxo alxl . . a,-lx,,-l (mod 2), where xi is the number stored in Fi before the clock pulse. The operation of a feedback shift register may be described by a matrix equation. If xi is the number stored in Fi after the tth clock pulse and xi' 1)th clock pulse, is the number stored in F, after the (t

+

+

+

+

or in matrix form, x' =

Tx

(mod 2),t

where

X =

The (column) vector x is called the state or content of the shift register, and T is called the characteristic matrix of the shift register. It is easily

t All arithmetic in this chapter is modulo 2; all codes to be studied will be binary. If the results of the chapter are to be generalized to codes whose words have components in an arbitrary finite field, the coefficients a , in the last row of the matrix T should be replaced by -ai in order that the characteristic (and minimal) polynomial of T still be . . . a , - l P - l Am (see Section 5.2). given by a, a l l

+

+

+

+

136

INFORMATION THEORY

seen that the determinant of T is a,; we assume that a, = 1 so that T is nonsingular. If x , is the state of the shift register after the tth clock pulse ( t = 0, 1, 2, . . .), then as the clock pulses appear the register content traces out a sequence x,, x,, x,, . . . , where x, = Tx,, x, = T2x0,etc. We claim that

Fig. 5.1.2. Feedback shift register and associated cycle set.

if x, # 0, then x, must equal x, for some n > 0;that is, the state must eventually return to x,. T o prove this, note that the shift register has only a finite number 2" of possible states. If the state never returns to x,, then eventually one of the other states must occur twice in the sequence, that is, there must be two distinct states xi and x i such that Tx, = x,, and T x , = xi+,. Thus T(x, - xi) = 0,and hence by the nonsingularity of T we have xi = xi,a contradiction. Thus the shift register sequences are periodic. If the initial state of the register is x, and a sequence of clock pulses is applied, the resulting sequence of states forms a cycle x,, x,, . . . , x,, x,, . . . , with the states x,, x,, . . . , x, distinct. The set of all possible cycles is called the cycle set of the feedback shift register, or of the associated matrix T.

,

FURTHER THEORY OF ERROR CORRECTING CODES

137

A feedback shift register and its associated cycle set is shown in Fig. 5.1.2. For simplicity, we have identified state vectors with binary numbers, for example,

There are two cycles of length 6 , one cycle of length 3, and one cycle of length 1 consisting of the zero vector alone. The cycle consisting of zero alone will always be present in any cycle set, since T - 0 = 0. We now establish a connection between feedback shift registers and a certain type of coding system.

A parity check code is said to be cyclic if it has the property that whenever w = (r,, r,, . . . , rn-,, rn-,) is a code word, the sequence W' = (rn-l, rO,r l , . . . , rn-2) obtained from w by a "cyclic shift" is also a code word. Definition.

A feedback shift register "generates" a cyclic code, in the following sense. Theorem 5.1.1. Let a parity check code be formed by taking, as the columns of the parity check matrix, one or more repetitions of a cycle generated by a feedback shift register. In other words, consider a parity check code with matrix A = [x 1 Tx T2x1 . . ( Tn-lx] where x is a binary m-vector, T is a nonsingular matrix of the form (5.1.1), and T n x = x . The resulting code is cyclic.

I

We may use the feedback shift register of Fig. 5.1.2 to generate various cyclic codes; for example, we may construct codes with parity check matrices 0 0 0 1 0 1 0 0 1 1 1 1

1 0 1 0 0 0

1 1 1 0 0 1

138

INFORMATION THEORY

It may be verified that A , and A , define the same code, namely the code whose words are 000000, 101010, 010101, and 111 11 1. (This is no accident; see Lemma 5.3.3b). The matrix A, is degenerate in the sense that the rank of A , is two while the number of rows is 4. Since the last two rows of A, may be expressed as linear combinations of the first two rows, the parity check equations associated with A, become :

Thus we may take one digit, say r,, as an information digit; the remaining digits r, and r, will be check digits. The code defined by A , is therefore (000, 111). We may also form a cyclic code by taking two repetitions of the cycle of length 3. The corresponding parity check matrix is

Since A, is of rank 2, the code defined by A , has 16 words, and thus differs from the code defined by A , and A,. Proof of Theorem 5.1.1. If w = (r,, r,, . . . , rn-,) is a code word, then AwT = 0, hence r,x r,Tx . - rn-lTn-lx = 0. Multiplying by T , we obtain (since T n x = x ) rn-,x r,Tx . . . rn-,Tn-lx = 0 ; hence (rn-,, r,, r,, . . . , rn-,) is a code word.

+

+

+

+

+

+

The converse of Theorem 5.1.1 is also true; namely, for every cyclic code there is a corresponding feedback shift register. We will not be able to prove this until we have developed some general properties of binary matrices. i

5.2.

General properties of binary matrices and their cycle sets

Before proceeding to a study of cyclic codes, we must apply some results from linear algebra to binary matrices and their cycle sets. Let T be a non-zero rn by m matrix of zeros and ones [not necessarily of the form (5.1.1)]. The characteristic polynomial of the matrix T is defined by

d4 = 1 T - 111 9

139

FURTHER THEORY OF ERROR CORRECTING CODES

where I is an identity matrix of the same order as T. For the matrix T of Fig. 5.1.2, we have 12. 1 0 01

In general, if the matrix T is given by (5.1. I), it may be verified by expanding IT - RII by minors of the last row that cp(l) = a,

+ a l l + a2A2+ - . . + am-,2m-1 + Am.

Note that if T is the characteristic matrix of a feedback shift register, then y(A) determines T uniquely. Now we define the minimal polynomial of T as the monict polynomial f (A) of lowest degree such that f (T) = 0. The reader may recall that q(T) = 0 (this is the Cayley-Hamilton theorem, which we shall use without proof) and that f(A) is a divisor of y(A) (see Lemma 5.2.2). In general, f ( l ) # ~ ( 1 ) . For example, for any square matrix T with minimal polynomial f(A) and characteristic polynomial cp(A), let

r

~ l o l T .

Then since

for any polynomial p, the minimal polynomial of To is f(A), but the characteristic polynomial of To is [cp(A)I2, which is always of higher degree than f (A). However, we do have the following result. Lemma 5.2.1. If a matrix T is of the form (5.1 I), then its minimal and characteristic polynomial are the same. Proof. Let r,(T') denote the first row of the matrix Tf By a direct computation it follows that

r,(TO) = r,(l) = [l 0 0 0 . . - 01, rl(T) = [0 1 0 rl(T2) = [O 0 1 0 . . - 01,. . . , r,(Tm-l) = [0 0 0 - . . 0 11.

+ +

+

+

--

01,

t A nomia mi alp(^) = b~ b ~ z b& . . . b,zn, b, # 0 , is said to be monk if b, = 1 . In modulo 2 arithmetic, all nonzero polynomials are monic.

140

INFORMATION THEORY

If f(A) # p(A) then since p(T) = 0, the degree of f(A) is at most m - 1. But if f(T) = b,Tk = 0, the linear independence of the r,(T'), i = 0, 1, . . . , m - 1, implies that all b, = 0, a contradiction. Thus f ( 4 = 944. In view Lemma 5.2.1, our main interest will be in matrices whose minimal and characteristic polynomials are the same. However, the results to follow will be stated in general terms. Let T be a binary matrix whose minimal polynomial has degree q (note that if T is m by m and y(l) =f(A) then necessarily q = m). Let Z be the set of matric polynomials in T of degree _
z :,

sf

+

+

+

9-1

4-1

z=o

c =O

2 bzT" 2 bb,'TZ

then

P-1

p(T) = 2 (b, - 6,')T"

0.

2=0

Thus we have a polynomial p(A) of degree
+

+

Lemma 5.2.3. Let f(R) be the minimal polynomial of a matrix T. Suppose that f (A) is irreducible, that is, the only polynomials which divide f (A) aref(A) itself and the constant polynomial I . Let p(T) be any matric polynomial in T such that p(T) is not the zero matrix. Then p(T) is nonsingular, and the inverse of p(T) is also a matric polynomial in T. (In particular, T itself is nonsingular.)

FURTHER THEORY OF ERROR CORRECTING CODES

141

+

Proof. Let S be the set of all polynomials of the form a(A)f(A) b(A)p(A),where a(A) and b(A) range over all polynomials. Let d(A) be the monic polynomial of lowest degree in S. Then: a. d(A) divides both f(A) and p(A). b. Any polynomial that divides f ( A ) and p(A) is a divisor of d(A), thus d(A) is the polynomial of highest degree which divides both f (A) and p(A); d(A) is called the greatest common divisor off (A) and p(A).

+

Since d(A) 6 S, d(A) = a(A)f (A) b(A)p(A) for some polynomials a(A) and b(A); (b) follows immediately. T o prove (a), write f ( A ) = g(A) d(A) h(A) where the degree of h(A) is less than the degree of d(A). Observe that the set S is closed under the operations of addition, subtraction, and multiplication by a polynomial; hence h(A) = f (A) - g(A) d(A) E S. But by the minimality of d(A), h(A) = 0 so that d(A) divides f (A). Similarly d ( l ) divides p(A). Now since f ( A ) is irreducible, the greatest common divisor of f(A) and p(A) is either 1 or f (A). If d(A) =f (A), then f ( A ) divides ~ ( 1 ) But . p(A) = g(A)f (A) implies p ( T ) = g ( T )f ( T ) = 0, contradicting the hypothesis. Thus d(A) = 1. Since d ( l ) = 1 E S, it follows that there are polynomials a(A) and b(A) such that a(A)f (A) b(A)p(A) = 1. Replacing A by T we have a ( T )f ( T ) b(T)p(T) = I ; but f ( T ) = 0,so that b(T)p(T) = Z a n d p ( T ) is nonsingular. The inverse of p ( T ) is b ( T ) , a matric polynomial.

+

+

+

We now return to the set Z. Theorem 5.2.4. Let Z be the set of all 2q matric polynomials in T of degree
c. The set Z , of all nonzero elements of Z is a n abelian group under multiplication. Proof. The modulo 2 sum of a matric polynomial of degree r and a matric polynomial of degree s 2 r is a polynomial of degree at most s. Hence Z is closed under addition. Addition of matric polynomials is associative since matrix addition is associative. The zero polynomial serves as the additive identity and each element of Z is its own additive

142

INFORMATION THEORY

inverse. Commutativity of addition is clear, and hence (a) is proved. Part (b) is a consequence of the distributivity of matrix multiplication over addition. To prove (c), note that the product of two matric polynomials p, and p 2 of degree
Let p,(T) = T3

Now f (T) = T4

+ T + I, p2(T) = T2 + I ;

then

+ T3 + I = 0,hence +

thusp,(T)p,(T) = T3 T2 = p3(T). We must show thatp,, p, E Zo implies plpz eZ0, that is, Zo is closed under multiplication. But if p,(T) and p2(T) are nonzero matric polynomials, then p,(T) and p2(T)are nonsingular by Lemma 5.2.3. Hence pl(T)p2(T) is nonsingular, and hence nonzero. [Alternately, ifpl(T)p2(T) = 0 then f (1) dividesp,(il)p,(l) by Lemma 5.2.2; since f (ljis irreducible, the unique factorization theorem for polynomials implies that either f (1) divides p,(il) or f (A) divides p2(l); this contradicts the assumption that the degrees of p,(l) and p,(l) are
FURTHER THEORY OF ERROR CORRECTING CODES

143

(Any finite field is referred to as a Galois field; it can be shown that a finite field must contain pq elements where p is a prime and q is a positive integer; furthermore, any two finite fields of the same size are isomorphic.) Now consider any nonsingular m by m binary matrix T and its powers T ; i = 0, 1 , 2 , . . . . We claim that for some r , T r = I. For since T has only finitely many components, there must eventually be repetition in the

L1 0 0

11

1

0 0 0 1 1 1 1 0 1 0 1 1 0 0 1

Parity check matrix of associated cyclic A code :

I

0 0 1 1 1 1 0 1 0 1 1 0 0 1 0

=

0 1 1 1 1 0 1 0 1 1 0 0 1 0 0 1 1 1 1 0 1 0 1 1 0 0 1 0 0 0

Fig. 5.2.1. A matrix with maximal period and its associated feedback shift register and cyclic code.

sequence I, T, T 2 ,. . . , that is, for some i and j we must have T i = T i . If say i > j, then the nonsingularity of T implies Ti-j = I. We define the period of T as the smallest integer r such that T T= I. (The period cannot exceed 2a - 1, where q is the degree of the minimal polynomial of T. See Problem 5.10.) Now if x, is any nonzero binary m-vector, then just as in Section 5.1 the vectors x,, x, = Tx,, x2 = Tx, = T2x,, . . . form a cycle, that is, x, = x, for some n > 0. We have the following theorem. Theorem 5.2.5. Let T be a nonsingular m by m binary matrix with period r . If the minimal polynomial f (A) is irreducible, then every nonzero vector x lies on a cycle of length r. Since the number of nonzero m-vectors is 2" - 1, it follows that r 5 2" - 1, and in fact r is a divisor of 2" - 1. Proof. Suppose x lies on a cycle of length j < r. (Observe t h a t j > r is impossible since T T= I, hence TTx= x for all x.) Then Tix = x, o r

144

INFORMATION THEORY

(Tj - I)x = 0. Sincej < r, Tj - I # 0,so that Tj - I is nonsingular by Lemma 5.2.3. But then x = 0, a contradiction. If the period of T is 2" - 1, T is said to have maximalperiod. Equivalently, T has maximal period if and only if its cycle set consists of one cycle of length 2" - 1 together with the trivial cycle of length 1. The maximal-period case is of interest in coding theory because if a matrix T of the form (5.1.1) generates a cyclic code, then if T has maximal period the resulting code has the largest possible word length. For an example of a matrix with maximal period, see Fig. 5.2.1. Necessary and sufficient conditions for T to have maximal period are given in the following theorem. Theorem 5.2.6. Let T be a nonsingular m by m binary matrix with minimal polynomial f(A). Then T has maximal period if and only if f(A) is irreducible and f(A) is not a divisor of ;lk- 1 for any k < 2" - 1. Proof. First suppose f(A) is irreducible and does not divide A' - 1 for any k < 2" - 1. Let r be the period of T. By Theorem 5.2.5, r 2" - 1. Dividing AT - 1 by f(A) we obtain where degree h(A) < degreef(A). Replacing A by T we have 0 = T r - I = g ( T )f ( T ) h ( T ) = h(T). By the minimality of f (A), h(A) = 0; hence f(A) divides AT - 1. But by hypothesis, r cannot be less than 2" - 1 ; hence r = 2" - 1 and T has maximal period. Conversely, suppose T has maximal period. If Ak - 1 = g(A)f ( A ) for some k < 2" - 1, then T k - I = g ( T )f ( T ) = 0, contradicting the fact that the period of T is 2" - 1 . It remains to show that f ( A ) is irreducible. If not, there are two possibilities:

+

CASE1. f (A) = g(A)h(A)where g(A) and h(A)are relatively prime [that is, the greatest common divisor ofg(A)and h ( l ) is unity], and the degrees of g(A) and h ( l ) are each greater than 1.

CASE2. f(A) is [g(A)lkwhere g(A) is irreducible and k

2 2.

+

In Case 1, there are polynomials a(A) and b(A) such that a(A)g(;l) b(A)h(A)= 1 (see the argument of Lemma 5.2.3). Now let S, be the set of all m-vectors x such that g(T)x = 0, and S, the set of all x such that h(T)x = 0. We will show the following: Any m-vector x may be written as y, where

+ y,

Y I E S1, Y 2 6 S2.

FURTHER THEORY OF ERROR CORRECTING CODES

+

145

+

(Note that y, and y, are unique for a given x, for if y, y, = y,' y,' where y,, y,' E Sl and y,, y,' E S,, then y, - y,' = y,' - y, E S , n S,, SO by (5.2.2h Y l = Y,', Y 2 = y,'.)

+

To establish (5.2.2), (5.2.3), and (5.2.4), we first note that a(il)g(L) b(il)h(l)= 1 implies (5.2.5) a(T)g(T) b(T)h(T)= I.

+

If x is any binary m-vector, then if x E S , n S , we have g(T)x = h(T)x = 0 ; hence, by (5.2.5), x = 0. This proves (5.2.2). Again by (5.2.5), x = Ix = a(T)g(T)x

+ b(T)h(T)x= y, + y,.

Since ~ ( T ) Y=z h(T)a(T)g(T)x= a(T)h(T)g(T)x = a(T)f ( T ) x = 0, and similarly g(T)y, = 0, we have y, E S , and y, E S,, proving (5.2.3). T o prove (5.2.4), observe that if x E S,, then g ( T ) x = 0 so that g(T)Tx = Tg(T)x = TO = 0 , whence T x E S,; similarly x E S, implies Tx E S,. Finally, we note that S , and S , each contain at least one nonzero vector. For suppose S , = (0). Then by (5.2.3), S , is the entire space V of all binary m-vectors, that is, g ( T ) x = 0 for all x. I t follows that g ( T ) = 0, contradicting the rninirnality of f(R). Now let x, c S,, x, E S,, with x, # 0 , x, # 0. By (5.2.2) and (5.2.4), x, and x, must lie on different cycles, so T cannot possibly have maximal period. In Case 2 we must appeal to a theorem in Birkhoff and MacLane (1953, p. 329) which states that if the minimal polynomial of T is [g(il)lk where g(il) is irreducible, then the entire space V may be written as a direct sum of nontrivial subspaces S,, . . . , ST, r 2 2 (that is, each x E V has a unique expression x = y, - . . y,, y, E S,) where x E Si T x E Si, i = 1,2, . . . , r. Now just as above, if we pick x, E S,, x, E S, with x, # 0 , x, # 0, then x, and x, must lie on different cycles. The proof is complete.

+

+

If T is nonsingular and has maximal period, the set of nonzero matric polynomials of degree <m - 1 coincides with the set of powers T i , 0 j i 5 2" - 2. For the matrices Ti, 0 j i j 2" - 2 are distinct. (If T' = T' where 0 5 j < i 5 2" - 2, then Ti-j = I where 0 < i - j 2'" - 2, contradicting the fact that the period of T is 2" - I.) Since

146

INFORMATION THEORY

Table 5.2.1. Binary irreducible polynomials and their periods (Elspas, 1959)

Coefficients Polynomial

Period a0

a1

a2

a,

a4

f(T) = 0,each Ti can be expressed as a (nonzero) matric polynomial of degree < m - 1. (See the proof of Theorem 5.2.4.) Since there are exactly 2m - 1 nonzero matric polynomials of degree < m - 1, these polynomials must be in one-to-one correspondence with the powers of T. For example, consider the matrix

1

!]

0 1 0 0

T=

with f(A) = y(A) = 1

+ +

+ A3 + 14.

(See Fig. 5.2.1 .) Using the fact that T4 T 3 I = 0,we may express the powers of T i n terms of matric polynomials of degree 5 3 as follows:

and so on.

FURTHER THEORY OF ERROR CORRECTING CODES

147

Polynomials f (A) satisfying the conditions of Theorem 5.2.6 exist for all possible degrees. (See, for example, Peterson, 1961.) A brief table of irreducible polynomials and the periods of the corresponding matrices is shown in Table 5.2.1. Note that A4 A3 A2 A 1 is irreducible but is a divisor of A6 1; the associated matrix does not have maximal period. We remark that if T is m by m and has maximal period, then the minimal polynomial f (A) is of degree m, and furthermoref (A) divides /I2"-' - 1. For iff (A) is of degree q, then the period of T is at most 2q - 1 (Problem 5.10), which proves the first assertion. In general, if T is a nonsingular binary matrix with minimal polynomial f(A) and period r, then we have seen (at the beginning of the proof of Theorem 5.2.6) that f(A) divides IT- 1 ; this proves the second assertion.

+ + + +

+

5.3. Properties of cyclic codes In this section we apply some of the results of Section 5.2 to the study of cyclic codes. First we develop a convenient algebraic characterization of such codes. If b(x) = b, blx - . - b,-,zn-l (b, = 0 or 1) is a polynomial of degree < n - 1, then we say that b(x) represents the sequence (b,, bl, . . . , bn-,). Now let K be the set of all binary polynomials of degree In - 1. It is clear that K is an abelian group under (componentwise) modulo 2 addition (in fact K is a vector space over the modulo 2 field). We define a multiplication operation "." in K as follows. If b(x) and c(x) are in K, let b(x). c(x) be the ordinary product (with arithmetic modulo 2), subject to the relation xn = 1. More precisely, b(x) . c(x) is the remainder after the ordinary product of b(x) and c(x) is divided by xn - 1. For example, if n = 4 then

+

+

+

+

since x4 = 1, x5 = x, x2 x2 = 0. (For those familiar with algebra, what we are actually doing here is considering the set K* of residue classes of binary polynomials modulo xn - 1 ; if [b(x)] is the residue class of the polynomial b(x), then addition and multiplication in K* are defined by

+

+

[b(x)l [c(x)l = [b(x) ~ ( 4 1 , [b(x)l . [c(x)l = [b(x)c(x)l. With this definition of addition and multiplication, K* (or K) is a vector space that is also a ring; that is, K* is an algebra over the modulo 2 field.) For the remainder of this section, the notation b(x). c(x) will always signify multiplication in K ; the notation b(x)c(x) will always indicate ordinary multiplication.

148

XNFORMAT~ONTHEORY

Now in K,

+

+

+

+

+

+ .- +

x . (bo b1x . bn-pn-') = bn-l bg blxa . bn-2xn-a so that multiplication by x corresponds to a cyclic shift. This idea is the basis of the following algebraic characterization of cyclic codes. Theorem 5.3.1. Let S be a cyclic code, and let S* be the set of polynomials in K which represent code words of S. Let g(x) be the monic polynomial of lowest degree in S*. (In the degenerate case in which S consists of the zero vector alone, take g(x) = xn - 1, which is identified with 0 in K). Then S* is precisely the set of all multiples of g(x) in K. Furthermore, g(x) divides xn - 1 ; that is, xn - 1 = h(x)g(x) (ordinary multiplication) for some polynomial h(x). g(x) is called the generator of S*, or the generator polynomial of the code S. Conversely, let gl(x) be any monic polynomial that is a divisor of xn - I. Then the set S* of multiples of gl(x) in K represents a cyclic code, and furthermore, the generator of S* is gl(x). Proof. We prove the direct assertion first. If b(z) E S*, then since x . b(x) represents a cyclic shift, the hypothesis that S is cyclic implies that x b(x) E S*, and hence that xi b(x) E S* for all i 2 0. Since S is a group code, it follows that if c(x) = co clx . . cn-,zn-' is an arbitrary binary polynomial of degree I n - 1, then

+

+

+

n-l

In particular, any multiple of g(x) in K belongs to S*. Now let b(x) be any element of S*. Dividing b(x) by g(x) we have b(x) = q(x)g(x) r(x) with deg r(x) < deg g(x). It follows that in K we have b(x) = q(x) g(x) r(x). Again using the group property, we have r(x) = b(x) - q(x) .g(x) E S*. It follows by the minimality of g(x) that r(x) = 0, so that b(x) is a multiple of g(x) in K. [Note that since deg b(x) n - 1 , q(x) - g ( x ) = q(x)g(x) = b(x).] T o show that g(x) divides 2" - I, write xn - 1 = h(x)g(x) r(x), with deg r(x) < deg g(x). In K this equation becomes 0 = 4 x 1 .g(x) r(x), hence r(x) = -h(x) g(x) E S*. Again by the minimality of g(x), r(x) = 0. To prove the converse assertion, given the set S* of multiples of gl(x) in K, let S be the set of binary sequences represented by the polynomials of S*. Since S* is closed under modulo 2 addition, it follows that S*, hence S , is an abelian group. Thus S is a parity check code. If b(x) E S* then x . b(x) E S*, hence a cyclic shift of a sequence of S also belongs to S. Thus S is cyclic.

+

+

<

+

.

+

149

FURTHER THEORY OF ERROR CORRECTING CODES

T o show that g , ( x ) is the generator of S*, we must show that gl(x) is the monic polynomial of lowest degree in S*. Any element of S* may be written as b ( x ) . g l ( x ) , where b(x) is a polynomial of degree _
+

<

+

We now examine generator and parity check matrices for a cyclic code. Theorem 5.3.2. Let S be a cyclic code with words of length n, and let g(x) = a , a,x . . am-lxm-l xm be the generator of S*. Then the word w = (a,, . . . , am-,, 1,O, . . . ,0 ) represented by w, together with k - 1 = n - rn - 1 cyclic shifts of w, form a set of generators for S, that is, a maximal linearly independent set of code words. Furtherhlx . - - hk-lxk-l xk more, if xn - 1 = h(x)g(x) where h(x) = h, ( k rn = n), then

+

+

+ +

+

+

+

+

+

is a parity check matrix for S. (In the degenerate case in which S consists of the entire space of 2" binary n-vectors, so that g(x)= 1, h(x)=xn - 1 = 0 in K, we may take w = (1 0 0 . . . O), and we may take the matrix (5.3.1) to consist of one row of zeros.) Prooj By Theorem 5.3.1, every polynomial in the set S* is a multiple of g(x). In fact, the argument in the proof of Theorem 5.3.1 shows that each b(x) E S* may be written as c(x)g(x) where the multiplication is ordinary polynomial multiplication. If rn is the degree of g(x), then the degree of c(x) is necessarily In - 1 - rn = k - 1. It follows that S* consists of all polynomials of the form ( c , c,x - . C~-~X~-~)~(X), that is, the set of all linear combinations of the polynomials g(x), xg(x), . . . ,

+

+ - +

150

INFORMATION THEORY

xk-lg(x). Translating this result back to S, we conclude that if g(x) = a, alx - . . a , - , ~ ~ - ~xm, then (a,, a,, . . . , am-,, 1,0, . . . ,O), (0, ao, a,, . . . ,a,"-,, l , O , . . . ,01, . . . ,(0, . . . , 0 , ao, a,, . . . , an,-,, 1) generate S. This proves the first part of the theorem. To prove the remaining assertion, note that a sequence (bob, . b,-,) satisfies the equation given by row t of the matrix A if and only if

+

+

+

+

k

2 h,bn-t-i = 0

(define h, = 1).

i=o

k

Since 2 hi bn-,-i is the coefficient of x"-~in the ordinary product of i=O

-

we have A(bo b, . . bn-l)T = 0 if and only if h(x)b(x) has zero coefficients multiplying x"-~, t = 1,2, . . . ,rn. Since h(x)g(x) = xn - 1 , it follows that b(x) = xig(x) satisfies this condition for i = 0 , 1, . . . ,k - 1. In other words, w and its k - 1 cyclic shifts satisfy the equations corresponding to A ; and hence every code word w, r S satisfies AwlT = 0 . Now the rank of A is rn = n - k , hence AwlT = 0 has 2'" solutions; since S has 2k words, the set of solutions to AwlT = 0 is precisely the set S, that is, A is a parity check matrix for S. The proof is complete. Observe that the rank of A = the degree of g(x). As a first step in showing that the class of cyclic codes is identical with the class of codes generated by feedback shift registers, we examine the relationship between the minimal polynomial of a matrix and the generator polynomial of the associated cyclic code. Lemma 5.3.3. Let T be a nonsingular binary rn by rn matrix whose minimal polynomial f (A) is of degree q. If x is a nonzero rn-vector and Tnx = x, let A be the matrix [x ( Tx I T2x ( . . ( Tn-lx]. [Notice that although T may not be of the standard form (5.1.1), the argument of Theorem 5.1.1 may be applied verbatim to show that the code S defined by A is cyclic.] To avoid degeneracy assume that n > q.

-

a. The binary sequence w represented by the polynomial f (x) is a code word of S ; hence f ( x ) is a multiple of the generator polynomial g(x) of the code. b. If the rank of A is q, then f ( x ) = g(x) so that we may identify the minimal polynomial of T with the generator polynomial of the associated code. In particular, if the rn-vectors x, and x, lie on cycles of the same length t, and n = st for some positive integer s, then the parity check matrices [x, Tx, . . . Tn-lx,] and [x, Tx, Tn-lx,] define the

1

1

I

1

1

I

FURTHER THEORY OF ERROR CORRECTING CODES

151

same code [namely the code whose generator polynomial is f(x)] provided that both matrices have rankq. Conversely iff (x) = g(x) then A has rankq. c. If f(z) is irreducible, then j'(x) = g(x) and hence the rank of A is q.

that is, the binary sequence w = (a, a, - . a,-, 1 0 . . - 0) satisfies the parity check equations of the code. This establishes (a). Part (b) is a n immediate consequence of (a) and the observation that the rank of the parity check matrix coincides with the degree of the generator polynomial. If f(x) is irreducible, then since g(x) divides f(x), g(x) is either 1 o r f(x). If g(x) = 1, then the code consists of all 2" binary n-vectors, a n impossibility since the parity check matrix A does not consist entirely of zeros. This proves (c). Alternately, we may argue that the first q columns of A are linearly independent. For if 2 ~b ,:T k = 0 with not all b, = 0, then Lemma 5.2.3 implies that b,T" 0, contradicting the minimality off. On the other hand, any polynomial in T of degree 2 q may be expressed as a linear combination of I, T, T2, . . . , Tq-I, using the fact that f ( T ) = 0. It follows that any column of A may be written as a linear combination of the first q columns, and thus the rank of A is q. The lemma is proved.

21;

We remark that if the rank of A is t = n - k, the code word w* corresponding to the generator polynomial g(x), together with k - 1 cyclic shifts of w*, generates the code. Consequently, g(x) is the nonzero polynomial of lowest degree such that g(T)x = 0. We are now ready to prove the converse of Theorem 5.1.1. Theorem 5.3.4. Let S be a cyclic code. Then there is a feedback shift register that generates S. More precisely, if S has 2k words of length n, then there is a nonsingular m by m matrix T of the form (5.1.1), with m = n - k, and a n m-vector x, such that T n x = x and A = [x 1 Tx I . - . I Tn-lx] is a parity check matrix for S. Proof. If S consists of all 2" binary n-vectors, we may take any T with x = 0. If Sconsists of the zero vector alone, we may take T t o be an n by n matrix of the form (5.1.1) with a, = 1, ai = 0, i = 1, 2, . . . , n - 1, and we may take x to be the (column) n-vector (0, 0, . . . , 1). The corresponding parity check matrix is square and nonsingular and therefore defines the code consisting only of the zero vector. Thus assume that S has 2k words of length n, where 0 < k< n.

152

INFORMATION THEORY

Now as in Theorems 5.3.1 and 5.3.2, xn - 1 = h(x)g(x), where g ( z ) = adxi is the generator polynomial of the code and h(x) = It-,h,zi is the polynomial whose coefficients appear in the parity check matrix A of (5.3.1). If we compare coefficients of xn - 1 and h(x)g(z),we obtain the identities

, : I

Now let 0

1

0

0

0

1

0

0

0

a, a, a,

... ...

0

... ..

1

0

am-

and take

We claim that the parity check matrix A of (5.3.1) is

I

I

[x T x T2x

I - I Tn-lx].

We may prove this by induction. The first column of A is clearly x. Now suppose that the rth column of A is Tr-lx. We note that

where z = a d o

+

. + am-lym-l.

FURTHER THEORY OF ERROR CORRECTING CODES

By the induction hypothesis, TTx= TTr-lx = T(co1umn r of A)

= column r

+ 1 of A.

Finally, since xn - 1 = h(x)g(x), we have h, = a, = 1 ; thus the last column of A is

Consequently,

Tnx = column 1 of T

Since a, is not zero, T is nonsingular and the theorem is proved.

154

INFORMATION THEORY

COROLLARY 5.3.4.1. a. The class of cyclic codes (with the exception of the code consisting of the entire space) coincides with the class of parity check codes whose parity check matrices are of the form

I

[x Tx

I . - . I Tn-lx],

where T is a nonsingular matrix of the form (5.1. l),

and Tnx = x. b. For any such code S in the latter class (excluding the degenerate case in which S consists of the zero vector alone), the minimal polynomial of T may be identified with the generator polynomial g(x) of the code. If xn - 1 = h(x)g(x), then the parity check matrix [x ( Tx . . ( Tn-lx] coincides with the matrix (5.3.1). Proof. Part (a) is an immediate consequence of the argument used in the proof of Theorem 5.3.4, along with Theorem 5.1.1. To prove (b), let A = [x Tlx I . . I T,"-lx]be the parity check matrix of a cyclic code S, where Tl is a nonsingular matrix of the form (5.1.1),

I

I

x =

,

and

Tlnx = x.

If Tl is rn by rn, it may be verified directly that the rn by rn submatrix formed by the first rn columns of A is triangular, and therefore A has rank rn. If n = rn, then the code consists of the zero vector alone, so assume n > rn. Since Tl is of the form (5.1.1), Lemma 5.2.1 implies that the minimal polynomial of Tl is of degree rn. By Lemma 5.3.3b, the minimal polynomial of Tl may be identified with the generator polynomial g(x) of the code. It follows that the matrix T which corresponds to S in the

FURTHER THEORY OF ERROR CORRECTING CODES

155

argument of the proof of Theorem 5.3.4 is the same as T I . The final assertion of (b) now follows. As an example, consider the code associated with Fig. 5.2.1. Since the first column of the matrix A of that figure is

A is of the form (5.3.1). The generator polynomial of the code is g(x) = 1 x3 x4. Also, x15 - I ( = x15 1) = h(x)g(x), where h(x), determined from the first row of A, is 1 x3 x4 xs + 9 + x9 + x10 + xl1. The code word w = (1001 10000000000) corresponding to g(x), together with 10 cyclic shifts of w, generate the code, that is, form a maximal linearly independent set of code words. As another example, the parity check matrix

+ + + +

+ +

generated by the matrix

L1

0 1 0 0 0-

0 1 0 0 -

0 0 1 0

T=

0 0 0 1 -1

0 1 0-

(see Fig. 5.1.2) is of the form (5.3.1). The generator polynomial of the code isg(x) = 1 x2 x4, and x6 - 1 = h(z)g(x) where h(x), determined from the first row of A,, is 1 x2. The words (101010) and (010101), corresponding to g(x) and xg(x), generate the code. As a final example, let

+ +

+

156

INFORMATION THEORY

be the parity check matrix whose columns consist of two repetitions of the cycle of length 3 in Fig. 5.1.2 (see p. 138). The minimal polynomial of the matrix T of Fig. 5.1.2 is f(A) = 1 A2 A4, but the rank of A, is only 2. By Lemma 5.3.3, the generator polynomial g(x) is a proper factor of f(x), necessarily of degree 2 since the degree of the generator coincides with the rank of the parity check matrix. Since

+ +

+ +

we have g(x) = 1 x x2. Hence a maximal linearly independent set of code words consists of (1 11000),(01 1loo), (001 1lo), and (0001 1 1). Since the number of check digits is 2 and the generator polynomial is 1 x x2, a parity check matrix A may be generated by the matrix

+ +

T=

[i :]

and the vector x =

.

Since the code word length is 6,

we have

(which coincides with the first two rows of A,). Thus h(x) = 1 x3 x4,

+

+x +

5.4. Bose-Chaudhuri-Hocquenghem codes In this section we describe a class of multiple-error correcting codes that are very easy to construct and that perform very well at least up to word lengths of several thousand. The procedure to be described yields, for any e and q, an e-tuple-error correcting code with word length n = 2q - 1. Let T be a q by q matrix of the form (5.1.1). Assume that the minimal polynomial of T satisfies the conditions of Theorem 5.2.6 so that T has maximal period. The parity check matrix A , of the Bose-Chaudhuri-Hocquenghem code is shown in Table 5.4.1 (x is any nonzero vector). As an example, suppose e = 3, q = 4, and the minimal polynomial of T is f (A) = A4 A 1. The associated Bose-Chaudhuri-Hocquenghem code is shown in Table 5.4.2. It may be verified that the Bose-Chaudhuri-Hocquenghem codes are cyclic (see Problem 5.3). To establish that the matrix A, does in fact define an e-tuple-error correcting code, we prove the following theorem.

+ +

Theorem 5.4.1. Every set of 2e columns of the Bose-ChaudhuriHocquenghem matrix A, is linearly independent.

FURTHER THEORY OF ERROR CORRECTING CODES

157

Parity check matrix of the Bose-Chaudhuri-Hocquenghem code

Table 5.4.1. -

x

A, =

Tx

T2x

T3x

T3x

7'6~

TsX

~5~

p O X

~

~2e-lX

~4e-ZX

A, is qe by n,

1

... 6 . . .~

... = 2q - 1

~6e-SX

n

-..

T n-lx ~ 3 ( n - 1 ) ~

T5(n-UX

~(Ze-l)(n-l)~ -

.. ., Tkrx add to

Proof. Suppose the columns headed by Tklx, Tkax, zero, with r j 2e. Then

Table 5.4.2.

Parity check matrix for a triple-error correcting Bose-Chaudhuri-Hocquenghem code

158

INFORMATION THEORY

Each equation of (5.4.1) is of the form p(T)x = 0 where p(T) is a matric polynomial. We claim that p(T) must be zero, for if p(T) were nonzero, thenp(T) would be nonsingular by Lemma 5.2.3. If the equation p(T)x = 0 is premultiplied by the inverse ofp(T) we obtain x = 0, a contradiction. Thus we may "get rid of" the vector x in (5.4.1). If for the sake of abbreviation we replace Tki by y,, we obtain

+

+

+

We claim that y12k yZzk . . yrZk= 0 for k = 1, 2,. . . , e. To see this, we square both sides of the first equation of (5.4.2) and note that the cross terms y,y, y,y, are zero (modulo 2). Hence

+

[What we are saying here is that if p(S) is any polynomial in the binary . . y,' = 0 matrix S, then [p(S)I2 = p(S2).] Similarly, to obtain y14 we square both sides of y12 . . y,2 = 0 ; to obtain y16 y,B = 0, we square both sides of y,S . . y,3 = 0, etc. Thus we have

+

+

+

+

+- + + - .+

We write only r equations in (5.4.3) so as to obtain a square array. The equations (5.4.3) may be written in matrix form as follows:

FURTHER THEORY OF ERROR CORRECTING CODES

159

Equation (5.4.4) is of the form By = 0, where the components of the (Theorem matrix Band the vector y are elements in the Galois field GF(2-)' 5.2.4). The elements y,, . . . ,y, are nonzero; hence the equations Bz = 0 have a nontrivial solution in the GaloisJield GF(2'-'). Consequently the determinant of B is zero. But det B is a standard determinant called the Vandermonde determinant; its value is Hi,, (y, - y,). [To see this, note that det B is a homogeneous polynomial g(y,, y,, . . . , y,) of degree 1 2 - . . (r - 1) = r(r - 1)/2. If yi = y,, then columns i and j of det B are identical, hence det B = 0. Thus for all i # j, Y, - y j is a divisor of g(yl, . . . , y,). Since the degree of

+ +

+

rI(Y, - Y,) i>j

is r(r - 1)/2, the result follows, u p to a constant factor.] Now in the present case the elements y,, y,, . . . y,, being different powers of the matrix T, are distinct. (See the remarks after Theorem 5.2.6.) Hence none of the products y, - y,, i > j, is zero. The conclusion is that det B # 0, a contradiction. Thus our assumption that a set of 2e or fewer columns is linearly dependent is untenable. Notice that if r were greater than 2e the above proof would break down, since we would have no way of proving that

The matrix A, of Table 5.4.2 is 12 by IS, but the rank of A , is less than 12 since row 10 of the matrix consists entirely of zeros, and rows 11 and 12 are identical. I t is possible to give an explicit and computarionally feasible procedure for obtaining the rank of A,. First we establish a preliminary result. Lemma 5.4.2. Let T be a binary matrix whose minimal polynomial is irreducible. Ifp(T) is any matric polynomial in T such thatp(T) is not the zero matrix, then the minimal polynomial of p(T) is also irreducible. Proof. If the minimal polynomial f ( l ) of p(T) is not irreducible, then there are nonzero polynomials g(l) and h(l), of lower degree than f(2), such that f (1) = g(l)h(l). Replacing ilby S = p(T) we obtain g(S)h(S) = f(S) = 0. Now g(S) = g(p(T)) may be expressed as a matric polynomial in T; since g(S) is not zero, we may (by Lemma 5.2.3) premultiply by the inverse ofg(S) to obtain h(S) = 0, which contradicts the minimality of f ( l ) . This proves the lemma. [For those familiar with algebra, what we are saying here is that the minimal polynomial of any element u of a field Fl, such that a is algebraic over a field F c F,, is

160

INFORMATION THEORY

irreducible. In this case, a = p(T), F is the modulo 2 field, and F, is the Galois field GF(2v) where q is the degree of the minimal polynomial of T. (See Theorem 5.2.4.)] The next theorem allows the precise determination of the rank.

Theorem 5.4.3. Let A, be the Bose-Chaudhuri-Hocquenghem parity check matrix. If f,(A) is the minimal polynomial of T' (i = 1, 3, 5, . . . 2e - l), and f(A) is the least common multiple of thef,(A), that is, the polynomial of lowest degree which is a multiple of each f,(A), then the rank of A, is the degree off (A). Furthermore, if i # j, thenf,(A) and fj(A) are either relatively prime or identical, so that f(A) is the product of the distinct-members of the set {f,(A), i = 1, 3, . . . , 2e - 1). Proof. Suppose that we have a finite collection of cyclic codes with parity check matrices Ai and generator polynomials g,(x), each code having words of length n. A binary sequence w represented by the polynomial b(x) satisfies AiwT = 0 for all i if and only if b(x) is a multiple of each g,(x). Now it is immediate from the definition of a cyclic code that the intersection of the codes defined by the matrices Ai is cyclic. Furthermore, the above argument shows that the associated generator polynomial g(x) is the least common multiple of the gi(x). To obtain the BoseChaudhuri-Hocquenghem codes we may take

by Lemma 5.4.2, the minimal polynomial f,(A) of Ti is irreducible; by Lemma 5.3.3c, f,(x) = g,(x) for each i, and consequently f(x) = g(x). Since the rank of the parity check matrix of a cyclic code is the degree of the generator polynomial of the code (see Theorems 5.3.1 and 5.3.2), the first part of the theorem is proved. Now iff,(l) andf,(A) are not identical, they must be relatively prime, for if f,(A) and f,(A) were to have a nontrivial common factor, the polynomials could not be irreducible, a contradiction. The theorem is proved. For the code of Table 5.4.2, we find (see Problem 5.9) that

so that the rank of A, is 10. The rank of the submatrix A,, which by the above argument is the degree off,(A), is 4 for i = 1 and 3, and 2 for i = 5. Thus two rows of the matrix A, must be linearly dependent on the other rows of A,, which agrees with our previous observations that row 10 of A, contains only zeros, and row 11 is identical with row 12.

FURTHER THEORY OF ERROR CORRECTING CODES

161

A procedure for obtaining the minimal polynomial of the matrix T' without actually computing the powers of T is indicated in the solution to Problem 5.9. If 2q - 1 is prime, then the rank of each matrix A, is q. (As above, Lemma 5.4.2 implies that the minimal polynomial of T q s irreducible; Theorem 5.2.5 then implies that T 9 a s maximal period, and consequently the columns of each A, constitute a permutation of the set of all 2" - 1 nonzero q-vectors. Thus all the matrices A , have the same rank. But the minimal polynomial of T is of degree q; therefore A, has rank q.) However, whether or not 2q - l is prime, it is possible forf,(l) and f,(l) t o coincide when i # j. For example, if q = 5, then T5 and T9 have the same minimal polynomial. For ifp(S) is any polynomial in the binary matrix S, p(S2) = [p(S)]Z. [See the discussion after (5.4.2).] Thus if p(T5) = 0, thenp(TlO)= p(T20) = p(T40) = p(T9) = 0. (Note T ~ ' - = ~ T31 = I ) . Sim= p(T5) = 0. Thus the set of ilarly, if p(T8) = 0, then p(T18) = polynomials annulled by T5coincides with the set of polynomials annulled by F'. Similarly, if q = 6, we may verify that T13 and TI9 have the same minimal polynomial. 5.5.

Single-error correcting cyclic codes; automatic decoding

A shift register may be used to generate a cyclic code that corrects single errors and at the same time entirely eliminates the storage problem

""'I d (ri,,r2,,...,rnP) Fig. 5.5.1. Automatic decoder for the cyclic code of Fig. 5.2.1.

162

INFORMATION THEORY

at the decoder. As an example, consider the shift register and associated code of Fig. 5.2.1. An automatic decoding circuit may be constructed as shown in Fig. 5.5.1. The original shift register of Fig. 5.2.1 is combined, through a number of AND gates, with another set of binary storage devices Go, G,, G,, G,. The Gi are constructed so that their state changes if a "1" input is applied. In other words the content of the G register after application of an input

y=' i r~

Y3

is the modulo 2 sum of y and the content of G before application of the input. In order to examine the operation of the decoding circuit, suppose that the received digits r,', r,', . . . , r,' arrive at times t = 1, 2, . . . , n respectively. The columns of the parity check matrix are x, T x , T 2 x ,. . . , Tn-lx, where x is some nonzero vector.

(In Fig. 5.2.1, x =

El.)

The F shift register is adjusted so that its content at time t = 0 is x. The G register is initially set to have content zero. At time t = 1. digit r,' appears on the input line. If r,' = 0, the AND gates will be closed and the G register will still have content zero. If r,' = 1, the AND gates will allow the content of the F register to be transferred to the G register. In short, the content of G at t = 1 will be r,'x. At t = 2, F has content Tx and r,' appears at the input line; hence the input to G is r,'Tx. The content of G at t = 2 is therefore rllx rZfTx.Continuing in this fashion, the content ofGatt=nis rllx rzlTx - . . rn'Tn-lx, which is the corrector c associated with r,', r,', . . . , r,'. If c = 0, the received sequence is accepted as the input code word. If c # 0, we employ a comparison circuit (not shown in Fig. 5.5.1) to determine which digit is in error. The corrector is compared with the content of F as F cycles through the entire set of columns of the parity check matrix. If there is a correspondence at column j, that is, if c = Tj-Ix, then a single error in digit j is assumed.

+ +

+

+

FURTHER THEORY OF ERROR CORRECTING CODES

163

This brief discussion does not do justice to the subject. For further details concerning efficient encoding and decoding of cyclic codes, see Peterson (1961). 5.6. Notes and remarks Cyclic codes are treated in detail by Peterson (1961), who uses the machincry of abstract algebra. Theorems 5.3.1 and 5.3.2 form the basis of Peterson's approach. A feedback shift register is a special case of a device called a "linear sequential circuit"; see Elspas (1959), Friedland (1959), and Friedland and Stern (1959). Such devices were applied to the construction of error correcting codes by Abramson (1961), Melas (1960), and others, but Theorem 5.3.4 does not seem to have been stated explicitly anywhere. Thc results of Section 5.4 are due to Bose and Chaudhuri (1960) and Hocqucnghctn ( 1959). Bose-Chaudhuri-Hocquenghem codes can be constructed in the nonhinary case, but for the proof of Theorem 5.4.1 to remain valid, additional blocks generated by T2x, T4x,. . . , TZexmust be added to A,,. Thus in the nonbinary case the number of check digits for a given word length is greatly increased. Vcry little is known about the asymptotic properties of Bose-ChaudhuriHocquenghcm codes. According to the data of Peterson (1961), the relationship between the word length n, the number of check digits m, and the nurnbcr o f errors corrected e, is approximately that of the Varsharmov-Gilbert-Sacks condition for values of n up to 1000. If this behavior were to be maintained for arbitrarily large n, the asymptotic properties of the codes would be very desirable. (See the remarks at the cnd of' Section 4.10). However. Peterson (1960) has proved that if e l n is kept constant, the transmission rate R = k / n (k = number of information digits) approaches zero as n approaches infinity. But this does not necessarily imply that the asymptotic properties are bad. The difficulty is that we have proved that the errors corrected by the codes include at least the errors of magnitude r o r smaller. It is possible that sufficiently many error patterns of weight greater than e are corrected to allow a positive transmission rate to be maintained with an arbitrarily small probability of error. PROBLEMS 5.1 Give a parity check matrix for a cyclic code generated by a single cycle of a feedback shift register whosc characteristic polynomial is a. "1 Sl + l

b.

c. i"

+ a:' + a2 + A + 1 + I"+ 1 (three possible codes).

164

INFORMATION THEORY

5.2 Find the generator polynomial g ( x ) of the code in Problem 5.lc which uses words of length 7, and find the polynomial h(x) such that h(x)g(x) = x7 - 1 . 5.3 Show that the Bose-Chaudhuri-Hocquenghem codes are cyclic. 5.4 Give the parity check equations for a double-error correcting BoseChaudhuri-Hocquenghem code with a code-word length of 31. 5.5 The binary sequence w = (101001 1) is a code word of a cyclic code with 3 information digits.

a. Find a parity check matrix for the code, and a feedback shift register that will generate the code. b. Find the generator polynomial g ( x ) of the code, and the polynomial h(x) such that h(x)g(x) = x7 - 1 . 5.6 [The application of Cyclic Codes for Burst-Error Correction; Melas (1960).] An error burst of width q 1 beginning at digit i is an error pattern such that ri and ri+, are in error, a digit r j (i < j < i q ) may or may not be in error, and all digits before ri or after r i + , are correct. For example, the error bursts of width 4 may be represented by the following binary sequences:

+

1001 101 1 1101 1111

+

ri and ri+, in error ri, ri+2, ri+3 in error ri, ri+l, ri+, in error ri, ri+2, in error.

Binary sequences like these, which describe the distribution of errors within a burst without regard to the position of the burst within the code word, are called burst-error patterns. These are 2'1-I burst-error patterns of width q 1. Now consider a code generated by a feedback shift register with maximal period, as shown below.

+

FURTHER THEORY OF ERROR CORRECTING CODES

165

Consider a typical burst-error pattern, say 1101. If the burst begins at digit 1, the associated corrector is

I:]

If the burst begins at digit 2, the corrector is

I-1-I

(T

+ T2 + T4)x =

=

Tsx.

+

+

If the burst begins at digit 5, the corrector is (T4 T5 T7)x = Tllx, etc. a. Show that a translation of an error burst by t digits results in a translation of the associated corrector by t digits. All possible burst-error patterns of width 4 or less are listed below, along with the associated correctors (assuming that the burst begins at digit 1). Burst-Error Pattern

Corrector

b. Show that if a cyclic code is generated by a maximal-period feedback shift register with q stages, each error burst of width q or less results in a distinct corrector (assuming all error bursts begin at the same digit). It follows from (a) and (b) that knowledge of the corrector and the digit at which the burst begins completely specifies the error pattern. The various burst-correcting schemes are all based on the use of additional check digits in order to determine the position of the burst. Note that a burst may begin at the end of a code word and end at the beginning of the word. In the above example, the burst-error pattern 1101 beginning at digit 14 represents an error in digits 14, 15, and 2, and results in the corrector ( I T T3)T13x = TSx. Because of the cyclic structure of the codes, bursts of this type will also be corrected. (Note that it is sufficient to assume that T has an irreducible minimal polynomial; however, the maximal-period case gives the longest code words.)

+ +

166

INFORMATION THEORY

5.7 (Abramson, 1959). A parity check matrix A is generated by a maximalperiod q-stage shift register. The matrix A is then augmented by the addition of an "all-check" digit, so that the parity check equations become

Show that the above code may be used to correct single errors and double adjacent errors. For example, if y = 3, the parity check equations might be

The matrix A is generated by a feedback shift register with characteristic polynomial R + i+ 1. A corrector

would be associated with a single error in r,. A corrector

would be associated with a double adjacent error in r, and r,. Decoding can be accomplished since the "all-check" digit distinguishes between a single error and a double error, and no two double adjacent errors result in the same corrector.

FURTHER THEORY OF ERROR CORRECTING CODES

167

5.8 (Fire, 1959). A parity check matrix A is generated by a maximal-period q,-stage feedback shift register. An identity matrix Iq2of order q, = 27, - 1 is formed. The parity check matrix for the Fire code is

The length of the code words is n = least common multiple of 2q1 - 1 and q,. q,. For example, if A is the matrix of ProbThe number of check digits is 7, lem 5.7 [q, = 3, qz = 5, n = Icm (7, 5) = 351, the parity check matrix of the associated Fire code is

+

-

loo00 loo00 loo00

01000 01000 01OOO 00100 00100 00100 OOO10 m 1 0 m 1 0

1 OOOO1 oOOO1 m 10010 11100 10111 00101 11001 01110 -01011

10010 11100

1.

Show that the Fire code will correct all error bursts of width q, or less. For example, suppose that in the above code the corrector is

-

The top five digits identify the burst-error pattern as 101, and indicate that the burst starts at digit 4, 9, 14, 19, 24, 29, or 34. Now considering only the matrix A, the corrector associated with a burst-error pattern of 101 beginning at digit 1 is

(I + T2)x =

[i]

= T6x.

168

INFORMATION THEORY

Sina

a burst-error pattern of 101 beginning at digit 3 will produce a corrector of

Hence we conclude from the last three digits of the corrector that the burst begins at digit 3, 10, 17, 24, or 31. Thus we have identified the 101 pattern beginning at digit 24, that is, a double error in digits 24 and 26. 5.9 If

I-

1 1 0 0-1 find the minimal polynomials of T3and T5. 5.10 Show that the period of a nonsingular binary matrix cannot exceed 29 - 1, where q is the degree of the minimal polynomial of the matrix.

CHAPTER SIX

Information Sources 6.1.

Introduction

We have not considered in any detail the model for a source of information. We have assumed at various times (Chapter 2) that'the information to be transmitted consisted of the outcome of an experiment involving a random variable X; if successive values of X were to be transmitted, the values were assumed to be chosen independently. In other words, an information source has at times meant for us a sequence of independent, identically distributed random variables XI, X,, . . . . At other times (Chapter 3) we have completely ignored the statistical properties of the source and focused instead on the assignment of a code word to all possible sequences the source could produce in a given interval of time; we required that the resulting code have a probability of error which was uniformly small for all possible code words so that statistical fluctuation of the source would not affect performance of the decoding scheme. In this chapter we examine more closely the properties of information sources. We shall find that of all the sequences a source can conceivably produce in a given length of time, only a fraction are "meaningful"; the remaining sequences may be ignored when we are considering the problem of transmitting the information produced by the source. This result will allow us to take advantage of the statistical structure of a particular source to achieve a higher transmission rate for a given probability of error. We will also try to use the concepts developed in the chapter to say something about the properties of a language as a means of communication. In addition, the results will be needed in Chapter 7 in the analysis of channels with memory. The mathematical equipment we shall use is the theory of finite Markov chains, which will be developed in this chapter. 6.2. A mathematical model for an information source Let us begin by observing that whatever model we use for an information source, the source must produce a sequence of symbols

170

INFORMATION T H M R Y

If we like, we may imagine the symbols being produced at a uniform rate, one per unit of time, with X , the symbol produced at time t . The structure of the source may be described by giving the statistical dependence of X , on the past symbols X,-,, X ,-,, . . . . As an example, suppose we have a primitive language with two symbols A and B. Suppose that in a sample of text in the language, the probability distribution of X , is determined

Fig. 6.2.1. Information source.

by the previous two symbols Xi-, and Xt-, but is not affected by the additional knowledge of Xi-,, Xt-,, etc. Let us assume that the conditional distribution of X , given Xt-, and Xt-, is as follows:

I I I

P ( X , = A XtPz = A, Xi-, = A ) = 114 P ( X t = A Xt-, = A, Xi-, = B ) = 115 P { X t = A X,-, = B, Xi-, = A ) = 117 P { X , = A ( Xi-, = B, X,-, = B) = 213 P { X t = B Xt-2 = a, X,-, = 8 ) = 1 - P { X , = A Xi-, = a, X,-I = /I) (a, p = A or B) P { X t = A ( Xt-3 = a, = B, Xt-l = Y) = P { X t = A ( Xt-, = b, =y) (a, B, Y = A or B) and so on. The source just described may be characterized by four "states" and a set of "transition probabilities." (See Fig. 6.2.1.) The source is in state 1 at time t - 1 if X,-, = X,-, = A, in state 2 if X,-, = A, X,-, = B, etc. If the source is in state 2 at time t - 1 , then with probability 115 the source moves to state 3 at time t and X , = A ; with probability 415 the source moves to state 4 at time t and X , = B.

I

I

INFORMATION SOURCES

171

An object of the type shown in Fig. 6.2.1 is called a Jinite Markov chain. More precisely, given a finite set S = {s,, . . . ,s,) called the set of states and a matrix rI = [ p , , ] , i, j = 1, . . . , r, called the transition = 1 for all i, we define a matrix, satisfying p i , 2 0 for all i, j ; & p i , sequence of random variables Z,, Z,, Z,, . . . as follows. Each random variable Zi takes on values in the set of states S. Let 2, have an arbitrary distribution. Having specified the joint distribution of Z O , Z 1 ,.. . , Z n we define P{Z,+, = s, = s,,, . . . , Z n = s,,} = p .a,k rthis determines the joint distribution of Z,, . . . , Z,,,. The sequence of random variables Z,, Z,, . . . is called a j n i t e Markov chain. A square matrix whose elements are nonnegative and whose row sums are unity is called a Markov matrix or stochastic matrix; any such matrix may serve as the transition matrix of a Markov chain. Intuitively we may think of the process beginning at time t = 0 by choosing an initial state in accordance with a specified probability distribution. If we are in state si at time t = 0 then the process moves at t = 1 to a (possibly) new state, the state being chosen using the matrix ll; the number p,, is the probability that the process will move to s , at time t = 1 given that we are in state si at time t = 0. If we are in state s , at t = 1 we move to s, at t = 2 with probability p,,; in this fashion we generate a sequence of states. The random variable Z n represents the state of the process at time t = n ; we have constructed the process so that the distribution of Z n depends only on the state at time n - 1 (and on the given matrix II) and not on the state at times t < n - 1. Formally, we have the Markou property:

Iz,

1

P { Z , = an 2,-, = a,-,,

. . . ,Z , = a,)

= P(Zn = a , IZn-, = an-,) for all a,, ul, . . . , un E S. (6.2.1)

The information source of Fig. 6.2.1 could be characterized as a finite Markov chain with transition matrix

together with the specification that if the source enters state 1 or 3 at time t, an A is produced at that time, and similarly if the source moves to state 2 or 4 at time t, a B is produced. We may describe this situation by considering a function which assigns to the states 1 and 3 the letter A and to the states 2 and 4 the letter B. We are thus led to the concept of a Markov information source.

172

INFORMATION THEORY

By a Markov information source we shall mean a finite Markov chain, together with a function f whose domain is the set of states S and whose range is a finite set F called the alphabet of the source. (In the example of Fig. 6.2.1, S has 4 members and r = { A , B}.) In Section 6.4 we are going to impose an additional requirement on the Markov chain, so the above description is not to be considered a formal definition of a Markov information source; such a definition will be given in that section. In Section 6.4 we shall also discuss more general information sources.

6.3. Introduction to the theory o f j n i t e Markov chains We are aiming at an eventual definition of the "uncertainty" associated with an information source. For this we shall need several concepts from Markov chain theory. Let us for the moment forget about the alphabet of the source and analyze a finite Markov chain with states s,, s,, . . . ,s, and transition matrix H. As an example, consider the chain of Fig. 6.3.1. Suppose that we start at t = 0 by assigning an initial distribution to the states, say P{Zo = s2) = 213. P{Zo = s,) = 113, (2,= state at time t ) Then the distribution of states at t = 1 is

For a general chain with r states, let

win' = P{Zn = sj} = probability of being in state s j at time t = n.

Fig. 6.3.1. A finite Markov chain.

173

INFORMATION SOURCES

At time t = n - 1, the state of the chain must be 1, 2, we may write w!q)

. . . , or r .

+ ,$-l)pZj + . . . + win- pTj.

= w~n-l)pl,

Hence (6.3.1)

In matrix form, (6.3.1) is

If we let W ( " )be the "state distribution vector" at time n, that is, W ( " )= . . . w,(")],then (6.3.2) becomes [win)

WF

By iteration of (6.3.3) we obtain

In other words, the state distribution vector at time n is the product of the state distribution vector at the initial time t = 0, and the nth power of the transition matrix. In particular, suppose we start the chain in a specific state si at t = 0. Then

where the "1" occurs in position i. From (6.3.4) we see that W'") is the ith row of Hn, that is,

Thus the element in row i and column j of I I n is the probability that the chain will be in state j at time t = n, given that the chain is in state i at t = 0. In symbols,

Thus ply) is the "n-stage transition probability" from state i to state j. It immediately follows that I I n is a stochastic matrix, that is, p!y) 2 0 for pi;) = 1 (i = 1, . . . ,r ) . all i, j and

174

INFORMATION THEORY

Let us return to the example of Fig. 6.3.1. A direct computation yields

An examination of 114 reveals that p z ' = p): and pfi' = p z ' ; in other words, the probability of being in states, at time t = 4 is almost independent of the starting state. It -appears as if for large n, the state distribution at t = n will approach a "steady state" condition; the probability of being in a particular state s j at t = n will be almost independent of the initial state at t = 0. Mathematically, we express this condition by saying that limp$)=wj

(i,j=l,

..., r),

(6.3.6)

n-rm

where wj is independent of i. The number w, is called the steady state probability of state s,. If there is a set of numbers w,, . . . , w, such that limp:;) = w, (i, j = 1, 2, . . . ,r) then we shall say that steady state n-

m

probabilities exist for the chain. We shall presently investigate the conditions under which (6.3.6) is valid; it does not hold for an arbitrary Markov chain. For the moment we concentrate on the consequences of the existence of steady state probabilities. Theorem 6.3.1. Suppose that for a Markov chain with transition matrix II = [p,,], i, j = 1, . . , r, we have steady state probabilities w, ( j= 1, . . . , r). Then

.

b. W = [w,, . . . , w,] is a "stationary distribution" for the chain, that is, WII = W. Consequently, if the initial distribution is W(O) = W then W'") = W for all n. c. W is the unique stationary distribution for the chain, that is, if Z = [z,, . . . ,z,], with all z, 2 0 and zi = 1, then ZII = Z implies

z=w.

z;=l

175

INFORMATION SOURCES

Proof. T o prove (a), we use the fact that &p$) to approach infinity, we obtain

= 1. Allowing n

To prove (b), we note that

+ ~ 1 2 " '+~ ~ .~. + pi,"'pTj( i , j = 1, . . . , r).

p'?+" =

'

23

(6.3.7)

The above formula may be derived by probabilistic reasoning or by observing that (6.3.7) in matrix form is simply

Keeping i and j fixed in (6.3.7), let n + a. We obtain

or in matrix form, W = W n . Finally, if the initial distribution is W(O)= W, then =)

~

(

1

~

(

2= )

w(o)n= wn = w w(l)n= wn = w

To prove (c), we note that if ZII = Z , then

zlP;;'

Let n

-+

+ z 2 p ; ~+) . . . +

2r p(n) T3

=zj

j = I, . . . , r

(6.3.8)

. . . , r).

(6.3.9)

co in (6.3.8), to obtain zlwj

+ z z w j + . . + z T w j = zj

( j= 1

Since 2;=l zi = 1 , (6.3.9) yields w j = z j for all j, completing the proof. (Note that the assumption that the zi are 2 0 is not used.) Theorem 6 . 3 . 1 ~ is very useful in the actual calculation of w l , . . , wT.

.

176

INFORMATION THEORY

We remark that the condition lim n-

PI;)

=

w, for i, j = 1, 2,

. . . ,r

is

m

equivalent to the statement that for each j, column j of IIn approaches a vector all of whose components are w j , that is,

alternately the matrix IIn approaches a matrix with identical rows, that is,

We now return to the problem of finding conditions under which steady state probabilities exist for a finite Markov chain. We have the following general result. Theorem 6.3.2. Let TI be an r by r transition matrix associated with a finite Markov chain. Then steady state probabilities exist if and only if there is a positive integer N such that I I N has a positive column, that is, a column all of whose elements are >O.

The above condition is equivalent to the existence of a state s, and a positive integer N such that starting from any initial state we can reach sj in N steps. An immediate corollary of Theorem 6.3.2 is the result that if n has no zeros, that is, every state may be reached in one step from any other state, then steady state probabilities exist; more generally, if for some n, TIn has no zeros, then steady state probabilities exist. Before proving Theorem 6.3.2 let us take an example. Consider a chain with transition matrix

To determine whether or not the condition of Theorem 6.3.2 is satisfied, we need not worry about the exact values of the transition probabilities

177

INFORMATION SOURCES

' :I=[:

p,,; we need only know which probabilities are strictly positive. Indicating positive elements by x and zero elements by 0 we compute

nz=

x xO x 2 x 0

x x o

:1,

x x x

which has a positive column. (In fact, n3has all elements >O.) To find the steady state probabilities w,, . . . , w r we use the fact that W = [w,,. . . , w,] is the unique stationary distribution for the chain (Theorem 6.3. lc). The equations WII = W become

From (6.3.10) we obtain w, = $w, and wl = gw,. Since w, must be 1 we have W 1 = 9, W z = 7, W3=281.

+ w, + w,

f

We remark that if IIN has a positive column, say column j, then column j of nn is positive for all n 2 N. (Consider nN+'= II IINand proceed inductively.) Proof of Theorem 6.3.2. First suppose that IIN has a positive column, say column j,,. If we write the equations corresponding to the matrix relation IIn+N = I I N I I n we obtain r

p:;+N)

= ~ p $ " ) p ~ ) i,; j = 1, .

. .,r ;

n = 0, 1,2, . . . .

(6.3.11)

lr=l

Let i, m, j be any integers

E

{1,2, . . . , r ) . From (6.3.1 1) we have

Define sets S , and Szby S , = {k: p/:' S , = {k:

- p!,") 2 0)

pr' - p::)

< 0).

We may then write (6.3.12) as (n+N) Pit

-

(n+N)

Pmj

(N) - (W (n) ( N ) - ( N ) (n) - 2 [pik ~ k 1 kP k j + 2 [ ~ i k Pmk 1 P k j ' kr.5'1

k€Sz

(6'3'13)

178

INFORMATION THEORY

Now let

M$") = max pi,")= the largest element in column j of IIn l 5 i l C

m y ) = min pi:) = the smallest element in column j of ITn. ll,
is

Let qk = ( p i f ' - ~2'1; note that

Hence we may write (6.3.14)as

~ i ( i n + ~-) Pmj (n+N) < (MY)- m y ) )

xq,= ( M y ' - m y ) )2 qk.

kfSl

(6.3.15)

kfRa

Now since column j, of ITN is positive, there is some E > 0 for which pi:' 2 E > 0 for s = 1,2,. . . , r (we may take E = m::)). If jOhappens to belong to S,, then

On the other hand, if jo c S, then

Thus in either case we have, from (6.3.15), (n+N) Pi j

(n+N) - pmi

I; ( M : . ~-) mjn))(1- E).

Since (6.3.16) holds for arbitrary i and m we have

INFORMATION SOURCES

179

Now we note that the sequences Mjn' and min', n = 1,2, . . . are monotone. To see this, suppose that pi;) is the largest element in column j of IIn, that is, pi'$ = Mjn'. Then, using the fact that I I n = I I I I n - I we have

so that M:"', n = 1,2, . . . , is a nonincreasing sequence. Similarly m:"', n = 1,2, . . . , is a nondecreasing sequence. Hence - men' , n = 1,2, . . . , is a nonincreasing sequence; since the M,!"' and m?' are probabilities, 0 j mjn' j Mjn' j 1. But every monotone bounded sequence converges so that

Nn'

,

lim My),

lim m p ) ,

lim [My) - m y ) ]

n-tm

n-r m

n-tm

all exist. But lim (MY) -

= lim (MyN)

n-m

- myN))= 0

I-00

by (6.3.19). Hence Iim n-rm

MY) = nlim my' -rm

= (say) w,.

pi;'

= W , and steady state probM,'"' for all i, lim Since m;") 5 n- m abilities exist. Conversely, suppose that steady state probabilities w,exist. By Theorem w, = 1 so that at least one w,,say wjo,is >O. Now 6.3.la,

z;,,

limp!~~=w,,

( i = l , 2 , . . . , r).

n-rm

Hence for each i there is an integer n, such that p&' > 0 for n 2 n,. If N = max n, then n 2 N implies pi: > 0 for all i; hence column j, of l0, and the proof is complete. Notice that if column j, of IIN is positive then m:$' > 0, hence m::' 2 mi:' > 0 for n 2 N; thus wio = lim m::' > 0. In particular, we have n- m

the following corollary to Theorem 6.3.2. COROLLARY 6.3.2.1. If for some N, IIN has all elements positive, then all steady state probabilities are positive. As an example of a chain with states having a steady state probability of zero, consider the chain characterized by the transition matrix (6.3.20)

180

INFORMATION THEORY

T o compute w,, w,, w3 we write the equations WII = W :

The solution to (6.3.21) is w, = 0, w, = $, w, = 4. The significance of the zero value for w, is that s, is a "transient state"; the chain must eventually reach a point where a return to s, is impossible. We shall have more to say about transient states at the end of this section. A question arises as to whether the criterion of Theorem 6.3.2 is effective. Suppose that we wish to determine whether or not a given transition matrix has steady state probabilities. If the 1 matrix satisfies the criterion of the theorem, then Fig. 6.3.2. A Markov by computing the powers of the matrix we will be ,-hain without steady state able to verify this fact in a finite number of steps. However, if the criterion is not satisfied, then probabilities. conceivably we shall never be able to come to a decision, since the fact that none of the first n powers of the matrix ])st have a positive column does not tell us anything about the (n power. It would be desirable to have a theorem which states that if the kth power of a transition matrix II does not have a positive column, then none of the matrices IIn, n 2 k, has a positive column and hence steady state probabilities do not exist. In fact, we may take k = 2", where r is the number of states of the chain associated with n. This will be a special case of a theorem to be proved in Chapter 7 (Theorem 7.2.2). It is possible for a Markov chain to have a unique stationary distribution without having steady state probabilities. For example, consider the oscillates back and forth from 0 to I , chain of Fig. 6.3.2. Since steady state probabilities do not exist. There is however, a unique stationary distribution for the chain, namely the distribution that assigns probability 112 to each of the states s, and s,. We now try to identify the class of Markov chains which have unique stationary distributions. Consider a finite Markov chain with states s,, s,, . . . ,s,. A set of states B is said to be essential if every state in B is reachable? (possibly in more than one step) from every other state in B, and it is not possible to reach a state outside of B from a state in B. It

+

pi,"'

..

t sj is said to be reachable from s, if there is a sequence of states s,,,,~,,,, . s,,,,,with s # ( ~=, s,, s,(,, = sf, such that the transition probability psiln,,i,n+l, is > 0 for n = 1, 2,

..., 1-

1.

181

INFORMATION SOURCES

is not hard to verify that any finite Markov chain has at least one essential set; see Problem 6.10. A chain is said to be indecomposable if it has exactly one essential set of states. For example, the chain characterized by the matrix (6.3.20) is indecomposable, since the states s, and s, form the unique essential class. In fact, a chain with steady state probabilities is always indecomposable. For suppose that a chain has at least two essential sets B, and B, (see Fig. 6.3.3). Suppose that B, consists of the states s,, . . . ,.,s If the process starts in B, it must remain forever in B,, and therefore Cπ' = 1 for all n. Allowing n t o approach infinity, we conclude that w, = 1 , and

z,"=,

Fig. 6.3.3. A decomposable Markov chain.

consequently at least one state s, E B, must have a positive steady state probability w,. But if we select any state si E B,, we have p:,"' = 0 for all n ; hence w, = 0, a contradiction. However, an indecomposable chain need not have steady state probabilities, as the example of Fig. 6.3.2 shows. We now prove that an indecomposable chain always has a unique stationary distribution. Theorem 6.3.3. Let an indecomposable Markov chain have states s,, . . . ,s, and transition matrix II. Then the chain has a unique stationary distribution; that is, there is a unique vector W = [w, w, . - . w,] such that all wi 2 0, wi = 1 , and WII = W. Proof. Let B be the essential class of the chain. Since it is not possible to move from a state in B to a state outside of B, the submatrix of the transition matrix corresponding to the states of B is still a Markov matrix, that is, the states of B form a Markov chain. If we can find a stationary distribution for the reduced chain consisting of the states of B, we can form a stationary distribution for the original chain simply by assigning probability zero to the states that do not belong to B. Thus for the purpose of establishing the existence of a stationary distribution, we may assume

z:=,

182

INFORMATION THEORY

without loss of generality that B is the entire set of states. We break the proof into several parts. a. It is possible to move from any state si to any state s, (s, = s, is allowed) in at most r steps. For if we have a sequence of states S ~ ( ~ ) S ,.( ~. )si(t ) where s ~ (=~s,,) s , ( ~=, s,, si(,+,) is reachable in one step from s , ( ~(n) = 1, 2, . . . , t - I), and t > r, then there is at least one state s,,,) (1 < m < t ) which appears two or more times in the sequence. Thus within the sequence we have a subsequence si(,)sasb . . sit,,. We may drop the states sasb- . . st(,) to form a shorter path from si to sj. The process may be continued until a path of length at most r results. b. If pi, > 0 for all i then all elements of IIr are

> 0.

For given any states si and s,, we know by (a) that s, is reachable > 0 for some n j r. Then it is from si in at most r steps. Thus possible to reach si from si in exactly r steps by moving from s, to itself r - n times, and then moving from si to s j in n steps. Therefore plf' > 0 for all i and j. Now we define a matrix ll' by II' = &(II I ) , where I is an identity matrix of order r. The row sums of II' are unity and the elements of II' are nonnegative, so that II' is a Markov matrix. Sincep,, > 0 implies pi,' > 0, every state in the chain defined by II' is reachable from every other state, so that the chain corresponding to II' is indecomposable, with the essential class consisting of all the states. Since pti' > 0 for all i, it follows from (b) that all elements of (II')' are positive. By Theorems 6.3.2 and 6.3.1, II' has a stationary distribution, that is, there is a vector Wwhose components are nonnegative and add to unity, such that WII' = I ) = 4W n g W; hence WII = W and W W. But WII' = 3 W(II also forms a stationary distribution for II. Thus the existence of a stationary distribution is established. To show that the stationary distribution is unique, we proceed as follows. First assume that B does not consist of all the states in the chain. It is possible to enter the class B from any state outside of B, for otherwise the chain would contain another essential set. (If B cannot be reached from the state s t q' B, let R be the set of states which are reachable from st. If sa E R and s, is reachable from s,, then s, must be reachable from st, hence s, E R. Thus the states of R form a Markov chain, which must have at least one essential set, necessarily disjoint from B.) Thus there is a positive integer k such that P{Z, E B 1 Zo = s,) > 0 for all si # B. (Of course P(Z, E B 1 Zo = si) = 1 for si E B since it is impossible to leave B.)

+

+

+

183

INFORMATION SOURCES

Let E = max P{Zk f B I Zo = si} < 1. Then for any state si, l
<

I

-

By iteration, we obtain P{Zm, f B Zo = s,} E" -+ 0 as m co. Now given any positive integer n, let m be the largest integer such that mk n. Since it is impossible to leave B, Z n f B implies Z,, f B; consequently

Since m + a as n

-

<

I

w , lim P { Z , f B Zo = si} = 0. In particular if n-- a

s, f B, lim pi;' = 0 for all i. Thus all states outside of B have a steady n- m

state probability of zero. It follows that any stationary distribution must assign probability zero to these states. (If s, f B, let n -+ w in the jth equation of W I I n = W to obtain w j = 0.) Therefore we may ignore the states not belonging to B, in other words we may again assume without loss of generality that B consists of all states in the chain. We now form the matrix II' as in the proof of the first part of the theorem. We claim that a vector forms a stationary distribution for n' if and only if it forms a stationary distribution for I I . We have already seen that Wn'= W implies W n = W. Conversely, if W n = W then writ= gw(n + ~ ) = g w + n g w = g w + gw= w. B U ~ n ' has steady state probabilities, and therefore the stationary distribution for H' is unique by Theorem 6.3.1. Thus the stationary distribution for n is unique. (In addition, if the essential set consists of all the states of the chain, the fact that all elements of (n')' are positive implies by Corollary 6.3.2.1 that all components of the stationary distribution are positive.) The above discussion shows that the states in an indecomposable chain can be divided into two disjoint classes; a (never empty) class B of essential states and a (possibly empty) class T of transient states. Within the class of essential states, any state may be reached from any other state; from each state of T i t is possible to reach any state of B, but it is not necessarily the case that every state of T is reachable from every other state of T. A; we have seen above, the probability that the chain will be in a state of T after n steps approaches zero as n approaches infinity, hence the probability is zero that the chain will remain forever in the set T. If the chain has steady state probabilities, then it turns out (see Feller, 1950, or Chung, 1960) that each state s of B is aperiodic in the sense that if F is

184

INFORMATION THEORY

the set of positive integers n such that it is possible to move from s back to s in exactly n steps, then the greatest common divisor of the members of F is 1. On the other hand, if the chain is indecomposable but does not have steady state probabilities, then each state of B is periodic with the same period d > 1, that is, the greatest common divisor of the members of F (the so-called "period" of the chain) is d. Note that the chain of Fig. 6.3.2 has period 2. In the periodic case, the chain corresponding to the matrix IId has d essential sets S,, . . . , S,; the original chain moves cyclically through the sets S,, . . . , S,; that is, we may label the indices so 1 belongs that if the state at time t belongs to S,, then the state at time t to Sk+,(with the indices reduced modulo d, that is, S,,, = S,, S,,, = S,, etc.). We have a weak version of steady state probabilities in the following sense. If si E Skand s j E Sk+T,t h e n ~ i j n ~approaches +~) a limit wi, independent of s, E Sk, as n -+co. In general, any finite Markov chain has a stationary distribution, but if the chain is not indecomposable, the distribution cannot be unique. For if the essential sets of a chain are B,, . . . , Bk, the chain formed by considering only the states of Bi is indecomposable, and hence has a stationary distribution. If we assign probability zero to the states outside B,, we have a stationary distribution Wi for the original chain (i = 1, 2, . . . , k ) . Since the Wi are distinct, we have found k different stationary distributions. In fact, if W = aiWi where the a , are nonnegative numbers whose sum is unity, then

+

z:=,

so that there are uncountably many stationary distributions. Thus the class of indecomposable jinite Markov chains coincides with the class of jinite Markou chains having a unique stationary distribution. We close the discussion at this point, referring to Feller (1950), Chung (1960), or Kemeny and Snell (1960) for further details. 6.4. Information sources; uncertainty of a source

In defining what we mean by a source of information, we try to capture the physical idea of a mechanism which emits a sequence of symbols in accordance with specified statistical rules. The symbols are assumed to belong to a definite finite set, the "alphabet" of the source, and the statistical properties of the source are assumed to be time-invariant. Definition. An information source is a sequence of random variables X,, X,, X,, . . . , such that

INFORMATIONSOURCES

185

1. Each X, takes values in a finite set I',called the alphabet of the source, and 2. The sequence is stationary, that is,

for all nonnegative integers i,,

. . . ,i,, h, and all y,, . . . ,y, E I'.

An information source may be derived from a Markov chain as follows. Given a finite Markov chain Z,, Z,, . . . , and a function f whose domain is the set of states of the chain and whose range is a finite set I', assume that the initial state Z, is chosen in accordance with a stationary distribution W = [w,, . . . , w , ] ; that is, P{Z, = s,) = wj for all states s,. Then the stationary sequence X, =f(Z,), n = 0, 1, . . . , is said to be the Markov information source corresponding to the chain Z,, Z,, . . . , the fucction f,and the distribution W. (Of course if the chain is indecomposable there is only one stationary distribution, and the source is determined by the chain together with the functionf Note also that all states s, such that w, = 0 may be removed from the chain without affecting the distribution of the sequence {X,).) If the chain has steady state probabilities, we call the resulting information source a regular Markov information source. (We may also call a Markov chain with steady state probabilities a regular chain.) If the chain is indecomposable, the resulting information source will be called an indecomposable Markov source. Physically, we might expect that if an information source is in operation for a long period of time, a steady state condition should be reached. Thus Markov sources met in practice should have steady state probabilities. However, it is convenient mathematically to allow an information source to be derived from an arbitrary Markov chain. This will avoid annoying difficulties in Section 6.5, when we approximate an arbitrary information source by a Markov source. The approximating sources need not have steady state probabilities and in fact are not necessarily indecomposable (although if the properties of the given source are "reasonable," the approximating sources will also be "reasonable"; see Problem 6.9). We remark that a sequence of independent, identically distributed random variables X,, XI, . . . ,taking values in a finite set S = {s,, . . . , s,), becomes a regular Markov information source if we take pi, = w, = P { X , = s,), i, j = 1,2, . . . , r, k = 0, 1, . . ; we may take I' = S, and f = the identity function. We now try to arrive at a meaningful notion of "uncertainty" of an information source.

.

186

INFORMATION THEORY

We are going to define the source uncertainty as (intuitively) the uncertainty of a particular symbol produced by the source, given that we have observed all previous symbols. Formally, we have the following. Definition. Given an information source Xo, XI, . . . , the uncertainty of the source, denoted by H{Xn, n = 0, 1, . . .) or for brevity by H { X ) , is defined as lim H(Xn Xo, X I , . . . , X,-,).

I

n-m

Note that since

H(Xn 1 Xo, XI, . . . , X,-I]

5 H(Xn 1 XI, . . . , Xn-d by Theorem 1.4.5 or Problem 1 . 4 ~

n 1 X , . . . ,X

=X

by stationarity,

I

the sequence ( H ( X n Xo, X I , . . . , Xn-,), n = l , 2 , . . .) is nonincreasing. Since the terms of the sequence are nonnegative, the limit H { X ) exists. In addition,

I

H { X ) I H(Xn Xo, . Xn-1) j H(X,) by Theorem 1.4.5

1

5; log I?[

by Theorem 1 A.2,

where is the number of letters in the alphabet I?. We remark also that if { X , ) is a sequence of independent, identically distributed random variables, then H { X ) = H(XJ (for any i). It is not apparent at this point exactly what properties of an information source are revealed by knowledge of its uncertainty. This matter will be clarified in Section 6.6. First we give an alternate expression for H { X ) . Theorem 6.4.1.

If Xo, X I , . . ., is an information source, then

Proof. Let gn = H ( X , I X,, X,, then by Theorem 1.4.4,

. . . , Xn-,), n = 1,2, . . . ;go = H(x,);

INFORMATION SOURCES

But by definition of H { X ) , g ,

+

H { X ) , and hence

being the arithmetic average of a sequence converging to H{X), also converges to H { X ) . We note that the sequence h,+,/(n 1 ) = H(Xo, . . . , X,)/(n 1) is nonincreasing. For by (6.4.1) we have h,,, = gi. Thus

+

z:=o

+

Since the sequence {g,) is nonincreasing we have gi 2 g , for i = 0, 1, . . . ,n - 1 . Hence ng, -;:2 ; gi ng, - ngn = 0, and the assertion is proved. The uncertainty of an information source is difficult to compute in most cases. However, for a certain class of Markov sources the calculation may be greatly simplified. Definition. Consider a Markov information source with set of states S = {s,, . . . , s,), alphabet I?, associated function f : S -+,'I and stationary distribution W = [w, w, . . . w,]. For each state s,, let s,,, s,,, . . . , sknkbe the states that can be reached in one step from s,, that is, the states s, such that p,, > 0. The source is said to be un8filar if for each state s,, the letters f(skl), . . ,f (sknk)are distinct.

<

.

Fig. 6.4.1. (a) Unifilar Markov source. Take

188

INFORMATION THEORY

In other words, each state reachable directly from s, is associated with a distinct letter of the alphabet. Equivalently, the state at time t and the symbol produced at time t 1 determine the state at time t 1. Examples of unifilar and nonunifilar Markov sources are shown in Fig. 6.4.1. In Fig. 6.4.1b, states s, and s, are directly reachable from s, but f (s,) =f (s,) = A , and therefore the source is not unifilar. Given a unifilar Markov source, as above, let s,,, . . . , sknkbe the states that are directly reachable from s,, k = 1, 2, . . . , r. We define the uncertainty of state s, as H, = H(p,,,p,,, . . . ,p,,J.

+

+

Theorem 6.4.2. The uncertainty of a unifilar Markov source is given by H{X} = ZL, W k H k . Proof. The underlying Markov chain of the source is a sequence of random variables Z,, Z,, . . . (recall that Z , represents the state of the process after n transitions). Now

x

ao.

2

. . . , a,€,$

. . . ,Z,

P{Zo = a,,

= a,}log

P{Zo = a,,

. . . ,Z ,

= a,).

Furthermore, we may write P{Zo = a,,

. . . ,Z,, = a,}

= P{Z, = a,}P{Z, = a,,

. . . ,Z,

I

= a, Z, = a,).

Now suppose that the initial state is a,, and that the state after i transitions is a i (i = 1, 2, . . . ,n); let y, = f(a,) < F. The hypothesis that the source is unifilar means that no other sequences of states a,', . . . , a,' can produce the sequence of letters y,, . . . , y,. In other words, for afixed initial state there is a one-to-one correspondence between sequences of states and sequences of symbols from the alphabet of the source. We may therefore write (6.4.2) as

x P{X1 = yl,

. . . , X,

+

I

= yn Z, = a,}

x [log P{Z0 = a,} log P{Xl = y,, . . . . X, = y,l Z , = a,)]) = (n I)-' [H(Z,) H(Xl, . . . , X , Z,)].

But

+

+

I

I89

INFORMATION SOURCES

I

Now H(Zo X,, n 4 c~ so that

. . . , Xn) 5 H(Z,) 5 log r and (n

+ I)-'

log r 4 0 as

lim H(Z0, Z1, . . ., Zn) - lim H(X1, . . . , Xn) = lim H(Xo9 . . . Xn-1) n + f n-m n+l n- w n+l o . n n = lim - H{X) n-rm n n+l We conclude that H{X) = H{Z). In this case H{Z) is easily computable. We have H ( Z ) = lim H(Z, Z,, Z , , . . . , Z,-,) 3

n-w

I

n-rm

= H(Z, ( 2,-,) =

I

H(Z, Z,) =

by the Markov property

2 P{Z,

=

I

sk)H(Zl Z, = s,)

k=l

r

=

2 wkH,. k=l

This completes the proof. For the source of Fig. 6.4. l a we may compute that the steady state probabilities are w, = 7/24, w, = 113, w, = 1/24, w, = 113. The source uncertainty is therefore H{X) = AH(O.2, 0.8)

+ iH(0.9, 0.1) + hH(0.8, 0.2) + +H(0.7, 0.3)

6.5. Order of a source; approximation of a general information source by a source offinite order

We expect that the statistical dependence of a symbol Xn produced by an rnformation source on the previous symbols XI, X,, . . . , Xn-, should co. In this section we examine unifilar Markov sources "wear off" as n in which the distribution of Xn is determined by a finite number of previous symbols Xn-,, . . . , Xn_,; such sources are referred to a sources of finite order o r j n i t e memory. We then show that given an arbitrary information source we can construct in a natural way a unifilar Markov source of finite order whose uncertainty is as close as we wish to that of the original source. Let us begin by analyzing the source shown in Fig. 6.5.1. In the diagram, iff&) = y, then all arrows entering state s j are marked with a y. The transition probabilities are unspecified in Fig. 6.5.1, except that we assume p,, > 0 if states s, and s, are connected directly. In Table 6.5.1 we have listed all possible sequences of two and three symbols, with the corresponding final states for a given initial state. For example if Z,-, = s,, X, = B, Xt-, = A , Xt = A, then Z , = s,. Blank entries correspond

,

INFORMATION THEORY

Fig. 6.5.1. Unifilar Markov information source of order three.

to impossible transitions; for example, if Z,-, = s, then it is impossible that Xt-, = B, Xt = B. It can be seen from Table 6.5.1 that the state 2, is determined by the three inputs X,-,, Xi-,, Xt regardless of the initial state Z For instance, if Xi-, = A, Xt-, = B, X , = B, then 2, = s, since all entries in column ABB are identical. However, the present state is not determined by the previous two inputs, since if X,-, = B, X , = A, then Z , can be either s, or s,.

,-,.

Table 6.5.1. State transitions for the source of Fig. 6.5.1

xt-,x t

Xt-a Xt-1 X t Zt-%

AAA AAB ABA ABB BAA BAB BBA BBB

s1

$1

s2

S1

Sa

$1

S?

S4

S8

S1

S*

S1

S*

S1

Sa

S1

St

S1

sa

-

s2 -

-

-

S4

S1

Sa

S1

Sa

S1

Sz

S4

-

Zs-2

AA

AB

BA

BB

$1

S1

sa

s1

s,

Sa

S1

S2

S4

-

S3

S1

Sz

-

-

S4

S1

Sa

s1

sa

We define a unifilar Markov source to be of order M if the state Z , is determined by the symbols X,, Xt-,, . . . , Xt-,+,, but not by the symbols X,, X,-,, . . . , X,-+ where i < M - 1. The source of Fig. 6.5.1 is therefore of order three. A unifilar Markov source is of inJinite order if it is not of order Mfor any finite M. The source of Fig. 6.5.2 is of infinite order since the sequence AAA . . . A does not determine the final state; depending on the initial state, the final state could be either s, or s,. If a source is of order M, it follows that

191

INFORMATION SOURCES

w Flg. 6.5.2. Unlfilar Markov source of infinite order.

Thus if a source is of order M, its uncertainty can be expressed as the difference of the uncertainty of M 1 successive symbols and that of M successive symbols. In general, the uncertainty H(Xl, . . . , X,) of n successive symbols produced by a source is sometimes referred to as the n-gram uncertainty of the source (For n = 1, the term "unigram" is used; for n = 2, "digram"; for n = 3, "trigram"). Thus the uncertainty of a source of order M is the difference of the (M 1)-gram and M-gram uncertainties. As a numerical example, we evaluate the various n-gram uncertainties for the source of Fig. 6.5.3, which may be seen to be of order 2.

+

+

P ( A A ) = P { X , = A and X,,, = A ) 4

= zw,P{X,= A, X,,,= AIZ,-, = s,) I-1

Fig. 6.5.3. Numerical evaluation of uncertainties.

192

INFORMATION THEORY

[Also P(AA) = P{Z,+, = s,) = w,.] P(AB) = ,',(0.2)(0.8) ,S,(O.9)(0.3)

+

+ &(0.8)(0.3) + ,",(0.7)(0.8)

= 8/24 = w,.

Similarly P(BA) = 8/24 = w,, and P(BB) = 1/24 = w,. P(AAA) = P { X , = A , X,+, = A, Xt+z = A ) = P{Z,+, = s,, X,+Z = A). P(ABA) = P { X , = A, Xi+, = B, Xt+z = A ) = P{Z,+, = s,, X,+Z = A ) ,

and so on. Thus

J1 = unigram uncertainty = -P(A) log P(A) - P(B) log P(B) = -#log g - #log #. .Iz = digram uncertainty = -P(AA) log P(AA) - P(AB) log P(AB) -P(BA) log P(BA) - P(BB) log P(BB) = H(7/24, 8/24, 8/24, 1/24). J, = trigram uncertainty = -P(AAA) log P ( A A A ) - P(AAB) log P(AAB) - . . -P(BBB) log P(BBB)

1 8 + 24 - H(0.7,0.3) + - H(0.8,0.2) 24

(by the generalized grouping axiom; Problem 1.10). H{X) = J , - Jz = zz4~(0.2,0.8) za4~(0.9,o.1) h ~ ( 0 . 80.2) , : 4 ~ ( 0 . 7 ,0.3),

+

+

+

which agrees with the expression computed at the end of Section 6.4.

193

INFORMATION SOURCES

T o determine the order of a source, we examine the final states associated with symbol sequences of length 1, length 2, length 3, etc., until we find a number M such that for all sequences of length M, the final state is independent of the initial state; M is then the order of the source. It would be desirable to know when to stop this process and declare that the source is of infinite order. A criterion is provided by the following theorem. Theorem 6.5.1. If a unifilar Markov information source with r states is of finite order M, then M 5 &r(r- 1). Proof. Let Q = (a,, a,, . . . , a,,(,-,)) be any sequence of &r(r - 1) symbols. We must show that the final state associated with Q is the same for all initial states. If u, and v, are two distinct initial states, the sequence

"1 OLfr(r-1) A B A u , - + u , - u , - ~ ~ - + U f , ( , ~ , S) 1 - S 3 - S 3 ~ S 1 vo

+

U1

- U2

' ' '

a

Utr(r-1)

S2

--S7

S4

S2

b

Fig. 6.5.4

Proof of Theorem 6.5.1

Q takes u, through a sequence of states u,, u,, . . . , ut,,,-,,, and v, through (see Fig. 6.5.4a). If ui = vi for some a sequence v,, v,, . . . ,vh i [0 < i < Br(r - I)] then u, = u, for i 5 j 5 &r(r- 1 ) . In particular, ~ r r ~ , - l )= vbr,,-,, and there is nothing further to prove. Thus let us assume that ui # 0, [0 5 i < &r(r- I)]. If for a particular i and j [0 2 i < j 5 &r(r- I)] we have u, = u, and vi = v,, then the source must be of infinite order, which contradicts the hypothesis. To see this, suppose for example that the sequence ABA takes state s, into itself, and also takes s, into itself (see Fig. 6.5.4b). Then we cannot determine the final state at the end of the sequence ABAABAABA . . Similarly we cannot have ui = v, and vi = u, for 0 5 i < j 5 ir(r - 1 ) . (If the sequence ABA takes s, into s, and s, into s,, then ABAABA takes s, into s, and s, into s,, which implies that the source must be of infinite order.) Hence if 0 i < j 5 &r(r- I), the (unordered) sets {u,, v i ) and {u,, v,} are unequal, and furthermore ui # vi for 0 i < +r(r - I).

,,- ,,

<

However, there are only

(9

= ;tr(r - 1 ) possible ways of choosing two

distinct states out of r. Consequently (u,, v,), (u,, v,), (u,, v,), . . . , ( u ~ ~ ( , - ~u)~- ,~(,, - ~ ) exhaust -~) all possible pairs of distinct states. Thus u~,,,-,, = v ~ , , , - ~and , , the theorem is proved. Now given an arbitrary information source S = { X , , n = 0, 1, . . .), we construct a sequence of finite-order unifilar Markov sources S, as follows.

194

INFORMATION THEORY

Let the states of S, correspond to the n-tuples (a,, . . . , a,) of symbols from the alphabet I' of S, such that P ( X , = a,, . . . , X, = a,) > 0. Let the alphabet of S, be I? for each n. For each state (a,, . . . , a,) of S, we may in one step reach states of the form (a,, . . . , a,, a,,,). The transition probability from (a,, . . . , a,) to (a,, . . . , a,, a,+,) is given by We define the function f associated with S , by f(a,, . . . , a,) = a,; if the state of S, at a given time is Z , = (a,, . . . , a,), the corresponding symbol produced by Sn is Y , =f ( Z , ) = a,. Thus moving from state (a1, . . . , a,) t o (a,, . . . , a,, a,,.,) corresponds t o moving from a condition in which the previous n symbols were a,, . . . , a, to a condition in which the previous n symbols are (a,, . . . , a,, an+,). We may verify that a stationary distribution is formed by assigning to the state (a,, . . . , a,) the probability P{X, = a,, . . . , X, = a,) (see Problem 6.4). With this distribution, S, becomes a Markov information source. Now if we know the state Z , of S, at a given time t , the symbol Yt+,determines the last coordinate of the state Z,,,; since the first n - I coordinates of Z,,, are the same as the last n - 1 coordinates of Z,, we conclude that 2, and Yt+, determine Z,,, so that the source is unifilar. Thus S, is a unifilar Markov source. By construction, S, has order at most n. (The order of S, may be less than n ; see Problem 6.1 1 .) To find the uncertainty of S,, we observe that if {Y,, n = 1,2, . . .) denotes the sequence of symbols produced by S,, then

Thus the conditional distribution of a letter, given n preceding letters, is the same for S, as for S. It follows that the uncertainty of S, is H,{X} = H(Y,+,

I Y,, . . . , y,)

Thus

= H(X,+,

1

lim H,{X} = lim H(X,+, X,, n+ m

n+ m

I X I . . . . , x,).

. . . , X,)

= H{X),

so that the sources S, have uncertainties which form a nonincreasing sequence converging to H { X } . S, is called the nth-order approximation to S. If the distribution of the symbol Xt depends on a finite number M of previous symbols, then the approximation S , coincides with the original source.

INFORMATION

195

SOURCES

As an example, let us compute the first-order approximation to the source of Fig. 6.5.3. (In this case the second-order approximation coincides with the source itself.) We have (p. 191)

The first-order approximation is shown in Fig. 6.5.5.

Fig. 6.5.5. First-order approximation to the source of Fig. 6.5.3.

The uncertainty of the first-order approximation is

6.6. The asymptotic equipartition property

In this section we try to give a basic interpretation to the uncertainty of an information source. We shall try to make precise the notion of a "meaningful" sequence produced by a source, and we shall show that under certain conditions there are asymptotically 2Hn such sequences, where H i s the uncertainty of the source. First consider a sequence of independent, identically distributed random variables X,, X,, . . . , taking values in a finite set I?. Define another sequence of random variables Vl, V,, . . . , as follows: If Xi = a,, . . . , X , = a,, let V , = Vn(Xl, . . . , X,) =

-n-I

log P { X l = a,,

. . .,X ,

= a,).

It follows from the independence of the Xi that V,(X,,

. . . , X,)

n

= n-'

2 U(X,), i=l

where U(X,) is a random variable that assumes the value -log P{Xi = a # ) when Xi = a,. Thus V,(X,, . . . , X,) is the arithmetic average of n independent, identically distributed random variables U(Xi),with E[U(Xi)]= - 2 P { X , = a ) log P { X , = a } = H(XJ = H { X } . u

196

INFORMATION THEORY

By the weak law of large numbers, V,(X,,

. . . , X,)

In other words, given

E

converges in probability to H = H { X ) .

> 0, 6 > 0, then for n sufficiently large,

For a fixed E and 6, let us divide the sequences (a,, a,, into two sets:

. . . , a,), ai E I',

S1 = {(a,, . . . , a,): I-n-'logP{X, = a,,.. . , X , = a,,) - HI < 6) S 2 = {(a,, . . . , a,): I -n-' log P { X , = a,, . . . , X , = a,) - HI 2 0). Then P{(X,, . . . , X,) E S,) 2 I - E. Furthermore, if (a,, . . . , a,) E S , then, writing p ( a l , . . . , a,) for P{Xl = a,,. . . , X , = a,), we have (by definition of S,) llog [p(a,, . . . , a,)]-' - n H J < nh, or z-n(H+a) < p(a,,

. . . ,a,) < 2-n'N-6'.

<

p(aI,. . . ,an)

Since 1-E

.

2

( a ~ ,. . .anF6'l

< 1,

(6.6.1) (6.6.2)

it follows from (6.6.1) and (6.6.2), just as in the discussion of "typical sequences" in Section 1.3, that the number of sequences in S1is at least

and at most 2n'Hta'. Thus for large n we can divide the sequences (a,, . . . , a,) into two categories: a set S , consisting of approximately 2"'l sequences, each with probability roughly 2 P H , and a set S , whose total probability is arbitrarily small. If the sequence { X n ) is regarded as an information source, then the above argument indicates that when we are transmitting long sequences produced by the source by means of an encoding and decoding scheme, we can restrict our attention to a set consisting of approximately 2"" "meaningful" sequences (that is, members of S,) and neglect those remaining. The probability of correct transmission will not be unduly affected by this restriction. We are going to prove that the above results hold for a very wide class of information sources, including all regular Markov sources. Let X,, X,,. . . , be an information source with alphabet F, that is, a stationary sequence of random variables taking values in a finite set r. For each fixed sequence a = (a,, . . . , a,), a, 6 r, define a random variable N," (X,, . . . , X,) as the number of integers i ( I i 5 n - m 1) such that Xi = a,, Xi+, = a,, . . . , Xi+,-, = a,.

<

+

197

INFORMATION SOURCES

In other words, N,'"X,, . . . , X,) is the number of times ( a , , . . . , a,) occurs (in sequence) in the block consisting of the first n symbols. For example, if n = 13, m = 2 , a = ( A B ) , then if ( X I , .. . , X,,) = A A ABBABAA A ABB we have Nf,3,,(Xl, . . . , XI,) = 3 ; similarly if a = AA and ( X , , . . . , X,,) is as above, N$,,(X1, . . . , Xi3) = 5. The information source is said to be ergodic if, for all sequences a = ( a 1 ,. . . , a m ) , Nan(X1,. . . , X,)/n converges in probability t o P { X , = a,, . . . , X , = a,). In other words, the long-run relative frequency of a sequence converges stochasticaIly to the probability assigned to the sequence. We shall show later that any regular Markov information source is ergodic. For an example of a nonergodic source, see Problem 6.5. The previous discussion concerning the number of meaningful sequences which a source can be reasonably expected to produce carries over to ergodic sources. Specifically, we have the following theorem. Theorem 6.6.1. (Asymptotic Equipartition Property; also called the Shannon-McMillan Theorem.) Let (X,, n = l , 2 , . . .) be a n ergodic information source with alphabet r and uncertainty H ( X ) . As before, define a sequence of random variables Vn(Xl,. . . , X,) as follows: If XI = a ] , . . . , Xn = a,, let Vn(Xlr. . . , Xn) = -n-I log P { X l = a,,

. . .,X ,

= a,).

Then V,(X,, . . . , X,) converges in probability to H { X ) . Proof. We break the argument into several steps. For convenience we set If = H { X ) . a. lim E(V,) = H . n+ m

Proof of (a) P { X l = a , , . . . X , = a,}

E(V,) = a l p . . ..an

-(

n

'-

x n-' log P { X l = a,, . . . , Xn = a , ) H

by Theorem 6.4.1.

Our ultimate objective is to show that for every E > 0, P { H - E 5 V, 5 H E ) 4 1 as n 4 c o. As a preliminary reduction, we are going t o show that only the upper half of the inequality is needed, that is, if for every E > 0 P { V , 5 H E } -t 1 as n -,co then the general result follows. As the first step in the reduction we prove the following result.

+

+

198

INFORMATION THEORY

b. If

E

> 0 , 6 > 0, ,!I 2 0, then

E(V,) - B

I 6 - (6 + E)P{V, < B - E) + (log Irl)P{Vn > B + 6) -n-l~{V, > ,!I + 6) log P{Vn > ,!I + 6). (6.6.3)

( I r l is the number of elements in r . ) Proof of (b): E(Vn) = 2 kP{V, = k) k

=

2

k < B-E

kP{Vn = k}

Thus

E(I/,) I (B - E)P{Vn Since P{B - E we have (B E(VJ

+ B-ESKB+~ 2 kP{V,

= k)

+2

kP{Vn = k}.

k>B+d

< B - E) + (B + b)P{B - E 5 Vn < B + 6 )

+

I V, I /3 + 6) 1 P{B - E 5

x

kP{Vn = k).

k>P+d

V,) = 1

+ 6) - (6 + E)P{V, < B - E} + 2

- P{V, <

- E)

kp{Vn = k } (6'6.4)

k>B+b

Now

2

k>

kP{Vn = k} = ~+d (el,.. . , a n ) c S (-n-' log P{X, = a,, . . . , X, = a,))P{X, = a,,

. . . , X,

= a,]

(6.6.5) where S = {(a1, . . . ,a,): -nP1 log P{X, = a,, . . . , X , = a,} = {(a,, . . . , a,): PIX, = a,, . . . , X, = a,) < 2-'@+ ')

> fl + 6) 1

+

6). The (that is, S is the set of sequences corresponding to Vn > summation (6.6.5) is of the form -n-I z g , p , logp, where Z,"=,p, = 6) = p, and m is the number of elements in S. Now by P(V, > Theorem 1.4.2.

+

m

- z&1ogP" < log m, 2=1p

p

or m

Since the number of sequences of length n with components in we have m _< I r 1 ", hence

is

I rln,

Then (6.6.3) follows from (6.6.4) and (6.6.6). c. If P{Vn /, H E}4 1 for every E > 0,then Vn -t H in probability.

+

199

INFORMATION SOURCES

+

H E)-. 1 for every E > 0, then P{Vn Proof of (c). If P{V, E(V,) E)-+ 1 for every E > 0 since E(V,) -t H. In (6.6.3) replace B by co, to obtain E(V,) and let n

-

+

0

5 6 - (6 + E) lim

sup P{Vn < E(Vn) - E), or

n+ m

6 lim sup P{Vn < E(Vn) - E) I, -. n-r w 6 - k ~ Fix E and let 6 -* 0. We conclude that P{V, < E(Vn) - E) 0 for any E > 0. Since E(V,) -+ H, we have P{V, 5 H - E) -+0 for all E > 0. But by hypothesis, P{V, 2 H E) -t 0 as n -+ co. Thus P{\v, 2 €1'0 -+

+

HI

for all E > 0, and the result follows. d. Let I"") be the set of sequences (a1, . . . , a,), a, E I'. For any 2 0, let on(/?) be the largest probability assigned to any subset of I?") which has at most 2"@elements, that is, D,(B) = max P{(X,, FTri")

Then given

E

> 0, B 2 0,

. . . , X,)eF)

+ +

D,(B) I P{Vn 5 B E) 2-"'. ) at most 2"8 elements and with Proof of (d). Let F be a subset of P n with probability D,(P), that is, P{(X,, . . . , X,) E F ) = D,(/?). Then

+ E)) + P ( F n {V, > B + E)) + + P ( F n {V, > /? + E)).

D,(P) = P ( F ) = P ( F n {V, I, B 5 P{V, j B E)

As in (b), if Xi= a,, i = 1, 2,

+

. . . ,n, then

) < 2-,(@+'). V, > /? E if and only if P{X, = a,, . . . , X ,= a , Thus each point in {V, > /? E) (and hence in F n {v, > E)) has probability less than 2-n(Bf'). Since F has at most 2nSpoints,

>+

+

+ +

<

E) 2n@2-n(Btc), ' on(/?) P{ V, I, proving (d). e. Let H , = H(Xm ( XI, . . . , X,,-,), m = 1,2, . . . , and let M be any positive integer. If for every E > 0 and every positive integer m 2 M, lim Dn(Hm E) = 1, then V, 4H in probability. n- m

+

+

Proof of (e). By (d), Dn(Hm E) 4 1 for every 6 > 0 implies P{V, 5 H, E)-* 1 for every E > 0. By definition of the uncertainty of a source (Section 6.4), (H,, m = 1, 2, . . .) is a nonincreasing sequence converging to H. If we choose r 2 M such that H,. 5 H €12, then P{Vn I, H E) 2 P{Vn H,. 42) -+ 1 for every E > 0. The result now follows from (c).

+

+

+

+

200

INFORMATION THEORY

Note that the results ( a ) through ( e ) hold for an arbitrary information source,. the ergodic hypothesis has not yet been used. Proof of Theorem 6.6.1. By (e) it suffices to show that lim D , ( H ,

+

E)

=

71 --r m

>

1 for all E > 0 and all m 2. To d o this we will find (for each fixed E and m) subsets B , of r"")such that each B , has a t most 2n(H*+c)elements [hence D,(H, E) 2 P(B,)] and P(B,) + 1. Let 6 > 0 (to be specified later). If a = (a,, . . . , a,) is any sequence in I"")and y = (y,, . . . ,y,) is any sequence in I""), let z(y; a ) be the number of times that (y,, . . . , y,) appears (in sequence) in the block consisting of the n symbols a,, . . . , a,. Define B, as the set of all a E Fn) such that

+

I 2Y -

P { X , = y,,

. . . , X,

= y,)

5 8 for all y E I""'

The probability of B, is then

< 8 for all y c I""' Since I"") is a finite set, P(B,) -,1 by the hypothesis of ergodicity. It remains t o show that for an appropriate choice of 6, B , has at most 2n(Hm+E) elements. Now let S,-, be the (m - l)th-order approximation to the original source (Section 6.5); as before we form a stationary distribution for Sm-, by assigning to the state (a,, . . . , elm-,) the probability w ( , ~ ,. . . ,dm-,, P{Xl = a,, . . . , X,_, = a,-,). Let {Y,, n = 1 , 2 , . . . ) be the sequence of symbols produced by S,-I and let {Z,, n = 1 , 2, . . .) be the corresponding sequence of states. [Recall that if Z , = (a,, . . . , am-,), then Y , = a,_,.] If n 2 m we have

= P{Y, = a,,. .

. , YmP1= a,-,)

n

x i=m

= P{Z,-,

P{Y, = a.1 Yl = a,, . . . ,

= (a,,.

. . ,a,-,))

since the order of Sm-, is j m - 1.

x-l = ai-,}

INFORMATION SOURCES

Now

-.

am-1)) = w ( a l , .. . = P{Xl = a,, . . . , Xm-I = am_,),

P{Zm-l =

9

and

by definition of Sm-,; thus

P(Yl = al,.

. . , Y , = a,} X ycI'("'

. . . , X,-, = am-,} [P{Xm = ym I XI = Yl, . . . ,Xm-l = Ym-1 }]z(u;a) = P{Xl = a,,

(6.6.7) [For example, if a = (a,,

. . . , a,,)

= (ABAAAABABA) and m = 3, then

P{ Yl = a,, . . . , Y,, = a,,) = P { X , = A, X, = B ) (P{x, = A X, = A , X, = B ) ) ~ X(P{X,=A~X,=A,X,=A))~P{X~=A~X~=B,X~=A) x P { X , = B I XI = A, X2 = A)P{X3 = B ( Xl = B, X, = A).]

I

<

Now by definition of B,, if a = (a,, . . . , a,) E Bn then z(y; a ) n(6 P{X, = y,, . . . , Xm = ym)) for a l l y = (y,, . . . , y,) E I""); hence by (6.6.7),

+

. . . , Y, = a,) 2 P { X , = a,, . . . , X,-, = a,,-,) n(8+P(Xl=sl, . . . , X,=s,H x l-I [P{Xm= ym I X I = ~ 1. , . ,xm-l=ym-,}I vrI"m) for all (a,, . . . ,a,) B,. (6.6.8)

P{Yl = a,,

E

Now

as n -+ c~ and 6 -+ 0. Thus we may find no and 6, such that if n and 0 < 6 I; So, the above expression is 22-' for all possible a,, am-,. If 6 is fixed 6 (0, 6,], then for n 2 no, (6.6.8) yields

2 no .. .,

or

P { Y 1 = a,,

. . . , Y , = a,) 2 2-nE2-nHm for (a,,

. . . ,a,)

r B,.

(6.6.9)

202

INFORMATION THEORY

Now 1 2 P { ( Y 1 , .. . , Y , ) E B,) =

2

..

P I Y , = a,,

. . . , Y,

= a,)

( ~ 1 , . .Qn)
> 2-n'2-nHm#(B,) Thus B, has at most 2n(Hm+c)

by (6.6.9).

elements; this completes the proof.

T o show that a regular Markov source obeys the asymptotic equipartition property, we prove the following result. Theorem 6.6.2. A regular Markov information source is ergodic. Proof. Let XI, A,, . . . be a regular Markov source. Then we may assume that X , =f ( Z i ) where Z,, Z,, . . . , is a finite Markov chain with steady state probabilities and f is a function from the set of states S = (s,, . . . , s,) of the chain to the alphabet r of the source. For each sequence a = (a,, . . . , a,) E I?(,), we must show that Nan(Xi, Xz, n

. . X,)

+

P I X , = a,,

. . . ,X ,

= a,)

in probability.

Now for each a , let t,(a), . . . , tk(,,(a)E S(,) be all possible sequences of states which produce a. [In other words, if

then ( f ( s i c l ) ( a ) ). . . ,f (si(,)(a)) = (a1, . . . , a,) = a , ] It follows that

If we can show that for each i,

it will follow that n-IN,"(X,,

. . . , X,)

prob

k'lr)

2 P{Zl = s,,,,(a), . . . , Z ,

= si(,,(a))

i=l

= P { X 1 = ct,,.. . , X , = a,)

Thus the problem reduces to showing that the underlying Markov chain Z,, Z,, . . . , is ergodic. Now construct the mth-order approximation S, to the chain just as in Section 6.5; that is, construct a new Markov chain whose states are

203

INFORMATION SOURCES

the elements of S(") and whose transition probabilities are P(si(l)'

'8i(m))(8r(ll'

.

'~a(rn)8t(m+l))

( 2 1 = ~ a ( l ) , ..

= P{Zm+l = st(,+,) = P{Zrn+l = Sa(m+l)

I

Zm = ~ ~ { r n ) }

= ~a(m)>

by the Markov property. Since the original chain has steady state probabilities, there is a state s,(,) and an integer No such that s,(,) is reachable in No steps from any initial state. It follows that if s = (s,(,,, . . . , s,(,,) is any state of S , having s,,,, as its first coordinate, then s is reachable from any initial state of W, in No m - 1 steps. Thus S , has steady state probabilities. (These probabilities are w(,,, ,,, . . . ,S,(m l ) = P{Z1 = st(,), . . . ,Z , = s,(,)); see Problem 6.4.) Now we may regard Z,, . . . ,Z , as the components of the initial state U , of S,, and the Z,, n 2 m , as the last coordinates of the succeeding states U,, . . . , Un-,+, of S,. Consequently, if s = (st(,,, . . . ,s,(,)) E S'"), then n-I Nnn(Zlr. . . , Z n ) = n-lNFm+l(U1, . . . , UnPm+,).

+

It follows that verifying the ergodicity condition for sequences s = (s,,,), . . . , st(,,) of length m in the original chain Z,, Z,, . . . , is equivalent to establishing the condition for sequences of length 1, that is, single states, in S,. Thus without loss of generality we may assume m = I ; it remains to show that for every state s , E S , n-lN;(Z,,

. . . , Z,)

in probability.

P { Z , = s,} = w,

-+

This is just the weak law of large numbers for Markov chains. We state this explicitly:

Theorem 6.6.3. Let Z,, Z,, . . ., be a finite Markov chain with steady state probabilities. Let the initial distribution be arbitrary. Then for each state s,, the relative frequency n-IN," of s, converges in probability to the steady state probability w,, that is, n-lN?(Z1,

. . . ,Z,)

prob.

---+w5.

(6.6.10)

Proof. It suffices to show that (6.6.10) holds when the initial distribution is such that Z , = s, with probability 1 , where s, is an arbitrary but fixed state. We will show that E[(n-IN," - w,),] -t 0 as n -,co; since P{ln-lN," - w,l 2 E) 5 c2E[(n-IN5" - w , ) ~ by ] Chebyshev's inequality (or Lemma 3.6.1) the result will follow. Define

U > = 1 if Z , = s,, =O

if Z , # s , .

( k = 1,2,. . .)

204

INFORMATION THEORY

Then n

Now

n

= np2

2 a,,

k,l=l

where a k l = E[(U,"

- wi)(U,"

wj)].

Let m = min ( k , I ) , d = Ik - I [ . Then E[Ujk] = P { U : = 1 ) = P { Z , = s i ) = pi;'

and E [ u j k u j l ]= P { U r = U j = 1) = P { Z k = Z , = .Sj) = pi,(m)p (jdj) . Thus a,, = pij")pE) - w 3.p!'f) - wjPj:) + W j 2 . 19 (6.6.12) Let M:") = max p,(,"),m y ) = min p!;). [see after (6.3.13)] 15iSr

lSi5r

If the transition matrix of the chain is such that Hflo has a positive column, then by (6.3.18), Mi(2nu) - m3, ( , n u ) < (1 - E ) l - ~ where E is the smallest element in the positive column of IInu. Given n < (I I)n,. Then by the any positive integer n, choose I so that In, monotonicity of the sequences M j ( n )and m , ( n ) ,

<

Thus there is a positive number b and a number t

+

E

(0, 1) such that

INFORMATION SOURCES

Thus by (6.6.13) there is a positive constant c such that lak,l

I c(tm + td + tk + tl).

Now

Similarly

This proves Theorem 6.6.3 and thus finishes the proof of Theorem 6.6.2. The asymptotic equipartition property (AEP) allows us to draw several basic conclusions about the encoding of the information produced by a source. Suppose that the information produced by an ergodic source with uncertainty H i s to be transmitted through a (discrete memoryless) channel with capacity C. Suppose H < C, and choose R such that H < R < C. By the AEP, for sufficiently large n we can divide the sequences of length n produced by the source into two sets S, and S,, such that Sl has roughly 2"" sequences; more precisely, for large enough n, Sl has at least 2n'H-d' and at most 2n'"+d' sequences, where 6 is any preassigned positive number. In particular, we may choose 6 so that S1 has fewer than 2nR sequences. The total probability of the sequences in S, can be made < ~ / 2for any ) the channel. preassigned E > 0. Now we can find a code ([2nR],n, ~ 1 2for Let us assign a code word of the code to each sequence in S , ; assign an arbitrary channel input n-sequence and an arbitrary decoding set to each sequence in S,. With this coding scheme it is possible to transmit the information produced by the source with an overall probability of error < E. Thus a source with uncertainty H can be handled by a channel with capacity C provided H < C. The encoder assigns a code word of length n to each source sequence of length n; hence, if the channel can transmit 1 symbol per second, the source may be run at a rate not to exceed 1 symbol per second. T o look at it another way, suppose that we wish to run the source at a rate of u letters per second. If the alphabet of the source has m symbols, then the equivalent binary rate of the source is u log m = R

206

INFORMATION THEORY

binary digits per second, since in n seconds the source will theoretically produce any one of mnu = 2nRpossible sequences. However, by the AEP, we may restrict our attention to 2Hnu= 2nRHl'og of these. To be able to maintain the transmission rate and transmit with high reliability through a channel with capacity C, we must have RH/(log m) < C, or R < C[(log m)/H]. (Note that H 5 log m so that C[(log m)/H] 2 C.) In Chapter 3 we showed that if a source produces at a rate of R binary digits per second, the rate could be maintained with an arbitrarily small error probabiity as long as R < C. By using the AEP, we may take advantage of the statistical structure of the source to achieve higher transmission rates; the source rate is now bounded by the larger quantity C[(log m)/H]. Notice, however, that if the source is a sequence of independent random variables XI, X,, . . . , where each X, takes on all values in the source alphabet with equal probability, then H = log m so the upper bound remains at C. Hence if we know nothing at all about the structure of the source, the requirement R < C cannot be weakened. We may use the results of this chapter to say something about the properties of languages. For example, we may regard a portion of text in a particular language as being produced by an information source. The probabilities P{Xn = a, X, = a,, . . . , Xn-, = an._,)may beestimated from the available data about the language; in this way we can estimate the uncertainty associated with the language. A large uncertainty means, by the AEP, a large number of "meaningful" sequences. Thus given two languages with uncertainties Hl and H, respectively, if Hl > H , then in the absence of noise it is easier to communicate in the first language; more can be said in the same amount of time. On the other hand, it will be easier to reconstruct a scrambled portion of text in the second language, since fewer of the possible sequences of length n are meaningful.

I

6.7. Notes and Remarks The notion of an information source and the idea of a Markov source are due to Shannon (1948). The present formulation of the Markov model follows Blackwell, Breiman, and Thomasian (1958). The fact that a Markov source must be unifilar for the formula of Theorem 6.4.2 to hold was pointed out by McMillan (1953). Basic properties of Markov chains may be found in Feller (1950), Chung (1960), or Kemeny and Snell(1960). Theorem 6.5.1 is related to a theorem of Moore (1956, Theorem 8) about sequential machines; the proof in the text was suggested to the author by D. Younger.

INFORMATION SOURCES

207

Theorem 6.6.1 was discovered for the case of Markov sources by Shannon (1948); the general result is due to McMillan (1953). The proof in the text follows Thomasian (1960). McMillan (1953) showed that Vn -. H in L,, that is, E[I Vn - HI] 40. Breiman (1957, 1960) showed that Vn + H with probability 1. The criterion we have given for a process to be ergodic can be shown to be equivalent to the standard definition of ergodicity of an arbitrary stochastic process, specialized to the case of a stationary sequence of random variables with values in a finite alphabet. For a proof of equivalence see Khinchin (1957, p. 51). Further properties of ergodic processes may be found in Doob (1953). Theorems 6.6.2 and 6.6.3 hold for indecomposable Markov chains; see Chung (1960). PROBLEMS 6.1 Consider the unifilar Markov information source in Fig. 6.P.1. a. Show that the source is of order 2.

b. If the system is in state s, at t = 0, find the probability that the symbol A will appear at t = 3. c. Find the steady state probability of the sequence ABBAA. 6.2 Find the order of the unifilar Markov information source in Fig. 6.P.2 and calculate the steady state probabilities and the uncertainty of the source.

208

INFORMATION THEORY

6.3 A "language" conslsts of four symbols: A, B, C, and the space. The symbols occur Independently except for the following constraints: I. The space cannot follow Itself. ii. If the symbol A occurs lmmedlately after B, C,or the space, then the next symbol must be an A also, that IS, P{Xt = A1 X,-, = A

and Xt-, # A )

= 1.

a. Draw a unifilar Markov state dlagram for the informat~onsource described above and find the order of the source. b. Assume that ~f the precedlng symbol is the space, then A, B, and C are equally likely; ~fthe precedlng symbol 1s B or C, or ~fthe prevlous two symbols are both A's, then A, B, C, and the space are equally llkely Calculate the uncertamty of the source. c. It 1s deslred to make a first-order approxlmatlon to the above source, that is, to consider the dependence of the present symbol on the lmmed~atelyprecedlng symbol, but to Ignore the dependence of X , on symbols occurring at tune t - 2 or before. Draw a state d~agramcorresponding to the first-order approxlmatlon and calculate the associated uncertamty. 6.4 If S , 1s the nth-order approx~matlonto an lnformatlon source {XI,show that a stationary dlstrlbution for S, may be formed by taklng

6.5 Define a sequence of random varlables as follows: Toss an unb~ased coln. If the result 1s heads, let X, = 1

for all n

If the result is talls, let X, = - 1

for all n.

Show that the resultmg sequence 1s statlonary but not ergodic. 6.6 Cons~deran mformatlon source wlth an alphabet of two symbols A and B. Show that P{XtVl = A, X , = B ) = P{XGl = B, X , = A).

6.7 Conslder the Markov informat~onsource of Fig 6 P 3. Let X,, X,, X,, . . . , be the associated sequence of random varlables. Suppose that every t h ~ r dsymbol of the sequence {X,, n 2 0 ) 1s erased, the lnltlal symbol to be erased 1s chosen wlth equal probablllty from among X,, X,, and X,.

209

INFORMATION SOURCES

The process of erasing generates a new sequence Yo, Y,, Y,, if X, is the first symbol to be erased, then if

{X,,n 2 0) then

= ABAABBBAA

. . . . For example,

.-,

{Y,,n 2 0 ) = A B e A B e B A e . . . ,

Find the uncertainty H{Y). 6.8 (Thomasian 1960.) a. As in the derivation of equation (6.6.3), show that if { X , , n = 1,2, . . .) is an arbitrary information source and V, is defined as in Theorem 6.6.1, then

-

b. If V, in probability, show that necessarily /? = H{X). 6.9 If an information source is ergodic, show that for each m, the mth-order approximation to the source is indecomposable. 6.10 Show that a finite Markov chain always has at least one essential set of states. Does this result hold for a chain with an infinite number of states? 6.11 Show that it is possible for the nth-order approximation to an information source to have order less than n. 6.12 A random variable Y takes the values y,, . . . ,yM with probabilities ,pMrespectively. The symbol y, is assigned the code word a,,a,, . . .a,,, p,, (i = 1, . . . , M). Assume that each code word begins with a different letter, that is, a,, # a,, for i # j . Suppose that successive values of Yare chosen independently and the resulting sequence of code characters written down. For example, if M = 3 and the code words assigned to y, y,, and y, are A, BA, and CAB respectively, then the sequence y,y,y~,y,y, would correspond to ABACABAACAB. Let {X,) be the sequence of code characters generated according to the above scheme. Show that {X,) may be regarded as a unifilar indecomposable Markov information source, and show that

...

where A

=

Zf/f,p,n,

is the average code word length. (Intuitively,

H ( Y) "bitslcode word" -- ri "symbols/code word"

=

"bits/symbol.")

6.13 (The Maximum Uncertainty of a Unifilar Markov Source; Shannon 1948.) Given a unifilar Markov information source with states s,, . . . , s, and transition probabilities p,,. We define the connection matrix A = [a,,] of the source as follows. Let

a,, = 1 if p,, a,,

=0

>0

if p,, = 0.

( i , j = 1, . . . , r)

210

INFORMATION THEORY

In other words the connection matrix is formed by replacing the positive elements of the transition matrix by 1's. For a given connection matrix, we wish to choose the probabilitiesp,, so that the uncertainty of the source is maximized. Assume that every state may be reached from every other state, so that the source is indecomposable with the essential class consisting of all the states. It follows from the theorem of Fro-

benius (see Gantmacher, 1959, p. 65) that the matrix A has a positive real eigenvalue IZ, which is greater than or equal to the magnitude of all other eigenvalues. Furthermore, the eigenvectors corresponding to A, consist of all multiples of some vector x whose components x, (i = 1, . . . , r) are all positive. a. Let N(T) be the total number of sequences of length T the source can produce (considering all possible initial states). Let N,(T), i = 1, . . . , r, be the number of sequences of length T the source can produce when the initial state is s,. By setting up a difference equation for the N,(T), show that there is a positive constant K such that N(T) < K ( Q T . b. Show that for any assignment of the probabilities p,, consistent with the given connection matrix, the uncertainty of the source is 5 log A,. C. Let p,, = a,,x,/l,x,, i, j = 1 , . . . ,r. Show that under this assignment, the uncertainty H { X ) attains the upper bound log lo. d. For the source of Fig. 6.P.4, find the maximum uncertainty and a set of transition probabilities for which the maximum is achieved.

CHAPTER SEVEN

Discrete Channels with Memory 7.1. Introduction In this chapter we try to analyze channels whose behavior at a given time may depend on the "state" of the channel at that time as well as on past inputs and outputs. In the case of the discrete memoryless channel (Chapter 3), the distribution of a channel output symbol is completely determined by the corresponding input letter. In the more general situation, the distribution of the output symbol may depend on events happening arbitrarily far back in the past. We shall try to develop reasonable models for discrete channels with memory, and prove coding theorems analogous to those derived for the memoryless case. We shall then talk about the general notions of capacity, weak and strong converses for arbitrary discrete channels. Let us begin with two examples. Example 7.1.1. (Blackwell 1961a). Given two "trapdoors" as shown in Fig. 7.1.1. Initially (Fig. 7.1. la), a ball labeled either 0 or 1 is placed in each of the two slots. Then (Fig. 7.1.lb) one of the trapdoors opens. (Each door has the same probability of being opened.) The ball lying above the open door then falls through and the door closes. Another ball is placed in the empty compartment (Fig. 7.1.1~)and the process is repeated.

Fig. 7.1.1.

"Trapdoor channel."

211

212

INFORMATION THEORY

We may regard this description as defining a channel whose inputs correspond to the balls placed in the empty compartment and whose outputs correspond to the balls which fall through the trapdoors. Suppose that we start counting time after one of the doors has opened and a ball has dropped through. Let the symbol b, correspond to the condition in which a ball labeled i remains in the occupied slot. We define four states Table 7.1.1. Channel matrices and output function for the "trapdoor" channel so0

Input = 0

Input

1

Sl0

1

Sl0

B

Sl0

so1

$11

0 0 0

0

3

0

= Ml so, 0 0 0 1 s,, -0 0 0 1 Let g(k, s , ~ = ) output when an input k is applied and the channel moves to state s,,. Then =

g(0, so,) = 0 g(O,s10) = 1 g(0, sol) = anything (this condition will never arise) g(0, ~ 1 1 )= 0

g(1, so01 = 1 g(1, slO)= anything g(1, sol) = 0 g(1, s11)

=1

s , ~i,, j = 0 or 1, as follows. We say that the channel is in state s,, at time t = n if condition b , holds at t = n and b, held at t = n - 1. An input k corresponds to placing a ball labeled k in the unoccupied slot. The opening of a trapdoor then determines the corresponding output (that is, the identity of the ball which falls through) and the next state. The above information is summarized in Table 7.1.1. The meaning of Table 7.1.1 may be clarified by means of an example. Suppose that at time t = n, the channel is in state slo, and an input 1 is applied. Then one ball labeled 0 and one ball labeled 1 rest over the trapdoors. With probability 112 the 1 ball falls through, leaving the 0

213

DISCRETE CHANNELS WITH MEMORY

ball in the occupied slot. The channel then moves to state so, and emits an output g(l, so,) = 1. With probability 112 the 0 ball falls through, sending the channel into state sol. An output g(l, sol) = 0 is emitted. Thus we may describe the behavior of the "trapdoor" channel by giving matrices Mo and M,, called "channel matrices," whose components are the state transition probabilities under the inputs 0 and 1, respectively, and a function that associates an output with each input-state pair (x, s). The reader may feel that this channel could be described using two states instead of four. For the purpose of analysis, however, we'want to associate a unique output with each pair (z, s ) ; this cannot be done using only two states. Example 7.1.2. (Gilbert 1960). Consider the following model for a channel in which errors occur in bursts. We start out with two states G (for good) and B (for bad). Assume that binary digits are to be communicated at the rate of 1 per second. If we are in the good state at t = n, P1 <

42

41 /

Perfect transmission

Pz

Error probabihty = j3

Fig. 7.1.2. Burst noise channel.

the digit transmitted at that time is received perfectly. A coin is then tossed; if the result is heads (say this happens with probability p,), we move to the bad state at t = n 1 ; if the result is tails (with prob1. In the bad ability q, = 1 - p,) we stay in the good state at t = n state, a transmitted digit will be in error with probability B. After transmission, a coin is again tossed; with probability p, we move back to the good state, and with probability q, = 1 - p, we stay in the bad state. (See Fig. 7.1.2.) Assume p,, q,, pz, q2,B E (0, 1). Let us try to analyze this example by giving state transition matrices for each input and a function defined on input-state pairs. To do this, we need four states, say GO, GI, BO, B1, where the first coordinate indicates the "good" or "bad" condition and the second coordinate indicates the most recent output. The channel matrices and output function are indicated in Table 7.1.2. For example, if we are in state G1 and the input to follow will be 1, then with probability q, we remain in the "good" condition. Transmission is then perfect so that an output 1 is received and we remain in state G1. With probability p, we move to the "bad" condition. Having done this,

+

+

214

INFORMATION THEORY

Table 7.1.2. Channel matrices and output function for the "burst noise" channel GO G1 BO Bl

Input

=

1

G1 0 71 P I P p,(l - B) BO 0 p2

728 72(1 - B)

The output functlon asslgns to each state ~ t second s coordinate, regardless of the input. with probab~lity/I an error IS made so that the output 1s 0 and the new state is BO; w ~ t hprooablhty 1 - /I the Input 1s transmitted properly and the new state 1s B1. Notice that In thls case the output may be regarded as a funct~onof the state alone rather than the input-state palr. It 1s posslble to reformulate Example 7.1.1 so that the output 1s determined by the state; however, more than four states wlll be required. (See Problem 7.1.) Now let us "connect a source" to one of the prevlous channels, say the channel of Example 7.1.1. Consider a Markov ~ n f o r m a t ~ osource n wlth two states a, and a,, an alphabet conslstlng of 0 and 1 an assoc~ated funct~ondefined by f(ao) = 0, f(a,) = 1 , and Markov transltlon matrlx

Suppose that at a glven moment the source 1s In state a, and the channel 1s In state s,,. With probability 314 the source moves to state a, and emlts the d l g ~ 0. t T h ~ sd l g ~ 1s t apphed as the Input to the channel; necessarily (see Table 7.1.1) the channel moves to state so, and emlts the output g(0, so,) = 0. With probabhty 114 the source remalns In state a, and emits a "1." If t h ~ sdiglt is appl~edto the channel then wlth probabhty 112 the channel moves to state so, and produces an output g(1, so,) = 1, and wlth probability 112 the channel moves to state so, and produces an

215

DISCRETE CHANNELS WITH MEMORY

output g(l, sol) = 0. T o summarize, if the source is initially in state a,, and the channel is in states,,, then with probability 314 the next source and channel states are (a,, so,); with probabiity (1/4)(1/2) = 118 the states are (a,, s,,), and with probability 118 the states are (a,, sol). The corresponding inputs and outputs are determined from the states. In general, the probability of transition from the pair of states (akl, s i l j l )to the pair (a,, sij) is the product of the source transition probability pklk and the channel transition probability from sil j l to sij under the input k. The results are summarized in Table 7.1.3. A source-channel matrix for example 7.1.1

Table 7.1.3. (ao, s o o )

( a 0 7 ~ 1 0 )(00, s o l )

J

0

B

0

0 0

d

b

0 0

P

B B

0 0

o 0 a

B B

o 0

*B

0

0

0

o

o

B

0

0

0

it

0

s

0 0

f f

o

t

o

(ak,si,) + input f (a,) = k output g(z,, si,) as given in Table 7.1.1

0 0 0 0

o

(a19 ~ 1 1 )

f

0 0

a

o

( 0 0 , ~ 1 1 ) ( P I , ~ 0 0 ) ( a 1 9 ~ 1 0 ) ( ~ 1 s ,o l )

0 0 0 0

t

o

3 o 0

t t

Thus the action of the source on the channel may be described by the matrix of Table 7.1.3, which is called the "source-channel matrix" corresponding to the given source. In the next section we are going to use the concepts developed here to define a general model for discrete channels with memory. 7.2. The finite-state channel

The model we are trying to develop is that of a channel with a finite number of internal states. The present state of the channel will in a sense represent a summary of its past history. The application of an input will result in a transition to a (possibly) different state and the production of an output. Specifically, given a finite set S = {s,, . . . , s,) called the set of states and two finite sets r = {b,, . . . , b,) and A called the input alphabet and the output alphabet, respectively, we define a finite-state channel as a collection of r by r transition matrices (that is, matrices of nonnegative numbers with all row sums equal to unity) MI, . . . , M,, one for each input letter, together with a function g that assigns to each pair (b, s), b E IT, s E S, an element g(b, s) E A. (The matrix M, will also be denoted by M,,,,, and is called the channel matrix corresponding to the input b,.)

216

INFORMATION THEORY

The interpretation of this definition is the following (compare Examples 7.1.1 and 7.1.2). If the channel is initially in state s, and an input bi is applied, the channel makes a transition governed by the matrix Mi. The probability of moving from s, to s, is the element M,(s,, sk) in row j and column k of the matrix Mi. If the new state is s,, an output g(b,, s,) is produced. Notice that given an initial state and a sequence of inputs, the distribution of the corresponding output sequence is determined by the channel matrices. Thus the finite-state channel may be regarded as a special case of the general discrete channel defined in Section 3.1. Now consider a Markov information source S with set of states SO = {a,, . . . , a,), transition matrix lI = [p,,], i, j = 1, . . . , q, alphabet r = {b,, . . . ,6,) = the input alphabet of the channel, and function f from So to r. As in the discussion in Section 7.1, the operation of connecting the source to the channel is described by the source-channel matrix Q corresponding to S. Q is defined as a qr by qr matrix (q = number of source states, r = number of channel states) whose rows (and columns) correspond to pairs (a, s), a r So, s r S. The element in row (ai, s,) and column (aj, s,) of Q is ptjMf(,,,(sk,s,). In other words, if initially the source is in state ai and the channel in state s,, then with probability pi, the source moves from ai to a j and emits an input f (a,); the channel then moves from s, to s, with probability M,(,L,,(sk,s,). For purposes of analysis we must impose restrictions on the channel. We say that a finite state channel is regular if every finite product Mi(1)Mi(2) . . , Mi(k),i(l), . . . , i(k) E (1, 2, . . . , t ) , k = I, 2, . . . , is regular; that is,steady state probabilities exist for the Markov chain determined by the matrix Mi(1,Mi(2). . . Mia,. (The integers i(l), . . . , i ( k ) are not necessarily distinct.) We remark that the product of regular matrices need not be regular. For example, take

(The z's stand for positive elements.) The product MlM2 is

which is not regular; in fact the chain determined by M,M, has two essential sets and is therefore not indecomposable.

DISCRETE CHANNELS WITH MEMORY

217

The following two theorems provide the justification for the restriction of regularity. Theorem 7.2.1 implies that if a regular Markov source is connected to a finite-state regular channel, the associated input sequence, output sequence, and sequence of input-output pairs are all regular Markov sources and therefore obey the asymptotic equipartition property. This will be essential in the proof of the coding theorem for the channel. Theorem 7.2.2 provides a criterion that allows us to determine in a finite number of operations whether or not a given finite state channel is regular. Theorem 7.2.1. If a regular Markov source is connected to a finite-state regular channel, the corresponding source-channel matrix is regular. Proof. Consider any regular Markov source with alphabet J? = {b,, . . . , b , } . By Theorem 6.3.2, there is a source state a* and an integer k such that a* is reachable from any initial state in k steps. In particular, a* may be reached starting from a * ; hence it is possible for the source to proceed through a sequence of states of the form a*aicl, . . a,(k-l,ai(kj with airk,= a*. Assume that the input corresponding to is b,(,)E r. Let M = M i ( l , M i ( 2.j. . Mi(,,; by hypothesis, M is regular, and consequently there is an integer n such that M n has a positive column. But

hence there is some channel state s* such that s* is reachable in nk steps from any initial state under the sequence of source states

Finally, given any source-channel state pair ( a , s ) we may in k steps reach (a*, s , ) for some i ; but for any i, (a*, s*) is reachable from (a*, s,) in nk steps. Thus (a*, s*) is reachable in k nk steps from any initial state, so that if Q is the source-channel matrix, the column of Q(n+l)k corresponding to (a*, s*) is positive. The proof is complete.

+

The definition of regularity may be restated as follows. Given any finite input sequence b , ( l j b , ( 2. j. b,(,,, there must exist a channel state s* and an integer n such that for any initial channel state, s* is reachable under the input sequence

Using this criterion, we may check that the channels of Examples 7.1.1 and 7.1.2 are regular. In Example 7.1.1, if a given input sequence contains at least one "1," then it is possible to reach and remain forever in state

218

INFORMATION THEORY

s,,; similarly, if the input sequence contains at least one "0," it is possible to reach and remain forever in state so,. In Example 7.1.2, for any input sequence we may reach and remain permanently in state BO (or equally well state Bl). Thus in both cases the channels are regular. The definition of regularity seemingly involves the checking of an infinite number of matrices. We may however, replace the definition by a criterion involving only a finite number of steps, as shown by the following theorem. Theorem 7.2.2. Let M I , M,, . . ., M , be arbitrary r by r transition matrices. Define sets of matrices A(k), k = 1, 2, . . . , as follows. Let

If k > 1, let A(k) = the set of all matrices that can be expressed as a product of a t most k, not necessarily distinct, elements of A(1). Also define 00 A = UA(k) k=l

Let R be the set of all regular r by r transition matrices. Then if ~ ( 2 " )c R then A c R. Thus in the definition of regularity we may restrict k to be 52'" so that only a finite number of operations will be involved. Proof. As noted in Section 6.3, in deciding whether or not a matrix has steady state probabilities, all we need to know about a given element of the matrix is whether it is zero or positive. In other words, without loss of generality we may assume that we are dealing with matrices with only two possible elements 0 and x (with xx = x , x x = x). More precisely let A'(k) be the set of matrices formed by replacing all positive elements of the matrices in A(k) by x. Define A' and R' similarly from A and R. It follows that ~ ( 2 ' 7) R if and only if ~ ' ( 2 ' " ) R'. Since the number of r by r matrices with elements 0 and x is 2'" AA'(k)must equal A1(k I ) for some k 5 2'" Now by definition of the sets A(k), A(k 1) A(k 2) for all k, and hence A'jk 1) c Af(k 2). But if M c A1(k 2) and M # A1(k 1) then M = M,M, where M a E Af(k 1) = A1(k), M,, E A'(1). Thus M c A'(k 1). Consequently A1(k 1) = A'(k 2), and similarly A1(j) = A1(k) for j 2 k, so that A' = ~ ' ( 2 " ) . Consequently

+

+

+

+

+

+

+

+

+ +

+ +

If we take t = 1 in Theorem 7.2.2, we obtain as a corollary the result that in applying the criterion of Theorem 6.3.2, we need only examine the powers IIn, n 5 2'" of a transition matrix rI in order to determine

219

DISCRETE CHANNELS WITH MEMORY

whether or not the associated Markov chain has steady state probabilities. For if S is the set of r by r transition matrices not having a positive column, ~ )S , an argument identical to that of Theorem 7.2.2 shows that if ~ ( 2 ' c then A c S. Thus if none of the matrices nn,n 5 2'', has a positive column, then no power of II has a positive column so that is not regular.

n

7.3.

The coding theorem forfinite-state regular channels

Suppose that a regular Markov source is connected to a given finitestate regular channel, and Q is the resulting source-channel matrix. Q then determines a finite Markov chain with states (a,, s,) where the a, are the states of the source and the s, are the states of the channel. By the assumption of regularity, the chain associated with Q has steady state probabilities. Given the pair (a,, s,) we can determine the corresponding input f,(a,, s,) =f (a,) and output gl(a,, s,) = g(f (a,), s,). Thus the chain associated with Q determines a stationary sequence of inputoutput pairs (X,, Y,), n = 1,2, . . . . We may therefore define an input uncertainty H { X ) = lirn H ( X n X,, . . . , Xn-,), an output uncertainty

I

I

n- m

H { Y ) = lim H ( Y n Y,,

. . . , Yn-,), and a joint uncertainty

n- w

We define the information conveyed about the process { X } by the process { Y ) as

It follows from Theorem 6.4.1 that

I

I

Z[{X) { Y ) ]= lim n-l Z(Xl, . . . , Xn Y,, n-

. . . , Yn)

m

and hence that I [ { X )I { Y ) ]is nonnegative. Furthermore, H { X , Y ) - H{x), which may be regarded as the conditional uncertainty of { Y ) given { X } , is given by lim r 1 H ( Y 1 ,. . . , Y n X,, . , Xn) and also by

I

..

n-m

lirn H( Y n n 4m

(see Problem 7.2).

1 x,, . . . , Xn, Y l , . . . , Y n J

220

INFORMATION THEORY

We define the capacity of the given channel as C = the least upper bound of I [ { X ) {Y)], taken over all regular Markov sources. We are going to prove that it is possible to transmit information at any rate less than C with an arbitrarily small probability of error. First we define a code for the finite-state regular channel just as in the memoryless case; that is, a code (u, n, E) is a set of input n-sequences x"), . . . , x'"), together with a decision scheme, that is, a collection of disjoint sets B,, . . . , B , of output n-sequences, such that

I

for all i = 1, 2, . . . , u and any initial state. Thus, no matter what the past history of the channel, decoding is accomplished with a probability of error not exceeding E. We now prove a lemma that will be useful in the proof of the coding theorem. If Q is the source-channel matrix corresponding to a regular Markov source which is connected to a finite-state regular channel, then Q has steady state probabilities w(,,,,, a E So = set of source states, s E S = set of channel states. The steady state probability of the channel's being in state s is given by ws

=

2 w(a,sl. a

Define a code (u, n, E ; Q) as a set of input n-sequences x"), . . . , x("), together with a decision scheme determined by sets B,, . . . , B,, such that if the initial channel state is chosen at random, with state s assigned probability w , (s 6 S), and the sequence x(" is transmitted, the probability of error, that is, the probability that the corresponding output sequence will not belong to B,, is < E for all i. Lemma 7.3.1. Let Q be a source-channel matrix for a finite-state regular channel. If for a given positive number R there is a sequence of codes ([2nR],n, E,; Q) such that E, + 0 as n -+ CO, then for any positive number R' < R there is a sequence of codes ([2""], n, E,') such that E,' + O as n + CO. Proof. Consider the following random experiment. Let xi be any input n-sequence, and let Bi be any set of output n-sequences. Let (a,, so) be any source-channel state pair. Allow the source and channel to make N transitions starting from (a,, so)with the transition probabilities governed by the matrix Q . After N transitions, send the sequence xi through the channel. If (A,, 2,) is the source-channel state pair after k transitions, then the probability that the output sequence corresponding to xi will not belong to

22 1

DISCRETE CHANNELS WITH MEMORY

a. 8

X

1

P { ( Y N + ~. ., . , YN+,) B B, ( X N + I ., . . , XN+,) = x z , ( A N , Z.") = ( a , s)) (7.3.1) (N- m) + 2'"(a,a)P{(Yi,. . . Yn) B Bi ( ( X i , . , X,)

= xi, (A,,

20)

= (a,s))

a.8

=

2 w8P{(Yl,. . . , Y,) # B, I ( X i , . . . , X,) = xi, Z , = s).

(7.3.2)

Note that the second factor of the summand of (7.3.1) is independent of N and hence coincides with the second factor of the summand of (7.3.2). Furthermore, since the number of source-channel state pairs is finite, the convergence of (7.3.1) to (7.3.2) is uniform in n, xi, B,, and (a,, so). By hypothesis there are codes ( [ 2 n R ]n,, E,; Q ) with E , -+ 0 as n 4 co. But by definition of such codes, the expression (7.3.2), evaluated for a code word xi and a decoding set B,, is just the probability of error when x, is sent. For each n, consider the code ([2nR],n, E,; Q ) with code words xi and decoding sets B,. Given E > 0, fix N = N ( E ) such that (7.3.1) and (7.3.2) differ by less than E , uniformly in n, xi, B,, and (a,, so). Let n be large enough so that s, < s (and hence (7.3.1) is less than 2 ~ ) . Now observe that if i and a, are fixed,

and the summation is over all input N-sequences x. Let r be the total number of channel states. Since 2,p(x)gSo(x)< 28, the set of all x such that g,,(x) 4 r ~ must have a probability <1/2r. For if not, we would have

>

>

>

2 4reP{x:g,,(x) 4rs) 2 ~ , a contradiction. If D" = { x : gso(x) 4 r ~ )then , Z80P(D,o)5 r(1/2r)= $. In particular, the sets Daocannot exhaust the entire collection of input N-sequences, so that there is an N-sequence xi' belonging to none of the Ds0; for such a sequence, gso(x,') < 4re for allpossible initial states so.

>

222

INFORMATION THEORY

It follows that if we form codes with words x ( ~=) (xi1, x,), i = I , 2, . . . , [2nR],and decoding sets Bi* = {(yif,yi) : y, E Bi) (in other words we decode by applying the original decision scheme to the last n coordinates of the received sequence), we will have constructed codes ([2nR],N n, 4rs). If we let m = N n and observe that 2nR = 2m(R-m-1NR) = 2mK' where R' -,R as m -+ co, we have the desired result. Lemma 7.3.1 is analogous to Lemma 3.5.3 in that it asserts that if we can make the "average" probability of error arbitrarily small for a given transmission rate, we can also make the maximum probability of error arbitrarily small. We may now prove the coding theorem for the finite-state regular channel.

+

+

Theorem 7.3.2. Given a finite-state regular channel with capacity C > 0 and a positive number R < C, there is a sequence of codes ([2"'], n, E,) with E , + 0 as n -+w . Proof. By definition of C, there is a regular Markov source S which, when connected to the channel, yields an information I = I [ { X ) {Y)] which is greater than R. Let Q be the corresponding source-channel matrix. Now suppose that the initial channel state is chosen at random, with state s assigned the steady state probability w,. Let p(y I x) be the probability that the n-sequence y is received when the n-sequence x is transmitted. The system of probability functions p(y I x) forms a discrete channel as defined in Chapter 3. Thus Lemma 3.5.2 applies (recall that we observed at the time that the assumption that the channel be memoryless is not essential to the lemma). In the lemma, we need to specify a probability function p ( ~ defined ) on the set of all input n-sequences. Let us takep(x) to be the probability that the source S produces the sequence x, that is, p(x) = P{(X,, . . . , X,) = x), when the initial state of the source is chosen at random in accordance with the stationary distribution induced by the matrix Q. [Specifically, the source state a is assigned probability w, = 2, w (,,,,, where w is the steady state probability of the source-channel state pair ( a , s).] We conclude that for each real number b and positive integers u and n, there is a code (u, n, E,; Q) such that

I

,,,

+

Let b = &n(R I), u = [2nR]. Define a sequence of random variables V , as follows. If (XI, . . . , X,) = a and ( Y , , . . . , Y,) = P, then

V, = n

-1

log-

I

P(P a ) P(P)

-1

=n

p(a9 P) log P ( ~ > P ( P' >

223

DISCRETE CHANNELS WITH MEMORY

It follows from the asymptotic equipartition property (Theorem 6.6.1) and the fact that a regular Markov source is ergodic (Theorem 6.6.2) that Vn converges in probability to H{X} H{Y} - H{X, Y} = I. Since

+

P K x , y):log [p(y ( x)lp(y)l I b j = P{Vn V, H R

we have

E, +

+ 0)

+

0 as n + co,

0 as n + CO. The result now follows from Lemma 7.3.1.

The weak converse to the coding theorem holds, that is, any code (u, n, E ) for the finite-state regular channel must satisfy (nC log u I

+ log 2)/(1 - E)

where C is the channel capacity (Blackwell, Breiman, and Thomasian, 1958); hence, if the transmission rate is maintained above capacity, the probability of error cannot be made to approach zero (compare Theorem 3.7.3). In fact, the probability of error must approach one with increasing code-word length (that is, the strong converse to the coding theorem holds) as shown by Wolfowitz (1963b). 7.4.

The capacity of a general discrete channel: comparison of the weak and strong converses

It is possible to construct a definition of the capacity of a discrete channel in such a way that the definition will not depend on the properties of any specific channel. First, a code (u, n, E) for an arbitrary discrete channel is defined in exactly the same way as a code for the finite-state channel, that is, a collection of u input n-sequences together with a decision scheme such that the maximum probability of error is <E, regardless of the state of the channel. A nonnegative number R is called a permissible rate oftransmission for the channel if there exist codes ([2""], n, E,) such that E, + 0 as n + co (n is allowed to approach infinity through a subsequence of the positive integers). We define the channel capacity C as the least upper bound of all permissible transmission rates. (Note that 0 is always a permissible rate, so that the set of permissible rates is not empty.) Thus if R < C , the rate R can be maintained with an arbitrarily small probability of error; this is not possible if R > C. In the case of the discrete memoryless or the finitestate channel, C agrees with the expressions given previously. In general, we establish that a particular number C, is the capacity of a given channel by proving a coding theorem (such as Theorem 3.5.1 or Theorem 7.3.2) and a weak converse (such as Theorem 3.7.3). In other words, we prove first that for any R < C,, there exist codes ([2nR],n, E ~ ) with en + 0 as n + a,and hence that C C,. We then prove that given

>

224

INFORMATION THEORY

any sequence of codes ([2""], n, E,), R > Co, there is an E > 0 such that for sufficiently large n, E, 2 E ; consequently C Co. We say that the strong converse holds for a given channel if any sequence of codes ([2nR],n, A,), with R > C, must be such that lim 1, = 1. To

<

n-c m

gain further insight into this concept, we introduce the notion of the k a p a c i t y of a discrete channel. Given any discrete channel, let N(n, 1) be the maximum possible number of code words in a code that uses sequences of length n and that has maximum probability of error at most 1. If 0 < A < 1, the 1-capacity of the given channel is defined by C(1) = lim sup n-I log N(n, 1)

(7.4.1)

n-t m

If one is willing to use codes whose maximum probability of error is < A (instead of demanding that the maximum probability of error be made arbitrarily small), then C(1) may be regarded as the least upper bound of all permissible transmission rates. Specifically, we call a number R a A-permissible rate of transmission if given any positive integer no, there exists a code ([2nR],n, 1,) with n 2 no and 1, 5 1. It follows directly from the definition of the lim sup of a sequence of numbers that C(1) is the least upper bound of all 1-permissible transmission rates. Since any code (u, n, 1,) is automatically a code (u, n, 1,) if 1, < A,, it follows that C(1) is nondecreasing. We have the following result.

Lemma 7.4.1. The channel capacity is given by C = lim C(1).

(7.4.2)

1-0

Proof.

Let Co = lim C(1). If R

< Co and 0 < 1< 1 then R < C(1),

1-0

hence R is &permissible. Thus given any no we can find codes ([2nR], n, 1,) with n 2 no and 1, 5 1. Since 1may be chosen arbitrarily small, it follows that C 2 Co. On the other hand, if R > Co there is a 1r (0, 1) such that R > C(1). Since R is not 1-permissible, it follows that for sufficiently large n, no code ([2""], n, 1,) can be such that A, 5 A. Consequently no number greater than Co can be a permissible transmission rate, and hence C Co. The strong converse has a striking interpretation in terms of the 1capacity.

Lemma 7.4.2. The strong converse holds for a discrete channel if and only if (7.4.3) C(1) = C for a11 1c (0, 1). Proof. If the strong converse holds, then given any A c (0, 1) and any R > C, for sufficiently large n no code ([2""], n, A,) can exist with A, 5 1.

DISCRETE CHANNELS WITH MEMORY

225

Therefore R is not I-permissible for any 1 ; consequently C(I) 5 R for all I. Since this holds for any R > C, we have C(A) 5 C for all I. But since C(1) is nondecreasing, Lemma 7.4.1 yields C(1) = C for all il E (0, 1). Conversely, suppose that C(1) = C for all A E (0, 1). Given R > C and 1E (0, I). Since R > C(il), R is not I-permissible and therefore if we have any sequence of codes ([2nR],n, An), we must have In> I eventually. In other words 1, 4 1 as n -,oo, and the strong converse holds. We are going to give an example of a channel for which the strong converse fails to hold. Our example will be a binary channel with the curious feature that lim C(I) = 0, lim C(I) = 1 [thus violating (7.4.3)l. a+o

a+1

Hence if we desire to transmit information with an arbitrarily small probability of error, the only permissible transmission rate is zero, while if we are willing to put up with an arbitrarily high probability of error, we can approach the maximum transmission rate of a noiseless binary channel. Consider a channel whose inputs and outputs are binary sequences of length n(n = 1,2, . . .). The action of the channel is described by specifying that if the output sequence y differs from the input sequence x in exactly i places (i = 0, 1, . . . , n) then

Thus the number of transmission errors is uniformly distributed between 0 and n, and all error patterns of the same weight have the same probability. (A formal definition of the channel may be readily supplied.) We shall establish the following result. Theorem 7.4.3. For I > 2, C(R) 2 1 - H(21 - I), where for brevity we write H(x) for the uncertainty -x log x - (1 - x) log (1 - x); consequently lim C(1) = 1. Furthermore, for all A, C(I) 1 - H(i(1 - 1)); a-

<

1

consequently lim C(I) = 0. 1-0

Proof. that if

The Varsharmov-Gilbert-Sacks condition (Theorem 4.5.3) states

it is possible to construct a binary code with 2kwords of length n which will correct t-tuple and all smaller errors. Now

$ (;)

r O

2 2nH'a) if 0


(Lemma 4.7.2)

a6

INFORMATION THEORY

~h~~ if 2"-k > 2(n-i)H((2t-l)l(n-l)), in particular if 2"-k > 2"fWln) (and 2tln < +), we can find a binary code with 2" words of length n which corrects all errors of weight t or less. Set R = k/n and a = 2tln. Then the above condition implies that if R < 1 - H(a), for sufficiently large n we can construct a code with at least 2nR words of length n which corrects all errors of magnitude
I t follows that R is (I - +a)-permissible, where R may be chosen arbitrarily close to 1 - H(a). Hence C(l - &a)2 1 - H(a) = 1 - H(l - a) (a < h), and the first part of the theorem is proved. T o prove the second part of the theorem, consider any code for the given channel, say a code with N words of length n and maximum probability of error 1. Since each error pattern of weight i or n - i has probability

which decreases as i varies from 0 to in, it follows that if r is the smallest integer such that 2r/(n 1) 2 1 - 1,each decoding set of the code has at least 2 2;:;

(7)

+

sequences. [The way to achieve a probability of correct

transmission of 1 - 1with the smallest possible number of sequences in a decoding set is to correct the errors of weight 0 and n, 1 and n - 1, . . . , r - 2 and n - (r - 2), and some or all of the errors of weight r - 1 and n - (r - 11.1 Hence, as in Theorem 4.2.2,

Using the fact that

DISCRETE CHANNELS WITH MEMORY

(Lemma 4.7.2), we obtain C(4) = lim sup n-' log N(n, 4) n-9

5 1 - H(g(1 - A)),

m

and the theorem follows. In the example just given, the variance of the number of transmission errors is of order greater than n (actually of order n2), as opposed to the case of a binary symmetric channel, where the variance is of the order of n. Wolfowitz (1963a) suggests that this fact is essentially responsible for the nonconstant behavior of C(4). 7.5.

Notes and remarks

The finite-state channel was introduced and analyzed by Blackwell, Breiman, and Thomasian (1958). These authors achieved greater generality than we have in this chapter by allowing the Markov chains involved in the definition of the channel to be indecomposable rather than simply regular. However, they used several ideas from martingale theory in their presentation, and we have avoided this here. Theorem 7.2.2 is due to Thomasian (1963). Just as in the case of the discrete memoryless channel (Theorem 3.6.4), there is an exponential bound on the rate of approach toward zero of the probability of error of codes with a fixed transmission rate; see Blackwell (1961b). A channel that fits naturally into the finite-state model is the so-called discrete jinite-memory channel. Intuitively, a channel with memory m should have the property that the distribution of the state of the channel at the end of an input sequence of length n, n 2 m, depends only on the last m inputs, and not on any previous inputs or states. Now if an input sequence x,(*). . xi(m)is applied, the distribution of the state at the end of the sequence may be found from the matrix M = M i ( l ) M i ( z.) . M i ( m ) , where M,(,) is the channel transition matrix corresponding to the input xio). T o say that the state distribution at the end of the sequence is independent of the initial state is to say that the matrix M has identical rows. Thus a finite-memory channel may be regarded as a finite-state channel with the property that for some positive integer m, each matrix M i ( l ) M I ( 2. ) . M i ( m )(and hence each matrix Micl,Micz). . Mi(,), n ;t m) has identical rows. A transition matrix with identical rows has at least one positive column, and thus a finite-memory channel is regular. Our approach to the discrete finite-memory channel is that of Blackwell, Breiman, and Thomasian (1958). Alternate treatments may be found in Khinchin (1957), Feinstein (1959), and Wolfowitz (1961).

-

-

228

INFORMATION THEORY

The results of this chapter have given no clue as to how to compute the capacity of a given finite-state channel. Very little is known in this area. An important initial step has been taken by Wolfowitz (1963b), who showed the existence of a procedure whereby we can achieve any preassigned approximation to the capacity in a finite number of steps. However, practical algorithms do not exist at present. One of the few explicit results is that of Gilbert (1960), who found the capacity of the channel of Example 7.1.2. The capacity of the channel of Example 7.1.1 is apparently unknown. There is a channel that in a sense combines some of the features of the discrete memoryless channel and the discrete channel with memory. Suppose we have a collection of discrete memoryless channels, one for each element s in some arbitrary set S. A code word is to be transmitted through one of the channels, but the identity of the channel to be used is unknown to the sender and may be regarded as being chosen by "nature." We wish to construct codes that maintain a given transmission rate and at the same time have a low probability of error regardless of which channel is used. Formally, we define a compound channel (the terminology is that of Wolfowitz, 1961) as a system of probability functions pn(& . . . , Pn I a l , . . . , a n ; s), where a,, . . . , a, belong to a specified finite set r called the input alphabet, PI,. . . , 8, belong to a specified finite set r' called the output alphabet, s belongs to an arbitrary set S called the set of states, n ranges over the positive integers, and finally for each s and n we have pn(B1,.

. . r B n I ~ l , .

..,an; s)=

1 a2; S) . .pl(Pn 1 an; S) P ~ ( P ~ ;J ~)~1(/32 The compound channel is thus a special case of the general discrete channel, and a code for the compound channel is therefore defined in the usual way. Notice that we require that the input alphabets be the same for all the individual memoryless channels in the collection indexed by the set S, and similarly for the output alphabets. Now if p(x) is any probability distribution on the input alphabet, then for each s r S we may calculate the information Z(s; p(x)) processed by the discrete memoryless channel corresponding to s. The following theorem was proved by Wolfowitz (1960, 1961) and independently by Blackwell, Breiman, and Thomasian (1959). The capacity of the compound channel is C = sup inf I ( s ; p(x)). P(Z)

ses

In accordance with the discussion in Section 7.4, the proof may be accomplished by proving a coding theorem and a weak converse. The strong

DISCRETE CHANNELS WITH MEMORY

229

converse to the coding theorem also holds, as was shown by Wolfowitz (1960, 1961). The notion of the &capacity of a channel and the discovery that there are channels for which the strong converse does not hold are due to Wolfowitz (1963a). The example discussed in Theorem 7.4.3 is the author's (1965). In certain physical situations we may have a discrete channel whose state a t the beginning of an input sequence is known to the sender or to the receiver, or to both. For a discussion of this type of problem, see Wolfowitz (1961). PROBLEMS 7.1 a. Reformulate Example 7.1.1 so that the output is a function of the channel state alone rather than the input-state pair. b. Show that any finite-state channel model may be reformulated so that the output is a function of the channel state alone, and furthermore, if the original channel is regular so is the new channel, and the capacity of the new channel is the same as that of the original. 7.2 Let {X,, n = 1 , 2, . . .) be a sequence of inputs produced by a regular Markov source that is connected to a finite-state regular channel, and let { Y,, n = 1 , 2, . . .) be the corresponding output sequence. Define the conditional uncertahty of the sequence { Y,) given the sequence {X,) as

+

Hence show that H{X, Y } = H { X ) H ( { Y ) ( { X } ) . 7.3 Let X,, n = I , 2, . . . ,and Y,, n = 1 , 2, . . . , be stationary sequences of random variables defined on the same sample space, such that the sequence of pairs (X,,Y,) is also stationary. (Assume that the X, and Y , can take on only finitely many possible values.) Define the conditional uncertainty of the sequence {Y,} given the sequence { X , ) as in Problem 7.2. Do the results of that problem still hold?

CHAPTER EIGHT

Continuous Channels 8.1.

Introduction

In this chapter we cons~dercommunication channels that are "contlnuous" In two senses. F m t we allow the Input and output alphabets to contain a n uncountably lnfinlte number of elements, although we requtre that the material to be transmitted through the channel be In the form of a dlscrete sequence of symbols In other words, once every second (say) we transmlt a symbol from a posslbly uncountable alphabet. Thls IS the so-called tlme-dfscrete, amphtude-continuous channel Second, we allow the transmission of mformatlon to be continuous In tlme. In other words, we transm~ta "tlme function"f (t) defined on an lnterval of real numbers, whose range IS also an interval of real numbers. T h ~ sis the so-called ttme-continuous, amplitude-continuous channel The first channel mentioned above IS somewhat easler to formalize. Given an arbitrary set S called the set of states, we define a t~me-dmrete, ampl~tude-contmuouschannel as a system of p r o b a b ~ l ~ dens~ty ty functions s), s E S, n = 1,2, . . , the x, and y, real; . , xn, pn(yl, . . . ,yn 51, that is, a system of nonnegative (Borel measurable) functions such that

1

-m

-m

for all n,

F,

x,,. . . , 5,.

Thus for each Input sequence and each state, we have a probabtlity measure on the set of all output sequences (In t h ~ case, s Euchdean n-space) (We could have achleved greater generality by allow~ngarb~traryprobabll~tymeasures on the set of all output sequences for each Input sequence and state.) A code (u, n) for such a channel is defined in the now famdiar way as a collection of sequences x"), . . .. x(") belongtng to Eucltdean n-space En, together with a dec~slonscheme, that IS,a partltlon of E" Into dlsjolnt (Borel) sets B,, . . . ,B,. A code (u, n, a) IS a code (u, n) wtth maximum probabhty of error 51,that IS, a code (u, n) such that

for all possible code words x(" and all posslble states s.

23 1

CONTINUOUS CHANNELS

In a given problem, physical considerations may require that some restriction be placed on the set of allowable inputs. For example, if each component of a code word is represented by the amplitude of a pulse, we may wish to require that 12, I 5 some constant M for all i, or possibly xi" M. we may have an "average power" limitation of the form n-l Given (Borel) subsets F, of Euclidean n-space (n = 1 2 , . . .) we may define a restricted time-discrete, amplitude-continuous channel corresponding to the F , as a system of probability density functions

zr=l

defined only for (x,, . . . , x,) 6 F,. A code (u, n) and a code (u, n, A) are defined in exactly the same way as before, except that the code words are now required to belong to the set F,. 8.2. The time-discrete Gaussian channel In many physical communication situations, disturbances enter in the form of an unwanted "noise" superimposed on the signal to be transmitted, thus producing a result in the form of signal plus noise, from which the desired signal is to be extracted. The assumption that the noise is Gaussian seems to be reasonable in a wide variety of physical settings; the assumption is definitely desirable from the mathematical point of view, as we shall see in the course of the chapter. A time-discrete Gaussian channel is a time-discrete, amplitude-continuous channel whose probability density functions exhibit no dependence on the state s, and are given explicitly by

.

I

pn(yl, . . . yn z l , . . . , x,) = (277N)-

t

" erp

[- f (Y' 2N- x')2].

(8.2.1)

,=I

Thus the output sequence is obtained from the input sequence by the addition of a sequence of independent Gaussian random variables with mean 0 and variance N. We shall see that if no restriction is placed on the choice of code words, arbitrarily high transmission rates can be maintained with an arbitrarily small probability of error. We are going to make a restriction on the "average power" of the input sequence; in other words, we introduce a restricted channel determined by sets F, = {(x,, . . . , xi2 M ) . The resulting channel is called a time-discrete xn): n Gaussian channel with average power limitation M . We may establish a relationship between the number of words in a code and the probability of error by proving a result which is analogous to the Fundamental Lemma 3.5.2 and its extension in Problem 3.1 1. The result is not restricted to the Gaussian case.

z?=, <

232

INFORMATION THEORY

Lemma 8.2.1. Given an arbitrary restricted time-discrete, amplitudecontinuous channel whose restrictions are determined by sets F, and whose density functions exhibit no dependence on the state s, let n be a fixed positive integer, and p(x) an arbitrary probability density function on Euclidean n-space. Writep(y I x) for the densityp,(y,, . . . ,y, ( x,,. . . ,x,), and write F for F,. For any real number a, let

(x, y): logThen for each positive integer u, there is a code (u, n, words belonging to F), such that

1 5 ue-" where

+ P{(X, Y) I A) + P{X I F }

A

and

A) (with all code

P{X F} =

(8.2.2)

1.-

-JP(Xl dx.

F

[For convenience we assume that all logarithms in this chapter are to the base e: thus we have a term ue-" in (8.2.2) instead of u -2-".I Proof. The argument proceeds exactly as in Lemma 3.5.2 (and Problem 3.11), with sums replaced by integrals, but we will go through the details for convenience. Let E be the expression on the right side of (8.2.2). If E >_ 1, there is nothing to prove, so assume 0 < E < 1. Pick (if possible) a sequence x") E F such that P{Y

E

A,w 1 X = xtl))

>1-

E

where A, = {y:(x, y) c A } ;

choose the decoding set Bl to be A,,,,. Having chosen x"), and B,, . . . ,Bk-,, select x ( ~c )F such that

.. .,

x(k-l)

set Bk = AXw - Uf:; BE. If the process does not terminate in a finite number of steps, then the sequences x'" and decoding sets B,, i = 1,2, . . . , u, form the desired code. Thus assume that the process terminates after t steps. (Conceivably t = 0.) We will show t 2 u by showing that

CONTINUOUS CHANNELS E

j teca

B=

+ P{(X, Y ) f A ) + P{X f F ) .

Ui=, B j . (If t

We proceed as follows.

233

Let

= 0, take B = 0.) Then

Since B n A x c B, the first term of (8.2.3) cannot exceed

But since B, c Axcir,

Now if y E then ( x i i ) ,y) r A ; hence log [p(y I x(q)lp(y)] > a, or p(y) < p(y I x"))e-". Consequently

Therefore the first term of (8.2.3) is Ire-". To estimate the second term, we claim that

First assume x = one of the code words xck). Then we have

234

INFORMATION THEORY

To establish (8.2.5), we show (as in Lemma 3.5.2) that if y E A,w then y necessarily belongs to B. Since Bk = A,w - U:.z: B,, if y E A,w and y $: U Bi, then y E Bk c B; on the other hand,if y E A X l k , and y 6 Ui:: Bi, then, since B is the union of all the decoding sets, we have y E B. Thus (8.2.4) is proved when x is a code word. Now assume that x # any of the

1

p(y x) dy 2 1 - E , then, since x is assumed rF and

x i . If YPBnA,

= A, - U:=,B,, we could enlarge the code by taking xtt+l)= X, B,,, = B n A,, contradicting the assumption that the process terminates after t steps. Thus the second term of (8.2.3) is 5 I - E. We estimate the third term of (8.2.3) by observing that B n A , is a subset of the entire space En, and hence the third term cannot exceed

B n A,

We conclude that P{(X, Y) E A ) result follows.

I te-"

+ 1 - + P{X t F}, and the E

As in the discussion of Section 7.4, we say that a number R is a permissible rate of transmission for a given time-discrete, amplitude-continuous channel (restricted or not) if there exist codes ([eaR],n, 1,) with An -t 0 as n + ao (where n is allowed to approach infinity through a subsequence of the positive integers). The channelcapacity is defined as the least upper bound of all permissible transmission rates. Let C be the capacity of the time-discrete, amplitude-continuous Gaussian channel with average power limitation M and "noise" variance N [see (8.2.1)]. We are going to show that

As indicated in Section 7.4, in order to establish (8.2.6) we shall prove a coding theorem which states that C 2 & log (1 M/N) [or equivalently, if R < log (I MIN), we can find codes ([enR],n, 1,) with A, -+ 0 as n -+ a], and a weak converse which states that C j4 log (1 MIN). We first prove the coding theorem.

+

+

Theorem 8.2.2. C 2 4 log (1 Proof. In Lemma 8.2.1, take

P(X) = (2n~.)-'^ exp

+

+ MIN).

(- 2 A) 2M, 5.2

,=I

where M, < M.

235

CONTINUOUS CHANNELS

In other words, choose the components XI, . . . , Xn of the input independently, each component having a Gaussian distribution with mean 0 and variance M, < M. The third term of (8.2.2) then becomes

z;=,

But n-I Xj2 is the arithmetic average of n independent, identically distributed random variables with expected value E(X,") = M, < M. By the weak law of large numbers, the arithmetic average converges in probability to M,, hence P{X # F ) -+0 as n -+ w. Now we note that the output sequence Y is given by Y = X Z, where the components of X are independent Gaussian random variables with mean 0 and variance M,,, and the components of Z = Y - X are independent Gaussian random variables with mean 0 and variance N [see (8.2.1)]. By the form of (8.2.1), the conditional distribution of Z given X is the same as the (unconditional) distribution of Z, and hence X and Z are independent. Since sums of independent normally distributed random variables are also normally distributed, the components of Y are independent Gaussian random variables with mean 0 and variance N M,. With this in mind, let us examine the second term of (8.2.2). We have

+

+

P(Y I x)


log P(Y)

+?)+ii[

e+nlog(~

2 i=1 (N

+y; M,)

- ( ~ -i xiI2]

+

< a.

N

+

Since E(Y,2) = N M, and Let W , = Y,2/2(N M,) - Z,"/2N. E(Z,2) = N, W , has mean 0. We may compute the variance of W , as follows :

Since

Y,/JN + M, and z,/JN

are each normal (0, I),

236

INFORMATION THEORY

+

Also E ( Y j 2 Z j 2= ) E[(Xj2 2XjZj dependent,

+ Z:)Z:];

since X j and Z , are in-

Thus =-, Mo

zj"=,

zj"_, Var

Now let Vn = W,. Then E(Vn) = 0, Var Vn = nMo/(N M,) by independence of the W j ; furthermore,

+

W, =

+

Let us choose a = i n log (1 M o / N ) - nd, where 6 > 0. With this choice, and with the aid of Chebyshev's inequality, the second term of (8.2.2) becomes

Thus P{(X, Y ) ( A ) -+ 0 as n + co. Finally, in Lemma 8.2.1 let u = [enR]where R The first term of (8.2.2) becomes

. e-a

j

< 4 log(1 + M / N ) .

e n ~ e - % nlog ( l - l - M ~ l N ) ~ n d

If M,, is chosen sufficiently close to M, and d sufficiently close to 0, we will have R - -2 log (1 + M o / N ) 6 < 0, and u . e-" will approach 0 as n -t co. The theorem follows.

+

8.3. Uncertainty in the continuous case

Before proving the converse to the coding theorem, we must extend the notion of uncertainty to the continuous case, and examine some of the properties of the resulting uncertainty function. If X is an absolutely continuous random variable, that is, a random variable having a probability density function p(x), we define the uncertainty of X as

237

CONTINUOUS CHANNELS

In contrast to the situation in the discrete case, H(X) may be arbitrarily large, positive or negative. To see this, let X be uniformly distributed between 0 and b. We then have H(X) = log b, which takes on all real values as b ranges over (0, a ) . In fact H(X) may be co or - co. For example, let p(x) = x-l (log x)-~,x 2 e ; p(x) = 0, x < e . Note that

+

Then x-'(log

log x - 2 log log 2) nx m

2 e

m

dx -x log x

- [d(log --

x) log x

+ 00.

P

As another example, let I, (n = 2 , 3 , 4 , . . .) be any system of disjoint intervals of real numbers such that the length of Inis (n log n)+. Let

r

x-l (log x)-2 dx < K = I,"=, nn-l(log n F 2 ; note that K < co since Definep(x) = n/K if x E I,, (n = 2, 3,4, . . .); p(x) = 0 otherwise. Then

S

m

p(x) dx = K-'

2 n-'

(log n)-2 = 1,

n-8

so that p(x) is a probability density function. Furthermore, m

m

H(X) = - 2 (n/K) log (n/K) = log K - K-12 (n log n)-l VL=Z (n log n)' n=2 Rut

x,",,(n log n)-I

= co since

J

(x log x)-I dx = co, so that 2

Note that in the above example, U,"=, Inmay be taken to be a bounded set; if this is done, then all the moments of X are finite, and yet H(X) = - co. We shall see later in this section that the finiteness of the variance of X implies that H(X) < co. If p(x) :) L < co for all x then H(X) 2 -log L > - co, so that boundedness of the density is sufficient to eliminate the case H(X) = - co.

238

INFORMATION THEORY

Now if X and Y are random variables which are defined on the same sample space and which have a joint density p(x, y), we may define the joint uncertainty of X and Y as

and the conditional uncertainty of Y given X as

I

where p(y x) = p(x, y)/p(x) is the conditional density of Y given X. We may prove results that are the continuous analog of the results of Section 1.4. The following lemma is useful. Lemma 8.3.1. Let p(x) and q(x) be arbitrary probability density functions. a. If -

j-mm p(x) log q(x) dx is finite, then -J:mP(x)

logg(x) dx exists,

and furthermore

with equality if and only if p(x) = q(x) for almost all x (with respect to Lebesgue measure). b. If -j:mp(x)

logp(x) dx is finite, then -

lWm

p(x) log q(x) dx exists, and

(8.3.1) holds. Proof. We will prove (a); the argument of (b) is quite similar. Since log b 5 b - 1 with equality if and only if b = 1 (see Fig. 1.4.1) we have p(x) log [q(x)/p(x)] 5 q(x) - p(x), with equality if and only ifp(x) = q(x). (Define q(x)/p(x) = 0 when p(x) = q(x) = 0, and take 0 co = 0.) Thus

-

with equality if and only if p(x) = q(x) for almost all x. Now

239

CONTINUOUS CHANNELS

(Note that there can be no set of positive measure on which q(x) = 0 and p(x)

> 0,

for if there were,

-J

r m

p(x) log q(x) dz would be -m

+ m.

the right side of the previous equation will not be of the form - oo except possibly on a set of measure 0.) It follows that

Thus

+ CQ

m

Finally, if equality holds in (8.3.1), the finiteness of -

[

p(x) logq(x) dx

J-m

allows us to conclude from (8.3.2) that

and hence thatp(x) = q(x) for almost all x. The lemma is proved. Lemma 8.3.1 has the following immediate consequences. Lemma 8.3.2. Let X and Y be random variables with a joint density p(x, y). Assume that H(X) and H(Y) are both finite. Then:

+

a. H(X, Y) exists, and H(X, Y) 5 H ( X ) H ( Y ) , with equality if and only if X and Yare independent. b. H(Y I X) and H(X I Y) exist, and H(X, Y) = H(X) + H(Y X) = H( Y ) H(X Y). c. H(Y / X) 5 H(Y) [also H(X Y) 5 H(X)], with equality if and only if X and Y are independent.

+

I

I

I

Thus if we define the information conveyed about X by Y by Z(X 1 Y) = H(Y) - H(X I Y), we have I(X Y) = I(Y 1 X), and I(X I Y) 2 0, with equality if and only if X and Yare independent. Proof. We note that by Lemma 8.3.l(a),

I

with equality if and only if p(z, y) =p(x)p(y) for almost all (x, y). The hypothesis that H(X) and H(Y) are finite implies that the integral on the right is finite, so that the application of Lemma 8.3.1 is legitimate. This proves (a).

240

INFORMATION THEORY

T o prove (b), we note that

-m -m

[The first integral on the right exists and is < cr, by (a); the second integral on the right is finite by hypothesis. Thus the integral on the left exists, and we have H(Y X) = H(X, Y) - H(X).] Finally, (c) follows immediately from (a) and (b) since H(X, Y) = H(X) H(Y X) H(X) H(Y), with equality if and only if X and Y are independent.

I I

+

+

Just as in Section 1.4, we may extend the notion of uncertainty to random vectors. For example, if X = (X,, . . . , X,), and X,, . . . , X, have a joint density p(x,, . . . ,x,), we define m

H(X)= -

J

-m

1 m

p(x1, . . . , 2,) log p(x,, . . . , r,) dx,

. . . dr,.

-m

The above results will remain valid in the more general context. We shall now prove a series of results which will be needed in the proof of the converse to the coding theorem for the time-discrete Gaussian channel. First, we show that the Gaussian distribution has maximum uncertainty among all distributions with a given variance. Theorem 8.3.3. Let X be an absolutely continuous random variable with density p(x). If X has finite variance a2, then H(X) exists, and H(X) $log (2nea2), with equality if and only if X is Gaussian (with variance a2). Proof. Let X be an arbitrary random variable with density p(x), mean p, and variance a2. Let q(x) = (2na2)-~e-(x-~)a~2a2. But

<

m

-

J

log q(x) dx = Jp(r) =

[i

log ( h a 2 )

+(x 2a2 -

a2 3 log ( 2 n d ) + = + log (2nea2). 2a2

The result now follows from Lemma 8.3.la. We are going to be interested in situations in which a code word is chosen at random and then transmitted through a Gaussian channel.

24 1

CONTINUOUS CHANNELS

Since the number of words in a code is finite, we are applying a discrete input to a channel and receiving a continuous output. We must examine the interplay between the uncertainty functions in the discrete and continuous cases. Let X be a discrete random variable that takes on the values z,, . . . , x,. Suppose that Y is a random variable such that for each i, Y has a conditional density given X = x,, say p(y xi). (In the case that will be of most interest to us, Y = X 2, where Z is a Gaussian random variable.) Then Y is absolutely continuous, with density

I

+

We may define the uncertainty of Y given that X = xi as

and the uncertainty of Y given X as

Furthermore, we may define the uncertainty of Xgioen that Y = y as

I

I

where p(xi y) =p(xi)p(y xi)/p(y) is the conditional probability that X = xi given Y = y. The uncertainty of Xgiven Y is defined as m

I

H ( X Y) =

j P(Y)H(x 1 Y

= Y) dy.

-w

[Notice that the above definitions closely parallel the corresponding definitions in the discrete case; furthermore, H(X Y) is always finite and in fact is bounded by log M.] Now assume for simplicity that H ( Y X = xi) is finite for each i, and that H ( Y ) is finite. It follows that

I

I

with equality if and only if X and Yare independent. For

242

INFORMATION THEORY

As in the proof of Lemma 8.3.1, m

m

I

with equality if and only ifp(y) = p(y xi) for almost all y. Thus

with equality if and only ifp(y) = p(y 1 xi) for almost all (x,, y ) ; in other words, H ( Y X) I; H ( Y ) , with equality if and only if X and Y are independent. Furthermore,

I

For

H(Y)

+ H ( x I y ) = H ( X ) + H ( Y ( x).

(8.3.4)

We define the information conveyed about X by Y , just as in the discrete case, by ' Z(XIY)=H(X)-H(X By (8.3.4), Z(X Y ) = H( Y ) - H(Y

I

I

furthermore, by (8.3.3), Z(X Y ) 2 0, that is,

I

H ( X Y ) I; W X ) , with equality if and only if X and Yare independent. We now prove a convexity result that is analogous to Theorem 3.3.1. Let X be a discrete random variable with values i, Y has a conditional density p(y xi) given that X = xi. Assume H ( Y X = x,) is finite for i = 1,2, . . . , M. Suppose that for each k = I, 2 . . . , n, pk(a), a = xl, . . . , X M , is an assignment of probabilities to the values of X. Let pO(a)be a convex linear combination of the p,(a), that is, p,(a) = z:=l akpk(a),where all a, 2 0 and EL, a, = 1 . Theorem 8.3.4.

XI,.

. . , X M ; let Y be a random variable such that for each

1

I

CONTINUOUS CHANNELS

Then, assuming Hk(Y) is finite for each k = 1,2,

.. .,n,

where the index k indicates that the quantity in question is computed under the assignment p,(u). Proof. We first compute Ho(Y X) :

I

I

I

Sincepo(y xi) is just the conditional density p(y xi),

We must show that

z:=,a k l k ( ~Y)l - Zo(X( Y) 5 0. But

By hypothesis, Lemma 8.3.lb, -

S-mm

p,(y) log pk(y) dy = Hk(Y) is finite, hence by

pk(y) logpo(y) dy exists and

.

If we multiply the above inequality by a, and then sum over k = 1,2, . . , 2~~akHk(Y). The theorem is proved. Note that Theorem 8.3.4 and the discussion preceding it extend immediately to the case in which X and Y are replaced by random vectors.

n, we see that Ho(Y) exists and Ho(Y) 2

8.4.

The converse to the coding theorem for the time-discrete Gaussian channel

We now apply the results of Section 8.3 to prove the converse to the coding theorem. Just as in the case of the discrete memoryless channel, the key step is Fano's inequality (compare Theorem 3.7.1).

244

INFORMATION THEORY

Theorem 8.4.1. Given an arbitrary code (s, n) for a time-discrete Gaussian channel, consisting of words x'l), . . . , x @ ) . Let X = (XI, . . . , X,) be a random vector which takes on the value x ( i )with probability p(x(')),i = I , 2 , . . . , s , where Z;=,p(x(") = 1. Let Y = ( Y , , . . . , Y,) be the corresponding output sequence. If p(e) is the overall probability of error of the code, computed for the given input distribution, then

Proof. The first part of the proof proceeds exactly as in Theorem 3.7.1. If g(y) is the input selected by the decoder when the sequence y is received, then by the grouping axiom (Section 1.2, Axiom 3),

I

where q = P{X = g ( y ) ( Y = y ) = 1 - p(e y) and the qi are of the form

We now must deviate slightly from the proof of Theorem 3.7.1. As in the discrete case, we must show that the uncertainty of a convex linear combination of probabilities is greater than or equal to the convex linear combination of the uncertainties, but in this case the convex linear combination is an integral rather than a summation. However, we may reason as follows. Let V be a random variable that equals 1 if a decoding error occurs, and 0 otherwise. Then

Thus if we multiply (8.4.1) by p(y) and integrate over all y, we find using (8.4.2) that H(X Y ) H(p(e), 1 - p(e)) p(e) log ( s - I), as required. (The fact that the channel is Gaussian implies that all uncertainties appearing in the above proof are finite, and thus there is no difficulty in applying the results of Section 8.3.) We shall need one more preliminary result.

1

+

245

CONTINUOUS CHANNELS

Lemma 8.4.2. Let X = (X,, . . ., X,) be a random vector with a finite number of possible values. Suppose that X is applied as the input to a time-discrete Gaussian channel, and that Y is the corresponding output. Then m

(compare Lemma 3.7.2). Proof. We have I(X Y) = H(Y) - H(Y X); note that since the channel is Gaussian, H(Y) and H(Y X) are automatically finite. We compute

I

I

I

m

2. .

H(YIX)=-

51,.

,En~

(

m

~

~

-m

~

~

~

~

-m

x logp(y1, - . ., Y% I 21, . .

1

xn) dyl

' '

dyn

where p(yl, . . . , y, XI, . . . ,x,) is given by (8.2.1). Since p(yl, . . . ,y, x,, . . . , x,) = p(yl x,) . . .p(y, x,), it follows that H(Y X) = H(Y, Xi). Since H(Y) = H(Yl, . . . , Y,) 5 22,H(Yi) by Lemma 8.3.2a, the result is established. We now prove the weak converse.

1 z:=,

I

I

I

1

Theorem 8.4.3. Given a time-discrete Gaussian channel with average power limitation M and noise variance N. Any code (s, n) for such a channel must satisfy nC, log 2 log s I 1-p(e>

+

+

where C, = 4 log (1 MIN) and -p is the average probability of error of the code. Proof. Let the code words be x(l) = (x,,, x,,, . . . ,x,,), x ' ~ = ) = ( ~ , 1 , ~ , 2., . ,x,,). Let X = (XI,. . . , X,,) (xZlrxZ2,. . . , x2,), . . . , be a code word chosen at random, with all words equally likely, and let Y = ( Y , , . . . , Y,) be the output sequence when X is transmitted. Now the overall probability of a decoding error associated with the transmission of a code word chosen according to the uniform distribution is p(e) (see Section 3.5). Thus by Theorem 8.4.1,

I

H(X Y)

5 ~ ( ~ - P3O ) ~ + p m 1 log (s - 1) 5 log 2 + p(e) log s.

(8.4.3)

Again using the fact that all words have the same probability, we have

1

Z(X Y) = H(X)

- H(X I Y) = l o g s - H(X I Y).

(8.4.4)

~

~

246

INFORMATION THEORY

By Lemma 8.4.2,

Thus far the proof is an exact duplicate of the argument of Theorem 3.7.3, but we must now deviate. We cannot conclude that I(Xi Yi) j Co for all i, but we shall show that

I

I t will then follow from (8.4.3)-(8.4.6) that nCo 2 log s - log 2 p(e) log s, as asserted. - To prove (8.4.6), we proceed as in Problem 3.11b. Let X be chosen with uniform probability from the sn digits xij, i = 1, 2, . . . ,s,j = 1, 2 , . . . ,n, of the code vocabulary. Since X may be selected by first choosing a digit j at random and-then selecting a code word at random and examining the jth digit of the word, we have n

P{X = a ) = n-' 2 P { X 3 = a ) . ?=I

Let Y be the output when Xis transmitted. It follows from Theorem 8.3.4 that R (8.4.7) I ( x 1 y ) 2 n-'21(x3 1 y,h 3=1

Since the average power limitation of the channel is M, we have n-I x , , ~j M, i = 1, . . ,s. It follows that the variance of X is S M , for

z;==,

.

s

E(X2) = (sn)-'

n

(I

2 2 xZ3'j s-' 2 M

= M,

r=l

1=13=1

and Var X = E ( x ~ )- EX)^ j E ( x ~ )j M.

+

Now we recall that Y = X 2, where Z is Gaussian with mean 0 and variance N, and X and Z are independent. (See the proof of Theorem N. By Theorem 8.3.3, 8.2.2.) Thus Var Y = Var X Var Z j M

I

+

+

H( Y ) I i log [277e(M

+ N)1.

But H(Y X) = H(Z) (compare Problem 1.9); by Theorem 8.3.3, H(Z) = log (277eN), so that Z(X Y) = H(Y) - H(Y X) j $ log (1 MIN) = C,,. By (8.4.7), Z,"=,Z(X, ( Y,) j nCo, and the proof is complete.

I

I

+

We now prove the strong converse to the coding theorem, that is, the result that if the transmission rate is maintained above Co, the probability of error must approach 1 with increasing code-word length. Specifically, we have the following theorem.

247

CONTINUOUS CHANNELS

Theorem 8.4.4. If E > 0 and 0 5 I < 1 , then for sufficiently large n, any code (s, n, I ) for the time-discrete Gaussian channel with average power limitation must satisfy

+

l o g s < n(C, E) where C, = 4 log (1 MIN). Proof. The proof is a "sphere-packing" argument similar to the proof of Theorem 4.8.1. In that proof we estimated the number of sequences in the decoding sets; here we estimate the volume of the sets. Let the code words be x ( l ) ,. . . , x(*)and let the corresponding decoding sets be B,, . . . , B,. First we approximate each B, by a bounded set. Let 6 be a fixed positive number, and define B,* = Bi n D,, where D, is the sphere in Euclidean n-space with center at the origin and radius [n(M N a)]%. It follows that if X = xci) is transmitted, the probability that the output sequence Y will fall outside of B,* is

+

+ +

+

P{Y # B , * IX = x ' ~ )j) P{Y # B , I X = x ' , ) ) P{Y # D,IX = x ( ~ ) ) 5 I P{Y # D, X = x ( , ) ) . (8.4.8) We shall show that

+

I

P{Y # D, I X = x ( , ) )-t 0 as n + co.

(8.4.9)

+

Let x ( ~= ) (xil, . . . , xi,), Y = (Yl, . . . , Yn) Then Y = x(,) Z, where the components Z,, . . . ,Z, of Z are independent Gaussian random variables with mean 0 and variance N. It follows that

The first term of (8.4.10) is < M by the average power limitation; the second term converges in probability to N by the weak law of large numbers. The variance of the third term is

By Chebyshev's inequality, the third term converges in probability to N 8 ) 40 as n + co. But since zero. Thus P{n-I zl=,Yk2> M this probability is P{Y # D, X = xci)), (8.4.9) is established. Thus we have a new system of decoding setst Bi* such that Bi* C D, for i = 1, 2, . . . , s. For n sufficiently large, the new code (s, n, A*) has, 1)/2 < 1. by (8.4.8) and (8.4.9), I* arbitrarily close to I , say I* 5 (1

I

+ +

+

t Strictly speaking, the sets B,* do not form a decision scheme as defined in Chapter 3, since the sets do not exhaust the space En. However, the B,* may be extended to form a partition of En. The nature of the extension has no effect on the proof.

248

INFORMATION THEORY

Now we shall find a lower bound on the volume of the decoding sets. The lower bound arises from the fact that the decoding sets must be large enough to make the probability of correct transmission at least 1 - 1* for each code word. This result, plus the upper bound on the volume resulting from the requirement that B,* c D,, will yield the strong converse. Let E , be the smallest positive number such that

Note that to N .

E , +0

as n + co since n-l

I:='=, Zk2 converges in probability

Fig. 8.4.1. Proof of Theorem 8.4.4.

Let G, be a sphere of radius [nN(1 - €,)I% and center at xci). We claim that (8.4.11 ) i = 1 , 2 , . . . , s. volume (B,*) 2 volume (G,), To establish (8.4.1I), suppose that the volume of B,* is less than the volume of G, for some i. We shall then prove that P{Y

6

I

B,* X = xc"}

But P{Y

E

< P{Y E G, I X

I

B,* X = x"'} 2 1 - (1

= xc')}.

(8.4.12)

+ 1)/2

and

This contradicts (8.4.12), and therefore establishes (8.4.1 1 ) . To verify (8.4.12), we examine Fig. 8.4.1. Let yo be any point on the surface of G,. Since the Gaussian density p,(y x ) of (8.2.1) increases as

1

CONTINUOUS CHANNELS

the distance from x to y decreases, it follows that

(B,* - G, is the set of points belonging to B,* but not to G,.) Similarly,

S

P{Y r G, - B,* 1 X = x'") = p,(y a,-B~*

j

2

PAYO1 x'")

I x'")

dy (8.4.14)

I

dy = pn(y0 x"')[volume (Gz - B,*)l

n,-B,' Subtracting (8.4.13) from (8.4.14) we obtain

2 pn(yo( volume (G, - Bi*) - volume (B,* - G,)], or P{Y r G i I X = x ( Q )- P{Y r B i * IX = x ' ~1)

2 pn(yoI volume (Gi) - volume (B,*)].

(8.4.15)

Thus if volume (B,*) < volume (Gi), then (8.4.12) follows from (8.4.15). Finally, the bound (8.4.11) and the requirement that each B,* be a subset of D, imply that I

2 volume (G,) I volume i-

volume (D,).

(8.4.16)

1

If V, is the volume of an n-dimensional sphere of radius 1, then (8.4.16) yields s Vn[nN(l - E,)]"/~j V,[n(M N 6)]"12, or n M + N + b log s j- log 2 N(l - En)

+ +

If d is chosen sufficiently small and n is sufficiently large, log s The proof is complete.

< n [ t log (1 + MIN) + EI.

INFORMATION THEORY

8.5.

The time-continuous Gaussian channel

In this section we shall try to construct a reasonable model for a channel that has as its input a function s(t) defined on the interval [- T/2, T/2], and whose output is obtained from the input by the addition of a sample function n(t), -TI2 5 t 5 T/2, of a stationary Gaussian random process. We shall try to show that such a channel may be regarded as a natural generalization of the time-discrete Gaussian channel. In the discussion to follow, we must use certain results about Hilbert space and second-order random processes. The relevant material is summarized in the Appendix. Let n(t), - w < t < oo, be a stationary Gaussian process with zero mean, continuous covariance function R(T), and spectral density N(o). We assume that for each T, 0 is not an eigenvalue of the integral operator on L,[-T/2, TI21 associated with R, that is, the operator defined by

-TI2

By the Karhunen-Lokve theorem (Appendix, Theorems 3.4, 3 . 9 , the restriction n,(t) of n(t) to the interval [- T/2, TI21 may be represented by w the equation nT(t> = Zkek(t;T ) ,

2

k=l

where the ek are an orthonormal basis for Lz[-T/2, TI21 consisting of eigenfunctions of the integral operator mentioned above, and the Zk are orthogonal (hence independent) Gaussian random variables with mean 0 and variance 1,(T) = the eigenvalue of e,. Specifically, 2, =

I,, TI2

n(t)ek(t; T) dt, where the integral is defined in a mean square sense

(Appendix, Section 3). The Karhunen-Lotve series converges in mean square (and since the process is Gaussian, the series also converges with probability 1) for each t E [- T/2, T/2]. Now (since 0 is not an eigenvalue), the functions e, span Lz[- T/2, T/2] (Appendix, Theorem 1.14). Thus each function s, E Lz[- T/2, T/2] has an expansion of the form R=l

where the series converges in L,[- T/2, T/2]. Thus the function s, is determined by the sequence of numbers s,(T), and the process n, is essentially determined by the sequence of random

251

CONTINUOUS CHANNELS

variables Zk. The above discussion suggests that in effect the channel adds to the sequence of numbers sk(T) the sequence of random variables Z,, or equivalently adds to the sequence of numbers sk(T)[Ik(T)]-% the sequence of random variables Zk*= Zk[Ik(T)]-%. Just as in the time-discrete case, it will turn out that if no constraint at all is put on the channel inputs, it will be possible to transmit at arbitrarily high rates with arbitrarily small error probabilities. Let us examine the form an "average power" limitation might take. The "energy" xkek(t; T) is in a signal of the form x(t) =

z:?,

and hence the "average power" in the signal is

We are led to the following definition. Let K be a positive real number, fixed for the remainder of the discussion. Any sequence {Zk*,k = l , 2 , . . .) of independent Gaussian random variables with mean 0 and variance 1 is said to define a timecontinuous Gaussian channel. A code (u, T) for such a channel is a set of real sequences x(') = (xil, xt2,. . .), i = 1, 2,. . . ,u, called code words, such that m

together with a decision scheme, that is, a partition of the set of all sequences of real numbers into disjoint Bore1 sets B,, . . . ,B,. The ) P{x(~) Z # B,), probability of error when x(') is transmitted isp(e x ( ~ )= where Z = (Z,*, Z2*, . . .). A code (u, T, 8) is a code (u, T) with maximum probability of error 5 B, that is, max p(e x")) 5 B. A number R is

I

+

I

lSiSu

called a permissible rate of transmission for the channel if there exist codes ( [ e H T ]T, , BT) such that BT 4 0 as T -+ CQ. ( T is allowed to approach infinity through a subsequence of the positive reals.) The channel capacity C is the supremum of all permissible transmission rates. It is shown in the Appendix (Theorem 4.5) that a way of generating inputs which satisfy the "average power" constraint for the time-continuous Gaussian channel is the following. Let s be any function in L,(- CQ,w) whose Fourier transform S vanishes whenever the spectral

252

INFORMATIONTHEORY

density N of the process n(t) vanishes. Assume in addition that

Then for any T, the restriction sT(t) = interval [- T / 2 , T / 2 ] satisfies

z,"=,

sk(T)ek(t;T ) of s to the

and thus corresponds to an allowable channel input. The capacity is established by the following theorem. Theorem 8.5.1. C = K/2. Proof. We first prove a coding theorem which states that C 2 K/2, or equivalently if R < K / 2 there exist codes ( [ e R T ]T, , b,) with b, 0 as T + co. Let T be a positive integer, and let n = aT, where a is a fixed positive integer. Consider the time-discrete Gaussian channel with noise variance 1 and average power limitation KT/n = K / a . By Theorem 8.2.2, if R' < log (I K / a ) , there exist codes ( [ e n K ]n, , E,) for this channel with L, 0 as n -+ co. If (x,, . . . ,x,) is a code word for the time-discrete channel, we may form a code word for the time-continuous channel by taking x = (xl, x2,. . . ,x,, 0,O, . . .). Since

-

-+

+

such a procedure is allowable. If Bn is the decoding set of (x,, . . . , x,) in the time-discrete channel, we take the decoding set B of x to be the cylinder with base B,, that is, B = { y : ( y l ,. . . ,y,) c B,). In this way we construct a code for the time-continuous channel with precisely the same error probabilities as the corresponding code for the time-discrete channel. Since nR' = (aR1)T, it follows that if R = uR' < ( a / 2 )log(1 Kla), there exist codes ( [ e R T ]T, , pT) with BT + 0 as T -+ m. Hence C 2 ( a / 2 )log (1 Kla). But this holds for arbitrary a ; therefore we may let a co to obtain C 2 K / 2 . We now prove the weak converse to the coding theorem, that is, C K/2. To do this we shall show that any code (u, T , P) with 0 < b < must satisfy &KT log2 log u < 1 - 28

+

-

+

+

+

253

CONTINUOUS CHANNELS

Let x"), . . . , xcU) be the code words and let B,, . . . ,B, be the corresponding decoding sets. The idea is to approximate the given code by a code for the time-discrete Gaussian channel. To do this, we first approximate the sets B, by "finite-dimensional" sets, that is, cylinders. We appeal to a standard approximation theorem [for example, Halmos (1950), p. 56, Theorem Dl which impliest that for each Bi there is a cylinder B,*, whose base B,*,, is a Borel set in Euclidean n,-space, such that

Bi* = (B, - B,*) u (B,* - B,) is the symmetric difference where B, between B, and B,*. (The notation Bi - B,* signifies, as before, the set of elements which belong to Bi but not to B,*.) The sets B,* may not be disjoint, but if we define sets A , by A, = Bi* -

U B,*, j#i

the A, are a disjoint collection of cylinders. If we set Yi = x ( , )

+ Z, we have

B - 2 P{Y, E Bi* n B,*). 2 P{Y, E B ~ }- 2~

j+i

We claim that Bi*

n B,*

c (B,

A B,*) u (B,

B,*)

for i # j.

To see this, write B,* n B,* = [(Bi* n B,) U (Bi* n I,)] denotes complementation). Since the Bi are disjoint, Furthermore, Bi* n B,

n B,*

c Bi*

n B,* (the bar

- B,. The assertion follows. Thus

Since the number of code words is finite, there is an integer n such that the base of each cylinder A i is n-dimensional, that is, for each Ai there is a Borel set A,, in Euclidean n-space such that (y,, y,, . . .) E A, if and only if (yl, yz, . . . ,y,) E Ain. We may now form a code (u, n, 28) for a timediscrete Gaussian channel with noise variance 1, using the code words

t If

p, is the measure induced by xu) B,* such that p(B, A Ed*)j p/2u.

U

+ Z, and ,u = 2 pk, then there is a cylinder k=l

254

INFORMATION THEORY

(xi,, . . . ,xi,) = the first n coordinates of x ( ~ )and , the decoding setst A,. Since

w(%) =

Theorem 8.4.3 yields log u

+ log 2 < ( 4 2 ) log (11+-KTIn) 28

+

-

But f(x) = x log (1 blx) is an increasing function for x > 0 if b is positive [the derivative f '(x) is decreasing for x > 0 and f '(x) 0 as x -,oo, so that f '(x) > 0 for x > 01. Now lim x log (1 blx) = b, *-a hence KT12 log 2 log < 1 - 28 The proof is complete.

+

+

It is possible to give an explicit procedure for constructing codes for the time-continuous Gaussian channel which maintain any transmission rate up to half the channel capacity with an arbitrarily small probability of error. Before giving the procedure, we need some preliminary results. Lemma 8.5.2. Given any code (u, n) for the time-discrete Gaussian channel, the ideal observer for the code, calculated for all code words equally likely, is a minimum-distance decoder; that is, for each received sequence y = (y,, . . . ,y,) the decoder selects the code word x = (x,, . . . , x,) which minimizes the Euclidean distance (xi - Y,)2]'/6between x and y. Proof. The discussion in Section 3.4 may be taken over essentially in toto; the ideal observer for a given input distribution selects the input x which maximizes the conditional probabilityp(x y) that x was transmitted given that y is received. If all inputs are equally likely, then the decoder selects the input which maximizes p(y x), where p(y x) is given by (8.2.1). The result then follows from the form of the Gaussian density.

[z&, I

I

I

It follows from Lemma 8.5.2 that if all code words are equidistant from the origin, the ideal observer with equally likely inputs selects the code word x which maximizes xiyi. Let us call this procedure the correlation detector. We have the following result.

z:=,

Lemma 8.5.3. Let Y = (Y,, . . ., Y,) be the received sequence when the input x(l) = (x,,, . . . ,x,,) is transmitted through a time-discrete

t The footnote to Theorem 8.4.4 is applicable here also.

255

CONTINUOUS CHANNELS

+

Gaussian channel, that is, Y = x(') Z, where Z = ( Z , , . . . ,Zn) is a sequence of independent Gaussian random variables with mean 0 and ) (xZ1,. . . , xZn)is any point in Euclidean n-space, variance N. If x ( ~= define y = y(x('), x ' ~ ) = ) 2:=, xlkx2,. Also define G(x(~)) = 2:=, xtkYk, i = l,2. Assume that 2,"_, x , ~ = L, i = 1,2. Then P{G(X(~)) 2 G(x('))}, which is the probability that the correlation detector will favor x ( ~over ) x'l), is given by

where F*(r) =

1-

(2n)-'*e-8212ds is the distribution function of a

normally distributed random variable with mean 0 and variance 1. Proof.

=y

-L

+ 2 ( ~ 2 ,- z,,)Z,. k=l

Thus G ( x ( ~ )) G(x(l)) is normally distributed with mean y - L and (x,, - x,,)~ = 2N(L - y). variance N Hence

z;==,

and the result follows. Note that lyl sums.

5 L by

the Schwarz inequality for

Now suppose that we have a code (u, n) for the time-discrete Gaussian channel, with the code words x(') = (x,, . . . ,zi,) mutually orthogonal; that is, y(x('), x")) = 0 for i # j. Assume that xik2= L for all i = 1,2, . . . , u and that correlation detection is used. We may estimate the probability of error as follows. Suppose that x(l) is transmitted (the argument will be the same for any code word). By definition of the correlation detector, an error in decoding implies that least one G(x(~)), i > 1, is >G(s(l)). Thus, using Lemma 8.5.3, we have

zc=l

p(e I x'")

5 P[G(x'") 2 G(x'")

for at least one i

2 21

256

INFORMATION THEORY

Since F*(-t) = 1 - F*(t),

J

dL/2N (2n)-xe-xz/z dz 5 (2n)-%a-l eca'Iz if a

m

But

[

> 0.

[This is a standard

Ja

property of the normal distribution; see, for example, Feller (1950), Chapter 7.1 Thus the maximum probability of error cannot exceed

We may use this result to construct explicit codes for the time-continuous Gaussian channel. Theorem 8.5.4. Given a time-continuous Gaussian channel (with an associated fixed constant K) and a positive real number T, let u = [ e H T ] , where R < K / 4 . Construct a code (u, u) for the time-discrete Gaussian channel with noise variance 1 such that the code words are mutually orthogonal and the Euclidean distance between each word and the origin is (KT)%. (For example, we could take the ith component of the ith code word to be KTand all other components to be 0.) Assume that correlation detection is used. Now convert the code to a code for the time-continuous Gaussian channel, as described in the proof of Theorem 8.5.1. If BT is the resulting maximum probability of error, then pT 4 0 as T + oo. Proof. We use (8.5.1) with N = 1, L = KT, and u = [eRT]to obtain

pT 5 ( 7 T ~ ~ )eRT - x e-KT/4, and the result follows. 8.6. Band-limited channels

It is perhaps appropriate to close the book on a note of controversy. Many communication situations involve signals that are essentially "band-limited," that is, signals whose frequency spectra (Fourier transforms) are essentially confined to a finite interval. However, there is no agreement on a proper model for a channel whose inputs and outputs are band-limited. We shall examine and compare several suggested models. Let n(t), t real, be a stationary Gaussian process with zero mean, continuous covariance R(T), and spectral density N(w). The process is said to be band-limited if there is a finite interval [-2n W, 2n W] such that

CONTINUOUS CHANNELS

257

N(o) vanishes outside the interval. W is called the bandwidth of the process. We might propose as a model for a band-limited channel the time-continuous Gaussian channel associated with the given process n(t). However, the inputs to such a channel correspond to functions defined on the interval [-- T/2, T/2]. I n effect the signals to be transmitted are "time-limited," that is, vanish outside a finite interval. Now a function in L,(- oo, oo) (other than the zero function) cannot be both time-limited and band-limited. For if the Fourier transform F of the function f vanishes outside the interval [ - B , B] then f is given by

The integral in (8.6.1) may be regarded as defining a function of the complex variable t ; we may verify by direct differentiation that the function is analytic everywhere in the complex plane. If such a function vanishes for It1 > T/2, t real, then it must vanish everywhere, and hence f must be the zero function. Thus the inputs to the time-continuous Gaussian channel cannot correspond to band-limited signals. Shannon (1948) takes the following approach. He considers "flat" band-limited noise; that is, he takes the spectral density to be N(w) = N/2, -277 W 5 o 5 277 W; N(w) = 0,101 > 277 W. The corresponding covariance function is R(T) = NW

sin 2rW-r 2n WT

Shannon forms the allowable inputs to the channel in the following way. He takes a signal which is limited to the same frequency band as the noise, that is, a function s E Lz(-co, oo) whose Fourier transform vanishes outside the interval [ - - 2 ~W, 2.rr W]. The function s is assumed to satisfy the constraint a0

( 2 ~ ) - ' / ~dw ' 5 KT, N(w) where the integrand is defined to be zero when N(w) = S(w) = 0. If we set K = 2MIN for convenience, the constraint becomes

258

INFORMATION THEORY

or, using the Parseval relation,

The signal transmitted through the channel is the restriction of the function s to the interval [ - T / 2 , T / 2 ] . The output is obtained from the input by adding n(t), It1 5 T/2. However, implicit in Shannon's development is the assumption that the "tail" of s, that is, the function s(t), > T/2, is not eliminated, but remains to interfere with signals which may be transmitted in the future. T o arrive a t an expression for a channel capacity, Shannon reasons as follows. He first establishes the sampling theorem, which states that if s E Lz(- a,a)and the Fourier transform of s vanishes outside [-27r W, W],then

(11

,,

where the series converges in L,(- co, co). Thus the functions is determined by its values a t a sequence of "sample points" spaced 112 W apart. Now the Fourier transform of (sin 2 n W t ) / 2W~t is ( 2W)-lu(a)),where u(w) = 1, 1 0 1 5 2 n W ; u(w) = 0, Iwl > 2nW. Thus the Fourier transform of e-'oki2w sin 2nW(t - k / 2 W ) is -4 27rW(t - k / 2 W ) 2W

~).

The Parseval relation shows that the functions gk(f) =

sin ( 2 nW t - k7r) , 2nWt - k n

k = O , f l , f 2 ,...,

are mutually orthogonal, and

I t follows that

Now suppose that in forming code words, we restrict our attention to functions whose sample values vanish outside the interval [ - T / 2 , T/2].

CONTINUOUS CHANNELS

259

Specifically, assume that the s(k/2 W) are 0 for k/2 W 5 - TI2 or k/2 W > T/2, so that there are at most 2 WT nonzero samples. If we let the 2 WT numbers s(k/2 W), - T/2 < k/2 W _< T/2, be labeled x,, . . . , x,, with n = 2 WT, then the constraint (8.6.2) becomes, with the aid of (8.6.3), n

n-I

2 xs2I M. r=l

Since the transmitted signal is determined by 2 WT sample values, it is possible to decode by examining the output at 2 WTpoints spaced 112W apart. But R(7) = 0 when 7 = k/2 W, k = f 1, f2, . . . ; thus for any t the random variables n(t k/2 W), k = 0, f 1, f2, . . . , are orthogonal, hence independent. Since the variance of n(t) is R(0) = NW, the situation is completely analogous to the case of a time-discrete Gaussian channel with average power limitation M and noise variance NW. Thus if R, < 4 log (I MINW), or equivalently if R = 2 WR, < W log (1 MINW), we can find [enR1]= [eHT] code words and a corresponding decision scheme with a maximum probability of error which approaches zero as n (and hence T) approaches infinity. Since each code word is derived from a band-limited signal whose nonzero sample values are concentrated in the interval [- T/2, T/2] there will be no interference with the transmission of signals whose nonzero sample values are restricted to the intervals [T/2,3T/2], [3T/2,5T/2], etc. Shannon concludes that the channel M/N W). However, since Shannon assumes a capacity is W log (1 specific decoding procedure and a specific method of choosing code words, it is possible to conclude from his results only that the capacity is 2 W log (1 MINW); the evaluation of the actual capacity is an open problem. [Observe that Shannon's expression approaches M / N as W+ m ; this is the capacity of the time-continuous Gaussian channel with K = 2MIN. [Recall that we set K = 2M/N in the discussion preceding (8.6.2).] This fact may be justified physically if we observe that in the case of infinite bandwidth, we may solve the interference problem by using time-limited code words. Hence in the extreme case of infinite bandwidth (the so-called "white noise" case), the channel we are studying should be equivalent to the time-continuous Gaussian channel as defined in Section 8.5.1 Recently, Wyner has found a model for a band-limited channel in which the expression W log (1 MINW) appears as a true capacity. We shall briefly outline his results. Consider the time-continuous Gaussian channel with flat band-limited noise and K = 2MIN. The definition of a code is modified by placing another constraint on the code words. In addition to the restriction that ZF=, xk2 (2M/N)T, it is required that , 2 W T xk2 = Tq(T), where q(T) 0 as T-+ a. Wyner shows that the allowable code words correspond to band-limited signals, truncated

+

+

+

+

+

+

zr

-

260

INFORMATION THEORY

to the interval [-T/2, T/2], with the restriction that the fraction of the signal energy which is concentrated in the interval must approach 1 as T+ co. [The argument uses the properties of band-limited signals derived by Slepian and Pollak (1961) and Landau and Pollak (1961,1962.)] If we examine the steps in the derivation that C K/2 for the timecontinuous Gaussian channel (the direct half of Theorem 8.5.1) for the special case a = 2 W, we see that the capacity of the channel just described is at least W log (1 MINW). The proof of the converse to the coding theorem and a further discussion of models for band-limited channels will appear in a forthcoming paper by Wyner.

>

+

8.7. Notes and remarks The time-discrete Gaussian channel was introduced by Shannon (1948), who observed that the expression & log (1 MIN) is the maximum information conveyed about the input X to the channel by the output Y, when X i s restricted to be an absolutely continuous random variable with variance j M . (See the end of the proof of Theorem 8.4.3.) A complete analysis of the channel, including precise exponential bounds on the probability of error, was given by Shannon (1959). The approach we have taken here is that of Thomasian (1961), who gave a proof, similar to that of Theorem 3.6.4, that the probability of error approaches zero exponentially with increasing code-word length if the transmission rate is maintained below channel capacity. The proof of Theorem 8.4.4 is due to Wolfowitz (1961). As indicated in Section 8.6, Shannon (1948) considered time-continuous channels with additive band-limited Gaussian noise. The general timecontinuous Gaussian channel was first analyzed by Bethoux (1962) [see also Fortet (1961)], who, using a different approach from that of Section 8.5, showed that the capacity was at least K/2. The approach in the text is that of the author (1963, 1964), who proved that the capacity is in fact K/2 and noted that the probability of error approaches zero exponentially as T - t co for transmission rates below capacity. (This follows directly from the corresponding exponential bound for the time-discrete Gaussian channel.) A proof of the strong converse for the time-continuous Gaussian channel has been given by Yoshihara (1964). Lemma 8.3.1 may be used to advantage in certain extremal problems involving the uncertainty function. For example, if we restrict ourselves to the class of random variables having densities that vanish outside the finite interval [a, b], and we take q(x) to be the uniform density over [a, b], we find that H(X) 5 log (b - a), with equality if and only if X is uniformly distributed between a and b. As another example, if we restrict

+

CONTINUOUS CHANNELS

26 1

ourselves to the class of nonnegative absolutely continuous random variables with finite mean p = l/A, and we take q(x) = Ae-" (x (z O), we have the result that H ( X ) I; 1 - log A, with equality if and only if X has the exponential density Le-"". Some results on the explicit construction of codes for the time-discrete Gaussian channel are given by Slepian (1965). An adaptation of an argument in Fano (1961), pp. 200-206, shows that the orthogonal coding technique of Theorem 8.5.4 actually realizes the channel capacity, i.e., the theorem holds under the hypothesis that R < C = K/2.

Appendix We collect here enough material from Hilbert space theory to allow a proof of the Karhunen-Loive expansion theorem for random processes, and to obtain some specialized results that are needed for the analysis of the time-continuous Gaussian channel in Chapter 8. As a byproduct, we shall derive some results from the classical theory of integral equations. The material in the first three sections is standard, but does not all appear in a single place. The major references we have used are Liusternik and Sobolev (1961), Taylor (1958), and Riesz and Sz.-Nagy (1955). We assume the elementary facts about Hilbert space which can be found, for example, in Liusternik and Sobolev (1961), pp. 73-80. 1. Compact and symmetric operators on L,[a, b]

We consider the real Hilbert space L, = L2[a, b] of all real-valued (Borel) measurable functions x on the finite or infinite interval [a, b] such that p 2 ( t ) dl

< a. The innerproduct of two functions x and y is (2, Y) = ]x(tMt) dt a

and the norm of a function x is

<

By the Schwarz inequality, l(x, y)l llxll 11 yll for all x, y r L,. Let A be an operator on L,, that is, a function which assigns to each x E L, another element A(x) r L,. A is said to be linear if A(u,x, a,x,) = cr,A(x,) u,A(x,) for all 2,, x, E L2 and all real a,,u,. A is said to be continuous if x, -+ x (that is, llx, - xi1 -+ 0) implies A(x,) -+A(x) for every sequence of points x, r L, converging to a point x r L,. The norm of the linear operator A is defined by llA/l = sup llAx/l; A is said to be bounded if IIAII < co.

+

1lxll=1

262

+

263

APPENDIX

The results to be derived in this section will be valid in any Hilbert space, but since our main interest is in L,, the results will be stated in this context. Lemmu 1.1. If A is a linear operator on L,, then A is continuous if and only if A is bounded. Proof. If A is unbounded, there is a sequence of points x , c L, such that llxnll = 1 for all n, and llAxnll + m. Let y, = xn/11Axnll. Then y , + 0, but since A y , has norm 1 for all n, A y , -0. Thus A is not continuous. Conversely, if A is discontinuous there is a sequence of points x , -+ 0 such that Ax,+ 0. Hence there is an E > 0 and a subsequence of points x,(,, such that IIAx,(,,ll 2 E for all k. Let yk = x,~,,/llx,~,,ll. Then llykll = 1, but 11 Aylcll 2 e/1Ixn,,,11 4 m. Consequently A is unbounded.

Note that if x # 0, then

so that for all x we have 11 Ax11

< IlAll

11x11 ; consequently

A linear operator on L, is said to be compact (also called completely continuous) if for every bounded sequence of points x , c L,, (that is, every sequence such that for some K < w we have llxnll K for all n), the sequence of points A x , has a convergent subsequence. Note that a compact linear operator is continuous. For if A is discontinuous, then A is unbounded by Lemma 1.l, and hence there is a sequence of points x , c L, such that llxnll = 1 and 11 A x J + w. Then { A x , ) could not possibly have a convergent subsequence, for such a subsequence would have to be bounded. As an example, the identity operator [ A ( x ) = x ] on L, is not compact. Let x,(t) = T-% sin nt on [0, 2 ~ 1 ;then llxn(l = 1 for all n, but

<

Thus the distance ) J x ,- xmJJbetween x, and zrnis hence { x , ) can have no convergent subsequence.

J2 for n # m,

and

Now let A be a compact linear operator on L,. For each real number A, let A, = A - I t , where I is the identity operator. We define two sets

264

INFORMATION THEORY

p(A) and a ( A ) as follows. We take p(A) to be the set of all real 1 such that the operator A , is both one to one and onto, that is, the set of all A such that A,-l exists. We take a ( A ) to be the set of all real A not belonging to p(A), that is, the complement of p(A) with respect to the reals. [Note that we are not using the terminology "resolvent set" and "spectrum" for p(A) and a ( A ) since we are dealing exclusively with real scalars.] We shall use the following two results without proof.

Theorem 1.2. If B is a continuous linear operator on L, and B-I exists, then B-I is continuous. [See Liusternik and Sobolev (1961), p. 881. Theorem 1.3. If A is a compact linear operator on L , and A is any nonzero real number, then A , is one to one if and only if it is onto. Theorem 1.3 supports the contention that compact operators are a natural generalization of matrix operators on a finite-dimensional space. For a proof, see Taylor (1958), p. 279, Theorem 5.5E. Lemma 1.4. Let A be a compact linear operator on L,, and A a nonzero real number. Then 1 E a ( A ) if and only if there is a sequence of points x , E Lz such that llxnll = 1 for all n and A,%, + 0. Proof. Assume a sequence with 11x,11 = 1 and A,%, -t 0. If 1 f a ( A ) , then A,-l exists. Since A,-l is continuous by Theorem 1.2, we have x , = AA-lA,x, -+ 0, contradicting llx,l/ = 1. Conversely, if 1 E a ( A ) then A,-l does not exist. By Theorem 1.3, A , is not one to one, hence there is a nonzero x E Lz such that A,x = 0. If we take x , = x/11x11 for all n, then llxnll 3 1 and A,%, = 0. The result follows. A real number 1 is called an eigenualue of the linear operator A if Ax = 1x for some nonzero x . Any x E L , such that Ax = Ax is called an eigenuector or eigenfunction corresponding to 1 . Theorem 1.5. If A is a compact linear operator on L , and A r a(A),

A # 0, then 1 is an eigenvalue. Proof. The result is immediate from Theorem 1.3. However, an alternate argument based on Lemma 1.4 is of interest. By Lemma 1.4, there is a sequence of points with llx,ll = 1 and A,x, -+ 0. Let y , = A p , ; since 1 # 0 we have x , = Akl(Ax, - y,). Since A is compact, { A x , ) has a convergent subsequence; since y , -t 0, { x , ) has a convergent subsequence. Suppose that x,(,, = A- AX,,,, - y,(,,) -+ x . Now y,(,, -+ 0, and hence A - ~ A x , ( ~-+, x, or Ax,,k) -+ Ax. But -+ Ax by continuity of A ; thus A x = Ax. Since x,(,, -+x, we have llxll = 1 ; in particular, x # 0. The theorem follows.

265

APPENDIX

A linear operator A on L, is said to be symmetric (or self-adjoint) if ( x l , Ax,) = (Ax,, x,) for all x,, x, E L,. The quadraticform of a symmetric linear operator A is a function from L , to the reals defined by

Q ( x ) = ( A x , x ) = ( x , Ax).

The norm of the quadratic form Q is defined by 11 Qll = sup lQ(x)l. Il*!l=l

Theorem 1.6. lf Q is the quadratic form of the symmetric linear ( (IA((. operator A on L,, then ( ( Q (= Proof. We first note that

Now by the Schwarz inequality, I(Ax7 x)l

I l l A 4 llxll I IlAll llx1I2;

if llxll = 1 we obtain lQ(x)l I IIAll; consequently IlQll 11 Qll 2 11 A 11 we must introduce a bilinear form

I IIAll.

To prove

= 4(x1,Ax,).

Thus

(The last step may be verified by direct expansion; it is an instance of the so-called parallelogram law in Hilbert space, that is, the statement that the sum of the squares of the diagonals of a parallelogram equals twice the sum of the squares of the sides.) Therefore if Ilxlll = llx,ll = 1 then la(% %>I I Il Q l l . Now

266

INFORMATION THEORY

Theorem 1.7. Let A be a compact symmetric linear operator on L,. Then A has at least one eigenvalue. Proof. Let A, = sup Q(x), po = inf Q(x), where Q is the quadratic lixIl=l

llxI=l

form of A . By replacing A by -A if necessary, we may assume A,, 2 0. [Note that 11 Q 11 = max (Ao, Ip,I ).I We claim that if A, 2 Ip,l, then A,, is an eigenvalue, and if Ip,l 2 A, then pois an eigenvalue. First suppose that A,, 2 IpoI. If A,, = 0, then A =-0;hence every vectorin L, is an eigenvector for the eigenvalue A = 0. Thus assume A, # 0. By definition of A,, there is a sequence of points x, E L , such that llxnll = 1 for all n and Q(x,) -+ A,. Since IlAll = IlQll by Theorem 1.6, and ((Q((= A, by the assumption A, 2 (po(,we have

By Lemma 1.4 and Theorem 1.5, 1, is an eigenvalue. If (pol 2 A, a similar argument shows that p, is an eigenvalue.

Theorem 1.8. Let A # 0 be a fixed eigenvalue of the compact linear operator A on L,. Let E, be the subspace of eigenvectors corresponding to A, that is, E, = {x:Ax = Ax). Then E, is finite dimensional. Proof. We shall show that every bounded sequence in EL has a convergent subsequence. If {x,) is such a sequence, then Ax, = Ax,, or x, = AklAx,. Since A is compact, {Ax,) has a convergent subsequence, and therefore so does (x,). Now if E, were infinite dimensional, we could construct (for example, by the Gram-Schmidt process) an infinite sequence el, e,, . . . , of mutually orthonormal vectors [that is, (en, em) = 0 for n # m and Ile,ll = 1 for all n] in E,. The sequence {en) has no convergent subsequence since

thus the distance between en and em is J2 for n # rn, so that there can be no convergent subsequence. It follows that E, must be finite dimensional. We remark that E, is a closed set since if Ax, = Ax, and x, by continuity of A we have Ax = Ax.

-t

x then

Lemma 1.9. Let A be a symmetric linear operator on L,. Then eigenvectors corresponding to distinct eigenvalues of A are orthogonal, that is, if Ax = Ax and Ay = py where A # p , then (x, y) = 0. Proof. Since A is symmetric, (Ax, y) = (x, Ay). Hence (Ax, y) = (x, py), or (A - p)(x, y) = 0. The result follows.

APPENDIX

267

Theorem 1.10. Let A be a compact symmetric linear operator on L,. Then the set of eigenvalues of A is either finite or countably infinite. In addition, the only possible limit point of eigenvalues is 0. Proof. Suppose that c is a nonzero limit point of eigenvalues. Let {A,) be a sequence of eigenvalues converging to c, and let xn be an eigenvector corresponding to A,. We may assume that 1, # 1, for n Z: m , and that llxnll = 1. Now

Therefore (as in the argument of Theorem 1.8) {Ax,) can have no convergent subsequence, contradicting the compactness of A. Thus 0 is the only possible limit point. Now if S is the set of eigenvalues, then

If S is uncountable, then one of the intervals [n-l, n] or [-n, -n-l] must have uncountably many points of S. But a bounded infinite set of real numbers has a limit point, and hence there is a nonzero limit point of eigenvalues, a contradiction. Lemmu 1.21. If 1 is an eigenvalue of the linear operator A , then lil IIAll. (Note that if A is compact and symmetric, then by the argument of Theorem 1.7, either IIAJI or - JJ AIJ is an eigenvalue.) Proof. If Ax = Ax where x # 0, then 1 1 1 = ll Axlllllxll I IIAll.

<

Lemma 1.12. The inner product is a continuous linear functional, that is, if {x,} is a sequence of points in L, and x, 4 x, then for any y E L, we have (x,, Y) (x, Y). Proof. By the Schwarz inequality, l(x, y) - (x,, y)] = 1(x - xn, Y)]5 Ilx - xnll IIYII 0. -+

+

The converse of Lemma 1.12 is also true. I f f is a continuous linear functional on L,, that is, a continuous linear mapping from L, to the reals, then f is given by an inner product, that is, there is an element y E L, such that f(x) = (x, y) for all x E Lz. [Liusternik and Sobolev (1961), p. 110.1 The space spanned by the subset S of L, is defined as the smallest closed subspace of L, containing S. Theorem 1.13. If A is a compact symmetric linear operator on L,, then the eigenvectors of A span the entire space. In other words, if E is the

268

INFORMATION THEORY

smallest closed subspace which contains all the eigenvectors of A (corresponding to all the eigenvalues) then E = L,. Proof. Let E l be the orthogonal complement of E, that is,

EL = {x E L 2 : (x, y) = 0 for a l l y E E). If E # L,, then E'- contains a nonzero element. [Liusternik and Sobolev (1961), pp. 75-76.] We note that EL is a closed subspace of L,. For if x, E ~ l - i n x, d -+x then for any y E w e have, by Lemma 1.12, (x, y) = lim (x,, y) = 0. We now show that EL is invariant under A , that is, n-

w

x E E l implies Ax E EL. First of all if x E E L and y is an eigenvector corresponding to the eigenvalue A, then (Ax, y) = (x, Ay) = (x, Ay) = 1(x, y) = 0. Thus Ax is orthogonal to all eigenvectors. In general, if y E E there is a sequence (y,) of finite linear combinations of eigenvectors converging to y. By Lemma 1.12, (Ax, y) = lim (Ax, y,) = 0; thusAx E E l . n-

m

Since E L is a closed subspace of L, (hence is itself a Hilbert space), and E L is invariant under A , we may consider A as a linear operator on El-. The restriction of A to E l is still compact and symmetric, and hence by Theorem 1.7 has at least one eigenvaluc. Therefore there is at least one nonzero eigenvector x E E L . B U ~we also have x E E (by definition of E). Since an element of EL is orthogonal to all elements of E, (x, x) = 0, which implies x = 0, a contradiction. It follows that E = L,. Theorem 1.14. (Expansion Theorem). Let A be a compact and symmetric linear operator on L,. Let {en,n = 1,2, . . .) be an orthonormal basis for the space spanned by the eigenvectors corresponding to the nonzero eigenvalues of A . [For example, we might construct such a basis by forming, for each nonzero eigenvalue A, an orthonormal basis for the finite-dimensional space E, (Theorem 1 . Q and then taking the union of all such bases. The resulting set of eigenvectors is countable by Theorem 1.10, and orthonormal by Lemma 1.9.1 Then if x is any element of L,, and h is the projection of x on the space E, = {y :Ay = 0) [see Liusternik and Sobolev (1961), pp. 75-76], then x can be represented as 4)

2

=h

+ 2 (5, en)%, n=l

where the series converges in L,. If en is an eigenvector of the eigenvalue A,, then the continuity of A implies that

269

APPENDIX

z:=,

Proof. We first show that ( 2 , en)en converges in L,. Using the orthonormality of the en we have, for any positive integer r,

Now if m

Thus

< n,

z,"=,( x , en)enconverges. Consequently we may write

,=" , z ( x , en)en, then y is orthogonal to all en. For If y = x - h (h, en) = 0 by Lemma 1.9, and ( x , ek)e,, en) = ( x , en) by Lemma 1.12; thus ( y , en) = ( x , en) - ( x , en) = 0. Furthermore, y is orthogonal to all elements of E,. For if h' E Eo then ( y , h') = ( x , h') - (h, h') = ( x - h, h'). Since h is the projection of x on E,,, x - h is orthogonal to E,, hence ( x - h, h') = 0. Therefore y is orthogonal to all eigenvectors, so by Theorem 1.13, y is orthogonal to all elements of L,. In particular, (y, y) = 0 and thus y = 0. The result follows.

(z,"?,

2. Integral operators

We shall consider integral operators on L,[a, b ] ,a, b j n i t e , defined by

where R is a real-valued function which is continuous in both variables. Note that the formula (2.1) does in fact define a linear operator on L,. For let x E L,, and M = max JR(t,T ) / . Then by the ~ c h w i r inequality, z t,rr[a,b]

x 2 ( ~d)~ I ~

~- a( ) l j b~ 1

Hence h

I I Y I= I ~J y z ( t ) dt a

5 ~ ' ( -b al2 I I X I I<~ a.

1~.

270

INFORMATION THEORY

In fact if x E Lz then y is continuous for all t E [a, b]. For we have IR(t, T)x(T))5 M )x(T)),which is integrable over [a, b] by the Schwarz inequality : b

b

Thus we may apply the Lebesgue dominated convergence theorem [for example, Lokve (1 955), pp. 125-1261 to obtain

In particular if e is an eigenfunction of (2.1) corresponding to the eigenfh

value

A, then Ae(t) =

~ ( t7 ,) e ( ~d) ~ Hence: .

Ja

The eigenfunctions of (2.1) corresponding to nonzero eigenvalues are continuous.

(2.2)

We remark that any function continuous on [a, b] has a maximum on [a, b] and hence belongs to LJa, b]. Theorem 2.1. The integral operator (2.1) is compact. Proof. We use the Arzela-Ascoli theorem [for example, Liusternik and Sobolev (1961), p. 391, which states that if {y,, k = 1, 2, . . .) is a sequence of continuous functions on [a, b], and the y, are uniform[y bounded(that is, Iyk(s)l 5 L < m for all k, s) and equicontinuous (given E > 0, there is a 6 > 0, where 6 depends only on E, not on k, such that if s,, s, r [a, b] and Is, - s,l < 6, then (y,(s,) - yk(sz)l < E for all k), then {y,) has a uniformly convergent subsequence. Now let {xn) be a bounded sequence in L,; say llxnll < N for all n. We must show that {Ax,} has an L,-convergent subsequence, where A is the integral operator (2.1). Let yn(t> = (Axn)(t) =

<

Then [see the discussion after (2. I)] 1 y,(t)l MN(b - a)% < co. Hence the functions y , are uniformly bounded. T o prove equicontinuity, we

APPENDIX

write

Now R is uniformly continuous on [a, b ] x [a, b ] ; therefore given c > 0 there is a 6 > 0 such that if It, - t,l < 6 and IT, - r21 < 6, then IR(tl, r l ) - R(t2,r2)1 < ~ N - l ( b- a)-%. In particular, if It, - t,l < 6, then 1 R(t,, 7 ) - R(tz, T ) / < ~ N - l ( b- a)-% for all T . Thus by (2.3), 1 1 c2~ It1 - t21 < 6 implies lyk(t1)- y,(t2)12 sWP2(b- a)-l(b - a ) 1 1 ~ ~ < for all k. Thus the functions yn are equicontinuous. By the Arzela-Ascoli theorem, there is a subsequence y,,,, converging uniformly to a continuous function y. But

<

since the uniform convergence allows the limit to be taken inside the integral sign; thus y,,,, + y in L, and the result follows. Theorem 2.2. Let R be a real-valued function on [a, b] x [a, b] which is symmetric [R(t,7 ) = R(T, 1) for all t , T ] as well as continuous, so that the associated operator A defined by (2.1) is symmetric as well as compact. Then all eigenvalues of A are nonnegative if and only if

J'S

R ( t , T ) x ( ~ ) x ( Td)t d~ 2 0 for all continuous functions

a a

x on [a, b].

(2.4)

Proof, Suppose that the double integral is nonnegative for all continuous s. Let x be an eigenfunction corresponding to the nonzero eigenvalue 1 ; x is continuous by (2.2). The double integral (2.4) becomes 1 p ( t ) dr = 1

11~11'. We may take x

f 0; it follows that 1

> 0.

Conversely, suppose that all eigenvalues of A are nonnegative. If x is continuous on [a, b ] , then z E L2[a,b]; and hence by Theorem 1.14, x =h c,en where the en are an orthonormal basis for the space

+ x;=l

272

INFORMATION THEORY

spanned by the eigenvectors corresponding to the nonzero eigenvalues, cn = (x, en), and Ah = 0. Now the double integral (2.4) may be written as (x, Ax) ; by Lemma 1.12 we have m

(2, Ax) = (2, Ah)

+ 2 cn(x, Ae,) n=l

+

m

Ln

= (x, ~ h )

cnAn(x,en) = n-1

2 A,c,2

2 0.

n=l

The theorem is proved. Note that in Theorem 2.2, the phrase "for all continuous x on [a, b]" in (2.4) may be replaced by "for all x E L,[a, b]"; essentially the same proof may be used.

Theorem 2.3. Let A be the integral operator (2.1) corresponding to a function R which is nonnegative dejnite as well as continuous and symmetric. (R is said to be nonnegative definite if

for all possible choices of t,, . . . , tn E [a, b ] , and all possible real-valued functions x on [a. b].) Then all eigenvalues of A are nonnegative. Proof. If x is continuous, then

is an ordinary Riemann integral; the approximating Riemann sums are of the form 2:j=lx(ti)R(t,, t,)x(t,). Since R is nonnegative definite, the approximating sums are all nonnegative, and hence the integral is nonnegative. The result follows from Theorem 2.2. Theorem 2.4. (Mercer's Theorem). Let A be the integral operator (2.1) corresponding to a function R that is continuous, symmetric, and nonnegative definite. Let {en,n = 1, 2, . . .} be an orthonormal basis for the space spanned by the eigenvectors corresponding to the nonzero (hence positive) eigenvalues of A. If the basis is taken so that en is an eigenvector corresponding to the eigenvalue An, then

273

APPENDIX

where the series converges absolutely, converges to R(t, T ) uniformly in both variables, and also converges to R(t, T )in L,([a, b] x [a,b]),that is,

Proof. We break the proof into several parts. ), the series converges in L, in each a. R(t, T ) = z z l A n e n ( t ) e n ( ~where variable separately. Fix t and apply Theorem 1.14 to R(t, T ) . We obtain

R(t, 7 ) = h(4 7 )

+ t:cn(t)en(.r), n-1

where

(2.6)

and the series converges in L,[a, b]. It follows from (2.6) and Lemma 1.12 that h

h

By Lemma 1.9,

1

h(t,T)en(T)

= 0.

n = 1,2, . . . .

a

S."

Hence 0 = h2(t,T ) d ~ .Therefore for each t , h(t, .) = 0 in L,[a, b] and the result (a) is proved. b. Let Rn(t, 7) = R(t, T ) - 2;=lAlei(t)et(~); then Rn(t, t ) 2 0 for all t. BY(49 m

2 4ez(t)er(~), %=n+l

Rn(t, 7 ) =

with L, convergence in each variable separately. I f x E L,[a, b] then, using Lemma 1.12, we obtain

Now if Rn(to,to) < 0 then by continuity, Rn(t, T ) 5 - E < 0 for some positive E and for t, T in some neighborhood U of (to,to) described by

274 to - a to - a

INFORMATION THEORY

< t < to + a, to - a < < to + a. Now < t < to + a ; x(t) = 0 elsewhere. Then x T

E

define x(t) = 1 for L,[a, b] and

contradicting (2.7). c. The series jl,e,(t)e,(~) converges absolutely; the convergence is uniform in T for each fixed t , and uniform in t for each fixed T . BY (b),

z:=,

n

Rn(t, t ) = R(t, t ) -

Z: Iie:(t)

2 0,

i=l

hence

Now by the Schwarz inequality for sums,

uniformly in T for each fixed t. The assertion (c) follows. d. The series z~=,I,e,(t)e,(~) converges pointwise to R(t, T). Fix t. By (a), ~ ~ = , ; l , e , ( t ) e , ( ~=) R(t, T ) with L, convergence. By (c), z:=ljlnen(t)en(~)= some function S(t, T ) with uniform, hence L,, convergence. It follows that for each t , R(t, T ) = S(t, T ) for almost every 7. But R(t, .) is continuous and S(t, .), the uniform limit of continuous functions, is also continuous. Thus R(t, T ) = S(t, T ) for every T , that is, R(t, 7 ) E S(t, 7). e. (Dini's Theorem). Given a sequence of real-valued functions {g,, n = 1, 2, . . .} continuous on [a,b] and a real-valued function g continuous on [a, b] such that g,(x) 5 g,+,(x) for all x and all n, and lim g,(x) = g(x) for all x E [a, b]. Then g, g uniformly on [a, b]. The

-

n-m

same conclusion holds if the sequence is monotone decreasing instead of increasing. Given E > 0, let U , = { x E [a, b]: Ign(x) - g(x)l < E } . Since g,(x) g(x) for all x, U Un = [a,b]. Thus we have a family of open sets that cover [a,b]; by the Heine-Bore1 theorem, a finite subfamily covers [a,b]; that is, ULl Un = [a,b] for some N. By the monotonicity of the sequence {g,}, Un c Un+l for all n, and hence U$='=, U , = U,.

,"=,

-

275

APPENDIX

Consequently U, = [a, b], so that lgN(x) - g(x)I < E for all x. By monotonicity, Ign(x) - g(x)l < E for all x if n 2 N; the result follows. ilnen2(t)with pointProof of Mercer's Theorem. By (d), R(t, t) = wise convergence. Let gn(t) = il,ei2(t). Then gn is continuous, g,(t) 5 g,+,(t) for all t, and gn(t) 4 R(t, t) for all t. By (e), gn(t) -+ R(t, t) uniformly on [a, b]. Now as in (c),

z:=,

z,"=,

uniformly in t and T. The proof is complete.

3. The Karhunen-Loeve theorem Let X(t), t E [a, b] (a, b finite), be a continuous-parameter second-order random process with zero mean and covariance function R(t, T). In other words, for each t E [a, b], X(t) is a (real-valued) random variable; E[X(t)] = 0 for all t r [a, b], and E[X(~)X(T)]= R(t, T) for t, T E [a, b]. Note that R is nonnegative definite, since

If g is a real-valued function on [a, b], we define Let A:a = to < tl < - - max It, - t,-,J. Define l
< tn = b

1."

g(t)X(t) dt as follows.

be a partition of [a, b], with lAl =

n

=

2g(tk)X(tk)(tk - tk-1).

k=l

-

If the family of random variables {Z(A)) converges in mean square to a random variable I as IAI -+ 0 [that is, E [IZ(A) - 1l2] 0 as IAl 4 0] then we say that g(t)X(t) is integrable over [a, b]. I is called the integral of Ph

g(t) X(t) dt. J : If L 2 (with an appropriate sigma-algebra of subsets and an appropriate

g(t) X(t) over [a, b] and we write I =

probability measure) is the probability space on which the X(t) are defined, then mean square convergence is convergence in the Hilbert space L 2 ( Q of random variables on l 2 with finite second moments. Theorem 3.1. If g is continuous on [a, b] and the covariance function R is continuous on [a, b] x [a, b], then g(t)X(t) is integrable over [a, b].

276

INFORMATION THEORY

Proof. Let

be two partitions of [a, b ] . Then

Thus

The right side of (3.1) is an approximating sum to a Riemann integral. Thus lim

EII(A)I(Af)]=

g ( t ) g ( ~ ) R (7t ,) dt d ~ .

(3.2)

Thus { I ( A ) )is a generalized Cauchy sequence in L2(R), and hence [(A) converges in mean square to a random variable I a s lAl -* 0. The theorem is proved.

Lemma 3.2. If g and h are continuous on [a, b ] , and R is continuous on [a, 61 x [a, b], then

) = 0. Furthermore, E [ l g ( t )X(r) dt] = E [l\(7) ~ ( 7d7] Proof Let

As in Theorem 3.1, let A and A' be two partitions of [a, b].

I = Sg(t)X(t)dt,

J = f h ( r ) X ( r ) dr.

-- -

277

APPENDIX

By Theorem 3.1, I(A) I, J(A1) J in mean square as IAl, IA'I follows that EII(A)J(A1)] E[IJ]. For we have IE[I(A)J(A') - IJI I

-+ 0.

It

I E [ I~(A)(J(A1>- J) I I + E[l(I(A) - OJI I

< (E[12(A)lEKJ(h') - J)21)x

+ (E[J2]E[(I(A)

(by the Schwarz inequality)

JE[J~I.o=o.

-+JE[12j.0+

But as in (3.2),

-

Similarly, I(A) I in mean square implies E[I(A)] -+ E(I). . Since E[I(A)] = 0 for all A, the lemma follows. Lemma 3.3. If h is continuous on [a, b] and R is continuous on [a, b] x [a, b ] , then

Ii

1=i

E X ( t ) h(~)X(r)d~

Proof. Let J(A') =

h(~)R(t,T) d ~ , t E [a, b].

z;",, ~ ( T ~ ) X ( T ~-) (~T~~ - and 3 , let

-

-

a

Then J(A1)-+J in mean square as (A'( 0. Just as in the argument of Lemma 3.2, EIX(t)J(A1)] E[X(t)J]. But [compare (3.2)]

and the lemma follows.

a

Theorem 3.4. (Karhunen-Lotve Expansion). Let X(t), t e [a, b], (a, b finite), be a continuous-parameter second-order random process with zero mean and continuous covariance function R(t, T). Then we may write

278

INFORMATION THEORY

where the e, are eigenfunctions of the integral operator (2.1) corresponding to R, and form an orthonormal basis for the space spanned by the eigenfunctions corresponding to the nonzero eigenvalues. The Z , are given by rh

X(t)ek(t)dt, and are orthogonal random variables ( E [ Z J , ] = 0 for

k # j) with zero mean and variance ilk, where 1, is the eigenvalue corresponding to e,. The series z,"=,Zkek(t) converges in mean square to X(t), uniformly in t , that is,

uniformly for t E [a, b ] . Proof. By Theorem 3.1,

Jbb

X(t)ek(t)dt exists and defines a random

variable 2 , . By Lemma 3.2, E[Z,] = 0 and

S

= 1, e,(t)e,(t) dt =

n

=

2 &e,"(O

k-1

0 for k # j 1,

for k = j

n

'

+

- 2 2 EIX(t)Zklek(t) R(t, t ) k=l

By Lemma 3.3,

Thus

a

uniformly in t, by Theorem 2.4. The result follows. The Karhunen-Lobve expansion assumes a special form when the process X(t) is Gaussian.

279

APPENDIX

Theorem 3.5. If X ( t ) is a Gaussian process, then the Z , of the Karhunen-Lokve expansion of X(t) are independent Gaussian random variables. Proof. The approximating sum to Z , = n

Since X(t) is a Gaussian process, the I,(A), k = 1 , 2, . . . ,form a Gaussian sequence, that is, Ikl(A),. . . , I,JA) are jointly Gaussian for all finite sets of indices {k,, . . . ,k,}. Now the joint characteristic function of Zkl(A),. . . , IkSA) is

[

M(t,, . . . , t,) = E exp

i

= exp

where i =

47

11

2 t,Ikj(A)

(- 21

tiojrn(~)trn)

-

.+,,=I

and o,,(A) = E[Zkj(A)Zkm(A)].

Since Zk(A) converges in mean square to Zk as IAl -+ 0 ( k = 1,2, . . .), it follows as in the argument of Lemma 3.2 that EIIkj(A)Ik,(A)]4 E[ZkjZkm].Thus the joint characteristic function of Zkl(A),. . . ,I,JA) approaches exp (-4 tioimtm) where o,, = E[ZkjZkm]. Thus Z,,, . . . ,Zk, are jointly Gaussian for each finite set of indices {k,, . . . , k , } ; that is, the Z,, k = 1,2, . . . , form a Gaussian sequence. Since the Zk are orthogonal random variables, they are independent.

z~,,=,

Thus in the Gaussian case, the Karhunen-Lokve expansion is a series of independent random variables. It follows (Lobe, 1955, p. 251) that for eachfixed t , the series (3.3) converges with probability 1, that is, for each t , the series converges pointwise to X(t) for almost all points w in the probability space 61. As an example of the Karhunen-Lokve expansion, we take R(t, T ) = min ( t , T ) , with the expansion interval = [O, 11. [If in addition we take X(t) to be Gaussian, we obtain the Brownian motion process.] T o find the eigenvalues of the integral operator associated with the covariance function, we must solve the integral equation

0

or equivalently JTe(T) d r 0

+t

S

e ( 3 d r = Ie(t),

t

05 t 5 I.

(3.4)

280

INFORMATION THEORY

If we differentiate (3.4) with respect to t , we obtain

Differentiating again with respect to t , we have

Thus any solution of the integral equation, that is, any eigenfunction, must be of the form t t e ( t ) = A sin - B cos -T . (3.7)

42

+

d1

[Note that if 1= 0 then e ( t ) 0 by ( 3 . 6 ) ; thus 0 is not an eigenvalue.] Now if we let t = 0 in (3.4) we obtain e(0) = 0, and therefore B = 0 in (3.7). If we let t = 1 in (3.5) we obtain de(t)/dt = 0 when t = 1, and therefore 1 cos - = 0,

JJ

Thus the eigenvalues are

and the orthonormalized eigenfunctions are en(?>= J2 sin ( n

- i)?~t.

(3.9)

[It may be verified by direct substitution that the functions (3.9) satisfy the integral equation (3.4).] Finally, if we define Zk* = z,/ A,, where the Zk are as given in Theorem 3.4, then the Karhunen-Lotve expansion becomes

2 Z , ,sin( n( n--S)Ti)n-t m

X ( t ) = JZ

n=1

where the Z,* are a sequence of orthogonal random variables with mean 0 and variance 1. If X(t) is Gaussian, we may show that for almost all w , the series (3.10) converges uniformly for 0 5 t 5 1, and therefore may be used to define a process whose sample functions are continuous. For we have

APPENDIX

It follows that the serles

2 n=l

lZn*l ( n - +)n

converges In mean square, and consequently, belng a serles of Independent random varrables, converges wlth probabllrty 1. The result now follows from the Wererstrass crlterlon for unlform convergence. For a further drscusslon of the theory of second-order random processes, see Loeve (1955), Chapter 10. 4.

Further results concerning integral operators determined by a covariancefunction

In thrs sectlon we shall need some standard facts about Fourier transforms on L,(-oo, co). [See, for example, Goldberg (1961), Tltchmarsh ( 1 937).] Conslder the H~lbertspace of allcomplex-valued Bore1measurable functrons f defined on the entire real h e , such that

I:a 1

f ( t )1 dt

< co ;

we denote t h ~ sspace by L,*(- co, a). [The norm off 1s

and the inner product of two functions f and g is

where the bar denotes complex conjugate.] Iff E L,*(- oo, co), then the functlon FN defined by Y

FN(w) =I f (t)e-zot dr -N

converges In L,*(- co, co) as N -+ co to a function F, that is, 11 FN - Fll 0. Conversely, the functlon fN defined by

-+

N

f,,,(t) = ( ~ n ) - l b ( w ) e ' " 'dw -N

converges In L,*(- co, co) to f. F 1s called the Fourrer transform off, and f the rnuerse Fourrer transform of F. The paw (f,F ) 1s called a Fourrer

282

INFORMATION THEORY

transform pair. A function in L,*(- co, co) is uniquely determined by its Fourier transform. Iff, g E Lz*(- co, co) with Fourier transforms F, G then the Parseval relation states that

consequently, 11 Fll = (2n)X 11 f 11. Iff, g E L,*(- co, a), the convolution off and g is the function h defined by

I

h(t) = f(s)g(t - s) ds. ,

-m '

[By the Schwarz inequality, h exists but is not necessarily in L,*(- C Oa, ) . ] If F and G are the Fourier transforms off and g respectively, and H is the product FG, then the convolution theorem is a name for a class of results which state that under certain conditions, (h, H) is a Fourier transform pair. In particular, iff

L1*(- CO,a ) , that is,

JIrnIf

(t)l dt

< CO,then

h c L,*(- co, co) and the Fourier transform ofh is H. Also, if H E L,*(- a, co), then so is h, and (h, H ) is a Fourier transform pair. Let n(t), - c~ < t < co, be a second-order stationary random process with continuous covariance function R(T) = E[n(t)n(t T)]. Assume that the process has a spectral density N(w), that is, assume that R(T) can be expressed as

+

m

(2~)-I J N ( W ) e ~ ~ ~ d W , -OD

where N is a nonnegative function integrable over ( - a , a ) . Let T be a positive real number, and let AT be the integral operator on L,[-T/2, TI21 corresponding to the covariance function; that is, the operator defined by

T

T s t < - . y(t) = T R ( t - T)Z(T)d ~ ; - 2

(4.1)

-TI2

We assume that 0 is not an eigenvalue of AT. (Later we shall investigate conditions under which this result holds.) It follows (Theorems 1.14, 2.3) that the eigenfunctions corresponding to the positive eigenvalues of

283

APPENDIX

AT span the Hilbert space L2[-T/2, T/2]. Let {en(?;T), -TI2 I :t I : T/2) be an orthonormal basis of L,[-T/2, T/2], where en(?;T) is taken as an eigenfunction corresponding to the positive eigenvalue An(T). Let G(w) = [N(w)lX and let g be the inverse Fourier transform of G. (Since G is an even function, g is real valued.) Define

We shall establish several properties of the functions q,. Lemma 4.1. For each T, the functions qn(t; T) are orthonormal over (-m, 00). Proof. For simplicity we write qn(t) for q,(t; T), en(?) for en(?; T), and An for An(T). We have

i

yn(t)qm(t)dt = A, W A,%

Since the Fourier transform of the function h(t) = g(t the Parseval relation implies that

- s) is e-'""G(w),

An application of the Schwarz inequality shows that

It follows that Fubini's theorem may be applied to the integral (4.2), and we obtain

Since en is an eigenfunction corresponding to A,, the integral has the value 0 if n # m, and 1 if n = m. The lemma follows.

284

INFORMATION THEORY

If L is a subspace of L,(-co, co) and f r L,(-co, co) we will use the notation f 1L to indicate that f is orthogonal to L, that is,

I

f(t)h(t) dt = 0 for all h r L.

-m

Lemma 4.2. For each T , let L , be the subspace of L,(- co, co) spanned by the functions {qn(t;T ) } . Iff E L,(- co, co) and T' < T , then a. f 1L , implies f 1L,.. b. LT* c LT. ProoJ To say that f 1L , is to say that

I

n = 1 , 2 ,...,

f(t)yn(t;T)dt=O,

-m

or

TI2

I

[nn(~)rN J f ( t ) [ o[S g(t - s)en(s;T ) ds -w

-TI2

dt = 0 ,

n = 1,2, . . . .

(4.3) We may reverse the order of integration (invoking Fubini's theorem as in Lemma 4.1) to obtain

i

T/Z

[ l n ( ~ ) l N en(.; T ) [ I g ( t - s ) j ( t )it]ds

= 0,

n = 1, 2,

....

-T/Z

Thus the function h(s) =

1-:

(4.4) g(t - s)f ( t ) dt,

- T/2 j s 5 T/2,is orthog-

onal to all the en(s; T). Since the en span L,[- T/2, T / 2 ] ,h is orthogonal to itself so that h = 0 almost everywhere, that is,

( J-

m

g(r - s)f(r) dt = 0 m

for almost every s E [- T/2, T / 2 ] ,hence for almost every s r [- T f / 2 ,T1/2]. It follows that (4.4), hence (4.3), holds with T replaced by T', and consequently f 1L,., proving part (a) of the lemma. We now prove (b). Given any f E Lz(- co, a), by the projection theorem we may write f =f,+f,, where f , r LT and f , 1LT. By (a), f,1L,. Using the projection theorem again, we have f , = f3 f,, wheref 3 r LT,, and f , 1L,. If P, is the linear operator on L,(- co, co) which assigns to a function its projection on L,, we have PTf =f,and P T ~ P= T ~P ~ f =l f a . But f =f 3 ( f , +f4), where f 3 r L,. and f , f, 1L,., and thus f 3 = P,$ It follows that

+

+

+

285

APPENDIX

+

Now assume f r L,,. By the Pythagorean relation, ll f 112 = ll fill2 If f, is not zero, then JIP,f JJ = Ilf,JJ< Jl f 1 1 ; since projection cannot increase the norm, we have IIPT,pTf ll 5 llPTf ll < I l f l l . But by (4.9, IJPT.PTf11 = 11 f 11, a contradiction. Thus f, = 0 and therefore f = fl E LT, completing the proof. Note that the above argument shows that the statements '71LBimplies f 1La" and "La c L T are equivalent for any closed subspaces L,, LB of L2(- co, a ) . Lemma 4.3. Let H be the subspace of L,(- co, co) consisting of those functions whose Fourier transforms vanish whenever G(w) (equivalently, N(w)) vanishes. With LT as in Lemma 4.2, let Lm be the subspace spanned by U LT. Then Lm = H. Proof. Fix T, and define

Then en* r L2(- CQ,co); let En* be the Fourier transform of en*. The functions y n may then be written as

Since the Fourier transform of a convolution is the product of the Fourier transforms, the Fourier transform of yn is [I,(T)]-MG(~)E,*(~). It follows that yn E H for all n = 1,2, . . . , and all T, hence L, c H f o r all T, or L , c H. To show that H c L,, we shall show that i f f c L,(-CQ, CQ),then f L , implies f 1H; the result then follows as in Lemma 4.2b. If f 1L, then f 1L , for every T; thus by the argument of Lemma 4.2ay

- t)f (t) dt = 0

for almost every s E [- T/2, T/2].

[Note that

since G(w) is real for all w , g is an even function, so that g(s g(t - s).] Since this holds for arbitrary T, j?g(s -OD

- t)f(r)

d t = 0 for almost all real numbers s.

- t) =

286

INFORMATION THEORY

Again using the convolution theorem for Fourier transforms, we have G(w)F(w) = 0 for

almost all o,

(4.6)

where F is the Fourier transform of$ Now let f, E H , and let F, be the Fourier transform off,. By definition of H, G(w) = 0 implies Fl(w) = 0. But by (4.6), G(w) # 0 implies F(w) = 0. Thus F(w)Fl(w) = 0 for almost all w , and consequently F and Fl are orthogonal. By the Parseval relation, f and f, are orthogonal. Since f, is an arbitrary element of H , f 1H and the lemma is proved. Lemma 4.4. Given a function s E H with Fourier transform S , define F(w) = S(o)/G(w)if G(w) # 0 ; F(w) = 0 if G ( o ) = 0 [= S ( o ) ] . Assume that F E Lz*(- m, a),that is,

Let f be the inverse Fourier transform of F (note f

/

H). Define

m

TI2

sn(T) =

Q

J

s(t)en(t; T ) dt, f n ( T ) = f ( t ) p n ( t ; T ) dt, n = I, Z, . . . .

-TI2

-m

Then [I,(T)]-%,,(T) =fn(T), n = l , 2 , . . . . Proof. By the convolution theorem for Fourier transforms,

/

m

~ ( t= )

Thus

g(t

- s)f ( s ) ds.

-m

Reversing the order of integration and using the fact that g(t - s) = g(s - t), we obtain

and the result follows.

-m

287

APPENDIX

Theorem 4.5. Let s be a function c H with Fourier transform S. If sn(T) is as defined in Lemma 4.4, then

lim

2

Is (T)I2

T + m n=l

An(T)

= (2n)-'

1 -m

-d o (assuming the integral is finite), N(w)

where the convergence as T + w is monotone from below. [The integrand (~(o)l"N(w)is defined to be zero if N(w) = S(w) = 0.1 Proof. Let F,S, and fn(T) be as in Lemma 4.4. Define

[Note that since the p n are orthonormal and

(Bessel's inequality; see the argument of Theorem 1.14), the series fn(T)pn(t; T) converges in L2(-w, a ) . ] We claim that h is the projection off on LT. T o see this, use the projection theorem to write f =f, f, where fl E LT and f, _L L,. Since f, is orthogonal to all the functions pn(t; T), it follows that

x;Zhl

+

By definition of L,, the p,(r; T) are an orthonormal basis for L,;

where fIn(T) = J

hence

fl(t)pn(t; T )dt =fn(T) and the series converges in

-a

L,(- w , co).(Again this can be justified by an argument similar to that of Theorem 1.14.) Thus fl, the prqjection off on L,, is in fact h. Now write PTf for the projection off on L,. If T' < T, then Ly c L, by Lemma 4.2b; the projection theorem then implies that (I f - PTf 11 j 1 f - P ,f 11. Therefore lim 11 f - PTf (1 exists. If this limit is E > 0, then T-m

given any T a n d any fl L, we would have JJ f -flJI 2 )If - PTf 1) 2 E. This contradicts Lemma 4.3, which states that any function in H can be approximated arbitrarily closely (in the L, sense) by functions in (J LT. T

288

INFORMATION THEORY

Therefore lim Jl f T-m But

- PTf 11

= 0, and consequently lim l l p ~ 1f' =

1 f /I2.

T- w m

2 Ifn(T)l2= 2 An(T)

IIp~jll"

n=l

by Lemma 4.4.

n = ~

Furthermore if T f < T, then by (4.5), IIPT,f 11 = IIPy p ~ 11 f 5 IIPT~1 1 . The theorem follows. We conclude with some conditions under which the zero eigenvalue is excluded.

Theorem 4.6. Let AT be the integral operator defined by (4.1). If (a) the set on which N(w) = 0 has measure zero, or if (b) the covariance function R belongs to L,(- co,co), and the set on which N(w) > 0 is of positive measure and is contained in a finite interval, then 0 is not an eigenvalue of AT. In particular, (a) implies that if N(w) is a rational function of o , then 0 is not an eigenvalue. Proof. If G(o) = [N(o)]M and g is the inverse Fourier transform of G , then by the convolution theorem, R(t) =

1

g(s)g(t - s) ds.

-m

Hence

If 0 is an eigenvalue then there is a function f such that

E

L2[-T/2, T/2],f # 0,

consequently TI2

TI2

II

-TI2 -TI2

But this integral is

R(t

- 7)f (t)f (7) dt d7 = 0.

APPENDIX

or, after the order of integration is interchanged,

TI2

- t)f(t)

It follows that

dt = 0 for almost every s. If we define

f * ( t ) =f(t), - T/2 j t F; T/2, f *(t) = 0 elsewhere, and if F * is the Fourier transform of f *, then the convolution theorem implies that [N(w)]xF*(w) = 0 for almost every w. If N is positive almost everywhere, F* must be zero almost everywhere, so that F*, hence f, is 0 almost everywhere, a contradiction. This proves (a). To prove (b), again suppose that R(t - s)f (s) ds = 0, t E [ - T/2,T/2], for some function f, f # 0. Define TI2

fl(t) =

J

~

(

-t s)f(s) ds,

-m

< t < m;

-TI2

then fl r L,(- m, m) and fl(t) = 0, - TI2 5 t 5 T/2. If Fl and F * are the Fourier transforms of fl and f * respectively, where f * is as defined in part (a), then the convolution theorem yields F,(w) = N(w)F*(w), and hence F, vanishes outside a finite interval [-B, B]. But in this case fl is given explicitly by B

fl(t) = (27r)-'

Fl(W)eio' dm.

(4.7)

-I3

We may use the formula (4.7) to define a function fl which assigns to rR

each complex number t the complex number ( 2 ~ ) - IJ - F1(w)eid dw. -B

Since the limits of integration are finite, it may be verified by direct differentiation thatfl is analytic everywhere. sincef1 agrees with f, on the real line and hence vanishes on the real interval [-T/2, T/2], is zero everywhere and hence so is fl. Consequently Fl(w) = 0 for almost a11 w, and hence N(w) > 0 implies F*(w) = 0. Rut since f * vanishes outside a finite interval, the same argument as above shows that F* is the restriction to the real line of a function which is analytic everywhere. Finally, let D be the set on which N(w) > 0; by hypothesis, D has positive measure and is therefore uncountable, and consequently has a

yl

290

INFORMATION THEORY

finite limit point. Since a function that is analytic everywhere and vanishes on such a set necessarily vanishes everywhere, it follows that F* = 0, so that f = 0, a contradiction. A result similar to Theorem 4.6 was proved by Root and Pitcher (1955); the other results of this section are due to Kelly, Reed, and Root (1960).

Values of -log, p and -p 10gz p

Solutions to Problems 1.1 p(A) + p ( B ) = 1

+ + +

~ ( Y I= ) 0.5p(A) 0.311 -p(A)1 ~ ( Y z=) 0.3p(A) 0.W - p(A)1 p(ya) = 0.2p(A) 0.2[1 - p(A)] = 0.2. ) = - 10.3 0.2p(A)]log 10.3 H( Y ) = - z - , p ( ~ ~logp(yt) -10.5 - 0.2p(A)]log [0.5 - 0.2p(A)] - 0.2 log 0.2.

+

= A ) + P ( B ) H ( Y ~ X= B ) H(YIX) =~(A)H(Y/X = H(0.5,0.3,0.2) = -0.5 log0.5 - 0.3 logO.3

+ 0.2p(A)]

- 0.2log0.2.

Z(X 1 Y ) = H ( X ) - H(X I Y ) . T o maximize I(X ( Y ) , differentiatewith respect t o p ( A ) and set the result equal to zero. The maximum occurs when p(A) = 0.5; hence Zmax =

-0.8 l0g0.4

+ 0.5 l0g0.5 + 0.3 l0g0.3

X

= 0.036.

Y Yi

Tell the truth

y2

Lie

y3

Refuse to answer

B

Problem 1.1

(No heads)

(1 head)

(2 heads)

Problem 1.2 293

294

INFORMATION THEORY

a. Z(X( Y ) = H ( Y ) - H(YI X )

b. Z(X I Z )

= H(0.2,O.g)

- 0.75H(0.1,0.9) - 0.25H(0.5,0.5)

= 0.12

= H(Z)

- H(Z I X ) = H(0.52, 0.48) - 0.75H(0.46, 0.54) - 0.25H(0.7,0.3)

= 0.03 c. First digit conveys H ( X ) = H(0.75,0.25) = 0.81 1 Second digit conveys H(YI X ) = 0.75H(0.1,0.9) 0.25H(0.5,0.5) = 0.602 Third digit conveys H(Z I X, Y ) = H(Z ( Y ) [sincep(z, I x,, y5) =p(zk I y,)] = 0.2H(1) 0.8H(0.4,0.6) = 0.8H(0.4,0.6) = 0.777.

+

+

Pass

xl

Fall

x2

5

yl

Own car

zl

Fratern~ty

yz

Do not own car

22

Nonfraternlty

Problem 1.3

By Lemma 1.4.1, for each i,

with equality if and only if p(y3, zk I x,) = p(y5 I x,)p(zk I x,) for all j, k. Thus H ( Y I X) H(Z I X ) 2 H ( Y , Z 1 X ) with equality if and only if

+

so H(Z I X, Y ) 2 H(Z I X ) with equality if and only if

SOLUTIONS TO PROBLEMS

1.5 By Lemma 1.4.1,

- f a, log a, i - 2 at log t=l

2=1

Thus

and the desired inequality follows. By Lemma 1.4.1, equality holds i f and only if azx, for all i, a, = -

i

3=1 n

2,

=

2 a , ~ =, constant

for all i.

5=1

+

l o g p d Pz + AP

2 -P1h

(PI

- Ap)

M

- pz log (pz + ApP) - I p , logp, 133

since the last term is 2 0

1.7 p,'

2 g l a,,p,; hence M H@,', . . . , p M 0 = - zpzt logp,'

=

M M

=

2-1

M

- r=lr=l 2 I a,,p, logp,'

M

I P. [Ia,J l0gp;].

= - 3-1

2=1

Now by problem 1.5, M

(plt)alj . . . ( p M W < 2 az3pzf. ,=l

(Note 22, a,,

=

1.) Hence, taking logarithms,

296

But ~

INFORMATION THEORY

~

,

( a,,p,') z ~ = ,z z l p , ' ( x E l a,,)

=

1

SO

Lemma 1.4.1 yields

so that H(pl', . . . ,pM1) 2 H(pl, . . . ,p M ) Clearly if (pl8,. . . ,pu') is a rearrangement of @,, . . . ,pM), then H(plf, . . . ,pMr) = H(pl, . . . ,pM). On the other hand, if the uncertainties are equal, then equality must hold in both (1) and (2), the equality in ( 1 ) holding for allj. Equality in ( 1 ) implies (see Problem 1.5) that for each j, allptf corresponding to positive a,, are equal, that is, M

p,' =

2 a,,p,'

= a constant c5

,=1

whenever a,, > 0. (Note ifp,' = c, then But equality in (2) implies

a,,p,' = c,

zz,

a,, = c,

= p,'.)

M

pj

2 a,,p,' ,=l

=

for all j.

(4)

Therefore, by (3) and (4), a,, > 0 implies p,' = p,. Since for each i, there is at least one j such that a,, > 0, each p,' is equal to some p,; hence (p,', . . . , pM') is a rearrangement of (p,, . . . ,pM). In Problem 1.6 we have

+

P1' = p1 - Ap = up1 Bp, Pz' = pz + AP = BPI + ap, i =3,. . . , M Ptf=p,, matrix

A =

is doubly stochastic, and therefore Problem 1.6 is a special case of this result. 1.8 H ( X , Y ) = H ( X ) + H ( Y IX ) = H ( Y ) + H ( X I Y). B U ~H ( Y I X ) = O since Y is determined by X, hence H ( Y ) H ( X ) . H ( Y ) = H(X)if and only if H(X I Y ) = 0, that is, if and only if X i s determined by Y, that is, for each y,

SOLUTIONS TO PROBLEMS

297

there is an xi such that p(zi I y,) = 1. Hence H(Y) = H(X) if and only if g is one to one on {z,, . . . ,xM), that is, xi # x j impliesg(xi) #g(xj). 1.9 P{Z = z, 1 X = 2,) = P{Y = z, - xi 1 X = 2,). Given X = xi, as zk runs over all possible values of Z, z, - z i runs over all possible values of Y; hence H(ZI X = x , ) = H ( Y I X = x J for eachi. Consequently H(Z I X) = H ( Y I X). The remaining conclusions of part (a) follow from Theorem 1.4.5. For part (b), take Y = -X, then H(Z) = 0 so in general H(Z) < H(Y). H(Z) < H(X), 1.10 For k = 2, this is the ordinary grouping axiom 3. The general assertion follows by induction, since if the result has been established for k - 1, then by induction hypothesis,

Applying the ordinary grouping axiom, the second term on the right becomes

298

INFORMATION THEORY

Again using the induction hypothesis, the sum of the first two terms on the right in this equation is H@1

+ . . . + Prl,

Prl+l

+ ..'

L

group 1 ( 2 terms)

--

p ~ Pre+l ~ ,

+ . . . + Pr,, .

9

Prk-l+i

+

'

. .prJ

P group k - l

group 2

( 1 term)

(1 term)

and the result follows. The above argument uses the ordinary grouping axiom but none of the other axioms for the uncertainty function. If we use all four axioms, the derivation t . the right-hand side of the statement to be is easier: Let q3 = ~ ~ & r , - l + l pThen proved is

+

1-11 Let h'@) = h(p) C logp. Then 2 ~ l p 2 h ' ( p t=) 0. Taking all = l / n we have h'(l/n) = 0,n = 1 , 2, . . . . If r/s is a rational number, then, taking 1

p,

S

we have

hence hl(r/s) = 0. The result now follows by continuity. 1.12 Let f(n) = h(l/n). Then the requirements (a) and (b) imply f(nm) = f ( n ) + f ( m ) , and n < m - f ( n ) < f ( m ) , n , m = 1 , 2 ,.... The proof of Theorem 1.2.1 in the text shows that f ( n ) = Clog, n where C > 0,b > 1. If p is a rational number r/s, then

Thus

Hence h(p) = -Clogp continuity.

for rational p .

The general assertion follows by

SOLUTIONS TO PROBLEMS

2.1

a.

so Sl s 2 s 3 1 100 11 010 11 10 oool 01 10 0101 1 1loo OOO11 00110 11110 10101 1 The code is not uniquely decipherable. A.

-

0001

Al

--

A , = 01011

1

5'4

s 5

00 110

01 011 110 0

S6 0 10 001 110

&code

A, = 11

A, = 110

A, = 0

A6 = 0110

w, = 00011 W , 101011 w, = 010 W,= 11110 W,= 1100 W,= 00110 Ambiguoussequence = AoWlW4W6 =0001 101011 1100 0110 = wow,w3w5= oooll 010 11110 00110 '0

abc abcd e dba bace ceac ceab eabd

word

w, = 0110

sl

'8

'3

'4

'5

'6

s7

s8

sg

s l ~

d abd

ba

ce

ac ab

cd c

eac eab

ac ab d

c cd ba

eac eab ce

ac ab d

S7 = Slo; hence Si = Si+3,

i27

None of the sets S , contains a code word, so the code is uniquely decipherable. 2.2 a. Let w, be the number of code words of length i (w, = 1 , w, = 2, wn = 0, n 2 3). Then a message whose coded form has exactly k letters must begin with a code word of length 1 or a code word of length 2; hence N(k) = w,N(k - 1 ) w2N(k - 2), k 2 3. Now N(2) = w,N(l) w2; hence if we define N(0) = 1 , the above equation is valid fork 2 2. Thus we must solve the linear homogeneous difference equation N(k f 2) -wlN(k 1 ) - w,N(k) = 0, k = 0, 1 , . . . , subject to N(0) = 1, N(1) = 1 . Assuming a solution of the form N ( k ) = Ak we obtain Ak+, - w1lk+l - wZAk= 0 or Ak(A2 - A - 2) = 0. The two nonzero roots are A, = 2, A, = - 1. Hence N(k)=A.2k+B(-1)k, k=0,1, Since N(0) = A B = 1 , N ( l ) = 2A - B = 1 , we have A =$,B=$. S 0 N ( k ) = ( g ) 2 ~ + ( ( g ( - l ) ~ , k = 0 , 1 , .... b. As in (a), N(k) = w,N(k - 1 ) w,N(k - 2) w3N(k - 3) ( o , = 1, w2 = 1 , w3 = 2, wn = 0, n 2 4 ) or N(k 3) - N(k 2) - N(k 1 ) - 2N(k) = 0, k 2 0.

+

+ +

....

+

+

+

+

+

+

300

INFORMATION THEORY

The assumption N(k) = Ak yields the characteristic equation R-j12-1-2=(A-2)(12+1+1)=0

with roots j13 1, = -4 + @2/3 = e"2"13, 1, = 2, - -4 - h 1 2 / j = e-~2n13. Thus N(k) = A . Zk B cos ( 2 ~ / 3 ) k C sin ( 2 ~ / 3 ) kk, = 0, 1 , . . . . The boundary conditions are N(0) = 1 = A + B N ( l ) = 1 = 2A - &B 42/5 C N(2) = 2 = 4 A - + B - * 2 / 5 ~ . These equations have the solution A =$ , B = $, C = 2/3/21, so

+

+

+

+

2~ ~ ( k =) ( $ 1 2 ~+ cos - k 3

4 3 sin . 277 +k, 21 3

Symbol Probab~hty

2

2

I

Problem 2.3

k

= 0,1,

... .

301

SOLUTIONS TO PROBLEMS

2.3 The process of constructing the code is indicated schematically in the pini = 3.419. accompanying figure. The average code-word length is ri = The uncertainty may be computed to be H ( X ) = 3.355. 2.4 By Theorem 2.5.2 there is a binary code whose average word length is less than H ( X ) 1 ; since the Huffman code minimizes the average word length for a given set of probabilities, the result is immediate. 2.5 Add fictitious symbols with zero probability to the given set until the number of symbols is congruent to 1, modulo (D - 1). An optimal code for the enlarged set will then yield a n optimal code for the original set. Now we shall prove that given any optimal code we may construct another code with exactly the same word lengths such that, among the words of maximal length, at least D words agree in all places but the last. Therefore without loss of generality we may assume that an optimal code has this property. In particular there are at least D words of maximal length, and consequently (note that Lemma 2.6.2a holds just as in the binary case) the D least probable symbols have code words of equal length. The proof for the binary case now goes through word for word, except that we combine symbols D at a time instead of 2 at a time. . T o prove the required assertion, pick any word W of maximal length n ~ Let Q be the sequence consisting of all digits of W except the last. Say the code alphabet is (0, 1, . . , D - 1) and W = Q0. If there are more than D words of maximal length we may, without changing the instantaneous property of the code, take any (D - 1) words of length n~ other than W a n d replace them by Q l , Q2, . . . , Q(D - 1); thus it suffices to consider the case in which there are at most D words of maximal length; we may assume without loss of generality that these words are Q0, Q l , . . . , Qi, i < D. We may assume that i > 0 since if i = 0, QO may be replaced by Q. Now as in the proof of Theorem 2.3.1, the number of terminal points on the tree of order D and size n~ excluded by a code with word lengths n,, . . , n~ is T = z;yl Dcn~-"l). Since D = 1 mod ( D - 1) and (after the fictitious symbols are added) M = 1 mod ( D - I), it follows that T r 1 mod ( D - 1). If R is any sequence of length nnf - 1 other than Q, then if R is not a code word and no code word is a prefix of R we may replace one of the words Q j by R. The resulting code is still instantaneous and has an average word length at most that of the original code. (Possibly Q j corresponds to a symbol with probability zero so that the average word length is unchanged.) Thus without loss of generality we may assume that every such sequence R is either a code word or has a code word as a prefix. In either case, the terminal points corresponding to the sequences Rj, j = 0, 1, . . . , D - 1 are all excluded. Therefore the only terminal 1), . . . , Q(D - 1). points not excluded are those corresponding to Q(i Since the total number of terminal points is Dnnr = 1 mod (D - I), the number of terminal points not excluded is congruent to zero modulo (D - 1). Hence D - 1 - i = 0 or D - 1 ; the second case is impossible since i > 0; consequently i = D - 1 and the proof is complete. For the given numerical example, we first group 2 symbols at a time (or add one fictitious symbol and group 3 a t a time) and afterward we combine 3 symbols a t a time. The code-word lengths are 1, 2, 2, 2, 2, 2, 3, 3.

zizl

+

.

.

+

302

INFORMATION THEORY

3.1 Since 8

P(Y,) =

2 p(x,)p(y, 1 x,)

1-1

and p(x, 1 Y,) =

P(x,)P(Y, I xi) P(Y5 )

we compute

Thus the ideal observer is g(yl) = xl,g(y2) of error is

xl, g(y3)

= x3.

The probability

I

3.2 If y, is received, the probability of correct transmission is p(er y,) = ~ ~ 1 q 3 1=P2,{( ~Y,). (Necessarily q,, 2 0 for all i, j and Zfl_,q,, = 1, j = 1,. . . , L.)

equality can be achieved by taking q,,o = 1, qj2 = 0, i # i,. Thus p(e' 1 y,) is maximized for each j, and hence the overall probability of correct transmission is maximized, by the above choice of q,,; however, this choice is precisely the ideal observer.

The essential consequence of the cascading process is that p(z I x, y) = p(z I y) for all x, y, z; in fact, this condition should be used as the definition of a cascade combination. It follows thatp(x I y, z) = p(x I y), hence H(X I Y, Z ) = H ( X I Y). Thus I(X1Y)-I(XIZ) = H(X1Z) - H ( X ) Y,Z) 2 0 by Problem 1.4. Since I(X ) Y) 2 I(X 1 Z ) for any input distribution, the capacity of K, 2 the capacity of the cascade. Now write I(Y[Z) Z(X I Z)

= H(Z)

- H(ZI

Y) = H(Z) - H(Z I X).

SOLUTIONS TO PROBLEMS

I

I

I

Since p(z Y) = p(z I x, Y), H(Z Y) = H(Z X, Y). ~ h u s I(Yl z) - I ( x / Z ) = H(zI X) - H(ZI X, Y) 2 0, so that the capaclty of Kz 2 the capac~tyof the cascade.

X

4

Y

Kz

z

Problem 3 3 3.4 We may regard Y as the mput and X the output of a dlscrete memoryless channel, the output X may be thought of as being apphed as the mput of a determlnlst~cchannel wlth output Z (see d~agram) The cond~tlonsfor cascadmg are sat~sfied,that IS, p(z I y, x) = p(z I x), the result then follows from the argument of Problem 3 3. Alternately,

I

I

H ( Y Z) = E[ -logp( Y a 1 = E[ -logp( Y l g(X))1 = - ~ p ( x , , Y3) log P{ y = Y) z = g(xz>>

I

K1

Y Problem 3 4

X

K2

Channel matr~xof K ,

= g(X) = [p(z I y)]

+

3.5 p, = (1 - B)pnPl p(1 - P,-~) or pn - (1 - 2p)pn-, = 8. The solut ~ o nto t h ~ sd~fferenceequat~onIS p, = A(l - 2p)" + B where the "part~cular solut~on"B IS determmed by B - (1 - 2p)B = p or B = &, and A 1s determmed by taklng n = 0 A + B = p, Thus pTL= .$ + ( p o -&)(I -2p)" -4 as n a. Now the channel matrlx of the cascade cornb~nat~on of n channels 1s 0 1 -+

+

where pn* = & (po - &)(I - 2p)", evaluated when po The capac~tyof the n-stage cornblnatlon 1s

Problem 3 5

=

1.

304

INFORMATION THEORY

3.6 Let

ni = channel matrix of K,, i = 1 , 2.

We have

Thus the matrix of the cascade combination is IIITI,; similarly if channels . . . ,K, are cascaded (in that order) the matrix of the cascade combination is the product 11111, . . . II,.

K,,K,,

X

KI

Y

K2

Problem 3.6

Remark Concerning Problems 3.5 and 3.6

+

If Xiis the output of the ith channel [ = the input of the (i l)th channel] of a cascade combination, the conditions for cascading are expressed by re1 x,,, . . . ,xL1) = p ( ~ 1,xkP1)for all x,, . . . ,x,, k = 1 , 2, . . . , n. quiring~(~ 3.7 Without loss of generality assume B > a. We may use Theorem 3.3.3. We have

Let Ha = H(a, 1 - a) = H ( Y I X = xl), H p = H(p, 1 - 8) = H ( Y I X = %I. We must show that the hypothesis of Theorem 3.3.3 is satisfied. We have [see (3.3.4)]

Thus dl 1

> 0 if and only if

-8

log -<

B

1 -B -BH, +aHp - ( 1 -B)H, - ( a - I ) H a , or log B B -a

Upon expansion, this becomes -a log a - ( 1 - u ) log ( 1 - a ) < - log B - ( 1 - a ) log ( 1 - B), which is true by Lemma 1.4.1. Consequently d , > 0. A similar argument shows that d, > 0. Thus by ( 3 . 3 3 ,

3.8 Let the input alphabets of Kl and K, be x,, . . . ,x, and x,,~, . . . , X M respectively; let the output alphabets of Kl and K, be yl, . . . , y, and ,,,y, ...,

SOLUTIONS TO PROBLEMS

yL. If p(x) is any input distribution, a n d p = z:=lp(x,), then M

H ( X ) = -Z.p(x,) logp(x,) ,=1 = -p logp - (1 - P ) log (1 - p ) + p H l ( X )

+ (1 - p ) H z ( X )

by the grouping axiom 3 of Section 1.2 where

is the input uncertainty of K , under the distribution {p(x,)/p, i

=

1,

. . . , r} and

is the input uncertainty of K, under the distribution {p(x,)/(l - p ) , i 1, . . . , M). (Alternately

=r

+

leading to the same expression as above.) Now

Now if 1 < i I; r, 1 I; xl I Y = y,) is

i

I; s, the conditional probability p(x, I y,) = P{X =

I

I I

P ( ~ , ) P ( Y , 2,) - prp(~1)lpIP(Y, 2,) P(Y,) p@,) -P(Y* xz) p ,=1 P = P { X l = x , Y l = Y,),

I

p(x, Y,) =

* Z.

I

that is, the conditional probability that the input of Kl is x , given that the output of K , is y,, under the distribution { p ( x J p , i = 1, 2, . . . , r). Thus H ( X I Y) = pHl(X I Y ) = H@, 1

+ (1 - p)H,(X I Y ) ,and I(X I Y ) =H ( X ) - H ( X I Y ) + pZl(X I Y ) + (1 - p ) M X I Y ) ,

- p)

where the subscript i denotes that the indicated quantity is calculated for K, under the appropriate input distribution. Now for a givenp, we are completely

306

INFORMATION THEORY

free to choose the probabilities p(x,)/p, i = I , . . . ,r, and p(z,)/l -p, i = r 1, . . . ,M. We can do no better than to choose the probabilities so that I,(Xl Y) = c,, I,(X I Y) = c,. Thus it remains to maximize H(p, 1 - p) + pCl (1 - p)C,. Differentiating, we obtain 2", -1 -1ogp + 1 + l o g ( l - p ) + C l -C, = O , o r p = 2c1 + 2C1 '

+

+

Hence

Therefore C = log (2"' + 2C2). 3.9 Let X, be the input to Kt, and Y, the corresponding output. By defin~tion of the product channel,

NOWH(Y,, Y,) < H(Y,) + H(Y,) with equality if and only if Yl and Y2 are I(X2 I Y,) 5 C1 C,. independent. Therefore I(X,, X, I Y,, Y,) 5 I(X, I Y,) If we take X, and X, to be independent, with X, having the distribution that achieves channel capacity for K,, then Y, and Y2 are independent (by the same argument as in the proof of the fundamental theorem), and equality is achieved in the above expression. C,. Thus C = C, 3.10 Channel (a) is the sum of

+

+

+

c =log4 Erase

7

1-1

1-P

Problem 3.10

+plog&p +(1 -p)logM

-PI,

or C = 1 - H@, 1 -p) = the capacity of a binary symmetric channel with error probability p. For channel (c), see the accompanying figure. If PIX = 0) = a then H(X) = H(a, 1 - a). H(X I Y) = p(erase)H(X I Y = erase) = pH(a, 1 - a).

Thus I(X I Y) = (1 - p)H (a, 1 - a), which is a maximum for a = 4. Therefore C = 1 - p. 3.11 a. Proceed exactly as in the proof of the fundamental lemma, with the set A replaced by A* = A n F*, where F* = {(x, y):x e F). Observe that all

307

SOLUTIONS TO PROBLEMS

code words selected in the algorithm must belong to F, since i f x IF, A,*(= { y : ( x ,y) E A*)) is empty. W e obtain a code (s, n, A) such that I I; s .2-" P{(X, Y ) I A*). Since

+

P{(X, Y ) I A*) I, P{(X, Y ) A )

+ P{(X, Y )4 F*) = P{(X, Y )

A)

+ P{X

4 F),

the result follows. b. W e shall prove that the maximum value o f Co,that is, the least upper bound o f all permissible rates o f transmission, is

-

which is the information processed by the binary symmetric channel when the input distribution is p(1) = r,p(O) = 1 - r. Notice that as r + i, Co C = 1 - H(P, 1 - P). W e shall first prove a coding theorem, which states that given R < Co*,there exist codes ([2nR], n, A,), with all words belonging to F,, such that lim A, = 0. n- m W e use the results o f part (a), with F = F,. Let X = ( X I ,. . . , X,) where the X, are independent and P{X, = 1) = p < r, P{X, = 0 ) = 1 - p. Then P{X 4 F,) = ~ { n - l X, > r) 4 0 as n -+ a by the weak law o f large numbers. Let us take a = kn. Then as in the proof o f the fundamental theorem, P { ( X , Y ) 4 A ) = P{nU, 5 k ) , where the U , are independent, identically distributed random variables with expected value

2kl

x;=,

-

+

Given R < Co*, we take s = [2nR],k = 3(R Co*),andp sufficiently close to r so that I(X, I Y,) > k. Then by choice o f s and a = kn, s . 2-a 0 as n + a. Also P{(X, Y ) # A) 0 as n oo by the weak law o f large numbers. The result follows. W e now prove a weak converse, which states that any code (s, n), all o f whose words belong to F,, must satisfy nCo* + log 2 logs 5

- -

1-p(e)

'

where p(e) is the average probability o f error o f the code. As in Theorem 3.7.3, this implies that i f we require that all code words belong to F,, it is not possible to maintain a transmission rate above C* , with an arbitrarily small probability of error. Given a code (s, n) with all words in F,. W e can proceed just as in the proof o f Theorem 3.7.3 (with Co* replacing C ) except for one difficulty. Although we have I(X I Y) < I(X, I Y,) as in (3.7.4), it is not true that I(X, I Y,) < Co* for all i. For example, i f s = 2, n = 5, r = 5, and the words o f the code are w1 = 1 1000 and w2 = 10000, then P{X2 = 1) = P{X2 = 0 ) = 4, and therefore I(X2I Y2)= C > Co*. However, we may prove that I(X, I Y,) < K O * . Thus I(X I Y ) I, K O * ,so that the proof in the text will go through. T o see this,

z:=,

308

INFORMATION THEORY

let X be a symbol chosen with uniform probability from among all sn digits of the code vocabulary w,, . . . ,w,. Since X may be selected by first choosing a digit j at random, and then selecting a code word at random and examining the jth digit of the word, we have

x;=,

By Theorem 3.3.1, I(X I Y ) 2 n-l I(Xi I Y,). But since all words belong to Fn,P { X = 1) 5 r, and hence I(X I Y ) < C,*. The result follows.

3.12 The information I(X I Y) processed by a channel may be expressed as a function of the input probabilitiesp(x,), . . . ,p(x&,).[See (3.3.6).] The domain of I is the subset of Euclidean M-space on which all p(x,) 2 0 and

Thus the domain is a closed and bounded subset of Euclidean M-space. Since such a set is compact, and a continuous function defined on a compact set attains a maximum value on the set, the result follows. 3.13 The easiest example is that of a useless channel. In this case Y,,. . , Yn are independent regardless of the input distribution. To see this, write

.

Since the channel is memoryless the second factor in the summand is

Since the channel is useless, P { Y i = pi I Xi = a i ) = P { Y , = pi). It follows that P { Y , = Dl,. . . , Yn = /In) = P { Y i = B,), and therefore Y,, . . . , Yn are independent. 4.1 There are 2" - 1 nonzero binary sequences of length m. Hence for a given m, the maximum n is 2" - 1. Note that since 2" = 1 n, the Hamming bound (Theorem 4.5.2) for a single-error correcting code is met. For m = 4 we have

IT:==,

+

~ 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 1

+

4.2 a. It is possible to correct e-tuple errors and detect (e 1)-tuple errors e yields a distinct corrector and in if and only if every error pattern of weight 1 has the same corrector as an error addition no error pattern of weight e

+

309

SOLUTIONS TO PROBLEMS

+

pattern of weight < e. (For then an (e 1)-tuple error would be confused with e and would therefore not be detected.) This is an error pattern of weight 1 columns of the parity check precisely the condition that every set of 2e matrix be linearly independent. b. Since the original code corrects e-tuple errors, every set of 2e columns of A is linearly independent. Now suppose

<

+

2e+l

2 Aikc(rik)= 0

(modulo 2),

k= 1

+

where l j = 0 or 1 and c(rj) is the jth column of A,. Since 2e 1 is odd, and each column of A, has a "1" in its bottom element, there must be an even number of nonzero A's, in particular there are at most 2e nonzero A's. This implies a linear dependence among 2e or fewer columns of A, a contradiction. c. Augment a parity check matrix of a single-error correcting code to obtain

("

> ~~~, I).

d. The condition is 2"

In view of part (a), the proof

proceeds exactly as in Theorem 4.5.3.

b. Diagonalize the parity check matrix A (see Section 4.5) to obtain a new parity check matrix

r1

O O O

0

1 0

0

0

1 0

LO o o If the last column of A' is

0

1

1

+ a,, 1 al, + a,, o a,, + a,, + a,,J 1 a,,

310

INFORMATION THEORY

then every set of 3 columns of A' is linearly independent and the code will correct single errors and detect double errors. We obtain

111

+

+

+

Now in A', c(r,) c(rJ c(r,) c(r,) = 0 regardless of the choice of the elements of column 6; hence not every set of 4 columns of A' (hence of A) 1s linearly independent so that not all double errors can be corrected. 4.4 Let w be a nonzero code word of minimum weight, and let h be the weight of w. Since dist(w,, w) = h, (where w, is the zero code word), the minimum distance is 5 h. Suppose the minimum distance is h, < h. Then there are distinct code words w,, w, such that dist(w,, w,) = h,. But dist(w,, w,) = dist(w, w,, w, w,) = dist(w,, w, w,) = weight of (w, w,). Thus the w, is h, < h, a contradiction. weight of wi 4.5 a. An error in decoding will be made if and only if an error of magnitude e is confused with the "no-error" condition. This will happen if and only if some set of e or fewer columns of the parity check matrix adds to zero. Hence e-tuple (and all smaller) errors can be detected if and only if every set of e columns of the parity check matrix is linearly independent. b. To say that columns k,, k,, . . . , k, of a matrix A add to zero is to say that AwT = 0 where w is a row vector with ones in positions k,, . . . , k, and zeros elsewhere. Thus the equivalent condition on the code words is that every nonzero code word have weight > e. (Alternately, the minimum distance between code words is > e.) c. An error pattern z is detectable if and only if z does not yield the same corrector as the zero vector, that is, if and only if AzT # 0, that is, if and only if z is not a code word. Therefore the number of detectable errors is 2" minus the number of code words, or 2" - 2k. 4.6 a. If w is a code word, then w = I,u, (modulo 2) where the u, are the rows of G*, and I, = 0 or 1. Thus

+

+

+

+

+

<

z:=,

SOLUTIONS TO PROBLEMS

and therefore

Thus [I, I BT]wT = 0 for every code word w. Since rank [ImI B ~ =] m, [I, I BT]wT'= 0 has 2k = 2*-" solutions; thus the vectors w satisfying [ I , I B ~ ] w ' = 0 are precisely the code words of the given code, so that [ImI BT] is a parity check matrix. b. (i) Diagonalize G to obtain

.*=I 1 0 1 0 0 0

1 1 .0 I 0 10 . 01 .

m c e A =

[

01 10 01 1 1 10

I

0 1 0 0 0 1

(ii) C*

=

G

=

[I

1

1

1

I]

so A

=

4.7 Let A = [ I , I A,]. Delete row 1 and column 1 of A (or equally well, row j and column j where 1 5 j 5 m). If columns i and j of the reduced matrix are the same, then columns i and j of the original matrix A agree in all digits c(ri) c(ri) = except the first, where they must disagree. It follows that c(r,) 0, contradicting the hypothesis (see Problem 4.2) that every set of 3 columns of A is linearly independent. Furthermore, no column of the reduced matrix can be zero. For if column i of the reduced matrix is zero, then column i of the original matrix would be zero or else would coincide with column 1 ; each possibility contradicts the hypothesis. 4.8 a. Take

+

+

312

INFORMATION THEORY

+

n = 7, k = 4, rn = 3; 2m = 1 n, 2k = 16 = number of code words. The Hamming bound is met, hence the code is lossless. b. The Hamming bound is

Since m

= 4,

n2 + n

- 30 = 0 or n

= 5.

Then k

=

n

- rn

=

1

l o 0 0 1 11 (compare Problem 4.6b). c. If n = 2m - 1 the Hamming code meets the Hamming bound and is lossless. In general, let 2m-1 5 n < 2m. [Note that by construction of the code (Problem 4.1), n cannot be less than 2"-l.] Given any binary column vector c of length m, if c is a column of the parity check matrix A then c is the corrector associated with a single error. If c is not a column of A then c is the modulo 2 sum of the column vector corresponding to 2m-1 and a column vector c' corresponding to a binary number less than 2"-l. For example if m = 4, n = 10 then

The vector

is not a column of A, but

Thus all correctors correspond either to single or to double errors, so the code is close-packed.

SOLUTIONS TO PROBLEMS 4.9

B(1, 8 ) = - H ( I , 1

- 1) - 1 log 8 - (1 - 1) log (1 - 8 )

aB a1

Now 1

-R

= H(1,

1

-log-

1 - 1 1

- 1) so dRld1 aB = - a= I~ - / aR dRld1

1-8 + log 8

= log

a ~

[1/(1 - A)].

log (1 log (1

Thus

- BY8 - 1)/A

Since

Therefore the tangent to B(1, 8 ) at 1 must be

=

1, has slope -1, so that the tangent

which is the lower bound of (4.7.28) for R < Rc. 4.10 Given a code (s, n) with average probability of error less than 1, choose any number k > 1 such that k 1 < 1. Then it is not possible that as many as s/k code words w have p(e I w) 2 k l , for if this were the case, we would have

I1, a contradiction. and consequently the average probability of error would be ; Thus we may construct a subcode having at least s - s / k words and a maximum probability of error < k l . If s = [2nR],R > C,then there is a 6 > 0 such that for sufficiently large n, s(1 - k-') 2 2n'C+*'. (Compare Lemma 3.5.3.) In other words, we are able to construct a sequence of codes that allows us to maintain a transmission rate above channel capacity with a maximum probability of error < k l < 1 . This contradicts the strong converse (Theorem 4.8.1). 4.11 Let Bi be the decoding set of w i(i = 1 , 2, . . . , s) in the original code. Since the first and second words are assigned the same probability, the average probability of error is unchanged if we replace B2 by B, LJ B2 and B, by the empty set 0. Now consider the new code in which w,' replaces w,. If we take the-decoding set of w,' to be 0, the decoding set of w 2 t o be B, u B2,and the decoding set of w ito be Bi, i > 2, then the average probability of error is the same as that of the original code. But the ideal-observer decision scheme has an average probability of error at least as small as the scheme just given. The result follows.

314

INFORMATION THEORY

Choosing

we obtain

A =

1 : i,

0 1 0 0

b.

T - 1

period-..

With

Since there are 3 cycles of length 5, we may get different parity check matrices for different choices of x. However, all such matrices define the same code (see Lemma 5.3.3).

315

SOLUTIONS TO PROBLEMS

There is one cycle of length 21, one cycle of length 7. one cycle of length 3, and the trivial cycle of length 1 . Hence the period of T is 21. If

then

!

0 0 0 0 1 1 1 1 1 0 1 0 1 0 0 1 1 0 0 0 1

I

0 0 0 1 1 1 1 1 0 1 0 1 0 0 1 1 0 0 0 1 0

A = A , = 0 0 1 1 1 1 1 0 1 0 1 0 0 1 1 0 0 0 1 0 0 . 0 1 1 1 1 1 0 1 0 1 0 0 1 1 0 0 0 1 0 0 0

1- I :I

1 1 1 1 1 0 1 0 1 0 0 1 1 0 0 0 1 0 0 0 0

If

x =

0 1 1 1 0 0 1

then A

: :0 A

= A. =

(A2has rank 3).

1 0 0 1 0 1 1

0 0 1 0 1 1 1

If

x

I] 1;

= 0

, then A

= A, = 0

+ +

1

(A, has rank 2).

1

5.2 By Lemma 5.3.3a, f (x) = 1 x4 x6 is a multiple of the generator polynomial g(x). Now x6 x4 1 = (x3 x l)(xe x 1 ) ; since the rank of the parity check matrix A, of Problem 5.lc is 3,g(x) is of degree 3, and 1. We may therefore generate the code by taking thus g(z) = x3 x

+ +

+ +

+ +

+ +

[I : 1 [I]

T= 0 0 1

and x =

0 .

The corresponding parity check matrix is

Ea;K;sj which is of the form (5.3.1). It follows that A(x)

=

1

+ x + x2 + d.

316

INFORMATION THEORY

5.3 Let A be as in Table 5.4.1. If w = (r,, r,,

. . . rn-,)

is a code word then

If A, is premultiplied by the matrix

we obtain (noting that T S n x = x, s = 1,2, . . . , 2 e

- 1)

Thus A, is obtained from BA, by a cyclic shift of columns. Examination of the equation B A , W ~ = 0 shows that (rn-,r0. . . r,,) is a code word. 5.4 e = 2, n = 31 = 2q - 1, so q = 5. From Table 5.2.1 we may take

SOLUTIONS TO PROBLEMS

with characteristic polynomial A5

+ A2 + 1.

If

the corresponding parity check matrix is

5.5 a. There are 7 - 3 = 4 check digits, hence the code may be generated by a nonsingular 4 by 4 matrix T of the form (5.1.1), and a vector

+

+

+

(See Corollary 5.3.4.1.) If T has characteristic polynomial a,, a l l ... am-,Am-l Am, then (a,a, . . . a,-l10 . . 0 ) together with k - 1 cyclic shifts generates the code. By shifting the given word w = (1010011) we find that ( 1 110100) is a code word and that (1110100), (0111010), (0011101) generate the code. It follows that an appropriate choice of T is

+

INFORMATION THEORY

the parity check matrix is

b. By Corollary 5.3.4.1, g(x) is the minimal polynomial of T, that is, g(x) = 1 + x + x2 + x4.- Since the above parity checkmatrix is of the form (5.3.1), we

obtain h(z) = 1 + x + x3. 5.6 a. The corrector associated with a given error pattern is Tix for some i. (By the remarks after Theorem 5.2.6, every matric polynomial in T is a power of T.) If the error burst is translated by t digits, the associated corrector is TtT'x = Tt+'x. b. The corrector associated with a burst of length q beginning at digit 1 is p(T)x wherep(T) is a matric polynomial of degree 5 q - 1. If pl(T)x = p,(T)x, then ifpl(T) # p2(T),pl(T) - p2(T)isnonsingular by Lemma 5.2.3 and Theorem 5.2.6, hence x = 0, a contradiction. Thus pl(T) = pz(T) so that the corrector determines the error burst uniquely. The above argument holds-under the weaker assumption that T has an irreducible minimal polynomial. 5.7 Clearly, single eirors are correctible. Assume a double adjacent error. The associated corrector is

<

This could not coincide with the corrector of a single error, since the last digit of thelatter wouldbe 1. If ( I T)Tix = (I T)Tix, 0 I i,j 5 n - 1 = 20 - 2 then by Lemma 5.2.3, I T is nonsingular, hence Tix = Tjx. Another application of Lemma 5.2.3 yields Ti = Tj, hence i = j. Thus distinct double adjacent errors have distinct correctors. 5.8 We must show that the corrector determines uniquely (a) the length of the burst, (b) the burst-error pattern, and (c) the starting position of the burst. Let

+

+

+

319

SOLUTIONSTO PROBLEMS

be the corrector corresponding to an error burst of width q,' 2 q,. Since q2 = 2q, - 1, c2 will contain a block of zeros of width 2q1 - 1 - q,' 2 ql - 1 (considering the first component of c, as following the last component); that is,

C2

=

Hence, a. The length of the burst is q2 minus the number of zeros in the block. b. The burst pattern is Iblbz . . . bqlp-,l. No other allowable burst pattern could produce the same c,, for such a pattern would have to begin with one of the b , and end at or after the "1" before b,. The length of the burst would then be 2 q1 1. c. The starting position of the burst is the position of the first nonzero entry after the block of zeros tqz where t is an integer and 0 5 t < n/qz. We claim that t is determined by c, and therefore the corrector specifies the error pattern uniquely. For (see Problem 5.6) the correctors c, = c,, produced by a fixed burst-error pattern of width 5 q, beginning at digit j ( j = 1,2, . . .,241 - 1) are distinct. (If clj and clk are two such correctors, with j < k, then c,, = Tax and c,, = T a + k - 3 ~ ;hence c,, = cIk implies Tk-jx = X, or Tk-j = I, with 0 < k - j < 2q1 - 2, a contradiction.) Thus c, tells us that the starting position of the burst is i tq, and cl tells us ~ 1I), where i and j are known and t < that the starting position is j ~ ( 2 n/q,, u < n/(2*1 - 1). If there were more than one possible value for t (and hence u) we would have i + tq2 = j + ~ ( 2 ~ - 11)

+

+

+

+

Hence ( t - t')qz = (u - u')(2*1 - 1) = r. Assuming that t > t' and hence u > u', r is a multiple of both q2 and 2q1 - 1 and r < n, contradicting the fact that n is the least common multiple. The result follows.

320

INFORMATION THEORY

5.9 We illustrate the calculation for T6. The characteristic polynomial of T5is of degree 4; hence the minimal polynomial is of degree 5 4. Let S = T5. Since T4 T I = 0,

+ +

The "coordinates" of S with respect to I, T, T2, T3are therefore (01 10). Now

We shall try to find bo, b,, b,, b3 such that boI

+ blS + bzS2 + b3S3 + b4S4= 0;

that is,

This yields bo b1

+ b2 + b3 = 0 + bz + b4 = 0. + +

We may take b3 = b4 = 0, bo = bl = bz = 1. Thus S2 S I = 0. Clearly, no polynomial of degree 1 can be satisfied by S. Hence the minimal polynomial ofSis12+l+1. A similar calculation shows that the minimal polynomial of T3 is A3 A2 + A 1. 5.10 (Solution due to A. Wyner and B. Elspas.) Let {(A) be the minimal polynomial of T. Write

+ +

+

where the degree of rk(A) is less than the degree of f(A) = q. If rk(l) = 0 for some k we may replace l by T to obtain T"I=

qk(nf (TI = 0

and we are finished. If rk(A) is the constant polynomial I ( = -1) for some k, then

which is impossible since T is nonsingular. Thus assume that none of the rk(l) is 0 or 1. There are 29 distinct polynomials of degree < y, leaving 2O - 2 when 0 and 1 are removed. Since there are 2q - 1 of the rk(l), we must have rk,(A) = rk2(A)for some k, > k,. It follows that Tkl = Tkr, or Tkl-li* = I, with 1
SOLUTIONS TO PROBLEMS

6.1 a. To find the order, construct the following table:

Thus Xt-, and X t determine Z t ; hence the source is of order 2.

C.

P { X t + , = A , Xt+2 = B, Xt+3 = By Xt+4 = A, Xt+S = A ) = (since Xt+, = A, Xt+, = B e Z , + , = s,) W t + 2 = s,)P{Xt+3

= B, Xt+4 = Xt+s = A I Zt+2 = ~

2 )

= P{Zt+, = s2)P{Zt+3 = s3,Zt+4 = s49 Zt+6 = S1 I Zt+2 = ~ =

w 2 p z s p ~ r=l a(.1)(.8)(.7) = .0187

The source is of order 1.

WII = W becomes

and

2 )

INFORMATION THEORY

Problem 6 . 3 . q b .

6.3 Let o = space. A possible state diagram figure. If X , = o, then X t = B, X , = C, XtPl # A, X t = A XtP1 = X , = A

is shown in the accompanying Z , = s, Z t = S, z, = Z t = s2 Z , = s,

4

Thus Xt-, and X , determine Z , and the source is of order 2. The steady state probabilities are w , = w, = b, w, = w, = w, = 2. The source uncertainty is H { X ) = * ~ ( 1 / 3 ,113, 113) iH(1) + $H(1/4, 114, 114, 114) = % l o g 3 + $ l o g 4 = % l o g 3 + $ = 1.5975. A state diagram for the first-order approximation is shown in the accompanying figure. Note

+

-

SOLUTIONS TO PROBLEMS

Problem 6.3.c. The steady state probabilities are w, = wl = 116 = p { X t = u) wg = w3 = 219 = P { X t = B ) wc = w4 = 219 = P { X t = C ) W A = w2 w6 = 7/18 = P { X t = A).

+

The uncertainty of the first-order approximation is

which is larger than H { X ) , as expected. 6.4

P { X I = a ,,..., X, = a n ) = ~ P { X ~ = U = ~ a, l X , . .~. , X,, = a n ) a0tr

.

Now in S n , p ( a o a , , a n - , l ) ( d l l a t a n ) = Ounless at' = a,, i = 1 , 2 , . . ,n may write the above expression as

- 1, SO we

324

INFORMATION THEORY

This is of the form

... a,) Hence the w ( , .., ~

=P(X1 =

=

Z: WsPa(.l ...

an).

a], . . . , Xn = a,) form a stationary distribution.

6.5

P{Xil = xip= . . . = xi, = 1) =P{X,. = X i = . . . = Xin = -1) = 112 for allii, . . . , in

so the sequence is stationary. Let a be the sequence consisting of the single digit 1 . Then n-'Nan(X1,

Let 0

. . . ,Xn) = nln = 1

with probability 112 = 0 with probability 112

< E < 112; then P{(n-'Nan - P{Xl - 41 = 4.

= a)( 2

E)

= P{(n-'Nan

- 1/21 2 E )

= 1 since In-lNan

Thus &Nan cannot converge in probability to P{Xl = a), and the sequence is not ergodic. 6.6 P{XtPl = A ) = P{Xt-] = A , X , = A ) P{X,-I = A, X t = B ) and

+

The result follows. 6.7 The original source is of order 1, so that H { X ) = H ( X n I X,-,).

Now

Break the summation into 3 parts: 1 . an.-, # e, an-, # e :

2. anPz # e, an-, = e : H(Yn

I

Yo

=

a,,

.,.

YnP1 = an-1) and P{Y,-,

= H ( X n I Xn-, = an-,)

= an-,, Yn-, = e ) = iP{X,-,

= a,,).

SOLUTIONS TO PROBLEMS

Thus H { Y ) = f[H(Xn 1 Xn-,)

+ H { X ) ] . Now

+

H { X ) = &H(5/7,2/7) &H(4/9, 519) and H(Xn I X n 4 ) = w,H(Xn I XnWz = A ) W B H ( X , I X*, = B) .1 6r H 4p8 + -2 5 11 + +g) + I@&@ + ip, $5 + +!). 633 4 8 This determines H { Y ) . 6.8 a. As in the proof of (6.6.3), write

+

2

2

~(v,)=Ck~{v,=k)= ~P{v,=x)+ kp{vn=k) k k<&r .B-cSkl~+a kP{Vn=k). k>B+d Thus

+ 2

and the result follows. b. Let n -+ oo in (6.6.3) and part (a) of this problem. We obtain -E

j -@

+ liminf E(Vn) 5 -j3 + limsup E(V,) n-

n- m

2 6.

m

Smce E and 6 are arbitrary positive numbers, lim E(Vn) = 8. n- w

By part (a) of the proof of Theorem 6.6.1, B = H{X). 6.9 Let {X,, n = 1, 2, . . .) be the given information source. Let a = (a,, . . . , a,) and 3f = (PI, . . . , B,) be any two states of the mth-order approxi1, mation 8,. Consider the set of all sequences y = (Y,, . . . ,y,), r 2 m such that (y,, . . . , y,) = a and (y,,, . . . ,y,) = p. We claim that we can find an r and a y such that p(y) = P { X , = y,, . . . , X , = y,) > 0. For by ergodicity, n-'Nan(X,, . . . , Xn) + p(a), n-'Npn(X1, . . . , X,) + p(P) in probabihty; by definition of S,, p ( a ) > 0, p(P) > 0. Therefore if 0 < 6 < min(p(a),p(p)), then P{n-'Nan 2 6 and n-lNpn 2 6 ) + 1 as n-. w . This implies that there is a sequence y with positive probability such that either y beg~nswith a and ends with P, or vice versa. In the former case, the claim is established. In the latter case, we observe that by ergodicity, the relative frequency of y converges in probability to p(y) > 0, and if yoccurs sufficiently many tunes among the symbols X,, . . . , Xn, a must precede f3 somewhere in the sequence. NOWlet s, = ( y l , . . . ,Y,), s, = (Y,, . . . , Y,+,), . . . ,,,,s = (Y,-,+I, . . . , Y,), and let p(%, st+,) = P { X , = Y,, . . . , X,+,-, = yi+-~, XI+, = ~t+rn)r i = 1 , 2 . . . , r - m. Since X , = y,, . . . , X , = y , o ( X I , . . . , X,) = s,, (X,, . . . , Xm+,) = s2, . . . , (X,-,+,, . . . , X,) = sr-,+,, it follows thatp(s,) > 0 and p(s,, st+,) > 0 for all i. But by definition of S,, the transition probability from S , to s , + ~in S, is

+

326

INFORMATION THEORY

Since s, = a, ,,,s, = p, and p ,, +, > 0 for all i, j3 is reachable from a in 8,. Thus every state in the mth-order approximation is reachable from every other state, and hence S, is indecomposable with the essential class consisting of all the states. 6.10 If the set S of all states is not essential, then there are two distinct states, let us call them s, and s,, such that s, cannot be reached from s,. Let S, be the set of states reachable from s,. Then the states of S, form a Markov chain (if s r S, and s' can be reached from s, then s' can be reached from s, and hence s' r S , ; thus the submatrix of the transition matrix corresponding to the states of S, is a Markov matrix). Since s, ( S,, S, is a proper subset of S. Apply the above argument to S,, and continue in this fashion until an essential set is found. The process must terminate since the original chain has only a finite number of states; if we come down to a chain containing only one state, the set consisting of that state alone is essential. An infinite chain need not have an essential set. For example, take the set of = 1 for all integers i (that is, at each transistates to be the integers, with tion, move one unit to the right). If state j can be reached from state i, then state i cannot be reached from state j, so no set of states can be essential. 6.11 For the source S shown in the accompanying figure, the order of every approximatior, S, is 1.

Problem 6.11. Any initial distribution [w,, w,] is stationary; take w , w,

+ w,

> 0,

w,

> 0,

= 1.

6.12 A state diagram for the source {X,}is indicated in the accompanying figure. The states are aij, j = 1, 2, . . . , n,, i = 1, 2, . . . , M, and the transition probabilities are p(atj, a%,j+l) = 1 for = 1,2, . . . ,ni - 1 and all i; p(ain,, a,,) = pk for k = 1, 2, . . . ,M and all i. Since each code word begins with a different letter, we have a unifilar Markov source. Every state may be reached from every other state, and hence the source is indecomposable. Thus there is a unique stationary distribution with probabilities wij assigned to the states aij. By inspection of the state diagram, Wil = Wi2 = . . . - winL= (say) bi, and M

bk =

2 bip(ain,, akl) = pK

i=l

M

bi for all k. %=1

SOLUTIONS TO PROBLEMS

z!:,

bi) = 1. ~ u t b,ni = 1, and hence ( IElpknk)( = pJri for all i. It follows from Theorem 6.4.2 that

Consequently bi

+

6.13. a. If we form a sequence of length T 1 beginning at state si, the first transition must lead to some state sj. If we are in state s j after one transition, 1 by constructing a sequence of we may complete the sequence of length T length T starting from s,. Since the source is unifilar, all sequences formed in this way are distinct. Thus

+

This is a system of linear homogeneous difference equations with constant coefficients, and therefore the solution is a linear combination of exponentials AT. To find the particular I's, we assume a solution of the form Ni(T) = yiAT to obtain AT(Ay,) = (AT) z = , aijyj, i = 1, . . . ,r, or l y = Ay. Thus the allowable I's are the eigenvalues of A. The result (a) now follows from the observation that N(T) 5

2N m .

i=l

b. Since H(X,, . . . , XT) cannot exceed the logarithm of the total number of possible sequences of length T (Theorem 1.4.2), we have T-lH(X1,.

. . , XT) <

T-I log N(T) I, T-l (log K

+ Tlog A&.

Using Theorem 6.4.1 we have

H { X ) = lim T-l H(Xl, . . . , XT) T-c ca

< log lo.

329

SOLUTIONS TO PROBLEMS

7.1 a. Let the channel be in state skl,, k, i, j = 0 or 1, at time t = n if the ball labeled j is in the occupled slot at t = n, the ball labeled i was in the occupied slot at t = n - 1, and an input k was applied before the transition at t = n. The transition probability from s,,, to s ~under ~ the~input~kl is,given ~ by Mk1(st3,s , ~ ~where ~ ) , the matrices Mkl are given in Table 7.1.1. The transition ~under ~ an ,input~ other ~ than ~ k, is zero. The output probabilrty from s,,, to s functiong' is given byg'(skr,) = g(k, s,,) whereg is the original output function of Table 7.1.1. b. We proceed as in (a). Given a finite-state regular channel with states s,, j = 1,2, . . . , r , and input alphabet r = {b,, . . . , b,), we define a new channel with the same input alphabet r, and states s,,, i = 1, 2, . . . , t, j = 1, 2, . . . ,r . The transition probability from s,, to s , ~under , ~ the input bLlis given by M,l(s,, s,~); the transition probability under any other input is zero. If g(b,, s 3 is the or~ginaloutput function, the new output function is gl(s,,) = g(b,, s,). Since the original channel is regular, for any finite input sequence b,, . . . b,, there is a is channel state sjoand an integer N such that for any initial channel state, s , ~ reachable under the input sequence b,, . . . b,, . . . b,, . . . b,, (N times). In the new channel, the state sPk10 is reachable under the same input sequence, so the new channel is regular. If a source 1s connected to the new channel, the distribution of input and output sequences is (by construction) exactly the same as if the source were connected to the original channel, and thus the capacity is preserved. 7.2 First observe that

by Theorem 1.4.5 or Problem 1 . 4 ~

by stationarity

Thus lim H(Yn I X I , . . . , Xn, Y,, . . . , Yn-,) n- m

Yn 1 XI,.

. . , Xn) = n-I

n

exists.

2 g t n , wheregIn = H(Yz 1 XI,.

Now n-lH(Y,,

. . , Xn,

Yl,.

. . .,

. . , Y,-&

r=l

Since the channel is nonanticipatory, that is, the distribution of a given output letter does not depend on future inputs,

Since lim g, = H({Y) I {X)), it follows that r-m

lim ne1H(Y1,. n- m

. . , Yn 1 XI,. . . , X,J

1

= H({Y) {X}).

330

INFORMATION THEORY

Now by Theorem 6.4.1, H{X, Y } = lim n-lH(X1, . . . , X,, Y,, n-+ m

= lim n-lH(X,,

=%?X}

. . . , Y,)

. . . ,X,) +n+lim n-lH(Y,, . . . , Y, I X I , . . . , X,)

+ H({Y}I {X}).

03

7.3 The results of Problem 7.2 do not hold in general. To see this, let Z o , Z l , . . . ,Z,, . . . , be a sequence of independent, identically distributed random variables, and take X, = Zn-,, Y , = Z,, n = 1,2, . . . We then have H(Yn I X,, . . . , X,, Y l , . . . , Y,-,) = H(Z, I Zo, . . . ,Zn-,) = H(Z,) = a constant H for all n. But

.

limn-lH(Y,,. n--. m

. . , Y, I X I , . . . ,X,)

=

lim n-'H = 0 # H n--. m

in general. However, if we make an assumption of "nonanticipatory" behavior, that is, H ( Y i I X,, . . . , X,, Y,, . . . , Y,-,) = H(YiI X I , . . . , Xi, Y l , . . . , YiJ for all n, and all i = 1 , 2 , . . . ,n, then the argument of Problem 7.2 will go through intact.

References Abrarnson, N. M. (1959), A Class of Systemat~cCodes for Non-Independent Errors, I R E Trans Inform. Theory, IT-5, 156157. Abramson, N. M. (1961), Error-Correct~ngCodes from Lmear Sequentla1 Networks, Proc , Fourth London Symposrum on Informatron Theory, C. Cherry, Ed., Butterworths, Wash~ngton,D C Abrarnson, N. M. (1963), Informatron Theory and Codmg, McGraw-HIU Book Co., New York. Albert, A A. (1956), Fundamental Concepts of Hrgher Algebra, Unw. of Ch~cagoPress, Ch~cago,111. Ash, R B. (1963). Capac~tyand Error Bounds for a Tme-Contmuous Gauss~anChannel, Informatron and Control, 6 , 14-27. Ash, R. B (1964), Further Drscuss~onof a Tlme-Contmuous Gausslan Channel, Informatron and Control, 7, 78-83. Ash, R. B. (1965), A Smple Example of a Channel for Wh~chthe Strong Converse Falls, I E E E Trans. Inform. Theory, In press. Concernant Certalnes Funct~onsAleatoms en Bethoux, P. (1962), Test et Est~rnat~ons Partlcuher Laplac~ennes,Ann. Inst. Henrr Porncard, 27, 255-322. Blrkhoff, G., and S MacLane (1953), A Survey of Modern Algebra, The Macm~llanCo., New York Blackwell, D (1961a), Informat~onTheory, In Modern Mathematrcs for the Engrneer, Second Series, E F. Beckenbach, Ed., McGraw-H111Book Co., New York. Blackwell, D. (1961b), Exponentla1 Error Bounds for Flnlte State Channels, Proc. Fourth Berkeley Symposrum on Math. Statrstrcs and Probabrlrty, Unlv. of Callforn~a Press, Berkeley, Cahf , 1, 57-63. Blackwell, D., L Brelman, and A. J. Thomas~an(1958), Proof of Shannon's Transrnrssron Theorem for Fmte-State Indecomposable Channels, Ann. Math. Stat.. 29, NO 4, 1209-1220 Blackwell, D , L. Brelrnan, and A J ThomasIan (1959), The Capac~tyof a Class of Channels, Ann. Math S t a t , 30, No. 4, 1229-1241. Bose, R. C., and D K Ray-Chaudhur~(1960a), On a Class of Error Correct~ngB~nary Group Codes, Informatron and Control, 3, 68-79. Bose, R C., and D K Ray-Chaudhur~(1960b), Further Results on Error Correct~ng Bmary Group Codes, Informatron and Control, 3,279-290. Breman, L. (1957), The Ind~vldualErgod~cTheorem of Informat~onTheory, Ann. Math. S t a t , 28, No. 3, 809-811; correction to thls paper, 1960, Ann. Math. Stat., 31, NO 3, 809-810 Chernoff, H (1952), A Measure of Asymptot~cEfficiency for Tests of a Hypothes~s Based on the Sum of Observat~ons,Ann. Math. Stat., 23,493-507. Chung, K L (1960), Markov Churns wrth Statronary Transrrron Probabrlrtres, SpringerVerlag, Berlm. Dobrushm, R L (1959), General Formulat~onof Shannon's Mam Theorem In Informat~onTheory, Usp. Math. Nauk, 14, No 6(90), 3-104, translated m Am. Math. Soc Translatrons, 33, Senes 2, 323-438 Doob, J L (1953), Srochasrrc Processes, John Wlley and Sons, New York.

332

INFORMATION THEORY

Elsenberg, E. (1963), On Channel Capac~ty,Technrcal Memorandum M-35, Electron~cs Research Laboratory, Un~v.of Cal~forn~a, Berkeley, CalL Ehas, P. (1955), Cod~ngfor NOISYChannels, IRE Conoentron Record, Part 4, pp 3746. Ellas, P. (1956), Cod~ngfor Two Nolsy Channels, In Informatron Theory, Colm Cherry, Ed., Academ~cPress, New York, pp. 61-74. Elspas, B. (1959), The Theory of Autonomous Lmear Sequential Networks, IRE Trans. Crrcurt Theory, CT-6, 45-60. Fadlev, D. A. (1956), On the Not~onof Entropy of a F m t e Probab~l~ty Space (In Russ~an),Usp. Math. Nauk, 11, No. 1 (67), 227-231. Fano, R. M. (1961), Transmrssron of Informatron, MIT Press, Carnbndge, Mass Felnstem, A. (1954), A New Baslc Theorem of Informat~onTheory, IRE Trans. PGIT, 2-22, Sept. Fe~nste~n, A. (1958), Foundatrons of Informatron Theory, McGraw-H111 Book Co , New York. Felnstem, A. (1959), On the Cod~ngTheorem and ~ t sConverse for F~n~te-Memory Channels, Informatron and Control, 2 , No. 1 , 2 5 4 . Feller, W. (1950), lntroductron to Probabrbty Theory, John Wdey and Sons, New York F ~ r eP. , (1959), A Class of Mult~pleError Correct~ngBmary Codes for Non-Independent Errors, Sylvanla Electr~cProducts, Mountam h e w , Cahf, Report RSL-E-2, March for Laplaclan Funct~ons,Fourth Fortet, R (1961), Hypothes~sTest~ngand Est~mat~on Berkeley Symposrum on Mathematrcal Statrsrrcs and Probabrlrty, 1 , 289-305, Un~v.of Cahfornla Press, Berkeley, Calf. IRE Tram C~rcurtTheory, Fnedland, B. (1959), Lmear Modular Sequentlal C~rcu~ts, CT6,61-68. Fnedland, B , and T E. Stern (1959), On P e r ~ o d ~ c ~ oft yStates In Lmear Modular Sequentla1 C~rcu~ts, IRE Trans. Igorm. Theory, IT-5, 136137. Gallager, R. G. (1963), Low Densrty Parrty Check Codes, MIT Press, Carnbndge, Mass. Gantrnacher, F. R. (1959), Applrcatrons of the Theory ofMatrrces,Intersc~encePubl~shers, New York. Alphabets, Bell System Tech J . , G~lbert,E. N. (1952), A Compar~sonof S~gnall~ng 31,504522. G~lbert,E. N. (1960), Capac~tyof a Burst No~seChannel, Ben System Tech. J , 39, 1253-1265. Goldberg, R. R. (1961), Fourrer Transforms, Cambndge Un~v.Press, London and New York. Halmos, P. R. (1950), Measure Theory, D. Van Nostrand Co., Princeton, N.J. Hamm~ng,R. W. (1950), Error Detectmg and Error Correct~ngCodes, BeNSystem Tech. J. 29, 147-160. Hocquenghem, A. (1959), Codes Correcteurs D'erreurs, Chrffres, 2, 147-156. Huffman, D. A. (1952), A Method for the Construct~onof Mln~mumRedundancy Codes, Proc. IRE, 40, No. 10, 1098-1 101 Karush, J. (1961), A Slmple Proof of an Inequality of McMlllan, IRE Trans. Inform. Theory, IT-7, No. 2, 118 Katz, M., and A. J. ThomasIan (1961), A Bound for the Law of Large Numbers for D~screteMarkov Processes, Ann. Math Stat., 32, No. 1, 336337. Kelly, E. J., I. S. Reed, and W. L. Root (1960), The Detect~onof Radar Echoes In No~se, J. SOC.Ind. Appl. Math., 8, 309-341.

REFERENCES

333

Kemeny, J. G., and J. L. Snell (1960), Frnrte Markov Chains, D. Van Nostrand Co., Prlnceton, N.J. Khlnchln, A. (1957), Mathematrcal Foundatrons of Information Theory, Dover Pubhcatlons, New York. Kraft, L. G. (1949), A Device for Quanrrzmng,Grouprng and Coding Amplitude Modulated Pulses, M. S. Thesis, Electrical Englneenng Dept., MIT. Landau, H J., and H. 0. Pollak (1961), Prolate Spheroldal Wave Functlons, Fourler Analysrs, and Uncertainty-11, Bell System Tech. J., 40, 65-84. Landau, H. J., and H. 0. Pollak (1962), Prolate Spheroldal Wave Functlons, Founer Analysis, and Uncertainty-111, Bell System Tech. J., 41, 1295-1336 Lee, P. M. (1964), On the Ax~ornsof Informatlon Theory, Ann. Math. Stat., 35,415-418. Ltusternlk, L. A. and V. J. Sobolev (1961), Elements of Functional Analysrs, Frederlck Ungar Publlshmg Co., New York. Ldve, M (1955), Probabrlrty Theory, D Van Nostrand Co , Pnnceton, N J McMlllan, B. (1953), The Baslc Theorems of Inforrnatlon Theory, Ann. Math. Stat., 24, No 2, 196-219. McMlllan, B. (1956), Two Inequalltles Implled by Unlque Declpherablhty, IRE Trans. Inform. Theory, IT-2, 1 15-1 16. Melas, C. M. (1960), A New Group of Codes for Correctlon of Dependent Errors ~n Data Transm~sslon,IBM J. Res. Development, 4, 58-65. Moore, E. F. (1956), Gedanken Experiments on Sequentla1 Machines, In Automata Studres, Ann. Math. Studres, 34, Prlnceton Unlv., Pnnceton, N.J. Muroga, S. (1953), On the Capaclty of a D~screteChannel, J. Phys. Soc. Japan, 8 , 484-494. Peterson, W. W. (1960). Encodlng and Error-Correction Procedures for the BoseChaudhurl Codes, IRE Trans. Inform. Theory, IT-6,459470. Peterson, W. W. (1961), Error Correctrng Codes, MIT Press, Cambridge, Mass. Rlesz, F., and B. Sz.-Nagy (1955), Functronal Analysu, Frederlck Ungar Publishing Co., New York. Root, W. L., and T. S Pltcher (1955), Some Remarks on S p a 1 Detection, IRE Trans. Inform Theory, IT-1, No. 3, 33-38. Sacks, G . E. (1958), Multlple Error Correctlon by Means of Panty Checks, IRE Trans. Inform Theory, IT-4, 145-147. Sardmas, A. A., and G. W. Patterson (1950), A Necessary and Sufficlent Condltlon for Un~queDecomposlt~on of Coded Messages, Research Drvrsron Report 50-27, Moore School of Electrical Eng~neenng,University of Pennsylvania, Phlladelphla, Pa. Sardmas, A A. and G. W. Patterson (1953), A Necessary and Sufficlent Condlt~onfor the Unlque Decomposltlon of Coded Messages, IRE Conventron Record, Part 8, 104-108. Shannon, C. E. (1948), A Mathematlcal Theory of Cornmun~cation,Bell System Tech. J., 27, 379423, 623-656. Reprmted In C. E. Shannon and W. Weaver, The Marhematrcal Theory of Communrcatron, Unw. of Illlnols Press, Urbana, Ill., 1949 Shannon, C. E. (1957), Certaln Results In Codlng Theory for Nosy Channels, Informatron and Control, 1, No. 1, 6-25. Shannon, C. E. (1959), Probabll~tyof Error for Optlmal Codes In a Gausslan Channel, Bell System Tech. 5 , 38, No. 3, 611-656. Sleplan, D (1956). A Class of Bmary Slgnallng Alphabets, Bell System Tech. J., 35, 203-234.

334

INFORMATION THEORY

Sleplan, D. (1965), Permutation Modulation, Proc. IEEE, 53, No. 3, 228-236. Sleplan, D., and H. 0. Pollak (1961), Prolate Spheroidal Wave Functions, Fourler Analys~s,and Uncertainty-I, BeN System Tech. J., 40,43-64. Taylor, A. E. (1958), Introductron to Functronal Analysrs, John Wlley and Sons, New York. Thomaslan, A. J. (1960), An Elementary Proof of the AEP of Information Theory, Ann. Marh. Stat., 31, 452-456. Thomaslan, A. J. (1961), Error Bounds for Contmuous Channels, In Fourth London Symposrum on Informatron Theory, C. Cherry, Ed., Butterworths, Washrngton, D.C., pp. 4660. Thomaslan, A. J. (1963), A F m t e Crlterlon for Indecomposable Channels, Ann. Marh Stat., 34, No. 1, 337-338. T~tchmarsh,E. C. (1937), Introductron to the Theory of Fourrer Integrals, Oxford Unlv. Press, London. Varsharmov, R. R. (1957), Estlmate of the Number of Signals m Error Correcting Codes, Dokl Akad. Nauk SSSR, 117, No. 5, 739-741. Welss, L. (1960), On the Strong Converse of the Codlng Theorem for Symmetric Channels w~thoutMemory, Quart. Appl. Math , 18, No. 3, 209-214. Wlener, N (1949), The Extrapolatron, Interpolatron, and Smoothrng of Statronary Trme Series, John Wlley and Sons, New York. Wolfowltz, J (1960), Simultaneous Channels, Arch. Ratronal Mech. Anal, 4, No. 4, 371-386. Wolfowltz, J. (1961), Codrng Theorems of Informatron Theory, Prentlce-Hall, Englewood Chffs, N.J. Wolfowltz, J. (1963a), On Channels wlthout Capacity, Informatron and Control, 6, 49-54. Wolfowltz, J. (1963b). The Capac~tyof an Indecomposable Channel, Sankhya, 25, Serles A, 101-108. Wolfowltz, J. (1964), Codrng Theorems of Informatron Theory, second edlt~on,SprmgerVerlag, New York. Wozencraft, J M., and B. Relffen (1961), Sequentral Decodrng, MIT Press, Cambridge, Mass. Wyner, A (1964), Improved Bounds for Mlnlmum Dlstance and Error Probabdrty In Dlscrete Channels, Bell Telephone Laboratones Internal Report, Murray H111, N.J. Wyner, A. (1965), Capabllltles of Bounded Discrepancy Decoding, Bell System Tech. J., 44, pp. 1061-1122. Yoshlhara, K (1964), Slmple proof of the strong converse theorems In some channels, Kodar Marh. Sem. Rep., 16, 213-222.

INDEX Abramson code, 166 Algebra, of b ~ n a r yp o l y n o m d s , 147 Alphabet, of a channel, 46, 215 of d code, 27 of d source, 172, 185 A m b ~ g u o u ssequence, 30 A r ~ t h m e t ~and c geometric means, Inequal~ t yof, 25 Arzela-Ascol~theorem, 270 Asymptot~cequlpartltlon property (AEP), 197, 223

Channel@), cascade c o m b m a t ~ o n of 85 code for, see Code compound, 228 connection of a source to, 214, 216 continuous, 230 IT determlnlst~c,5 1 d~screte,47 memoryless, 47 wlth finlte memory, 227 with memory, 21 1 IT f i n ~ t estate, 215 ff Band-l~rn~tcd funct~ons,256 fT regular, 216 Bessel's lnequal~ty,269 mformatron processed by, 50, 219 Bmorn~ald ~ s t r ~ b u t ~ 3, o n 14, , 15 Input n-sequence for, 65 estlmate of "tall" of, 114 lossless, 50 B ~ t s ,1 I, 84 m a t r ~ xof, 49, 212-215 Base-Chdudhur~Hocquenghem codes, source-channel matrlx, 215-217 156 ff no~seless,27, 51 output n-sequence for, 65 C a p a c ~ t yof a chdnnel, 1, 3, 50 product of, 85 ~ d k ~ l d t l 0Of, n 53 ff states of, 46, 215, 230 c d p d ~ ~ t 224 y, sum of, 85 of a brnary s y m m e t r x chdnnel, 54 symmetrlc, 52 o f d d~scretememoryless channel, 50 t~me-contmuousG a u s s ~ a n 251 , of a fin~testate regular channel, 220, 228 t m e - d ~ s c r e t e ampl~tude-contmuous, , 23C o f a general dlscrete channel, 223 tlme-d~screteG a u s s ~ a n 23 , 1 of a symmetrlc channel, 53 with average power l ~ m ~ t a t ~ 23 o n1, of a tlme-cont~nuousG a u s s ~ a nchannel, trap door, 21 1 251, 252 useless (zero-capac~ty),52 of a t ~ m e - d ~ s c r e tampl~tude-contmuous e, Chebyshev's ~ n e q u a l ~ t y15, channel, 234 Code, 4 of a t ~ m e - d ~ s c r e G t ea u s s ~ a nchannel, 234 Abramson, 166 Cascdde ~ o m b r n a t r o n ,of d ~ s c r e t ememoryalphabet, 27 less channels, 85 block, 4, 39, 63 Cdyley-Hdm~ltontheorem, 139 Base-Chaudhurl-Hocquenghem, 156 Chdnnel(s), 1 characters, 27 alphabet of. 46, 215 check d ~ g l t sof, 93 b d n d - l ~ m ~ t e d256 . fT close-packed, 132 blndry erasure, 86 cychc, see Cychc code blndry syrnmetrlc, 53, 54, 87 ff error correcting, 87 ff burst nolse, 213 Flre, 167 cdpaclty of, see C a p a c ~ t y for a bmary symmetric channel, 87 ff 335

336

INDEX

Code, for a discrete memoryless channel, 65, 66 for a finite state regular channel, 220 for a general discrete channel, 223 for a noiseless channel, 27 for a time-continuous Gaussian channel, 25 1 explicit construction of, 256 for a time-discrete, amplitude continuous channel, 230 geometric interpretation of, 90 group, 93, 99 coset of, 100 Huffman, 42 information digits of, 93 instantaneous, see Instantaneous code lossless, 132 (n, k), 99 nonbinary, 126 optimal, absolutely, 38 instantaneous, 41 parity check, see Parity check code random, 66, 67, 74, I10 single-error correcting, 105 cyclic, 161 uniquely decipherable, 28, 29, 35, 40 word, 4, 27, 63, 65 Coding theorem, see Fundamental theorem of information theory Communication entropy, 24 Communication system, I Conditional uncertainty, 19, 219, 229, 238, 24 1 Convexity, of information and uncertainty, 54, 81, 242 of the logarithm, 17 Corrector, 94 Correlation detector, 254 Coset, of a group code, 100 relation to correctible error patterns, 102, 103 Coset leader ( = correctihle error pattern), 103 Covariance function, of a random process, 250, 256, 275 ff. Critical rate, for the binary symmetric channel, 117 Cycle set, of a binary matrix, 136, 143, 144 of a feedback shift register, 136 Cyclic code, 137 application to burst-error correction, 164

Cyclic code, decoding of, 161 generation by a feedback shift register, 137, 151 generator polynomial of, 148 properties of, 147 ff. set of generators for, 149 single-error correcting, 161 Decision scheme, 1, 4, 60, 65, 220, 230, 25 1 ideal observer, see Ideal observer dccision scheme maximum likelihood. 62 minimum distance, see Distance randomized, 84 with uniform error bound, 62, 66 Decoder, see Decision scheme Decoding sets, 65 Dini's theorem, 274 Distance (between code words), 87 ff., 126 minimum, bounds on, 129 minimum distance decoder, 88, 126, 254 for a parity check code, 95 minimum distance principle, 87, 88 relation to error correction. 89 Effective procedure, for determining existence of steady state probabilities in a finite Markov chain, 180, 2111 for determining whether a finite state channel is regular, 218 Eigenvalues, eigenfunctions, and eigenvectors, 210, 250, 264 Encoder, 1, 4 Entropy, 24 Ergodic information source, see Source Error bounds (exponential), 77 8.. 227, 260 for general binary codes, 113 8. for nonbinary codes, 127 Error patterns, 95 correctible, relation to cosets, 102, 103 Fano's inequality, 80, 243 Feedback shift register, 134 ff. characteristic matrix of, 135 cycle set of, 136 generation of a cyclic code by, 137, 151 state (content) of, 135 Field, 126, 142 Galois (finite), 142 Fire code, 167

INDEX

Fourier transforms, 25 1, 257, 258, 28 1 8 . convolution, 282 Parseval relation, 258, 282 Fundamental lemma (in the proof of the fundamental theorem), 68, 86, 232 Fundamental theorem of information theory, 1 for the discrete memoryless channel, 66, 307 algorithmic proof, 68 random coding proof, 74 Shannon's original proof, 66 for the finite state regular channel, 222 for the general discrete channel, 223 for the time-continuous Gaussian channel, 252 for the time-discrete Gaussian channel, 234 strong converse to, see Strong converse weak converse to, see Weak converse Generator matrix, of a cyclic code, 149 of a panty check code, 132 Generator polynomial (of a cyclic code), I48 relation t o the minimal polynomial of a matrix, 150 Generators, set of, for a cyclic code, 149 for a parity check code, 110 Greatest common divisor, of two polynomials, 141 Group, 96 abelian (commutative), 96, 141, 147 Group code, 93, 99 coset of, 100 Grouping axiom, for the uncertainty function. 8, 80. 81 generalized, 26 Hamming bound, for general bmary codes, 90. 133 for panty check codes, 107 strengthened, 127 Hamming code, 130, 133 Hdmmlng d~stance,see Dlstance H~lbertspace, 250, 262 ff Bessel's ~ n e q u a l ~ t y269 , mner product, 262 norm, 262 orthogonal complement of a subspace, 268

337

Hilbert space, orthonormal family, 266 parallelogram law, 265 space spanned by a subset, 267 Huffman code, 42 Ideal, in the algebra of binary polynomials, 149 Ideal observer decision scheme, 61, 73 for the binary symmetric channel, 88 for the q-ary symmetric channel, 126 for the time-discrete Gaussian channel, 254 Independent random variables, 14, 39, 195 as a Markov information source, 185 uncertainty of, 18, 19, 21, 239, 241, 242 Information, 4 convexity of, 54, 242 conveyed about o n e random variable by another, 22, 23, 239, 242 conveyed about one stationary sequence of random variables by another, 219 processed by a channel, 50, 219 source of, see Source Instantaneous code, 28, 39 conditions for existence of, 33 optimal, 41 relation t o sequence of "yes o r no" questions, 40 Integral operator, 269 ff. associated with a covariance function, 250, 282 ff. Integration, of second order random processes, 275 ff. Irreducible polynomial, 140, 146 Joint uncertainty, 18-21, 238-240 Karhunen-Lobve theorem, 250, 277 for Brownian motion, 279 for Gaussian processes, 279

LZ space, 250, 257, 262 ff. Language, uncertainty of, 206 Linear operator (on a Hilbert space), 262 ff. compact (completely continuous), 263 continuous (bounded), 262 integral operator, 269 ff. associated with a covariance function, 250, 282 ff. symmetric (self-adjoint), 265

338

INDEX

Linear operator (on a Hilbert space), bilinear form of, 265 quadratic form of, 265

Nonnegative definite function, 272

Markov chain(s) (finite), 171 essential sets of, 180, 183 indecomposable, 18 1, 184 periodic and aperiodic states of, 183, 184 regular, 185 states of, 171 stationary distribution of, 174, 181, 184 steady state probabilities of, 174, 176 effective determination of existence of, 180, 218 theory of, 172 ff. transient states of, 180, 183 transition matrix (Markov matrix, stochastic matrix) of, 171 Markov information source, see Source Markov property, 171 Matrix, 138 ff. binary, 138 characteristic polynomial of, 138 cycle set of, 136, 143, 144 minimal polynomial of, see Minimal polynomial period of, 143 polynomial in, 140 powers of, and matric polynomials, 145 with maximal period, 144 channel, 49, 212-215 connection, of a unifilar Markov source, 209 doubly stochastic, 26 Markov (stochastic, transition), 171 parity check. 93, 106 source-channel, 21 5-2 17 Maximal period, 144 Maximum likelihood decision scheme, 62 Mercer's theorem, 272 Message, 1, 27, 63 Minimal polynomial of a matrix, 139 irreducible, 140, 141, 143, 144 relation to generator polynomial of a cy. clic code, 150 Minimum distance decoder, see Distance

Parity check code(s). 91 ff., 126 adequacy of, 110 ff. bounds on error-correcting ability of, 105 K. generator matrix of, 132 set of generators for. 110 Parity check matrix, 93, 106 Period of a matrix, 143 maximal, 144 *-sequence, 24 Polynomial(s), algebra of, 147 characteristic, 138 generator, see Generator polynomial in a binary matrix, 140 irreducible, 140, 146 minimal, see Minimal polynomial representation of, by a binary sequence, 147 Prefix, 28 Probability of error, 65 average, 65. 73 exponential bounds on, see Error bounds for a group code, 104 for general binary codes, 113 8. maximum, 66

Noise, 1 Noiseless coding, 27 ff. theorem, 13, 37

Order of a unifilar Markov source, 189 ff.

Random coding, 66, 67, 74, l I0 Random process, 275 ff. covariance function of, 250, 256, 275 K. Gaussian, 250, 256, 279 second order. 250. 275 If. spectral density of, 250, 256, 282 stationary, 185 Random variables(s), conditionally independent, 25 Gaussian, 231 uncertainty of, 240 independent, see Independent random variables noiseless coding problem for, 27 uncertainty of, 5 K. Random vectors, 19, 25, 39, 240, 243 Rate of transmission, 1, 3, 63 critical, for the binary symmetric channel, 117 h -permissible, 224 permiss~ble.223, 234. 251

INDEX

339

Uncertainty, 4, 8 Sampling theorem, 258 average, 5 Schwarz inequality, 78, 255, 262 axioms for, 5 ff., 24, 26 (Problem 1.12) Sequence, input, 65 grouping axiom, 8, 80, 81: generalized, "meaningful," produced by a n informa26 tion source, 195, 196, 206 conditional, 19, 219, 229, 238, 241 output, 65 typical, 14, 24, 83. 196 convexity of, 54, 81 input and output, 50 in Shannon's original proof of the funinterpretations of, 12 IT. damental theorem, 66 joint, 18-21, 238-240 Sequential circuit, linear. 163 maximization of, 17 Shannon-McMillan theorem, see Asymptotic equipartition property for a unifilar Markov source, 209 n-gram, unigram, digram, a n d trigram, Shift register, see Feedback shift register Source of information, 1, 63, 169 ff., 184 191 alphabet of, 172, 185 of a discrete random variable, 5 ff. approximation of, by a source of finite of a discrete random variable given a n absolutely continuous random variorder, 189 ff. asymptotic equipartition property, 197, able, a n d vice versa, 241 223 of a function of a random variable, 26 ergodic, 197, 202, 207, 208, 223 of a Gaussian random variable, 240 Markov, 172, 185 of a language, 206 indecomposable, 185 of a n absolutely continuous random variregular, 185, 202, 223 able, 236 unifilar, 187: connection matrix of, of a n information source, 186, 219 209; maximum uncertainty of, 209; of a unifilar Markov source, 188 order of, 189 ff.; uncertainty of, 188 properties of, 16 ff. uncertamty of, 186, 219 Unifilar Markov source, see Source Source-channel matrix, 215-217 Uniform error bound, 62, 66 Spectral density, of a random process, 250, Varsharmov-Gilbert-Sacks condition, 108, 256, 282 122, 130, 131, 163, 225 States, of a channel, 46, 215, 230 of a Markov chain, 171 Vector space, 126, 147 Stationary distribution, of a finite Markov chain, 174, 181, 184 Weak converse t o the fundamental theoStationary Gaussian random process. 250 rem, for the discrete memorvless Stationary sequence of random variables, 185 channel, 82, 307 Steady state probabilities of a finite Markov for the finite state regular channel, 223 chain, 174, 176 for the time-continuous Gaussian chaneffective determination of existence of, nel, 252 180. 208 for the time-discrete Gaussian channel, Stirlrng's formula, 113 234, 245 Storage requirements, of a decoder, 92, 161 Weak law of large numbers, 3, 130 Strong converse t o the fundamental theoexponential convergence in, 83 rem, 83, 223, 224 for regular Markov chains, 203 failure of, 225 Weight, of a binary sequence, 102 for the binary symmetric channel, 124 for the time-discrete Gaussian channel, "Yes o r no" questions, 13, 14 246 relation to instantaneous binary codes, Syndrome (=corrector), 94 40