INFORMATION THEORY FOR CONTINUOUS SYSTEMS
INFORMATION THEORY FOR CONTINUOUS SYSTEMS
Shunsuke lhara Department of Mathematics College of General Education Nagoya University
V ^ g World Scientific I T
Singapore
* New Jersey
• London-Hong
Kong
Published by
World Scientific Publishing Co. Pie. Ltd. POBo* 128, Farter Road, Singapore 912S USA office: Suite IB, 1060 Main Street. River Edge, NJ 07661 UK office: 73 Lynton Mead, Totteridge, London N20 SDH
Library of Congress Cataloging-! n-Publicalion Data
Information theory for continuous systems / author, Shunsuke Diaia. p. cm. Includes bib1io|raphical references and index. ISBN 98102O985L I. Entropy (Information theory)-Mathematical models. 2. Neural trans mission-Mathematical models. I. Ihaia, Shunsuke. Q370.156 1993 003\54~dc20 93-23179 C1P
Copyright © 1993 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This bode, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording orany information storage and retrieval system now kiunvn or to be invented, without written permission from the Pubiiiher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Ciearance Center, Inc., 27 Congress Street, Salem, MA 01970. USA.
Printed in Singapore by Utopia Press.
Acknowledgments I would like to express my gratitude to Professor Takeyuki Hida for introducing me to this important research area, and for his valuable suggestions and encouragement, w i t h o u t which I could not have completed the present work. Among the many others who have made contributions, I would particularly like to thank Professors Tunekiti Sirao, Hisao Nomoto, Nobuyuki Ikeda, Hiroshi Kunita, Izumi Kubo, Masuyuki Hitsuda, Charles R. Baker and Kenjiro Yanagi, for valuable comments.
Preface This book is intended to provide a comprehensive and up-to-date treatment of information theory with special emphasis on the study of continuous information transmission systems, rather than discrete ones. Information theory in the strictest sense was largely originated by C.E. Shannon. In the paper t i t l e d "A mathematical theory of communication" [103], published in 1948, Shannon defined such fundamental quantities as entropy and mutual information, introduced general communication systems models, and proved coding theorems. Since then Shannon's basic idea and results have been extended and generalized by information theorists, both in mathematics and engineering. There are two large trends i n the development of information theory. The first one is the analysis of information theoretic quantities like entropy and mutual information. The concept of entropy was proposed, i n information theory, as a measure of uncertainty or randomness of a random variable, in other words, as a measure of the information involved i n the random variable. A generalization of the idea of entropy called relative entropy was introduced by S. Kullback {see [83]). This form of information measure, which is also referred to as information divergence or Kullback-Leibler information number, can be interpreted as a measure of similarity between two probability distributions. Many results for entropy and mutual information can be viewed as particular ones for relative entropy.
The
second trend is a generalization of coding theorems for more general and more complicated information transmission systems. The field of information theory has grown considerably.
Now modern infor-
mation theory deals w i t h various problems of information transmission, based on probability theory and other mathematical analysis, and is more than a mathematical theory of communication. I t is tied with estimation theory, filtering theory, and decision theory more closely than ever. vii
viii
PREFACE
Information theory is based on mathematics, especially on probability theory and mathematical statistics. Information transmission is, more or less, disturbed by noises which are arisen from a number of causes. Mathematically, the noises can be represented by suitable random variables or stochastic processes. Moreover, the significant aspect i n communication theory is that the actual message or signal to be transmitted is the one selected from a set of possible messages or signals. The system must be designed to operate for every possible selection, not just the one which will be actually chosen, since this is unknown at the time of design. The sets of possible messages and signals are given w i t h probability distributions. Thus messages and signals are also considered to be represented by random variables or stochastic processes. This indicates why the stochastic approach is quite useful for the development of information theory. A t the same time, information theory has fundamentally contributed not only to communication theory but also to statistical mechanics, probability theory, and statistics. The contents of this book fall roughly into two parts. I n the first part, entropy, mutual information and relative entropy are systematically analyzed and a unified treatment of these quantities in information theory, probability theory and m a t h ematical statistics is presented. Links between information theory and topics t h a t come from probability theory or mathematical statistics are frequently expressed in terms of relative entropy. It is shown that relative entropy is of central i m portance i n large deviation theorems, hypothesis testing, and m a x i m u m entropy methods i n statistical inference. The second part deals mostly w i t h information theory for continuous information transmission systems, based on the most recently available methods i n probability theory, and analysis developed i n the first part. The Gaussian channel is a communication channel disturbed by an additive noise w i t h Gaussian distribution. I t is known, as the central limit theorem, that a noise caused by a large number of small random effects has a probability distribution that is approximately Gaussian. This indicates that the study of Gaussian channels is especially i m p o r t a n t not only from the theoretical point of view but also from the viewpoint of application. One of the most interesting and complicated information transmission systems is a system w i t h feedback. We can handle feedback systems on the basis of causal calculus and innovation approach which have been developed i n probability theory. There are many excellent books on information theory.
While most of them
deal w i t h discrete information transmission systems, few of them treat continuous (both i n state spaces and time parameter spaces) systems. One of the major aims of this book is to cover the most recent developments i n information theory for continuous communication systems including Gaussian channels.
ix
PREFACE
Topics on information theory and stochastic processes are dealt w i t h i n moderately rigorous manner from the mathematical point of view. A t the same time, much effort is p u t into a rapid approach to the essentials of the matter treated and the maintenance of a spirit favorable to application, avoiding extremes of generality or abstraction. Most of the results i n this book are given as theorems and proofs. It is not necessary to understand the proofs completely to have a general grasp of the subject. The proofs can be skipped on the first reading when desired. They can be studied more carefully on the second reading. It is my hope that this book is accessible to engineers who are interested in the mathematical aspects and general models of the theory of information transmission and mathematicians who are interested i n probability theory, mathematical statistics and their applications to information theory. In Chapter 1, we introduce the fundamental quantities of information theory - entropy, relative entropy and mutual information - and study the relationships and a few interpretations of them.
They play important roles throughout the
text. These quantities are defined for stochastic processes i n Chapter 2. Especially stationary processes are studied from the information theoretical point of view. Chapter 3 is devoted to the study of large deviation theorems, maximum entropy or the m i n i m u m relative entropy spectral analysis, and hypothesis testing. Relative entropy plays fundamental roles i n the investigation of these topics. A large deviation theorem, which is a limit theorem in probability theory, shows that the speed of convergence is described i n terms of relative entropy. A model selection that maximizes the uncertainty or entropy yields the maximum entropy approach to statistical inference, and the minimum relative entropy method can be viewed as a generalization of the maximum entropy method. Interpretations of basic concepts i n communication theory, such as channel capacity and achievable rate of information transmission, are given in Chapter 4. Fundamental problems on information transmission over a communication channel are also investigated i n Chapter 4. The last two chapters are devoted to a study of information transmission over communication channels w i t h additive noises, especially Gaussian channels w i t h or without feedback. Such subjects as mutual information, channel capacity, optimal coding schemes, and coding theorems are investigated i n detail.
Chapter 5 is for the discrete time case.
I n particular.
Chapter 6 on the continuous time Gaussian channels is unique in this book. Some terminologies and results, which are used i n the text, i n probability theory are prepared i n the Appendix.
Contents
Preface
vii
C h a p t e r 1. E n t r o p y
1
1.1
Information Transmission
1
1.2
Entropy
4
1.3
Entropy for Continuous Distributions
13
1.4 1.5
Relative Entropy Properties of Relative Entropy
20 29
1.6
M u t u a l Information
35
1.7 1.8
Rate-Distortion Function Entropy for Gaussian Distributions
41 47
Historical Notes
56
C h a p t e r 2. S t o c h a s t i c Processes a n d E n t r o p y
57
2.1
Entropy Rate
57
2.2
Discrete Time Stationary Processes
63
2.3
Discrete Time Markov Stationary Processes
69
2.4
Entropy Rate of a Stationary Gaussian Process
76
2.5
Continuous Time Stationary Processes
83
2.6
Band Limited Processes
90
2.7
Discrete Time Observation of Continuous Time Processes Historical Notes
93 96
xi
CONTENTS
C h a p t e r 3. M a x i m u m E n t r o p y A n a l y s i s 3.1
M a x i m u m Entropy and M i n i m u m Relative Entropy
98 98
3.2
Large Deviation Theorems
3.3
M a x i m u m Entropy Spectral Analysis
118
3.4
Hypothesis Testing
127
3.5
Hypothesis Testing for Stationary Gaussian Processes Historical Notes
C h a p t e r 4. T h e o r y o f I n f o r m a t i o n T r a n s m i s s i o n
110
134 143 144
4.1
Model of Communication Systems
4.2
Information Stability
144 148
4.3
Source Coding Theorems
153
4.4
Channel Capacity
158
4.5
Channel Coding Theorems
162
4.6
Fundamental Theorem in Information Transmission
171
Historical Notes C h a p t e r 5. D i s c r e t e T i m e G a u s s i a n C h a n n e l s
175 177
5.1
M u t u a l Information in Channels w i t h Additive Noise
177
5.2
Discrete Gaussian Channels
183
5.3
M u t u a l Information in Gaussian Systems
188
5.4
Capacity of Discrete White Gaussian Channels
195
5.5
Capacity of Discrete Gaussian Channels without Feedback
200
5.6
Optimal Codings i n Discrete W h i t e Gaussian Channels
207
5.7
Capacity of Discrete Gaussian Channels w i t h Feedback
210
5.8
Rate-Distortion Function of a Discrete Gaussian Message
220
5.9
Coding Theorem for Discrete Gaussian Channels
223
Historical Notes C h a p t e r 6. C o n t i n u o u s T i m e G a u s s i a n C h a n n e l s 6.1
226 228
Continuous Gaussian Channels
228
6.2
M u t u a l Information in White Gaussian Channels (I)
231
6.3
M u t u a l Information in White Gaussian Channels ( I I )
236
6.4
Capacity of White Gaussian Channels
250
6.5
O p t i m a l Codings i n White Gaussian Channels
254
6.6
Gaussian Channels without Feedback
258
CONTENTS
Sill
6.7
Stationary Gaussian Channels
263
6.8 6.9
Gaussian Channels w i t h Feedback Rate-Distortion FAinction of a Gaussian Process
268 277
6.10 Coding Theorem for Gaussian Channels Historical Notes Appendix. Preparation from Probability Theory
280 285 287
A.l
Probability and Random Variable
287
A.2
Conditional Probability and Conditional Expectation
289
A.3
Stochastic Integral
292
Bibliography
293
List of Symbols
300
Subject
304
Index
Chapter 1 Entropy 1.1 I N F O R M A T I O N
TRANSMISSION
A mathematical foundation of information theory was presented by C.E. Shannon w i t h his famous paper "A mathematical theory of communication", published in 1948. I n a strict sense, information theory may be understood as a mathematical theory of communication, or a theory of information transmission. As the field of information theory has grown and developed, we shall consider information theory here i n a somewhat broader sense. Communication theory deals w i t h systems that transmit information or data from one place to another. Physically speaking there are various types of communication systems.
Most of such systems can be described by the block diagram
in Figure 1.1, which is known as Shannon's scheme. A message from the information source is encoded at the encoder. The encoded signal is input into the channel. Then the received signal is decoded at the decoder. The fundamental problem of communication is that of reproduction of the message either exactly or approximately. I t is significant to note that the actual message is the one selected from a set of possible messages w i t h probabilistic structure. The system must be designed so as to be ready to operate for each possible selection, not just the one which will actually be chosen. It is essential to evaluate the performance of communication for all possible selections as a whole, not for the actually chosen one. The fundamental problem of information transmission w i l l be discussed i n Chapter 4 i n detail. It is essential i n information theory to measure the quantity of "information". 1
2
ENTROPY
For this purpose, Shannon introduced the concepts of entropy and m u t u a l information from the viewpoint of communication theory. Entropy is defined as a measure of "uncertainty" or "randomness" of a random phenomenon. Suppose t h a t some information about a random variable is received. Then a quantity of uncertainty is reduced, and this reduction in uncertainty can be regarded as the quantity of transmitted information, which is called the mutual information. The capacity of a communication channel is defined to be the maximum value of the information that can be transmitted over the channel. Before Shannon's paper, i t was generally thought that decreasing the required probability of error decreases the transmission rate of information over the channel. However Shannon has shown that i t is not true; rather, an arbitrarily small probability of error is achievable at any transmission rate less than the channel capacity.
Source
Encoder
Channel
Decoder
User
Noise F i g u r e 1.1. A communication system
Developing the idea of entropy, the concept of relative entropy was introduced by Kullback to discriminate two probability distributions and to apply information theory to statistical inference. This form of information measure is also referred to as information divergence, Kullback-Lei bier information number, or information gain. Relative entropy is better interpreted as a measure of "distance"
between
two probability distributions. Mathematically speaking, relative entropy is not a true metric, but i t enjoys some of the properties of a metric. Many results for entropy and mutual information can be viewed as special cases of results for relative entropy and the formula for relative entropy arises naturally i n some proofs. Three mathematical notions, entropy, mutual information, and relative entropy, introduced as measures of "information", have played significant roles in information theory. We note that these quantities, i n particular entropy and relative entropy, play important roles not only i n information theory but also i n relevant fields such as probability theory, mathematical statistics, ergodic theory, statistical mechanics, and so on. Entropy, relative entropy and mutual information play central roles throughout
I.J INFORMATION
3
TRANSMISSION
the text. I n this section we introduce them only briefly, and we shall study each of them i n detail i n later sections. Let X be a random variable taking values on a set {a ,a ] x
= Oi) = pi,
P(X
w i t h probability
m
t = l,...,m.
Then H(X)
n = - f > t a B A
= H(p ...,p ) u
m
(1.1.1)
•=i
is called the e n t r o p y o f t h e r a n d o m v a r i a b l e X or the e n t r o p y o f t h e p r o b ability d i s t r i b u t i o n (pi, ...,p ).
The H(X)
m
was named entropy by Shannon
because he recognized some analogies between the H{X)
and the entropy i n sta-
tistical mechanics, and the symbol H came from the H i n Boltzmann's famous H theorem. Let X and Y be two random variables w i t h probabilities P(X
=
P(X
= - a , , r = 6,) = r y ,
a
i
) =
P i
,
P{Y
Then the m u t u a l i n f o r m a t i o n I(X,
= b) = j
q, j
t = l , - , m , j = l,...,n.
Y) between X and Y is defined by n
m
It can be expressed I(X,Y)
= H{X)~H{X\Y),
(1.1.3)
where TI
The quantity H(X\Y)
m
is called the conditional entropy of X given Y.
I t can be
explained as the remained uncertainty of X after we are informed of the outcome of Y. Hence we can say that /(X, Y) — the reduction of the entropy of X due to the knowledge of Y. In other words, I(X, in Y.
Y) describes the amount of information about X contained
4
ENTROPY
Let p. = ( p j , . . . , p ) and 1/ = (qj,---,q ) be probability distributions. T h e n , the r e l a t i v e e n t r o p y H{y.;v) of p. w i t h respect to c is defined by m
m
We shall see that the entropy and the mutual information can be expressed as a relative entropy. Here we have defined relative entropy and mutual information only for discrete probability distributions. We shall repeat these definitions i n later sections where we deal w i t h probability distributions of more general types.
1.2 E N T R O P Y
This section is devoted to a study of various properties of entropy and of axiomatic characterization of entropy. Suppose that we have a set ( a , , a
m
} of possible states whose probabilities of
occurrence are given by ( p i , . . . , p ) . Then the random experiment is described by m
a random variable X with probability P{X
= a,)=p„
i = i
(1.2.1)
m.
The ( p i , . . . , p ) is the probability distribution of X. Throughout this book, we denote by P{A) the probability of an event A . Here we repeat the definition of entropy. m
D e f i n i t i o n 1.2.1. Let X be a random variable w i t h distribution ( p i , . . . , p ) . Then the e n t r o p y o f t h e r a n d o m v a r i a b l e X or the e n t r o p y o f t h e d i s t r i b u t i o n ( p i , . . . , p ) is defined by m
m
m
H(X)
= H{ ,..., ) Pl
Pm
= -£p,logp,.
(1.2.2)
1=1
Suppose that a probability distribution ( p i , . . . , p ) is known and t h a t we do not m
know which event will occur. Then the entropy H(pi,
. . . , p ) shows how much m
1.2
S
ENTROPY
freedom one is given i n the selection of an event, or how uncertain the outcome is, or how difficult to predict the outcome. The logarithm may be taken to the base e or to the base 2. I n the case of base e, the entropy is measured in units of n a t s , while in the case of base 2, the entropy is measured i n units of b i t s . We take the logarithm to the base e throughout the text unless otherwise stated. B i t is usually more convenient for the practical use, and nat is more convenient for theoretical developments.
We use the notational
convention 0 log 0 = 0. Quite often i n the present text, we simply denote
JVj
f ° H H i > XI^=i i f no confusion occurs. Some basic properties of the entropy are now i n order. r
T h e o r e m 1.2.1. (Properties of Entropy)
The following properties ( H . l ) - (H.7)
hold for a probability distribution ( p i , . . . , p ) . m
(H.l)
tf( ,..., )>G. Pl
The equality H{pj, ( i 10(H.2) H(p
. . . , p ) — 0 holds if and only 'if pi=
1 for some i and p , = 0
m
. . . , p ) is a continuous function of ( p i , . . . , p ) .
u
(H.3)
Pm
m
m
The entropy is symmetric: ff£Pi)
- >Pm)
=
# ( P O ( 1 ) , •">Pl(m))l
where a denotes a permutation of ( 1 , ...,m). (H.4)
Let { ( f l i j i ,
4n|>); i = 1,—,"»} be a set of probability distributions. Then
it holds that H(Pl
P m Qn\m)
9 l | Li - - - . P i 9 n | l . P 2 9 l | 2 . m
= i 7 { p i , . . . , p ) + ^2piH{qn,
?„(;)•
m
(H.5)
If p
m
= q + r > 0 and
tf(p,,...,p _i,0\r) m
(H.6)
r > 0, then = i f ( p , , . . . , p „ , p ) - r p / Y (—, — ] . \Pm Pm / m
1
m
m
Let ( j j i , ...,9m) be an arbitrary probability distribution. Then m
H(pi,...,Pm)
< -^Pilog?.. i=]
6
ENTROPY
w i t h equality i f and only if p,- = qi for a l l t = l , . . . , m . (H.7)
H(p,,...,p ) m
w i t h equality if and only if pj = • •• =;pm = j j j . P r o o f . Properties ( H . l ) , (H.2) and (H.3) are obvious by definition. (H.4)
Clearly ^f(pi9ili,
-,pig |i,P2gi|2,-.,pm?»|m) n
-5I
Jl'
P i 9
I o s
t
p ; 9
>i^
•j
= - ^ P . 9 > | i l o g P i - X^P.9>[i '°g?j|, m + ^Pii/(9i|i,-,g |i).
= H{Pi,-,p ) m
n
Thus we have (H.4). Property (H.5) is a special case of (H.4). (H.6)
To prove (H.6) we use the inequality l o g i < x - 1,
x > 0.
(1.2.3)
The equality holds i n (1,2.3) if and only if x — 1. We may assume that p, > 0 for all i . Using (1.2.3) we have m
ff(pi,...,p )
in
+ ^p,log , = ^p.log —
m
9
i=l
.=1
P
I
The equality holds i n (1.2.4) if and only i f p, —
m
= i
in (H.6).
•
We will explain the intuitive meanings of these properties which suggest to us that the entropy H(pi,
. . . , p ) is a suitable quantity as a measure of uncertainty m
of the outcome of a random phenomenon. It is reasonable to expect that the properties ( H . l ) - (H.3) are satisfied by the quantity to measure the uncertainty. The inequalities i n (H.6) and (H.7) are quite interesting and useful. Indeed, many of the basic properties of entropy follow from (H.6).
1.2
7
ENTROPY
Let X be a random variable satisfying (1.2.1) and Y be a random variable w i t h probability m
= kj) = ^
J = l,...,n.
(1.2.5)
Denote by qj
V
= P(Y
= bj\X
=
a)
the conditional probability of Y = b given X = a^, and by }
p =P{X
= o \Y =
tij
i
b) j
the conditional probability of X = a; given Y = bj. Then the joint probabilities are given by r , j = P(X
= a ,Y {
Thus the entropy H(X, B(X Y) t
= bj) = p i j j j i = qj y,
i - l,...,m, j = l,...,n.
Pi
(1.2.6)
Y ) of (X, Y ) is given by
fffru,
=
...,r ,r ,...,r „) ln
tt
m
= #(Pl?l|l> ••-,P]«n|l.P2?l|2, •;Pmq \m) n
(1.2.7)
Once we are informed that X = a; has occurred, while the uncertainty of X disappears, a reduced uncertainty H(qn„ ,.., g„|j) of y still remains. Therefore the second term on the right-hand side of (H.4) can be explained as the average value of the entropy of Y when X is known, in other words, the remained uncertainty of Y after X is known. Thus (H.4) explained that the uncertainty of (X, Y ) is equal to the sum of the uncertainty of X and the remained uncertainty of Y after X is known. We now introduce the concept of conditional entropy. D e f i n i t i o n 1.2.2. For a pair {X, Y ) of random variables with probabilities (1.2.1), (1.2.5) and (1.2.6), the c o n d i t i o n a l e n t r o p y H(X\Y)
r 7 m n = - I > '
Using (H.4), we see that H(X\Y) H(X\Y)
l
o
of X given Y is defined by
6 ^ -
(1-2-8)
has an alternative expression = H{X,Y)-
H(Y).
(1.2.9)
g
ENTROPY
The quantity m
H{X\Y
...,
= bj) = H( j, Pli
P H i
) - -^ntflogpiU
(1-2-10)
is called the conditional entropy of X given Y = bj. Then H(X\Y) tation of H(X\Y
is the expec-
= •): H{X\Y)
= j2liH(X\Y
(1.2.11)
= bj).
We collect some of the basic properties of conditional entropy. T h e o r e m 1.2.2. Let X,X\,X ,
Y be random variables w i t h discrete proba-
n
bility distributions. Then (H.8)
H(X\Y)
<
H(X)
and (H.9)
H(X,Y)
+
H[Y).
The equalities hold i n (H.8) and (H.9) i f and only i f X and Y are independent. More generally n
(H.10)
HiX^.^XAY)
•
= Y,H(X \Y,X k
X _,)
i f (JT»|K).
t
i
k=i
P r o o f . Applying (H.6) to H(X\Y
k=i
= b,) = H(p ,p |,) lU
and using T,j1jPi\j
m
=
Pi, we obtain n
H[X\Y)
< - £
>=] Thus we have (H.8).
/ m
\
\i=i
/
9,
}=
m
~ £ w
SP.
b
=
H(X).
i=i
The inequality (H.9) follows from (1.2.9) and (H.8).
By
(H.6), the equality holds in (H.8) (and consequently i n (H.9)) if p i y = p , for a l l i and j such that q, ^ 0. Clearly this condition is equivalent to the independence of X and Y. Using (1.2.9), we expand H{X\,X„, H(X ...,X ,Y) u
n
Y) in different ways to obtain = H(Y)
+
H(X ,...,X \Y) l
n
1.2
8
ENTROPY
and B(Xi>-,
X, n
Y)
= H(X,, =
ff{K)
Y) + H(X ,X |r, 2
4- H(X,\Y)
X,)
n
+
H(X ,...,X„\Y,X ). 2
1
Consequently H(X,
X \Y)
= H(X
n
}
\Y) +
H(X ,...,X \Y X ). 2
Repeating the same procedures, we obtain (H.10).
n
}
1
•
The property (H.8) says that the uncertainty of X is decreased when we have some information about Y, unless X is independent of Y. The equation i n (H.10) is often referred to as a chain rule for entropy. As we have seen above, the properties ( H . l ) - (H.10) show that the entropy denned by (1.2.2) is quite suitable as a measure of uncertainty of the outcome of a random experiment.
On the other hand, it seems to be reasonable to say
that a function to measure the uncertainty should possess at least the properties ( H . l ) - (H.4).
More importantly Shannon showed that the entropy defined by
(1.2.2) is only the function that satisfies ( H . l ) - (H.4). To state the Shannon's characterization of entropy, we introduce weaker conditions ( H . l ' ) and (H.2') than (H.l)and(H.2). (H.l')
There exists 0 < p < 1 such that H(p, 1 - p) > 0.
(H.2')
H(p, 1 — p) is a continuous function of p e [0,1].
T h e o r e m 1.2.3. Assume that a real valued function H(pt,
. . . , p ) , defined on m
the class of all discrete probability distributions, satisfies ( H I ' ) , (H.2 ), (H.3) and 1
(H.5). Then H(p,,...,p )
is the function of the form
m
m H(pt,...,p ) m
= -c^Tpilogpi,
(1.2.12)
i=i
where c is a positive constant.
P r o o f . A t first we shall prove (1.2.12) in the case of m = 2. For this purpose we put f( ) P
= 7/(p, 1 - p),
0 < p < l .
10
ENTROPY
Let p, q, r be non-negative numbers w i t h r > 0, p + q + r = 1. Then, by (H.3) and (H.5), H(p,q,r)
( j ^
= H(p,q + r) + (q + r)H = H
i
,
q
p
+
r
)
+
i
p
r
+
)
H
t
^
^
)
.
Accordingly, /(p) satisfies /(p) + (1 - ) f (j^j
= f(q) + ( 1 - ) / ( ^ )
,
5
P
(1.2.13)
where p , q > Q and p + q < 1. Let p be fixed and integrate both sides of (1.2.13) from 0 t o 1 — p w i t h respect t o q. Then we obtain (l-p)/
f Jo
" f{q)dq
+P
2
(1.2.14)
I t- f(t)dt, Jp 3
where we have rewritten the second integral on the right-hand side by changing variables and a=
f Ja
f(t)dt.
Since all terms, except the first term on the left-hand side, are differentiable w i t h respect to p, /(p) is differentiable. Differentiating both sides of (1.2.14), we have (I
- P)f'(p)
- fl?)
- 2"{1 - P)
= - / ( l - ) + 2p f P
(1.2.15)
r'f{t)dt-p-'f(p).
Similarly as above i t can be seen that f'(p) is differentiable. N o t i n g f{p) = / ( l — p ) , differentiate both sides to obtain (1 - P)f"(p)
+ 2a = 2 / ' r f(t)dt-p- f( )-p- f(p). 3
~ f(p)
2
(1.2.16)
s
P
Subtracting (1.2.15) from p times of (1.2.16), we have
m
= - p ^ p - y
0
<
p
<
1
'
Integrating twice, we obtain /(p) = - 2 { p l o g p + (1 - p ) l o g ( l - p)} + C 0
iP
+ C,
0 < p < 1,
2
(1.2.17)
I S
and C
where
2
ENTROPY
are integral constants. By the continuity o f / ( p ) , (1.2.17) is valid
for all 0 < p < 1. Substituting p = 0 i n (1.2.13), we know that /(0) = 0, so that C
2
= 0. Moreover C\ = 0, because /(p) = f(\ - p). Therefore, /(p) should be
written i n the form f(p)
= H(p,l-p)
= -c{plogp+(l-p)log(l-p)},
0
1,
w i t h a constant c. By ( H . l ' ) , c > 0. Thus we have proved (1.2.12) for m = 2. Next, we shall prove (1.2.12) by induction w i t h respect to m , the number of components of distribution ( p i , . . . , p ) . Suppose that (1.2.12) holds for all m < k. m
Let ( p i , . . . , p t ) be a probability distribution. We may assume that p * i > 0. I t + 1
+
follows from (H.5) that H(pi,
. . . , p t - i , p * , p t + i ) =H(pi, ...,pk-i, k P
+p*+i)
By the assumption of induction, this can be rewritten as
tf(pi.—iP*+i)
= -
c
£pi
I o
g P i - (P* + P * + l ) ' ° 6 ( P i t C
+Pk+l)
•=i
meaning that (1.2.12) is true for m = k -f 1. Thus the proof is complete.
•
Here we prove an interesting inequality which is known as Fano's inequality. T h e o r e m 1.2.4. Let X and Y be random variables taking values aj,a , m
and
denote by *
e
=
P(X^Y)
the probability of error. Then (1.2.18)
ENTROPY
12
where h(x) = —x log x — (1 — x ) l o g ( l — i } P r o o f . Let Z be a random variable defined by 1,
Y*X,
I o,
Y = X.
Then P ( Z = 1) = 7r . Using the chain rule (H.10), we expand H(X, e
Z\Y) i n two
different ways to obtain fi(X,
Z\Y) = H(X\Y)
+ H(Z\Y, X) =
H(X\Y)
and H(X,
Z\Y) = ff (Z|K) + H(X\Y,
Z).
Thus //(XIV) = H{Z\Y)
+ H{X\Y, Z).
(1.2.19)
Using (H.8) and (H.7), we have H(Z\Y)
< H(Z)
= /»(«•«)
(1.2.20)
and H{X\Y,Z)
= P{Z
= 0)H(X\Y,Z
= 0) + P{Z
= \)H{X\Y, Z = 1)
= F ( Z = l ) / J ( X | r , Z = 1) < jt, b g m .
(1.2.21)
The desired inequality (1.2.18) follows from (1.2.19) - (1.2.21).
•
Fano's inequality w i l l play an important role to establish the channel coding theorem (cf. Proof of Theorem 4.5.2). If X is a random variable taking infinitely many values a i , a > — w i t h probabil2
ities pi,|>2,—, then as a natural extension of (1.2.2), the entropy of X is defined by //(*)
=
Note that, in this case, H{X)
ff(p,,p ,...) 2
CO = -X>logp .
is not necessarily
;
finite.
(1.2.22)
I n the case where Y is
also a random variable taking values of at most countable number, the conditional entropy H(X\Y)
is defined by (1.2.9).
1.3 ENTROPY
FOR
CONTINUOUS
1.3 E N T R O P Y F O R C O N T I N U O U S
13
DISTRIBUTIONS
DISTRIBUTIONS
I n this section we introduce the notion of continuous entropy (or differential entropy), which is defined for random variables w i t h continuous probability distributions. Let Xi,...,Xi (Xi,...,Xj)
If the probability of X
be real (valued) random variables.
S
e A is given by = J
P(X£A)
— j
p{x„...,x )dx,...dx , d
i
A
for any d-dimensional
Borel set A C R
( R = (—°o> cc) denotes the set of all real
d
numbers), then we say that X = IXi, ...,X )
is with d-dimensional
d
distribution, or simply that X is a continuous random variable. p(x) = p{x\, . . . , i d ) (x = [xi,Xi)
continuous
The function
e W*) is called the probability density. The
density function is nonnegative; p(x) > 0, and satisfies
J tL*
j p(x ,...,x )dx ...dx 1
d
l
d
= 1.
D e f i n i t i o n 1.3.1. The c o n t i n u o u s e n t r o p y of a if-dimensional continuous random variable X — (Xt,...,Xj)
w i t h probability density function p(x) is defined
as h(X)
= k(p) = -
[
p{x)logp(x)dx,
(1.3.1)
provided the integral exists. I t is often referred to simply as the e n t r o p y of A" or the e n t r o p y of p ( x ) . The continuous entropy k(X) tropy. We w i l l write j f(x)
has been traditionally called the differential en-
dx as the domain R
d
of the integral /
R J
/(x) dx if no
confusion occurs. Here we give two examples to calculate continuous entropy. E x a m p l e 1 . 3 . 1 . (Uniform Distribution)
Let X be a real random variable with
uniform distribution on an interval (a, a + L\ of length L. Then the probability density function p(x) is given by •L,
a
+ L,
0,
x < a, x > a + L,
14
ENTROPY
and the entropy is given by h(X)
h(p)=logL.
=
E x a m p l e 1.3.2. (Exponential D i s t r i b u t i o n ) Let tp( ) x
/
exp{-(x))dx < co,
/
JR.*
be
a
function such that
\ip(x)\exp{-
< oo.
J R *
Let X = (X\,.... Xd) be a d-dimensional
random variable w i t h density function
p(x) of exponential type given by
where S is a domain of R ' and A~
= j " e x p { — i p ( x ) }
1
1
s
that = k(p)
k(X)
= - log A +
I
dx
JR*
= -\o A S
where E[Y]
+ E[p(X)l
(1.3.3)
denotes the mathematical expectation of a random variable Y.
We have defined the entropy of a continuous random variable i n an analogous form w i t h that of a discrete random variable. Continuous entropy is similar i n many ways to the entropy of a discrete random variable. B u t i t is not the l i m i t i n g case of the entropy of a discrete random variable and there are i m p o r t a n t differences. Here we investigate the relationship between the entropy of a continuous random variable and that of a discrete one. Let X be a real random variable w i t h probability density function p(x). take approximating discrete random variables X , n
X
n
= ^
if
2
^
*
<
^
(fc = 0 , ± l , ± 2 , . . . ) .
Set A , n
t
= [&/2 ,(fc + l ) / 2 " ) = { T ; * / 2 " < x < (k + 1)/2"J n
and Pn.k = P(X
€
=
/
We
n = 0,1,2,..., defined by
p(x)dx.
(1.3.4)
1.3 ENTROPY
Then the entropy of X
FOR
CONTINUOUS
15
DISTRIBUTIONS
is given by
n
-H(X )=-
^ Pn.llogPn.lt=-oo
n
As n tends to infinity, X„ converges to X in some sense. On the other hand, since p
goes to 0, the entropy H(X„)
nk
h(X).
A relationship between H(X )
T h e o r e m 1.3.1.
diverges to infinity and never converges to and h{X)
n
is given in the following theorem.
Assume that the density function p(x) satisfies (a) - (c):
1°)
p(x) is bounded and continuous almost everywhere.
(a) ( )
/R>(*)l°gp(z)l
(c)
H(X )
B
< co.
= - £ n = - / > o , * l ° g p o , * < CO.
0
DO
Then l i m (H<X )
- n l o g 2 ) - hlX).
n
(1.3.5)
Let Y also be a real random variable w i t h density function satisfying (a)
2°)
- (c) and Y
a
(n = 0 , 1 , 2 , . . . ) be defined by (1.3.4) replacing X by Y. Then lim (H(X )
- H(Y ))
n
Proof.
= k{X)
n
Define p „ ( z ) , n = 0,1,2
1°)
p (x)
= 2 p ,k n
n
if
n
- h(Y).
(1.3.6)
by
ieA , n
(fc = 0 , ± l , ± 2 , . . . ) .
t
Then /V*'°g(2'Vn,*) =
/
p (a:)logp„(i)^. n
Noting that Kta-eoPW,* = 1. e have w
H(X )-nlog2
oo V
= -
n
. p \og(2 p , ) n>k
n
n k
= -
I f p(x) is continuous at i , then clearly limn—co p„(x)
p (x)log (x)dx. n
= p(x).
For any fixed positive
integer K, we can exchange the order of l i m i t and integration:
-
lim °°J-K
/
p (z)logp„(z)
p(x)]og,p(x)dx.
/ J-K
Pn
IB
ENTROPY
Here we have applied Lebesgue's bounded convergence theorem. Using ( b ) , we have lim "
—
0
0
/ p(x)lofp<x)tfa = 0. J\z\>K '|r|>
Thus, to derive (1.3.5), i t suffices to prove lim
Pn(^) logPn(^) dx = 0
/
n.
uniformly i n
(1.3.7)
Since / ( i ) = — r l o g z (x > 0) is a concave function and since P ,k = P(l<X
Y,
n
+ l) =
,,
Po l
i t is shown that p (z)Iogp (r)
£
n
< f(Pd.i) ~ Since j
A
^ p(z) (fi = p ^ n
-
-Po,|logPo,i.
and the length |A t| of A „ , t is equal to 2 W|
p(z)logp(z)dz = j
/
/(2>M)
we have
f(p(x))dx
< lA^I/dA,,,*!- ,,,,.*) = - /
p„(z)logp (z)
1
n
Combine these two inequalities to obtain -
p(z)logp(T)di < - /
/ J\z\>K
p (z)logp {z)dz n
n
\'\>K
J
< -
£
?o,*logp ,*0
(1.3.8)
\k\>K
Letting K tend to oo i n (1.3.8), we obtain (1.3.7). 2 ) s
Since H(X )
follows from 1°).
- H(Y„)
n
= (H{X„)
- n l o g 2 ) - {H{Y ) n
- n l o g 2 ) , (1.3.6)
•
Intuitively speaking, uncertainty or randomness of a continuous random variable is infinite. This fact corresponds to l i m , , ^ , * , H(X ) n
= oo. The continuous
1.3 ENTROPY
entropy h(X)
FOR
CONTINUOUS
17
DISTRIBUTIONS
itself does not work as a measure of uncertainty. However, Theorem
1.3.2 tells us t h a t the difference k(X) — h(Y)
of the entropies indicates the difference
of the uncertainties of X and Y This is an important difference between continuous entropy and the discrete one. I n the discrete case entropy measures the uncertainty i n an absolute way. In the continuous case the measurement is only in a relative way. I n the sequel, it is often required to know which random variable has a greater or the greatest entropy. I n this situation, i t is enough to know the difference of entropies, and continuous entropy may play important roles as well as discrete entropy. The conditional entropy of a continuous distribution can be defined i n a similar manner as i n the discrete case. Let X and Y be, respectively, k and J dimensional continuous random variables w i t h density functions p(x), x g R* and g(y), y (z R'.
Moreover, assume that (X,Y)
function r ( x , y ) , (x, y ) e R * ' .
also has continuous distribution w i t h density
Then the conditional probability of
+
given Y = y
has a continuous distribution w i t h density function ' ' ( x , y ) / g ( y ) . D e f i n i t i o n 1.3.2. The c o n d i t i o n a l e n t r o p y of X given Y is defined by h(X\Y)
= - Jj^
dxdy.
r(x,y)log
(1.3.9)
Here we assume that the integral exists. Define the conditional entropy of X given Y = y by
Then ti(X\Y)
is the average value of h{X\Y h{X]Y)=
= y ) with respect to Y:
(y)h(X\Y
I
=y)dy.
q
JR<
We w i l l now summarize the properties of continuous entropy. We denote the continuous entropy h((Xi,...,X )} n
the conditional entropy h(X\{Y Theorem
u
of (Xj,.... X„) ...,Y )) n
by
simply by h(Xi,X„), 1}
1.3.2, (Properties of Continuous Entropy)
( h . l ) - (h.7) hold. ( h . l ) The entropy h(X)
can be negative.
and
h(X\Y ...,Y ). n
The following properties
18
(h.2)
ENTROPY
Let X and Y be d-dimensional
random variables such that Y =
where tp = {ip\, ». ,
= h(X)-
p(x)1j>(*))|dx,
f
fiX),
Then
J
(1-3-11)
J R '
where p ( x ) is the density function of X and J ( y ) = det
(
-M, -
is the Jacobian matrix of the inverse transformation 0 = ( ^ i , i j > d ) of iyj (det A denotes the determinant of matrix A),
i n particular, i f
then h(Y)
= h(X)
+ log |det A\,
(1.3.12)
where u>i(x) = £ j O i y i j , i = 1,-..,<*, and A = ( a ; , k j = i *• (h.3) Let p ( x ) and g(y) be
h( ) = - /
p(x)logp(x)dx<^ /
P
JR."
p(x)log,(x)dx,
(1.3.13)
JR'
w i t h equality if and only if p(x) — g(x). n(A", Y) = h{Y)
(h.4)
+ h{X\Y)
h{X\Y Yi)
(h.5)
= h(X)
< h(X\Y,)
u
h{Y\X).
+
<
h'X).
The first inequality turns into equality i f and only if Y
2
is independent of X when
Y\ is given, and the second inequality turns into equality if and only i f V] is independent of X. n h(X ,...,X„)
(h.6)
1
n
=
Y H *\Xn-,Xk- )
The equality holds if and only i f X,,X„ (h.7)
Proof,
(h.l)
n
l
l l
l
h X
are mutually independent.
/ (x,,...,A- |y) = ^ / ( x | y x i
x
1
1
.*•*_,) < £ h ( x | r ) . t
Example 1.3.1 provides an example of a random variable w i t h
negative continuous entropy.
1.3 ENTROPY
(h.2)
FOR CONTINUOUS
19
DISTRIBUTIONS
For any Borel set V C R , we have d
P(Y e r ) = P{x
e
P
( x ) d x = / pW.y))\J(y)\ dy,
where i£(T) = {i/>(y); y € I"). This means that Y has the probability density function p(^>(y))j J(y)|. HY)
Consequently
= -
f
p(^(y))|/(y)|log[ W ))|J(y)|] f P
P X
=
y
x
h(X)-jp(x)\ \J(
Thus (1.3.11) is obtained. 1
I
j ( )\og\pW\J(
= -
detA"
y
X
I f
Then J"(y) —
= 1/det A, and (1.3.12) follows from (1.3.11).
Property (h.3) can be proved i n the same way as (H.6) of Theorem 1.2.1, replacing the summations w i t h the integrations. (h.4)
Noting that / r ( x , y ) d x = ?(y), we have h{X, Y) = - jj — h{X\Y)
r(x, y ) l o
g g
( y ) dxdy
4- h(Y).
The equation h(X, Y) — h(Y\X) (h.5)
d x d y - jj
r ( x , y ) log
+ h{X) can be shown in the same way.
Denote by ? i ( y i ) , « ( y i , y j ) , p ( x , y , ) , r ^ y , ^ ) the density functions
of Yi, (YuY,),
( - f , K , ) , {X,Y,,Y }. 2
Then using (h.3) we obtain
2
dyidy = -
=
ffpi^y^og^^dxdy,
hiXlY,)-
Here the equality holds if and only i f T-(x,yi,y )/q(y y ) 2
u
= p(x,yi)/si(yi).
2
condition is rewritten as r{x,yi,yg) 5i(yi)
=
g(yi,y2)p(x,y ) l
?i{yi)
2
5i(yi) '
This
20
ENTROPY
meaning that Y is conditionally independent of X when Yt is given. The second 2
inequality can be shown i n the same way. (h.6)
By (h.4), h(X X ,...,X ) u
2
= hiX )
n
+ k(X
1
X \Xt).
2
n
In the same manner as i n the proof of (h.4), we can show h{X ,....
X \X,)
2
a
= k(X \X ) 2
t
+
h(X ,X \X X ). 3
n
u
2
Repeating the same arguments, we have n
h(X, ,x ,...,x ) 2
n
= Yl ( *
The inequality of (h.6) is clear from (h.5). h{X \X -,X -i) t
u
X\,...,X„
-
k
h(X ) k
for all k =
)•
h x
The equality holds i f and only if l,...,n.
This occurs i f and only if
are mutually independent.
Finally (h.7) can be proved similarly to (h.6).
•
As was mentioned earlier, the difference of continuous entropies is a natural extension of that in the case of discrete distributions. The properties (h.3) - (h.7) are common w i t h (H.4), (H.6), (H.8), (H.9), (H.10). This is because (h.3) - (h.7) are properties concerning the difference of entropies. On the contrary, ( h . l ) and (h.2) are concerning the values of continuous entropy itself, and these properties do not hold in the case of discrete distributions. I n particular, (h.2) implies that the continuous entropy is not invariant under a coordinate transformation.
1.4 R E L A T I V E
ENTROPY
In the case of discrete probability distributions, the relative entropy was defined in Sec. 1.1. This section extends the definition to the case of general probability distributions. Intuitively speaking, the relative entropy is defined to measure how close two probability distributions are. I t will be shown that the discrete entropy and the continuous entropy studied in the preceding sections can be expressed as a relative entropy.
M
RELATIVE
ENTROPY
Let l i and v be probability measures on a measurable space ( X , B ( X ) ) .
We
say that /x is a b s o l u t e l y c o n t i n u o u s w i t h respect to e, i.e., h -< c, i f /i(A) = 0 for every ,4 e B ( X ) such that i/{A) = 0. I t is known as the Radon-Nikodym theorem that, i f \L is absolutely continuous w i t h respect to v, then there exists a i/-integrable function
V A € B(X).
The function tp{x) is called the Radon-Nikodym derivative and is written i n the form
D e f i n i t i o n 1.4.1. For probability measures fx and v on ( X , B ( X ) ) , the r e l a t i v e e n t r o p y H(p.\ u) of n w i t h respect to v is defined by j
r
^
^
i f
L oo,
( 1
if
The measure e is referred to as the reference
. . 4
1 }
/ c
measure.
Note that the relative entropy is defined in quite general cases, including the cases of discrete probability distributions and continuous probability distributions which are studied in the preceding sections. I n these cases the relative entropy can be expressed explicitly, using the discrete probability distributions or probability density functions. A t first we consider the discrete case. Let \i and w be probability measures on a finite set X = { O i , a } m
The measures ji and v are specified by f i ( { a , } ) = p;
and f ( { a i } } - qi (i — 1, ...,m). Note that >i is absolutely continuous w i t h respect to V i f pi = 0 whenever qi = 0, and the Radon-Nikodym derivative is given by —(x) dv
= — qi
if
x — ai
( i = l,...,m).
Now we can easily see that the relative entropy H(p; u) coincides w i t h the one introduced i n Sec. 1.1: ff(ii;*)
= X>l°g^.
(1.4.2)
This can be rewritten as m
H(p; u) = -H(p,,
...,p ) - £ p f m
log qi,
(1.4.3)
ENTROPY
22
If, as a reference measure, we take the uniform distribution Co which is given by "o({o0) — l /
m
i
!
=
l i —)' *i T
e n (1.4.3) is w r i t t e n as
t f l
H{(i; uo) = -H(pi,p ) m
+ log m.
Therefore, once i n is fixed, the entropy H(pi,...,p ) m
(1.4.4)
is essentially equal to the
relative entropy w i t h respect to the uniform distribution, ignoring the additive constant l o g m and the multiplicative constant — 1 . Next, we consider the continuous case. Let ft and u he continuous distributions on W w i t h density functions p ( x ) and g{x), respectively. d
Assume t h a t /i is ab-
solutely continuous with respect to v. Then i * ( { x £ R \ q ( x ) = 0 } ) = 0 and the d
Radon-Nikodym derivative is given by
The relative entropy H{ji\ u) is now given i n the form mm
f
")=
p( )\cg^\dx.
(1.4.5)
X
If the continuous entropy k(p) = — Jp(x) l o g p ( x ) ifx is finite, then H(p-; c) can be rewritten as H{n;v)
= -k(p}-
j
p(x)logs(x)dx.
(1.4.6)
JR'
We consider a special case where p ( x ) vanishes outside a domain G of I f
1
with
finite volume \G\ and FO is the uniform distribution on G. T h e density function 9o(x) of v is given by 0
?
0
(
X
)
1 0 ,
=
x*G.
The relative entropy H{p\ v ) is given i n the form a
"») = - (p) f t
/ P( ) Hloix) x
dx = -k{p) + log \G\.
(1.4.7)
Thus we see that the continuous entropy h(p) coincides w i t h the relative entropy with respect to the uniform distribution, except the additive constant log\G\ and the multiplicative constant — 1 . We t u r n to study some of the properties of relative entropy. theorem is of fundamental importance.
T h e following
l.f
RELATIVE
2:i
ENTROPY
T h e o r e m 1 . 4 . 1 . Let /j and u be probability measures on (X,/5(X)). Then ff(**;f)>0,
(1-4.8)
w i t h equality i f and only i f ji = v.
P r o o f . We may assume that /i is absolutely continuous with respect to V. Then using the inequality (1,2.3) we have
= j
dii(i)- J
du(x) = 1 - 1 = 0.
Since the equality holds in (1.2.3) if and only i f x = 1, the equality holds i n (1.4.8) if and only if d^/dp(i) — 1 for //-a.e. x (fi-a.e. stands for ^[-almost everywhere). Since ^t(X) — c ( X ) — 1, this condition is equivalent to fi — v. •
The non-negativity of relative entropy enables us to understand that the relative entropy expresses a " s i m i l a r i t y " or a "distance" of two probability measures. I n other words, the relative entropy can be considered as a measure of how different the two measures are. Since relative entropy is not symmetric and does not satisfy the triangle inequality, i t is not a metric at all. Even though relative entropy is not a metric i n the usual sense, the understanding interpreted above is quite helpful to have an insight into various problems concerning relative entropy. The concept of relative entropy was first introduced by Kullback for applications of information theory to statistics (see [83, 84]). Since then it has played important roles not only i n information theory but also i n probability theory and mathematical statistics. There are many names and notations for relative entropy used in various fields. Some names i n common use for this quantity are Kullback-Lei bier information number, information divergence, information for discrimination, information gain, and cross entropy. We call A = {A ,...,A } t
a finite partition of X , i f A; € B ( X ) , A; O Aj = tj>
m
( i / j ) and \J1L Ai = X . l
Let p and v be two probability measures.
partition A = { A ] , . . . , A } , we define the relative entropy H^(ii\u) m
Given a associated
w i t h A by Ha(W
= X>(^)
log
(1-4.9)
24
ENTROPY
{Bi, ...,B„]
We say that a p a r t i t i o n F = {Aj,A ] m
is a refinement of a p a r t i t i o n A
=
i f for each Bj g T there exists Ai e A such that Bj C -4,-. I n this case
we can rearrange the sets B\,S
'
n
n
such a way that r = {Ajy; t = 1, . . . , m , j —
1 , f c ; } where ( J j = i A f j = Ai. The relative entropy H&(ii; v) increases according to the refinement of partitions. L e m m a 1.4.2. Let A = {A\,A }
be a finite p a r t i t i o n of X and T = {Ay;
m
i =
1 , m , j = 1,.... ki} be a refinement of A . Then H^^^Hr^-u).
Proof.
For simplicity, we denote S t = 3 "•-.(Ai)'
Then, since £ • jCtiV^Aij)
b
-, u(Aij)'
" ~
= ^,i j ij '{Aij) a
(1.4.10)
l
y X!i j -
P* ' 1
i = l,...,m, j = — /i(X) — 1, i n the same way as i i
the proof of (H.6) we see that Y ^ii^{A. )lo {aMAij)} l
1)
S
<
^Qiji/fAijJlogtaijcfAij)}
On the other hand
Combining these two relations we obtain (1.4.10).
•
This lemma enables us to give a characterization of relative entropy. T h e o r e m 1.4.3. Let p and v be probability measures on ( X , £ f ( X ) ) .
Then
H(n\ u) = s u p t f a O * ; k ) , A where the supremum is taken over all finite partitions A of X .
Proof,
(1.4.11)
We denote the right-hand side of (1.4.11) by H. First we assume that p.
is not absolutely continuous w i t h respect to v. Then there exists a set A £ B ( X ) such that i/(A) = 0 and ii(A) M > H (m &
> 0. Hence, for a p a r t i t i o n A = {A,
V) = fiA)
log ^ |
+ rtA*) log f@
A ),
= cc,
c
1.1 RELATIVE
25
ENTROPY
meaning that (1.4.11) holds i n the sense of oo = oo. Next we assume that ji is absolutely continuous w i t h respect to v. Denote by tp — dji/dv the Radon-Nikodym derivative. Given an arbitrary partition A
=
{Aj,.... A } of X , we define a probability measure v& by m
VA(A)
=
A e 0(X),
^(x)<Mx),
where
Note that H&(fi\ v) can be written i n the form Ho.(p\v) =
/ log^ (x) Jx A
d)i{x).
Moreover i t can be shown that ji is absolutely continuous with respect to
and
the Radon-Nikodym derivative is given by P
d
I) x
-
By Theorem 1.4.1
This is equivalent to H (fi;v)=
/ log \b (x)dfi(x) Jx
A
A
< / logi^(i)d/i(i) = Jx
H(fi,v).
Since the partition A is arbitrary, we have H < H(ii-v).
(1.4.12)
We now proceed to prove the inverse inequality of (1.4.12). We put C„ = {x e X ; |logp(r)| > 7i}. Since Yim„ —
ao
n(C ) n
« ( c ) i o g 4£4 n
= 0, for any e > 0, there exists JV such that > M o i o g / i c c . ) > -e,
y» > m
(i.4.i3)
26
ENTROPY
Fix an integer n > N and define a partition A = {An i = 0, ± 1 , ± m ] At = ix e X ; ( A_ A_
m + i
m
=
( l
- ^ " < logv(i) < — } , m mj
by
- m + 2 < i <m,
= { z 6 X; - r . < log^(z) <
,
C„.
Then lo =(r) < -
< log 444
g ¥
xGA
+ -.
iy
~m
(1.4.14)
Thus, using (1.4.13) and (1.4.14), we have
/ ^ l o g ^ w ^ J ^ ^ o i o g ^ ^ < Sofa
u) + e + - < H m
+ e + —. m
Letting m —• oo first, and then e —» 0, we finally obtain the desired inequality B(m»)=
(\o (x)dtix)<x. SV
J X
The proof is complete.
•
Relative entropy has an alternative expression which has played i m p o r t a n t roles in the large deviation theory (see e.g.
Theorem 3.2.4). We shall show that the
relative entropy is equal to D{n\v)
= sup j
# ( * ) d t f s ) - l o g / exp(#[>J) dv'x)
(1.4.15)
where the supremum is taken over the set B of all bounded measurable functions defined on X . In the large deviation theory, D(fi, v) is called the entropy function or the rate function. T h e o r e m 1.4.4. Let fi and V be the probability measures on ( X , S ( X ) ) . Then H(p,i>) =
D(r,»).
(1.4.16)
1.4 RELATIVE
27
ENTROPY
To prove the theorem, we introduce auxiliary notations. Define classes U U ( f ) and V = \(u)
=
by
U = {u G Z ^ ) ;
> 0 (>-a. .), e
f u(x)dt>(x) = 1}, Jx
V = { u € U ; logueB}, where L ' ( f )
is the class of all v-integrable functions.
And define D(p;v)
and
D (ft; u) by 0
5 ( / i ; e ) = sup / l o g u ( x ) d u ( r ) , u€U J x D ( u ; c ) = s u p / logii{x)d/i(x). aev *ev J x 0
For every # € B , there corresponds a function u 6 V given by -l
«(*) =
exp(#(x)).
[^exp(
(1.4.17)
Then i t is easily seen that j
$(x) d/i(x) - log J
exp((i))tu/(x) = J
logu(x)dp(x).
Jx
Therefore we know that D(ti-,v) = D (fi;u).
(1.4.18)
0
P r o o f o f T h e o r e m 1.4.4. To show (1.4.16) i t is enough to prove the following three assertions. 1°)
I f D{fi; V) < co, then B(p\ v) < 5 ( u ; v).
2")
I f D (/i; u) < oo, then D(p\ u) < D (ii; v).
3")
I f ff(fi\ v) < oo, then D (y.;
V) p-iv.
0
0
0
Assume that D(fi;v) Let v(A)
Noting that v(A)
= 0(Ae
u) < H(j*l
v).
< oo. First we shall show the absolute continuity
S ( X ) ) . For any fixed k > 0, we put k,
x £ A,
0,
x$A.
= 0, we know that the function u(x) e V
(1.4.17) coincides w i t h exp(
logv(t)M*)
< D(it;i/)
< oo.
C U defined by
28
ENTROPY
Since k is arbitrary, ji{A) should be zero. Consequently /i -i v. We p u t
and ff&qv}-
2°)
/ logv(*)«^(*)<5( i/). Jx Assume that DQ(fi\f) < oo. I t suffices to show W
logu(x)d/i(z) < Dot/*! " ) ,
Vti e U .
(1.4.19)
ix Let u £ U . We may assume that (\ogu(x))~ — — m i n ( l o g u ( i ) , 0 ) € L ^)1
We
define «»(>) = f«$pJvijfMh
n = l,2,....
Then - f ( z ) = l o g u ( i ) £ B and n
n
/ # ( a : ) 4 ( * ( * ) < l < « / exp(#,,(ir))<M») + Z>o(ii;i'). (1.4.20) Jx Jx Since 0 < u „ ( i ) = e x p ( # „ ( z ) ) < u ( i ) + 1 for all n and lim„_oo exp(!f„(x)) = u ( i ) B
(i<-a.e. x), using Lebesgue's convergence theorem, we have
n
l i m / e x p { # „ ( i ) ) di/(i) = / u ( i ) d i / { i ) = 1. -°°Jx Jx
Noting that 0 < ( # ( * ) ) n
+
< (* +i(x)) n
and 0 < ( # „ ( » ) ) " < ( # „ + i ( s ) )
-
+
< (log (x)) U
+
(u(x)
Assume that H(LL;U)
= max(t,(x), 0))
< ( l o g u ( i ) ) - , we obtain
l i m / #„(x)d/x(x) = j l o g u ( i ) d o ( i ) . ™ Jx Jx The inequality (1.4.19) follows from (1.4.20) - (1.4.22). 3")
+
(1.4.21)
(1.4.22)
< oo. Then /i - I if. For any u £ V , we define a
probability measure 0 (c-a.e. x). We can easily show that 0 < f f ( / i ; < r ) = j^\og^{x)dp{x) = -
= ^ ( l o g ^ i j + log^fz))^*)
/ logii{x)<^(a:) + /r(/ ji*). Jx t
Thus we have Oof/'; *) = / l o g « ( x ) ^ ( x ) < / f ( ; i/). Jx The proof is complete. • U
1.5 PROPERTIES
OF RELATIVE
1.5 P R O P E R T I E S O F R E L A T I V E
ENTROPY
29
ENTROPY
This section is devoted to studying properties of relative entropy. Throughout the section we denote the set of all probability measures on a measurable space (X,B(X])byM, M,(X,I!(X)). =
First we shall show the convexity of relative entropy. For probability measures (j], fi-2 € M j and a constant 0 < a < 1, a probability measure ji — api + ( 1 — a)/*2 is defined by f*{A) = am'A) + ( 1 - a)ii,(A),
A 6 0(X).
T h e o r e m 1.5.1. 1°) The relative entropy H(fi; u) is convex in (p,v). precisely, for any m, i/j e M , {i = 1,2) and constant 0 < or < 1, Hfav)
< « f f ( ( * , i * , ) + ( l - o)Jf{/i ;c ), 2
2
More
(1.5.1)
where jx = afX\ + (1 — a ) f i j and i/ = ai/, + (1 — Ot)v$, 2°)
The relative entropy H{ji\v)
is strictly convex in ji. More precisely, i f
f/{/i]; i/) and H{ix \ v) are finite, ^i] ^
and 0 < o < 1, then
2
mm where 3")
") < * * G » i ; " ) + ( ! -
(1-5.2)
= ar/ii + (1 — a)y.i. I f // (^]; f ) and H(^2\ v) are finite, then inequalities 0 < aH{ii,-u)
+ {l -a)H(fi ;v)2
Bffcv)
< h{a)
(1.5.3)
hold for any 0 < a < 1, where ^ = oifii + (1 — or)>i2 and k{a) = —a log a — (1 — o ) l o g ( l - a).
To prove the theorem, we need the following lemma. L e m m a 1.5.2. Let at and i ; ( i = l , . . . , m ) be non-negative numbers. Then the inequality m
5>l*>g£>al°gp i
1
(1.5.4)
30
ENTROPY
holds, where a = YT=l <*i and 6 = £ ™ ,
T
m
m
equality holds i n (1.5.4) i f and
m
P r o o f . We may assume that a, b > 0. t
e
a /b .
only i f a, /6) = • • • =
u = (a /a,...,a /a)
h
For discrete probability distributions
and v = (b,/b,...,b /b),
from Theorem 1.4.1, we know that
m
Clearly this inequality is equivalent to (1.5.4). Moreover from Theorem 1.4.1 we know that the equality holds i n (1.5.5) and (1.5.4) if and only i f aj/o — h/b for all i , or equivalently a,/b\ = • • - = a / 6 . m
P r o o f o f T h e o r e m 1.5.1.
1°)
^
m
To prove (1.5.1) we may assume that m is
absolutely continuous w i t h respect t o i>, ( i = 1,2). We p u t
^ ' j j g w ,
P W - f W
(x € X , i = 1,2). Then a/, ( z ) + ( l - a ) / i ( s ) =
andaj7,(x)+(l-ft)s (s) = L s
Using Lemma 1,5,2, we have afi{x)log
+ {l -
a)f (x}\ogifi {x) 2
2
= « / i ( x ) l o g ^W + f l - «)/a(*)log 9\\x)
/,( )+(l- )/ ( )
fl
>(a/ (z) + (l-o)/ (z))log. l
^ ffs(i)
I
Q
!
I
2
O f f l ( l ) + (1 -
<*)?2(Z)
= ¥>(x)log^(x).
(1-5.6)
Integrating both sides of (1,5,6), on the space X , by the measure f , we obtain the inequality (1.5.1). 2°)
Since U] ^ /i
2
and 0 < a < 1, the strict inequality holds i n (1.5.6) on a
set w i t h V positive measure. Thus the inequality (1.5.2) can be shown i n the same way as i n 1°). 3°)
The first inequality of (1.5.3) has been shown. By assumption,
(t = 1,2)
is absolutely continuous w i t h respect to v. We use the same notations as above. Since (s + t)log(s + t) > slogs + t l o g t , s,t > 0, ¥>(*) log fix)
> a / i ( * ) l o g ( a / i f ; x ) ) + (1 - o ) / i ( x ) l o g ( ( l - a ) / ( x ) ) 2
=af ( )iagfi(x) l X
+
(l- )f (x)\ogh(x) a
2
+ a/i(x)logo+(l -a)/j(e)log(l
-a).
I.S PROPERTIES
OF RELATIVE
ENTROPY
31
The second inequality of (1.5.3) is now obtained by integrating b o t h sides by the measure v.
•
As we have mentioned, relative entropy can be considered as a "distance" between two probability distributions. Here we investigate convergence of probability distributions. Let ft, c, ji
n
that the sequence {/!„}
£ M ] ( n = 1,2,...) be probability measures. We say
c o n v e r g e s t o /i i n e n t r o p y , if l i m /Y(^ ;^) = 0. n
The
t o t a l v a r i a t i o n of p — v is defined by m
- " I I = s u p X J HfU)
- K*)l<
(1-5-7)
where the supremum is taken over all finite partitions A = { A ] , . . . , A ) m
is easily seen that \\ti —
of X . I t
defines a metric on the space M i . It is said that
(u ) n
c o n v e r g e s t o (i i n v a r i a t i o n , if
lim ||fi„ - H| = 0. By definition, i t is clear that if {fi„} l i m fi (A) n
converges to ji i n variation then
= >,(A),
V A £ S(X).
In the case where X is a metric space and B ( X )
(1.5.8)
is the topological Borel
field,
namely the m i n i m u m
It is said that {/j„} c o n v e r g e s t o j i i n l a w , or
weakly
c o n v e r g e s t o ft, i f
l i m / f{x)dfi {x)= "— J x n
00
I
f{x)dii{x),
J X
for all bounded continuous functions /. It is known that { J I „ } converges to u i n law i f and only i f lim
u (G)>^(G), n
for all open sets G. Therefore, if ( u „ ) converges to fi in variation, then i t converges to ji in law.
32
ENTROPY
L e m m a 1.5.3. I f measures p e M i and e £ M i are absolutely continuous w i t h respect to a measure a € M i , then
Jx where ^
= g(*3
and
0(x)=gf>),
P r o o f . For any p a r t i t i o n A = {A\,...,A },
i t is easily seen that
m
T\v(A )-v{A )\ i
= Y,\
i
/
< £ / i =
Mz)-V>(>)I<M*)
i
= / |^>-^)1<M*)Jx Therefore <
/ |v(*)-*(*)l
To prove the converse, we put s =
{xeX
; ¥
Kx)>^(x)}.
Then (fJ, B } is a partition of X , and c
= /i(B)-«{S) + " ( 5 ) - M » 1 B
m
^sup^l^AO-KA,)!. Thus the proof is complete.
•
Here we show that the total variation of /i — f is dominated by their relative entropy.
1.5 PROPERTIES
T h e o r e m 1.5.4.
OF RELATIVE
Si
ENTROPY
Let ft, V G M , . Then
1°)
|| -c|| <2tf( ;c). i
H
2")
(1.5.9)
u
I f { f i } converges to /i i n entropy, then i t converges to u i n variation.
Proof.
n
1*)
There exists a measure a e M ] such that w -< a and v < a. For
example we may take a = 1/1 + \v as such a measure. Denoting as
and 0 0 ) = dv/do, we put S = { i £ X ; y j ( x ) > ${x)}.
M-B) + ti(B ) c
= v{B)
&-H=
+ v(B ) c
Noting that
= 1, we have
f iv(?)-'K*))M*)
+ j
= , i ( S ) - v{B) + v(B')
-
U
Mx)-
(B<)
= 2{MB)-"(5)).
(1-5.10)
By Theorem 1.4.3, i t is clear that fffj.;
„ ) > ,i(B)log
^ l o g ^ l .
+
(1.5.11)
For any numbers 0 < s, t < 1, the following holds slogy + { l -
3
>2( -tf.
) l o g ^
S
(1.5.12)
Combining (1.5.10) - (1.5.12) we have
ff( i,)£20i(S)-^fl)) >^ll^-HI 1
W
1
Thus (1.5.9) is obtained. Assertion 2°) is an easy consequence of 1°).
•
Lower semi-continuity is an important property of relative entropy. T h e o r e m 1.5.5. I f {ii„} and [u ) n
converge to ii and v, respectively, i n the sense
of (1.5.8), then H((*\v)<
l i m H(p ;u ). n
a
(1.5.13)
ENTROPY
3*
In particular, if {/!„} and {u }
converge to n and v i n variation, then (1.5.13) is
n
true.
Proof.
A t first, we assume that H(fi;v)
< co. Then, for any e > 0, there is a
partition A = {A\, —, -<4 } such that m
H (p\v)> A
H{p\v)-e.
(1.5.14)
See (1.4.9) for the definition of H&(ti; v). By the assumption, for each A
i:
l i m ft (Ai) n
= p{Ai),
l i m f„(Ai)
n—-co
we have
=
TJ—-co
Therefore, i f n is large enough, then |Jf (/i;i/)i/„)| a
< e,
and
This means that lim
H& -,v„)>H((ii»)-2e. n
n—-co
Since e > 0 is arbitrary, we have (1.5.13). Next, we assume that H(/i\ v) = oo. Then, for any fixed positive number M , i n place of (1.5.14), we have S a O ; V) >
M.
Using the same arguments as above, we can show that lim H(flni "n) = oo = ff(u; v). n—-co
The proof is complete.
•
1.6 MUTUAL
1.6 M U T U A L
35
INFORMATION
INFORMATION
Shannon successfully introduced the concept of mutual information as a measure of information. I t may not be too much to say that this made i t possible to build up information theory, a mathematical theory of communication. I n this section we shall define m u t u a l information and study most of the properties of mutual information. Let X be a random variable taking values a , , a
w i t h probabilities p j , . . . , p .
m
m
Suppose that we cannot observe the outcome of X directly, and what we can observe is another random variable Y. w i t h probabilities qi,...,q
n
We assume that Y takes values bt,.... b
n
The entropy H{X)
respectively.
uncertainty of X before the observation.
of X expresses the
Suppose we are informed that Y = bj
occurred. Then the uncertainty of X is now reduced to the conditional entropy H(X\Y
= bj).
We can say that the amount of information w i t h respect to X,
contained i n the information ¥ — bj", is equal to H{X)-H(X\Y
= bj).
(1.6.1)
The average value £ { t f ( . Y ) - H(X\Y
= bj))P(Y
= by) = H(X)
-
H(X\Y)
of (1.6.1) is the amount of information w i t h respect to X we may expect to obtain by observing Y
We denote this quantity by I{X,Y)
Using (1-2.2) and (1.2.8), I(X,Y)
= H(X)
I(X,Y):
- H(X\Y).
can be written as n
m
j = l i=l where r,j = P{X
= a„Y
(1.6.2)
y , H
>
= bj). Therefore, we have I(XX)
=
I(Y,X),
meaning that the information about X contained in Y is equal to the information about Y contained i n X.
This allows us to call I(X,
or the average m u t u a l information between X and V".
Y) the mutual information
ENTROPY
36
We denote by px
the probability distribution of X.
In the same way we denote by pxY
Then fix ~ (pi.•••>Pm)-
— {>",j; ' = l,...,m,j
probability distribution of X and Y. A n d denote by fix
l , . . . , n } the joint
=
PY the product of y.
x
x
and /iy: P-x x w ( { ( « < A ) j ) = Pi!fcThen from (1.6.3) and (1.4.2) we know that the mutual information I(X,
Y)
can
be expressed as a relative entropy: I(X,Y)
= H(HXY;IIX
x py).
(1.6.4)
This suggests to us how to extend the definition of mutual information to continuous random variables or stochastic processes. Let ( X , B ( X ) ) and ( V , 6 ( Y ) ) be measurable spaces, and let X and Y be random variables defined on a basic probability space (ft, B, P) taking values in X and Y , respectively. We denote by ji
and jiy the probability d i s t r i b u t i o n of X and Y:
x
p (A)
= P(X
x
Let B(X)xB(Y)
e A),
A£B(X).
be the product of S ( X ) and B(Y),
theo-field of the product space
X x Y = { ( * , » ) ; i € X , j i e Y } generated by {Ax
B- A e B(X),
B € B(Y)}.
The joint probability distribution /ixv of X and K is now defined as a probability measure on ( X x Y , f J ( X ) x B(Y)) = P({X,
PXY(C)
The product fix x /iy of /i
x
( X x Y , B ( X ) x B(Y))
fix x M
A
such that Y)
eC),
G € 0(X)
k
B(Y).
and p.y is also defined as a probability measure on
satisfying
* ) 5
= MA)/MB),
A e B(X), B e 0(Y).
Note that X and F are mutually independent if and only i f fx Y X
D e f i n i t i o n 1.6.1. The m u t u a l i n f o r m a t i o n I(X,
Y)
— P-X X f l y .
between X and Y is de-
fined by (1.6.4).
Note that, since we may take ( X , B ( X ) ) and ( Y , B ( Y ) ) arbitrarily, the mutual information I(X, stochastic processes.
Y)
is well denned even if X and Y are random vectors or
1.6 MUTUAL
37
INFORMATION
By the definition of relative entropy, the mutual information is w r i t t e n as I(X,
Y) =
log ,
/
( « , y) dfixvix, V%
d t l X Y
(1-6-5)
if CXY ri. t*X x fiy \ otherwise I(X, Y) is infinite. Throughout the text we denote by E\ £ ] the mathematical expectation of a random variable £:
Then, i n tenns of the integral on the space f i , (1.6.5) can be rewritten as I(X,Y)=
f fag
X
=
(X(u) Y(»))dP{w)
^ dll X /l
t
Y
£ | i o ^ - ( x , F ; d/ix x u y g
(1.6,6)
J
M u t u a l information plays a central role in the theory of information transmission. Let {A\,A )
and {Bi,B„}
m
be finite partitions of X and Y , respectively.
Then A = {Ai x By, i = l , . . . , m , j = l , . . . , n ) is a partition of X x Y .
(1.6.7)
Choosing points a; £ A , 6j £ £7, arbitrarily, define f
discrete random variables X& and 1 a by
x
A
=
if
ai
X £ Aj,
Kb = iy
if
y £ Bj.
Note that, by taking fine partitions, one can approximate X and Y by discrete random variables X& and 1 ^ . We shall show that the mutual information I(X, is approximated by I{X ,Y ), &
meaning that the definition (1.6.4) of I{X,Y)
a
Y) is a
natural extension of (1.6.3). T h e o r e m 1 . 6 . 1 . The following holds I(X,Y)
= sup I(X ,Y*),
(1.6.8)
A
A
where the supremum is taken over all the finite partitions A of the form (1.6.7), P r o o f . For discrete random variables X& and K a , i t is clear that I(X ,Y ) A
&
=
Hinx^Yi.Px*
x nvt)
= H&(PXY;I*X
x
p )Y
(1.6.9)
38
ENTROPY
Hence (1.6.8) follows from (1.6.4), Theorem 1.4.3 and (1.6.9). Here we should note that the supremum i n (1.6.8) is taken, not for all, but only for all partitions by means of rectangles of X x Y . However i t is known that this restriction is not necessary (see Dobrushin [31]).
•
For discrete random variables, the mutual information is given by (1.6.3). For continuous random variables, i t is calculated as follows. T h e o r e m 1.6.2. Let X = (X ...,X ) it
and Y -
k
(*,,...,Y|) be, respectively, k
and / dimensional random variables such that (X, Y) has continuous distribution w i t h density function r ( x , y ) ( ( x , y ) £ R * ' ) . Denote by p ( x ) = J " r ( x , y ) d y and +
q(x)
= jr(x,y)dx
the density functions of X and Y, respectively.
mutual information I(X,Y)
T h e n the
is given by = f
I(X,Y)
r f x . y j l o g - ^ ^ x d y .
(1.6.10)
= h(X) + h(Y) - h(X,Y).
(1.6.11)
Moreover the following holds I(X,Y)
Proof. 0, lixY
= h{X) - h{X\Y)
Since fix X py has the density p(x)(y) and p ( x ) g ( y ) = 0 implies r ( x , y ) = 's absolutely continuous w i t h respect to /ix x fiy and d w x
^
i
y
(
X
•
y
,
p(x) (y)' 9
l*,y)€H
"
Now (1.6.10) follows from (1.6.5). We can easily show (1.6.11), using (1.3.1) and (1.3.9).
•
We proceed to define conditional mutual information. Let X, Y, Z be random variables taking values on measurable spaces ( X , f J ( X ) ) , ( Y , B{Y)} and (Z, 0 ( Z ) ) , respectively. Given a condition that Z — z occurred, we denote by /ix|z("l )> z
PY\Z{-\ ),
PXY\Z(-\Z)
Z
(X,Y).
the conditional p r o b a b i l i t y distributions of X,
Y and
Then the quantity l(X
t
Y\Z = z) = H(LL ^{ xy
• \z);fix\z( • |f} x fi \z( • \4) Y
(1-6.12)
expresses the conditional mutual information between X and V when Z = z occurred. Taking the average o f I(X,Y\Z conditional mutual information.
= z) w i t h respect t o Z , we define the
1.6 MUTUAL
39
INFORMATION
D e f i n i t i o n 1.6.2. The quantity = J
I(X,Y\Z)
I(X,Y\Z
= z)dp (z)
(1.6.13)
z
is called the c o n d i t i o n a l m u t u a l i n f o r m a t i o n between X and Y given Z.
We collect most of the properties of mutual information. T h e o r e m 1.6.3. (Properties of M u t u a l Information)
Let X, Y, Z, X , Y (i = s
f
1, ...,n) be random variables. The following properties hold. I(X,Y)
(1.1)
=
(1.2)
I{Y,X).
T(X,Y)>0,
w i t h equality if and only i f X and Y are mutually independent. (1.3)
I f Z is a function of Y, namely Z is given by Z(w) = f(Y(uj)),
us € ft, where
/ is a measurable mapping from Y into Z . Then I(X,Y)>I(X,Z). If / is a bisection and the inverse mapping /
- l
(1.6.14) is also measurable, then the equality
holds i n (1.6.14). I(X,Y\Z)>0.
(1.4)
The necessary and sufficient condition for I(X,Y\Z)
= 0 is that X and Y are
independent when Z is given, in other words X, Z, Y forms a Markov chain. (1.5)
I f (X, Y) is independent of 2 , then I(X,Y\Z)
(1.6) (1.7)
KX,{Y,Z))
=
= I(X,
I(X,Y).
Z) +
HX,Y\Z).
I f A\ Y, Z are discrete random variables, then I(X,Y\Z)
=
H(X\Z)-H(X\Y,Z)
= H(X\Z)
+ H(Y\Z)
- H(X,
Y\Z).
10
ENTROPY
(1.8)
I f X, Y, Z are continuous random variables, then I(X,Y\Z)
h(X\Z)-h{X\Y,Z)
=
= h(X\Z)
(1.9) (1.10)
7(x,(r ,...,y )|2) = 1
-
h{X,Y\Z).
^/(x,y;]2,(r ,...,F -.)).
n
I f (Xi,Yi),...,(X ,Y„)
+ h{Y\Z)
1
i
are mutually independent, then
n
n
I((X ...,X»),(Y ...,Y )) u
1}
=
n
Y- '^ I(
Xi Y
i=i
P r o o f . We shall prove assuming that the mutual informations and the conditional informations are finite. Otherwise we can show that the equalities hold i n the sense of oo = oo. The symmetry ( L I ) is clear by definition. Since I(X,Y)
is a relative entropy,
(1.2) follows from Theorem 1.4.1. (1.3)
For each finite partition A = { A ; x Cj)
of X x Z by means of rectangles,
there corresponds a finite partition T = ( A ; x f i j } of X x Y , where Bj = f~ {Cj) {y e Y ; f(y)
e Cj}.
Clearly I(X ,Y ) r
from Theorem 1.6.1. I{X,Z) (1.4)
= I{X^Z^).
r
I f / is a bijection and
> I{X,f~'(Z))
= I(X,Y),
is measurable, then we have
and consequently I(X,Y)
' W)
' \ ) = Px\zl Z
x
=
I(X,Z).
PY\z{ * \ ): Bnd we know from Theorem 1.4.1, z
(1.6.12) and (1.6.13) that this is equivalent to I(X,Y\Z) Let (X, Y)
be independent of Z.
PY\Z( * |*) and HXY\Z(
Px\z{'
equal to px, HY and fi YLet I(X,
Z) and I(X,
= 0.
Then the conditional distributions
• l»0 do not depend on z, and are respectively
Thus I(X,Y\Z)
X
(1.6)
=
follows
The inequality is clear by definition. X, Z, Y forms a Markov chain i f and
only i f IIXY\Z( (1.5)
i
Thus (1.6.14)
=
I(X,Y).
Y\Z) be finite. Then there exist the Radon-Nikodym
derivatives ,
Consequently jixYZ
>
dpxz
.
,
< P-X * PYZ and
dpx
*
(x, ,z) V
PYZ
= a(x, )ft*,y,*)V
(1.6.16)
1.7 RATE-DISTORTION
FUNCTION
41
It follows from (1.6.5), (1.6.15) and (1.6.16) that I(X,
(K, 2 ) ) =
log a{x, z) diixvzix,
/
y, z)
JXxYxZ
+ -
/
\ogff(x,y,z)dp z{x,y,z) XY
JX-YxZ
I(X,Z)
+
I(X,Y\Z).
Properties (1.7) and (1.8) are easily shown by using (1.6), (1.6.2) and (1.6.11), and (1.9) is proved by using (1,6) repeatedly. Put Xi = (X
(1.10)
,X„)
aadYi=(Yi,...,Y ).
Then by (1.6)
n
/((x ,...,x ),(r ,...,y„)) = /{(ji£- ,^ ),(y ,? )) 1
-
I1
I(X,
1
,Y,)
1
+ I(Xi,Yi
[Yi) + I(X ,
2
1
Y\ \X ) + I(X ,
2
t
By (1.4), (1.5) and the assumption, we see that I(X , l
and I(%S,%lXuYi')
= i(Xl>%%
n
2
%\Yi)
Y 1*, 2
,Yi).
= I(X ,Y,\Xj)
= 0
2
so that
/((x ,...,x ),(y ,...,y„)) = /((x,, X ),(Y Y )) 1
1
1
2
U
= I(X Y,)
2
Repeating the same argument we obtain (1.10).
U
+I(X Y ). 2}
2
•
The properties (1.7) - (1.10) are immediate consequences of (1.6). I n this sense (1.6) is fundamental. Properties (1.6) and (1.9) are referred to as chain rules for mutual information. Moreover, since (1.6) is equivalent to (1.6.13), we have an alternative expression I(X,Y\Z)
= / ( X , ( y , Z ) ) - I(X,Z)
(1.6.17)
of the conditional mutual information.
1.7 R A T E - D I S T O R T I O N
F U N C T I O N
In practice, since information transmission over a communication channel is perturbed by noise, we may not expect exact recovery of messages at the o u t p u t of the channel. Practically speaking, we are not interested i n exact transmission, but only i n transmission w i t h i n a fidelity of reproduction. To discuss information transmission w i t h fidelity, we introduce the concept of rate-distort ion function.
ENTROPY
42
Mathematically speaking, messages t o be transmitted a n d the corresponding received messages are represented by random variables. Denote by X a message to be transmitted and by Y the received message. The fidelity of the communication is evaluated by using a function d(X, Y), which is called the r e p r o d u c t i o n e r r o r or the d i s t o r t i o n . The distortion d(X, Y) should be determined corresponding to the joint probability distribution of X and Y, The distortion needs not have metric properties b u t should be defined i n such a way that smaller distortion corresponds to better performance of the communication. Let us see some important examples o f distortions to measure the reproduction error. I f the messages X and Y are discrete random variables, then one of the natural ways to evaluate the reproduction error is to use the e r r o r p r o b a b i l i t y P(XjLY)
(1.7.1)
of reproduction of the message X. Mean square error is another useful measure to evaluate the reproduction error. Let X = (X^, ...,Xiv)
and Y — (Y\,...,Ys)
be N-dimensional
random variables.
Then the mean square error is given by f 1
1
N
1 / 2
More generally, i f X is a metric space w i t h metric p(x,y) and Y = (Y\,...,YN)
and X =
(X ,XN) t
are JV-dimensional X valued random variables, then the
reproduction error can be measured by I
N
d(X,Y)=^~Y, {p( -' -) )\ E
x
Y
•
2
Let ( X , X) be a measurable space. A d i s t o r t i o n m e a s u r e p(x,y)
(1-7-3) is a function,
defined on X x X , which satisfies the following properties (a) and (b): (a)
p(x,y)
= (y,x), P
Vx,y € X .
(b)
p(x,y)
> 0,
V x , y € X , and p(x,y)
Note that p(x,y)
= 0 i f and only i f x = y.
is not necessarily a metric, because the triangular inequality is
not assumed. And the d i s t o r t i o n d{X,Y)
is defined by
d(X,V) = { £ [ p ( X , n ] } 2
1 / 2
.
We can easily see that (1.7.2) and (1.7.3) are special cases of (1.7.4).
(1.7.4) Moreover,
the error probability (1.7.1) can also be considered as a special case of (1.7.4). I n
1.7 RATE-DISTORTION
43
FUNCTION
fact, i f we define a distortion measure by / 0,
if .
1,1,
if
•*=& ijiy,
then P{X?Y)
=
E\p(X Y?}=d(X,Y)\ l
I n this section, we shall be concerned w i t h the relationship between mutual information and reproduction error. We are interested i n the minimum quantity of the m u t u a l information we need to send for information transmission w i t h i n a required fidelity. Definition
1.7.1.
R(D;X)
For a given message X, a function R(D; X)
= in{{I(X,Y);
d{X,Y)
defined by
0 < D < co,
(1.7.5)
is called the r a t e - d i s t o r t i o n f u n c t i o n of X, where the infimum is taken for all Y satisfying d(X,Y)
< D. The d i s t o r t i o n - r a t e f u n c t i o n D{R,X)
of X is defined
by D(R\ X)
= i n f { d ( X , K ) ; I{X,Y)
0 < R < co,
< ft},
where the i n f i m u m is taken for all Y satisfying I(X,Y) occurs, we simply denote R(D)
and D(R)
Note that the rate-distort ion R(D,X)
for R{D;X)
< R.
(1.7.6) If no confusion
and D(R\X)
respectively,
is the minimum mutual information we
have to transmit i n order to reproduce X w i t h distortion not greater than D. Moreover the distortion-rate D(R\X)
is the minimum distortion attainable by
sending m u t u a l information not greater than R. For a Gaussian random variable X we can find an exact expression of R(D\
X)
as a function of D (see Theorem 1.8,7). I n general it may be a difficult task to find an exact expression of the rate-distort ion function. By definition, we know that the inequalities I(X,
Y)
> R(d(X, Y); X),
d(X,Y)
> D(I(X,Y);
X)
(1.7.7)
hold. Let a message X be fixed and denote by /IXY the joint probability distribution of X and Y. Then i t is seen that the distortion d(X Y) 1
1
= E\p(X,Y) \= 1
f f p(x,y? Jx Jx
dn y{ ,y) X
X
(1.7.8)
44
ENTROPY
is determined by ji YX
mined by pxY
Recall that the mutual information I(X,Y)
is also deter-
(see (1.6.4)). Denote by Lty\x the conditional probability distri-
bution of Y given X.
Then, since (txY
= l*x * PY|X> 1 ( ^ , 1 ' ) and d(.X, Y )
are
determined by iiy\X once the distribution /ix of X is specified. Let c( • | •) be a conditional probability distribution, namely v(B\x) satisfies the properties (a) and (b): (a)
For each fixed i e X , c( • |i) is a probability measure on X ;
(b)
For each fixed measurable set B, v(B\• ) is a measurable function denned
on X . Denoting by Q the class of all conditional probability distributions on X , we define
where Co is a measure given by = J
MB)
v(B\x) dnx(x)
(1.7.9)
and dc( - |i}/di
Then by (1.6.5) and (1.7.8) I(X,Y)
= /(n
v l x
),
Therefore, if we define classes Qj(Z?) C d p ) = (c e Q; d(u) < D],
d(X,Y)
= dfjiy,*).
(1-7.10)
Q/(/i) by Q,(R)
= {u€Q;
/(c) <
R],
then we obtain alternative expressions of the rate-distortion function and the distort ion-rate function: R(D)
= i n f { / ( c ) ; c € Qd(D)),
JJ(fl) = mi{d(v)-
c g C/(B)}-
(1.7.11) (1.7.12)
Fundamental properties of rate-distort ion function and distortion-rate function are given i n the following theorem.
1.1 RATE-DISTORTION
T h e o r e m 1.7.1. The functions R(D) decreasing, and the function r(D)
FUNCTION
= R(D;X)
and D{R)
45
= D(R;X)
are non-
= R(\f~D) is convex i n D. The function
R[D)
is continuous on (0, oo), and continuous on [0, oo) i f R{0) < oo. Proof.
I t is evident that R{D)
and D(R)
positive constants such that A] + A Di and D ,
5
are non-decreasing. Let Ai and A
a
be
= 1. For arbitrary non-negative numbers
we put D = A i D , 4- A J D J . By (1.7.11), for any e > 0, there exists
2
v, e Q i t v ^ ) (' = 1,2) such that I[v )
+ e.
i
(1.7.13)
Define u £ Q by u(B\x) ^ ^MBlx) then i t is easily seen that v € Qi{sfD). r(D)
+
X^(B\x),
Therefore, by (1.7.11),
= R(y/D)
<
(1.7.14)
Define |r« ( i = 1,2) by (1.7.9) replacing v by i/j. Then J { f ) can be written as
(1.7.15) Substituting dc( • |i) dc ^ ^ ^ d ^ T l i ) ^ i0
into the inequality logt < t — 1, we obtain
from (1.7.15). Since i/ = Ai*^o + A i d o , i t follows from (1.7.16) and (1.7.13) that 0
I{v)
< A i J ( " i ) + AJ/(I/ ) < A , r ( O ) + A r ( D ) + c. 2
l
2
2
(1.7.17)
Since £ > 0 is arbitrary, from (1.7.14) and (1.7.17), we obtain r(\iDi
+ A j D ) < A , r ( D i ) + >,!•(£,), a
meaning that r ( D ) is a convex function. Note that a bounded convex function is continuous. Since r{D)
= R( \/D) is non-increasing, i t is bounded on the outside
A6
entropy
of a neighbourhood of D — 0 and continuous on (0,oo). I f ft(0) < oo, then r ( D ) is bounded and consequently continuous on [0, oo).
•
R
0
Om«
F i g u r e 1.2.
D
Rate-distort ion function
For a random variable X, define D „ M
by
= D (X)
D ™ , = [ i n f jj^
MII
<<Mx); v e
x
Then i t is seen that i t ( I > „ ) = 0, m
If D „ , „ D > S
= oo, then (1.7.18) is obvious. B U
|
there exists y
0
D(0) = D „ „ .
(1.7.18)
Suppose that D
is finite.
MILL
e X satisfying /
p ( r , s/o) d u x ( x ) < O 2
x
a
be a conditional probability distribution such that K i l f o } ) * )
=
Then d(c) < D and /(f) = 0. Hence R(D)
< D ( 0 ) < d(c) <
= 0 and D „ m
Since K ( D ) is continuous, letting D tend to D
m l I
1 f°
For any
Let v £ Q
r
every
i£X. D.
, we obtain (1.7.18).
Summarizing the properties stated above, we see t h a t the rate-distort ion function is as sketched i n Fig. 1.2. Finally we shall show that rate-distortion function and distortion-rate function are mutually inverse functions of one other. T h e o r e m 1.7.2. The following relations are satisfied. R(D(R))
= R,
D{R{D))
= D,
Q
0 < D < D
m
i
l
.
(1.7.19) (1.7.20)
1.8
P r o o f . I f ft = D{I(X,X))
ENTROPY
FOR
GAUSSIAN
47
DISTRIBUTIONS
If R = I{X,X),
0, then (1.7.19) is dear from (1.7.18).
= 0 and (1.7.19) is evident. Let 0 < R < I(X,X).
e > 0, there exists v 6 Q i ( R ) such that d(i/) < D{R)+e.
then
Then, for any
Since c € Qj(i)(fl)-l-£),
A ( D ( i I ) - r £ • ) < / ( * ) < ft.
(1.7.21)
Note that J(c) > f t i f d(y) < D { R ) - e. This means that (1.7.22)
R < R ( D ( R ) - E ) .
Since J t ( D ) is continuous, (1.7.19) follows from (1.7.21) and (1.7.22). The equation (1.7.20) can be shown i n the same way.
•
1.8 E N T R O P Y F O R G A U S S I A N
DISTRIBUTIONS
The Gaussian distribution is known as the most important continuous distribution. Its mathematical structure allows a much more thorough analysis than is possible w i t h many other distributions. Indeed for Gaussian distributions the information functions, such as continuous entropy, relative entropy, mutual information, and rate-distort ion function, can be calculated exactly.
I n this section
the formulas are derived to calculate these quantities for Gaussian distributions. Moreover, we give various characterizations of Gaussian distributions i n terms of entropy and mutual information. These results suggest to us that the Gaussian distribution occupies a quite important position i n the development of information theory. Definition
1.8.1. If X
= (JSj,Xj)
is a d-dimensional continuous random
variable w i t h probability density function
9
{
x
)
= (
2
, ) ' / W / s
e
x
p
H
(
x
-
a
)
r
_
i
(
x
-
a
)
,
| •
x
e
r
j
'
( i
-
8 i
»
then we say that X is a d - d i m e n s i o n a l G a u s s i a n r a n d o m v a r i a b l e (or v e c t o r ) , or that X is w i t h i f - d i m e n s i o n a l G a u s s i a n d i s t r i b u t i o n N(a, F). Here a = ( a i , . . . , a j ) £ R*, r = (7i>)i,j=i,...,
48
ENTROPY
that X has mean E\Xi\ — a; and covariance E\(Xi — at){Xj
—
The
distribution iV(a, T) is called a Gaussian distribution w i t h mean vector a and covariance m a t r i x V. Especially, in case of d — 1, the density function of (1.8.1) is of the form x e R (a e R , a > 0).
(1.8.2)
I n this case, we say that A" is a (one-dimensional)
random variable w i t h distribution N(a,c ),
Gaussian
where a is the mean and a
2
2
is the
variance of X.
If [X\,X )
is a fc-dimensional Gaussian random vector and X +\,X&
k
are
K
linear combinations of X\,...,X
(k < d), then we say that X
k
is a degenerate Gaussian random vector.
—
(Xjt...,Xj)
I t is known that X = (Xi,Xj)
is
Gaussian (including the degenerate case) i f and only i f every linear combination of Xi,Xi
is a Gaussian random variable.
The importance of Gaussian distribution i n applied probability is usually attributed to a fundamental result, known as the central limit theorem, which states, roughly speaking, that if a random variable X is generated by adding together a large number of mutually independent random variables then X w i l l always have an approximate Gaussian distribution, irrespective of the form of the distribution of the individual component random variables. I n view of this remarkable result, it is reasonable to suppose that the errors involved i n the measurement of most physical quantities would have distributions very close to Gaussian distributions, because such errors would generally arise as the combined effect of a large number of independent sources of error. I t is also reasonable to employ a Gaussian random variable as a mathematical model of a physical noise i n communication system. Let us start w i t h the calculation of the continuous entropy of a Gaussian distribution. T h e o r e m 1.8,1, Let X — (X),Xd)
be a d-dimensional Gaussian random vec-
tor w i t h distribution iV(a, T). Then the continuous entropy of X is given by h(X)
= ±\og{(2« ) \r\}. e
(1.8.3)
d
In particular, i f X is a one-dimensional Gaussian random variable w i t h distribution N(a,o- ), 2
then (X)=
h
-log(27re<7 ). 2
(1.8.4)
1.8
ENTROPY
FOR
GAUSSIAN
49
DISTRIBUTIONS
P r o o f . Since the density function g(x) of X is given by (1.8.1),
= \ log{(2^) |r|} + | j (x) {(x - a ) r - ' (x - a ) ' } dx i
fl
(1.8.5)
Noting that the covariance is given by T.j = jff(*){*,
- ai)(ij -
and denoting by vjj the ( i , j ) component of T
- 1
a,)dx
, we can see that
(1.8.6) where Tr(A)
denotes the trace of a matrix A. The formula (1.8.3) follows from
(1.8.5) and (1,8,6). The equation (1.8.4) is obtained as a special case of (1.8.3), where d = 1 and |r| = a . 2
•
It is noticed that the entropy h(X) does not depend on mean vectors. The relative entropy of Gaussian distributions is calculated as follows. T h e o r e m 1.8.2. Let ji and u be d-dimensional Gaussian distributions N(a.,T) and N(b,
A ) , respectively. Then
In particular, i f y. and v are one-dimensional Gaussian distributions N(a,a } 2
and
JV(fc,r ), then 2
(1.8.8) P r o o f . Denote by p ( x ) and q(x) (1.4.5), H(ji\f)
the density functions of ft and v.
Then, by
is given by
(1.8.9)
ENTROPY
50
Since g(x) is given i n the form (1.8.1) (replacing a and F by b and A respectively), we have
- Jp(x)}ogq(x.)dx = ^\og(2w) + ^log\A + I Jp(x){(x
- b ) A - ( * - b ) ' } dx.
(1.8.10)
l
Note that |p(x){(x-b)A-'(x-b)'}dx =
/ ( x ) { ( ( x - a) - (a - b ) ) A " ' ( ( x - a ) - (a - b ) ) ' } dx
=
/ p(x){(x - a J A - ' O " < 0 ' } * t + ( » - b ) A " ' ( a - b)'.
P
(1.8.11) I n the same manner as i n (1.8.6) we have J p ( x ) { ( x - a ) A " ' ( x - a)'} dx = TV ( r A " ) .
(1.8.12)
1
Combining (1.8.10) - (1.8.12) we obtain -
f p ( x ) l o g g f x ) dx = | log(2x) + - log |A| l
+ -Tr ( r A " ) 1
l
+ i((a-b)A- (a-b)'}.
(1.8.13)
1
Since, using (1.8.3) or similarly as above, we have - j P ( X ) l o g ( x ) d x = ± i o ( 2 ^ ) + \ bs |r| + \ T T ( r r - ) , P
1
g
the equation (1.8.7) follows from (1.8.9) and (1.8.13). The equation (1.8.8) can be obtained as a special case of (1.8.7).
•
We now proceed t o calculate the mutual information. T h e o r e m 1.8.3. Let X = (Xi,...,X % k
Y = (Y ...,Y,), U
{X,Y)
be, respectively
k, I, k + l dimensional, Gaussian vectors w i t h covariance matrices A, B, C. Then « « - f i « B f f i .
( 1
. .„, 8
1.8
ENTROPY
FOR
GAUSSIAN
51
DISTRIBUTIONS
In particular, i f k = I = 1, then I(X,Y)
= - -log(l-p(X,Y) ), l
(1.8.15)
2
where p(X, Y) is the correlation coefficient of X and Y. P r o o f . The equation (1.8.14) is an immediate consequence of (1.6.11) and (1.8.3). Assume that k=
I = 1. Then the covariance matrix C is given by r - (
°
(" \ T
2
where a and T are the variances of X and Y, respectively, and p = Thus (1.8.15) is obtained by substituting 2
2
\A\ = a , 2
into (1.8.14).
\B\=r ,
\C\ =
2
p(X,Y).
* r\l-p ) 2
2
•
It is shown t h a t , among all continuous distributions w i t h given covariance matrix, the Gaussian distribution has the maximum entropy. We may say that this fact is one of the major reasons why the Gaussian distributions play so important roles i n information theory. T h e o r e m 1.8.4. Let g(x)
be the density function of a d-dimensional Gaussian
distribution JV(a, F) and p ( x ) be an arbitrary density function of a continuous distribution w i t h the same covariance matrix T. Then h( )
(1.8.16)
P
P r o o f . Since the entropy h(g) assume that g(x)
does not depend on the mean vector, we may
and p ( x ) are of zero mean. Then I x xj (x)dx
J i i p ( x ) d x = 0,
{
P
= 7,y,
where 7,j is the ( i , j ) component of T. Using (h.3) and (1.8.1), we have h(p) <-J
p(x)logg(x)dx P
( x ) i o g { ( 2 T ) / | r | / } dx +1 ' 1,
1
2
1
f p{x))xr-V}
2
i
J
dx. (1.8.17)
ENTROPY
52
Since p ( x ) and s ( x ) have the common covariance matrix, i t is clear that the righthand side of (1.8.17) is not changed by the replacement of p ( x ) by g(x) equal to h(g). Thus we have obtained (1.8.16).
and is
•
Another characterization of a Gaussian distribution is given i n terms of relative entropy. If a reference measure is Gaussian, then the Gaussian distribution minimizes the relative entropy over the set of all continuous distributions w i t h a specified mean vector and a covariance matrix. T h e o r e m 1.8.5. Let p and v be d-dimensional Gaussian distributions
N(a,F)
and JV(b, A ) , respectively. Assume that T is an arbitrary continuous distribution w i t h the same mean vector a and covariance matrix I " as p. T h e n H{n;v)<
P r o o f . We denote by g(x), g(x)
(1.8.18)
H{T;U).
and p(x) the density functions of u, ft and T ,
respectively. Then we have H(p; u) = -h(q)-
Jg(x)log (x)dx
(1.8.19)
9
and H(r-u)
= -h(p)
- j p ( x ) l o g ( x ) dx
(1.8.20)
g
(cf. (1.8.9)). Let us recall the proof of Theorem 1.8.2. Since p and T have the same mean vector and covariance matrix, we see that (1.8.13) is valid not only for Gaussian density g(x) but also for p ( x ) , so that j (x)logq(x)dx = J p(x)log (x)dx. 3
(1.8.21)
9
As we have already shown i n Theorem 1.8.4, the entropy h(p) is not greater than h(g). Therefore (1.8.18) follows from (1.8.19) - (1.8.21).
•
From the viewpoint of mutual information, Gaussian random variables have an interesting property. Given a fc-dimensional Gaussian vector X° =
(Xf,X®)
and a /-dimensional covariance m a t r i x , i t will be shown that the mutual information I(X°,Y)
is minimized by choosing the Gaussian random vector over the set
of all random vectors w i t h the given covariance matrix.
l.S ENTROPY
T h e o r e m 1.8.6. Let A"
FOR GAUSSIAN
= (X?, ...,X%) be a fc-dimensional Gaussian random
0
vector. A n d let Y = (Yj,Y,) (X ,Y) a
53
DISTRIBUTIONS
and Y° = (Y?,Y,°)
= (X°,...,Xt,Y ,...,Y,)
and (X"^)
1
be random vectors such that
are (fc + I)-dimensional continuous
random vectors w i t h covariance m a t r i x T = (fij)ijmi„..,k+l
and that { X ° , y ° ) is
Gaussian. Then I(X ,Y°)
Proof.
(1.8.22)
0
We may assume that X ,Y
and Y°
sue with mean zero. x)
5 W , 9 ( y ) , ?°(y), r ( x , y ) , r°(x,y) (x = f>,
e R * , y = (yi,...,y,)
t
the probability density functions of X", Y, Y°, (X ,Y), 0
Since g(x),
Denote by
(X°,y°),
6
R ) 1
respectively.
and r°(x, y ) are density functions of zero mean Gaussian distri-
q°(y)
butions, we know that l o g r ° ( x , y ) / { f f ( x ) g ° ( y ) ) is a quadratic function of ( x , y ) : r°(x,y)
log
(1.8.23)
?(x)?°(y)
i,J=l
where Z{ = n for 1 < i < k, Z{ = y i - k for k < i < fc+1, and o,j and 4 are constants depending only o n the matrix T. By assumption, / J R . '
/
z z r°(x,y)dxdy i
/
=
j
J R *
J R '
(1.8.24)
2 , Z j r ( x , y ) dxdy = 7,
J R *
Thus i t follows from (1.8.23) and (1.8.24) that (1.8.25)
jt»f°(y) O n the other hand, using (1.6.11) and (h.3), we obtain I{X°,Y)
= h(X°) , , M
'JJ
X
,
y
)
Jj
+ u
+
r(x,y) • «(y)
Kxpr) ?(y)
s
//[Kx,y),
yy
i^(y7
r (x,y) B
l
o
E
^(yr
q{y)dxdy q(y)dxdy
r fx v l D
(1.8.26)
'fl(x) °(y) g
The inequality (1.8.22) is now clear from (1.8.25) and (1.8.26).
•
This theorem enables us to calculate the rate-distortion function of a Gaussian random variable under the mean square error. We treat here only the one-
si
ENTROPY
dimensional case. The multi-dimensional case will be studied later. Under the mean square error, the rate-distortion function of a real random variable X is given by R(D)
= R(D;
X) = i n f { / { X , Y ) ;
E[(X
- Y) ]
< D },
2
(1.8.27)
2
where the infimum is taken over all real random variables Y satisfying the condition E\(X - Y) ] 2
< D
2
D(R)
(cf. Def. 1.7.1). The distort ion-rate function is given by X) = inf{<,/E[(X
= D(R-
-Y) \;
I{X,Y)
2
< R}.
(1.8.28)
T h e o r e m 1.8.7. Assume that X is a Gaussian random variable w i t h distribution N(a,
Then _3 '
D(R)
D > 0,
g log max ( 1,
fi(D)= =
(1.8.29)
R > 0.
R
(1.8.30)
P r o o f . We may assume that a = 0. If D > tr, noting that a random variable Y s 0 satisfies E[(X The error E\(X
- Y) }
= o
2
— Y) \ 2
< Ef,
2
we see that R(D)
0. Let D < a.
-
is determined by the covariance m a t r i x of (X, Y).
On the
other hand, once covariance matrix is specified, by Theorem 1.8.6 we see that the mutual information I(X,
Y) is minimized i f (X, Y) is Gaussian. Hence the infimum
in (1.8.27) is achieved by a Gaussian random variable. So we may assume that Y is a zero mean random variable such that E\(X
— Y) ) 2
< D
2
and (X,Y)
is
Gaussian. I t is known that the o p t i m a l mean square predictor of X based on the observation of Y is given by the conditional expectation X = E\X\Y]
=
^
Y
(see Sec, A.2), and the error is bounded by E[(X Denote by p — p(X,X) E[XX]
= E[X ], 2
- X) }
< E[(X
2
- Y) ] 2
< D. 2
the correlation coefficient of X and X.
one can show that ,
=
{E\XX\}
2
E[X }E[X } 2
2
=
E\&) E[X*Y
(1.8.31) Then, noting
1-8
ENTROPY
FOR
GAUSSIAN
DISTRIBUTIONS
55
so that ,
,2_E[X*)-B[X'\
E\{X-X) } 2
E[X \
P
<J<
2
Therefore, using (1.8.15) and (1.8.31), we have AX
Y) = / < * )
= \ log - J K J _ > l log
(1.8.32)
meaning that R(D)
> ^log^.
(1.8.33)
Let Z be a Gaussian random variable w i t h distribution iV(0, Z> (1 2
D a~ )) 2
2
and
independent of X , and define V by
Then i t is easily shown that £[(Jf - y ) j = / J and X = Y s
Therefore, i n this
2
case, inequalities (1.8.31), (1.8.32) and consequently (1.8.33) turn into equalities. Thus (1.8.29) is proved. By definition, D(R)
< a. Using (1.7.19) and (1.8.29) we
obtain
Solving this equation we have (1.8.30).
•
Here, using this theorem, we derive a lower bound for the mean square error which is given i n terms of mutual information. This result will play an important role to show the optimality of codings. T h e o r e m 1.8.8. Let X be a zero mean Gaussian random variable w i t h variance (7
2
Then, for any real random variable Y, the mean square error is bounded by E[(X
P r o o f . Put R = I(X,Y). y/E\(X
yielding (1.8.34).
•
-
> a
Y} } 2
2
e
-
2
,
l
x
^
(1.8.34)
Then by (1.8.28) and (1.8.30), -
Yf)
a
D
W
=
a
t
~
R
=
<**~
,
KX,Y)
ENTROPY
56
HISTORICAL
NOTES
Most part of the concepts and results here (as i n the other chapters) are due to Shannon (103, 106]. There are a number of excellent references on information theory. Some of them are Fano [38], Gallager [43], Pinsker [97], Blahut [15], Gray [47], and Cover and Thomas [23]. Most of the rather general concepts i n information theory, treated in this text, are also contained in these texts. The characterization of entropy, Theorem 1.2.3, was given by Shannon [103] i n a heuristic manner, and was proved by Khinchin [78] in a rigorous way. The proof given here is due to Faddeev [37] (see also [113]) which is different from Khinchin's proof. Fano's inequality, Theorem 1.2.4, was obtained by Fano [38]. Axiomatic treatment of various kinds of entropy is given i n [1]. Relative entropy was introduced by Kullback and Leibler [84] (see also [83]), The properties of relative entropy, given in Sees. 1.4 and 1.5, are found i n [24, 97). Theorem 1.4.4 is given i n [30, 115], Rate-distortion function was first introduced by Shannon [105] and has been studied i n detail by Berger [8]. Rate-distort ion function is also called e-entropy. Kolmogorov [79] and Pinsker [96] have studied e-entropy of stochastic processes, they have been especially interested i n the asymptotic behavior of s-entropy when e tends to zero because i t expresses the complexity of the processes. The asymptotic behavior of f-entropy of finite-dimensional random variables has been examined by Lin'kov [85]. Theorems 1.8.6, 1.8.7 and 1.8.8 are due to Pinsker [96], Shannon [105], Ihara [58], respectively.
Chapter 2 Stochastic Processes and E n t r o p y 2.1 E N T R O P Y
RATE
In this section, we introduce important classes of stochastic processes, like Gaussian processes, Markov processes, and stationary processes.
Then we dis-
cuss how to define entropy, relative entropy and mutual information for stochastic processes. A real (or complex) stochastic process is a system '{Xti t £ T } of real (or complex) random variables w i t h time parameter I f T .
If the time parameter space
T is equal to the set Z = (0, ± 1 , ±2,...} of all integers or a subset of Z, then {Xt',t
{Xt\t
(E T )
is called a discrete time stochastic process.
On the other hand,
e T } is called a continuous time stochastic process, if T is equal to the set
R of all real numbers or a subinterval of R. I n the following we assume that each X i is a complex random variable defined on a probability space (£1,8,/*), unless otherwise stated. D e f i n i t i o n 2 . 1 . 1 . A system { X ; t £ T } of real random variables is said to be a t
G a u s s i a n s y s t e m , if the joint distribution of finite set {f j , f „ }
(Xt , t
X
lrl
)
is Gaussian for every
C T . I n particular, if T is a time parameter set, a Gaussian
system is called a G a u s s i a n process.
For a Gaussian system X = { X , ; t EE T } , since the probability distribution of 57
STOCHASTIC
58
AND
ENTROPY
is determined by the mean vector and the covariance m a t r i x , the
(X, ,...,X, } l
PROCESSES
n
distribution of X is determined by m(t)
-
E[X,\
dP(u),
/ X,{w)
=
Jn
t £
T,
and 7
( f , a) - £ ( ( * " « - M ( f f t i & i - m(s))],
*, t € T .
As functions of t £ T and ((, s) £ T , m ( t ) and 7(1, s) are called the expectation 2
function and the covariance function of X, respectively. D e f i n i t i o n 2.1.2. A stochastic process ( X , ; t £ T } is called a M a r k o v process, if P(X,
£A\X ,
u<s)
U
=
(2.1.1)
P(X,€A\X,)
holds for every s < t and every Borel set A, where P{X
t
conditional probability of {X
£ A ) given Y (see Sec.
t
£ A \ Y ) denotes the
A.2 for the definition of
conditional probability).
To be precise, (2.1.1) holds i n the sense w i t h probability one.
We shall not
repeat such a remark i n the sequel. D e f i n i t i o n 2.1.3. Let T = Z or T = R . 1°)
A (strictly) s t a t i o n a r y process X = { X t \ t £ T ) is a stochastic process
whose distribution is shift invariant; that is, the distribution of ( J f ,
] + J
,Xt„+ ) a
is independent of s for any positive integer n and f i , ...,(„ £ T . 2°)
A stochastic process
X
—
{Xiit
£ T } is said to be w e a k l y s t a t i o n a r y ,
if £[|X,| ] is finite, and E [ X ] = m and E \ ( X , + , - m)(X, 2
S
- m)] = ( ( ) do not 7
depend on s, where z denotes the complex conjugate of a complex number z. We call the function 7 ( t ) , £ £ T , the auto covariance function, or simply the covariance function of X.
If E[|X,| ] is finite, then a strictly stationary process is clearly weakly stationary. 2
If a Gaussian process is weakly stationary, i t is also strictly stationary because the distribution of the Gaussian process is determined by its mean and covariance functions.
I n this book, both strictly and weakly stationary processes w i l l be
referred to as stationary processes.
S.l ENTROPY
59
RATE
E x a m p l e 2 . 1 . 1 . A B r o w n i a n m o t i o n is a Gaussian process such that
E\B(t)]
= 0 and
E[B(t)B(s)\
= t/\s
E[\B(t)-B(s)\ }
=
2
min(t,
(-
B = {B(t)\t
>
0}
I t is easily seen that
s)).
\t- \.
(2.1.2)
s
I t is also seen t h a t , i f Si < s < d < i , then 2
2
E{(B(t )-B(t ))(B( 2
namely B
=
-•£?(([) is independent of
B(t ) 2
{B(t)}
l
3
2
)-B( ,))]
= Q,
3
B ( s i ) . Thus the Brownian motion
B(s }2
is a Gaussian Markov process w i t h independent increments.
E x a m p l e 2.1.2. A n O r n s t e i n - U h l e n b e c k B r o w n i a n m o t i o n is a stationary Gaussian process {X,\ t € R.} w i t h zero mean and covariance function 7
( ( - s ) =
(2.1.3)
J
where c > 0 and a / 0 are constants. I t can be shown that the Ornstein-Uhlenbeck Brownian m o t i o n is a Markov process (see Sec. 2.5). Let X — {X„}
Even if each X„ is a
be a discrete time stochastic process.
discrete random variable, i n most cases, the entropy
diverges to
H(Xi,...,Xn)
infinity as N —» oo. I n this sense, the process X — {X„}
has infinitely large
entropy. This observation suggests that a quantity which plays an important role is, not the l i m i t of
H ( X \ , X
growth of the entropy. [X\,
...,X„)
N
b u t the rate l i m / j „ j V
) ,
—
of
H(X\,Xn)
We define the entropy rate as this rate of growth. I f
has a continuous distribution, limjv—oo
replaced by l i m w - . c N~ h{X\,..., x
D e f i n i t i o n 2.1.4.
- 1
(a)
N~ H(Xi ..., l
t
XN)
should be
Xfj).
Assume that each X
n
(n g Z
+
= { n 6 Z; n > 0 } ) is a
discrete random variable. Then the e n t r o p y r a t e or the p e r u n i t t i m e e n t r o p y of the process X — { A } is defined by n
H~(X)
= J '
m
^H(X ...,X ), it
N
(2.1.4)
N—»oo iV
when the l i m i t exists. (b)
Assume that
(X\,...,XN)
>s continuously distributed for each N
£ Z
Then the e n t r o p y r a t e or the p e r u n i t t i m e e n t r o p y of the process X =
+
{X„}
is defined by k(X}=
lim I f i ^ , , , , - ^ } ,
(2.1.5)
STOCHASTIC
60
PROCESSES
AND
ENTROPY
when the l i m i t exists. Here we shall show that the limits in (2.1.4) and (2.1.5) exist for any stationary processes. T h e o r e m 2 . 1 . 1 . Let X = ( X „ ; n e Z ) be a strictly stationary process. 1°)
I f each X „ is a discrete random variable, then the l i m i t i n (2.1.4) exists
and lim ± (V—oo iV
H ( X ) =
where X " =
( X
0
0
, X - i , X -
2
2°) it
If
( X , , . . . , X
N
1
, . . . , X
) =
N
1
0
T
(2.1.6)
H(X \XQ),
and r / ( X , | A " ) =
, - )
is the conditional entropy of X h(X ...,X ),
H ( X
timx-*,
given the past X$ of X =
H ( X , \ X
0
,
. - , X ^
N
)
{X„}.
n £ Z , has a continuous distribution w i t h finite entropy
) ,
+
then the l i m i t i n (2.1.5) exists and
n
£(*)= where /i(Xj |
X$)
lim
~ h ( X
N—oo JV
lim,v—oo fi.(JT] |
=
P r o o f . We shall prove only 2"). stationarity,
i
k 1
N
(2-1.7)
= h(,Xi\Xe%
)
X(,,X—Jf)-
Part 1") can be shown i n the same way. =
k(X \X ,...,X - ) k
, . . . , X
1
1
k+2l
By
Thus, using (h.6), we
h(X \X^ ...,X ). 0
have h(X ...,X ) u
Since h{Xi)
mean j V
=
N
> h(Xi\X )
_ 1
0
h(Xi\Xo,
h[X,)
+ h(X \X )+ 1
-
0
+
h(X \X ,...,X_ ). l
> ••• > h(Xi\X(,,...,X-. + ) N
0
N+2
(cf. (h.5)), the arithmetic
2
. . . j X - k + a ) monotonically converges to h(X j | X Q ) , as
N —• oo. Thus we have lim — - f t f j E i , X ) n—-cc iV The proof is complete. • N
=
Urn iV—»oo
| X ,X_w) a
This theorem states that the entropy rate h(X)
= n(A"! | X^).
is equal to the conditional
entropy of one step "future'' X i when the "past" X$ is known. Let us introduce the relative entropy rate and the m u t u a l information rate which w i l l play important roles i n the sequel. These quantities are also defined as the rates of growth of the relative entropy and the mutual information, respectively. Let X = {X„} ^
(
y
W
and Y = {Y„)
be discrete time processes, and denote by f i ^ ' and
' the probability distributions of
{ X , , X
N
)
and ( K i , Y ) , N
respectively.
S.I ENTROPY Definition 2.1.5.
61
RATE
The r e l a t i v e e n t r o p y r a t e H { p
1°)
x
\ o f
X w i t h respect
to Y is defined b y fejfWrfPi&l
StftXipr) =
(2.1.8)
when the l i m i t exists. 2°)
The m u t u a l i n f o r m a t i o n 7(X,Y)
=
r a t e 7(X,Y)
l i m ±I{(Xu.~,X UYl.~,X )), iV—-co JV N
when the l i m i t exists. T h e quantity mutual
between X and Y is defined by (2.1.9)
N
is also called the p e r u n i t
I(X,Y)
time
information.
It is noticed that the limits in (2.1.8) and (2.1.9) do not always exist, even i f X
= \X„) and Y = {Y„} are stationary.
For stationary Gaussian processes,
we shall give explicit formulas of entropy rate and relative entropy rate later (Sec. 2.4). The m u t u a l information rate will be investigated i n Chap. 5. The distribution fix
of the process X — { X } is a probability measure on X = R . Let p and z
n
v be shift invariant probability measures on X and denote by pfj the marginal distribution of p on W . The relative entropy rate H ( p ; v) of p with respect to v N
is defined similarly by ffO^J-KiD
lfffjw*»).
(2.1.10)
N—-oo JV
The relative entropy rate possesses the following properties. T h e o r e m 2.1.2. Let p, p ' , p " , v, i/', v" be stationary probability measures on X = R
Z
1°)
(Positivity)
The relative entropy rate H ( p ; u) is non-negative.
2°)
(Convexity)
If p = ap' + (1 - ot)p" and v = a v ' + (1 - a ) u " , then Hip;
u) < a f f f j l ' j f ' ) + (1 - a)H(p";
*"),
(2.1.11)
where 0 < a < 1 is a constant. 3")
(Affine property) H(p-
I f p = ap' + (1 - a ) p " and 0 < a < 1, then v) = oH(p';
i/) + ( l - <>)#(/."; c).
(2.1.12)
P r o o f . Properties 1°) and 2°) are clear from Theorem 1.4.1, Theorem 1.5.1 and the definition (2.1.10).
02
STOCHASTIC
3°)
PROCESSES
AND
ENTROPY
"w) "
ff0*ffi»w3
By 3°) of Theorem 1.5.1, 0 < V; " « ) + 0 ~
^ Mai-
Dividing by iV and letting N tend to infinity, we obtain (2.1.12).
For continuous time processes X = {X(t);t
•
6 R } and Y = {Y{t);t
ER},
the relative entropy rate and the mutual information rate are denned i n a similar f Ti {T\ i_ • • manner as i n the discrete time case. We denote by p and p the probability x
distributions of {X(t};0
D e f i n i t i o n 2.1.6.
The r e l a t i v e e n t r o p y r a t e
1")
and {Y{t)\0
x
Y
< t < T } , respectively. H{HX\I>-Y)
<&X ' w
t
n
respect
to Y is defined by
Bfarw)- J "
i'O^Pi^ ). 7
(2.1-13)
when the l i m i t exists. 2°)
r a t e I(X,
The m u t u a l i n f o r m a t i o n
7(X,K)=
Y)
between X and Y is defined by
Urn i / T C * . n
(2-1-14)
when the l i m i t exists, where I {X,Y) T
= niXffilQ
< t < T},iF(i);0 < t < r } ) .
The quantity 1{X, Y) is also called the p e r u n i t t i m e m u t u a l
(2.1.15) information.
In general i t is a difficult task to calculate the relative entropy rate and the mutual information rate for continuous time processes, even i f they are stationary Gaussian. Some results w i l l be given i n Chap. 6 when the processes are Gaussian. Note that we can define the relative entropy rate H ( p ; v ) for stationary probability measures ft and v a a X =• R
( _ 0 0 , O O )
in the same way as (2.1.13). Then we
can show that the relative entropy rate for continuous time case also possesses the properties stated i n Theorem 2.1.2.
8.S DISCRETE
TIME
STATIONARY
PROCESSES
63
2.2 DISCRETE T I M E STATIONARY PROCESSES This section is devoted t o studying the spectral representation, the canonical representation and the moving average representation of discrete time stationary processes. These representations are important tools to analyze stationary processes. In the following, we consider a discrete time weakly stationary process X = ( X , | n £ Z } . T o study spectral analysis, it is rather convenient to treat complex stochastic processes. So each X„ may be a complex random variable. Note that our discussions can be applied t o real stochastic processes without any changes. Since the process X — {X„} is stationary, the mean m = E[X ]
(2.2.1)
k
and the covariance „
r
= E[(X
- m)(X
n+k
k
- m)j
(2.2.2)
do not depend on k £ Z. We may assume, without loss of generality, that m = 0. Recall that, as a function of n £ Z , f„ is called the covariance function of X. B y definition, i t is clear that 7n = 7 - „ ,
" G Z.
(2.2.3)
The covariance function can be represented i n the form „ = J
T
e
i n A
dF(A),
neZ,
(2.2.4)
where F ( A ) (— IT < X < it) is a non-decreasing right continuous function with F(—7f) = 0, F(ir) = 7o- The function F(X) is uniquely determined by the covariance function. The expression (2.2.4) is called the s p e c t r a l d e c o m p o s i t i o n of the covariance function and F(X) is called the s p e c t r a l d i s t r i b u t i o n f u n c t i o n . I n particular, i f F(A) is absolutely continuous, then the function /(A) (-rr < A < TT) such that F(A) = j
f(t) dt,
- T < A < 7T,
(2.2.5)
is called the s p e c t r a l d e n s i t y f u n c t i o n ( S D F ) , or the p o w e r s p e c t r a l d e n s i t y of X. I n this case (2.2.4) turns to l n
= j\-»*f(X)dX,
n€Z.
(2.2.6)
6-1
STOCHASTIC
PROCESSES
AND
ENTROPY
Note that the SDF /(A) is non-negative and satisfies /(A)dA =
7 o
(2.2.7)
Furthermore, i t is known that the process X = {X }
itself can be expressed i n
n
the form X„=
f
e
i n A
d<(A),
neZ,
(2.2.8)
J-n
which is called the s p e c t r a l r e p r e s e n t a t i o n of X. We need t o note that neither d£ is a measure i n the usual sense nor the integral i n (2.2.8) is the usual one. The set (C(A); - w < A < TT) is a system of zero mean complex random variables C(A) s < ( A , u ) such that £[C(A!X(A )) = 0
if
2
A,nA
where A ; is a subinterval of [ - I T , JT] and ( ( A )
2
= ^,
(2.2.9)
= C(6) — ( ( a ) for A =
(a,b\.
In
general, if a system {£(A);— 7r < A < TT) of random variables satisfies the property (2.2.9), then
< A < T } is called a random measure. We denote by
{d({\);-TT
the class of all random variables £ such that / |£(w)|
L (Q,P) 2
2
n
and similarly denote by L {[ — jr, jr], F) such that jf_
|i^(A)| d F ( \ ) 2
ip £ £ ([—jr, IT], F).
<
oo,
< oo. Then, similarly to the Wiener integral (see
Sec. A.3), we can define the stochastic integral ?
dP(u)
the class of all measurable functions ip
2
J2
*fi(X)
V
dC(A) for each function
The right-hand side of (2.2.8) is the stochastic integral i n
this sense. For each ip, ip £ L ([—t, 2
T], F),
J _
belonging to £ ( f i , P ) and i t holds that 2
j
E
In particular, for each interval
^(AK(A)j =j A=
(2.2.10)
(ft,6]i we have
la(A)d^(A) — C(A)
J2
W
and
F[|C(A)|] = F(A) = Fib) - F(a), 2
where 1&(A)
is the indicator function;
' We denote by H (X) n
H (X) n
if
C
H +\(X). n
I
A $ A.
0,
the subspace of L (Q,
P) spanned by X ,
2
A stationary process
X
k
=
{X } n
k < n . Clearly
is said to be d e t e r m i n i s t i c ,
S.S
DISCRETE
TIME
STATIONARY
PROCESSES
On the other hand, i f
then X is said to be p u r e l y n o n d e t e r m i n i s t i c . I f X is purely nondeterministic, then H (X)%H ^{X), n
meaning t h a t , at each time n , X
provides new information not contained in the
n
past { X t ; k < n } of X.
neZ,
n
Every weakly stationary process X — {X„}
is uniquely
decomposed into the sum of a purely nondeterministic process X' = {X' } n
deterministic process X"
=
X =X'„
+ X';,
n
where X' and X"
and a
{X„}: n g Z,
(2.2.12)
are mutually orthogonal. If X is Gaussian, then X' and A " are
mutually independent Gaussian processes. The decomposition (2.2.12) is known as the W o l d d e c o m p o s i t i o n .
Using the decomposition we can remove the de-
terministic part which is less interesting.
Consequently we may always assume
that stationary processes to be discussed are purely nondeterministic. I t is known that a weakly stationary process X is purely nondeterministic i f and only if the spectral d i s t r i b u t i o n function is absolutely continuous and its SDF f(X)
I/:
log/(A)dA < oo.
satisfies (2.2.13)
For later use we denote by S the family of all SDF's of purely nondeterministic stationary processes. Noting (2.2.7) and (2.2.13), we can show that S = {/ e £ ' [ - » , » ) ; / ( \ ) > 0 and / satisfies (2.2.13)),
(2.2.14)
where £ ' [ — T , T ] is the space consisting of all measurable functions such that Denote by
H*(X)&H*-1
that is the subspace of to
H„.\(X).
Then
(X)
the orthogonal complement of K „ _ , (A") i n
n
H„(X)
exists a random variable £„
9
H (X), n
consisting of all elements which are orthogonal
Ti (X)
is a one-dimensional subspace and there
W„-,(X)
€ H (X) n
6
H*-i{X)
such that £[|t;„| ] = I . Clearly 3
n € Z } is a sequence of mutually orthogonal random variables. Especially, i f X
is Gaussian, then
G Z ) is a sequence of independent random variables
w i t h standard Gaussian d i s t r i b u t i o n N(0
1). I t is clear that X„ belongs to the
STOCHASTIC
66
PROCESSES
AND
ENTROPY
subspace W„(£) of L ( f l , P) spanned by £*, k < n. Using the stationarity we see 2
that X
n
is expanded i n the form
- = Y -*£*-
x
" '
c
-- >
e z
(2 2 15
k=-oo
where c*, k > 0, are constants such that {ck}
|cjtj < oo. Note that the constants 2
are uniquely determined except multiplicative constants of modulus one. The
expression (2.2.15) is known as the m o v i n g a v e r a g e r e p r e s e n t a t i o n of A*. From the definition of { £ „ } and (2.2.15), i t is seen that a significant equivalence H (X)
= H ((),
n
holds.
We denote by B„{X)
n e Z,
n
(2.2.16)
the o-field generated by {X ;k
< n).
k
Then, if
X = { X ) is Gaussian, (2.2.16) is equivalent to n
B (A)
= 0 (O,
n
n
«EZ.
(2.2.17)
The relation (2.2.17), or (2.2.16), indicates the causality of the representation (2.2.15).
It may be said that the significance of the representation is primarily
becav e of the causality. Let X = {X„}
be a Gaussian process but not necessarily stationary. If X is
represented in the form n
*=-oo
(£!!=_oo |a ,*| < oo), where { £ „ } is a sequence of independent Gaussian random n
a
variables w i t h identical distribution N{0,1),
and the causality (2.2.17) holds, then
(2.2.18) is called the c a n o n i c a l r e p r e s e n t a t i o n of the process X.
The moving
average representation (2.2.15) for a stationary Gaussian process X is nothing but the canonical representation. Intuitively speaking, (2.2.17) means that, at each instance n ,
< n ) and {Xk',k
< n ] contain exactly the same information.
Moreover £„ is independent of {£*; k < n - 1} and consequently of {X ;k k
n — 1}. Therefore £„ expresses the new information provided by X„.
<
W i t h these
implications, the process £ = { £ „ } is called the i n n o v a t i o n p r o c e s s of A". Let { „ be a random variable w i t h mean zero and variance a
2
> 0, and £„'s be
mutually orthogonal. Then £ = { £ „ } is a weakly stationary process w i t h
E[tn k£k] +
=
(2.2.19)
2.S
DISCRETE
TIME
STATIONARY
67
PROCESSES
Therefore f has the SDF 2 fW
=
< A <
-t
Since £ has a fiat spectrum, i t is called a discrete time w h i t e noise. I f
is
Gaussian then i t is called a Gaussian white noise. I f £ 's are mutually independent n
and w i t h an identical probability distribution n, then ( = {f } n
is called an i . i . d .
process w i t h distribution ft. Obviously a Gaussian white noise is an i.i.d. process. Canonical representation has been successfully applied to solve the problem of prediction.
Let X
=
be a purely nondeterministic stationary Gaussian
{X„}
process. One of the most important problems of prediction is to predict the i : step " f u t u r e " A "
n +
t , based upon the observation of the "past" Xj, j < n. I f the
prediction error is measured by mean square error, then the problem is equivalent to finding a random variable X which minimizes the error E{\X
-
n+k
over the set of all functions of {Xj\j
X\ ]
< n).
2
The random variable which mini-
mizes the error is said to be the best p r e d i c t o r and the minimum error is called the m i n i m u m p r e d i c t i o n e r r o r . It is known that the best predictor of
X„
+k
based on the observation of X ; , j < n, is equal to the conditional expectation E[X \Xj,j n+k
(see Sec.
A.2).
The canonical representation is useful to solve the mean square prediction problem. T h e o r e m 2 . 2 . 1 . Let X = {X„}
be a purely nondeterministic stationary Gauss-
ian process w i t h canonical representation (2.2.15). Then the best predictor A"„,j, is given by n
X,
n k
= E\X \X„j n+k
£ )=-oo
Cn+k-jti,
(2.2.20)
and the m i n i m u m prediction error t\ is equal to (2.2.21)
£l=£[\X + -X rf\^fl\c,\\ n
k
n
P r o o f . We put Xn.k
= X+ n
k
- X .k = n
>
Cn+fc-ifi.
68
STOCHASTIC
Since
X_
since
X„,
n
k
k
£ W„(f) -
PROCESSES
H„(X),
Hence, for any
X
n k
H (£)
=
n
H (X),
is independent of
X,
n
n k
Xj,
< n.
}
< n ) , we have
= X(Xj;j
E[\X + n
ENTROPY
is a Function of A j , j < n . On the other hand,
X,
is orthogonal to
AND
-
k
X\ }
= E[\X , \ }
2
+ E[\X
2
n k
-
ntk
>£flA ,,| ] = £ | 2
n
The equality holds i n (2.2.22) if and only i f
X
=
C i
|
X\ } 2
(2.2.22)
2
This means that X „ , » is
X„ . lk
the best predictor and the minimum prediction error is given by (2.2.21).
•
The m i n i m u m prediction error is directly related to the m u t u a l information between the past
I(X ,X~) n+k
X~
=
and the future
{X„,X -\,...} n
X
n
+
k
-
T h e o r e m 2.2.2. Let X = {X„} be the same stationary Gaussian process as i n Theorem 2.2.1. Then i t holds that I ( X
a
+
,X-)
k
=
I{X
where e\ is given by (2.2.21) and
X„
k
, X
) =
n
11*3,
(2-2.23)
n
is independent of
+k
+
= £[|A" p] is the variance of X„.
7 o
P r o o f . Since
n
if
X
n
is given, using (1.4), (1.6) and
X„
ik
(1.8.15), we have I(X ,X;}
= I(X ,X , )
n+k
where
7=
-y(X , n+i
n+k
= -ilog(l -
n k
7
3
),
(2.2.24)
is the correlation coefficient. I t is easily seen that
X„ ) :k
,
,.i
=
,
E[\X»M*] E[\X , \ \ n k
Thus (2.2.23) follows from (2.2.24).
2
ill
_
70 4-
•
Concerning the coefficients in the moving average representation, the following result has been known. The proof is omitted.
S.3 DISCRETE
TIME
MARKOV
STATIONARY
PROCESSES
69
T h e o r e m 2.2.3. Let X = { X } be a purely nondeterministic stationary process. n
Then the coefficients {c,-} i n the representation (2.2.15) are given as the coefficients of the Taylor expansion of an analytic function i n the unit disc:
g c j V s V S e x p j - ; j '
where /(A) is the SDF of X.
C^log/(A)<&J,
|*|<1,
(2.2.25)
I n particular, substituting z = 0 into (2.2.25), we
have c
a
2.3 D I S C R E T E
J"
= v^exp
TIME MARKOV
log/(A)dAj .
(2.2.26)
S T A T I O N A R Y PROCESSES
Among the stationary processes autoregressive processes and moving average processes are fundamental and important. In this section we shall give a characterization of these processes in terms of their SDF. I t will also be shown that these processes have multiple Markov property. Let X = {X ; n
n <E Z } be a zero mean weakly stationary process.
Definition 2.3.1.
1°)
The process A" is called an a u t o r e g r e s s i v e
o f o r d e r I, or simply an AR(/) j = 1.
process
process, if A" is a linear combination of n
X'n—fo
and £„: l X„
where £ = {( \ n
2°)
n
= Y , i J=I a
X
» - i
+
"
e
Z
-
(«i
0)
(2.3.1)
€ Z} is a white noise such that (.„ is orthogonal to X j , j < n - 1 .
The process X is called a m o v i n g average process o f o r d e r m ( M A ( m )
process), if the moving average representation of X is given i n the form
k=0
where £ =
n € Z ) is a white noise w i t h variance one.
70
STOCHASTIC
3")
PROCESSES
AND
ENTROPY
If X is represented i n the form I
m
the process A" is called an a u t o r e g r e s s i v e - m o v i n g (;,m)
a v e r a g e process o f o r d e r
( A R M A ( l , m ) process), where £ = {£„] n 6 Z ) is a white noise w i t h
variance one such that (2.2.16) holds and that £„ is orthogonal to X j , j < n — 1.
I t should be noted that the white noise £ = { £ „ } i n 1", 2"), 3°) are the innovation process of the process X, which appeared in the moving average representation (2.2.15). For example, let X = {X ) a
be the ARMA(/, m) process given by (2.3.3).
Then i t is clear from (2.3.3) that x„ e H -i(X)
u w»(0.
n
Ue
n (X) n
u
H -i(0n
From these relations, inductively we can show that X„ € K ( 0 , B
£« e H „ ( A - ) ,
so that (2.2.16) holds. Hence £ = { £ „ } is the innovation process. Roughly speaking, the A R part (all the first terms on the right-hand side of (2.3.1) and (2.3.3)) indicates the dependence on the past of the process. As mentioned above, the M A part (the right-hand side of (2.3.2) and the second terms on the right-hand side of (2.3.1) and (2.3.3)) corresponds to the innovation, namely it indicates the new information brought by X
n
at each time n.
As w i l l be studied later, the mathematical structure of A R M A processes is considerably simple. Especially A R processes can be analyzed rather easily. On the other hand, since every purely nondeterministic stationary process can be represented as (2.2.15), i t can be approximated by M A ( m ) processes by taking the order m large enough.
This allows us to employ an A R M A process as a
mathematical model for a random phenomenon w i t h stationarity. Therefore we can say that A R M A processes play important roles not only i n theoretical study but also i n practical applications. Historically speaking, Yule [117] analyzed a series of annual Wolfer sunspot numbers, and he has shown that i t can be regarded as an AR(2)
process.
Here we give a classification of M A , A R and A R M A processes i n terms of SDF.
2.3
DISCRETE
TIME
MARKOV
STATIONARY
71
PROCESSES
T h e o r e m 2 . 3 . 1 . Denote by /(A) the SDF of the stationary process X. 1°)
I f A" is the A R ( ' ) process of (2.3.1), then
/
where A(z)
(
A
)
=
2 T 1 W -
is a polynomial of order / given by I A(z)
= Y *<-i* J
(<*> = - 1 ) ,
>
(2.3.5)
j=o
and a is the variance of £„. 2
2°)
I f X is the M A ( m ) process of (2.3.2), then
/(A) = ^\B{e< )\\
(2.3.6)
x
where f?(:) is a polynomial of order m given by
SW = ^ L V .
(2.3.7)
t=0
3")
I f X is the A R M A ( / , m ) process of (2.3.3), then
(2.3.8)
where A{z)
and B ( z ) are the polynomials given by (2.3.5) and (2.3.7).
Conversely, i f X has the SDF /(A) of (2.3.8) then X is an A R M A ( ( , m )
4°) process.
I n this case the coefficients i n (2.3.3) do not necessarily coincide with
those of the polynomials A(z)
and B{z).
I f we choose A(z)
and B(z)
as i n the
Remark given below, then the coefficients of the polynomials coincide w i t h those in (2.3.3). The same assertion is valid also for M A and A R processes.
Remark,
(i)
We may assume that the polynomials -4(2) and B { z ) have no
roots outside the unit disc. (ii)
A(z)
has no roots on the unit circle. Because, if A(z)
has a root a with
\a\ = 1, then this is in contradiction to rj[|A" | j = 70 = /__ /(A) dA < 0 0 . n
2
72
STOCHASTIC
PROCESSES
AND
ENTROPY
(iii) We may assume i n 3°) that A(z) and B(z) have no common roots. Because, if A ( Z ) and B{z) have common roots, then the process is an A R M A process of lower order.
P r o o f of T h e o r e m 2.3.1.
1°)
Denote by F { X ) the spectral distribution func-
tion of the AR(() process of (2.3.1). Since f = { ( „ ; « G Z ) is a white noise w i t h variance (Tg, using (2.2,19), the covariance function p
n
of £ is given by (2.3.9)
On the other hand, i t follows from (2.3.1) that Pn =
E\f t \ n+t k
r '
=
'
S
E
_
p -+k-pY°1 -'i
a
Lp=0
X
Xk
1=0
-i:
(2.3.10)
Since (2.3.9) and (2.3.10) hold for all n, the function F ( A ) should be absolutely continuous and the density /(A) should be given by (2.3.4). A R case 1°) and A R M A case 3°) can be proved i n the same manner. 4°)
Assume that X has the SDF of (2.3.8), and that A(z)
and B(z)
satisfy the
properties (i) - (iii) of Remark. We p u t
In = X
n
Then
rj =
Efon'lj]
=
- y^OjA" -j, /=1 n
n G Z.
{)?„} is a stationary process satisfying
E[i)„X -j]
0.
Using this fact, we can con-
\
n
— j\ >
m
,
and
E\r)n~n-m]
7* °-
=
n
struct a white noise £ = (£,,} w i t h variance one such that
r)„
B[{,,X,-] = 0, J < n - 1. Hence A" is represented as (2.3.3).
=
0,
Y^T=o
j
<
n -
m,
&*fn-t
•
Since the right-hand side of (2.3.8) is a rational function of e' , the SDF of the A
form (2.3.8) is said to be a rational SDF. For an AR(/) process, the covariance function is uniquely specified by the first ; + 1 covariances, T o . T i i — i 7 l -
A
t
first,
suppose that A" is an A R { 1 ) process of
S.3 DISCRETE
(2.3.1). Since £ [ { „
Y
n
TIME
MARKOV
STATIONARY
PROCESSES
73
, J = 0 (Jfc > 0), multiplying both sides of (2.3.1) by A%_*
and taking expectation, we have i k>0.
7* = £ « i 7 * - i >
(2.3.11)
Especially we have a set of equations, known as the Y u l e - W a l k e r e q u a t i o n , - 7 2
+ «i7i
+
i27o +
r (7,-2 Q
=0 (2.3.12)
-7i+i
+ " i 7 i +
Q
2 7 i - i
+
aa\
• ••+
=
0-
Here we have used (2.2.3). This means that, since the simultaneous linear equation 7i*o + 7 o 2 i +
7 i * 2 + ••- r 7 , _ j x j = 0
72^0
70^2 +
+ 7 i
I
i +
=0
\-Jt-2 < x
(2.3.13) 71+1^0 +
7 i ^ i + 7 i - i ^ 2 H
has a non-zero solution ( z , x , , n ) 0
+
T i ^ i -
0.
= (—l,ai,...,«(), the determinant of the
matrix consisting of the coefficients of (2.3.13) should be zero:
Therefore
71+1
71
70
7i
••
7 i - i
72
7i
70
••
7 , - 2
73
72
7i
7(+i
ii
= 0.
7 i _ i
is determined from
...
70,71,...,71
(2.3.14)
71
by solving (2.3.14), Similarly one
can see that n + y f* > 1) is a function of t _ i , . . . , 7 i + t _ i , and consequently of 7
7o,7t>—.71-
Next, suppose that we are given definite. Here we say (7_
t
( 7 0 , 7 1 , 7 1 )
7o.7i.---.7l
= 7 ^ ) is positive definite.
k = 1,2,..., as functions of
s u c n
that
(70i7l»—«7l)
' positive s
is positive definite if the matrix ( 7 m - n ) o i , « = a
1
Using (2.3.14), successively we define 7 1 + * ,
7o,7i,
7i-
Then
(70,7i,
7i+t)
l s
positive
definite for every k > 0, and we can show that there exists uniquely a weakly stationary process X = [X„] (-l,a ...,ai) u
w i t h „ , n £ Z , as the covariance function. Let 7
be a solution of the Yule-Walker equation (2.3.12). Then (2.3.11)
holds and one can show that X is an A R process of order at most I with expression (2.3.1). Note that, i f a\ ^ 0, then the order is exactly equal to I. Thus we have obtained the following theorem.
71
STOCHASTIC
PROCESSES
AND
ENTROPY
T h e o r e m 2 . 3 . 2 . Let 7 0 , 7 1 , . . . , 7 1 be constants such that ( 7 0 , 7 1 1 —> 7 i ) is positive
definite. Then there exists uniquely a zero mean A R process X = {X ; n G Z } of a
order at most / having covariances
7*,
70,71,—,7)
as the first
k > J, are determined from
I
+ 1 covaxiances. The remaining
70,71,—>7i
by solving the equations of
type (2.3.14). The coefficients a],...,aj of the A R expression (2.3.1) are determined from (2.3.11). Multiple Markov property of stationary processes is also an i m p o r t a n t concept to analyze A R M A processes. I n the following, for simplicity, we shall consider only real processes. A discrete time Markov process A" = { X } is also called a simple n
Markov process or a Markov chain. We introduce here the concept of multiple Markov property as a natural generalization of that of simple Markov property. There are two different ways to define multiple Markov process. Definition 2.3.2.
1°)
A stochastic process X = { X } is called an a t m o s t n
JV-ple M a r k o v process i n t h e s t r i c t s e n s e , if P ( X
n
+
<x\X„,X _ ...)
l
n
u
= P ( X
n
+
< x \ X
1
n
, . . . , X
n
-
N
+
l
) ,
(2.3.15)
holds for all n and x; I f X is at most JV-ple Markov b u t not at most { N — l)-ple Markov, then X is called an JV-ple M a r k o v p r o c e s s i n t h e s t r i c t s e n s e . A process X — {X„;n G Z } is called an JV-ple M a r k o v process i n t h e
2°)
w i d e sense, i f
successive conditional expectations
N
E[X +i\X -j, n
j
n
>
k,...,k+N
— l, are linearly independent b u t JV+1 successive J3[A" j| X„-j,
t =
+ N, are linearly dependent for any n G Z and k > 0.
n+
fc,k
0], t = j > 0],
I t is clear that a simple Markov process is nothing b u t an at most 1-ple Markov process.
From (2.3.15) i t is easily seen that an at most JV-ple Markov process
satisfies P(X„ <x\X ,X _ ,...) +k
n
n
1
= P(X» <x\X ...,X - ), +h
ni
n N+1
(2.3.16)
for all x, n and k — 1,2,.... Obviously (2.3.16) is a natural extension of (2.1.1). A n d the multiple Markov property i n the strict sense seems to be a concept of a natural generalization of simple Markov property. However, in some cases, the strict sense Markov property is rather restrictive and the wide sense Markov property seems to be more reasonable.
2.3 DISCRETE
TIME
MARKOV
STATIONARY
PROCESSES
75
Let us investigate the multiple Markov property of Gaussian processes. X
=
be a Gaussian process.
{X„}
Then, given
distribution of X „ * (k > 0) is Gaussian.
X
n
, X
N
Let
the conditional
- N ,
Hence we see that, if A" is JV-ple
+
Markov i n the strict sense, then i t is JV-ple Markov i n the wide sense. Moreover, we shall show that, while a Gaussian A R ( J Y ) process is nothing but an JV-ple Markov process i n the strict sense, a Gaussian A R M A ( JV,M) (M
< JV)
process is an JV-ple Markov process i n the wide sense. T h e o r e m 2.3.3. Let X = {X„}
be a purely nondeterministic stationary Gauss-
ian process. 1°)
The following equivalent relations hold. X is an JV-ple Markov process i n the strict sense •4=^ X is an AR(JV) process X has a SDF of the form (2.3.4) w i t h I = JV
2°)
Put X *
-
E[X \X , k
j
}
(2.3.17)
< 0], k > 0. Then the following equivalent relations
are also true. A" is an JV-ple Markov process i n the wide sense The subspace of L (Sl,P)
spanned by X t , k > 0, is jV-dimensional
2
X is an ARMA(JV, M)
{ M < JV) process
X has a SDF of the form (2.3.8) w i t h I = N, m = M < N. (2.3.18)
Proof.
1°)
Let X = {X„}
be an AR(JV) process w i t h expression (2.3.1) (re-
placing i by JV). Then, since (,
= P(€ +i B
= P(X
n + 1
<* <
is independent of X j , j < n, i t is true that
n+i
N
Y, > "+i-j\x ...,x - )
x\X
a
n
,...,X
x
n
ni
-
N
+
l
a N+l
),
so that A" is JV-ple Markov in the strict sense. Conversely, let A be an JV-ple Markov process i n the strict sense. Then i t can be shown that X
n
is expressed i n
the form (2.3.1) (replacing / by JV) w i t h a random variable £„ which is independent
76
S T O C H A S T I C
of X
n
P R O C E S S E S
- j , } > 0. Therefore X is an A R ( N )
AND
E N T R O P Y
process. The second equivalence i n
(2.3.17) has been shown i n Theorem 2.3.1. 2")
Let X
be an ARMA(JV, Jlf )(Jlf < JV) process w i t h expression (2.3.3)
(replacing ( f , m ) by
Then, since
(N,M)).
(k
>
is independent of
—M)
( j < — M ) , noting M < N and using (2,3.3) we know that every X a linear combination of
XQ, X - t ,
Therefore
...,X-pi+i.
wide sense. Conversely, let X — { X „ }
X
K
Xj
, k > 0, is
is ./V-ple Markov in the
be an JV-ple Markov process in the wide
sense. Then, since JV + 1 random variables
are linearly dependent, there
X Q , ...,Xfj
exist constants a , k = 0 , J V , such that ( a , a w ) ^ ( 0 , 0 ) k
0
and
TV
£ a * A V * = 0.
(2.3.19)
1=0
Here
a a^/ a
^ 0, since
and
Xn,Xpj-i
X ,...,Xpj l
a stationary process TJ = {T)„} by T?„ = - £ ) " F(nnA"(c) = 0 for k < n — N.
are linearly independent. Define = 0
a
**--*-
Then, by (2.3.19),
I n the same argument as i n the proof of 4°) of
Theorem 2.3.1, we can show that X is an ARMA(JV, M ) process. Here the order M
is equal to the maximum integer m such that £[rj,,rf _ ] / 0.
M
< N. The first equivalent relation i n (2.3.18) is clear by definition and the last
n
one has been shown i n Theorem 2.3.1.
m
Moreover
•
2.4 E N T R O P Y R A T E O F A S T A T I O N A R Y G A U S S I A N P R O C E S S Entropy rate and relative entropy rate are defined i n Sec. 2.1. For stationary Gaussian processes they can be calculated exactly, and be expressed i n terms of their SDF. Let us start w i t h the calculation of entropy rate. For convenience we put
Mf)
l ° g { 4 » V ( A ) }
=
for every f g S. B y the definition of S (see (2.2.14)), h(f)
(2.4.1) is finite.
T h e o r e m 2 . 4 . 1 . Let A" = { X ; n 6 Z } be a stationary Gaussian process and n
let X
n
= X ' + X%, ft € Z, be the Wold decomposition, where X ' = { X ' J is the n
S.i
ENTROPY
RATE
A GAUSSIAN
OF
PROCESS
purely nondeterministic part and X" = {X%} is the deterministic one. Then the entropy rate h(X)
of X is given by h(X)
= h(X') = H(f),
(2.4.2)
where / is the S D F of X'. P r o o f . A t first we prove (2.4.2) assuming that X is purely nondeterministic, so that X has the moving average representation (2.2.15). Then, i n particular, o Xi
= c i,
+
0
Y
ci-itj-
j«-oo Note that £, is independent of X j , j < 0, and that X3>
E
^ a ( X ) is
a function of X j , j < 0. Hence, the conditional probability distribution of Xj given X
0
= (Xg, X i , X , ••-) is a Gaussian distribution with mean S;
2
and variance £l|cofi| ] = c\. Therefore, using (2.1.7), (1.8.4), (2.2.26), we have 2
h(X)
= k(X \X^)
= h ^ )
x
T\ f_
=
= I log(2»e ) C o
log{4^e/(A)}dA =
h(f).
Next we prove (2.4.2) i n the general case. Decompose X as (2.2.12). Then, for = n H (X")
every fc, i t holds that H (X")
ne2:
k
n
= n z~H {X) n£
n
C
n < 7i„(X). n
D
Hence, i f XQ is given, then X' \ k £ Z, and consequently X' = X t — X' , k < 0, k
h
k
are known. Therefore, using (2.1.7), we have h(X)
= h(X \X ~) 1
= h(X[\X«)
a
= h(X[](X%)
We have already shown that h(X') is equal to h{f).
=
h(X').
•
I t may be interesting to note that the entropy rate does not depend on the deterministic p a r t , and especially that h(X) = 0 if X is deterministic. The entropy rate of a Gaussian A R M A process is calculated more simply. Let X be a Gaussian ARMA(Z, m ) process w i t h SDF /(A) of (2.3.8). Denote by a the roots of A(z)
and by p\, ...,0
m
the roots of B(z).
x
, a
t
As we have remarked in Sec.
2.3, we may assume that |crj| < 1, \Q \ < 1 and there are no common roots. Now k
/(A) can be w r i t t e n as m
=
a
nr-i«"-ftr.
M
78
STOCHASTIC
Theorem
PROCESSES
2.4.2. Let I b e i
AND
ENTROPY
Gaussian A R M A ( i , m ) process w i t h SDF /(A) of
(2.4.3). Then the entropy rate of X is given by (2.4.4)
h{X) = \\ g{2^bl). 0
P r o o f . I t follows from (2.4.2) and (2.4.3) that k(X)
log|e- - & | dX
= i log(27rei ) 2
A
a
I' log^-atfdX. ,=i«-*
-iT, ™
(2.4.5)
B y Poisson's integral formula we have j
l o g | e - « | d A = 0, a
(2.4.6)
,
if |a| < 1. The equation (2.4.4) is now clear from (2.4.5) and (2.4.6).
•
For each SDF / e S , the JV x JV m a t r i x f
To
7-{«-l)\
7-1
7-(N-2)
7i
?M/) = ( 7 - ) m
n
m i
„ i =
(2.4.7)
=
N
\7/V-l
To
IN-2
/
e' f(X)dX,
7 n = J"
BX
is called the Toeplitz m a t r i x corresponding to /. I n order to calculate the entropy rate and the relative entropy rate, we need to study the asymptotic behavior of Toeplitz matrices. Here we briefly summarize known facts concerning the asymptotic behavior of eigenvalues, inverses, and products of the Toeplitz matrices. For an
N
x N
matrix
UN
=
("mn), we define a norm
H M ^
£
\\Upi\\
K. | .
by
2
(2-4.8)
\\U \\\\V \\,
(2.4.9)
n
tn,u—1
Then SchwarVs inequality yields the inequality \Tr(U V )\ N
where
V
N
= (u
m n
N
<
N
) is also an JV x JV m a t r i x and
N
T T ( U
N
)
denotes the trace of
UN-
2 j ENTROPY
RATE
OF A GAUSSIAN
79
PROCESS
L e m m a 2.4.3. Let /, g € S be SDF's. Then we have Km^ilpglT^/)! = £
/ " lc.g{2 /(A)}dA
(2.4.10)
w
and ±Tr(T (f)T (g))
lim
N
= 2*
N
("
f(\)g(\)d\.
(2.4.11)
Moreover, if / € L [ - T , 7 r ] and /(A) > a, VA £ [—TT,TT), for some a > 0, then 2
i*r*pr3K/jr
Jirn^ where Tjv ( / ) -1
Proof.
- f i » ( i T l f = 0,
(2.4.12)
is the Toeplitz m a t r i x corresponding to 1//(A).
Let A" = {X„} be a stationary Gaussian process w i t h SDF /. Then T\(f)
is the covariance m a t r i x of (A*i,..., A*jv). Hence, by Theorem 1.8.1, the entropy rate h ( X ) is given by MAT) =
lim !*<*,,...,3V}
= I
lim ilog{(2 re)' |T (/)|}. v
7
w
Combining (2.4.2) and (2.4.13), we obtain (2.4.10). We put 7n = f
e
/(A)«/A
and /.„ = |
e'" (A)rfA. A
s
Then 7V(/)
= (7r»-i»)m,»-J,...,W I
?*(&*) = ( P m - n ) „ , „ = ]
and i t is easily seen that |V Tr(T (f)T (g)) N
=
N
Y,
7m- />nn
m
m n=l h
JV
fc=-JV + l
Consequently we have 1
lim -Tr(T (f)T (g}) fV—oo JV N
N
=
°°
T , —
k=~oo
%%,
N•
(2.4.13)
80
STOCHASTIC
PROCESSES
AND
ENTROPY
By the Parsevai equality T
ftfe-fl*
f(A)g(\)dX.
f
-«
*=-»
J
Hence we have obtained (2.4.11). The assertion (2.4.12) has been proved i n Gray [45], under the assumption H ^ L - o o l T n l < °°-
Even i f this condition is not satisfied, we can choose /„ €
5n£ [-7r,7r] i n such a way that £ ™
| J * . e * / „ ( A ) d\\ < oo, /.(A) > a, VA,
2
iB
and Hm ||/-/„|| = 2
where
lim
;
|ja denotes the norm of L [ — t t . t ] . N o w (2.4.12) is satisfied by /„: 2
£ H^fT^A))-
- T
1
(2.4.14)
(/-»)|| = 0. 3
N
Moreover we can show that, for any fixed E > 0, there exists n such that jt\\(T (f)r N
1
-(T (f )y
l
n
N
||TiV ( / " ' ) - T ( / w
113
(2.4.15)
U
(2.4.16)
:
for every JV. Finally (2.4.12) follows from (2.4.14) - (2.4.16). We denote by X ,„, N
•
n = 1,...,JV, the eigenvalues of T (f).
Then |T (/)| = W
N
n £ = i ^N,n and (2.4.10) can be rewritten as
i
N
i
J ™ , N £ '°
S A w
'
n
=
r
27 /
l
o
^ "^ 2
dX
-
Although we omit the proof, more generally, the following formula has been known. If 'p(i)
is a continuous function, then J ™ , Jr E
v(Af/,n) = 5 -
n=l
/
V f > / ( A » dA.
•' ' -
(2.4.17)
r
We now turn to calculate the relative entropy rate for stationary Gaussian processes. For convenience we put
for SDF's / and g. We see that 0 < H(f; x > 0.
g) < oo, since x - 1 - log x > 0 for all
S.i ENTROPY
RATE
T h e o r e m 2.4-4. Let X — {X ;n
OF A GAUSSIAN
e 2 } and Y = {Y \n
n
81
PROCESS
n
e Z ) be zero mean purely
nondeterministic stationary Gaussian processes w i t h SDF's / and <j, respectively. Then the relative entropy rate W{jtx\ P Y ) is given by F(«x;ny)=S (/;g),
(2.4.19)
r
provided that at least one of the following conditions is satisfied, (a)
/(A)/<j(A) is bounded.
(6)
g(X)
> a > 0 for a l l A S | - J T , T ] and / €
L [-W,TI]. 2
P r o o f . We shall prove (2.4.19) i n the case ( i ) . probability distributions of (A*j,..., A*jv) and
Denote by o ^
(Yi,V/v),
W)
and u
(
y
N )
the
respectively. Then, by
Theorem 1.8.2,
j^pjj + Tr {T (/) ( ( T ^ ) ) " |Tw(/)l
= log
1
N
- (T^/))" ) } 1
(2.4.20)
Using (2.4.10), we have (2.4.21) Obviously
i T r { r ( / ) ( r ( / ) r ' } = i. w
(2.4.22)
w
Applying (2.4.11), we have J » > { ^ W ) T
W
( r n } =
2 . £ ^ d A -
(2.4.23)
B y (2.4.9), TrlTvifHTNigjf^-^TriTMWNig-')} (TN(9))-
- T ^ T
1
<
N
(2.4.24)
( g - ' )
\\TN(/)\\
Note that lim v-o Af- ||3«(/)|| = 2TT / % / ( * ) < * * is finite. Then, using (2.4.12) J
0
1
a
(replacing / by g) and (2.4.24), we know that l i m T f { T ^ / K I M s ) ) - ' } - ^ T r {T {f)T N—oo N
( a " ) ] - 0. 1
N
(2.4.25)
STOCHASTIC
82
PROCESSES
AND
ENTROPY
It follows from (2.4.23) and (2.4.25) that
l i m l 7 V {T
} = £
( W
N(f)
£
g§
dA.
(2.4.26)
Finally, combining (2.4.20), (2.4.21), (2.4.22) and (2.4.26), we obtain (2.4.19). I n case (a), (2.4.19) has been proved i n Pinsker [96j.
•
Here we present a result on the relative entropy rate o f a non-Gaussian process w i t h respect to a Gaussian process. T h e o r e m 2.4.5. Let X = { X ; n n
e Z } and Y — {Y»;n e Z } be stationary
Gaussian processes w i t h SDF / and g, and Z = {Z„;n 6 Z ) be a stationary process w i t h the same SDF f as X. T h e n , under the same assumption of Theorem 2.4.4, the l i m i t =
K m ^ H U
z
N
- , p
)
)
m
Y
(2.4.27)
exists and = H ( P X \ P Y )
H(PZ\PY)
H(f-,g)
=
H(ii ;/ix)
+
z
+
H{p ;p $ z
x
>~3(f;g).
P r o o f . We may assume that H(n
z
(2.4.28)
] /Xy
) ' finite, so that the distribution p s
has a probability density function p/v(x) ( x £
Denote by
R ). N
p° {x) N
and
z
q%(x)
the density functions o f Gaussian distributions / J ^ ' and ( i ^ ' , respectively. Since and / i ^ '
have a common covariance matrix, and l o g f p ^ f x j / ^ j x ) )
quadratic function of x = ( j r j ,
H ^ P ; ^ ) = f
= =
P
i
w
w
( x ) l o
g
^ d x
P , ( x ) l o g ^ | d x
Bi$hp$h
By Theorem 2.1.1, the l i m i t
is a
£ R , we see that
+
+
B($hpP%
/
p ? v
(x)log4^dx (2.4.29)
S.S CONTINUOUS
TIME
STATIONARY
PROCESSES
83
exists, and we see that
-N^hJ „ n
=
PN{x)logp
^
jf
L
-
x)dx
P » ( X > 1 ° 6 p M X ) dx =
k(X).
Consequently we have H(
f
l
Z
l
l
i
x
)
= ^ m ^ H { ^ P ; ^ )
= h{X)-h{Z).
(2.4.30)
Since X and Y are Gaussian, by Theorem 2.4.4, the l i m i t N-o*
^
H
^
l
x
}
^ Y
N
>
) - * W ; M -
K/vff)
(2.4.31)
exists. Therefore we know that the l i m i t (2.4.27) exists and that the assertion (2.4.28) is obtained from (2.4.29) - (2.4.31).
•
2.5 C O N T I N U O U S T I M E S T A T I O N A R Y P R O C E S S E S This section is devoted to studying continuous time stationary processes. Spect r a l representation, canonical representation and multiple Markov property are to be discussed. Let
X
— {X{t);t
G R ) be a continuous time weakly stationary process. I n
Gaussian case we assume that X is a real stochastic process, otherwise we assume that A"(t) =
X(t,w)
(t
G R ) is a complex random variable defined on a probability
space (ft, B, P). Throughout the section we assume, without loss of generality, that the expectation E\X{t)\=
j
X(t,u>)dP(u)
is zero. We denote by (t)
1
= E\X{t
+ s)XlJ)),
1ER,
the covariance function of X. Similarly as i n the discrete time case we denote by H(X)
and Ht(X),
t £ R, the subspaces of L (9.,P)
X(s),
s
2
spanned by X(s),
s G R , and
S4
STOCHASTIC
PROCESSES
AND
ENTROPY
As for the spectral decomposition, we have analogous representations as i n the discrete time case. The spectral decomposition of the covariance function 7 ( t ) is given i n the form 7(0= where F(A) linu
/
e
i , A
dF(A),
teR,
(2.5.1)
is a non-decreasing right continuous function such that f ( — o o ) =
^F(X)
= 0 and F(oo)
= l i m i - a , F ( X ) = -y(Q).
I f the function F(\)
is
absolutely continuous, then (2.5.1) is rewritten in the form 7(0=
where f(X)
e f(X)d\,
f°° J
t
i,x
(2.5.2)
R,
G
—OO
is the density function defined by F(X)
— f_ f(x)dx.
The functions
x
F and / are called, respectively, the s p e c t r a l d i s t r i b u t i o n f u n c t i o n and the spectral density function (SDF)
of X . Note that, while the spectrum is dis-
tributed on the finite interval [ — ir, IT] i n the discrete time case, i t is distributed on the whole line R = ( — 0 0 , 0 0 ) in the continuous time case. The stationary process X itself has the spectral representation i n the form
x(0 = f
t <E R,
(2.5.3)
J —c
with a random measure {d(|(A); X G R ) satisfying (2.2.10).
The integral on the
right-hand side of (2.5.3) is a stochastic integral i n the same sense as i n (2.2.8). The process X is said to be purely nondeterministic i f n R ? f , ( X ) = ( 0 ) , and I 6
deterministic i f H,(X)
— Ht{X)
for all s, t € R. Similarly as i n the discrete time
case, X has the Wold decomposition. I t is known that X is purely nondeterministic if and only i f X has a SDF / satisfying
£
log/(A) dX 1 + A*
(2.5.4)
< CO.
The SDF / w i t h this property is known as a function of Hardy class. Among the SDF's, rational SDF's are important. A SDF / is called a rational SDF, i f i t is given i n the form
/(A)
where A(z)
=
B{i\)
2
A(iX)
= a [ j J L i t ? + « j ) and B(z) 0
n£.iiA+fti'
(2.5.5)
"0 = b I l j L t C * + &i) 0
M
e
polynomials w i t h
no common roots. We may assume that Say < 0 and Si/3, < 0 (ftz denotes the real
2.5 CONTINUOUS
part of z). Note t h a t , since /
TIME
STATIONARY
II
PROCESSES
/(A) dX - -y(0) < oo, the order M of the polynomial
R
A(z) should be less than the order j V of B{z).
Analogously as i n the discrete time
case, the stationary process X w i t h SDF f of (2.5.5) is said to be an autoregressive moving average process of order (JV, M ) , or simply an A R M A ( J V , M ) process. I n particular, if the SDF / is given i n the form
where
A{z)
=
a
0
\\f (z =1
a,) is a polynomial of order
+
N
such that Ret, < 0,
then the process X is said to be an autoregressive process of order N, or simply an AR(N) Let
X
process. =
{X(t);t
g R ) be a purely nondeterministic stationary Gaussian pro-
cess. I t is known that, using a Brownian motion
B
=
{B{t}\t
6 R),
X
can be
represented i n the form A-(t) =
/'
G(t-u)dB(t),
<eR,
(2.5.7)
J—o
w i t h equality H,{X)
= n,(B),
teR.
(2.5.8)
This representation w i t h causality property (2.5.8) is called the c a n o n i c a l r e p r e s e n t a t i o n i n the sense of Levy-Hida-Cramer. called the canonical kernel. Since crements,
{dB(t)\t
B
=
{B{t))
I n (2.5.7) G(i)
is a real function
is a process w i t h independent in-
£ R ) is a random measure and the integral on the right-hand
side of (2.5.7) is a stochastic integral which is known as the Wiener integral (see Sec. A.3). Intuitively speaking, (2.5.8) means that { S ( s ) ; s < t] contains the same information as { X ( s ) ; s < (}.
Hence
{dB(t))
can be regarded as the innovation
process of X. The canonical representation enable us to solve the problem of prediction. We are required to predict A"(t) (( > 0) based upon the observation of {A"(s);s < 0). The prediction error is evaluated by the mean square error. Similarly as i n the discrete time case, we denote by X(t)
the best predictor that minimizes the mean
square prediction error. T h e o r e m 2 . 5 . 1 . Let X be a purely nondeterministic stationary Gaussian process canonically represented as (2.5.7).
Given {A"(s);s < 0 } , the best predictor A"(t)
for X { t ) (t > 0) is given by X(t)=
f J-DO
G((-u)d5(u).
(2.5.9)
8fi
STOCHASTIC
PROCESSES
The corresponding prediction error e(t) (i)
=
2
E[\X(t)
-
ENTROPY
is given by
2
£
AND
= / ' |G(u)f
X{t)\ ] 2
(2.5.10)
du.
P r o o f . It is known that the best predictor is given by the conditional expectation: X{t)
= E[X(t)\X{ ),
s < 0).
S
(2.5.11)
G ( t - u) d B ( u )
From (2.5.7) and (2.5.8) we know that
£ H (B)
==
0
H (X) 0
and that /„' G ( t - u) d B ( u ) is independent of B { s ) , s < 0, and of X ( s ) , s < 0. Consequently ELY(t)| X ( ) , s < 0] = s
G(( - u ) d B ( u ) .
/
(2.5.12)
J —oo
The equation (2.5.9) follows from (2.5.11) and (2.5.12). Since
Xm
-
X U =
f Jo
G(t-u)dB(u),
using (A.3.3) we have
z(t?
f\G(t-u)\ du=
= E[\X(t}-X(t)\ )= 2
2
Jo
Thus the proof is complete.
/'|G(u)| du. 2
Jo
•
By the use of canonical representation (2.5.7) one can derive a formula for the mutual information
(f
I(X(t),Xg)
"past" A"^" = {A"(s);s < 0}.
>
0) between the " f u t u r e "
X(t)
and the
It is given i n terms of the m i n i m u m prediction
error. Unfortunately, no explicit formulas for the mutual information I{Xf, between X ,
+
= (A"(s);s > (} and X^
X ) 0
-
have been obtained, except the case where
X is the process w i t h rational SDF. T h e o r e m 2.5.2. Let X be the purely nondeterministic stationary Gaussian process w i t h canonical representation (2.5.7). Then
1
°^-i*$-f*jjfc
,>0
'
(2
-
513)
2.5 CONTINUOUS
TIME
STATIONARY
H7
PROCESSES
where 7 ( 1 ) is the covariance function of X. P r o o f . I f the conditional expectation
X(t)
= E[X(t)\X(s),
s
< 0] is given,
X{t)
is independent of X^. Hence, using (1.4) and (1.6), we can show that I(X(t),X;)
Since X(t) is given by
-
= I(X(t),(X{t),X^))
is independent of
X(t)
= I(X(t),X{t)).
(2.5.14)
the correlation coefficient
X(t),
p ( X ( t ) J { t ) f =
l
W
-
{
*
t
)
2
.
p{X(t),
X(())
(2.5.15)
The equation (2.5.13) follows from (2.5.14), (1.8.15), (2.5.15) and (2.5.10).
•
We now proceed to study multiple Markov property. Similarly as in the discrete time case we can define multiple Markov property, b o t h in the strict sense and in the wide sense. Let
X
=
{X(t);t
£
R ) be a Gaussian process, not necessarily
stationary. We say that X is differentiable i n L ( f l , P ) at ( i f there exists X ' ( t ) e 2
L (Sl,P) 2
such that
lim
fc—
0
E
±.{X(t
+
h ) - X ( t ) } - X ' ( t )
= 0.
A n d X ' ( t ) is called the derivative i n i ( S l , / * ) . 2
Definition 2.5.1.
1°) The Gaussian process
X = {X(t)}
is called an a t m o s t
JV-ple M a r k o v process i n t h e s t r i c t sense, if X is JV — 1 times differentiable in
L (Q,P) 2
at every i 6 R and i f P(X(s =
+ t)<x\X(u), P{X(s
u < i )
+t)<x\X(t),
X ' ( t ) , * < " - " ( ( ) ) ,
(2.5.16) holds for any x, t 6 R, where X^-' denotes the k-th derivative. I f X is at most JV-ple Markov b u t not at most (JV - l)-ple Markov, then X is called an JV-ple M a r k o v p r o c e s s i n t h e s t r i c t sense. 2°)
The Gaussian process
X
=
{X(t);t
€
R } is called an 7V-ple M a r k o v
process i n t h e w i d e sense, i f for any f < ij < ••• < t u < f/v+i the conditional a
STOCHASTIC
88
expectations E [ X ( t i ) \ X ( u ) , i n H(X)
butE[X(ti)\X(u),
PROCESSES
AND
ENTROPY
u < („}, t = l , . . . , i V , a r e l i n e a r l y i n d e p e n d e n t u < t ] , i = 1,...,JV + 1, a r e l i n e a r l y d e p e n d e n t . 0
Note that a Markov process is an at most 1-ple Markov process (see Def. 2.1.2). I n principle, Markov property should be a property concerning the dependency on the past of the process. However i t should be noted that, to define the Markov property i n the strict sense, we are required to assume the differentiability which seems to be unrelated to Markov property. The multiple Markov property of a stationary Gaussian process is characterized by its SDF. Although we omit the proof, we can prove the following theorem which corresponds to Theorem 2.3.3 for the discrete time processes. T h e o r e m 2.5.3. Let 1°)
X
£ R ) be a stationary Gaussian process.
— {X{t);t
The process X is an A^-ple Markov process in the strict sense i f and only
if X is an A R ( i V ) process. A n d X is an jV-ple Markov process i n the wide sense if and only if X is an A R M A ( j V , M) 2°)
process.
I f X is an JV-ple wide sense Markov process w i t h S D F / of (2.5.5), then the
canonical kernel G ( t ) i n the canonical representation (2.5.4) is determined from
Finally we give three examples. E x a m p l e 2 . 5 . 1 . Let X — ( X ( ( ) } be a stationary Gaussian process with canonical representation X(t)
e— —
= bf
(t
1
(2.5.17)
dB{xt)
J-co
(a > 0 and 6 ^ 0 are real numbers). Then the covariance function 7 ( 1 ) and the SDF /(A) are given by 7(0 =
m
-
^exp(-a|t|),
27r(A *+ a ) 2
J
2jr|iA-a| ' 2
Therefore we see that A" is a Markov process and is an Ornstein-Uhlenbeck Brownian motion (see Example 2.1.2). I t is known that, for almost all w € 11, the sample path
B(t)
— B(t,w)
of the Brownian motion is not differentiable i n the usual
S.S CONTINUOUS
sense.
TIME
STATIONARY
However there exists the derivative B(t)
PROCESSES
89
i n the Schwartz's distribution
sense. Formally speaking, (2.5.17) can be w r i t t e n as e"B(u)du.
X ( t ) = 6
+ aX(t)
=
bB{t).
This (formal) stochastic differential equation can be regarded as an autoregressive representation of the continuous time process X. It is interesting to compare this equation w i t h the A R representation (2.3.1) in the discrete time case. E x a m p l e 2.5.2. Let X be a process with canonical representation X(i)=
{e-"-
I
u |
-e
2 ( ,
- }d-B(u). u )
J-oo
Then X has the SDF /(A)
=
27r(A2 + l)(A2 + 4 ) '
Thus X is a double Markov process both i n the strict and wide sense. Similarly as in Example 2.5.1 i t is seen that X satisfies a. (formal) stochastic differential equation
JT"(0 + 3A"(0 +
2X(t) =
B(t).
E x a m p l e 2.5.3. Let X be a process with canonical representation X(t)=
{2e-"-
I
u )
-e
2 | [
- >}d£(u). u
J-co
Then X has the SDF f ( X ) n
>
_
A
'+
9
2x(A + l ) ( A 2
3
+ 4)'
and A" is a double Markov process i n the wide sense, but not i n the strict sense.
90
STOCHASTIC
PROCESSES
AND
ENTROPY
2.6 B A N D L I M I T E D P R O C E S S E S In practical systems of communication, the frequency of signals is often limited w i t h i n a finite interval. I n this section we study the processes w i t h frequency-band limited spectrum. One of the advantages of the band limited model is t h a t we can use a sampling expansion to generate the signal. Let X = {X(t);
t €
R } be a zero mean weakly stationary process w i t h covariance
function 7(f) and spectral distribution function F ( X ) . D e f i n i t i o n 2.6.1. The process X is said to be b a n d l i m i t e d to \X\ < KW', i f the spectral distribution F ( X ) satisfies the condition
For the band limited process X
=
the spectral decompositions (2.5.1)
{X(t)}
and ( 2 . 5 . 3 ) turn into 7
(t)
=
e dF(X),
t e a
e" d<(A),
teR-
/ /I-*W -.IV
itX
and x
(2.6.2)
Note t h a t , i f X has a S D F /, then (2.6.1) is equivalent to f(X)
= 0,
\X\ > TTW.
(2.6.3)
In this section we develop the sampling theorem which provides a link between continuous time processes and discrete time processes.
The sampling theorem
states that the process X = {A"(t)f, limited to the band —KW to TrW, is completely determined by giving the values X{n/W),
n
£
Z, of the process at a
series of points spaced 1/W seconds apart. We define the discrete time process X
=
{X neZ}by n]
X
n
=
x
( w ) '
n
€
Z
-
{
2
-
6
4
)
Then clearly X is stationary and its covariance function 7 „ , n £ Z, is given by 7Concerning the spectra of X and A", we shall show the following theorem.
(2-8-5)
2.6
T h e o r e m 2 . 6 . 1 . Let A" =
BAND
LIMITED
91
PROCESSES
be aweakly stationary process of band limited
{X(t)}
to |A| < i r i y . T h e n the spectral distribution function F ( \ ) of X is related to the spectral distribution function F ( A ) of X by F ( A ) = F(W\),
-n<\<w.
(2.6.6)
In particular, i f A" has a SDF /, then X also has a SDF / such that /(A) = Wf(W\),
< A < TT.
(2.6.7)
P r o o f . By (2.6.5) we have
jrW
/ =
•itW
/
e *dG(A), in
neZ,
J—it
where G(A) — F{W\).
Using the uniqueness of the spectral distribution function,
we obtain (2.6.6). Since /(A) = ward of (2.6.6).
j^F(X)
and /(A) = ^ F ( A ) , (2.6.7) is straightfor-
•
Since the spectrum of the process X vanishes outside the interval [—irW, JrlF], Theorem 2.6.1 means that F (resp. /) is completely determined by F (resp. /). Therefore i t is a natural consequence that all the values of { A ( t ) J are reconstructed from { A n } . The precise formula of the sampling expansion is given i n the following theorem which is known as the s a m p l i n g t h e o r e m . T h e o r e m 2.6.2. Let A = { A ( t ) ; t 6 R ) be the same band limited process as in Theorem 2.6.1, and assume that the spectral distribution function F ( X ) is continuous at A =
±ir\V.
Then, for each ( £ R,
X
¥i
=
fej^
*
X(t)
W
is represented i n the form
( t - n W - i )
X
n
'
(
2
' ' 6
8
)
where A = (AT„} is the process defined by (2.6.4) and the l i m i t is taken i n the sense of mean convergence. Proof.
For notational convenience, we put
y2
STOCHASTIC
PROCESSES
AND
ENTROPY
By ( 2 . 5 . 1 ) , i t is easily seen that
E n--N f
TT Wl
e
i
J-nW
,
x
- Y f A t y - ^
dF(X).
(2.6.10)
-N
For every fixed [, as a function o f A 6 [ — t W , v W \ i
the function e " * can he
expanded i n the Fourier series s
= lim Y
ia
n
-irW < X < ww.
(2.6.11)
i=-N
The right-hand side converges to cos7rWt i f A = ±irW.
Since the integrand in
( 2 . 6 . 1 0 ) is bounded i n A and F ( A ) is continuous at ± T T W , i t follows f r o m ( 2 . 6 . 1 1 )
and Lebesgue's bounded convergence theorem that the right-hand side of ( 2 . 6 . 1 0 ) converges t o zero as JV —> oo. Thus ( 2 . 6 . 8 ) holds i n the sense o f mean convergence.
•
We note that, i f the band limited process A" has a SDF /, then F(X) = y_
f(x) dx satisfies the continuity assumption in Theorem 2 . 6 . 1 . So far we have assumed the stationarity of the processes. However a sampling
theorem also holds i n a class of nonstationary processes. Let us consider a nonstationary process X = ( ^ ( ' ) ) whose sample path X(-,u>)
= {A"(t,u);t g R }
satisfies the condition A(-,w) e L ' ( R ) n L*(R)
w i t h probability one, where L / such that j \f(x)\ R
p
F
(2.6.12)
( R ) denotes the space of all measurable functions
dx < oo. Then the Fourier inverse transform £ ( - , w ) of
X( • ,w) exists. We assume that the frequency of the Fourier inverse transform is limited to |A| < nW, so that X = (X(t)}
X{t,w)
=
/
is represented i n the form
e"^(A,^)dA,
t<ER,
(2.6.13)
w i t h probability one (cf. ( 2 . 6 . 2 ) ) . Note that a stationary process never satisfies (2.6.12).
2.7 DISCRETE
TIME
OBSERVATION
T h e o r e m 2.6.3. If a process X = each ( e R the sampling expansion
OF CONTINUOUS
92
satisfies (2.6.12) and (2.6.13), then for
{X(t))
Ji= -
PROCESSES
CO
holds w i t h probability one, where V n ( 0 is the function of (2.6.9).
P r o o f . Using (2.6.13), i t can be seen that £(A) has a Fourier expansion
— DO
The expansion (2.6.14) follows from (2.6.13) and (2.6.15).
•
It should be noticed that, if a process X = {X(t);t £ R } is time limited, namely X{t) = 0, |t| > T, then X = {X(t)} is neither stationary nor a process w i t h property (2.6.13), I t is known that the sampling theorem cannot be applied to time l i m i t e d processes.
2.7 D I S C R E T E
T I M E OBSERVATION
OF
C O N T I N U O U S T I M E PROCESSES Let us consider a stochastic process
X
— \X(t);
t
£ R ) as a model of a random
phenomenon. A l t h o u g h the phenomenon itself is with continuous time parameter, in practical situations i t is not a rare case that the observation is made only discretely. For simplicity, suppose that we make observations once every k seconds. Then the mathematical model of the observed values is given by A"' ' = 11
X(nh);
{xi* ^ 1
=
n e Z } . The observed process A ' ' ' may be called the discretization process 1
of X w i t h time interval k. As was shown i n the preceding section, i f X is a process of band limited to |AJ < KW, then all the values of X(t), process
X ^
denoted by
= X
t (E R, are recovered from the discretization
j J f ' ; r c € Z } , where A )
= {A„}}.
k
= 1/W ( i n Sec.
2.6 X ' > = A
I t goes w i t h o u t saying that, i n general,
{X(t);t
is
{X } ( h) n
g
R)
cannot be recovered from A " ' ' , and we want to estimate the statistical structure A
STOCHASTIC
94
PROCESSES
AND
ENTROPY
of X = ( X ( t ) } based on the information from the observed process X K
For
fk
this purpose i t is important t o investigate the relationships between statistical properties of X and X Let X = {X(t)}
w
.
be a purely nondeterministic stationary process w i t h covariance
function 7{() and SDF /(A). Then the discretization X
| f t )
is clearly a discrete time
stationary process w i t h covariance function yW=f(nk),
neZ.
(2.7.1)
Hence we have oo e"^f{X)d\
/
-co
This means that the SDF f (X)
of the process X W is given by
lh,
/C0(A)=i
/ ( ^ ^ ) -
f jt=—oo
^
- ' ^ < ,
(2.7.2)
'
Note that (2.6.7) i n Theorem 2.6.1 can be obtained as a special case of (2.7.2). Now let us study the Markov property of processes X and X * ' ,
A t first we
fc
consider a simple Markov stationary Gaussian process X = (X(t);t
6 R}.
By
Theorem 2.5.3 the SDF / is given i n the form 2
= A^Tfc "' 2
- o o < A < oo
(2.7.3)
(a, b > 0). I t is easily seen that the covariance function 7 ( 1 } is given by
7(0=^V
b|1
'
(2.7.4)
(see Example 2.5.1). The SDF / *>(A) of the discretization process X * ' = (
1
is equal to /W(A) = ^
£
^
^
= 1
£
7(nn)e-
{l,)
{X }
S.7 DISCRETE
TIME
OBSERVATION
OF CONTINUOUS
PROCESSES
95
Using (2.7.4), we can calculate the series on the right-hand side to obtain
Therefore, by Theorems 2.3.1 and 2.3.3, the process
X< > h
is an AR(1) process, so
is a simple Markov process. Thus we have known that the simple Markov property is inherited from X to
X . ik)
I t may be interesting to show that, as for the multiple Markov property, the wide sense Markov property of a stationary Gaussian process is inherited from X to X ,
whereas the strict sense Markov property is not necessarily inherited.
w
T h e o r e m 2 . 7 . 1 . Let X — {A"(();( £ R ) be a purely nondeterministic stationary Gaussian process and
X
i
k
X
the discrete time process
X
£ Z ) be the discretization process. I f the
{xi ;n
—
i
continuous time process
h>
=
=
W
is JV-ple Markov in the wide sense, then
[X(t)}
is also JV-ple Markov i n the wide sense.
{X } i l,) n
On the other hand, even if X is multiple Markov i n the strict sense, J f ' ' is not fi
necessarily multiple Markov in the strict sense.
P r o o f . Let
X
=
be JV-ple Markov in the wide sense. Then the SDF
{X(t)}
/ ( A )
of X is given i n the form (2.5.5). Since, even if the polynomial A(z)
=
has multiple roots, we can prove analogously, we assume that A(z)
has only simple
PIjl=]( "t" j) z
Q;
roots. T h e n / ( A ) can be rewritten in the form
where a, = Sicij > 0. I n the same manner as i n deriving (2.7.5) from (2.7.3), one can show that the SDF f (X)
of A"" ' is given by
lh)
Thus X
fh)
1
has a SDF of the form (2.3.8) and is an A R M A ( J V , m ) (m < JV) process.
To prove that the strict sense Markov property is not inherited from X to A " , ( k )
it suffices to give such an example in case of JV = 2. Let a stationary process X be double Markov i n the strict sense. Then the SDF / ( A ) is presented i n the form
F
W
=
A^ToT
+
STOCHASTIC
96
PROCESSES
AND
ENTROPY
where + c = 0,
C l
e , ^ + c a\ = C i H " 4)
2
and we assume that C « I , Q
C - - )
* °.
2
2
7
7
are positive numbers. Then, by (2.7.6), we can show
2
that the SDF of A ' ' ' is given by (
,
m
- Hi f
m
7
V
'
' " - - I
-
2 \o,|e'""-e' P
\
1
a je'""-e^P J
A
2
c ( A i - 2,4.2 cos A) t
~ 2a where A [ , A
hc
ix
ha
-e'*)| ' 2
are constants independent of A, particularly
2
A
2
Thus X^
\{e " - e )(e *
ia2
-
a,e '(e ka
- 1) - a e <
iha>
1
ha
(e
2 f c a
' -
1).
is double Markov i n the wide sense. However, by Theorem 2.3.3, i n
order that X -^ 1
is double Markov process i n the strict sense A
2
should be zero.
This occurs if and only i f «i
^"•(e
=
Since the function g(x) = exp(hx) (2.7.8) holds only i f a, =a . 2
in the strict sense,
2 A
°'-l)
=
e""' -
e- "" 1
— exp(—hx) is concave on (0, co) and g(0) = 0,
However this contradicts w i t h (2.7.7). Consequently,
cannot be double Markov.
•
This theorem suggests that the wide sense Markov property is rather satisfactory, especially i n continuous time case, to discuss multiple Markov property.
H I S T O R I C A L
N O T E S
There are many suitable references for the general theory of stationary processes, among them Box and Jenkins [16], Doob [32], Ibragimov and Rozanov [58], Priestley [98]. The spectral representation and the moving average representation are treated i n these texts. The canonical representation of Gaussian processes in the sense of Levy-Hi da-Cramer is studied in Hida and Hitsuda [51] in detail. The concept of multiple Markov property i n the wide sense was introduced by H i d a
HISTORICAL
NOTES
97
[50] (see also [51]). The definition of multiple Markov property i n the strict sense of a continuous time process is due to Doob. Formulas for the entropy rate and the mutual information of Gaussian processes were first derived by Shannon [103] (see also Gallager [43]). Entropy rate, relative entropy rate, and mutual information rate are studied in Pinsker [97] in a rigorous way (see also [45, 79]). Theorem 2.2.3 is due to Kolmogorov (see [51]). To derive the formula (2.4.19) for the relative entropy rate of a Gaussian process, we have examined the asymptotic behavior of eigenvalues of Toeplitz matrices. The formula (2.4.17) has been obtained i n Kac, Murdock and Szegd [72].
See also [47, 48]
concerning the asymptotic behavior of Toeplitz matrices. The sampling theorem (Theorem 2.6,2) is due to Shannon [103].
Chapter 3 M a x i m u m E n t r o p y Analysis 3.1 M A X I M U M
ENTROPY
MINIMUM
RELATIVE
AND ENTROPY
The m a x i m u m e n t r o p y p r i n c i p l e ( M E P ) was first introduced by Jaynes [71] in 1957. I n his paper, he stated that the M E P justifies the statistical inference in statistical mechanics. I t is also demonstrated by Kullback [82] t h a t the problems of minimizing relative entropy play a basic role i n the information theoretic approach to statistics. The M E P and M R E P ( m i n i m u m relative entropy principle) have been applied w i t h considerable success, i n various fields such as statistical mechanics, statistical inference, time series analysis, probability theory, and so on. Let us consider a random variable X .
We are given only some partial infor-
mation about the probability distribution of X . The M E P states that we should choose the probability distribution that is consistent w i t h the given information but maximizes the entropy. Since the entropy is a measure of uncertainty of probability distributions, the increase of information about the distribution results i n the decrease of entropy. Therefore, we may say that the M E P is the probability assignment, which is consistent w i t h the available information b u t is maximally noncommital w i t h regard to missing information. At first, we consider a discrete random variable X set X = { « i , . . . , a } . m
taking values i n a finite
Suppose that we are given some partial knowledge about
the distribution of X , and denote by F the set of all probability distributions which satisfy the given information. For a discrete probability distribution n = (Pi)-,J>m)
t h e
entropy is given by H{/i)
= - "£?
=l
as
P. f°gPi (see Sec.
1.2).
The
S.I
MAXIMUM
ENTROPY
AND MINIMUM
RELATIVE
ENTROPY
99
M E P says t h a t i n making inferences on the basis of partial information we must use the probability distribution v which has the maximum entropy i n F : H(u)
> H(tt),
¥/i € F .
This is the only unbiased assignment we can make.
(3.1.1) Because to use any other
would amount t o arbitrary assumption of information which by hypothesis we do not have.
T h e distribution V £ F is called the m a x i m u m e n t r o p y ( M E )
d i s t r i b u t i o n i n F . We should note here that, using (1.4.4), (3.1.1) is equivalent to H(D-,v )
V/i£F,
(3.1.2)
where v is the uniform distribution on X : f o ( { a i } ) = •-• = "a{{o. }) 0
m
— 1/m.
In other words, maximizing the Shannon entropy is equivalent to minimizing the relative entropy w i t h respect to the uniform measure. Next, we consider the same problem for continuous distributions. The entropy h(n) of a continuous distribution w i t h probability density function p { x ) is given by A(/i) — k(p) = — / p ( z ) logp(z) dx (see Sec. 1.3). Let F be a class of continuous R
distributions on an interval R
0
C R satisfying given constraints. The problem is
to find a distribution V £ F which has the maximum entropy in F : H?)>h(p),
V^eF,
(3.1.3)
where q{x) and p(x) are probability density functions of P and ft, respectively. I f R
0
is a finite interval, then using (1.4.7), again we know that (3.1.3) is equivalent
to (3.1.2), where Va is the uniform distribution on R n . In this section we study mainly the m i n i m u m relative entropy (MRE) analysis. As we have seen above, the maximum entropy analysis can be treated as a special case of the M R E analysis. To formulate our problem, we denote by M i the set of all probability measures on a measurable space ( X , 5 ( X ) ) . Given a reference measure v £ M ] and a subset F C M l i the problem is to find a measure v £ F which minimizes the relative entropy H( • ; v) in F : H(V;
v) < H(pt; i/),
V n £ F.
(3.1.4)
The measure 9 is called the m i n i m u m r e l a t i v e e n t r o p y ( M R E ) d i s t r i b u t i o n w i t h respect to v. The reference measure v may be regarded as a prior probability distribution. The prior distribution may be given on the basis of experience or i t may be obtained from theoretical considerations. I f the prior distribution is not known, then we
inn
MAXIMUM
ENTROPY
ANALYSIS
may take the uniform distribution as the reference measure, and then minimizing relative entropy is equivalent to maximizing entropy. The class F represents the set of all probability distributions consistent w i t h available information. T h e n the m i n i m u m relative entropy method is to choose the M R E distribution v as the optimal estimate of the unknown distribution. I n other words, i t is to choose the most similar distribution, in the sense of relative entropy, to the prior distribution. In the maximum entropy analysis we have to treat separately the case of discrete distributions and that of continuous ones, because the entropies are given i n the different forms. On the other hand, i n the M R E analysis, since the space X may be taken arbitrarily, a unified treatment is possible for any k i n d of probability distributions. This is surely an advantage of the M R E analysis. For convenience, we put J7(F;j/) =
inf{r/(u;i/);ueF}.
If there exists a measure F t F such that H(V; „) =
H(F;v),
then V is the M R E distribution and it is often called the H - p r o j e c t i o n of v on F. The concept of H-projection was first introduced by Csiszar [24] who has called it the I-projection because i n [24] the relative entropy is called the I-divergence (information divergence). Note that the H-projection does not always exist, and is not necessarily unique even i f i t exists. Concerning the existence and the uniqueness of H-projection, the following theorem is fundamental. T h e o r e m 3 . 1 . 1 . Let F C M , be a convex set such that 7f(F;i/) < oo. 1")
I f the H-projection of u on F exists, then i t is unique.
2°)
I f F is closed in variation, then there exists the H-projection v of c on F,
P r o o f . 1°)
Let V\ and v 2
H(v\u)
be H-projections of v on F.
2
v = j ( t ' i +P )
Then, by convexity,
belongs to F. Therefore, by Theorem 1.5.1, we know that H(F;
< £ ( # ( ? , ; K ) + H{v \v)) 2
= H(F;v).
u) <
Here the equality holds if and only
= u (see 2") of Theorem 1.5.1). Thus the H-projection is unique.
if
2
2")
There exists a sequence (fi„} C F such that lim H(«„;*)
= H{F;v).
(3.1,5)
I t is easily seen that H{ii ;u)
+
m
= 2 H (n , ;v) m
n
H^v) + H {n ;ii ,„) m
m
+ H(p: ;ii , ), n
m n
(3.1.6)
S.I MAXIMUM where u
m
, — ^(ii
m
n
1.5.1, 2Z?(jU ,»; m
<
v)
ENTROPY
AND MINIMUM
RELATIVE
101
ENTROPY
+«„)•
Since F is convex, /i „ belongs t o F. By Theorem
H(ii \i/)
+ i 7 ( w „ ; u ) . Therefore, the last two terms on the
rai
m
right-hand side of (3.1.6) must converge to 0 as m , n —• oo. By Theorem 1.5.4, we have < \\lt ~ U
\\Hm ~
m
< [ 2 J f (
N
miIl
;
| | + \\p - fl ,„||
F
n
v
) ] "
J
m
+
1/2
l2H(n„;n , )} . m n
Hence one concludes that \\fi — ti„\\ converges to 0 as m , n —i oo. Since F is m
closed i n variation, there exists a measure P £ F such that lim i V „ - P | | = 0 so that H ( F ; v) < H(V; u) <
u) =
l i m H(n ; n
H(F;!/),
where we have used lower semi-continuity of relative entropy (Theorem 1.5.5). This means that P is the H-projection of v on F.
•
We shall give a condition i n order that a measure P is to be the H-projection of V on F. This condition will play a fundamental role to find the M R E distribution. We define subclasses S(v,r)
and
S{u, r) = {pe
FQ UI
of M , by
M ; j f t p ; i/) < r]
(0 < r < oo)
t
and T ,p v
Note that S(v,r)
£ M , ; jf
=
log ~(x)dfi(x)
> R(?;v)j.
(3.1.7)
and F f are convex. The geometrical interpretation is given i n w
Fig. 3.1. I n the following, we simply write /ipdv or / ^ ( i ) d i / for
j f(x)dv(x), x
if no confusion occurs. T h e o r e m 3.1.2. Let F C M] be a convex set such that H(F;v)
< oo. Then a
measure P £ F Pi 5 ( f , oo) is an H-projection of u on F if and only if FnS(i/,oo) C F„, . p
(3.1.8)
In particular, P is the H-projection of v on F „ ^ . P r o o f , ("if" part)
I t suffices to show the inequality H{S;v)
< H{frv),
V(i£F.
(3.1.9)
102
MAXIMUM
ENTROPY
ANALYSIS
We may assume that H(ft; t>) < oo, so that fi -i f . Since /i £ F C F ^ e ,
Thus we have obtained (3.1.9). ("only i f " part)
Let P E F (7 S(e, co) be an H-projection of v on F . We prove
(3.1.8) by contradiction. Assume (3.1.8) were false. T h e n we would have a measure Pi S F such that H(pi; v) < co and p\ ^ F , i ; , so that v
/ log ^dUt-j Define a measure u
Q
log ^ d P < 0 .
(3.1.10)
(0 < « < 1) by n
Q
= q/ji + ( 1 - a)?.
Note that /i„ 6 F , since F is convex, and that p = v. We p u t Q
and ¥><«(*) = P o ( ^ ) l o g P o ( « )
-Pl*)logpW-
Then i t is clear that H(p ;v) 0
- H(u; u) = j
{x)dv.
(3.1.11)
Va
Since p„(x)
= api(x)
+ ( 1 - <*)p(x) = a ( p i ( x ) - pT*)) +
ftx)
is linear i n a and since r log t is a convex function of f > 0, for every fixed x € X , j i / ) ( i | is increasing i n q > 0 and 0
l i m -
lim-{H{p ;v) ala a 0
- H{v;v)}
= (Pi(x) -
=
ftOXfasft*)
+ !)•
f ]im-(p (x)dv J aio a a
x
= J( i(x)-p(x))(lag,p(x) P
=
/
1
° s f
+ l)dv
M - J h g § &
(3- - ) 1
12
3.1 MAXIMUM
ENTROPY
AND MINIMUM
RELATIVE
103
ENTROPY
Therefore B{p ;v)
<
a
for sufficiently small a > 0. Since ji
a
B(v;u)
6 F, this contradicts w i t h that the measure
V is an H-projection of v on F. Therefore (3.1.8) should be satisfied. Take the set F „
i C
as the set F. Then obviously (3.1.8) is satisfied and we know
that v is the H-projection of u on F „ n.
•
F i g u r e 3 . 1 . H-projection of u on F
I t may be of interest to give a geometrical interpretation (see Fig. 3.1).
We
denote by 3F„, = | / e M ; ^ I o g ^ ( r ) d i ( ) = S
1
/
1
I
7f(P;f)J
(3.1.13)
the " b o u n d a r y " set of F 5 . B i
T h e o r e m 3.1.3. Assume that H(V; i/) is finite. Then for any fi £ 9 F j ; i t holds V i
that JTfftw) = H( ;V) fi
P r o o f . Let fi e 3 F
t i S
. Then clearly
+ H(V;v).
(3.1.14)
104
MAXIMUM
ENTROPY
ANALYSIS
Note that (3.1.14) is an analogue of Pythagoras' theorem. Therefore Theorem 3.1.3 states that the relative entropy # ( • ; • ) , which is not a metric at all, behaves like the square of the Euclidean metric. Moreover the set dF^v
forms the "tangent
hyperplane" of the "sphere" S(i/,r) at P, where r = H{u\ v). One of the most important cases to apply the m i n i m u m relative entropy method is the case where the available information about a random variable A" is given in terms of average values fl(X),
of several random functions
E[f\(X)],E[f (X)] n
Here / i ( z ) , / „ ( i ) are real functions defined on X . For this
...,/ (X). n
case, Theorem 3.1.2 can be applied to find the probability distribution which minimizes the relative entropy. I t is shown that the distribution has a density of exponential type w i t h respect to the reference measure. To state the result, we introduce some notations. Define functions fli(A), i = l , . . . , n , of A = ( A i , . . . , A ) £ R " n
by
^ ^ ^ / x ^ ' ^ ' P ^ ' ^
^
1
1
' '
i = l,..,n,
(3.1.15)
where 4(A) = / e x p ^ A , / ; ^ ) } ^ ) -
(3.1.16)
We denote by A the set of all A for which the integrals i n (3.1.15) and (3.1.16) converge, and define a set A C R™ by
•4={(!?i(A),...A(A));A€A}. T h e o r e m 3.L.4. Let
v
£ M ] be a fixed reference measure, and let
fi{x),
i
=
1 , n , be real functions defined on X . Assume that F C M j is a set of the form
F = > e M i ; LfMM*)
= «i,
*= 1
n),
where a;, i = l , . . . , n , are given constants such that ( a . . . , < ! „ ) £ A. lt
(3.1.17) Then the
H-projection V on F is given by 1
du
£
(
T
)
= „ A l
" e
X
p
(
E
( 3 1 l g )
where A t= (A],...,A„) £ A is a vector uniquely determined by solving 1
/ J
x
" /*(*) e x p { £ J %lfM}i^)
•»«%»
I - I , . . . . n.
(3.1.19)
S.l MAXIMUM
ENTROPY
AND
MINIMUM
RELATIVE
105
ENTROPY
The corresponding minimum relative entropy is given by n
H(u; u) = E i « i - l o g * ( A ) .
(3.1.20)
A
i=i
P r o o f . Since ( f l i , ...,a„) g A, there exists A = ( A , , A „ ) satisfying (3.1.19). For the measure V defined by (3.1.18), the equation (3.1.20) can be easily derived by using (3.1.18) a n d (3.1.19). Noting (3.1.20) and
L
i
o
d%
§
{x)
M
we know that the set F s
x
= f ^
)
X
i
L
u
{
x
)
M
x
)
~
i o e
*
( a )
-
of (3.1.7) is described i n the form
v
n
F „ , = {p€
n
M i ; £*Ai I
s
fi(x)
dp(x)
> £
X% m
Since n
-
i=l
J
n
i=l
x
for every /i e F , i t is clear that F C F„, . c
Therefore, by Theorem 3.1.2, the measure V is the H-projection on F.
•
More generally, the H-projection is also given i n the form (3,1.18), even if the class F is presented as F.= L
e M i ; (J
h&M?)
f
/ « ( * ) « W * ) ) & -4 J ,
(3.1.21)
in place of (3.1.17), where A is an arbitrary set i n R™. We shall prove this fact i n a special case of n = 1, Let us consider the case where the available information is given i n the form c < E[f(X)\
< d,
with a real function / defined on X . The function *(()
= J^exp{tf(x))Mx),
(3.1.22)
106
MAXIMUM
ENTROPY
ANALYSIS
is called the moment generating function of v. We assume that there exist — co < ti < 0 < t-i < oo such that
f f { x ? e x { t / ( i ) } dv(x) < oo, V t € ( t , , t ).
* ( t ) < oo,
P
2
Jx
Then i t is easily seen that f | / { z ) j e x p { f / ( 3 ) } du{x) Jx
< co,
ti < t <
t, 2
so that $ ( t ) is differentiable and *'(()
=
/ /(z)exp{t/(*)}<Mz),
We define constants Ci and c Ci =
sup
2
d < t < l
2
.
by $•(() —--—
«,«<<, *(*)
and
c = 2
. , ml
$'{t) —, •
''<•<" *(0
T h e o r e m 3.1.5. Let C] < c < c be a given constant. If / 2
x
f(x)
dv{x)
< c, then
the H-projection P of v on F = {(i€Mi;e<
/ /U)dp(i) < d}. JX
(3.1.23)
is given by
where t is a constant uniquely determined by = c.
/ f{x)dV{x) JX
(3.1.25)
The m i n i m u m relative entropy is given by
J/(P;i-) = If J
X
f(x)dw(x)
fc-log$(t).
> c, then the same result holds for a set
F =
{/i6M ;6<^/( )d 1
I
f i
(z)
(3.1.26)
3.1 MAXIMUM
P r o o f . We assume /
ENTROPY
x
AND MINIMUM
f(x)du{x)
-S 8 D -
=
i s
increasing i n
ENTROPY
t £ (t t ). u
2
107
Since
< c.
I (If) m J *'(*)/*(') such that
RELATIVE
M x )>
°-
Therefore there exists a unique t £ ( f j . t j )
*'(*) _ *(,)
" C
< c, the constant t is positive. Since we have assumed that * ' ( 0 ) / 4 ( 0 ) = / f{x)dv Using this t we define a probability measure u by (3.1.24). Then i t is easily shown that v satisfies (3.1.25):
Therefore V 6 F . Now, for any /i £ F , i t is easily seen that
/ S ; £ ; ( z ) a V = j{a l o
where a = — log 4 ( f ) .
H(v\
V) =
+ tf(x)}dp
= a + t
J f(z)d(i
> a + tc
Moreover,
log-^-{x)
j
Hence F C E * ^ r , where F
V
dO — a + t J
f{x)dv
= a +
tc.
$ is the set defined by (3.1.7). Consequently, by Theo-
rem 3.1.2, we see that v is the H-projection of v on F .
•
We have already shown i n Theorems 1.8.4 and 1.8.5 that, i f a class F is given i n terms of covariance functions, a Gaussian distribution maximizes the entropy i n F and i t minimizes the relative entropy w i t h respect to another Gaussian distribution. I t is clear t h a t Theorem 1.8.5 can be obtained as a corollary of Theorem 3.1.4. In the sequel, applying Theorems 3.1.4 and 3.1.5, we shall give several examples of M R E distributions. Moreover, we see that these examples provide characterizations of some probability distributions such as Gibbs distributions, binomial distributions, Poisson distributions, etc.. The first example was examined by Jaynes (71) i n applying the M E P to statistical mechanics.
108
MAXIMUM
ENTROPY
ANALYSIS
E x a m p l e 3 . 1 . 1 . (Gibbs distribution) Let X = { s , , . . . , s w } be the set of all possible states and denote by e ( s i ) the energy of a particle sitting on the state Si. Based on observations we know the average value e of the energy, and this is all the information available for us. Then the set of a l l distributions subject to the constraint is given in the form w
F = { ; ] T < ) M { * . - } ) = e-}t f
e
S i
1=1
The distribution V which has the maximum entropy in F is given by
where t is uniquely determined by 1
W
The distribution V is called the Gibbs distribution which is known as the equilibr i u m distribution.
Let X — { 0 , 1 , . . . , « } and let v be the
E x a m p l e 3.1.2. {Binomial distribution)
binomial distribution of parameter ( n , p), where 0 < p < 1 is a constant. The distribution v is defined by
^ ) ) = ( ; ) / ( l - P r
l
* = 0,l,...,n,
and the mean of v is equal to np. I f the set F is given i n the form n
where 0 < o < b < 1 are constants, then the H-projection v of v o n F is also a binomial distribution of parameter ( n , a), ( n , p ) , and ( n , 6), accordingly as p < a, a < p < 6, and b < p, respectively.
3.1 MAXIMUM
ENTROPY
AND MINIMUM
RELATIVE
ENTROPY
109
E x a m p l e 3 . 1 . 3 . (Poisson distribution) Let X = { 0 , 1 , 2 , - . } , and let v be the Poisson distribution of parameter A, where A > 0 is a constant. The distribution v is given by
and the mean of v is equal to A, If the set F is given i n the form oo
F={n;a
E x a m p l e 3 . 1 . 4 . (Exponential distribution) Let X = [0, oo) and let v be the exponential d i s t r i b u t i o n of parameter A > 0. The distribution v is the continuous distribution on [0, oo) w i t h probability density function q(x)
=
XeT ", 3
x>0.
If the set F is given i n the form
where p(x) denotes the density function of /i and 0 < b < a are constants, then the H-projection P of v on F is also an exponential distribution of parameter b, A, and a, accordingly a s A < 6 , v < A < o , and a < A, respectively.
Examples 3.1.2, 3.1.3 and 3.1.4 state that, if the prior probability distribution is binomial, Poisson, or exponential, and the available information is given in terms of the mean value of the distribution, then the M R E distribution is again binomial, Poisson, or exponential, respectively.
MAXIMUM
110
3.2 L A R G E D E V I A T I O N
ENTROPY
ANALYSIS
THEOREMS
The large deviation theorem is a kind of limit theorem i n probability theory, and i t is closely related to the m i n i m u m relative entropy analysis. To explain the large deviation theorem, we start w i t h the law of large numbers. Let { X ; n — 1,2,...} be a real valued i.i.d. n
denote by m = -EpO,] =
process w i t h distribution v. We
the expectation and by
f xdi>(x) R
a
1
= E{(X
—
n
m) ] 2
= J ( i — m ) du(x) the variance. We are interested i n the asymptotic behavior of R
2
the sample mean ^L = I(A-,+•••+*-„). n
(3.2.1)
n
It is well known that S / n "converges" to the expectation m . This property is n
known as the law o f large n u m b e r s .
We give the precise statement of the
theorem without proof. T h e o r e m 3 . 2 . 1 . Let { X „ , n = 1,2,...} be an i.i.d. process w i t h distribution e and assume that the second moment -ELY ] = J z dv{x) is finite. 2
R
2
1") (Weak law) For any * > 0 lim
n—.oo
2°) (Strong law)
P
^ - m | < ^ = l .
(3.2.2)
I t is true that P f j m
^
= m j = 1.
(3.2.3)
The convergence i n (3,2,2) is called the convergence i n probability, and that (3.2.3) is the convergence w i t h probability one. I n general, i f a sequence of random variables converges w i t h probability one, then i t converges i n probability. Let (a, 6) be an interval. Then i t is clear from (3.2.2) that
taP^MiWi' y n
n-<x>
l Vff m
/
( 0,
a
m f
\e,b].
Moreover we may expect that, if m ^ [a, 6], then the probability P(S„/n goes to zero exponentially: P (J^- € (",6)^ ~ e x p ( - J n ) ,
as n - » oo,
(3.2.4, g (a, &))
3 2 LARGE
DEVIATION
111
THEOREMS
where / is a constant. More precisely, under some moderate conditions, i t will be proved later that
where I(A)
(3.2.6)
= rzd{I(x);xGA}
a n d 7 ( x ) is a function called the entropy function or the rate function (see Theorem 3.2.6). I t will also be shown that the entropy function is closely related to the relative entropy. We should note here that the limit l i m , , ^ ^ M o g P ( S / n 6 A) n
does not always exist. I n general, i n place of (3.2.5), we may expect to prove the upper estimate: Urn" - l o g P f &
* i — n
g F ] < -/(F),
\ n
(3,2.7)
/
for every closed (or compact) set F , and the lower estimate: l i m i l o g P (h. for every open set G.
G ] > -1(G),
e
I t is said that { S } N
(3.2.8)
has the large d e v i a t i o n p r o p e r t y
if (3.2.7) and (3.2.8) (or (3.2.5)) hold.
Especially (3.2.5) is called the Sanov
p r o p e r t y . I f the entropy function I{x)
is continuous, then (3.2.5) follows from
(3.2.7) and (3.2.8). We have explained the large deviation property i n the i.i.d. case. However i t should be noticed that the large deviation property holds more generally. Indeed, the large deviation theorems have been proved not only for the i.i.d. processes hut also for Markov processes and a class of stationary processes. Intuitively speaking, the large deviation theorem tells us the rate of convergence to zero of the probability. I t w i l l play important roles i n evaluation of error probabilities i n statistical inferences, for example, the error probability in hypothesis testing or the error probability of information transmission.
We are more
interested i n (3.2.5) than i n (3.2.7) and (3.2.8). Because, from the viewpoint of application, (3.2.5) seems to be much more useful than (3,2.7) and (3.2.8). Here we shall show a large deviation theorem only for a discrete time process, but not necessarily Markov nor stationary. Let us consider a sequence {S„}
of real
random variables, which are not necessarily sums of i.i.d. random variables. No assumptions are made about the dependency structure of the sequence. To state the theorem, we need some preparation. Define a function <^ (i) by n
ilog£[exp(<S )] = i l o g / «p(tx)A/ (x), n
11
t 6 R,
(3.2.9)
112
MAXIMUM
ENTROPY
ANALYSIS
where f „ is the probability distribution of S
n
(n = 1, 2,...). T h e function ip„ is
convex. Indeed, for any s, t € R and 0 < a < 1, using Holder inequality, we have E[exp{(as + 0 t ) S n ) ] = £ [ e x p ( a 5 5 ) e x p ( ^ r 5 ) ] n
n
< {£[exp( S )]) (E[exp(fS )])" , 3
a
n
n
where /? = 1 — a . Therefore
+ <9i) < ay (s)
n
0 (t)
+
a
Vn
meaning that ip„ is a convex function. Define
l i m
(3.2.10)
n
provided that the l i m i t exists. The function ip is also convex on the set where f(t) exists. Consequently,
is non-decreasing if
ip'(t)
tp(t)
is differentiable. We denote
by V the class of all ( € R such that the l i m i t (3.2.10) exists, tp(t) < oo, f(t) is differentiable and
and define a function ip by *(*) =
tSi>.
(3.2.11)
L e m m a 3.2.2. The function ^ ( t ) is non-decreasing on the set {( <E T>; t > 0} and non-increasing o n { t 6 D ; f < 0) Proof.
Let 0 < t < t + A and f, ( + k € V. Then there is a constant 0 < 0 < 1
such that
v»(* +ft)- v(0 = Wit Therefore, noting that
is non-decreasing on
+ T>,
oh).
we have
ip(t + ft)- -/>(*) = % ' ( < + ft) - *>'(*)} + V ' ( « + ft) - M * + ft) =
t{
+ ft) - (()} +
h{ '(t V
+
ft)
-
{t)}
v
0ft)}
> o. In the same way, we can show 1>(t +
h)-if>{t)>0,
for any t , t + ft £ 73 such that t + ft < t < 0.
•
We prepare a simple but useful lemma which follows from the Chebyshev i n equality.
3.S LARGE
DEVIATION
113
THEOREMS
L e m m a 3 . 2 . 3 . Let A" be a real random variable. Then P(X
>x)<
E[exp(tX)]exp(-<x),
(3.2.12)
for a l l t > 0 and a: 6 R.
Proof.
Denote by v the probability distribution of X. Since t(y - x) > 0 for all
f > 0 and y > x, we have P ( X > x ) <
/
e p{t(y
<
-
X
J[t
l 0 O
x)}
dv(y)
)
/
expiry-x)}d^{y)
= ElexpftA")] e x p ( - l x ) . Thus we have (3.2.12).
•
Now we are i n a position to prove the Sanov property. T h e o r e m 3 . 2 . 4 . Assume that 0 £ D and ^'(0) £ i n t ()), where int [if>'{V)) denotes the set of the interior points. 1°)
i f a < ip'(0) < 6, then l i m -\ p(?H. e(a,b)\ = l i m - hgP «—oo n \ n j n—oo n os
2°)
[^ €fab]\\ = 0. \ n J
(3.2.13)
I f v>'(0) < a and a £ int(i/(Z>)), then
l i m - l o g p f ^ €{a,b)) = l i m - l o g P (— £ [a, bU = - 0 ( ( ) , n—oo n \ n J n-.oo n y rt / o
for all 6 £ (a,oo), where t
0
is the constant uniquely determined by (3.2.15)
'(*»>) = a. 3°)
I f (0) > 6 and b £ int W(V)),
l i m i i o g P ( — £ (a,b)) for all a £ [ — 0 0 , 6 ) , where (
(3.2.14)
0
=
then
lim ^ l o g P ( ^
£ [o,i]) = - ^ ( i ) , a
(3.2.16)
is the constant determined by /(*„)
= b.
(3.2.17)
114
MAXIMUM
ENTROPY
P r o o f . We start w i t h the proof of 2").
ANALYSIS
For the proof of (3.2.14), i t suffices to
show an upper estimate Orn - l o g P f& > a) < -i/>(f ) n—oo n \ n f
(3.2.18)
0
and a lower estimate lim i ]
o
e
p ( ^
e
( a , a
> -^(( ),
+ e))
(3.2.19)
0
where e is an arbitrary positive constant. A t first we shall show the upper estimate (3.2.18). Since ip'(Q) < a = (t ) and
0
> 0.
Hence we can apply Lemma 3.2.3 to obtain P (~£«J
< £[exp(f S„)]exp{-n( ). 0
o a
Noting (3.2.9) and ( a =
P (^—
>
< exptn^n^oJJexpf-n^fio) - n0(( )} O
and | log P 01 Since f
£ V, ip {to)
D
a
> (A <
converges to f(to)
V n
(
t 0
) - ,p(t )
-
0
yp(t ). 0
as n —• co and we obtain (3.2.18).
Next we shall prove the lower estimate (3.2.19). Let (j £ I? be a non-negative number and a = <^>'((i). Denote by Z t
n
a random variable w i t h probability distri-
bution j i given by n
^(z)
= exp{-r^ ( ,)}exp{t x), n
t
where u is the probability distribution of S . n
n
Km^P
We shall prove that
£ f T ( a , ) ) = J f a n (nU ( )) n
s
(3.2.20)
1
= 1,
6 ai
for all 6 > 0 where U (ai) 6
=
{x;\x
-
a, | <
5}
and
nU
=
{nx\x
€U).
(3.2.21)
3.2 LARGE DEVIATION THEOREMS
For any
> 0,
8
115
we apply Lemma 3.2.3 to obtain
J.ln([n(al
+ 0), 00)) = P (~n ~
:::; exp{ -sn( al
+ o)}
fa
+ 0)
al
exp( SX) dJ.ln( x)
= exp{-sn(al + o)}exp{-n
fa
exp(tlx)exp(sx)dvn(x)
Therefore 1 lim -logJ.ln([n(al n
n-OQ
+ 0), 00))
+ 8) -
:::;
(al
+ 0)8.
(3.2.22)
Since O. Hence J.ln([n(a) + 0),00)) converges to zero exponentially fast as n --+ 00. In the same way we can show that J.ln« -00, neal - 0)]) goes to zero exponentially fast . Therefore (3.2.21) is proved. Noting (3.2.20), it is easily seen that P
(~n
= (
E
U6(a d ) = P (Sn E nU6 (at})
l nU6(a,)
exp{ -tlX
= exp{ -n(t l al
+ n
-
1
exp{ -t) (x - a) n)} dJ.ln(x)
nU 6(al)
~ exp{-n(tlal -
Therefore
Since a E int (
. lIm -1 log P (Sn - E (a, a n n
=
+ e:) )
0 such that
~
n-oo
. -log 1 lIm P (Sn - E U6 (al )) n n
n-oo
~
-t/J(td - tl o.
(3 .2.23)
MAXIMUM
116
ENTROPY
ANALYSIS
Let a, I a and S | 0. Then f.j | („ i n (3.2.23) and we obtain (3.2.19). Assertion 3°) can be proved in the same way, 1°)
Put f-i = 0 and
can be replaced by
S
n
a-, =
i n (3.2.20) and (3.2.21). Then
ip'(Q)
in (3.2.21). Taking & > 0 so small that
p„
=
t
and 2 „
v„,
C
U (ai)
(a,
f»), we
obtain {u>hi) > l i r r ^ P ^
b f ^ 6
implying that (3.2.13) is true.
e r/«0i)) = i.
•
Theorem 3.2.4 is obtained by modifying the following theorem due to Gartner [43] and Ellis [34]. T h e o r e m 3.2.5. Assume that the l i m i t
ip(t)
= lim„_
exists for all f. £ R,
tp {t)
0 0
n
allowing the case of co = co. 1°)
(upper estimate)
If there exists x £ [a, b] such that I(x)
=
up{tx
S
-
i
<
co,
then Em - l o g P (— £ k i n < - b f /(*). n—co n y n j r6[a,*l 2°)
(lower estimate)
and(a,6) C
=
^(D?)
Assume that
{>p'(t)it
lim i l o g P
£ D }. v
ip
is differentiable on
D^,
(3.2.24) =
< oo}
{t;
Then £ (a, 6 ) ) > -
inf
J(x).
(3.2.25)
We omit the proof. In Theorem 3.2.5, i t is assumed that limn^oo
exists for all
t
£ R.
This
condition seems to be rather restrictive. Let us apply Theorem 3.2,4 to the case which we considered at the beginning of this section. Let {X ; n
v
n = 1,2,...} he a real valued i.i.d. process with distribution
and define { S „ } by (3.2.1). Then the function
defined by (3.2.9) is now
given in the form 1 " f p „ ( t ) = - l o g T T £ [ e x p ( * . Y , ) ] = log / e x p ( t z ) d H z ) . i-l * n
J
3.2 LARGE
DEVIATION
THEOREMS
117
Thus does not depend on n, and the limit ^(() = lim,,—,*, V n ( 0 always exists and is equal to ¥>(() = l o g / JR
exp(fi)aV(z)slog*(t),
where / e x p ( i i ) dv(x)
4(i) =
JR
is the moment generating function. We assume that X
n
has a finite second mo-
ment: E[X$]
Let
D
=
{t
R; /
e
x
2
x
exp(tx)
=
f x dw(x}
< oo.
2
Jx
du(x)
<
(3.2.26)
co} and define ( , x
t i — i n f D,
(
2
i
2
by
= sup £),
Then i i < 0 < t . We also assume that a
ti < 0 < ti.
(3.2.27)
Then i t can be easily seen that 0 £ ( * i , t j ) C Z>, where 73 is the same set as in Theorem 3.2.4. We denote the expectation of X
n
= B[X } a
=
by
jxdvix).
Noting
and 4 ( 0 ) = 1, we know that m =
Thus the assumptions of Theorem 3.2.4 are satisfied. Now the large deviation property (3.2.5) of the i.i.d. process is straightforward of Theorem 3.2.4. T h e o r e m 3.2.6. Let {X„,n v and let S
a
— 1,2,...} be a real i.i.d. process w i t h distribution
= X i + - • + X„.
Assume that (3.2.26) and (3.2.27) are satisfied.
Then, for every —oo < a < b < oo such that (a,6) H ( t i , t i ) 5^ & l i m l\ gp(^.e(a,b)) 0
=
l i m - l o g P (—
= -H(F;y),
<E \a, b])
(3.2.28)
118
MAXIMUM
ENTROPY
ANALYSIS
where
F = {(i6M,;4< P r o o f . First we assume that m =
E[X \ n
J
xdp(x)
€
Then # ( F ; i / ) = 0.
(a,b).
Thus
(3.2.28) follows from 1°) of Theorem 3.2.4. Next we assume t h a t m < a. Let i
0
be the constant satisfying {3.2.15). Then (3.2.14) holds, where W'(io)-V(*o).
$(M=
O n the other hand, by Theorem 3.1.4, we see that H(F;
v) = t„a - log * ( t „ ) = 0 ( t ) . o
(3.2.29)
Thus we have (3.2.28) from (3.2.14) and (3.2.29). Finally, i f m > 6, then (3.2.28) can be proved i n the same manner as above. I n this case H(F; ff(F ,) ;l
where f
0
= r b-log*(fo) =
3.3 M A X I M U M
ENTROPY SPECTRAL
We have already seen i n Sec.
V(M.
0
is the constant determined by (3.2.17).
v) is given by
•
ANALYSIS
3.1 that the M E ( m a x i m u m entropy) method
and the M R E ( m i n i m u m relative entropy) method are quite useful i n statistical inference. A method based on the M E principle in spectral analysis of stationary processes was first proposed by B u r g [18] i n 1967. Since then the M E method has been applied i n spectral estimation w i t h considerable success. This section is devoted to study the M E and M R E methods in spectral analysis, which is called the maximum entropy spectral analysis, or the m i n i m u m relative entropy spectral analysis. Let X = {X }n€Z n
be a real valued stationary process, which is a mathematical
model of a random phenomenon w i t h stationarity. Suppose t h a t the true probability distribution of X is unknown, b u t only some p a r t i a l knowledge about the distribution is given.
I n practical situations, the knowledge is provided by the
information coming from observed data. Denote by Q the family of a l l stationary processes which are consistent w i t h the given p a r t i a l knowledge. As we have discussed i n Sec. 3.1, the M E method chooses the distribution that maximizes the
3.3 MAXIMUM
ENTROPY
SPECTRAL
119
ANALYSIS
"entropy" as the optimal estimation. For a stationary process i t is reasonable to adopt the entropy rate (see Sec.
2.4) as the "entropy"
Then the M E principle
suggests that, as the o p t i m a l model, we should choose that process i n Q which h(X).
maximizes the entropy rate
A t first we deal with one of the most important cases. Let us assume that the first JV + 1 values of the covariance function are exactly known. Assuming that the process is w i t h zero mean, denote by
f
n
=
«
B[X +itXk], n
£
Z, the covariance
function. Then our condition can be w r i t t e n as % =
n = 0,l,...,AT,
(3.3.1)
where % * s are given constants. As we have shown in Sec. 2.3 (Theorem 2.3.2), there exists uniquely a Gaussian A R ( A ' ) process which satisfies (3.3.1). Burg [18] has shown that the Gaussian A R ( i V ) process has the maximum entropy rate among all stationary processes which satisfy (3.3.1). Theorem
3 . 3 . 1 . Let 7o, 7 i ,
7/v he real numbers such that ( 7 0 , 7 1 , J N )
is
positive definite and let Q be the class of all stationary processes which satisfy (3.3.1).
Then the Gaussian AR(JV) process satisfying (3.3.1) has the maximum
entropy rate i n Q.
Theoretically speaking, the problem of M E spectral analysis may be formulated in a more general form. A generalized result will be shown soon later (Theorem 3.3.2) and i t w i l l be seen that Theorem 3.3.1 can be derived as a corollary of Theorem 3.3.2.
Several different arguments to prove Theorem 3.3.1 have been
given i n the literatures. Here we give an information theoretic proof. P r o o f o f T h e o r e m 3 . 3 . 1 . Note that the condition (3.3.1) is given i n terms of the covariance function. As is shown i n Theorem 1.8.4, once the covariance matrix is specified, the m a x i m u m entropy is attained by a Gaussian distribution. Hence it is enough to prove that the Gaussian A R ( A ) process has the maximum entropy r
rate among all stationary Gaussian
processes satisfying (3.3.1). Let X =
{X } n
be an arbitrary zero mean stationary Gaussian process belonging to Q. Since the distribution of ATj is A f ( 0 , 7 ) , using (2.1.7), (1.8), (1.8.4), we have 0
h ( x ) = h(Xt |x -) = h ( x ) e
1
/(*„x -)
= ^log^^-Zpf!,^")-
0
(3-3.2)
MAXIMUM
120
ENTROPY
ANALYSIS
This means that maximization of the entropy rate h ( X ) is equivalent t o m i n i mization of the mutual information
between the "present"
I(XI,XQ)
X\
and the
"past" X~. B y (1.6) and (1.4), I(X ,Xv)
= I(X ,(X ,...,X_ ))
1
s
ll
+
N+1
I(X ,XZ \X< ,...,X- ) J
>I(X„(X ,...,X- )). 0
ance matrix ( 7 i - j )
f J
=
I(X
,{X ,
t
-/v+i (
12
>
N+1
(3.3.3)
N+1
Since the mutual information
N
depends only on the covari-
•-• ,X-s+i))
a
Theorem 1.3.3), where 7 _ * = 7*, i t is con-
s e e
stant on Q. I f X — { X } e Q is the Gaussian AR(iV) process, then X is JVn
ple Markov (Theorem 2.3.3), ( X
0
, X - N + i )
X\
and
are conditionally independent when
XZfj
is k n o w n , and consequently
I(X XZ \X ,...,X- ) u
N
o
= 0.
N+1
(3.3.4)
Thus we know from (3.3.2) - (3.3.4) that the Gaussian AR(7V) process X minimizes I ( X I , X Q )
and maximizes
h(X).
•
It is noticed that the following equivalent relations for stationary Gaussian processes have been used i n the proof. X = { X } maximizes h ( X ) n
<=$•
X
minimizes
I ( X \ , X Q )
X is the iV-ple Markov process •^=*
X is the A R ( i V ) process.
Throughout the section, like as i n Theorem 3.3.1, we assume that the partial knowledge is given i n terms of auto covariance, equivalently given in terms o f SDF (spectral density function). Then, as a result of Theorem 1.8.4, the entropy rate is maximized by a stationary Gaussian process. Since every stationary Gaussian process is specified by the SDF, our problem is to find a SDF that maximizes the relative entropy under a given constraint. We may assume that the process X = { X } is purely nondeterministic and n
w i t h zero mean. Then X has a SDF /(A) satisfying the condition (2.2.14) and the covariance function is represented i n the form 7.=
/ J
—
e'-VWdA. TT
3.3 MAXIMUM
ENTROPY
SPECTRAL
Since the process is real valued, i t holds that 7_„ = f_
n
7 - = f
= f
e- fWd\ inX
J —*
J
= -f„ and n g Z,
e " f(-\)d\, i
—
121
ANALYSIS
x
It
consequently / ( - A ) = /(A),
a.e. A.
For convenience we denote by S the class of all SDF's of purely nondeterministic real stationary processes. T h e class S is described as follows: S = |/ €
[—*,*•};/(A) > 0, / ( - A ) = /(A), 1^"
log/(A)dA
< coj
(3.3.5)
Recall that the entropy rate h(X) of a Gaussian process X with SDF /(A) is given by h(X) (see (2.4.2)).
= h(f) = - L /
log{4* e/(-M} dA 2
(3.3.6)
Assume that the partial information about the process is given in
terms of the SDF. We denote by J- the class of all SDF's consistent with the given condition. Now the problem of M E spectral estimation is equivalent to finding the SDF
t h a t maximizes the entropy rate fi( /) i n T: h(f)
= sup{/T(/);/€JH.
(3.3.7)
For example, the constraint in Theorem 3.3.1 is represented in the form F=
j
' e f(\)d\ ,nX
= %,
r» = 0 , l , . . . , / Y | .
(3.3.8)
Now Theorem 3.3.1 can be restated that the entropy rate is maximized i n T by the A R ( j V ) S D F f
which is uniquely determined by Theorem 2.3.2. This means
that, under the condition (3.3.1), the M E spectral estimation is equivalent to the fitting of an A R model to the process. However, we should note that this is due to the condition (3.3.1), and not because of the M E principle itself. I n other words, under another condition, we may expect to have another type of SDF as a result of M E spectral estimation. The M E method discussed above can be generalized into two directions. One is to consider more general type of constraints than (3.3.8). The other is to use the relative entropy rate instead of the entropy rate. The m i n i m u m relative entropy ( M R E ) method is to choose, as the optimal estimate, a SDF which minimizes the
122
MAXIMUM
ENTROPY
ANALYSIS
relative entropy rate. The M R E method differs from the M E method i n its explicit inclusion of an initial (or prior) estimate of the spectrum and its using the relative entropy rate i n place of the entropy rate. Suppose t h a t , as an i n i t i a l estimate, we are given a Gaussian process w i t h S D F g e S. A n d suppose that we axe given a class 7 C S of possible SDF's, which is not necessarily the set of the form (3.3.8). Recall that, if the covariance matrix is specified, the relative entropy w i t h respect t o a given Gaussian distribution is minimized by a Gaussian distribution (Theorem 1.8.5). Hence the m i n i m u m relative entropy rate is attained by a Gaussian process w i t h a S D F i n F . The problem now is to find the S D F f ' e f
which minimizes
the relative entropy rate H( •; g) w i t h respect to g: H(f;g)
V/ € ?
(3.3.9)
When no i n i t i a l knowledge is given, i t may be reasonable to take the S D F
as the initial estimate. Note that g is the SDF of an i.i.d. process w i t h variance 0
a . I n this case, the relationship 1
H(f-,9o)
+ ~
= -Kf)+\log{2™ ) 2
j
J(X)d\
(3.3.10)
between the entropy rate and the relative entropy rate follows from (2.4.18) and (3.3.6). Therefore, if the variance i
a
relative entropy rate h(f).
= f_
/(A) dA is known, minimization of the
is equivalent to maximization of the entropy rate
H(f\g ) 0
Thus the M R E spectral estimation can be regarded as a generalization of
the M E spectral estimation. Analogously as in Sec. 3.1, we define H(r-,g)
A SDF
by
H(F\g)
=
iid{H(f-,g)J<ET}.
which satisfies (3.3.9), or equivalently H(r;g)
=
H(r-g),
is called the H - p r o j e c t i o n of g on !F. To find the M R E spectrum = H-projection, the following theorem is essential. We define a set T' = ^ ' ( f , g ) C S by r
U
\ 9 )
={f*S:
£
-
^)
IW
^
< c* ) ,
(3.3.11,
3.3 MAXIMUM
ENTROPY
SPECTRAL
ANALYSIS
123
where c* is a constant given by
Note that /" g j P " , because /" satisfies the inequality i n (3.3.11) w i t h equality. T h e o r e m 3 . 3 . 2 . Let g £ S be a given SDF and let J be a convex subset of S 7
such that UiF;
g) < co. Then a SDF fefis
the H-projection of g on T if and
only i f r c r ( r , g )
(3-3-13)
Note that there are some analogies between Theorem 3.3.2 and Theorem 3.1.2. Although the proof is a little more complicated, i t is similar to the proof of Theorem 3.1.2. P r o o f . First we assume (3.3.13). Then for any / g T, using (2.4.18) and (3.3.13), we have
< o. The last inequality follows from the inequality \ogx < x - 1 (x > 0). Thus (3.3.9) holds. The converse is proved by contradiction. Assume (3.3.13) were false. Then we would have a S D F /, g J n ( J * ) . Define a SDF / ( A ) (0 < a < 1) by r
r
c
a
/„(A) = « / , ( > } + ( ! - « ) / • ( * ) . Since fx, g € F and T is convex, the SDF f
a
is an element of !F We put
and
m
=
f
J —7
/•(A)
&
V
/"(A)
124
MAXIMUM
ENTROPY
ANALYSIS
Then i t is easily seen that Ix{H(f ;
g) -
a
a y - , £ )
=
«Z
+ /(*)•
(3.3.14)
Since / j does not belong to iF, f < 0 .
(3.3.15)
Here we shall show that lim ^
Q—o
= 0.
a
(3.3.16)
To prove (3.3.16) we put
For X £ A and 0 < a < j , one can show that
_
2 S
I
(
b 8
l +
d«M)
S 0
.
Therefore, by Lebesgue's bounded convergence theorem, we have
On the other hand, ( l / o ) l o g { l + a(/,(A) - / *(A))//*(A)} > 0 is decreasing i n a, for each fixed A € A .
Therefore, i n the same way as above, we have
c
1 ^
«
/ ,
(,
*
i1
to
, «(/,(AW(A))\ J
T^A)
+
/•
/i(A)-HA)
= A.
rfA
/-(A)
^
( 3
- ' 3
1 8 )
Thus (3.3.16) is easily derived from (3.3.17) and (3.3.18). I t follows from (3.3.14) - (3.3.16) that lim
l { H ( f
Q —0
or
a
;
g ) - H ( r
i
g
) } =
~
47T
This implies that HU ;g) a
for sufficiently small a > 0 . Since /„ *£ .F, this contradicts w i t h (3.3.9). Thus the "only i f " part is also proved.
•
3.3 MAXIMUM
ENTROPY
SPECTRAL
125
ANALYSIS
As a special case, let us consider the case where both /* and g are SDF's of A R processes of order N and JV , respectively. Then, using (2.3.6), we know that 0
1//'(•A) and
can be expanded in the form,
l/g(X)
,
where
and
p*
k
AT
/V„
are constants. I f JV > JV , then 1//'(A) -
p
i
m
-
m
=
S
/
is w r i t t e n as
l/g(X)
0
k
^
(
3
' 3
2
0
)
where A =p' -p k
k
( k = 0 i f |*| > j V ) .
k
P
(3.3.21)
0
The constant c* of (3.3.12) is now given as follows: N
N
where 7* is the fc-th covariance associated w i t h the SDF /*. Therefore the set T*
= F'{f,g)
of (3.3.11) is expressed as
f
r
N
I
k=-N
J
1
N
- '
k=-N
)
Therefore, as a corollary of Theorem 3.3.2, we have obtained the following result. T h e o r e m 3 . 3 . 3 . Let /* and g be SDF's of A R processes of order JV and JV
0
(JV > JV ), respectively, given by (3,3,19), Then /* is the H-projection of g on a 0
convex set T i f and only i f /* £ jF and T C T'', where F ' is the set of (3.3.22). In particular, /* maximizes the entropy rate h( • ) on every set T C iS such that / ' e J c f o n f , ,
(3.3.23)
where T = { f £ ,S; / * , f{X) dX = 70} and 0
\
^> =
/ e 5;
I
N
r
£
p* /
t=-/v
N
«'"/(A)dX <
•' ' -
r
£
1 pJt*
.
J
126
MAXIMUM
ENTROPY
ANALYSIS
The second half of the theorem can be proved by taking g(X) = 7o/(2ir) as the initial model. I t is clear that the set F of (3.3.8) satisfies (3.3.23). Hence Theorem 3.3.1 can be obtained as a corollary of Theorem 3.3.3. Another important class of stationary processes is that of A R M A processes (see Sec. 2.3). We now t u r n to study the relationship between M R E spectral analysis and A R M A model fitting. Let f
be an A R M A ( A , M ) { N = K + M > M ) SDF, r
which is given in the form
f(X)
rr
=
ii - 3
M
e\ iX
2
" ' 2™>nt ii-"n^i U
=
P
1
m
1
(cf. (2.3.8)). Then
2
can be w r i t t e n i n the form
l/f"(X)
TO=
£
* >
3
(3.3.24)
1
P l ^+ I I C I l - ^ r
k=-K
2
-
(3.3-25)
".=1
Assume that, similarly as in Theorem 3.3.3, the initial model g is an AR(iVo) SDF. Then
can be w r i t t e n as i n (3.3.19). Now, i f
l/g(X)
j}x\~Jx-)= ' J
where
A
k
K
* t ^ *
M
'
k=-K
N
0
then
< K,
+ E * | i - A ^ r * .
(3-3-26)
m=]
is given by (3.3.21). I n this case, the set
F*
-
F'(f',g)
of (3.3.11)
consists of all SDF's which satisfy the constraint
m=l
km-K K
J
-"
11
P
m
€
I
M
where 7 t and IT , are constants defined by 2
J*
e r(X)dX
= 7,
f
[1 - £ m e | - 7 - ( A ) d A = 4 ,
ikX
k
t = 0,l,...,A:, (3.3.28)
,A
m = 1
M.
Again, as a corollary of Theorem 3.3.2, we obtain the following theorem.
S.J HYPOTHESIS
T h e o r e m 3 . 3 . 4 . Let
and
f
g
137
TESTING
be the ARMA(JV,
SDF and
M)
AR(N ) 0
SDF given
by (3.3.25) and (3.3.19), respectively, where N = M + K > M and K > N . a
Then
/* is the H-projection of g on a convex set T i f and only i f /* S'JF and f
C P ,
where J " is the set consisting of all SDF's satisfying (3.3.27). I n particular, /* 7
maximizes the entropy rate h( •) on the set JFj of all SDF's which satisfy the condition (3.3.28).
Let X = { X } be a stationary process with SDF / and define processes F ' ' m
n
{Y™}
=
by
=
E
n e
z
Then we can easily show that K > is a stationary process w i t h SDF , m
|1 — ^mC'*! " 2
Hence (3.3.27) can be w r i t t e n as
E
^ E ^ x o i
+ E^^in " ! ^ E
t=-A"
1
1 1
2
m=]
Arw+Es^m= l
t=-h'
Therefore the constraint (3.3.27) can be explained as a condition given i n terms of the first K + 1 covariances of X and the variances of Y
3.4 H Y P O T H E S I S
K
TESTING
The large deviation theorem can be applied to evaluate the error probability in hypothesis testing. The problem of hypothesis testing is to decide which of the two hypotheses is true.
The decision is made based on the outcome of a
measurement that is characterized by a probability distribution on the space of possible measurement outcomes. Let us consider a random phenomenon which is described by a random variable w i t h unknown distribution i/, and we are required to test two hypotheses Ho and Hi w i t h the associated probability distributions Uo and i/i: H
:
v =
i/o,
H\
:
v =
v\.
0
(3.4.1)
128
MAXIMUM
ENTROPY
ANALYSIS
Our task is to decide which of Ho and H\ is true based on the observed data. We suppose t h a t , as a result of observation, we are given a sample x — (x\,...,Xft) size
N.
Let
X
— [ X
) „
n
€
be an i.i.d. process w i t h identical distribution
2
v.
of Then
the outcome x = (x-,, ...,x,v) of the measurement can be regarded as a realization of
( X \ , X ^ ) -
The hypothesis testing under this situation is most fundamental
not only from the theoretical point of view b u t also from the application viewpoint. For mathematical simplicity, we assume that each X„ is a real random variable, so that V is a probability distribution on R. The hypothesis testing procedure is essentially determined by a p a r t i t i o n of the space R
w
of the measurement outcome into two sets B R and ( B / v ) . Suppose that c
we are given a measurement x —
€ R .
(XI,...,XN)
If x is an element of
w
we decide that Hg is true; if x is an element of Bpj, we decide that The sets B ^ and [ B N Y accepting hypothesis H
when H
x
a
C
is true.
called the decision regions. The probability
a r e
(BJV) ,
of
is actually true is called the e r r o r p r o b a b i l i t y
o f t h e f i r s t k i n d . On the other hand, the probability 0 N of accepting hypothesis Ho when H\ is actually true is called the e r r o r p r o b a b i l i t y o f t h e kind.
probabilities
as I V —* oo.
and ff
N
probability distribution of of the independence of of
second
I n this section we are interested i n the asymptotic behavior of the error (X
,...,XN)
t
Let us denote by f , N ( M
under the hypothesis
the distribution
Xi,...,Xff,
v ,N M
H. m
M
1) the
=
Then, because
is the jV-ple product
v: m
N v
{At
m
i
N
(A
x
x • •• x A)
=
N
v (A ),
JJ
m
(3.4.2)
k
€ 6 ( R ) ) . Now the error probabilities are equal to
A
N
=
The problem is to determine possible. The Neyman-Pearson orjv is specified, say
a
N
,
0N
=
SO
(BN,(BN) ) C
(3.4.3)
"I,N{( NY)B
that ayv and
0N
are as small as
point of view assumes that a m a x i m u m value of
and determine
( B
N
{ B
:
N
)
C
)
so as to minimize
8fi
subject
to this constraint, ow < a* . N
(3.4.4)
A method to find the optimal decision regions is given by the following wellknown theorem.
3.4 HYPOTHESIS
T h e o r e m 3 . 4 . 1 . (Neyman-Pearson)
129
TESTING
Fix a positive integer iV. For any positive
number t , define sets B„{t) =
{ x € R
M
(B (t)y N
and let OfA/(t) and &iv{t)
4 l o g ^ W > i } , .
N
" 7
(3.4.5)
4 log P ^ ( x ) < t ) ,
= {x e R ; N
be the error probabilities of the first and the second
kinds corresponding to this choice of decision regions. Suppose ctfj and 3^ are the error probabilities corresponding to some other choice of decision regions. Then &N > ^jv(t), whenever ajv < O N ( ( ) . We note t h a t , throughout this section, no assumptions are made concerning the type of the unknown distribution and V
m
may be arbitrary, not necessarily
discrete nor continuous. Therefore the theorems derived in this section are valid for continuous distributions as well as discrete distributions. I t is reasonable to assume that Vo and i
m
has a finite second moment, j
K
x dv {x) 2
< oo (m = 0,1). We denote the
m
Radon-Nikodym derivative by *C*>-f*W.
seR.
(3.4.6)
The Ney man-Pearson theorem says that the optimal test is provided by determining the decision regions i n the form (3.4.5).
In order to investigate the
asymptotic behavior of error probabilities O N ( ' ) = fo,A/(BN(')) and /?Aj(t) = "i,w((JBiv(')) ) as JV — i oo, we need to examine the function e
a() N X
= log P^(x)
= T
loge( ),
x = <*
I n
*«) € R , w
(3.4.7)
which is called the log likelihood ratio function. Corresponding to sw(x), we define a random variable N
SN
= E
l o
Se(^).
(3-4-8)
>* i
for the i.i.d. process JV = { - f „ ) . Then, as a corollary of Theorem 3.2.6, we know the asymptotic behavior of S . We denote by P < ' ( •) and £< > [ - ] the probability m
N
and the expectation under the hypothesis H . m
m
no
MAXIMUM
ENTROPY
ANALYSIS
L e m m a 3 . 4 . 2 . Let {SJV} be the sequence of random variables o f ( 3 . 4 . 8 ) . Then under the hypothesis H
m
(m = 0 , 1 ) ,
and Jim l l o g P O ) ( f
l
t
o
g
^
l
W
« B
w
( t
)
n
= -ir((G«) ;n),
(3.4.10)
0
where G, =
> tJ .
| p g M ;jjogg(x)dp(x) 1
( G , ) = {/* € M ; p £ G « } . c
t
(3.4.11)
The large deviation property suggests to us that the error probabilities go t o zero exponentially as JV —* oo. So we specify the maximum error probability a '
N
of the first kind as a
= exp(-nriV),
N
(3.4.12)
where or > 0 is a constant. The large deviation theorem (Theorem 3 . 2 . 6 ) enables us to find a useful expression for the asymptotics of the m i n i m u m error probability 8"
N
of the second kind. In order to give the statement of the main theorem of this section, we prepare
a lemma concerning the relative entropy. We define a function Qa(x) on R by « ( * )
= A(e) (x)Tb, e
X
€
R
,
(3.4.13)
where A(0)
= ^ e ( z ) ^
dv (x) a
is a normalizing constant. I t is easily seen that the integral
j
R
g(x) ^ ^ e
J+9
dva(x)
is finite for all 9 > 0 , A { 8 ) is continuous i n 8, A(0) = 1 , and l i m ^ ^ , A(8) = 1 . We may choose a constant
8
0
< 0
so that
J
K
e( ) ^' x
e
+
e
>
d"o( ) x
< co for all
0 >
8. a
The family {"g ',8 > 8 ] of probability distributions defined by 0
^k(z) = «(*),
x<=R,
(3.4.14)
S.4 HYPOTHESIS plays significant roles i n the sequel. "geodesic c u r v e " connecting v
1 3 1
TESTING
I n t u i t i v e l y speaking {fj;<> > 0 } forms the
w i t h f , . W e note here that t h e R a d o n - N i k o d y m
0
derivative of i/J w i t h respect to v\ is given by
- « • > ( » * •
<»•">
Noting log » ( z ) = l o g o f f ) + ~
log
(3-4-16)
we see t h a t t h e relative entropy r / ( f j ; c ) is presented in the form 0
= iogA(fl) + - v ^
/
e(*)^iogfl(*)<M*). (3.4.17)
T o e x a m i n e the behavior of J 7 ( f f ; H>), we introduce sets
F
9
= |
H
6M
i
;
A'oge*(*)<W») > * K i M }
(3-4.18)
J
(3.4.19)
and 3F, -
U
€ Mi;
\Dgm0f)4rt*)
=
ffW;%)J.
I t is clear that * j e 3 F , C P#.
(3.4.20)
T h e o r e m 3.1.2 s a y s that i/J is the H - p r o j e c t i o n of i^, on F » : H(F,;vt)
= H{v*,\*).
(3.4.21)
A t the s a m e t i m e , using (3.4.15), we c a n show that v\ is also the H-projection of i/i on F , = {p € M r , / l o g p ( z ) d n ( x ) < R
B{(F,)';vi)
e
H(v}\v„)):
= H(F#;vi) =
(3.4.22)
A g a i n noting (3.4.16), it is easily shown t h a t , for each $ > 0, there exists a constant f = t(tf) for w h i c h F, holds.
= G,
(3.4.23)
132
MAXIMUM
ENTROPY
ANALYSIS
L e m m a 3.4.3. The relative entropy / f ( e j ; e ) (resp.
is continuous on
H{vl\vi))
0
(flo.oo) and increasing (resp. decreasing) on [0,oo). Moreover * ) - * ( < « * ) ' - 4
(3.4.24)
and lim
S-oo
H{vl;va)=
f
gix)
J
log g(x)
du (x)
=
0
H{m
"*)•
(3.4.25)
R
P r o o f . The continuity of /f(|/J;Vo) and the relations (3.4.24) and (3.4.25) can be easily shown by using (3.4.17). By (3.4.23) and the definition of G , , we have the alternative,
dFe
the continuity of
=
or
8Fe>
H{v' ;
v ),
9
a
dF»
=
D
for every pair(f?,0'). Hence, noting
(3.4.24) and (3.4.25), we conclude that Ffl^F,-,
9<6'.
if
Finally we see, from (3.4.20) and (3.4.26), that Similar properties associated w i t h
H(i/g]
(3.4.26) is increasing i n 0 > 0.
H(v' ;v ) 9
0
V\) can be shown analogously.
•
We are now in a position to show that the error probability converges to zero exponentially and give a precise expression of its exponential rate. Theorem
3.4.4. Let 0 <
a
<
and let
H(v\\ a) v
8 =
> 0 be a constant
8(a)
satisfying Bpl^V^-m,
If the first kind error probability
api
(3.4.27)
satisfies (3,4.4) w i t h
a'
N
=
exp(-aN),
then
the exponential rate of the m i n i m u m error probability 0^ of the second kind is given by j m ^ i i o g ^
=
~m#* w>-
<-- > 3 4 28
m
P r o o f . Since H(i/g \ i/) is increasing in 0 > 0 (Lemma 3,4.3), the number 8 ( a ) > 0 0
satisfying (3.4.27) is uniquely determined. For each N there exists a unique number t/v such that "D,A/(BN('N)) = e x p ( - o J V ) . We shall show, by contradiction, that the sequence say t(a), as N —» oo. Assume that
(3.4.29) converges to a constant,
did not converge. Then we would have t
and t such that lim
t
N
< t < i <
Urn t . N — oo N
(3.4.30)
3.4 HYPOTHESIS
133
TESTING
We can choose a subsequence {t/v,} such that t <
so that
( M i C B w ( T ) 3 > " , w ( * N { < N ) ) = exp(-aNiO. t
u
4
t
(3.4.31)
t
By Lemma 3.4.2 hm
1
rJ-lo i/o,N (5 (()) = -/7(G ;i/ ), g
1
T
W l
(3.4.32)
0
where G , is the set defined by (3.4.11). I t follows from (3.4.31) and (3.4.32) that H(G ,
< a = H( \ ;u ).
M
r
Vt
a)
(3.4.33)
0
Similarly we can show that E(Gt;vo)>a.
(3.4.34)
On the other hand, since G t D G j , ff(G, / )
0
(3.4.35)
r;1
Combining (3.4.33) - (3.4.35), we have equalities ;i/ ) = H(Gj;
fl(G,
)
= a = H(vfa,
Va
0
M-
(3.4.36)
By (3.4.23) there exist 0 and 8 such that F
£ = <>
*V = < -
G
G
and consequently H(F ;
u) = H ( F z
t
0
=
(3.4.37)
As we have seen i n Lemma 3.4.3, H[Fe\ "a) = # ( f j ; "a) is increasing in 0. Hence (3.4.37) means that 8 = 8. On the other hand, i t follows from G , ^ G j that Fg
/
F j , so that 8 ^ 8 .
This contradicts. Therefore we conclude that {t^r}
converges: lim t
N
= t(a).
(3.4.38)
N—.00
Furthermore we have obtained from (3.4.36) that lim
^rzlogv ,N{B\(t )) a
=
N
lim
-j= log fo,w(i?w (((<*))) = S(vi\ y,
c ).
a
N—oo JV
JV—oo JV
0
* '
I n a similar manner, using Lemma 3.4.2, (3.4.15), (3.4.22) and (3.4.38), we can show that lim
—logjt?^ =
A/—oo iV
lira 7 7 l o g f i . / v ( ( B j v ( * N ) ) ) e
N—oo JV
=
lim
^log^«((B«(t(«))) )
/V—oo JV
Thus (3.4.28) is obtained.
•
e
134
MAXIMUM
ENTROPY
ANALYSIS
3.5 H Y P O T H E S I S T E S T I N G F O R S T A T I O N A R Y G A U S S I A N PROCESSES
The hypothesis testing problem is discussed i n the preceding section for i.i.d. processes. I n this section we study a problem of testing hypotheses on the SDF of a stationary Gaussian process. I t is shown i n Theorem 3.5.2 t h a t , similarly as in the i.i.d. case, the error probabilities decrease exponentially w i t h sample size N.
Moreover the exponential rates are explicitly evaluated. I n order to examine
the asymptotic behavior of the error probabilities, applying Theorem 3.2.4, a large deviation theorem (Theorem 3.5.4) is proved for stationary Gaussian processes. It is noticed that the probability distribution of a stationary Gaussian process is determined by the covariance function, or equivalently by the SDF. Hence, for stationary Gaussian processes, i t is natural to describe the hypothesis testing problem in terms of SDF. Let
X
be a stationary Gaussian process w i t h
{X }„ez
—
n
unknown S D F /. We consider two hypotheses Ho and H i and their associated SDF's /„ and /,: Ho-
f = fo,
« 1
f = f,
We are required to decide which of the H
a
data x = ( i ] , . . . , i j v ) 6
, (
N
Denote by f , i V ( m = 0,1) the probability distribution of m
hypothesis H .
"
5
1
)
and H i is true based o n the observed
that can be regarded as a realization of
R.
K
3
( X \ , X N )
( X \ , X N ) .
under the
Then c , j v has the covariance m a t r i x
m
m
where e f„Wd\,
= J
(3.5.3)
ikx
and has the Gaussian probability density function
P
"
,
<
X
)
=
( 2 ^ | r
where x = ( I , , . . . , I ) e R n
w
m
,
w
| ^
e * p { 4 <
and ( x , y ) = £ *
= 1
r
" >
X
)
} '
( - - > 3
5
4
i y . t
k
We are interested i n the asymptotic behavior of the error probability i n the hypothesis testing (3.5.1). For each /V, let B N the observed data x = H
a
( X , , . . . , I N )
is true; if x is an element of B
€ R w
N
and ( B N )
C
is an element of
be decision regions. I f ( B
N
)
C
,
we decide that
, we decide that H i is true. We denote by
3.5 HYPOTHESIS
a«
TESTING
FOR GAUSSIAN
135
PROCESSES
the error probability of the first kind, which is the probability of accepting
hypothesis f/j when Ha is actually true. A n d denote by 3N the error probability of the second k i n d , the probability of accepting hypothesis Ho when Hj is actually true. Then clearly
« N =
voM *)>
0N
B
=
vt,N(( n) )B
(--)
e
3
Similarly as i n Sec. 3.4, assuming that the maximum value a'
N
5
5
of ojv is specified,
the problem is to find the decision rule that minimizes the error probability of the second kind. We denote by 3 8'
=
N
N
the minimum:
ipl{ M(BN) );
<
V ,N{BN)
C
Vl
0
a
N
(3.5.6)
).
Although the process is not i.i.d., the Ney man-Pearson theorem (Theorem 3.4.1) is still valid and the optimal decision regions
= J3jv(t) and B
N
are given in
the form B
w
, « ) = { x
R - I l o g ? | g >
6
} ,
(
(3.5.7)
w i t h a parameter t which is called a threshold. Similarly as i n the preceding section, specifying the maximum error probability a*
of the first k i n d as
N
a
N
- exp(-oJV)
N
(3.5.8)
w i t h a constant a > 0, we are interested i n the asymptotic behavior of the minimum error probability 3
N
of the second kind. I t w i l l be shown that 3
N
goes to
zero exponentially as N —• oo. Moreover the exponential rate lim ^ l o g ^
(3.5.9)
A/—oo iV
w i l l be evaluated explicitly. To avoid unnecessary technical complexities, throughout the section, we assume that
f
m
£ S,
m
family {g ; 0 < 8
O
= 0 , 1 , and that /o(A)//,(A) and f,(X)/foW
are bounded. A
< 0,} of SDF's defined by
1
1
M
~
m
t
(i-g) , g /o(A) * /,(A)
1
1
(3.5.10)
136
MAXIMUM
ENTROPY
ANALYSIS
w i l l play significant roles. Here 9 < 0 and 9\ > 1 are constants determined i n a
such a way that go€S
and
is bounded (m = 0,1).
(3.5.11)
Note that go = fa and j i = / j . Intuitively speaking, { g e ; 0 < 6 < 1} forms a "geodesic curve" connecting fa w i t h / i on the space o f SDF's. We introduce here two classes (3.5.12) and (3.5.13) where c
+
* - L \ ( i - e ) h W
<>foW)
In order to give the statement of the main theorem, we prepare a lemma concerning the relative entropy rate H(f\g) Lemma 3.5.1. B(ge\fiY)
(see (2.4.18) for definition).
1°) As a function of 6, the relative entropy rate H(gg\ f ) (resp. a
is continuous on [#o,#i], increasing on [0,0]] (resp.
on [ l , # i ] ) and
decreasing on [#o,0] (resp. on [0o,l])2°)
For each 0 < 9 < 1, gg is the H-projection of f
0
on T$ and at the same
time the H-projection of /[ on T$: H(9s;f ) 0
Proof.
= H(^Jo),
H(g h) 9!
= H{T- -,h). 9
(3.5.14)
1°) B y (2.4.18) and (3.5.10)
7-„ I W
) + ( 1 - f )/i(A)
g
C/o(A) + ( 1 - f?)/,(A) J
Note that a function
off+ * ( ! - * )
6
afl-f-i(l-fl)
_
3.5 HYPOTHESIS
TESTING
FOR GAUSSIAN
PROCESSES
137
(a j i 6 are positive constants) is differentiable and F-ffll = Since F'(fl) > 0 i f fl > 0 and
< 0 i f 0 < 0, the integrand of (3.5.15) is
increasing in 8 > 0 and decreasing in 8 < 0, for each fixed A.
Consequently
H{9e\ SO) ' increasing i n 8 > 0 and decreasing in fl < 0. Similar properties can be s
proved for H(gs;fi)
i n a parallel way.
The
continuity of H(gt\fo)
H(g \fi)
and
s
is obvious. 2")
Noting the relations 1
_ 1
9*W
UW
/ ~
1
1
\ftW
\ _
foW)
8
~ 8-1
I \g,(\)
1
1
\
MX))
•
we know that T I and TI can be rewritten as TI
=
Tt
=
with
and
{'^••L(^)-m) ^"<} fWd
nth
-•'JJ^~m)'
Mdy
Now (3.5,14) can be obtained as a direct consequence of Theorem 3.3.2.
•
Now we can state the theorem on the asymptotic behavior of the error probabilities for the hypothesis testing (3.5.1). T h e o r e m 3 . 5 . 2 . Assume that the first kind error probability a/v for the hypothesis testing (3.5.1) satisfies (3.5.8), where 0
(3.5.16)
Then the exponential rate of the minimum error probability 0^ of the second kind is given by l i m ^z\og0' N—-oo 1\
N
= -H{gn )\fi), a
(3.5.17)
138
MAXIMUM
ENTROPY
ANALYSIS
where g is the SDF of (3.5.10) and 8 ( a ) is uniquely determined by 9
ff(fl«(a);/o)
(3.5.18)
= <*•
For the proof we apply the large deviation theorem (Theorem 3.2.4).
From
(3.5.4) i t is easily seen that
l o g
4-777 = I N P>/( ) x
2
r W I- 1 ^ ir^l +
n
L
r-'^x.x)},
X E R " .
For the stationary Gaussian process X = {X },
2
-
1
(3.5.19)
let us consider a random variable
l »."l
J (3.5.20)
r
To evaluate the error probabilities of (3.5.5) corresponding to the decision regions of the form (3.5.7), we apply Theorem 3.2.4 to the random variables { S N } of (3.5.20). For this purpose, we examine the function P - , i f ( * ) = £ 1"S E
, m
W«>S«)],
(3-5.21)
and the l i m i t function V m
(9) = j i *
(3.5.22)
where J S ' ' [ J denotes the expectation under the hypothesis H . m
m
The function
Vm.ivffl) can be rewritten as
e^Llog^l^lp^xJdx.
p™,W<*) = " U g /
(3.5.23)
We can show that, for each 8 < 8 < 8i, an N X N m a t r i x B
iV* = [ ( l - Q ^ - H B ^ j , ] is positive definite. We prepare a lemma.
- 1
(3.5.24)
3.5 HYPOTHESIS
L e m m a 3.5.3.
TESTING
FOR GAUSSIAN
1°) The functions
PROCESSES
139
m = 0 , 1 , are expressed i n the form
tf),
VO,N(V) = ¥ I . N ( C - 1 ) ,
= ^
{ffio |r , | g
0
N
+(i-
«)iog|r
I i W
|
- loglfliv* + ( i -
0)TI,N\)
do < 0 < T h e l i m i t s
2*)
= l i m
0
w
-
0
o
(«o < * < * i ) and ip,(6)
0 N
.
(3.5.25) = lim^-oo
Vl.iv(^) ( H J - 1 < S < 9 | - 1 ) exist, and they are given i n the form
. J L / [
4
o
r J
The function yo(#) and ipi(8 — 1) are differentiabie on (0 , 0,) and
3°)
A
VoW = 9',(<>-i) i
r
f
h w - u w
4 y_.U/o(A) l r
,
+ (l-fl)/ (A) 1
+
M A ) i
° /,(A)J 6
•
(
3
5
2
7
)
In particular ri(0) = V ' , ( - 1 ) = -H(hlfi), Moreover,
4°)
4 , ( 1 ) = V',(0) = F ( / i ; A ) .
(3.5.28)
is continuous.
Define a function ip (0) M
by
lM*)» V-(*)-V-(«)
(3-5-29)
(cf. (3.2.11)). T h e n WO) =
ff(?»;/o),
S
(3.5.30)
6>o < 9 < 0,.
(3.5.31)
and ^ , ( 0 - 1) = H ( Proof.
S 9 ;
/,),
1°) Denote by u , w ( 0 , x ) the integrand in (3.5.23). Then, using (3.5.4), m
(3.5.10) and (3.5.24), one can easily show that no,M*,x) =
=
exp(flIog4^}p ; (x)
( 2 7 W « P { i r . . » l -
(
,
|r, | lW
\ V ' > < * > }
(
3
5
Z
2
)
MAXIMUM
140
ENTROPY
ANALYSIS
Since [ - \ ( K j , * , x ) }
<*P
j
(2*) /*\Ve, \ i
dx =
N
N
l/2
using (3.5.23) and (3.5.32), we obtain V o . w W = 4 log /
u
0 i K
(i?,x)dx
= 4 ? Oog iv^.jvi - ( i - fl)iog i r , j v i - # i o g i r » , | } . 0
w
"
2
(3.5.33)
It is clear that v-
= (i -
e N
tf)r-J,
= r - { ( i - 9)T j» +
+
0
N
t
so that |Ve, | = |r,,»| \Vi,N\l(i
(3.5.34)
- « ) r , , « + or , \- . 0 N
w
1
The equation (3.5.25) follows from (3.5.33) and (3.5.34). I n the same way we can show that 2°) Tpj(fm)
1) is also equal to the right-hand side of (3.5.25).
—
Since the covariance matrix T j y is nothing but the Toeplitz m a t r i x m :
corresponding to the SDF /„,: r . j v = Tiv(/ ), m
m
using Lemma 2.4.3, we have
tim
i%|(i-B)r,, +«t , | ,
K
0
w
h / _ " l ° 6 [ 2 M ( l - 0 ) / i ( A ) + 0/o
=
From (3.5.25) and (3.5.35) we know that, for every
0 e [8a,8i\,
(3.5.35) the l i m i t
=
lim/v_oo i^o,w(0) exists and (3.5.26) holds. 3")
Using (3.5.26) and exchanging the order of integration and differentiation,
we obtain (3.5.27):
l_
m ^
-LW
l
o
s
/
o
'
A
)
+
(
(A) (A)
- ^ s / - w -
1
/i(A)-/„(A) ff/ (A) 0
l
o
S * W ,
+ ( 1 - 0 ) / , (A) '
,
)
+ ( i - W * ) } ] ^
3 . 5 HYPOTHESIS
TESTING
FOR GAUSSIAN
141
PROCESSES
The equations i n (3.5.28) are evident. The continuity of tp'^S) is clear from (3.5.27). 4°) The equations (3.5.30) and (3.5.31) can be easily shown by using (3.5.26), (3.5.27) and (3.5.29). •
We now prove a large deviation theorem which is similar t o Lemma 3.4.2. B (t)
T h e o r e m 3 . 5 . 4 . For each t £ ( - f f ( / o ; / i ) , f f ( / i ; / o ) ) , define the region by (3.5.7). T h e n N ^ L iv" w
l o g
"°> W
lim ilog^, o
»
N
j Y
-
A).
e
w
< - - ) 3
= -H{ ;fi),
((fl {t)) )
N
5
36
(3.5.37)
gil
where 8 £ (0,1) is a constant uniquely determined by VW)
= t.
(3.5.38)
P r o o f . Using (3.5.27) one can easily show that U,»(G\-
implying that
1
(/Q( )-/I(A))
n
A
is an increasing function.
2
i
-
Hence, noting (3.5.28), we know
that there exists uniquely a constant 8 € (0,1) satisfying (3.5.38). Now we can see that the assumptions of Theorem 3.2.4 are satisfied. Therefore, applying 2°) of Theorem 3.2.4, (3.5.36) is obtained: Jim I
log u B (t)) oA
N
=
to
i
log P<°> ( f
>
,)
= - M * )
=
-*(«;/»),
where P'°*( •) denotes the probabihty under the hypothesis H , and we have used a
(3.5.30). The equation (3.5.37) can be proved similarly.
•
We now t u r n t o prove the main theorem of this section. P r o o f o f T h e o r e m 3.5.2. Since H(g \ f ) is increasing i n 8 > 0 (Lemma 3.5.1), a
a
the number 8 ( a ) £ (0,1) is uniquely determined by (3.5.18). For each iV there exists a unique number tjv such that fo,w(-Bjv{fjv))
=exp(-QiV).
142
MAXIMUM
ENTROPY
ANALYSIS
We shall show, by contradiction, that the sequence { t ; v } converges t o a constant, say t(a). Suppose that {t^} does not converge. Then we have t and ( such that lim
t
N
< t < t <
Inn t .
JV—oo
lV —00
(3.5.39)
N
Corresponding t o ( and t there exist constants 0 and 0 such that M m Trlog"a.Jv(BjvU)) = - j f f ( f l * ; / o ) Jv—-oo /V Jim
T 7logf
0 i J
v(B {t))
=-/f{ffj;/ ).
w
0
Similarly as in the proof of Theorem 3.4.4, using Theorem 3.5.4, we can derive the inequalities Hlgj-.M
<<* = H(g yJ )
(3.5.40)
H(gi\h)
> c - H(g y,f ).
(3.5.41)
Ha
0
and Ha
0
On the other hand, the inequality t ^/vf-Sivft)) and consequently H(gs;f )
0
(3.5.42)
0
By (3.5.40) - (3.5.42) we have H{g ; t
h)
= H{g-
e
; /„) = H
^
y
/„),
yielding that 0 = 8 =
0( ). a
From this we conclude that t = t. B u t this contradicts (3.5.39).
Therefore we
know that l i m (jv = (for) JV—oo
(3.5.43)
and w
lirn ^logi' o
0 > w
-H(ggf y f ).
( B j v ( f j v ) ) = Jmi ^-logi'o,N(S/v(i(o)) =
a
o
In a similar manner, using Theorem 3.5.4 and (3.5.43), we can show that
= ^
]y log"i.w((B«(«(a))) )
=
-H(ge( y,fi).
e
a
t 0
HISTORICAL
Thus (3.5.17) is obtained.
HISTORICAL
NOTES
143
•
NOTES
The m a x i m u m entropy principle and the minimum relative entropy principle were introduced by Jaynes [71] and Kullback [83]. The minimum relative entropy principle is axiomatically studied i n Shore and Johnson [107], Geometrical properties of relative entropy has been investigated by Csiszar [24], Theorem 3.1.1 Theorem 3.1.3 are due to Csiszar [24]. There are suitable references for large deviation theory, among them Varadhan [115], Ellis [36], Deuschel and Strook [30], Bucklew [17]. The texts [115, 30] are w r i t t e n rather i n mathematical flavor.
Theorem 3.2.4 and Theorem 3.2.5 are
essentially due to Gartner [44] and Ellis [35] (see also [17, 36]). The Sanov property is due to Sanov [101], and related topics are treated i n [25]. The m a x i m u m entropy method i n spectral analysis was first proposed by B u r g [18]. The fundamental theorem (Theorem 3.3.1) is due to Burg. Several interesting papers on the maximum entropy analysis are reprinted in [21]. I n this book we have not discussed the problem of how to select the order of an A R model (see the discussion by Ulrych and Bishop [112]). To estimate the order, Akaike information criterion (see [2]) may be useful. As a result of the maximum entropy analysis, we are given only A R models, as long as we rely on the Burg's theorem.
In this sense, Theorem 3.3.2 seems
to be of interest. This theorem was proved by Franke [40] and Ihara [65, 67] i n different ways. Franke proved by using convex optimization theory. Franke [41] gave another characterization of A R M A models i n the maximum entropy analysis. In the case of i.i.d., there are a number of works on the error probability i n the hypothesis testing (see [14, 15, 17]). The same problem is studied in Natarajan [90] for Markov chains. Theorem 3.5.2 is essentially due to Dacunha-Castelle [27]. A special case of Theorem 3.5.2 has been investigated i n Bucklew [17], where
f
a
is the SDF of a noise process, fi = Jo + go and go is the SDF of a signal process. In this case the problem is a kind of detection of the signal.
Chapter 4 T h e o r y of I n f o r m a t i o n Transmission 4.1 M O D E L O F C O M M U N I C A T I O N
SYSTEMS
There are communication systems of various types like telephone, radio, television, satellite communication, etc.. Physically, they have different aspects.
How-
ever, as Shannon [103] pointed out, there is a unified model for these communication systems. This enables us to discuss various kinds of information transmission under a unified treatment. The model is given as in Figs. 1.1 and 4.1. A communication system consists of essentially five parts; an information source, an encoder, a channel, a decoder, and a user. A n encoder and a decoder are also called a transmitter and a receiver, respectively. The i n f o r m a t i o n s o u r c e produces a m e s s a g e
to be communicated to a u s e r .
The
encoder
(transmitter)
operates on the message i n some way to produce a signal suitable for transmission over the channel. The operations includes compression, quantization and encoding. A signal input into the channel is called an i n p u t s i g n a l or a signal to be transmitted. The c h a n n e l is merely the medium used to transmit the signal from transmitter to receiver. I n general an input signal is perturbed by a n o i s e coming from a noise source. A signal received at the output of the channel is called an o u t p u t s i g n a l or a received signal. The d e c o d e r ( r e c e i v e r ) ordinary performs the inverse operation of that done by the encoder. The received signal is decoded by the decoder to reproduce the message. The reproduced message is called the
144
J.J MODEL
OF COMMUNICATION
145
SYSTEMS
received message.
We should note that the actual message is the one selected, w i t h a probability, from a set of possible messages. The system is designed to operate for all possible messages, not just the one which w i l l actually be chosen. The possible messages and the noise are accompanied with statistical structure. Thus, mathematically speaking, messages, signals and noises are described by random variables or stochastic processes. The actual message or the noise is the outcome of the random variable or the process.
Encoder
message
Source
input signal Source Encoder
r
Channel Encoder
X
Channel Source Decoder
User 1 |
received message
Channel Decoder Decoder
-Noise
Y
output signal
F i g u r e 4 . 1 . The model of communication system
Let us consider a communication system of Fig. 4.1. We may denote a message by a process £ = by Y = {Y ;t t
€ S}, an input signal by A" = {X,;t
6 T } , an output signal
e T } , and a received message by r/ = (i),;s € S}. These processes
are defined on a probability space ( f i , 0, P ) . ff we need to clarify that these random variables are functions of u G f l , we write, for example, f, = £j(w). As the time parameter spaces S and T , we may take Z = (0, ± 1 , ±2,...}, R = (—oo,co), or subsets of these sets, accordingly as the features of sources or channels. We have assumed that £ and If, X and Y have the same time parameter spaces because the assumption is satisfied i n most practical situations. A n d we may not lose generality of theory w i t h this assumption. The state space, the set of possible values which £ takes, is called the a l p h a b e t space of £, and each element of the alphabet space is called an a l p h a b e t . The state spaces of (, jj, X, Y state that what kind of alphabets are produced at the sources and the channels. The fundamental problem of communication is that of reproduction of the mes-
THEORY
148
OF INFORMATION
TRANSMISSION
sage either exactly or approximately. Practically, we are not interested i n exact transmission when the channel is noisy, but only i n transmission w i t h a certain fidelity of recovery. Rate-distortion functions, introduced i n Sec. 1.7, play central roles to evaluate the fidelity of reproduction. A n input signal X is given as a result of an encoding. Mathematically speaking, an input signal
X,
s
at time t £ T is a function of
X,(UJ)
t
and £. Hence, we can
write the input i n the form x( )
= x (lU^y,"es}).
t ul
(4.1.1)
t
Similarly, a received message n , s n,(w) is a function of 9 and an output Y, and written i n the form ^ H
= n.({r,( );ieT)).
(4.1.2)
W
For theoretical development of coding theory, i t is more useful to discuss the encoding scheme decomposing into two operations; source encoding and channel encoding, as is pictured i n Fig. 4.1. Similarly, we decompose the decoding into channel decoding and source decoding. Denote by £ and 7j, the output of source encoding and the output of the channel decoding, respectively. Then, symbolically, the information transmission can be expressed as follows. f
^
i
^
X ^
Y ^ v ~ * V
(4.1.3)
Operations by source encoder, channel encoder, channel decoder, source decoder are represented by £ —• £, £ —• X, Y —» Jj, rj —t ij, respectively.
Coding theorems
for information sources are studied i n Sec. 4.3. A channel is corresponding to X
—' Y.
If a channel is noiseless, then just
one o u t p u t signal corresponds to each input signal. I n general, input signals are perturbed by a noise. I n this case, at the output of the channel, we have a set of possible output signals w i t h assigned probability, corresponding to each input signal. Therefore a channel is specified by a conditional probability P(Y
6B\X)
of Y given X, or the corresponding conditional probability distribution t*Y\x(B\x)
= P(Y
e B\X
= x)
(4.1.4)
(see Sec. A.2 for the conditional probability). One of most interesting channels is a channel w i t h additive noise. The input signals of this channel are perturbed by an additive noise. The channel is expressed in the form Y, = X, + Z„
t G T,
(4.1.5)
4.1 MODEL
OF COMMUNICATION
147
SYSTEMS
where Z — { Z \ t 6 T ) is a stochastic process representing an additive noise. Sometimes (4.1.5) is more simply w r i t t e n as t
Y = X
+ Z.
(4.1.6)
A channel given by (4.1.5) or (4.1.6) is called a c h a n n e l w i t h a d d i t i v e noise. I n this case, i f -Y is independent of Z, then the conditional probability (4.1.4) is w r i t t e n as P(Y
G B\X
= x) = P(ZeB-
x\X = x) = p {B z
- x),
(4.1.7)
where J 3 - i = {z;x+z e B}. Thus the channel with additive noise is characterized by the probability distribution nz of the noise Z. In particular, i f a noise i n (4.1.5) is a Gaussian process, then the channel is called a G a u s s i a n c h a n n e l , and the noise is called a G a u s s i a n noise. Gaussian channels w i l l be studied i n detail i n Chapters 5 and 6.
backward channel
Source
Encoder message
Channel
X
input signal
F i g u r e 4.2.
,
Noise
Y
Decoder
output signal
User 1
received message
A channel with feedback
In many practical channels, the encoder receives some information from the receiver. Schematically the channel is described as i n Fig. 4.2. A channel, to transmit an information about the output signal, from the receiver to the transmitter may be called a backward channel. The main channel is called the forward channel, i f we wish to distinguish. The channel as i n Fig. 4.2 is called a c h a n n e l w i t h feedback. A t the input of a channel w i t h feedback, the encoder operates on a message to produce an input signal w i t h the aid of an information which is sent back over the backward channel. I n other words, an input signal is a function of a message and an output signal, and the signal X
t
*«m
=
at time ( G T can be written in the form
x aw,' t
e s } , { n ( w ) ; « < *}).
(4.1.8)
THEORY
148
OF INFORMATION
TRANSMISSION
Compare (4.1.8) w i t h the equation (4.1.1) for the channel w i t h o u t feedback. Generally speaking, the backward channel may be noisy and the transmitter receives only partial information about the output signal. Moreover, there may be time lag, say S > 0, t o send back a signal from the output to the i n p u t . I n this case we must replace tt < f i n (4,1.8) by u < t — S. Therefore, under a mathematically idealized situation, a channel w i t h feedback is represented by (4.1.8). I n this case, we say that the feedback is noiseless and without time lag. Coding theorems for channels are studied i n Sec. 4.5. A source coding theorem and a channel coding theorem are combined, i n Sec 4.6, to obtain a fundamental theorem i n information transmission.
4.2 I N F O R M A T I O N
STABILITY
In general, communication channels are noisy and one cannot expect that messages are exactly reproduced by the receiver. T h e n the practical goal of communication is to transmit messages w i t h i n a given fidelity. As we have discussed i n Sec, 1.7] the fidelity of communication is evaluated by using a distortion. I n this section, we discuss on relationships between the distortion and the mutual information between a message to be communicated and a reconstructed message by a user. In the same manner as in the preceding section, we denote a message to be transmitted and a received message by stochastic processes j = {£,; a £ S ) , and r; = {ry,; S £ S}, respectively. I t is natural to assume that ( and 7; have the same G G, for all s 6 S and w £ Si,
state space (alphabet space) G, namely £,(w), If we denote by G = G
s
= {x = {x,; s € S}; x, £ G]
the space of G-valued
functions defined on S, the stochastic processes £ and n can be regarded as G valued random variables. Denote by
ji^ the probability distributions of £, TJ,
and denote by j i ^ the joint distribution of £ and n. One of most fundamental sources is a memoryless source. Let £ = {/;„; n € S} (S C Z) be a message w i t h discrete time. The source is said to be m e m o r y l e s s if £ — {(*', n £ S} is an i.i.d. process. Then the probability distribution of £ is given by Pit
E B) m l*e(B)
=
r W .
where B = f i n e s n and (i is the identical distribution of { £ „ } . S
(4-2.1) More generally,
4.8 INFORMATION
STABILITY
140
we may say t h a t a source is memoryless, even i f the probability distribution is given by Pt(B)
= I]
(*-2-2)
in place of ( 4 . 2 . 1 ) , where p ' s may be different probability distributions. n
As we have seen i n Sec. 1.7, a distortion d(£,rj) is denned in the form du, ) = {£[pu, } ]}'/ , n
7
2
s
where p(x, y), a function defined on G x G, is a distortion measure. Recall that, given the probability distribution of the message £ and a distortion measure, the rate-distortion function is defined by R ( D ; 0 = inf {/(£,,,); < £ , » , ) < D},
0 < D < oo,
(4.2.3)
where the infimum is taken for all rj = j n , ; s G S ) which satisfy d(£,7j) < D. The rate-distortion function describes the m i n i m u m mutual information we must transmit i n order to send £ w i t h distortion D. On the other hand, the distortionrate function Z > ( f l ; 0 = i n f f ^ . n ) ; J(£,T,) < R ) , i
0 < R < co,
(4.2.4)
indicates the m i n i m u m distortion which is achievable by transmitting mutual information not greater t h a n R . I t w i l l be seen that the rate-distortion function plays important roles to evaluate the performance o f channel coding. We need to treat the case where the time parameter space S is an infinite set such as S = Z"*" = { 1 , 2,...} or S — [0, oo). I n this case, in place of the rate-distortion function, the rate-distort ion function per unit time plays important roles. To define per unit time rate-distort ion function, we are required to introduce the distortion per unit time and the mutual information rate. s 6 S ) and r/ = ( n ; 3 £ S } be G-valued stochastic processes. T h e
Let f =
3
time parameter space S is taken as S = Z in continuous time case. Let p(x,y)
+
in discrete time case and as S = [ 0 , oo)
be a distortion measure defined on G x G,
and define p ( • , •) by s
s Ps(x,y)
2
=
Y,P( "-y^ ' x
2
n=l
in discrete time case, where 5 is a positive integer, and
Ps(x,yf=
[ Jo
m*)iV{*f?**>
150
THEORY
OF INFORMATION
TRANSMISSION
in continuous case. Denoting by
ds«,n) = { | £ [ ^ , ) ] J 2
0
,
(4.2.5)
we define the d i s t o r t i o n 3(£,n) p e r u n i t t i m e (or per step, or per second) by d\t,l) = . m d (£,l),
(4-2.6)
S
provided the l i m i t exists. On the other hand, the m u t u a l information rate /(£,r/) is defined by I(Li)= (see Def. 2.1.6), where S, s < 5 } and r j
( s )
Is(£,y)
lim i/s(£,n), 5—oo J
(4.2.7)
is the mutual information between
—
a £
= {TJ,; S £ S , S < S ) .
The r a t e - d i s t o r t i o n f u n c t i o n J?(.D;£} p e r u n i t t i m e is now defined by R ( D - 0 = inf{7(C,r,)i d t f . r , ) < D],
0 < D < oo.
(4.2.8)
In the same way, the d i s t o r t i o n - r a t e f u n c t i o n D(R; £) p e r u n i t t i m e is defined by D(R;0
= inf{d(£,7,)
i
;
7(^,7,) <
R),
0<
co.
R <
(4.2.9)
T h e o r e m 4 . 2 . 1 . Let £ = {£„;ri £ Z + } be a message from a memoryless source in the sense of (4.2.1). Then the rate-distortion function and the distort ion-rate function per unit time are given by
R(D;0
= R(DUnl
D(R;Q
= D(R;M,
n
€
Z
+
.
(4.2.10)
We prepare a lemma. L e m m a 4 . 2 . 2 . I f j i , £ w are mutually independent, then
/(«i,-,CW).(1I.-.1N))>
(4-2.11)
4.S INFORMATION
STABILITY
151
If ( 6 » > h ) . •••>({iVjIfiv) are mutually independent, then in (4.2.11) the equality holds.
P r o o f . By use of (1.6), we can show that /((ei,...,^),^,-,^))=/{fi.m)+/((6,-,eiv),(»?a,...,«?N))
-
> ii2>
- i CN)) -
>/(fi,«?i) + I ( ( 6
-.7/v))
U),(« ,...,^)), 2
where we have used / ( f i , (£2, . . . , £ « ) ) = 0. Repeating the same arguments, we obtain (4.2.11). The last assertion is nothing but (1.10).
•
4 . 2 . 1 . Since £„, n e Z + , are mutually independent, by
Proof of Theorem
Lemma 4.2.2, i t is enough to take the infimum in (4.2.8) only for the processes *) = {in}
such that ( £ „ , ! ) „ ) , n £ Z , are mutually independent. Thus, taking +
(4.2.5) into account, we obtain 1 R{D-0<
lim
N
TJ:£>(I>.,;60.
(4.2.12)
where D
=
2
lim
i y Z J * .
Using the convexity of R(\/~D; £„) (Theorem 1.7.1), we can show that the equality holds i n (4.2.12) i f D — Di = D
2
= * • •. Consequently we have the first equation
of (4.2.10). The second equation can be derived analogously.
•
Here we discuss the stability of mutual information and distortion. The stability plays significant roles i n proving the source coding theorem (Sec.
4.3) and the
channel coding theorem (Sec. 4.5). Denote by u ^ ' , / i ^ ' and jqj^* the probability distributions of S
T / ' and the
S
joint distribution of £
( 5 )
and I J
time case, and G * ' = G
s
s
( s )
, respectively. We put G <
5)
= Gl
s
0 , 5
l i n continuous
(S is a positive integer) i n discrete time case. If w ^ '
is absolutely continuous w i t h respect to u ^
5 1
x / i ^ , then denoting the RadonS )
N i k o d y m derivative by D
9 (*> y) = (S)
J
AS)
.si \ s ) ( * ' (
<*• f >
e
G < S )
*
G < 5
*'
<-- > 4
2
13
152
THEORY
OF INFORMATION
the m u t u a l information Is( , f
m , i )
=
TRANSMISSION
l ) is expressed as !
(4.2.14)
^S9 (x,y)d^(x,y). lS)
JO< >xGi ) s
s
Note that this can be written as /s(£,n) = £ [ l o g ^ ' ( ? , > , ) ] .
(4.2.15)
s
The function log
=
s)
j(*, ) 6 C V
is called the information density. For each £ > 0, the
x G< >; | | l o g ^ ( * , i O -
| S |
s
Is(Z,l)\
(4-2.16)
<
is cailed the ^ - t y p i c a l s e t . Each element (x, y ) of 7 T ' is said to be a n
e-typical
p a i r . The stability i n information and i n distortion is defined as follows. Definition
4.2.1.
1°)
A pair (£,n) of processes £ =
{£,; s 6 S} and n =
( n ; s E S) is said to be i n f o r m a t i o n s t a b l e , or s t a b l e i n i n f o r m a t i o n , if a
(4.2.17) for every e > 0. 2°)
The pair (£, T]) is said to be d i s t o r t i o n s t a b l e , or s t a b l e i n d i s t o r t i o n ,
if (
4
-
2
-
1
8
)
for every £ > 0, where V[
S )
= { ( « , » ) £ G< > x G< >; | | 5
s
P S
( * , „ ) - d (£, „ ) * | < !
s
•
(4.2.19)
Noting (4.2.15) and (4.2.5), we see that the stabilities (4.2.17) and (4.2.18) are a kind of weak law of large numbers (cf. Theorem 3.2.1). Hence we may expect that the stabilities are satisfied under rather moderate conditions. Here we examine a simple but important example where are i.i.d. random vectors. Then
{£„,7j„},
n = 1,2,...,
4-S SOURCE
CODING
153
THEOREMS
and
n=1
are sums of independent random variables. Moreover, for any iV,
and Hence, applying the weak law of large numbers (Theorem 3.2.1), we can show that (4.2.17) and (4.2.18) are true. Thus we have obtained the following theorem. T h e o r e m 4.2.3. Let { £ „ , n „ ) , n = 1,2,..., be i.i.d. random vectors. I f /(£,,tji) and di(£,T() are finite, then the pair (£,ij) of processes £ = {£„; n £ Z } and +
>7 = { n „ ; TI £ Z ) is stable i n information and distortion. +
4.3 S O U R C E C O D I N G
THEOREMS
This section is devoted to studying source coding theorems. We use the same s € S) a message to
notations as i n the preceding sections. We denote by £ =
be transmitted, and by r? — { n , ; s € S} the output of the source. We may assume that £ and ?j have the same state space G, namely
and T) are G-valued random
variables. Assume that a distortion measure
is defined on
3
p(x,y)
G x G.
For simplicity, i n this section, we consider only discrete time case, assuming that S = Z + . P u t £< > = { £ „ ; n
=
N
denote by
p! , N)
(
ft
l N) 9
and
joint distribution of £
( j V )
1,...,N)
and r)< ' = fa,,; w
w
and T] \ (N
tW
respectively.
For given length iV and rate R , a set v = {v
= (ti ],...,f jv) £ G ;
m
1 , M }
And
n = l,...,N}.
the probability distributions of £ ( > , n » and the
m
N
m
is called an (R, N) c o d e or simply a c o d e , where M = [e ] NR
the largest integer not greater t h a n x). Each v
m
a code w o r d of length N. A random vector £
= (y \,
( w )
m
m =
([x] denotes
"miv) £ G
N
is called
= { £ „ ; n = 1,...,/Y} uniformly
distributed on v : P(£
( N )
= v } = -^, m
m = l,...,M,
(4.3.1)
154
THEORY
OF INFORMATION
TRANSMISSION
is called a quantized message. Intuitively speaking the source coding theorem can be stated as follows. a rate
R
is greater than the per unit time rate-distort ion
R(D;f),
If
then at the
output of the source i t is possible to have a quantized message £ ' ' w i t h distortion N
This implies that, i n order to transmit the
message £ w i t h allowable distortion D, we may input the corresponding quantized message £ into the channel i n place of the original message £.
Moreover the
minimum rate is equal to the rate-distortion rZ(D;£). I t should be emphasized that the m i n i m u m rate depends only on the rate-distort ion
R(D\£),
and does not
rely on other properties of the probability distribution of the message £. To give a precise statement of the source coding theorem, we define the stability of a message. D e f i n i t i o n 4 . 3 . 1 . A message (or a source) £ = {£„; n £ Z ) is said to be s t a b l e , +
if for any fixed
D
> 0 and e > 0, there exists rj = ( n ; n £ Z } such that n
+
(£,77)
is
stable b o t h i n information and distortion, and satisfies (4.3.2) and (4.3.3)
Using Theorem 4.2.1 we can easily show that a memoryless source £ = { £ „ } is stable. Theorem 4.3.1.
(Source Coding Theorem)
Let a message £ = (£ ; n £
be stable and assume that the distortion measure
Z } +
n
p(x,y)
is bounded by
K
<
00.
If R>R(D,0,
(4.3.4)
then, for any 6 > 0 and any sufficiently large jV, there exist an (R, N) code and a quantized source output
satisfying (4.3.1) and
P r o o f . The proof employs a method known as the random coding argument. By the assumption (4.3.4), for any 0 < £ <
^(R-R(D;
£)), there exists rj = { n „ } such
13 SOURCE
CODING
155
THEOREMS
that (£, n) is stable i n information and distortion and satisfies (4.3.2) and (4.3.3). This means that, for sufficiently large JV, d ((,t)
N
+ 6,
2
(4-3.5)
0
where So is a positive number such that D
+ 3£ = (D + $) , and
2
2
0
^I ((,n)
+ 2e.
N
(4.3.6)
In the following we fix a sufficiently large jV, and to avoid notational complexity we abbreviate the suffix (JV) as far as no confusion occurs. Let random vectors V
= {V
m
m n
;
1,...,JV}, m = l , . . . , A f ,
n ~
rj = {rj ; n = 1,...,JV} and let u n
of V .
Note that v = {v }
m
x
—
€
minimizes
ps{x,v )
word
=
x(v)
N
k — 1, . . . , M .
o v e c
£
(xi,...,Xfj)
x(v)
m n
=
be independent copies of
{e \) NR
; n — 1,...,JV} € G
N
code.
be an outcome
Each source input code word
is mapped into that source output code word u
G
k
(M
— {u
is an (R,N)
m
(x\,...,xti)
m
= v
G
corresponding to
N
PN(x,v )
if
m
m
which
In other words, the source output code
=
m
is given by
x
mm
p (x,v ). N
(4.3.7)
k
we define the source output £ — £(v) by
Now, for the message
f(v) = x(v)
if
(4.3.8)
t = x.
Note that the distortion djv(£, £) = d/v(£ >, £ (A,
, w )
) depends on the randomly chosen
codes v — { v J . We denote by F ( • ) the probability w i t h respect to the choice m
c
of v = {i>m}- To evaluate the distortion d j v ( ( , f ) we define a set BN by
B
N
= |(v,
x)
e G
M
N
x G; N
?(v))
jjPN(x,
2
>
D
2
+
2S
Q
j-.
(4.3.9)
Then we shall prove E i / v(i,2(v)) d/ W —oo JV J MNy_QN P i
; i
>
£
x . (v, )
{
I
; !
(4.3.10)
1 )
G
Define sets T ""> and Z ^ c
A
N
w )
by (4.2.16) and (4.2.19), and define a set A
- {(*,«) 6 G "; £ * i f W f a « ) $ $ J
~PN(x,y)
2
< dj ,
N
by (4.3.11)
156
THEORY
OF INFORMATION
TRANSMISSION
= R - c ,
D +26 ,
where R
d =
2
and dp
here
a
(N)
Me
(- |i) denotes the conditional distribution of n
( l V
' given
R - K(.D; 0 > 4e, (4.3.6) and (4.3.5), we have
Then clearly
By the stability o f ( £ , n ) ,
fc.* (^)"JSL*,(»P)-',
Therefore M
Denote by ( A N ) Z
I i m ^
(
j
4
w
)
=
1
.
the z-section of A N (AN), = { y € G ; («,*) e A } . N
N
Then for each fixed i we have
p
c
(^^(^^vj)
1
>
ij
= n
p
£
^~ ( v ) PN Xl
m
> d)
2
.if
M
= fl - / L
/(Air).
4^%)
4-3 SOURCE
CODING
THEOREMS
Noting (4.3.11) and (4.3.12), we have
-
< exp
/
exp(-NR)dp^ \y\x) (
-Mexpl-NR)
/
f
d^.\y\x
d ^ ' ( ! , k ) + exp[-Afexp(-/V/£)],
here we have used the inequality exp(-Max)
1-
<
+ exp(-A/a),
x
0 < * < 1.
Therefore, i t follows from (4.3.14) and (4.3.15) that
*
f
JO"
1
_
/
d^>(y\x)
[
J(A„),
=
,
+
xp[-Mexp(-NR)]
e
K
p £\A%)+exp[-Mexp(-NR)}. i
Noting Mexp(-NR)
=
\ c
N
\ e -
R
N
R
+
N
>
e
t '(l N
-
«-"*),
we have l i m e x p l - A f exp(-JVJi)] = 0. N—oo
Combine (4.3.13), (4.3.16) and (4.3.17) to obtain l i m P « X | » i ' ° ( B j f ) = 0. (
It is clear that
i
/
^ f l ' + Zfo +
• |( ,9[T)] 4ftKMfv,s) J
a
>
i Y ' P c X ^ K ) .
158
THEORY
OF INFORMATION
TRANSMISSION
Now the desired inequality (4.3.10) follows from (4.3.18) and (4.3.19). I t is clear from (4.3.10) that
1
f
p (x,x(v))
for sufficiently large N. v = {v } m
e
c
2
+
3fi
0
=
(D
+
S) , 2
This means that there exists at least an {R,N)
code
for which
is satisfied, where complete.
x p (v,¥)
dP
2
N
is the quantized message corresponding to v. The proof is
•
The source coding theorem tells us that there exists an o p t i m a l source encoding £ —• £ (ci. the scheme (4.1.3)) and that the encoding is a quantization. Finally we note that the source coding theorem holds for a memoryless source, because it is stable. C o r o l l a r y . I f £ = {£„; n — 1,2,...) is an i.i.d. process, then the channel coding theorem holds for the message £ =
4.4 C H A N N E L
{£ }n
CAPACITY
We consider a communication channel and discuss the channel capacity. Using the same notations as i n Sec. channel output by stochastic processes
X
4.1, we denote a channel input and a =
{X(t);
t €
T } and
Y
=
{Y(t);
t
respectively. Let H be the state space of X and K, namely X ( t ) and Y(t) valued random variables. We may regard the processes Y
=
{Y(t)\
t
X
=
[X(t)\
t
£ T}, be H-
£ T } and
£ T ) as H = / f - v a l u e d random variables. T
A communication channel is, as was mentioned i n Sec. 4.1, essentially specified by a conditional probability P(Y
g B \ X ) of Y given JV, or the corresponding
conditional probability distribution p ]x(B\x) Y
= P(Y
£f?LY =
z),
(4.4.1)
4-4 CHANNEL
158
CAPACITY
where x £ H a n d B is a measurable set i n H . See Sec. A.2 for the conditional probability and also Sec. 1.7 for the conditional probability distribution. One of most fundamental channels is a memoryless channel. A channel is said t o be memoryless i f the probability distribution of the output at that time depends only on the channel input at that time and is conditionally independent of previous channel inputs or outputs. To be precise, let T = {1,2, —,N} and let u( - | •) be a conditional probability distribution. Then a m e m o r y l e s s
channel
is characterized by N
"<
PY\X(B\*)=n
f l
»M>
( - - ) 4
4
2
n=l
where B — Bi x • • • x Bs and x = (x\, ...,i/v). More generally, we may say that a channel is memoryless, even if i t is characterized by N
HY\x(B\x)=Y[»*(.B \ ), M
XN
(4.4.3)
n=l in place of (4.4.2), where e ( -1 - )'s may be different conditional probability distrin
butions. I n the case of continuous time, one can introduce a memoryless channel in an intuitive manner. However, in this case, we need mathematical justification (see Sec. 6.1). I n practical situations, a channel has some constraints imposed on the channel input. T h e simplest type of constraint is an amplitude constraint, where the input signal is l i m i t e d t o taking values not greater than some fixed value p:
\x,\< ,
v t e x
P
More common and important types of constraint are described as 1 —
or
f
T
I
T Jo
w i t h probability one,
X dt
2
E[X ]dt
ij£
2
2
(4.4.4)
(4.4.5)
where T = [0, T\. The conditions of type (4.4.4) and (4.4.5) represent constraints concerning the energy to transmit, and are called power constraints.
Especially
(4.4.5) is called an average p o w e r c o n s t r a i n t . More generally the average power constraint can be described as ±
£E[${X ) ]dt
2
2
(4.4.6)
160 where
THEORY
OF INFORMATION
TRANSMISSION
is a positive valued function defined o n H.
I n discrete time case,
important types of power constraint are represented by 1 — Y
N
X* - P ' 2
w
i
t
o
probability one,
n=l
n—1
and more generally
4£aw*«?j£r*,
(4-4.7)
where T = {1,...,JV}. In general, a constraint on the input can be described by a class o f all possible probability assignments. Denote by X the class of all input signals X = {X, ; t e T ) which satisfy the constraint. Each stochastic process X fj X is said t o be an admissible input. Thus we can say that a channel is specified by a conditional probability (4.4.1) and a set X o f all admissible inputs. The channel capacity is t o be defined as the m a x i m u m m u t u a l information one can transmit over the channel. A t first we give the precise definition of the capacity in the case where the channel is without feedback. D e f i n i t i o n 4 . 4 . 1 . The c a p a c i t y C = C { X ) of a channel without feedback is defined by C = sup{/(X,y-);Xe^},
(4.4.8)
where X is the class of all admissible input signals, and Y is the output corresponding to the input X. To be precise, Y is related t o X w i t h (4.4.1).
The capacity defined above is sometimes called the " i n f o r m a t i o n " capacity to distinguish from the "coding" capacity.
The latter is defined as the maximum
rate one can transmit over the channel w i t h arbitrarily low probability of error. Shannon's fundamental theorem establishes that the information capacity is equal to the coding capacity (see Sec. 4.6). Thus we drop the word " i n f o r m a t i o n " i n most discussions on the channel capacity, If a channel is w i t h feedback, the situation becomes a l i t t l e more complicated, because a channel input X at instant t depends not only on a message { b u t also t
on the channel output {Y,\ s < t} up to ( (cf. Fig. 4.2). Hence the channel is
4-4 CHANNEL
161
CAPACITY
specified, n o t by a class X of all possible inputs, but by a class A of all possible pairs (£,X)
of a message f and an input X.
Each pair ( f , X ) G A is called
an admissible pair. Moreover, i f the channel is w i t h feedback, then the mutual information I(X,
Y ) between an input and the corresponding output counts not
only the forward information transmitted from the transmitter to the receiver but also the information sent back over the backward channel. This implies that i t would be a mistake to use I(X, In (4.4.8), I(X,
Y ) to define the capacity of a channel w i t h feedback.
Y ) should be replaced by the mutual information
Y ) between
a message £ and the corresponding output Y. D e f i n i t i o n 4.4.2. The c a p a c i t y C
s
= C/(A)
of a channel w i t h feedback which
is specified by (4.4.1) and the class A of all admissible pairs is defined by C
= sx p{Itt,Y)-,{t,X)€A},
f
(4.4.9)
t
where Y is the output corresponding to the input X. Remark.
I f the channel is without feedback, then the right-hand sides of (4.4.8)
and (4.4.9) coincide w i t h each other. I n fact, since input X is encoded from the I(X,Y),
we know that the supremum i n (4.4.9) is
achieved i n the case where ( = X.
message £ so that /(£,
Thus the supremum i n (4.4.9) is reduced to
Y)
<
that i n (4.4.8). When the alphabet space H is a finite set, an algorithm has been known to calculate the capacity ([3, 13]). I n general, i t is not easy to give an explicit formula to calculate the channel capacity. Especially i t is a hard task i f the channel is with feedback. The capacity of Gaussian channels w i l l be treated i n Chapters 5 and 6. We have defined the capacity of a channel w i t h finite time interval. However i f the time d u r a t i o n is very long, then the capacity per unit time will play a more important role. Let T — Z
+
or T = [0,oo). Suppose that some constraints are
imposed on the channel input signals and denote by X the class of all admissible inputs X = {Xt\ i £ T ) . (£,X) I(X,Y)
We also denote by A the class of all admissible pairs
of a message £ and an input X for a channel with feedback. Denote by the m u t u a l information rate between
X
=
{X,;
t €
T } and
T } (see (2.1,14) for the definition). The mutual information rate
Y
I(£,Y)
=
{Y ; t
t £
between
a message £ and the o u t p u t Y is similarly defined by i(i,Y)=
hm i/ a,r), I —*oo J T
(4.4.10)
THEORY
OF INFORMATION
provided the l i m i t exists, where
I T ( * , Y )
-
TRANSMISSION /(£,
t
{Y,\
<
T}).
D e f i n i t i o n 4.4.3. The c a p a c i t y C = C ( X ) p e r u n i t t i m e of a channel without feedback is defined by C = sup{7(*, Y ) ; X G X } , where Y = { Y , ; ( e T )
(4.4.11)
is the output corresponding to the input X . Similarly,
if a channel is w i t h feedback, then the c a p a c i t y C
s
= Cj(A)
p e r u n i t t i m e is
defined by C = upCl(f,Y);(t,X)eA}. f
(4.4.12)
S
We may call C and C / the capacity rates. Finally, as an example, we consider a memoryless channel given by (4.4.2).
Let X be a class of /f-valued random
variables. We define a class X of all admissible messages by X
=
{X
=
{ X
N
} ; X
N
£
X,
Vn}.
Then the unit time capacity C of the memoryless channel is equal to c
=
c,
where C is the capacity of a single channel specified by the conditional probability distribution i>( -1 -) and the class X .
4.5 C H A N N E L
CODING
THEOREMS
I n this section we prove the channel coding theorem which is one of the most basic theorems i n information theory.
Intuitively speaking, the theorem states
that, i f a message has entropy less than the channel capacity, then one can transmit it over the channel w i t h very small error probability of reproduction at the channel output by using an appropriate encoding and decoding scheme. A n d conversely, if the entropy of the message is greater than the capacity, i t is impossible to transmit the message over the channel w i t h arbitrarily low probability of error. Let us consider a communication channel and, as i n the preceding sections, denote by X — {Xt\ ( £ T ) corresponding to X. each X
t
the channel input and by Y = {Y \ i g T } the output T
We assume that the state space of X and Y is H so that
and Y, are ff-valued random variables.
4.5 CHANNEL
CODING
163
THEOREMS
I n the investigation of the coding theorem, we need to discuss i n slightly different ways according to whether the time duration T of the communication is finite or infinite. A t first we study the coding theorem i n the case of infinite transmission d u r a t i o n , assuming that T = Z
+
or T = [0, cc). For simplicity, we shall
concentrate on the discrete time case. The continuous time case can be treated in a parallel way.
I n this section we always assume that the channel is without
feedback, unless otherwise stated. Let X = {X„\
n £ Z ) be an input and Y = {Y„; n € Z ) be the corresponding +
+
output. As usual, we assume that the channel has an average power constraint imposed on the channel input. We may assume that the constraint is presented i n the form lim
—
E
£ * ( x , )
N—oo JV
:
(4.5.1)
,n=l
where * ( i ) is a positive valued function defined on the state space H. For convenience, we denote by X(p) X(p)
the class of all admissible input signals: = { X = ( X „ ) ; X satisfies (4.5.1)}.
Then, by definition, the capacity C = C(p)
per unit time of the channel is given
by C(p)=sup{I(X,Y);XeX(p)}
(cf.
(4.4.11)).
Denote by
X< > N
=
and
{X ...,X } U
N
Y
t N )
=
the input and the
{Y,,...,Y } N
corresponding o u t p u t of the channel up to JV, and denote by ( and >; a message to be transmitted and a received message, respectively. For each JV, let A ' ' ( p ) be w
the class of all admissible pairs ( £ , X ^ ^ ) of a message £ and an input X^ ' N
that X^
such
N
is a function of £ satisfying the average power constraint (4.4.7). Here
we consider a quantized message £< > w i t h probability w
PtfW =
m
) =
^
,
where J f C ' > 0 is an integer. Suppose that and the corresponding o u t p u t y ' ematically speaking, X^ ^ N
l V
m = l,...,M< >, w
is encoded into an input
' is decoded into a received message
is a function of
and
The error probability i r ' ' of reproduction is now given by w
(4.5.2)
is a function of
X^ ^ N
if'MathY^ \ N
THEORY
L64
OF INFORMATION
TRANSMISSION
Since the message £< > has entropy # ( £ < " " ) = log M > > , if log Jtf< > is less t h a n N
w
w
the capacity of the channel of length JV, then we may expect that £ mitted w i t h very small error probability of reproduction.
< N )
is trans-
T o give the precise
statement of the coding theorem we need to define the achievability of information transmission rate, and also need to introduce the concept of stability of a channel. D e f i n i t i o n 4 . 5 , 1 . A rate R > 0 is said to be a c h i e v a b l e , i f for a message £
(ftr)
given by (4.5.2) w i t h M< > = w
there exist an input A " A >(p) (N
( w )
[c ] NR
and a received message n
( N
* such t h a t ( £
< w )
,X
( N i
) €
and
l i m TT"*) - 0.
(4.5.3)
N —»oc
We can say that, i f a rate R is achievable and the transmission duration JV is large enough, then M
{N)
= [e \ NR
different message sequences can be reconstructed with
very low probability of error at the output of the channel. Note that the maximum achievable rate is called the coding capacity of the channel. D e f i n i t i o n 4,5.2. A channel subject to the constraint (4.5.1) is said to be s t a b l e , if for any 6 > 0 there exists an information stable pair (X, Y ] of an input X = and the corresponding output Y = [Y ] n
{X„}
such that X satisfies the constraint (4.5.1)
and T(X,r)>C(p)-^, where C ( p ) is the channel capacity per unit time. We now give the precise statement of the channel coding theorem and its converse. Theorem
4.5.1.
(Channel Coding Theorem)
Let the channel be stable and
without feedback. Assume that the per unit time capacity C ( p ) under the constraint (4.5.1) is continuous at p. Then a rate R < C ( p ) is achievable. T h e o r e m 4.5.2. (Converse to the Channel Coding Theorem) achievable, then R < C(p).
If a rate R is
T h e channel coding theorem, with its converse, is also known as the Shannon's second theorem.
T h e theorem indicates that, if the entropy of a message ( is
4.S
CHANNEL
CODING
THEOREMS
165
less than the capacity, then there exist a channel encoding £ —• X and a channel decoding Y — rj (ci. the scheme (4.1.3)) by which £ is transmitted w i t h a very small probability of error. The theorem also means that the coding capacity is equal to the information capacity. P r o o f o f T h e o r e m 4 . 5 . 1 . Let 0 < 4e < C(p) - R. Then there exists a constant 0 < pa < p such that
Ze
(4.5.4)
0
and
there exists an information stable pair (X,Y)
and
the corresponding output Y = {Y }
of an input X = {X„}
£ X(pa)
such that
n
7(X,Y)>C(p )- -. 0
E
Therefore, < p 1
(4.5.5)
2
_
j I {X,Y)>C{p,)-c
(4.5.6)
f N
are true for sufficiently large N.
(X,Y):
?}
N )
-
{<*, y) G H ;
i
2N
Denote by T
e
( W 1
the e-typical set for the pair
| W * > ( * , y) - / * ( * , K ) | < e }
(4.5.7)
(cf. (4.2.16)), where
Let a message £< > be the random variable given by (4.5.2) w i t h M
{N)
N
Recall that a set u = (u ;
m = 1
m
H
N
M
( w |
}
of code words u
= (u
m
m
= i,u
\t ]. NR
m
w) €
of length iV is called an ( i t , N) code. We now define the encoding and decoding
scheme as follows. Encoding:
Given an (fl.JV) code u = ( u „ ) , to send £
inputs into the channel a signal =
u
m
if
= fW=m,
(U ...,U ) U
N
, w t
,
the transmitter
which is determined by m = l,...,M< >. w
(4.5.8)
THEORY
166
Decoding:
OF INFORMATION
TRANSMISSION
Receiving the channel output signal V ' ' ' = ( V j , V A T ) , the reV
ceiver declares that £
= m was sent i f ( « , V
( N )
m
Namely, the received message ft") =
| N |
| is the only e-typical pair.
is given by
if
Note that the average power of the input signal is given by
n—t
m=J
To evaluate the average power we define the set D
( l V 5
( p ) by
- )
2
< P
2
and we call each u £ Z ) ' ' ( p ) an admissible code. Now what we have t o show is the w
existence of an (R, N) code u = { u } £ D (p)
for which (4.5.3) holds. For this
m
m
purpose, similarly as in the proof of the source coding theorem (Theorem 4.3.1), we use the random coding method. Let t i = { u „ ; n = 1,...,N}, m = m
1,...,M' ' IV
m
1
IN)
be code words independently drawn according to the identical distribution / i ^ Note that u = { " m l TI = 1 , M '
f t r
' } is an (R,N)
code. Define the input signal
[/(AO _ (CTj,..., (7jv) by the encoding rule (4.5.8). Denote the corresponding output signal by V ' ^ ' = (Vj,...,Vjv) and define the received message £ t ' according to w
the decoding rule. I t should be noticed that the average power E [ $ ( [ / ) ] and the J
n
error probability
depend on the choice of code u = { u } . m
We denote by P ( •) and E [ - ] the c
probability and the expectation w i t h respect to the choice of u =
e
{u ). m
By
(4.5.5), X>(«™.)
J
for every m , and using the law of large numbers we have the inequality (4.5.9)
4.5 CHANNEL
CODING
167
THEOREMS
for sufficiently large JV. To evaluate the error probability 7 T B
m
=
{v
g H-
= {v!,...,v )
occurs i f E [*m\
< Pc(Bfl£
t
Since each u
m
n
1 ) + M^P (B \^ '>
-
( N )
= 1 was sent. Then the error
( N f
for some n # I , Therefore
€ B\ or V > €B nB l
C
=1).
N
1
is drawn according to the distribution
P (BZ\fW c
= ^
=
\
w
(4.5.10)
know that
e
1)
g H;
(^y)
= 1 - ^ W
.
m
We assume w i t h o u t loss of generality that £ ( N
, define events
(u ,„) e T ^ }
N
N
(|V)
|
N
)
|lo ><">(x,!,) - 7 ( X , r ) | > e ) gV
w
) .
(4-5-U)
Hence, i t follows from the stability of (X, Y) Hm P (Bt\C
yv—oo
= W =0.
(N)
c
that (4.5.12)
Since V ' " ' is independent of u when j f W = 1, we see that 2
P e C B )
[fW
= 1
)
=
<
J
i & x & t o , )
f J <»)
^-I (X,Yy
+
N
lt)dp g(x,v)
e
l
T
<exp(-I (X,Y)
+ eN),
N
(4.5.13)
and, by (4.5.4) and (4.5.6), that M' P (Bi\i N)
c
= 1) < exp{NR
(N)
- I {X,Y)
+ eJV} < exp(-eJV).
N
(4.5.14)
Combine (4.5.10), (4.5.12) and (4.5.14) to obtain lim
E [^ ]
N—oo
C
N)
=
0.
Denote by E [ir< >| £>(*>{/>)] the conditional expectation on the event c
w
Then i t follows from (4.5.9) that
U {p). lN)
168
THEORY
OF INFORMATION
TRANSMISSION
Consequently lim E [wW\D \p)]
(4.5.15)
= Q.
l,t
c
TV—-co
This means that there exists at least an admissible (N,R) that (f,< >,[/< >) £ A< >(p) and limjv_™ N
w
N
code u = { u } m
such
= 0, where t/ > is the channel
7r< >
(JV
w
input corresponding t o u . Therefore the rate R is achievable.
•
P r o o f o f T h e o r e m 4 . 5 . 2 . Assume that a rate R is achievable. Then there exist an input
X ~>
=
lN
{X„)
and a received message i f
such that ( f \ J C ' " ' ) ( w
( J v t
£
A > ( / j ) and (4.5.3) is satisfied. Noting (w
/ < £ < " > , =
fftf*))
- B$m,„(*>)
= log([e
W R
]) -
flK<"»|,W),
by Fano's inequality (Theorem 1.2.4) we have
(i - i r W j f c ^ M j ) _ fc »(«)) < C
r w,W) < (€
where ft(a) = - i l o g i - ( 1 - at)log(l - x) and Y sponding to X ^ K
I n { X i Y )
= {Y„)
( N )
,
is the output corre-
Therefore
N
1(1
- wW^NR
1/ (*,n
- i) - !*(*<">) <
N
Letting JV to infinity, we obtain the inequality R < C(p).
•
I t should be emphasized that the stability property of the channel has played an i m p o r t a n t role i n proving the coding theorem. The property of information stability has been investigated i n detail by Pinsker [97].
We may say t h a t , i f
a channel is stable, then the coding theorem can be established for the channel, whether the channel is of continuous time or discrete time, and whether the channel is w i t h or w i t h o u t feedback. In the case of continuous time, we replace the average power constraint (4.5.1) by hm
dt
— E
T—co T
Jo
(4.5.16)
2
Furthermore the stability of the continuous time channel is defined analogously as in Definition 4.5.2, using the distribution /j / (resp. p ^ ) of X^ 1
t < T ) (resp. K
, T )
)
r
xm
= {X,;
1
and the joint distribution f/gy of A"< > and K
< 7 - )
0 <
i n place of
p'x' (resp. /i^) and Pxy, respectively. Then the coding theorem can be proved for the continuous time stable channel without feedback.
4-S CHANNEL
CODING
169
THEOREMS
I D Chapters 5 and 6 we shall prove the coding theorem for Gaussian channels w i t h feedback, by showing the stability of the channels. As a fundamental and important example, let us consider a discrete time memoryless channel. The channel is specified by a conditional probability (or a transition probability) l/( • | • ) . Let X = {X„}
and Y = { Y „ } be an input and the corre-
sponding o u t p u t . Then the conditional probability of Y
= {Y,
LM
X
l
N
'
=
P
(r< >
x
€ B,
N
YN)
given
is given i n the form
{Xi,...,Xfj}
•x
B \XW
= <*,,
N
= j j N£BL>«3
...,x )) N
6 / f and B„ is a measurable set in H). has an average power constraint
(4.5.17)
Assume that the channel
lv
(4.5.IS)
Denote by C ( p ) the capacity of the memoryless channel subject to ( 4 . 5 . 1 8 ) . For any
0 <
p
0
< p, there exists a pair
(X,Y)
=
such that
({X },{Y„}) n
(X ,Y ), n
H
n = 1,2,..., are mutually independent, with the identical distribution, say VXY, and satisfy I { X
n
, Y
) > C {
n
P
0
) -
e
- ,
and that X satisfies ( 4 . 5 , 1 8 ) w i t h p in place of p. Define a functional tp(x, y) by 0
dvx
where i/x (resp. information
IQ
v y ) is the distribution of X„ (resp. n
Y ). n
Then the mutual
is given by
I(X ,Y )
—
x
n
h =
j
\ogy{x,y)dvx.
=
Y
E[\og
and the £-typical set T t i s now given i n the form 1
N
— ^log^(i„,j/ )-J n
0
n=i
where a: = ( z i , ...,XN),
y = ( y i , •-•> i//v) € H . N
1 <eV, J
Hence ( 4 . 5 . 1 1 ) becomes
/> (5fU = 1) c
(w|
1
\
W
— ^log¥>(i ,y )-/ n
n—
I
n
0
>el. /
THEORY
170
OF INFORMATION
TRANSMISSION
Let M i be the class of all probability measures on H
2
A
C
=
G M,;
<| 7
Then, noting that (X ,Y ), n
j
logf>(x,y)da(x,y)
and define a subclass -
7| > ej
.
0
n = 1,2,..., are i.i.d. and using the large deviation
n
theorem (Theorem 3.2.6), we know that (X,Y)
is information stable and that
Jim llo P,{SfiC< > = l) = -a«, w
g
where o
s
H(A \v )
=
t
XY
= 'mi{H(
VXY)',
e £A
C
) ,
and //(-;•) denotes the relative entropy. Therefore i f 0 < a < o
c
and jV is large
enough, then $$Bft&**
= 1) <
e- . a N
Moreover we can show
Ec&W\E0)ty))
<
2e~ + 2e~' , aN
N
yielding (4.5.15). Thus we have obtained the coding theorem for the memoryless channel. T h e o r e m 4.5.3. The discrete time memoryless channel given by (4.5.17) subject to the constraint (4.5.18) is stable. Assume that capacity C(p) continuous at p. Then a rate R is achievable if R < C(p). is achievable, then R <
of the channel is
Conversely, i f a rate R
C(p).
We note that, for the memoryless channel we can show the stability and the coding theorem, even i f the constraint (4.5.18) is replaced by (4.5.1). Finally let us study the coding theorem for a channel of fixed finite time length. As i n the preceding sections, we denote by X = {X,;
t £ T } a channel input and
by Y — {Yt', t £ T ) the output corresponding to X, where we assume that T { 1 , . . . , ! } or T =
=
[0,T] (L, T < oo). Suppose that the channel is w i t h o u t feedback
and has the constraint (4.4.6) or (4.4.7), replacing N by L . Denote by C =
C(p)
the capacity of the channel. Then, by definition, there exists an admissible pair (£,X)
of a message £ and an input A" such that the mutual information
I(£,Y)
between £ and the corresponding output Y is approximately equal to the capacity C. On the other hand, consider a message £ w i t h probability
P(( = m) = ~
>
m=l,,„,M.
4-6 FUNDAMENTAL
If M < e ,
THEOREM
IN INFORMATION
then the mutual information /(£,£) = H(Q
c
TRANSMISSION
171
= l o g M is less than the
capacity C. However, i n general, i t is impossible to send £ over the channel w i t h small error probability because of fixed finite time length. This implies that, to establish the coding theorem for the time fixed channel, we have to modify the definition of achievable rate. Now let us suppose that the channel is used repeatedly.
For simplicity, the JV
times independently repeated channel is called the JV-repeated channel. Evidently, the capacity of the JV-repeated channel is equal to NC.
Although we may not
expect to t r a n s m i t data reliably over every subchannel at a rate R close to the capacity C, we can expect that, if JV is large enough, messages w i t h entropy less than JVC are transmitted over the JV-repeated channel w i t h arbitrary low error probability. Now define achievability of a rate by Definition 4.5.1, understanding that J f ' ^ ' and y < l are respectively the input and the output of the JV-repeated w
channel. Denote H
=
Jf . T
T h e n the JV-repeated channel can be regarded as a H -
valued memoryless channel. Consequently, the argument we have adopted to prove Theorem 4.5.3 can be applied for this channel. Theorem
4 . 5 . 4 . Let the channel of fixed time length be without feedback and
have the constraint (4.5.19) or (4.5.20). Assume that capacity C[p) of the channel is continuous at p. I f R < C(p), rate R is achievable, then R <
then the rate R is achievable.
Conversely, i f a
C(p).
4.6 F U N D A M E N T A L T H E O R E M I N INFORMATION TRANSMISSION
We are interested i n transmission of a message over a given channel within a certain fidelity of recovery. There are two fundamental problems. The first one is to determine the m i n i m u m distortion one can achieve in transmission of a message. The second one is to construct an encoding and a decoding which are optimal i n the sense t h a t the m i n i m u m distortion is achieved by using these coding schemes. Although the second one is quite important, we do not go into detail. Here we treat mainly the first problem. For a message £, once a method of encoding and decoding is specified, the received message n is determined and the reproduction error (or distortion) d(£,n)
172
THEORY
OF INFORMATION
TRANSMISSION
is computed by (1.7.4). The infimum of reproduction errors D*(£)=infd(£,n)
(4.6.1)
for all possible methods of encoding and decoding is called the m i n i m u m r e p r o d u c t i o n e r r o r . Clearly D'(£) p(x,y)
is determined depending on the distortion measure
appeared i n (1.7.4) and the channel. Roughly speaking, i t will be shown
that D ' ( £ )
depends essentially only on the rate-distortion R(D;£)
of £ and the
capacity C of the channel, but not on other aspects of the message, the distortion measure and the channel. T h e o r e m 4 . 6 . 1 . Suppose that a message £ w i t h rate-distort ion function R[D; £) is transmitted over a channel of capacity C. Then D'(0>D(C;0,
(4.6.2)
where D(R\ £) is the distort ion-rate function of £. Moreover, i f R(D\f)>C,
(4.6.3)
then i t is impossible t o transmit £ w i t h reproduction error d(£, rj) < D, whatever encodings and decodings we use.
P r o o f . Since the m u t u a l information /(£, T/) between the message £ and a received message r/ is not greater than the capacity C, (4.6.2) is clear from the definitions (1.7.5) and (4.6.1) of R(D;
£) and D * ( f ) .
Suppose that there exist an encoding and a decoding which result a reproduction error of d(£, n) < D. Denote by Y the channel o u t p u t . Then I(t,v)
since n is a function of Y. On the other hand, R(DU)
by (1.7.7). Hence R(D;0
<
I(tr,),
< C. But this contradicts (4.6.3). Therefore, under the
condition (4.6.3), there are no encodings and decodings satisfying d(£, n) < D.
•
Since the rate-distortion function is continuous (Theorem 1.7.1), the last assertion i n Theorem 4.6.1 is equivalent to saying that R(D'{0\0
(4.6.4)
4-6 FUNDAMENTAL
THEOREM
IN INFORMATION
173
TRANSMISSION
holds. We know from Theorem 1.7.2 that (4.6.4) is equivalent to (4.6.2). Therefore, in reality, the two assertions i n Theorem 4.6.1 are equivalent to each other. Theorem 4.6.1 indicates that, i f we can find an encoding and a decoding for which d ( f , i f ) =
holds, then the equality holds i n (4.6.2):
D(C-£)
D*(0
= D(C;0
(4.6.5)
and the encoding and the decoding are surely optimal. For Gaussian channels, such examples of optimal codings will be given i n Chapters 5 and 6. However, as long as the communication duration is finitely fixed, we cannot expect to find such an optimal coding satisfying (4.6.5), except some rare cases where the message is well adjusted t o the channel. Rather surprisingly, i f the communication duration is sufficiently long, then (4.6.5) is approximately true under moderate conditions. This fact is known as the Shannon's fundamental theorem or the fundamental theorem i n information transmission. A t first, to avoid unnecessary technical complexity, we concentrate ourselves to discrete time case. Let £ = (£ ; n (E Z ) +
n
user. We denote by X = {X„;n
e Z } +
be a message to be communicated to the and Y = { Y ; n € Z ) n
+
an input signal
of a channel and the corresponding output signal. Roughly speaking, what we are going to show is that, i f the rate-distortion
R(D;£)
is less than the channel
capacity C, then there exist a source encoding £ —* £, a channel encoding £ —' X, a channel decoding Y —• 7, and a source decoding 7 —> 7, such that the resulting distortion d(£, 7) is less t h a n D, where the distortion is measured by the distortion d(£, 7) per unit time, defined by (4.2.6). The existence of such a source coding is assured by the source coding theorem (Theorem 4.3.1). While the existence of such channel coding and decoding is proved i n the channel coding theorem (Theorems 4.5.1 and 4.5.2). Theorem
4.6.2.
(Fundamental Theorem i n Information Transmission)
be a stable message w i t h rate-distortion function
R(D,£),
Let £
the channel be stable,
and the constraint (4.5.1) be imposed. Assume that the distortion measure p ( x , y ) in (4.2.5) is bounded and that the channel capacity C(p) R(D;t)
is continuous in p. I f (4.6.6)
then for any 6 > 0 there exist, for sufficiently large JV, a coding scheme and an auxiliary message T<
N|
= { 7 ; n < N}
such that
n
d (( \fi< ) N
iN
N)
< D
(4.6.7)
THEORY
174
OF INFORMATION
TRANSMISSION
and P <"> *
< S,
( n
where r /
(4.6.8)
= {Tj„;n < A'} is the message reproduced by the user.
( N |
P r o o f . By the assumption, there exist D' > D and R such that R(D';0
Applying Theorem 4.3.1, we know t h a t , for sufficiently large JV, there exist an (R,N)
code v = { u } m
and £
( w )
satisfying (4.3.1) w i t h M = Jtf< > = [e ] w
and
NR
d (f<">,£< >)
0
(4.6.9)
w
N
by
N>
g^-W
if
^ >
=
W
m = l,...,M -
»*,,
(4.6.10)
{N>
On the other hand, by Theorem 4.5.1, the rate R is achievable. This means that, for the message f J
W )
, there exist an input X
and a received message f/J^ such
(Ni
1
that
where $ ^ , X
m
sponding t o X<- K N
)
and
€ A< \p) N
is a function of the o u t p u t y l " > corre-
Now define n ' " * and ^
^
W )
by
=-»>
W - l
M< >,
(4.6.12)
w
and ijW = P>.
(4.6.13)
Then (4.6.7) is clear f r o m (4.6.9), and (4.6.8) is derived from
The proof is complete.
•
The coding scheme we have found i n the proof can be expressed i n the form e
t*J _
g*>
_
x
w
_
yW
_
qg*
_
,
(
4
6
1
4
)
HISTORICAL
175
NOTES
(cf. (4.1.3)). The intuitive meaning of (4.6.7) and (4.6.8) w i t h auxiliary message fj^ ^ is that one can change the received message I J ' ^ ' w i t h very small probability N
8 and obtain the result that the pair (f} \7* ') N
fidelity.
For a fixed length iV, define the m i n i m u m reproduction error D (f)
ing d(£, »j) by d {(, N
N,
and
N
the distort ion-rate function large
satisfies the required condition of
N
D' (() N
by (4.6.1) and (4.2.4), respectively, replac-
DN(R\(.)
n). Then Theorems 4.6.1 and 4.6.2 imply that, for sufficiently
is approximately equal to
namely (4.6.5) holds ap-
Dfj(NC;(),
proximately. The conclusion obtained i n Theorem 4.6.2 remains true whether the time parameter spaces S and T are discrete or continuous. I n the following, we assume that S and T X
are Z
+
=
or [0,co), and denote
= { X ; t £ T, f < T}.
{(.;s
£ S, s <
S } and
We can prove the following theorem. The proof is
t
omitted. T h e o r e m 4.6.3.
(Fundamental Theorem i n Information Transmission)
(£ ;s £ S} be a stable message w i t h rate-distortion function R(£*;£), 3
Let £ =
the channel
be stable, and an average power constraint be imposed. Assume that the distortion measure
p{x,y)
is bounded and that the capacity
is continuous i n the power
C(p)
p. Then, for any 8 > 0 and sufficiently large 5 and T ( 5 and T depend on 8) such that
S(AO<|c(p), there exist a source encoding £
—* £
( s |
( s )
, a channel encoding £ > —» X \ |5
iT
a
channel decoding y< ') —* T)* >, a source decoding T^ ^ —» r / ' , and an auxiliary 7
s
S
s
message rf ^ such that s
d
s
( t
(
S
)
r f
s
)
) <
D
and
P(r/ >/ip)<£. s
HISTORICAL
NOTES
Most of the results and concepts here are essentially due to Shannon [103, 105]. The basic idea of rate-distort ion theory was introduced i n [103] and was developed in [105] into the coding theorems based on the rate-distortion theory. More general
176
THEORY
OF INFORMATION
TRANSMISSION
treatments of these theorems are given i n the comprehensive book by Berger [8], Mathematical generalization of Shannon's idea i n this field has been provided by Kolmogorov [79, 80] and his colleagues i n Russia (Dobrushin [31], Pinsker [97]). Especially Dobrushin [31] proves the coding theorems and the fundamental theorem i n information transmission under a quite general formulation. I n the main theorems i n this chapter "stability" plays important roles. The information stability has been studied by Pinsker [97] i n detail. For discrete memoryless systems the coding theorems are studied i n Csiszar and Korner [26], where the concept of £-typical sequence plays fundamental roles. The e-typical sequence should be compared w i t h the e-typical set or the e-typical pair developed in the proof of coding theorems. I t seems to be an interesting task to discuss the behavior of the error probability of reproduction. We may expect that, if the rate-distort ion of the source is less than the channel capacity, the error probability goes to zero exponentially as the terminal time of the communication tends to infinity (cf. Proof of Theorem 4.5.1). Explicit formulas for the error exponents are given i n [14, 15, 90] i n the cases where the sources and the channels are memoryless, or Markovian. Similarly as in the hypothesis testing (Sec.
3.4, 3.5), the large deviation theorems would be
helpful to examine the asymptotics of the error probability.
Chapter 5 Discrete T i m e Gaussian Channels 5.1 M U T U A L
INFORMATION
WITH ADDITIVE
IN
CHANNELS
NOISE
This chapter is devoted to studying discrete (time) channels w i t h additive noise, especially Gaussian channels. The channel w i t h additive noise models w i t h many practical communication channels. Roughly speaking, the additive noise is due to a large number of random effects. The central limit theorem says that the noise caused by such large number of small random effects has a probability distribution of approximately Gaussian. This indicates that the study of Gaussian channels is especially important. Throughout the chapter we deal w i t h discrete time communication systems. We denote by N the time parameter space. Denote by X„, Y„ and Z„ a channel input, a channel output and a noise, respectively, at time n £ N . Then a channel with additive noise is presented by Y
n
(cf.
(4.1.5)).
We shall take
= X
n
{1,...,N},
+ Z,
n € N
n
Z , or Z as the time parameter set N . 4
Throughout the chapter, we assume that the noise process 2 = { Z „ } degenerate, namely 0,
Z
m
, Z
m
+
i ,
(5.1.1)
is non-
are linearly independent for every m and it >
I f the noise Z — { Z } is Gaussian, then the channel is called a Gaussian n
channel. We start w i t h the calculation of the mutual information transmitted over 177
DISCRETE
178
GAUSSIAN
CHANNELS
the channel (5.1.1). I n this section, formulas for the mutual information will be derived, w i t h o u t Gaussian assumptions. The Gaussian case w i l l be extensively studied i n the successive sections. We denote by £ = (£,; s e S} a message to be transmitted. I t is natural to assume that £ is independent of the noise Z. I f the channel is w i t h o u t feedback, then the channel input X = {X„} is independent o f Z. O n the other hand, if the channel is w i t h feedback, then the input X
at time n is a signal which is encoded
n
from the message and the output {Yf,k < n ) , and X is no longer independent of Z. We assume that the backward channel is noiseless. To be precise, we say t h a t the channel (5.1.1) is w i t h feedback i f the following conditions are satisfied. (a.1)
A message £ to be transmitted is independent of the noise Z.
(a.2)
A n input signal X„ is a function of £ and {?**; k < n — 1}: *„ = X (£,{n;fc
neN.
n
If the channel is w i t h o u t feedback, then (a.2) is replaced by (a.2'). (a.2')
A n input signal X
is a function of £:
n
Note that the case without feedback can be treated as a special case of the one w i t h feedback. We denote by i)v(£, V ) = -f (£, ( V i , - - • , I'w)) the mutual information between a message £ and the corresponding channel output K ' ' sa ( Y i , Y N ) w
we denote by
Hn(Y)
the entropy if
— H(Y%,...,YN)
Y^ * N
Moreover,
is a discrete random
variable, and denote by h w ( Y ) = h ( Y , . . . . Ypj) the continuous (or differential) T
entropy of Y^ > if i t is a continuous random vector. N
Theorem
5 . 1 . 1 . Let £ be a message t o be transmitted over the additive noise
channel (5,1.1) w i t h feedback. Then the mutual information //«(£, Y ) transmitted over the channel is given as follows. 1°)
I f the output Y = {Y„} and the noise Z = {Z„} are discrete random
variables, then I {£,Y) N
2°)
= H {Y)-H (Z). N
(5.1,2)
N
I f Y and Z are continuous random vectors w i t h finite continuous entropy,
then I (t:,Y) N
= h (Y) N
-
h (Z). N
(5.1.3)
S.I MUTUAL
INFORMATION
IN CHANNELS
WITH
ADDITIVE
NOISE
179
P r o o f , We shall prove only 2"), because 1°) can be shown analogously. I t follows from (1.8) and (h.7) that -
IN({>Y)
h (Y)-h(Y ...,Y \0 N
LT
N
N
C- - )
= hN(Y)-Y,KY»\M
1
S i n c e X , , . . . , ^ are functions of (£, Y j , Y „ _ , ) and Z =
Y -X ,((,Z ,...,Z - )
k
is a function of (£, Y i , Y „ _ , ) . (£,
Conversely,
4
k
k
l
n t
( £ , Y i , . . . , Y „ _ i ) is a function of
Therefore we have
Z\,Z„-i).
z u,Y ^t%^i)
i s C i ; i f , y , . . . , y _ ) = h(x + 1
n
u
n
n
1
= /i(Z„|Z,,...,Z„-i),
(5.1.5)
where we have used the fact that £ is independent of Z . Equation (5.1.3) follows from (5.1.4), (5.1.5) and (h.6).
•
R e m a r k . I t is emphasized that the formulas (5.1.2) and (5.1.3) hold whether the channel is w i t h or w i t h o u t feedback. If the channel is without feedback, then the mutual information I f f ( X , Y ) input J t ( ' = (XI,...,XN)
— / ( ( . X i , J V i v ) , ( Y i , Y j v ) ) between a channel and the output Y< > corresponding to X < - > is also
N
w
N
equal t o the right-hand side of (5.1.2) or (5,1.3). independent of the noise Z = { Z
N
Because, i n this case, X is
} and we can replace £ by X i n the proof.
Thus, the calculation of the mutual information f/v(£, Y ) or J ( X , Y ) is reduced w
to that of the entropy. I n Gaussian cases, the mutual information can be calculated more explicitly. We now t u r n to calculate the mutual information transmitted over an additive noise channel w i t h communication duration of infinity: Y If the noise Z — { Z
N
N
= X
N
+ Z„,
ngZ.
(5.1,6)
} is a stationary process, then the channel (5.1.6) is called
a stationary channel.
We are interested in the computation of the mutual
information rates 1 ( X , Y ) between the input JV = { X } and the output Y = { Y } , N
n
and I(f, Y ) between a message £ and the output Y . See (4.2.7) and (4.4.10) for the definition of 1 ( X , Y ) and 7 { ( , Y ) . As an immediate consequence of Theorem
DISCRETE
180
GAUSSIAN
CHANNELS
5.1.1, we know that the calculation of the mutual information rate is reduced to that of the entropy rate. For simplicity, we deal w i t h processes w i t h continuous distributions. Similar results can be proved for discrete distributions. T h e o r e m 5.1.2. For the channel (5.1.6) w i t h additive noise, whether the channel is w i t h or without feedback, T(£, Y ) is given by l(£,Y)
if (Yj, rates
Yfi) h{Y)
=
and
= k(Y)-h{Z),
(5.1.7)
{ Z \ , Z N )
are w i t h continuous distributions and the entropy
lim/i/_.oo N^hftfY)
M ^ ) exist. I n particular, i f the channel is and the noise Z = {Z }
without feedback and an input signal X = { X }
n
N
are both
stationary processes, then 7(X,Y)
where h(Y)
(5.1.8)
h(Y)-h(Z),
=
and k ( Z ) exist.
P r o o f . The formula (5.1.7) can be derived as an easy consequence of Theorem 5.1.1. Let the channel be without feedback and X and Z be stationary. Then as noticed i n the Remark of Theorem 5.1.1, we have I (X,Y)
=
N
Moreover Y — {Y„) h(Y)
h (Y)
~
N
(5.1.9)
h (Z). N
is also stationary. Consequently by Theorem 2.1.1 the limits
= l i m j v - o o / v - ' h w t F ) and k { Z ) = l i m * - * i V - V ( Z ) exist. Thus (5.1.8) l
follows from (5.1.9).
•
For the stationary channel (5.1.6), i t is interesting to analyze the behavior of I N { X , Y ) as a function of N.
For notational convenience we p u t Y£ = { Y * ; m
<
k < n } . Then note that / (X,Y) = /(X N
D
V
,Y
0
K
).
The conditional mutual information
L(N) is a useful quantity.
= /(x^.ri^yjU,
# >o
(s.i.io) is linear i n N.
Because, as w i l l be seen later, L(N)
introduce the following notations: JNMY)
=
I ( Y ^ Y ^
J o ( y ) = ^ o
0
+
N
\ Y
, n " ) .
0
N
) ,
N
>
0,
U
>
1,
We
5.1 MUTUAL
INFORMATION
IN CHANNELS
WITH
ADDITIVE
NOISE
181
monotonically increases t o J N ( Y ) as M —t co.
Note that JN,M(Y)
T h e o r e m 5.1.3. Let the stationary channel (5.1.6) be without feedback, the noise Z = {Z„} be purely non-deterministic and the input signal X — { X
N
} be also
stationary. Assume that Jn(Z) and J o { Y ) are finite. Then L{N)
= AN,
JV>0,
(5.1.11)
where A is a constant. The mutual information IN(X, Y ) is expressed i n the form IfliX,
Y)
= AN
+ J {Z)
=
AN
+ K (Y)
C
= J (Y)
-
N
J (Y)
+ C
N
- K (Z),
N
(5.1.12)
N
where C is a constant given by a
-
J (Z).
(5.1.13)
0
Now i t is easy t o answer a natural question "how does the mutual information I (X,Y) N
grow w i t h JV".
C o r o l l a r y . Under the same assumptions as i n Theorem 5.1.3, l i m (l (X,Y) N—'oo
-
N
AN)
= C,
(5.1.14)
and I(X,Y)=
l i m ±I (X,Y)
= A.
N
JV—'OO
(5.1.15)
JV
P r o o f o f T h e o r e m 5.1.3. For every N > 0 and M > 0 we have L{M
+ N)
® l
t{X?+ ,Y™4Y?H)
2 L{M) +
=
+
N
L{M)
r
( X
M
B
tifcg**,Y%\X^,Y3„)
+ L(N)
= L(M) + L(N).
+
I ( X
M
+
\ Z
M
t
N
^
N
\ Y ^
g
a
)
+ L(N)
J X ^ Z ° _
O
B
)
(5.1.16)
Here ( a ) is the chain rule (1.9), ( b ) follows from the chain rule and the stationarity of X and Y, and (c) follows from the independence of X and Z. Since M and JV
182
DISCRETE
GAUSSIAN
CHANNELS
are arbitrary, L(N) should be w r i t t e n i n the form (5.1.11) w i t h constant A = L(l). Using the chain rule (1.9), we have IM N(X Y)=I (X,Y) +
I
+
N
I(X ,Y \X ) ,+N
N
N
N
0
0
+ J(X^ ,Y^ \Y ). +N
+N
(5.1.17)
N
N
By (1.6), (5.1.10) and the independence of X and Z, we obtain /(^
+ w
,r
0
|X
w
TtvM+N '{A-o
o
w
) = 7(X#
+ N
,Z6"W)
= 0,
(5.1.18)
M+N, N~, 1*0 J
V
V
= I(Yi „,
+ T(Xt%"
Yjf* \Y^) N
r
-n{Xl ,Y° ),Y>f+ \X * Y ) ao
N
BO
= A M + J , (Y)
+N
0
l
N
Q
~ Woo,
N M
2L«). Z «
+ N
|X
0
M + N
, Z
0
N
), (5.1.19)
and
-liz^z^ix^^z^ =
+
iixl^z^ix^^z^) (5.1.20)
JN,M(Z).
I t follows from (5.1.17) - (5.1.20) that /
M
+
N(X,
Y) = I (X,
Y) + AM
N
+ J {Y)
-
MIN
J {Z), MLN
so that I + (X, M
N
Y) -
A(M
+ N)
= I (X,
Y)-AN
N
+ J , (Y)
Let N be fixed and M tend t o oo. Since J , ( Y ) M
JN(Z)
-
M N
N
~> J
N
(5.1.21)
J (Z). MLN
( Y ) and J\t, ( ) N
z
~*
as M —* oo, the l i m i t o n the right-hand side of (5.1.21) exists. Denote the
l i m i t by C = C(N).
Then, taking l i m i t on the right-hand side of (5.1.21), we have C{N)
= I (X,Y)
-AN
N
+ J (Y) N
-
J (Z). N
(5.1.22)
On the other hand, taking limit on the left-hand side, we know that C(JV) = J i m \I {X, M+N
=
\™JI (X,Y)-AM). M
Y) - A(M
+
N)}
(5.L23)
S.2 DISCRETE
GAUSSIAN
183
CHANNELS
Thus C = C(JV) does not depend on N, and the Erst equality of (5.1.12) is derived from (5.1.22). T h e second equality of (5.1.12) is obtained by substituting N — 0 in (5.1.19) and (5.1.20). Let JV = 0 i n (5.1.12). Then we have (5.1.13).
•
P r o o f o f C o r o l l a r y . Equation (5.1.14) has been shown i n the proof of Theorem 5.1.3 (see (5.1.23)). Equation (5.1.15) follows from (5.1.14).
5.2 D I S C R E T E
•
GAUSSIAN CHANNELS
We consider a discrete Gaussian channel presented by (5.1.1), where the noise Z = { Z ) is a Gaussian process, and X„ and Y n
n
represent a channel input and
the corresponding output, respectively, at time n. The aim of this section is to give various formulas to calculate the mutual information transmitted over the Gaussian channel. If Z„, n £ Z, are mutually independent Gaussian random variables, then the process Z =
{Z„}
is called a white Gaussian noise and the channel (5.1.1) is
called a (discrete) w h i t e G a u s s i a n c h a n n e l or a memoryless Gaussian channel. Although a white Gaussian noise has rather simple statistical structure, i t is the most fundamental and important Gaussian sequence.
Indeed, many problems
on the m u t u a l information transmitted over-Gaussian channels can be solved by reducing them to those over a white Gaussian channel. It has been shown that a Gaussian noise entropy output
hpi(Z)
N
t
of the Gaussian channel (5.1.1) has a continuous distribution
[Yi,Yfj)
and that h (Y)
has a finite continuous
lZ .....Z,\)
which is given i n Theorem 1.8.1. Moreover, we can see that the is finite as long as the variance of X , n
n = 1 , J V , are finite. I f
the output Y is also Gaussian, then the mutual information is calculated exactly. We denote by
T s{Z)
the covariance m a t r i x of
(Z\,...,Z ). N
T h e o r e m 5 . 2 . 1 . Let £ be a message to be transmitted over a Gaussian channel (5.1.1). 1°)
Assume that the output Y = {Y„)
is Gaussian. Then
rwvf,* )-!** 5
\?N(Y)\
|r»(z>r
(5.2.1)
184
DISCRETE
GAUSSIAN
CHANNELS
Assume t h a t the noise Z = { Z „ J is white Gaussian, the input X =
2")
is Gaussian, and each X
is independent of Yi,Y„_i.
n
+ ^I).
M ^ n - ^ E ^ ^
{X„}
Then
(5.2.2}
where p* and
n
Proof.
1")
Equation (5.2.1) is clear from Theorems 5.1.1 and 1.8.1.
Since Z = {Z }
2")
is a Gaussian white noise, by Theorem 1.8.1 and (h.6),
n
N
h (Z) N
N
= £
ft(Z ) n
= - £
n=l
B y ( a . l ) , (a.2) (see Sec.
log(2*e4).
(5.2.3)
n=l
5.1) and the assumption of the theorem, X
n
are mutually independent and they are independent of Y i , . . . , Y - i . n
X
+ Z
is a Gaussian random variable w i t h variance p + c
Y\,Yn-\
• Therefore, i n the same way as above, we obtain
n
n
n
h {Y)
= £
N
h(Yn)
n
+ °D]
Equation (5.2.2) follows from (5.1.3), (5.2.3) and (5.2.4).
and Z„
Hence Y„ =
and independent of
•
(5-2.4)
D
In the case where the channel is w i t h o u t feedback and the input is Gaussian, the mutual information is expressed in terms of the eigenvalues of covariance matrices. It is known that there exists an JV-dimensional regular m a t r i x A = A/« such that A'V (Z)A
= I,
N
(5.2.5)
where A* is the transposed m a t r i x of A and / is the identity matrix. T h e o r e m 5.2.2. Assume that the Gaussian channel (5.1.1) is w i t h o u t feedback and the input X = {X } n
I (LY) N
is Gaussian. Then
= I (X,Y) N
= ! £ l o g ( l + r„ ), 2
(5.2.6)
5.2 DISCRETE
T
where T , , 2
N
GAUSSIAN
are the eigenvalues o f A r ( X ) A , l
IBS
CHANNELS
here A is the matrix o f (5.2.5).
N
P r o o f . Since the channel is without feedback, i t follows from (1.6) and (1.4) that r « ( S , Y) = r {{t,X),
Y) = I (X,
N
(5.2.7)
Y).
N
Since A TN(X)A is a symmetric positive definite matrix, there is an orthogonal m a t r i x B such that
B'A'T (X)AB
=
N
Vo Let A B as G: as (sCijJj^j { X \ , X
N
)
by X
n
as
N
and define a Gaussian random vector X —
j CinJYi. Then X can be simply written as X = J V C ,
where JV = ( X i , J V J V ) .
Similarly we define Y = ( Y i , Y ) v ) and 2 = ( Z i , Z N )
by Y = YC and Z = Z C . Then we have another Gaussian channel + Z»,
Y„=X
N
Since the m a t r i x C is regular, X = X C ~ ' = IN(X,Y)
IN((,Y)
n = l,...,JV.
(5.2.8)
and Y — Y C ~ , and consequently L
(5.2.9)
= I (X,Y). N
I t is easily shown that the covariance matrices TjvfJV) and Tiv(Z)
are given by
0 r (x) N
= C T {X)C 1
r (z) N
=
N
= c'r (Z)c N
= B ' I B = i.
Therefore we can apply Theorem 5.2.1 to the channel (5.2.8) t o derive 1 I (X,Y) N
= k{Y)
-
h{Z)
N
= - £ l o g ( l + rl).
Equation (5.2.6) follows from (5.2.9) and (5.2.10).
(5.2.10)
•
We now t u r n t o calculate the mutual information rate transmitted over a stationary Gaussian channel.
DISCRETE
186
GAUSSIAN
CHANNELS
T h e o r e m 5.2.3. Let a channel (5.1.6) be stationary Gaussian and without feedback. Assume that the noise process Z = {Z„} is purely non-deterministic and w i t h SDF s(A), and the channel input X = { X
N
} is also a stationary Gaussian
process w i t h SDF /(A). Then the mutual information rate 7(X, Y ) is given by
P r o o f . Since
and Z are independent, Y = { Y „ } is also a stationary Gaussian
process w i t h SDF /(A) -f g(A). By Theorem 2.4.1, we have h(Y)
= JLy"
log[4» e(/(A) + ( A ) ) ] d A , a
ff
ft(Z) = i y " l o [ 4 r e ( A ) ] d A . r
g
5
1
Thus (5.2.11) follows from Theorem 5.1.2.
! J
•
R e m a r k . For the mutual information /jv(JV, Y ) i n the stationary Gaussian channel, by (5.1.15), we know that the formulas (5.1.12) and (5.1.14) hold w i t h
Let us compare an additive non-Gaussian noise channel: Y
N
= X
N
+ Z, n
n e N ,
(5.2.12)
n6 N,
(5.2.13)
w i t h a Gaussian channel: £ where X" = (X°)
= X° + Z l
and Z° = ( Z ° ) are Gaussian random processes. Let us de-
note by (//v and u
N
the probability distributions of (Z\,ZN)
respectively. To compare the mutual information IN(X,Y) use the relative entropy H {Z;Z") N
=
H{VN\V )N
and {Z\,
...,Z%),
w i t h IN(X°,Y ),
we
0
S.S DISCRETE
GAUSSIAN
[87
CHANNELS
T h e o r e m 5.2.4. Let the channels (5.2.12) and (5.2.13) be without feedback. Assume that X
A
= { X ° } and Z ° = {Z°} are Gaussian processes w i t h the same
covariance matrices as X = { X „ } and Z = { Z „ } , respectively. 1°)
H Z M Z ; Z ° ) < o o , then J w ( A \ Y ) - I (X ,
2°)
Y ° ) = H (Z;
A
N
N
(5.2.14)
- H (Y;Y°). N
I t holds that I (X,
Y) < I (X\Y°)
N
3°)
Z°)
Z°).
+ H (Z;
N
N
(5.2.15)
I f X is also Gaussian, then (5.2.15)
I (X\Y°)
N
1°) Since H~N(Z; Z ° ) < co, the distribution vn is absolutely continuous
Proof.
with respect t o the distribution V% (cf. Sec. 1.4), meaning that vjv is a continuous distribution. Denote by p ( x ) and p ° ( x ) the probability density functions of ( Z i , . . . , Z ) and ( Z , ° , Z % ) , respectively. Then, since {Z ) w
N
and { Z ° } have the
same covariance matrices and ( Z ° ) is Gaussian, /
P
(x)logp (x)dx =
/
D
J R *
J
-h (Z°).
p (x)logp°(x)dx = 0
R
N
"
Consequently, using (1.4.5), we have H (Z\Z°)
= -
N
p ( x ) l o g p ° ( x ) d x - M Z ) = M Z ° ) - k {Z).
I
N
(5.2.17)
J R "
In the same way, we see that (Yj,..., Yjv) is a continuous random vector and H (Y; N
Y ° ) = h (Y°) N
- h (Y).
(5.2.18)
N
On the other hand, by Theorem 5.1.1, I (X,Y) N
/
w
=
h (Y)-h (Z), N
( J ^ n = M n - M z
N
0
) -
"
1
'
Thus (5.2.14) follows from (5.2.17) - (5.2.19). 2")
I f H {Z\Z°) N
is infinite, then (5.2.15) is trivial. I f H (Z\Z°) N
is finite, then
(5.2.15) follows from (5.2.14), since H (Y\ Y ° ) is non-negative. N
3")
I n this case, X" may be identified w i t h X and IN(X, Y) =
I (X°,X°+Z). N
Since (X°, X° + Z) and {X°, Y") have same covariance m a t r i x , (5.2.16) is obtained as a special case o f Theorem 1.8.6.
•
188
DISCRETE
GAUSSIAN
CHANNELS
T h e o r e m 5.2.5. For the channels (5.2.12) and (5.2.13) w i t h N = Z , we assume +
the same conditions as i n Theorem 5.2.4. X = {X },
Z = {Z„},
n
Furthermore, assume that processes
X" = { X $ } , and Z° = { Z } are stationary. Then n
7(X,
Y)
=
I(X°,
< 1(X",
Y°)
+ H(Z;
Z°)
-
Y°)
+ H(Z;
Z°),
U(Y;
Y°)
(5.2.20)
where H( •; •) is the relative entropy rate. Moreover, if X = { X } is also Gaussian, n
then 1(X,Y°)<1(X,Y).
P r o o f . Since h{Y°)
and
Y
a
Z°
- h ( Y ) and H(Z;
are Gaussian,
H{Y\Y°)
=
]iia -. N-^H {Y\Y") N
oa
N
=
Z " ) = h ( Z " ) - h ( Z ) . Thus (5.2.20) is an easy conse-
quence of (5.2.14) and (5.2.15). (5.2.16).
(5.2.21)
The inequality (5.2.21) also follows easily from
•
5.3 M U T U A L
I N F O R M A T I O N IN GAUSSIAN SYSTEMS
There is another way to calculate the mutual information in Gaussian systems. Since, for Gaussian random variables, the orthogonality is equivalent to the independency, the simultaneous orthogonalization method i n a Hilbert space is useful to calculate the m u t u a l information. Let H be a separable Hilbert space w i t h inner product (-, •) and norm || - |iLet Ho be a subspace of H and denote by no the orthogonal projection on HoNote that n
0
is a linear operator on "H satisfying
where HQ is the orthocomplement of H . a
tion n
It is known that the orthogonal projec-
is the linear operator w i t h the following properties:
0
i.e., {U. x,y) a
= {x,U y} 0
for every at, y € H\
( I I o i , i ) > 0 for every i e H; Let Hi and H
2
and (hi) Til =
(i) f l is self-adjoint, 0
(ii) Wo is positive definite, i.e.,
U. a
be subspaces of H, and denote by 11; (i = 1,2) the orthogonal
projection on Hi. A n d we define linear operators B
t
B, = n , n , n , ,
B
2
and Bi by
=n n n . 2
1
!
5.3 MUTUAL
INFORMATION
IN GAUSSIAN
189
SYSTEMS
Then B i is self-adjoint:
= (n n rfia!,ff) = (*,n n>n if) =
(B\x,y)
1
a
l
1
and positive definite: (B ,x)
= ( n , n n , ^ i ) = $hUmU&)
lX
>
2
0.
Therefore the eigenvalues A„, n 6 Z , of B are all non-negative real numbers, here +
t
{ A „ } contains all eigenvalues appearing in accordance w i t h their multiplicity. We denote by u„ ( n £ Z ) the eigenvector of B\ corresponding to A ; B\u„ — A„u,,. +
n
Note that each vector on 7i\ is transformed by B\ into the same space Tij and each vector on H±
is transformed to 0. Hence we may take {u„; n 6 Z } i n +
such a way that { u } forms a complete orthonormal system (c.o.n.s.) of "H\. The n
operator B
if E ~
i K
has the same property. The operator B i is said t o be of trace class,
2
<
*»-
L e m m a 5 . 3 . 1 . The operators By and B (a)
0<
K
<
satisfy the following properties.
2
\,
(b)
The non-zero eigenvalues of B\ and B
(c)
If
(d)
I f 5,tt„ = 0 (i.e., A„ = 0), then u„ € H ^ .
m ^
n,
2
are the same.
then n u „ , is orthogonal to u„ and
n u„.
2
2
P r o o f . Property (a) follows from the fact O < B i < I , where / is the identity operator and A < B means that B — A is a positive definite operator. Since SjIliUn = IIjBiUn = we know that A„ is also an eigenvalue of B
2
X U U„, n
2
and the corresponding eigenvector is
n u . I n the same way i t is shown that every eigenvalue of B is also an eigenvalue 2
2
n
of B\. Thus B i and B? have the same eigenvalues. Noting u„ = I l i U n e Hi,
(c)
follows from (n u ,n u ) = {n w ,u } = (B,u ,u ) 2
m
2
n
2
m
n
m
n
= A (u ,u } —X S . m
m
If Biu„ = 0, then, by (5.3.1), ||n u„|| 2
n
m
(5.3.1)
mn
0. This means that n u 2
n
= 0 and
190
DISCRETE
GAUSSIAN
CHANNELS
L e m m a 5.3.2. There exist c.o.n.s.'s { u „ ; n £ 2 } of H i and {v„; n £ Z } of H i such that +
(v ,V ) M
= \A^X».
N
+
(5.3.2)
m,n£Z , +
where A„, n £ Z , are the eigenvalues of B i . +
P r o o f . There exists a c.o.n.s { u ; n £ Z } of 7 i i such that u„ is an eigenvector +
n
of B i corresponding t o A„. We define u„ £ H i by n if A / 0 .
3 M
,
T h e n , by (5.3.1),
n
Ps«»ll
= ( « . n u ) - A„,
a
n
2
n
and («m. »»> = - 7 ^ ( u , n u „ ) = \/A„£ . vA m
2
mn
n
We know from Lemma 5.3.1 that v„ is an eigenvector of B i corresponding to A„ and {v„; X„ / 0} forms an o.n.s. of H . 2
We can find a c.o.n.s.
{u„; n £ Z } +
w i t h supplement of eigenvectors of B j corresponding to the eigenvalue zero. If BiU
m
= 0 or
Biv„
0, then by ( d ) of Lemma 5.3.1 we know that u
=
ti„ 6 "Hi", so that ( u , u „ ) = 0. Thus the proof is complete. m
m
£
or
H
2
D
We now apply these arguments to calculate the mutual information i n a Gaussian system. Let X = {X„\ n £ N ) and Y == {Y ; n 6 N ) be Gaussian processes such n
that
{X ,Y ; n
Z or Z X„(u>)
n
n
£ N ) forms a Gaussian system, where the parameter set N is
I n this case we say that (JV, Y ) is pairwise Gaussian.
+
is a random variable defined on a probability space ( f i , B,
P)
Since JV = n
w i t h J3[JV ] = n
/ A^„(w) d P ( w ) < oo, JV„ = JV„(w) and Y — Y„(w) can be treated as elements n
!
n
of a Hilbert space L ( f i , P). Note that the inner product and the norm of L (Sl, 2
s
P)
are given by (V,V)=
f Ja
U( )V{w)dP{u)
\\Uf=
j Ja
\V{u>)\ dP(»)
=
u
=
2
E\Vn
E[\vn
Let H{X), H { Y ) and H be subspaces of L ( f i , P ) spanned by {A"„; n £ N } , {Y ; n £ N } and {X„,Y ; n £ N } , respectively. Define operators B and By on Hby J
n
n
B
x
x
= n
x
u
Y
n
x
,
B
Y
= n
Y
n
x
n
Y
,
(5.3.3)
5.3 MUTUAL
where l l
(resp. U )
x
INFORMATION
IN GAUSSIAN
SYSTEMS
is the orthogonal projection on H(X)
Y
(resp. W ( K ) ) . From
Lemma 5.3.2 i t is seen that there exist c.o.n.s.'s U = {U \ n £ Z n
V = [V
} of H(X)
+
and
n € Z + ) of H ( K ) such that
ni
m,n g Z .
(5.3.4)
+
Thus the pair ( [ / , V ) is a simultaneous orthogonalization of
This orthog-
(X,Y).
onalization enables us t o calculate the mutual information I{X, Y). 5.3.3. Let X = {A" ; n £ N } and Y m {Y„\ n £ N } be Gaussian
Theorem
n
processes such that
(X,
Y)
is pairwise Gaussian. The mutual information
is finite i f and only i f the operator B eigenvalue. The mutual information
x
l(X,Y)
is given by
oo I(X,Y)
>
/((/, V ) = £ /([/.. V . ) = n= 1
=
where { A } are the eigenvalues of B n
P r o o f . Since {U }
x
£ > g ( l - A ), n= 1
(5.3.5)
B
.
is a c.o.n.s. of H(X), each X
n
I(X,Y)
is of trace class and does not have 1 as an
n
is a linear combination of \ U ), n
and conversely each U„ is a linear combination of (JV }. Thus i t holds that n
I(X,Y)
= I(U,V).
(5.3.6)
Since {U„, V„; n £ Z } is a Gaussian system and ((/„, V ), n £ Z , are mutually +
n
+
orthogonal, ( [ / „ , V „ ) , n 6 Z , are mutually independent. Hence +
oo I(V,V)
= Y,HVn,V )
(5.3.7)
n
n=] (see (1.10)).
Now, from Lemma 5.3.2, we know that ([/„, V„) is a 2 dimensional
Gaussian vector, U
and V„ have variance one, and the correlation coefficient is
(>(V ,V )
=
n
a
N
= (U ,V ) n
n
Hence, by (1.8.15), we have I { V M
= -\\c,%(\-K).
(5.3.8)
Equation (5.3.5) follows from (5.3.6) - (5.3.8).
Since 0 < A„ < 1, the mutual
information l(X, Y ) = — y £ l o g ( l - A „ ) is finite if and only i f A„ / 1 and 52n=i
< • 00
Thus the proof is complete.
•
Note that Theorem 5.2.2 can be regarded as a special case of Theorem 5.3.3. Applying the simultaneous orthogonalization method, we shall prove the information stability of Gaussian processes. For this purpose we prepare a lemma.
192
DISCRETE
GAUSSIAN
CHANNELS
L e m m a 5.3.4. Let {£, ij) be a 2-dimensional Gaussian random vector w i t h covariance matrix
r where 0 < 7 < 1. Denote by p { x ) , q(y), r ( x , y ) the probability density functions °f £1 7i (£17)1 respectively, and define a function ip(x, y) by ^,y)
= T ^ ( ^ y ) = ^ ^ . d/i x / i p(i)9(y) f
M f f t * .
n
Then a random variable C — log i^(£, 17) has mean £lCW(£^> = - |
l
o
g
(
~
1
7
2
'
)
( 5
-
3 9 )
and variance Var[C3-=£[|C-/(£,n)| ] = 7 -
(5-3.10)
2
a
P r o o f . Equation (5.3.9) is clear from (1.6.6) and (1.8.15). We can easily show that log f f a ! , ) = ! { ^ ( s
2
+ „*) + ^ ^ y }
- 5 log(l -
7
J
)•
Therefore v
«lfl = i ( i r ^ )
3
/ /
(-^
l 2
+y ) + 2
2
i
v}
3
^,y)^y-
(5.3.11)
One can show the following equations /jR'
l 4 r ( a
\ y)
|» r(x, y) dxdy = 3,
Jj
x yr{x,y)dxdy
^
a ; y r ( i , y ) d i d y = l + 27 .
3
a
= JJ
2
xy r(x,y)dxdy 3
n
= 3 , 7
•
n £ N ) be a pairwise Gaussian system. We put j (JV)
and define the e-typical set T
(N)
t
(5.3.13)
(5.3.14)
2
Combining (5.3.11) - (5.3.14), we obtain (5.3.10). Let ( X , Y ) = {X„,Y \
(5.3.12)
4
by (4.5.7) w i t h R i n place of H.
5.3 MUTUAL
INFORMATION
IN GAUSSIAN
193
SYSTEMS
T h e o r e m 5.3.5. Suppose that (X, Y ) is the same pairwise Gaussian system as in Theorem 5.3.3. 1°)
For any JV and £ > 0, i t holds that
& W
2°)
N
)
) > 1 -
2
(5-3.15)
- ^ ^ .
The pair (X, Y ) is information stable, i f
P r o o f . 1°)
Let us apply Theorem 5.3.3 to the Gaussian pair ( X
( W )
,Y
< W )
)
=
((Jf i , J V j v ) , (Yi,.... Y N ) ) to obtain simultaneously orthogonalized Gaussian vectors
r/< ' N
= ( U
U
. . . , U
N
and
)
V '
N
= ( V i , . . . , V ) . Equation (5.3.5) turns to
I
N
« MX,
where
Y)
= I (U,
N
= Y /(£/„, V„) = - = Y . n=l n=l
V)
N
,
is the correlation coefficient of U
n
o
g
(
" »>'
1
A
(
and V„. We write U
n
and V
n
5
'
3
1
7
)
in the
form N
N
Then we can show that, under the transformation R
3 ( r , y ) — » (",")
2 F L I
€ R
2
W
given by N
N
u-n = Y , " a
k X k
'
« = 1,..., JV,
=
t=i
*=i
'(ir,v)
the Radon-Nikodym derivative
is invariant:
d/ifj
where u = ( t i i , . . . . u ) , v = (t>i,...,Vjv) £ R T,<"> = { ( « , w ) 6 R
2
N
; ^ jW
<
w
V
. Define a set f}
Ni
W
N
x \L
,
( « , t») - W
by
V)| <
•
Then
^
W
) = W )
(5-3-18)
134
Since
DISCRETE (U ,V ), n
n
n — 1 , N ,
GAUSSIAN
CHANNELS
are mutually independent, ^ ^ ' ( u , v) can be written
as a product
N
where
and random variables (
= log^(t7 ,V ),
n
n
n-l,...,JV,
n
are mutually independent. Using Lemma 5.3.4, we know that E[C»\
= I(U ,V ) n
and
n
Var[C„] =
V
Therefore, the random variable logi/>' '(f/, V ) is a sum of independent random w
variables: N
N
log^'ci/.v) = £
l o g o f f . . . K.) =
£c»,
with mean £[logi/> (I7, V ) ] = Jjv(r/, V ) and variance (JV)
iv
w
VarllogV'^ff/^)] = ^ V a r K n ] = £ n=l
V
n=l
Using the Chebychev's inequality, we see that
M$
N )
\ = * (|W (tf, v) - /«(tf,v)| < jv) (JV)
e
2: i - ^ r " -
(5-3.19)
Since l o g ( l - x) < - x for all 0 < x < 1, JV
$2
w A n
^ - E
n=l
M^
1
- «) = A
2 J
N ( ^ - V).
n=l
The inequality (5.3.15) follows from (5.3.18) - (5.3.20). 2°)
Let the pair (X, Y ) satisfy (5.3.16). Then, by (5.3.15), w
i ^ ^
{
r ^ > ) = i,
meaning that (X, Y ) is information stable.
•
(5.3.20)
5.4 CAPACITY
OP WHITE
GAUSSIAN
IBS
CHANNELS
C o r o l l a r y . Under the same assumptions as i n Theorem 5.2.3, the pair ( X , Y ) of the channel i n p u t X = \X„] and the output Y — {Y } of the stationary Gaussian n
channel (5.1.6) w i t h o u t feedback is information stable.
P r o o f . Since the l i m i t l i m j v _
a o
exists (see Theorem 5.2.3), the con-
N~'J (X,Y) N
dition (5.3.16) is satisfied. Therefore, the pair ( X , Y ) is information stable.
5.4 C A P A C I T Y O F D I S C R E T E
•
WHITE GAUSSIAN CHANNELS
I n this section the capacity of discrete time white Gaussian channels under average power constraint is studied. I t will be shown that the capacity is not increased by feedback. Let us consider a white Gaussian channel given by Y =X„
+ Z,
n
where
Z\,...,ZN
N
n=l,...,/V,
(5.4.1)
are mutually independent Gaussian random variables. As was
mentioned i n Sec. 4.4, i t is reasonable to assume that an energy or power constraint is imposed on the channel input. We deal w i t h the average power constraint given by E[Xl)
or
V«,
(5.4.2)
,
(5.4.3)
. i £ £ ( A 3 < „
J
n= 1
where p > 0 is a constant- A l i t t l e more general type o f constraint E\Xl\
< p i
Vn,
(5.4.4)
is also considered, where p„ > 0 is a constant. Denote by X the family of all inputs X = {X \ n = 1,.... JV} which satisfy the n
average power constraint and are independent of the noise Z. Then, by definition, the capacity C = C{X) of the channel without feedback is given by C= M N{X Y)-X£X), S
!
!
(5.4.5)
196
DISCRETE
GAUSSIAN
CHANNELS
(see Def. 4.4.1). When the channel is w i t h feedback, the capacity Cf = Cf(A) of the channel is given by C, = mip{I ((,Y);
U,X)
N
S A},
(5.4.6)
where A is the set of all pairs (£,X) which satisfy ( a . I ) , (a.2) (see Sec. 5.1) and the average power constraint (see Def. 4.4.2). Clearly C < Cf by definition. I t is shown that the capacity is not increased by feedback and is attained by sending a Gaussian signal. Theorem
5 . 4 . 1 . 1°)
Under the constraint (5.4.4), the capacity of the white
Gaussian channel (5.4.1) is not changed by feedback and is given by
c
=
c
/=^Z>s(
1
+
;|)'
I--) 5
4
7
where a is the variance of Z „ . n
2°)
Let X =
£ X be a Gaussian random vector such that
are mutually independent and
is w i t h distribution
N(0,p ). n
X\,.., Xti t
Then the
capacity C = Cf is attained by transmitting the input signal X without using feedback: C = C, = I {X,Y).
(5.4.8)
N
P r o o f . Let (£,X)
£ A be an arbitrary admissible pair and let Y = {Y } n
corresponding o u t p u t . Since X
n
be the
is independent of Z „ , E[Y \ = E\Xl] + E[Z„] < 2
p\ + o%. I t follows from (h.6) and Theorems 1.8.1 and 1.8.4 that
h {Y) N
< £
h(Y ) n
< I £
log [2*z( l
+ 4)]
P
.
(5.4.9)
Therefore, using (5.1.3) and (5.4.9), we have
l N ( ( , Y ) < \ j 2 l o
n=l
g
( l
+
S
£ ) . n
/
This means that ^ < Q < | £ l 0 5 ( l + § ) .
(5.4.10)
5.J CAPACITY
OF WHITE
GAUSSIAN
CHANNELS
137
Let X = { X } £ X be as i n 2°). Then the constraint (5.4.4) is satisfied and, using Theorem 5.2.1, we have n
c > i (x,Y) N
= ±Y, s lo
f
n= 1
^
+ §)
1
n
•
(5-4-nj
'
The desired equation (5.4.7) and (5.4.8) follow from (5.4.10) and (5.4.11).
•
Since the constraint (5.4.2) is a special case of (5.4.4), the capacity of the channel subject to (5.4.2) is given by
C
=c,-i£>4
g).
+
Under the constraint (5.4.3), the capacity of the white Gaussian channel is given by the following theorem. Theorem
5.4.2. Assume that
a\
>
-- • >
cr , 2
N
where
a
2
=
E[Z \. n
Then the
capacity of the channel subject to (5.4.3) is given by
n=M
a
where the positive number A and the positive integer M are determined by
M
=m\n{k- 4<
A }.
(5.4.14)
2
}
In particular, i f a\ - • - • = °% = a , then A 2
2
= p + a , M = 1 and 2
(? = % = | l p g f l + ^ ) .
P r o o f . For convenience, we put
2
(5.4-15)
DISCRETE
198
GAUSSIAN
CHANNELS
Comparing the constraint (5.4.3) w i t h (5.4.2), we see t h a t the capacity C = Cf of the channel subject to (5.4.3) is equal to the maximum value of C ( p i , P N ) for all {PU
- , P N ) satisfying p2
Clearly the maximum is attained under the condition i
N
(5.4.16) n=l
pl\
1 2
I
i
i
i
n
M
W
F i g u r e 5 . 1 . Assignment of power p„
Then note that the sum
53i]=i(/n + ,
the choice of ( p , , . . . , p ) , C ( W
P l
p
c „ ) = JVp -(2
N
2^=i
a
\ does not depend on
) = * E ^ , M p I +4)
" 5S ^ M
l o
S"n-
and log i is a concave function. From this observation, we know t h a t i f we could take p ! , P N i n such a way as pi+4
then
C ( p i , P N )
P % + V N ( = A -
2
(5.4.17)
1
would be maximized. I f the constant
specified by (5.4.13)
M
and (5.4.14) is equal to one, then (pi,...,pjv) can be chosen as (5.4.17) and the maximum value is equal to the right-hand side of (5.4.12). Now assume that M 1. Then ( p i , p w )
suppose that pi > 0 for some k < M. exists I > M such that p] + a] < A (< 2
p]
+p\
and p„ =
>
cannot be chosen as i n (5.4.17). Under the condition (5.4.16),
p„,
n /
k, I,
T h e n by the definition of A and M c l ) . I n this case, defining p
we see t h a t
C(P,,...,PN)
>
C(p
u
k
there
= 0, p
...,p )N
1
=
This
5.4 CAPACITY
OF WHITE
GAUSSIAN
CHANNELS
199
means that to maximize C(/>i, . . . , p ) we need to take pt = 0, jfc < jVf. A n d , as we w
have seen above, C ( 0 , 0 , pM+t,....
PN) is maximized by taking p„, M
as PM = • • • = PN = A (this is possible). Thus (5.4.12) is obtained. I n the case where tr% =
•==
•
The assignment of values p„, n = 1 , J V , is illustrated graphically i n Fig. 5.1. Let us proceed t o calculate the per unit time capacity of a white Gaussian channel represented by (5.1.6), where Z „ , n e Z, are mutually independent Gaussian random variables. Assume that the channel has the constraint
n=l
imposed on the inputs. T h e per unit time capacity C = C ( X ) without feedback is denned by (4.4.11), where /(JV, Y ) is the mutual information rate and the class X of all admissible inputs consists of all processes JV = (JV„) which satisfy (5.4.18) and are independent of the noise Z. The feedback capacity C j — C f ( A ) per unit time is defined by (4.4.12), where the class A of all admissible pairs consists of all pairs (fJ.JV) satisfying ( a . l ) , (a.2) and (5.4.18). T h e o r e m 5.4.3. Assume that the noise of the white Gaussian channel (5.1.6) has the variance E\Z \ 2
n
1°)
= cr , n € Z . 2
+
Under the constraint (5.4.18), the capacity of the channel per unit time is
given by C = C 2°)
ilog(l
/ =
+
£ ) .
(5.4.19)
The capacity is attained by i n p u t t i n g a white Gaussian signal X = {X„} €
X w i t h mean zero and variance p : 2
C = Cf=J(X,Y).
(5.4.20)
P r o o f . Let (£,X) £ A be an arbitrary admissible pair. Then, for any e > 0, YE\X ]
n
2
for large JV. B y (5.4.15) i n Theorem 5.4.2,
+ £) 2
DISCRETE
200
GAUSSIAN
CHANNELS
Since e > 0 is arbitrary, we have
C
+ ^ ) .
f
(5.4.21)
On the other hand, by taking an input signal X = {X } n
€ X as i n 2°), clearly we
have = ^log ( l + ^)
C > 7(X,Y)
•
Now (5.4.19) and (5.4.20) follow from (5.4.21) and (5.4.22).
(5.4.22)
•
In this section we have learned that the capacity of a white Gaussian channel has the following two properties. (a)
The capacity is not increased by feedback.
(b)
By inputting a Gaussian signal, we can transmit the mutual information
equal to the capacity. Property (a) is a result of the whiteness of the Gaussian noise. I n general the capacity of a non-white Gaussian channel is increased by feedback.
Such exam-
ples will be given later. O n the other hand property (b) holds not only for the white Gaussian channel but also for non-white Gaussian channels. O p t i m a l coding schemes, i n the sense that the maximum mutual information ( = capacity) is transmitted, will be studied later.
5.5 C A P A C I T Y O F D I S C R E T E CHANNELS
WITHOUT
GAUSSIAN
FEEDBACK
The aim of this section is to calculate the capacity of a Gaussian channel without feedback. The noise is Gaussian b u t not necessarily white. At first we consider a Gaussian channel represented by (5.4.1) w i t h a Gaussian noise Z — [Z\,...,ZN)-
Suppose that the average power constraint (5.4.3) is
imposed. Then the capacity C = C { X ) of the channel subject to (5.4.3) is given in the following theorem. T h e o r e m 5 . 5 . 1 . Let c
2
trix r (Z) N
> •-• > a
2
N
be the eigenvalues of the covariance ma-
= (7o)ij-i,...,N of the Gaussian noise Z = ( Z , , . . . , Z ) . N
Under the
S.S CAPACITY
OF GAUSSIAN
CHANNELS
WITHOUT
201
FEEDBACK
constraint (5.4.3), the capacity of the Gaussian channel without feedback is given by 1 A
A*
where a positive integer M and a positive number A are specified by (5.4.13) and (5.4.14).
P r o o f . There exists an orthogonal matrix B = (6y)<^=i 0
N such that \
In the same manner as i n the proof of Theorem 5.2.2, define X = Y
= (%,..., Y
N
) and Z = ( Z i
(X ,XN), S
ZN ) by
X = XB,
Y = YB,
(5.5.2)
Z = ZB.
Then Y„ = X
N
(5.5.3)
n = l,...,N.
+ Z, N
Gaussian random variable w i t h £[ZjZjj — Sij, (5.5.3)
Since Z is an N-dimensional
presents a white Gaussian channel. Noting the orthogonality of B and (5.4.3), we see that the i n p u t X satisfies the constraint
£ n=l
E[xi\ = e
E
h
^
m
m
=
(5.5.4)
< i=l
n=] i,j=l
Since B has the inverse m a t r i x , the transformation (5.5.2) preserves the mutual information: IN{X,Y)
=
I (X,Y). N
Therefore the capacity C is equal t o the capacity of the white Gaussian channel (5.5.3) subject t o (5.5.4) and, by Theorem 5.4.2, is given by (5.5.1).
•
Let us study the per unit time capacity of a Gaussian channel represented by (5.1.6), where the noise Z — {Z„; n g Z } is a Gaussian process. I t seems t o be natural t o require stationarity on the channel. So we assume that the noise Z =
202
{Z„)
DISCRETE
GAUSSIAN
CHANNELS
is a stationary Gaussian process w i t h SDF g(A) and assume t h a t the input
signals are also restricted to stationary (not necessarily Gaussian) processes. I n order to find a formula to compute the capacity under an average power constraint, at first we calculate the capacity under the condition that the input signals are restricted to stationary processes w i t h a given S D F /(A). Denote by X the class of possible input signals: X = { X = {X ; n
n e Z ) ; X is a stationary process w i t h S D F /(A)}.
(5.5.5)
Then the capacity C = C ( X ) is defined by (4.4.11). T h e o r e m 5.5.2. Under the constraint given by (5.5.5), the per unit time capacity of the stationary Gaussian channel (5.1.6) without feedback is
The capacity C is attained by choosing a stationary Gaussian process w i t h SDF /(A) as the channel input.
P r o o f . Note that all processes i n X have the same covariance function. Therefore, by Theorem 5.2.5, the mutual information
is maximized by sending a
I(X,Y)
Gaussian process X — {X„} g X. A n d , by Theorem 5.2.3, the m a x i m u m value of I{X,
Y ) is given by (5.5.6). Thus the proof is complete.
•
T h e o r e m 5.5.3. Under the constraint (5.4.2), the per unit time capacity C of the stationary Gaussian channel (5.1.6) without feedback is given by
C
=
i
L £ b g [
m
a x ( - A . , l
(5.5.7)
where A > 0 is a constant determined by (A- (A))dA = p .
(5.5.8)
2
9
JtXei-a.nh {X<=\-*,*];9(\)
The capacity C is achieved by i n p u t t i n g a stationary Gaussian process A ' = {X®} 0
w i t h SDF m
if-m I 0,
.
(5J 9)
if
g(X) > A .
v
'
S.S CAPACITY
OF GAUSSIAN
CHANNELS
WITHOUT
FEEDBACK
203
P r o o f . The capacity (j? is defined by (4.4.11) where the set X of admissible inputs consists of all weakly stationary processes satisfying (5.4.2).
We have already
known that the maximum value of I(X, Y) is achieved by choosing a Gaussian process JV € A* as an input. Moreover, by (5.1.8) and Theorem 2.4.1, we may assume that the process has a SDF. Let X = {-V„} e ~X be a stationary Gaussian process w i t h S D F /(A). Then E\Xl)
= j
' f(\)d\
(5.5.10)
Thus the capacity C is equal to the maximum value of the right-hand side of (5.5.6) w i t h respect to SDF's /(A) satisfying (5.5.10). Similar argument as in the proof of Theorem 5.4.2 shows that the right-hand side of (5.5.6) is maximized by taking /°(A) of (5,5.9) as /(A), and the maximum value is equal to the right-hand side of (5.5.7). Thus the proof is complete.
O
F i g u r e 5.2. Water-filling method
The assignment of f(
A) in this theorem has the graphical interpretation of Fig.
5.2, which is known as "water
filling"
method. We can think of the power p
2
as being a q u a n t i t y of water to be placed i n a reservoir w i t h a shaped bottom determined by ff(A). The level to which the water rises is A and /°(A) gives the depth of water at A. Here we give two examples of non-white Gaussian channels to calculate the capacity explicitly.
DISCRETE
204
GAUSSIAN
CHANNELS
E x a m p l e 5 . 5 . 1 . Under the constraint (5.4,18), denote by C the per unit time capacity of a stationary Gaussian channel (5.1.6) without feedback, where the noise Z = { Z „ } is a Gaussian AR(1) process represented as
Z„
here W — { W } n
P> aZn-i
+ W.,
1 6
(5.5.11)
Z.
is a white Gaussian noise with variance 1 and 0 < a < 1 is a
constant. The SDF g(A) of Z is given by
g(\) =
2jr|e - a \ ,A
2
2 u { l - 2a cos A + or )
(5.5.12)
2
(see Theorem 2.3.1). I f 2a
P> 2
(5.5.13)
(1-<*)»(!+«)'
then we can show that 1 C = - log •x
/>' +
(5.5.14)
1 —of
Note that S(0) =
max
g(X) = —
—*<\
Z1T(1 —
—a)'
and £
9
( A ) d A
=
T
J - , .
Thus, under the condition (5.5.13), the constant A of (5.5.8) is given by
Finally, by Theorem 5.5.3, we can show that the capacity C is given by (5.5.14):
5.S CAPACITY
OF GAUSSIAN
CHANNELS
WITHOUT
205
FEEDBACK
E x a m p l e 5.5.2. Let C be the capacity under the same constraint as in Example 5.5.1, but the noise is replaced by a Gaussian A R M A ( 1 , 1 ) process Z = {Z ) n
with
representation + W„ - ( l + a)e- W -i,
n
where W
=
n e Z,
a
Z„ = e-"Z .j
n
(5.5.15)
is a white Gaussian noise w i t h variance 1 and a > 0 is a
(W„}
constant. The process Z has the SDF | l - { l + a)e-V | A
9W
=
2JT|1 - e-^e *! 1
_1_
2
2
a - o r
2
"
a e~
-
2
2a
(5.5.16)
l_ -= .'A|2
2^
e
e
(see Theorem 2.3.1), Since a — ae max
a(A) = g(±7r) -
(l + o + e°)
3
If P > Pot *}' where 1
(5.5.17) then C
=
\log
/l + g +
e-V
+
p-po(a
(5.5.18)
By (5.5.17), i t is seen that the constant A of (5.5.8) is given by A =
g(ir)
+
2rt
and consequently —
Since J ^ l o g j l Cn
°&{2ng{X)}
]
re \ dX ix
dX =
2
1
f
=
2ng{7!)
+ p -
p {a' 0
l
(5.5.19)
dX.
0 for all - 1 < r < 1, noting (5.5.16), we have
0. Thus (5.5.18) follows from (5.5.19).
We shall discuss on the capacity
C
of an additive
non-Gaussian
noise channel
(5.2.12). I n general, i t is difficult to calculate the capacity C exactly.
Compare
the non-Gaussian channel (5.2.12) w i t h the Gaussian channel (5.2.13). Then we can give an upper and a lower bounds for the capacity C i n terms of the capacity C° of the corresponding Gaussian channel.
206
DISCRETE
GAUSSIAN
CHANNELS
T h e o r e m 5.5.4. Let C and C° be non-feedback capacities of the additive noise channels (5.2.12) and (5.2.13), w i t h N = {1,...,JV}, subject to (5.4.4).
Suppose
that the noise Z ° = { Z „ } is Gaussian and has the same covariance m a t r i x as the noise Z = {Z„}.
Then the capacity C is bounded by C
< C < C" + H (Z;
a
(5.5.20)
Z").
N
P r o o f . Let X be the class of all admissible input signals under the constraint (5.4.4). Let X £ X be an arbitrary input signal and X° £ X be a Gaussian signal with the same covariance m a t r i x as X, and denote by Y and Y" the outputs corresponding to JV and JV°, respectively. Then, by 2°) of Theorem 5.2.4, IN(X,
Y ) < / ( * , K ° ) + H (Z; W
0
N
Z°)
+ H (Z; N
Z°),
yielding that C
+ H (Z;
Z°).
N
On the other hand, by Theorem 5.4,1, there exists a Gaussian signal JV° £ X such that
C° = Jw(JC°,r '). H
Using 3") of Theorem 5.2.4, we know that C
= I (X°,Y°) N
Thus the proof is complete.
< I (X,X" N
+ Z)
<
C.
•
R e m a r k . Not only under the constraint (5.4.4), the inequalities of (5.5.20) hold so long as the constraint is given i n terms of covariance matrices of the channel inputs.
The first inequality of (5.5.20) indicates that the Gaussian noise minimizes the capacity, i n other words, is most powerful as a noise.
The same result can be
shown for the per unit capacity of stationary channels. T h e o r e m 5.5.5. Let the noise Z = { Z } of the channel (5.2.12), w i t h N = Z, be n
a stationary process w i t h SDF g{\) and the noise Z ° = { Z } of the corresponding n
Gaussian channel (5.2.13) be a stationary Gaussian process w i t h the same SDF
S.6 OPTIMAL
CODINGS
IN WHITE
GAUSSIAN
207
CHANNELS
g(A). Assume that the stationary channels (5.2.12) and (5.2.13) are without feedback. Then the per unit capacity C of the channel (5.2.12) under the constraint (5.4.2) (or (5.4.19)} is bounded by C °
+ H(Z;Z< ) ,
(5.5.21)
7
where C is the per unit capacity of the Gaussian channel (5.2.13) under the same constraint.
5.6 O P T I M A L
CODINGS
WHITE GAUSSIAN
IN
DISCRETE
CHANNELS
In this section, we shall investigate coding schemes, encoding and decoding, to transmit Gaussian messages over a white Gaussian channel (5.4.1) w i t h feedback. Let the noise Z = {Z„} of the channel (5.4.1) be a white Gaussian noise with variance c , and the constraint (5.4.2) be imposed on the input X — {X ). 2
Then,
n
by Theorem 5.4.1, the capacity Cf of the channel w i t h feedback is given by
C
= |log(l + £ ) .
/
(5.6.1)
Suppose that the message ( to be transmitted is a Gaussian random variable w i t h distribution N(Q, T ) . We assume that the reproduction error is measured by the mean square error 2
d(£,n) = { £ [ | £ - | ] } / , n
1
1
2
( . .2) 5
6
where TJ is a received message. The aim of this section is to construct an encoding scheme and a decoding scheme which are optimal i n the sense that the reproduction error (5.6.2) is minimized. Let us denote by
R(D;£)
and D(R;
£) the rate-distortion function and the
distort ion-rate function of £ associated w i t h the distortion d(£, 17) of (5.6.2). A n d denote by D*(() 4.6.1,
D'(£)
the minimum reproduction error (see (4.6.1)). Then, by Theorem
is not greater than
D(Cf,£).
As was mentioned i n Sec. 4.6, if one
can find an encoding and a decoding by which the equality D-(0
=
D(C,-0
(5.6.3)
208
DISCRETE
GAUSSIAN
CHANNELS
is satisfied, then the coding scheme is optimal (cf. (4.6.5)). We shall find such a coding scheme to transmit the Gaussian message £ over the white Gaussian channel. It is known that D(R;()
= r e x p ( - f l ) (see (1.8.30)), i n particular
Z>(C,;£) = r e p ( - C 7 ) = T ^ l + ^J
.
X
(5.6.4)
Once an encoding scheme is specified, then the input X = {^f }
and consequently
n
I t is known that the received message n =
the output Y = { Y } are specified. N
I(YI,...,YN)
which minimizes the mean square error d(£,n) is the conditional
expectation: i, = i=E[t\Yi,...,Y ]
(5.6.5)
N
(see Appendix, Sec. A.2). Thus the optimal decoding is always given by (5.6.5). The optimal coding scheme is provided i n the following theorem. At the same time it is shown that the coding scheme is also optimal i n the sense that the maximum mutual information is transmitted. T h e o r e m 5 . 6 . 1 . Suppose that a Gaussian message £ w i t h distribution JV(0, r ) !
is to be transmitted over the white Gaussian channel (5.4.1) w i t h feedback, under the constraint (5.4.3). Define an encoding by Xn = a (t-£ ), n
n
n=l,...,jV,
(5.6.6)
n > 2,
£,=0,
(5.6.7)
1,Jv":
(5.6.8)
where £„ = E[(\ Yi,.... Y - , ] , n
and a„'s are constants determined by E[Xi]
= a\E\\t -
= p\
n =
And define a decoding by (5.6.5). Then the resulting reproduction error satisfies (5.6.3): d{U)
= E[\(
2
- £\ ) = r 2
(l
2
+
= D(C ;0 . f
(5.6.9)
2
In other words, the reproduction error is minimized. Moreover the transmitted mutual information, under this coding scheme, is equal to the capacity: lit
0 = IN(1,Y)
= C
F
= ^ log ( l +
•
(5.6.10)
5.6 OPTIMAL
CODINGS
IN WHITE
GAUSSIAN
CHANNELS
209
P r o o f . Given an encoding scheme by (5.6.6), i t is clear that the input X — {X }
and the o u t p u t Y — {Y }
Yi,....
Y _i.
n
are Gaussian, and £„ is a linear combination of
n
n
Thus the encoding and decoding are linear. By (5.6.8), the constraint
(5.4.3) is clearly satisfied. Here we shall show
m
-
lf\
= T*
(i +
' .
«=i,
by induction. Note that (5.6.9) is obtained by substituting n = JV + 1 i n (5.6.11). Equation (5.6.11) is t r i v i a l for n — 1. Suppose that (5.6.11) is true for n = k. By definition, X
and also Y — Xi + Zt are independent of Yi,:.,Yk-t-
k
k
is a linear combination of Y\,...,Yk, 6 and E[lt\Y ] k
+
1
= E[i\ Y , , Y k ] = t + m
Y] k
k
- i - l - -
ewt-Mn)
= —(X -E[X \Y }). a* k
k
k + i
= E[£ \ = 0. Therefore
I
Since X
Since £
£t+i can be written as
k
(5.6.12)
k
is independent of Z , i t is easily shown that k
2 2
On the other hand, by the assumption of induction and (5.6.8),
4 - * - V ( l + fj)
•
(5-6.14)
Now (5.6.12) - (5.6.14) show that (5.6.11) is true for n = k + 1, so that (5.6.11) holds for all n. Thus (5.6.9) is satisfied and the reproduction error is minimized. It is clear that
/(f,D<Jw«,r)
=
R{D{C,;£);£)
Consequently we have (5.6.10).
=
•
R(d(thO
<
H U l
210
DISCRETE
GAUSSIAN
CHANNELS
The optimal encoding given i n the theorem is interpreted as follows. A t time rc, having received the full information about the channel o u t p u t Y\,Y„_i
which
have been sent back by using the backward channel, the encoder at first finds the best mean square predictor £„ of £. Next, to amplify £ — £„ as efficiently as possible under the average power constraint, the encoder multiplies £ — £„ by a constant Q„, and finally inputs the signal X„ = a„(£ - £ ) into the channel. n
There are various coding schemes which enable us to transmit the mutual information equal to the capacity over the white Gaussian channel. I n fact, such a coding scheme has been presented i n Theorem 5.4.1, Note t h a t the coding scheme in Theorem 5.4.1 is w i t h o u t using of feedback and the message is a Gaussian but JV -dimensional random vector. On the other hand the coding i n Theorem 5.6.1 is with feedback but the message is one-dimensional. I f a Gaussian message is onedimensional like i n Theorem 5.6,1, then without using feedback i t is impossible to transmit m u t u a l information equal to the capacity.
5.7 C A P A C I T Y O P D I S C R E T E CHANNELS WITH
GAUSSIAN
FEEDBACK
This section is devoted to discussing the capacity of a Gaussian channel with feedback.
I n the case of feedback channel, generally speaking, it is a difficult
task t o calculate the capacity exactly.
On the other hand, i n the case of non-
feedback Gaussian channel, as we have seen i n Sec. 5.5, by use of an orthogonal transformation the problem is reduced to t h a t of a white Gaussian channel and the capacity can be calculated.
Recall that the Gaussian noise is whitened by
the orthogonal transformation and that the mutual information is kept invariant under the transformation. To study the capacity of a Gaussian channel (5.1.1) w i t h feedback, we shall introduce another transformation which also maps the noise Z = { Z } to a white n
Gaussian noise.
Since the channel is with feedback, the channel input is
w i t h respect to the channel output, that is the input X
n
cauial
depends only on the past
{Yt; it < n ] of the o u t p u t and not on the future { Y i ; k > n } . This indicates that the transformation should keep the causality. Such a transformation can be realized by using the canonical representation of the Gaussian noise Z (cf. Sec. 2.2).
5.7 CAPACITY
OF GAUSSIAN
CHANNELS
WITH
FEEDBACK
211
Define a process W = {W„} by W
, Z " - t " l ' Z-il ^[(Zn-E^iZ,,...^^,]) ]'
=
n
£
Z
„ g +
2
(5.7.!)
z
2
(for n = 1, E\Z„\Z! W
Z „ _ , ) = E\Z„\ = 0). Then i t can be easily seen that
= ( W } is a white Gaussian noise w i t h variance 1 (namely, an i.i.d. n
pro-
cess w i t h distribution JV(0, 1)). Since Z is Gaussian, the conditional expectation £ [ Z | Z ] , Z „ _ i ] is a linear combination of Z i , Z „ _ ! , so that (5.7.1) is rewritn
ten i n the form n
Conversely, this equation is solved to give the expression •
Z =Y,CinWi, \=\
" 6 Z
n
Denote by 7i„(Z)
(5-7.3)
+
P ) spanned by Z „ 1 < i < n. Then, by
the subspace of
(5.7.2) and (5.7.3), K.(Z)-M»(H0.
n62 .
(5.7.4)
+
Thus (5.7.1) is the canonical representation of the Gaussian process Z. Operating the same transformation on the input X — [X ]
and the output Y = {Y } of the
n
n
channel we obtain =
Xn'f^tHnXu
i-i
neZ
(5.7.5)
+
1=1
Then Yn=Xn
+ W,
n € Z
n
(5.7.6)
+
Noting n (X)
= Hn(X),
n
HJX)
=
» e Z , +
(5.7.7)
we know t h a t (5.7.6) presents a white Gaussian channel w i t h feedback. Let X„ and Y„ be the orthogonal projection of X„ and K„, respectively, on the subspace « „ . , ( ? )
-
H -i{Y)
(Xi
n
= f , = 0)._Since
W
n
is independent of f i
it
is easily seen from (5.7.6) that Y = X„. We put n
u
B
= x„ - x , n
v
n
= Y _ ?„, n
ne z , +
(5.7.8)
212
DISCRETE
GAUSSIAN
CHANNELS
to define a white Gaussian channel V
N
= U
neZ .
+ W„,
N
(5.7.9)
+
Clearly V„ 6 W ( K ) , and conversely, noting V, = f\i we can show Y„ 6
H„(V)
n
inductively. Thus Hn(V)
= HniX),
n€Z .
(5.7.10)
+
It is an easy consequence of ( a . l ) , (5.7.4) and (5.7.10) that U„ is a function of £,Vi,..., V _ i . Thus we have three Gaussian channels (5.1.1), (5.7.6) and (5.7.9) n
w i t h feedback. The channels (5.7.6) and (5.7.9) are white Gaussian. Note that U
N
and V„ are orthogonal to Vj,..., V _ i , so that the input signals U„'s of the third B
channel (5.7.9) are mutually orthogonal. T h e o r e m 5 . 7 . 1 . The mutual information transmitted over the Gaussian channel (5.1.1) w i t h feedback is equal to those transmitted over the transformed white Gaussian channels (5.7.6) and (5.7.9), and they are bounded by M f , Y) =
1 V) < - £
Y ) = IN(C
N
l o g f l + rl),
N € Z , +
(5.7.11)
n=l
where r
2
is the variance of U„.
P r o o f . The equalities i n (5.7.11) are evident from (5.7.7) and (5.7.10). The inequality i n (5.7.11) follows from 1*) of Theorem 5.4.1.
•
We shall show that under an average power constraint the capacity of the Gaussian channel (5.1.1) w i t h feedback is achieved by sending a Gaussian message w i t h the use of a linear feedback. Let A be the class of all admissible pairs subject to the constraint (5.4.4): A
And let A
s
=
{(t,X}\
(t,X)
satisfies (a.l), (a.2), (5.4.4)}.
be a subclass of A consisting of all linear Gaussian pairs: A,
= {(£,X)
e A;t
= {(j} and
where
H ((, n
Y)
is Gaussian
X £H {(,Y), n
is a linear space spanned by f j ,
Vn},
n
j
< n,
and
Y
k
- X +Z , k
k
k <
n-l.
The following lemma plays a key role i n analyzing coding schemes for Gaussian channels w i t h feedback.
5.7 CAPACITY
OF GAUSSIAN
CHANNELS
L e m m a 5.7.2. For an arbitrary pair € A , such that { X \ , X
(£ ,X ) S
9
as (X\,JCjv,
has the same covariance matrix
for every N. Moreover, i n this case, it holds that
Z\,ZN),
lN(t,Y),Y'),
(5.7.12)
N
where Y* = {Y*}
is the o u t p u t corresponding to r* = ^
P r o o f . Let (£, X) process X m a t r i x as
(X
£' = (^)
by
«
9
S
9
Z , , Z
N
N
(5.7.13)
+
Then there exists a Gaussian has the same covariance
N
Define
) .
9
»6Z .
} such that ( X , . . . , X , Zj, ...,ZN)
...,X ,
L7
+ %
(£ ,X ):
e A be an admissible pair.
= { X
}
213
FEEDBACK
€ A , there exists a Gaussian pair
, Zi,Z/v)
N
WITH
Y'
=
by (5.7.13). We now define
{Y*}
XU,Y?,,*.,Y»„ ],
= Xi-B[Xi\X(
n>2.
l
<
5
7
1
4
)
Since X and JV have the same covariance and (5.4.4) is satisfied by X, the process 3
X
S
satisfies (5.4.4). Using (5.7.14), i t is shown that X
N
of f * , - . . , f J , Y i V - - , K * . _ , so that 1
is that
is a linear combination
satisfies (a.2). W h a t we have to show
(£',X»)
is independent of the noise Z = { Z „ } . By (5.7.14)
is independent
of Xf , Y f , i < n - 1, and of Z, = Yf - Jiff, i < rc - 1, so i t is independent of Clearly
Wi,i
W„
is independent of
X,
j
}
Y%,
k < n-1,
fc < n. Therefore the message £ is independent of W = {W„} 9
of Z = (Z ). N
Thus ( a . l ) is satisfied, and we have shown that ( ( » , P ) e
Note that Y * = {^n'} ' Y = {Y }.
and of g|,
and consequently
s
a
A. G
Gaussian process w i t h the same covariance function as
T h e n , using (5.1.3) and Theorem 1.8.4, we have
n
I ((,Y) N
= h (Y)
The proof is complete.
-
N
h (Z) N
< h (Y»)
-
N
h (Z) N
=
I {t',Y>). N
•
The lemma indicates that, even i f a Gaussian channel is w i t h feedback, the capacity is attained i n the Gaussian system
A. G
T h e o r e m 5.7.3. The feedback capacity Cf = Cf(A)
of the Gaussian channel
(5.1.1) subject to (5.4.4) is achieved by transmitting a Gaussian message using a linear coding scheme, namely C (A) F
=
Cf(A ). G
(5.7.15)
214
DISCRETE
Moreover Cj{A)
GAUSSIAN
CHANNELS
is equal to the supremum of the right-hand side of (5.7.11) taken
over all possible { r } . 2
P r o o f . For each pair (£,X) {t',X»)
e A
9
G .4, by Lemma 5.7.2, there exists a Gaussian pair
for which (5.7.12) holds. This means that (5.7.15) is true. Corre-
sponding to (£_ ,X ), S
we define { U * } and { V » } i n the same way as i n the proof
S
of Lemma 5.7.2. Then E[(Ulf]
= E[V ]
n € Z+,
= rl
n
and we are given a white Gaussian channel V* = U„ + W , Ufa, V * , V
(cf. (5.7.9)). Since (f/j\
N
Z ,
ne
n
)
(5.7.16)
+
is Gaussian and tf» is independent of
V * , V ' _ j , using Theorem 5.2.2, we obtain 1 = I {V,V)
INH',Y')
=
N
N
- £ > g ( l * T &
(5.7.17)
TJ=1
Thus the capacity Cj(A)
is equal to the supremum of the right-hand side of (5.7.17)
taken over all possible ( r } .
•
2
Using Lemma 5.7.2, we shall prove an interesting relationship between the mutual information in feedback Gaussian channel and i n non-feedback ones. T h e o r e m 5.7.4. Suppose that (£,-X) £ A is an arbitrary admissible pair for a Gaussian channel (5,1.1) with feedback.
Let X°
— { X ® } be a Gaussian process
independent of the noise Z = { Z „ } and w i t h the same covariance function as X = { X } , and let Y
= { Y ° } be the output, corresponding to the input X°,
A
N
of
a Gaussian channel Y* = X l + Zn,
n€Z , +
(5.7.18)
/ a,y)<2/ (X ,y°).
(5.7.19)
without feedback. Then w
w
0
P r o o f . Lemma 5.7.2 implies that, for the proof, we may assume that ( f , K ) is a Gaussian pair in A . Then, by Theorem 5.2.1, 9
Y
)
~ 2
l 0 S
WNW\
~ 2
I O g
|F (Z)| W
'
5.7 CAPACITY
where
OF GAUSSIAN
CHANNELS
denotes the covariance matrix of
T (Z) N
^
•
r
)
2
=
l o s
( Z
WITH , Z
t
FEEDBACK
Similarly
) .
N
215
ifMior-
Since X is independent of Z , Q
r
N
( r ° ) = r (x")
+ r
N
N
( z )
= r (X)
+
N
r
N
(z).
Therefore, i t is enough t o prove the inequality
| r v ( * + z)||r (z)| < w
\r (x)
(5.7.20)
+ r (z)\ .
N
2
N
We can easily show that T {X
+ Z) + V {X
N
- Z ) = 2T {X)
N
(5.7.21)
+ 2T (Z).
N
N
In general, i t is known that \A
+
a
{l-a)B\>\A\ \B\ - , a
l a
where A and B are non-negative definite matrices and 0 < a < 1 is a constant. I n particular, ±r (x
+ z) + ±r {x
N
z)| > \r (x
-
N
+ z)r (x
N
(5.7.22)
- z)\"\
N
In the same manner as i n the proof of Lemma 5.7.2, we can show that there is a pair
e
((',X')
A
s
such that
has the same covariances as { — X „ , Z „ } .
{X' ,Z„} n
Hence h (X
- Z ) - h (Z)
N
where Y
l
= M l -
N
1
) - h (Z)
= I
N
N
( ( \ Y ' )
= {Y*} is the output corresponding to ( f ' . X ) .
> 0,
By (1.8.3), this is
1
equivalent to (5.7.23)
> \r (Z)\.
\T (X-Z)\
N
N
The desired inequality (5.7.20) follows from (5.7.21) - (5.7.23): \r (X) N
+ r (z)\ N
2
-Y {X
>
1
>\r {x
+
>\r (x
+
N
N
The proof is complete.
+ Z) +
N
z ) r
N
\T {X-Z) N
( x - z ) \
z)\\r (z)\. N
•
We have already seen that the capacity of a white Gaussian channel is not increased by feedback (Theorem 5.4.1).
I n general, feedback can increase the
capacity of a Gaussian channel, but at most twice.
216
DISCRETE
T h e o r e m 5.7.5. 1")
Under an average power constraint, denote by Cf (resp.
GAUSSIAN
CHANNELS
C ) the capacity of the Gaussian channel (5.4.1) w i t h (resp. w i t h o u t ) feedback. Then C j is bounded by C
(5.7.24)
f
2°)
Under an average power constraint, denote by Cf (resp. C ) the per unit
time capacity of the Gaussian channel (5.1.6) w i t h (resp. w i t h o u t ) feedback. Then C
2C.
f
3°)
(5.7.25)
The factor " 2 " i n (5.7.24) and (5.7.25) is the best possible i n the sense that
the number 2 cannot be replaced by any other number less than 2. P r o o f . 1")
The first inequality of (5.7.24) is clear by definition. For any £ > 0
there exists a pair (£, X ) e A such that C,-e
(5.7.26)
N
where Y — {Y„} is the output corresponding to (£, X ) . Corresponding to X, we choose X° — { X } as i n Theorem 5.7.4. Then clearly 0
Z w ( f , r ) < 2I (X\Y°)
< 2C.
N
(5.7.27)
Since G is arbitrary, the second inequality of (5.7.24) follows from (5.7.26) and (5.7.27). Assertion 2") can be proved i n the same way. 3°)
For any e > 0, Example 5.7.2, given later, w i l l show that there exists a
Gaussian channel for which C
f
>
(2-ejC
holds. This means that the factor 2 i n (5.7.24) and (5.7.25) cannot be replaced by any other number less than 2.
•
Although the proof is omitted, i t is known that the feedback can increase the capacity of the Gaussian channel by at most half a bit. T h e o r e m 5.7.6. Under the same conditions as in Theorem 5.7.5, denote by C j ' W
(resp. C
( W
' ) the capacity of the Gaussian channel (5.4.1) w i t h (resp. without)
feedback. Then
^<«><^ <»> C
+
l
l o g 2
.
m
M
)
5.7 CAPACITY
OF GAUSSIAN
CHANNELS
WITH
217
FEEDBACK
R e m a r k . The constraint (5.4.4) is not necessarily essential t o prove Lemma 5.7.2 and Theorems 5.7.3 - 5.7.6. The statements remain true as long as the channel has a constraint given in terms of the covariance matrices of the inputs. Example 5.7.1.
Let the noise Z = { Z } be the Gaussian
(cf. Example 5.5.1)
n
AR(1) process given by (5.5.11). To give lower and upper bounds for the feedback capacity Cf (resp. C )
of the Gaussian channel (5.4.1) (resp. (5.1.6)) subject to
s
(5.4.2) (resp. (5.4.18)), we consider the equivalent white Gaussian channels (5.7.6) and (5.7.9). For each input X = { X } of the channel (5.4.1), the corresponding n
inputs X = { X } and U = {U„}
of the channels (5.7.6) and (5.7.9) are given by
n
X
- X
n
—
U
n
n
=
(X„
aXn-\,
-
vXn-,)-
E[X
-
n
aXn-^Yu.^Yn-ll
(I/, = X i ) , Clearly E[Ul\ < E [ X \ < (14- a ) V know that the capacity Cf is bounded by
U >
2
Thus, by Theorem 5.7.3, we
2
C ^ ^ l o g l l - K l + cOV].
(5.7.29)
To give a lower bound for Cf, let us consider a specific coding scheme [jj, X ] E A . Let £ be a Gaussian random variable w i t h distribution N(Q,1)
and X = {X„}
be
given by
where a
n
f
X,
=
p£,
1
X
=
-
n
a
„ ( X
n
.
Y ,...,Y„-i\),
n
n>2,
t
2
n
of the white Gaussian channel (5.7.9) is given as Un=
S
E[X -l\
is a constant determined in such a way that £LY;j] — p • I n this case
the input U = {U )
(i/i =X ).
-
}
(\
+
n>2,
^jXn,
Noting
<-«8-(t+i)"««l-(i+£)V and using Theorem 5.7.3, wc know that the capacity Cf is bounded by 1 C
f
> I ((.,Y)
=
N
, i
M
l
+
^
/#((,
+
V)
=
||^
N
- Y, n=l 1
+
M l
(
+
r
»)
i) ^. 1
1
+
(5.7.30)
218
DISCRETE
It can be shown that {a„}
GAUSSIAN
CHANNELS
satisfies
f 4 = l + p\ [ 4-l
+ (l + s£r)V.
«>3.
Moreover one can show that the l i m i t a = lim,,..™ a„ exists and a is the unique positive solution of
Letting JV -* oo i n (5.7.29) and (5.7.30), we know that C ; is bounded by
i^-jk.{i+(x gv} +
< i l o g { l + (l + a)V }-
(5.7.31)
2
E x a m p l e 5.7.2.
(cf. Example 5.5.2)
Let the Gaussian noise Z = {Z„}
of the
channel (5.1.6) be given by
\z
n
= e - Z „ . + f f „ - ( l + c)€-»H' . ,
where W = {W„) C}{p)
(resp. C{p))
0
1
0
1
n>2,
(5 7 32) ' " V
is a Gaussian i.i.d. process w i t h variance 1 and a > 0. Denote by the per unit time capacity of the channel w i t h (resp. w i t h o u t )
feedback under the constraint (5.4.18). The noise Z is slightly different from the noise i n Example 5.5.2, however we can show that the capacity C ( p ) coincides with the capacity C i n Example 5.5.2 and is given by (5.5.18). I n particular C ( M « ) ) - log where po(ct)
(5-7.33)
is given by (5.5.17). The transformation (5.7.2) can be w r i t t e n i n the
form W
n
where bn = 1, ojt = <*(1 + a )
n = Y,K~kZ ,
n
l!
t _ l
e * _
distribution JV(0,1), and X = {X„}
a
G
Z + ,
(k £ Z ) . Let f be a random variable w i t h +
be given by
= (! + « ) " ( £ - { „ - ! ) ,
S.7 CAPACITY
OF GAUSSIAN
where £t = E[f\ Yi,...,Yi].
where d = l , d 0
m
= £™
CHANNELS
WITH
219
FEEDBACK
Then elementary calculations show that
+ « ) * • and
= 1
IN{S,
t n
at
(E
= !(£, f ) = I log
Y)
w
•
Now, letting JV — i oo, we obtain lim
i
N £
^
= J *
= PiH-
(5
7
3 4
)
where .
o(2 + c.)(l + o ) ( l - e - ° ) 2
-
2
(! + « - « - ) »
'
and 7(^n =
^-^log/'x;^)
= l o g ( l + o).
(5.7.35)
is bounded by
By (5.7.34) and (5.7.35), C/(M«))
> 7((.,Y)
C{{?,(<*))
= log(l + a).
(5.7.36)
Note the following relations 1 2 1+ a + lim — l o g ( l + a ) = l i m — log
lira
aj.0
p (c.) 0
a
e
a
= 1,
.. p i ( c ) 1 = hm = -. a io a 2
Using (5.7.25), (5.7.33) and (5.7.36) we can show that, for any e > 0, i f Q > 0 is small enough, then (2 -
£
< log ( l + (1 -
} C ( p H ) = (2 - 6 ) log 0
< C , (p, ((1 - | ) « ) )
|)«)
< C/(p (o)) < 2 C ( 0
Therefore
£4^
lim = 2, ojo C ( p ( o ) ) 0
indicating that the capacity is almost doubled by feedback.
P o
( )). a
220
DISCRETE
5.8 R A T E - D I S T O R T I O N
GAUSSIAN
FUNCTION
A DISCRETE GAUSSIAN
Let £ —
CHANNELS
OF
MESSAGE
£jv) be an JV-dimensional Gaussian random vector. Denote by
n=l
the mean square distortion. A formula for the rate-distortion function
R(D,£),
denned by ( 1 . 7 . 5 ) , is given in terms of the eigenvalues of the covariance matrix of £. In case of N — 1 , the formula has already been given i n Theorem 1.8.7. T h e o r e m 5 . 8 . 1 . Let £ = ( £ i , b e
an JV-dimensional Gaussian vector with
distribution JV(0, T). Then, for any D > 0 ,
R(JJ £) = ^ l o g m a x f e , l ) ,
(5.8.1)
;
where A [ , . . . . \
N
are the eigenvalues of Y and 0 > 0 is a constant uniquely deter-
mined by iv £ m i n ( A , t ) ) = Z) . 2
n
(5.8.2)
J
The proof is based on a similar argument to the water-filling method which is employed to determine the channel capacity (cf. Proof of Theorem 5 . 4 . 2 ) . L e m m a 5.8.2. The m i n i m u m value of 1
N
—
J(Z>,,-,Ow)H-^log
1
(5.8.3)
n=l
on the set J{D)
= {(D,,D ); £* N
D
= l
2
< D,
D„ > 0 } is equal to the right-
2
hand side of ( 5 . 8 . 1 ) .
P r o o f . We may assume that the eigenvalues are arranged i n decreasing order: Aj > ••• > A N - T O minimize J(D ...,D ), U
N
manner that; (i) D „ < A„, V n ; (ii) £ ^ 0p for some N
a
we choose [D ...,Z) ) lt
D\ = D ; 2
w
i n such a
and ( i i i ) D\ = • • • — D
and On. Then we can prove the lemma.
2
•
No
=
5.* RATE-DISTORTION
FUNCTION
OF A GAUSSIAN
P r o o f o f T h e o r e m 5 . 8 . 1 . Denote by matrix T.
Let u„ = ( t i „ i , . . . , a
n W
MESSAGE
221
the ( i , j ) component of the covariance
) £ R
N
(n — 1,...,JV) be the eigenvector of
r corresponding t o the eigenvalue A„. We may assume that Ui,,.,,Ujv forms an orthonormal basis of R n —
(i]i,...,r) ) N
. Hence U — (ttjj)jj"=l,...,JVis an orthogonal matrix. Let
N
be an arbitrary random vector such that
Define random vectors £ — (fi,...,£iv) and ij = ( 5 i , . . . , ? « ) by N
£m=
N
/, u„ £ k
k
and
fj„ = ^
and
1),, = ^
"nlc'7*-
(5.8.4)
u t ^ ,
(5.8.5)
Then we can show that iV
N
in = ^
Uic 6 n
Jt=l
*=1
(5-8-6)
E l t t ] - M-»* and d ( f , T ) = d(c;, ,).
(5.8.7)
r
It is clear from (5.8.5) that the transformation (5.8.4) keeps the mutual information invariant: = /(£,•?)•
(5-8-8)
From (5.8.7) and (5.8.8) we conclude that R(DU)
(5-8.9)
= R(D;0-
Equation (5.8.6) means that Gaussian random variables i , n
mutually independent. tion of R(D\£)
i t is enough to take the infimum only for ij =
( f i i f l ) , —,(fjv, W )
1, ...,7V, are
n =
Hence, using Lemma 4.2.2, we know that in the defini-
are independent:
^A^=^(/(l55M(^
=
inf{/(f,7);?GK(D)}
=
inf{£/(6,,^);7eA:(I>}},
{rj } n
such that
222
DISCRETE
GAUSSIAN
CHANNELS
where K(D)
= {i},d(£,ij)
< D and ( & , j f i ) , . . . , ( f r , l f r r )
Once the values d(f„,»7„) = £[|{„ — 2
7f | ] n
s
independent}.
= D , n = 1,..., JV, are specified, where 2
( D i , . . . , D j v ) £ J { D ) , by Theorem 1.8.7, we know that the minimum value of I(£,ff)
is equal to J ( D ) , Z ? w ) -
minimum of J(D ,£>,v)
is equal to the
on the set J ( D ) . A n d , by Lemma 5.8.2, the minimum
X
is given by (5.8.1).
Therefore Jl(Z>;£) = R(D;Q
•
We now turn to give a comment on the rate-distortion function of a non-Gaussian random vector. There is a duality between the rate-distort ion function of a nonGaussian random vector and the capacity of a channel w i t h additive non-Gaussian noise. T h e o r e m 5.8.3. Let £ = (fi,...,£w) be an JV-dimensional non-Gaussian random vector, and £° — ( £ j , f ^ ) be a Gaussian vector w i t h the same covariance matrix as f. Then the rate-distortion R(D; {), w i t h the mean square distortion, is bounded by R{D-£ )
-
0
where H(f;£°)
Btf; f)
< R(D; 0 < R{D\ ff°),
(5.8.10)
is the relative entropy.
We do not give the proof here. However, (5.8.10) can be proved along a similar line as in the proof of the inequality (5.5.20) for the channel capacity. For a stationary Gaussian process £ = {£„\n £ Z } , the rate-distort ion function R(D;£)
per unit time is calculated exactly. Here R{D\£)
is defined by (4.2.8) and
the distortion d(£,n) per unit time is defined by
3(£,
n
f = j i m ^ £25fle„ - , „ f ] ,
(5.8.11)
T h e o r e m 5.8.4. Let £ = ( £ „ } be a stationary Gaussian process w i t h S D F /(A), Then, for any D > 0,
5.9 CODING
THEOREM
FOR GAUSSIAN
223
CHANNELS
where 6 > 0 is a constant uniquely determined by J
min(/(A),(? )dA = O 2
(5.8.13)
a
The formula (5.8,12) can be shown by using Theorem 5.8.1 and the asymptotic behavior (2.4.17) of the eigenvalues { A „ } of the covariance m a t r i x of ( { 1 , £ w ) as
—* oo.
N
5.9 C O D I N G
THEOREM
GAUSSIAN
FOR
DISCRETE
CHANNELS
For a Gaussian channel the information stability, which is essential to establish the channel coding theorem, has already been shown i n Sec.
5.3. Therefore we
may expect that the coding theorem is held for the Gaussian channels. I n this section, we shall prove a coding theorem for a Gaussian channel, generalizing the coding theorem (Theorem 4.5.1) into two directions. First, the coding theorem is proved for a Gaussian channel with feedback as well as without feedback. Next, we show the coding theorem i n a stronger form than as in Theorems 4.5.1 and 4.5.2. Let us consider a Gaussian channel presented by (5.1.1).
Suppose that the
channel is w i t h feedback and the constraint (5,4.3) is imposed on the channel input. For each terminal time JV, denote by pairs
of a message £
(^ \X^ >) N
N
{ N >
(5.4.3) are satisfied. We denote by
the class of all admissible
A^ \p) N
and an input A " Cpj,f{p)
, w )
such that ( a . l ) , (a.2) and
the feedback capacity of the Gaussian
channel (5.4.1) subject to the constraint (5.4,3), and denote by C j { p ) the per unit time capacity of the Gaussian channel (5.1.6) w i t h feedback under the constraint (5.4.18). Similarly as in Sec. 4.5, we consider a message £< > w i t h probability N
where M
> 0 is an integer. Suppose that the output Y^ ^
(£( \X( ^)
is decoded into a received message | < . Then the error probability
(Ni
N
N
N
corresponding to
w )
TT" '' is given by 1
= />(£<"> ^
|W),
The coding theorem for the Gaussian channel can be stated as follows.
224
DISCRETE
T h e o r e m 5.9.1.
1°)
GAUSSIAN
CHANNELS
Assume that Bm % ^ = 0
is satisfied. Let (^ ^
be a message given by (5.9.1) w i t h
N
M
= \mrfpB#%
fNi
For any e > 0, i f
R
l
N f
N )
such that (£<">,JY< >) £ W
l i m TS
l
N
(N {
= 0.
N)
Xtffi
and
A \p)
(5.9.4)
i n (5.9.3), then (5.9.4) does not hold for any
+£
= N~ C j(p)
N
(5.9.3)
- e i n (5.9.3), then there exist an input
= N- C , (p)
N
and a received message ?
Conversely, i f
(5.9.2)
encodings and decodings. Under the constraint (5,4.18), a rate R less than C/(p)
2°)
versely, i f a rate 3°)
R
is achievable, then J i <
is achievable. Con-
C/(p).
The same statements 1") and 2°) hold i n the special case without feedback
upon substitution of
for
Cy(p)
and C(p) for C/(p),
C\j(p)
where Cyv(p) and
C(p) are the capacities of the same channel without feedback.
Proof.
1°)
Using Theorem 5.7.3, i t is shown that there exists a pair (0, X ) £
A^\p)
such that 6 = (0i,...,0jv) is a Gaussian vector independent of Z, the
channel input X = (Xi, . . . , X M ) is w r i t t e n i n the form X„ where
(
a
£
Hn-W),
= 0„ + (
n
W
n
£
+ W,
tt -,(Z), n
n = X
n
(„ +
W
n
JV,
6 rf _ (r) n
1
(5,9.5) (Y
=
(Y ...,Y ) U
N
is
the channel o u t p u t ) , and W « , f ) > C
w
W -
£
,
(5.9.6)
Note that ( „ + W„ represents the feedback term. The e-typical set 7 i ' by = {(x,y) £ R *; 2
where
| |logv> »(x,y) (N
K)J
< e} ,
N )
is given
5.9 CODING
THEOREM
FOR GAUSSIAN
CHANNELS
Noting the assumption (5.9.2) and applying Theorem 5.3.5, we obtain lim p g p f t p O ) = L
(5.9.7)
Since C e W„_i(fl), i t can be w r i t t e n i n the form n
n-l
Cn = £ c „ * £ V
(5.9.8)
1=1 Here we employ the random coding method similarly as i n the proof of Theorem 4.5.1. Let u
m
= {u
m n
; n = 1,...,/V}, m = 1,...,A/' ', be code words indepenW
dently drawn according to the identical distribution f l j . For the message
f , lN>
define a channel input {X„} by X„ where u
m
= {v )
= u „ + v
mn
m
+ W
if
n
£
( w )
=m,
(5.9.9)
is given by
mn
n-l 1=]
using the same coefficients { c „ t } of (5.9.8). Given the output {Y„\ n = 1,...,JV}, the received message £ ' ' is reproduced by the same decoding rule as in the proof w
of theorem 4.5.1. Denote by 7r' * the error probability of reproduction. We denote w
by P { • ) and E [ • ] the probability and the expectation with respect to the choice c
r
of code u = { u } . Then, using (4.5.10), (4.5.11), (4.5.13), (5.3.15) and (5.9.6), m
we have E [^\ c
< exp(C,v./(p> - 2NE] exp[-/„(fl, Y ) + Nz\ + <exp(-(/V-l) ) £
^
+
^
2
I
^ P
.
Consequently, by (5.9.2), we obtain lim E l T t
( N )
] = 0.
(5.9.10)
N—oo
Since the channel has feedback, concerning the average power constraint, we need a slightly different treatment from that i n the proof of Theorem 4.5.1. P u t
226
DISCRETE
Then, since 0„ + t
n
GAUSSIAN
is independent of W
and (5.4.3) is satisfied by {X„}
n
N
CHANNELS
of (5.9.5),
.
Therefore E("mn
+«mn)
5
< V " 7
2
-
(5-9-11)
.n=l
We define the class D ( p )
by
w
Z><»>(„) = { „ = { u
Note that, i f u =
{u } m
m }
;
e D
*
( N ,
£
I
£ >
m
n
+ ,
m R
)
2
< P - 7 ) • 2
( p ) , then the corresponding input X =
given by (5.9.9) satisfies the constraint (5.4.3). Since { u „ + v ; m
m = I,Af< ', N
2
mn
{X„}
n = 1,...,JV),
are independently drawn according to a Gaussian distribution,
from (5.9.11) we know that
for a constant a > 0. Now i n the same way as i n the proof of Theorem 4.5.1 we have l i m E [^ W (P)) N->oo C
N
N)
= 0.
This means that there exist sequences of inputs { A ' ' ' ' } and of received messages v
such that (£<*>,-Y/< >) £ N
A {p) { N) f
and (5.9.4) is satisfied.
The converse part can be proved i n the same manner as i n the proof of Theorem 4.5.2. Assertion 2°) is straightforward of 1"). manner as above.
HISTORICAL
Part 3°) can be proved i n a similar
•
NOTES
Additive noise channels, especially Gaussian channels, were first investigated by Shannon [103, 104] (see also [38, 43]), and further mathematically analyzed
HISTORICAL
by Russian mathematicians (Kolmogorov
NOTES
227
[79, 80], Gel'fand and Yaglom [45],
Pinsker [96, 97]). The formula (5.1.12) i n Theorem 5.1.3 for mutual information i n a stationary channel is due to Solev [110] and Ihara [66], while (5.1.11) is due to Pinsker [97]. The simultaneously orthogonal expansion has been employed by Kolmogorov [79] and Gel'fand and Yaglom [45j (see also [97]) to calculate mutual information i n Gaussian system. Information stability has been investigated by Pinsker [97] i n detail.
Especially Theorem 5.3.5 is proved i n [97]. The comparison theorems
(Theorems 5.5.4 and 5.5.5), concerning the capacity of channels w i t h additive noise, are due to Ihara [61]. For Gaussian channels w i t h feedback, except some special cases including memoryless channels, no explicit formulas for the capacity have been obtained.
It
is shown in Theorem 5.7.3 that the feedback capacity of a Gaussian channel is attained by transmitting a Gaussian message w i t h the aid of a linear feedback (see [62]). Coding schemes i n feedback Gaussian channels have been proposed, and lower and upper bounds for the capacity have been given i n [19, 20, 93, 94, 102, 112], I n Theorem 5.6.1, the optimal coding scheme is obtained to transmit a Gaussian message over a white Gaussian channel with feedback. This result is proved based on the idea given by Ihara [58]. The interesting fact that feedback increases the capacity of a Gaussian channel by at most twice was first pointed out by Pinsker (his proof has not been published). Theorem 5.7.4, and 1"), 2°) of Theorem 5.7.5 are proved by Cover and Pombra [22]. The example (Example 5.7.2) to prove 3") of Theorem 5.7.5 was provided by Ihara [68]. Theorem 5.7.6 is also due to Cover and Pombra [22]. Feedback capacity and non-feedback capacity are also studied in [28]. The formulas for the rate-distortion function (or £-entropy) of Gaussian processes, given i n Theorems 5.8.1 and 5.8.4, have been derived by Kolmogorov [79] and Pinsker [96] (see also [8]). The comparison theorem (Theorem 5.8.3), concerning the rate-distort ion function, is due to Binia, Zakai and Ziv [11], The coding theorem (Theorem 5.9.1) for a Gaussian channel with feedback is proved i n Cover and Pombra [22].
Chapter 6 Continuous T i m e Gaussian Channels 6.1 C O N T I N U O U S G A U S S I A N
CHANNELS
A Gaussian channel is a channel w i t h additive Gaussian noise.
A Gaussian
channel w i t h continuous time parameter is called a continuous (time) Gaussian channel which is represented by Y® where
and
X(t),Y(t)
Z(t)
= X($
+ Z(®i
«€T,
(6.1.1)
are a channel input, a channel output and a noise,
respectively, at time t , the time parameter space T is a subinterval of R, and the noise
Z
=
{Z(t};
t
£ T } is a Gaussian process. This channel is the most important
continuous channel and a good model for some common continuous communication systems. Chapter 6 is devoted to developing the theory of information transmission in continuous Gaussian channels. In the study of the Gaussian channels, we shall be required to take different approaches according to, (i) the Gaussian noise is white or non-white, (ii) the channel is w i t h feedback or without feedback, and (iii) the message and the channel input are Gaussian or non-Gaussian. If the channel is w i t h o u t feedback, then the input of the noise Z = { 2 ( t ) } .
X
=
{X(t)}
is independent
I f the channel is w i t h feedback, then there arises a
problem which has never appeared i n the discrete time case. Let us denote by £ =
s 6 S} a message to be transmitted and by if = { n , ; s £ S} a received 228
6.1 CONTINUOUS
GAUSSIAN
229
CHANNELS
message. I t is natural t o assume that the message £ is independent of the noise Z. Suppose that the backward channel is noiseless and without time delay. Then an input signal X{i), at time (, is a function of the message £ and the output { Y ( u ) ; u < ( } u p to t. Intuitively, X(t) is w r i t t e n as
* M
=X
(t,
ti •
,"),r '-( )), 0
w
t
et,
£ n,
M
(6.1.2)
where £ ( - , w ) = { { ( a , w ) ; s e S} and K '~(w) = ( K ( u , w ) ; u < ( ) . Note that, 0
if
Y(t)
is continuous i n ( w i t h probability one, then
replaced by
Y '(ui) 0
=
i n (6.1.2) can be
Y '~(w) g
{ V ( u , u i ) ; u < t } . We should note that
X(
•
) is causal with
respect t o Y( •). Since X(t) depends o n Y(u), u < f, the channel equation (6.1.1) is a stochastic equation. Therefore, i n order t o develop meaningful discussions, the equation
(6.1.1) should
have a unique solution
Y
— {Y{t)}.
Mathematically
speaking, there arise some difficulties concerning the existence and the uniqueness of the solutions of the stochastic equation. However we do not go deep into this subject and proceed by assuming that the channel equation has a unique solution. The channel presented by (6.1,1) is called a c o n t i n u o u s G a u s s i a n c h a n n e l w i t h feedback i f the following conditions (A.O) - (A.3) are satisfied. (A.O)
Z = ( Z ( t ) ; f £ T } is a Gaussian process.
(A.l)
The message £ is independent of the noise Z.
(A.2)
The channel input X is a function of £ and the output Y, and causal with
respect t o Y. (A.3)
The stochastic equation (6.1.1) has a unique solution
Y = (Y{t)\
t
6T}.
If the channel is without feedback, the channel input X(t) depends only o n the message £ and is represented i n the form, X{t^)
= X(t,£{.^)).
(6.1.3)
We say that a channel (6.1.1) is a G a u s s i a n c h a n n e l w i t h o u t feedback, if (A.O), ( A . I ) and (A.2') are satisfied: (A.2')
The channel input X is given i n the form (6.1.3).
In the case where the channel is w i t h feedback and the backward channel has time delay of 6 > 0, the input X(t) can be w r i t t e n as X(t,»)
= X{t,£(-,u),Y <- ("))a
6
(S-l-4)
Clearly (6.1.3) and (6,1.4) can be treated as special cases of (6.1.2). Hence the arguments on the Gaussian channel w i t h feedback can be applied, without any change, to those channels w i t h time delayed feedback or without feedback.
CONTINUOUS
230
GAUSSIAN
CHANNELS
The most important Gaussian noise is a white Gaussian noise. Intuitively speaking, a Gaussian process v = {"(*)> ' € R-) is said to be a white Gaussian noise if it has independent values at every point, namely u ( s ) and v(t)
are mutually
independent for each s ^ t. Physically, such a noise does not exist and is just an idealized one. Mathematically, the white noise is justified as a generalized stochastic process i n the sense of Schwartz's distribution. A channel w i t h additive white Gaussian noise u is called a (continuous time) white Gaussian channel, which is presented i n the form 01= A Brownian motion
B
+
— {B{t);
t >
i6T,
(6.1.5)
0} is a Gaussian process such that incre-
ments B { s ) - B ( s i ) and B ( t ) - S ( * i ) are independent for each 0 < Si < s 2
tj
<
t, 2
2
and £ [ ( B ( t i ) —
B(t )) \ t
=
2
t
2
2
<
Roughly speaking, this means that
— t. }
the derivative B ( t ) (if it exists) of B ( t ) could be a white Gaussian noise. However it is known that the sample paths
are not differentiable w i t h respect to
B(t,w)
t. A n d we need a mathematical justification to define a white Gaussian channel. Formally integrating (6.1.5) we get Y(t) =
f i(u)riu + B ( t ) ,
t > 0.
(6.1.6)
Jo
This equation can be considered as a mathematical model of a white Gaussian channel. If the following conditions ( A , 0 ) - ( A , 3 ) are satisfied, then the channel pre0
0
sented by (6.1.6) is called a (continuous time) w h i t e G a u s s i a n c h a n n e l w i t h feedback. (Ao,0)
The noise
(A ,1)
The message £ is independent of the noise B .
(Ac,2)
The channel input i(()
0
{Y(u);Q
B
1
0
{B(i);
0} is a Brownian motion.
t >
(( > 0) is a function of £ =
{(,}
and Y * = 0
x(t w)
(A ,3)
=
=
x(t,t(-,w),Y *lti)i
Eq. (6.1.6) has a unique solution
Noting that the sample paths
t>0, weft.
0
B(t,uj}
Y
=
{Y(t);
t >
(6.1.7)
0}.
of the Brownian motion is continuous in
t w i t h probability one, so that the o u t p u t process Y ( f , w ) of (6.1.6) is continuous in t, we may use Y ' i n place of Y ~ Q
i n ( A , 2 ) . A white Gaussian channel (6.1.6)
g
0
is said to be without feedback, i f ( A , 0 ) , (Ad, 1) and (Ao, 2') are satisfied. 0
(A ,2') 0
The channel input x ( t ) is a function of the message £.
S.2 MUTUAL
INFORMATION
IN
WHITE
GAUSSIAN
CHANNELS
231
(I)
A fundamental problem i n Gaussian channels is to derive formulas to calculate mutual information and channel capacity. Formulas for mutual information w i l l be given i n Theorems 6.2.1, 6.3.2, 6.3.8, 6.7.1 and 6.8.3. Especially, when a Gaussian channel is w i t h o u t feedback (or w i t h linear feedback) and channel inputs are Gaussian, formulas to calculate the mutual information w i l l be derived in Theorems 6.3.2 and 6.3.8, i n a similar manner as i n the discrete time case. I t is a natural way to solve problems i n a Gaussian channel by reducing them to those i n a white Gaussian channel. Therefore i t is important to discuss the mutual information, the channel capacity, or the optimal coding schemes i n a white Gaussian channel. In this sense, the formula (6.2.2) i n Theorem 6.2.1 for the mutual information i n a white Gaussian channel w i t h feedback is most fundamental.
6.2 M U T U A L WHITE
INFORMATION
IN
GAUSSIAN CHANNELS
(I)
This section is devoted to deriving a formula to calculate the mutual information transmitted over the white Gaussian channel (6,1,6) w i t h feedback.
Let
£ = {£,\ s S S} be a message to be transmitted and denote by IT(£, Y) — ^(f.
Y )
the m u t u a l information between £ and the output Y
T
0
— {Y(t};
R
0
0 < t < T ) . The
formula (6.2.2) given in the following theorem plays a central role i n the theory of information transmission i n continuous time Gaussian communication systems. I t is noticed that the mutual information IT{£, Y) is related to a causal mean square filtering
error.
T h e o r e m 6 . 2 . 1 . Let £ = {£,; s <E S} be a message to be transmitted over the white Gaussian channel (6,1,6) w i t h feedback. x = {i(f)}
/ Jo
E[\x{t)\ ]dt
Then the m u t u a l information IT(£,Y)
h i t where x(t)
Suppose that the channel input
satisfies
= E[x(t)\
Y(u),
Y
) = \ [
2
< co.
(6.2.1)
is finite and is given by
* I M * > - m\\
0 < u < t] is the conditional expectation.
(6.2-2)
232
CONTINUOUS
GAUSSIAN
CHANNELS
The proof is based on a result concerning the absolute continuity of Wiener measure.
the probability distribution of a
Let T be fixed, and denote by
Brownian motion
B
=
T
{B(t);0
T } , namely
< t <
the sample paths of the Brownian motion
fi (A)
=
B
6
P{B^
Since
A).
is continuous almost surely,
{B{t)}
ft
B
is a probability measure on the space C([0,T]) of all continuous functions on [0,T]. The measure /ig is called a W i e n e r m e a s u r e . We denote by Sec. A.3). Let So / ( « W " )
f
a
Y =
f(u)dB(u)
=
f
0
the stochastic integral (see
f(v,ui)dB{u,u)
be the solution of (6.1.6). Then the stochastic integral
{Y(t)\
• Jo' / ( « , w ) d F ( u , w ) is given by
/ /(u,w)dV>,uO = / / ( u , u ) z ( , u ) d u - r / Jo Jo Ja U
Since the sample paths of the channel output
Y
=
(6.2.3)
f(u,w)dB(u,u).
are also continuous
{Y{t)}
functions of r almost surely, the probability distribution (iy of K
0
T
is a probability
measure on C([0, T}). Equation (1.6.5) suggests that the absolute continuity of /iy with respect to H B and the corresponding Radon-Nikodym derivative play important roles to calculate the mutual information. The following theorem, concerning the absolute continuity of the Wiener measure, has been known. T h e o r e m 6 . 2 . 2 . Under the assumptions of theorem 6.2.1, fiy is absolutely continuous w i t h respect to j i g and the Radon-Nikodym derivative is given by
dfiB
( Y V » = ex 0
/
P
x(t,w)dY{t,u)-
Ja
I
\ * Jo
m ^ f d i
£ fl.
(6.2.4)
We do not give the proof of Theorem 6,2.2 but proceed to prove Theorem 6.2.1. P r o o f o f T h e o r e m 6.2.1. Denote by of the message £ =
and fj^y the probability distribution
£ S} and the joint distribution of ( and KjT, re-
{£,,s
spectively, and denote by fiy^ the conditional distribution of Yj" given £.
It
follows from (6.2.1) and the property (A,2.5) of conditional expectation that £
E[E\\ (t)\ \(}]dt x
=
2
£
E\\ (t)\ ]dt x
2
< oo, meaning that
T E[]x(t)\ \(] 2
dt
< oo,
with probability one.
Hence we can apply Theorem 6.2.2 to j i y ^ and we know that fty^ is absolutely continuous w i t h respect to
fig.
Once £ is given, then
x(t)
=
x{t,f,,Y-}
depends
6.2 MUTUAL
INFORMATION
IN WHITE
GAUSSIAN
only on K '. Therefore, when we apply (6.2.4) t o iiy\t, ft
Y ')\ft
Y '}
0
= x(t).
0
o
x(t)
by
Thus we have
torn ( y ) = e x p ( 1
233
(I)
we should replace
0
E\x{t,
CHANNELS
r
jf
z(t)dY(t)-\j
\x{t)\*dt
(6.2.5)
Here we assume that p y - I p x /iy. Then, in general, i t is true that £
£
<^;v dfli
where p i x p
_ y| _
dii \(/dii
£
X fjy
d / l X HY
Y
denotes the direct product of /i\ and
2
B
d/iY/d/lB
duy
(
Hence i t follows from
(6.2.4) and (6.2.5) that
" ( f t ' ou ) — d p X p.y " ' fixp
[j-T ~
£
( f ) ( i
y
( ( )
_ I j-T |
£(<)
|2 *
(6.2.6)
We use (1.6.6) and (6.2.6) t o derive 1
All-,,
IHCY) =E
d u X fly "hi * r*Y
\
£
L
'ft
J
i
f
T ' dt 2
(6.2.7)
- E
Noting that
E
E[f
T
0
x(t)dB(t}]
=
0 (see (A.3.2)), i t follows from (6.2.3) that
= i j T E[\x(t)\ dt.
y\(t)dY(t)-l£\x(t)\*dt^
s
(6.2.8)
In the same way, i t can be shown that
E
-l>\
x(t)x(t)--\x(t)n
dt.
(6.2.9)
Now the formula (6.2.2) is an easy consequence of (6.2.7) - (6.2.9). To derive (6.2.2) we have assumed that ( i y •< f i x p (
£
Y
• Therefore, to complete the proof, we have to
CONTINUOUS
234
GAUSSIAN
CHANNELS
show that this is true. Oo the contrary, let us suppose that fi y e
there exists a set A , w i t h p-ey(A) > 0, on which dpy/dft
7$ H
x
(*$*'• Then
— 0 and dpyytldpB
B
> 0,
consequently log -
= oo
dpy/d/iB
on
A.
This means that dVY\(/dPB
E
/e
v
T
\
(6.2.10)
is infinite. On the other hand, (6.2.10) is equal to the right-hand side of (6.2.7) This is a contradiction. Therefore p y
which is finite by assumption (6.2.1).
(
should be absolutely continuous w i t h respect to / i x fiy.
•
£
R e m a r k . Since x(t) x(t))x(t)\
is a function of Y ', i t follows from (A.2.7) that E[(x(t) 0
-
= 0, and that E[\x[t)
-
x(t)\ ] 2
= E\\x(t)\
-
2
|2(«)| ].
(6.2.11)
2
Therefore i t is true that
m,y)-\l <|jf
T
{E[\ (m-E[x(t)\ ]}dt 2
X
E[\x(t)\ )di.
(6.2.12)
3
The formula (6.2.2) for the mutual information is quite i m p o r t a n t , especially the following points are worth being emphasized. (a) The formula can be applied even i f the channel is w i t h feedback. (b) The formula can be applied whether the channel input is Gaussian or nonGaussian. (c) Using the formula one can rather easily analyze how does the m u t u a l information Iriti
Y) grow w i t h T.
I t is known that the conditional expectation x(t) optimal mean square estimator of x(t)
= E[x(t)\K(ti),
u < f] is the
based on the observation of the output
K ( u ) , u < t. I n the filtering theory, i t is a fundamental problem to f i n d the best estimator x(t) and t o compute the error £[li(f.) — x(t)| ]. 2
is called the f i l t e r of x(t) and E\\x(t) - x(t)\ ] 2
is called the
The best estimator x(t) filtering
e r r o r . I t has
been seen that in white Gaussian channel to calculate the mutual information is equivalent to calculate the filtering error. Some examples t o calculate the mutual information will be given i n the next section.
S-t MUTUAL
INFORMATION
IN WHITE
GAUSSIAN
CHANNELS
2 3 5
(I)
Here let us consider a channel w i t h linear feedback. I t has a simple structure, b u t plays an important role i n practical situations. Let a message £ = { £ ( ( ) ; ' > 0} be a stochastic process. A white Gaussian channel w i t h l i n e a r feedback is presented by Y(*)=
f A(u)(£(u)-<(u))d + B(t), Jo
(6.2.13)
U
where A ( u ) is a non-random function and ( ( u ) = < ( u , F ) depends only on the 0
u
output Y . Intuitively speaking, £(u) represents the feedback term and A(u) is an U
B
amplitude function. I f feedback is not available, then the white Gaussian channel without feedback, corresponding to (6.2.13), is presented by
Vo(t) =
/ ' A{u)f(n)
du + B(t).
Ja
(6.2.14)
It is shown that the mutual information and the filtering error are not changed by using the linear feedback. T h e o r e m 6.2.3. Let £ = ( £ { ( ) } be a message, and
Y =
{Y(t)}
and
Y
a
=
{Y (t)} 0
he the outputs of the feedback channel (6.2.13) and the non-feedback channel (6.2.14), respectively. Assume that 0 < |A(u)| < M < oo (0 < u < T ) and
£
*< < oo,
[
£[IC(f)| ] du < co. 2
Then for any 0 < t < T I,(f,Y)
- (Wl } 1
= It&Yo),
(6.2.15)
= £ ( l « 0 - tm%
(6-2-16)
where ff(f) = £[£(f)| V » , « < t] and £o(f) = £[£(()! tt(u), u < (].
P r o o f . Since £ ( « ) is a function of Yg, the output K ( u ) = Y { u ) + f * A(s)£(s) da 0
of the non-feedback channel is a function of the output Y
g
of the feedback channel.
Consequently, we have I,((>Y)>It(t>Y )
(6.2.17)
0
and B [ m - m i < E [ \ m - i o ( t ) \
2
) .
(6.2.18)
236
CONTINUOUS
From (A.2.6) we know that
GAUSSIAN
E{£(u)\Y(s),
CHANNELS
s < u] =
Hence, applying Theorem
((s).
6.2.1, we see that /«(*,
y)=\f
A(u?mM
= \f
-
m
«km
-
-
cooi vm,
«in
* <
A(u) E[|£(u) - £(«)| ] du. 2
(6.2.19)
2
It follows from (6.2.17) - (6.2.19) and Theorem 6.2.1 that I (f,Y)>I (£,Y ) t
t
a
=
>
-^A(ufE[\£(u)-Uu)\ du
1
^'A(u) £[|£(u)-£( )| ]d 2
!
U
i
u
= /.(£, n
(6.2.20)
Thus the two inequalities i n (6.2.20) hold w i t h equality and we have the equations (6.2.15) and (6.2.16).
•
The output of the feedback channel is written as
Y(t)
=
Y (t)
— j ' A(u)C(u)
a
0
du.
The feedback term j j . A ( u ) ( ( u ) d u is specified by the o u t p u t Y * which is known a
for the receiver.
This means that the linear feedback brings to the receiver no
new information. This is the reason why the mutual information is not increased and the filtering error is not decreased by linear feedback. However feedback can be used to save the transmission energy. Suppose that the channel (6.2.13) has an average power constraint £[|i(!)| ] 2
— A(t) E[\£(t) 2
—
C(f)| ] < p imposed on 2
2
the input signals. We can reduce £[|£(i) — ((t)| ]> by choosing £ appropriately. 2
Then more effective amplitude function A(t)
is allowed. As a result, this make
i t possible to increase the mutual information transmitted over the channel and reduce the filtering error. This observation suggests to us how to construct an optimal coding scheme under an average power constraint. The optimal coding methods w i l l be studied i n Sec. 6.5 i n detail.
6.3 M U T U A L I N F O R M A T I O N W H I T E GAUSSIAN
IN
CHANNELS
(II)
6.3 MUTUAL
INFORMATION
IN WHITE
GAUSSIAN
CHANNELS
(It)
A message £ = {£(s);s > 0} is called a Gaussian message i f £ is a Gaussian process.
I n this section we consider a case where a Gaussian message is trans-
mitted over a white Gaussian channel. As we have seen i n the preceding section, calculation of the mutual information is directly related to calculation of the mean square filtering error. I t is known that some problems i n filtering can be solved w i t h the aid of results i n the theory of integral equation. Some useful known results, concerning the filtering error, will be put i n order. Then they w i l l be applied to calculate the m u t u a l information. A t first, a white Gaussian channel without feedback is considered. Then the channel input may be identified w i t h the message, so that the channel is described as t
r
Y(t)=
/ ' e ( ui ))d u + B(t), Jo
0
(6.3.1)
where £ = { £ ( ( ) } is assumed to be a Gaussian process w i t h mean aero and covariance function
= £[£(i)£(s)] such that
R(t,s)
R{t,t)dt
<
co. Note that the
covariance function R ( t , s) has the following properties: (i) i t is symmetric, i.e., R{t,
s) = R(s,t);
(ii) it is positive definite, i.e., ,T
and
(iii) R(t,s)
,T
R(t,s)f(t)f(s)d dt>0,
/
/
Jo
Jo
V/eL ([0,T]); 3
s
g L ( [ 0 , T ] ) , i.e., 2
r
Jo
J
fR^sf
dsdt
<
Ja
rr
Jo
Elatfmttsywadt
Jo
2 R{t,t)dt
T
< OO.
Here we denote by £ ([0, T]) the space of all real (measurable) functions /(t) such J
that j
a
T
f(t)
2
real functions
dt < oo. I n the same way £ ( [ 0 , T ] ) is defined as the space of all 3
g(t,s)
satisfying
£
$
g(t,
2
s)
2
dsdt
<
oo. Since
forms a Gaussian system, i t is seen that the filter (f(t) = £(t) is a linear function of Y(u), integral equation. A function
E[£{t)\Y(u),
{f(t),B{t),Y(t)} u <
t]
of
u < t , and is obtained by solving a Wiener-Hopf f{t,s)
is called a Volterra function if / ( t , s ) =
0
whenever t < s. L e m m a 6 . 3 . 1 . Let
R(t,s)
be a covariance function. Then the Wiener-Hopf equa-
tion h(r, s) + J
k(t, u)R(u,
s) du = J i ( i , s),
a<s
(6.3.2)
238
CONTINUOUS
GAUSSIAN
has a unique Volterra type solution
6
h(t,s)
CHANNELS
I f R(t,s)
L (\0,Tf). 2
is continuous i n
( i , s), then as the solution we may take a function h(t, s) which is continuous on the domain {((, s ) e R ; 0 < ^ < ( < T } . We omit the proof but briefly sketch the outline of the proof. O u t l i n e o f P r o o f . Define Ji< >(i,s) by / ^ " ( M ) = n
/E
( n + l ,
/ R^(t,u)R(u,s)d ,
(t, )=
Jo
Then we can show that the function = E(-ir
Ji
(6.3.3)
defined by
h(t,s)
+ l
and n>l.
u
s
m 4
R(t,s)
t n ,
(6.3.4)
3 < t,
((,s),
1=1
satisfies (6.3.2). The uniqueness of the solution is proved as follows. Let
ki(t,s)
and h i ( t , s) be solutions of (6.3.2). Then the function h (t, s) = h (t, s) — h\(t, s) 0
satisfies Ji ((, s)
=
0
— j
Q
k (t, u}R(u,
s) du.
0
2
Therefore, noting the positive definite-
ness of R { t , a), we have 0 <
meaning that
/ Jo
k {t,a) da
h (t,s) 0
0
= -
1
f j Jo Jo
R(u,s)h (t u)h (t,a)duda l)
>
v
0. Thus h , ( t , a ) = « i ( t , s ) .
=
< 0
•
T h e o r e m 6.3.2. Let the message £ = { £ ( ( ) } be a Gaussian process w i t h continuous covariance function
R(t,s).
Then the filter, the filtering error and the mutual
information i n the white Gaussian channel (6.3.1) without feedback are given by f(*)=
(6.3.5)
/ h(t,s)dY{s),
Jo
E[\at)-at)\ ) 2
(6.3.6)
= h(t,t),
respectively, where k{t, s) is the solution of the Wiener-Hopf equation (6.3.2). P r o o f . Since
{({t),
B(t),
Y{t);
t >
0} forms a Gaussian system and the right-hand
side of (6.3.5) is a function of {Y(u);u
< t],
to prove (6.3.5) i t suffices to show
that £(t) - /„' n(f, a) d Y ( s ) is orthogonal for all Y(u), u < t: = 0,
u < ( .
(6.3.8)
6.3 MUTUAL
INFORMATION
IN WHITE
From the independence of {£(*)} and
GAUSSIAN
CHANNELS
(II)
239
(6.3.1), (6.2.3) and (A.3.3) we have
{B(t)},
Right-hand side of (6.3.8) =
-
E
= jf
jf
h(t, )f( )ds s
-
s
j * h(t,s)dB(s) j
+
(J*t(v)dv
S
B(u)
)
j f h(^»)J^B «)«foi ds
(R{t,s)-k{t,s)-
)
= 0, and we have (6.3.8). Using (6.3.2) and the symmetry of fl(i,s), i n the same way as above we obtain E\\Z(t)\\
=
[
I
h(t u)h{t,v)R(u,v)dudv
+
i
Jo JO =
/
f
h{t,u)
2
du
Jo k{t,u)R(u,t)du.
Jo
Thus, using (6.3.2) again,
EW)
2
~ ?(t) ] 2
= R(t,
t) -
/'
h{t, u)R(u,
t) du = h(t,
t).
Jo Now (6.3.6) and (6.3.7) follow from (6.2.11) and (6.2.12).
•
It is not true to say that the mutual information is explicitly calculated by means of (6.3.7), because no general method to solve the Wiener-Hopf equation has been known. However, there is a rather simple but important case where the Wiener-Hopf equation can be solved to calculate the mutual information. E x a m p l e 6 . 3 . 1 . Let £o(^) be a zero mean Gaussian random variable w i t h variance c , A(t) be a non-random function w i t h Jg A{t} 2
dt < oo, and £ = { £ ( ( ) ; ' >
2
0) be a message given by £(t,w) = A(t){o(w),
r > 0, bi e 9 .
Then, since the covariance function of £ is
R{t,
s)
=
a A{t)A(s), 2
(6.3.9) i t is easily seen
that the solution of (6.3.2) is given by
^ • ' - i / f f i , 1 +
0
2
(
6
3
1
0
)
CONTINUOUS
240
GAUSSIAN
CHANNELS
Consequently, the m u t u a l information transmitted by sending the message (6.3.9) over the channel (6.3.1) without feedback is given by a
A(s)
t*
2
1
I n particular, if A ( t ) = 1, then r (to,Y)
=
t
= | l o g ( l +
IfcY)
(6.3.11)
a
If a message is a Gaussian Markov process, namely a Gaussian process with simple Markov property, then the filter is given as a solution of a stochastic integral equation and the filtering error is given as a solution of a Riccati differential equation which is rather easier to solve than the Wiener-Hopf equation. This is known as the Kalman theory. We give here the results without proof. Let a Gaussian process rj = { l ( t ) i i > 0} be a solution of a stochastic integral equation >?(<)= f
0
+
f
a(s)ri( )d + S
Jo
S
f
Jo
t>0,
b{ )dW(s), S
(6.3.12)
where n is a random variable w i t h distribution JV(0, t r ) , W = ( W ( t ) } is a Brownian motion independent of no, and a ( t ) and b(t) are real functions satisfying 2
0
fT
T
f
J
\a{t)\dt
< oo
j
and
b(t)
2
dt <
co.
It is known that n is a Markov process, and conversely a Gaussian Markov process w i t h mean zero is presented as a solution of a stochastic integral equation of this type if its covariance function is differentiable. Let a message £ = { £ ( ' ) } be given by * > 0, w g
« t , w ) = A(f)ij(*, ), W
where A ( t ) is a function such that
A(t)
2
ft,
(6.3,13)
dt < oo.
T h e o r e m 6.3.3. Assume that the message £ of (6.3.13) is t o be transmitted over the channel (6.3.1). Then the filter ij(/) =
E[t}(t)\
Y(u),
u < f] of rj(i) is a solution
of a stochastic integral equation flr)=
f{a(s)-A( ) {s)m )d + S
Jo
2
1
s
s
f
Jo
A(s)-y(s)dY(s),
t >
0,
6.3 MUTUAL
INFORMATION
IN WHITE
GAUSSIAN
CHANNELS
241
(II)
where 7(1) = £[|n(i) - i7(i)| ) is the filtering error and is given as the solution of the following Riccati equation 2
t 7(0)
( 6
- " 3
1 4 )
R e m a r k . The differential equation (6.3.14) is equivalent to the integral equation
a
+
(s)ds)
(6(s)2 j4(s) 7(s)3)exp
* i' " '
a(u)rfu ds
(- £ ) }•
E x a m p l e 6.3.2. Let the input £ = { £ ( ( ) } of the channel (6.3.1) be a Gaussian process given by £(t) = ft - <> / ( ( s ) d i + i.W((), Jo
' >0,
(6.3.15)
where £0 is a Gaussian random variable with distribution JV(0,a ). The message 2
£ is obtained as a special case of (6.3.12), by substituting o ( t ) = — a, 6(t) = 6, A(t) = 1. I n this case (6.3.14) turns out to be £ ( f ) = - 7 ( 0 - 2 n ( r ) + t> , 2
7
2
7
<>0
7(0) = <*\ Solving this equation, i t can be seen that the filtering error 7(f) — i?[|£(() — £(t)| ] 2
is given by 7
(i) = " '
g l Z cex (2 ^ +i r)-l 2
P
v
r
v
+
V
^ T ^ - a ,
7
where c
=
/ 2
" T T
• w •
(6.3.16)
Consequently the mutual information is equal to
1 c - e x p ( - 2 y ^ T i ' t ) , 1, , = xlog + ; ( v « ' + 6 ' - o)(. ~ /
1
c
l
2
f
1
(6.3.17)
242
CONTINUOUS
GAUSSIAN
In particular, i f a > 0 and b = 2ac , 1
CHANNELS
then we can show that £ = { £ ( * ) } is an
2
Ornstein-Uhlenbeck Brownian m o t i o n w i t h covariance function R{t,s)
= <7 e-'
M ,
2
(6.3.18)
We now consider Gaussian channels w i t h feedback. Suppose that the message £ = {£(t);t > 0} is a Gaussian process w i t h a continuous covariance function R(t, s ) . Let us start w i t h a white Gaussian channel (6.2.13) w i t h additive feedback. For simplicity we assume A(t) = 1. Then the channel is given by
Y(t,u)=
l {£(u,w)-C(u,w)}du + Jo
fl(r,w),
0 < ( < T.
(6.3.19)
Moreover we assume that £(u, w) = C(u, ^ " ( w ) ) is a linear function of K(t»,u>), 0 < v < u. I n other words, Q = { ( ( ( ) ; t > 0} is a process satisfying
C(t)eH {Y),
/ Jo
t
where H (Y)
is the subspace of
t
L (il,
spanned by Y(s),
P)
2
E[C.{t?\dt < m,
(6.3.20)
0 < s < t. I n this case
it is shown that the stochastic equation (6.3.19) has a unique solution Y = L e m m a 6.3.4. If (6.3.19) has a solution Y = {Y(t)},
U {Y)
= [J
t
f £ L ([0,(])|,
then
0 < i < T.
2
u
Denote by H (Y)
Proof. Ht(Y)
Hu)dY( );
{Y(t)}.
(6.3.21)
the set of the right-hand side of (6.3.21). T h e n clearly
t
For an element
D
n
V = J^c Y(t )£H (Y), k
k
(6.3.22)
t
=a
k
(0 = (
0
< ••-< t
n
< t, and c is a constant), let / ( i t ) = E ? = * + i <*> ** < « < * * + ! • k
Then n y
=
E
i
\
n+1
E
t = 0 \,=*+l
^
i
(n**+a-ni*)}= /
/
/ ( « w « > e w . ( n
6.3 MUTUAL
INFORMATION
IN WHITE
GAUSSIAN
CHANNELS
(II)
243
where t = t and c , = 0. Noting that the set of all elements of the form (6.3.22) is dense i n H,(Y) and that H {Y) is a closed subset of H,(Y), we conclude that H,{Y) = Ht{Y). • a
+
n +
J
t
This lemma implies that (6.3.19) can be replaced by r(t,u>) = j f ^ (
u
,
w
) - ^
l ( u , s ) d K ( s , ) ) du +
fl(t,w),
w
(6.3.23)
where, by (6.3.20), l{t s) is a Volterra function belonging to L ^ j O . T ] ) . We consider a Volterra equation 1
t
l(t, u)k(u,
fc(t, s) + j
s) du = -l(t,
s),
0 < s < t < T,
(6.3.24)
which is obtained from (6.3.2) by replacing the integral kernel fl(i,s) by a Volterra function i(*,s). We shall show a similar result t o Lemma 6.3.1. L e m m a 6.3.5. The Volterra equation (6.3.24) has a unique solution k ( t , s ) £ L ( [ 0 , T ] ) such that 2
2
k{t,s)
+ J
= 0 for ( <
k(t,s)
k(t,u)I(u s)du
s.
The solution fc(f,s) satisfies
= -l(t,s),
t
0<s
(6.3.25)
Moreover, i f i ( t , s ) is continuous in ( t , s ) on {((,s) £ R ; 0 < s < t < T } , then fc((, s) can be taken as a continuous function on the same set. 2
P r o o f . Define ('"'((,s) i n the same manner as i n (6.3.3), and define k ( t , s ) by
n=]
We can easily show that the right-hand side converges i n L ( [ 0 , T ) ) and fc(f,s) 2
2
satisfies (6.3.24) and (6.3.25). To show the uniqueness of the solution, let fc[((,.s) and
k {t,s) 2
be solutions of (6.3.24). Then fc (t,s) 0
Since ;
( n l
= (-1)" j
k [t, 0
l< >(t,u)k {u, ) n
a
s
s) = fci((,s) — du,
k (t,s) 2
Wn,
satisfies (6.3.26)
- » 0 i n i ( [ 0 , T ] ) as n - » co, we conclude that fc (f,s) = 0. Thus the 2
2
solution of (6.3.24) is unique.
0
•
The solution k ( t , s ) of (6.3.24) is called the r e s o l v e n t k e r n e l of the Volterra kernel (((, s).
244
CONTINUOUS
GAUSSIAN
CHANNELS
T h e o r e m 6.3.6. The stochastic equation (6.3.23), representing a white Gaussian channel w i t h linear feedback, has a unique solution Y(t,u>)
= %rt,vi+£
^J\u,»)dY (a,u)j
du,
0
(6.3.27)
where k(t, a) is the resolvent kernel of Z(r, s) and %fau)=
f(u,u>)du
I
Jo
+ B(t,u>).
(6.3.28)
Furthermore i t holds that H,{Y)
= H,(Y ),
0
0
(6.3.29)
P r o o f . Using Y o f (6.3.28), (6.3.23) can be rewritten i n the form Q
Y(t)
Let
Y = {Y(t)}
£
+ j
(J"
/(u, s) dYfrj)
du = Y (t).
(6.3.30)
0
be given by (6.3.27). Then, noting l(u,v)k(v, )dY {a)J s
0
dv = £
l(u,v)k(v,s)dY {a))
dv
from (6.3.27) and (6.3.24), we see that
= 0. Thus
Y
satisfies (6.3.30). Conversely, let
Y =
{Y(t)}
be a solution o f (6.3.30).
Then, using (6.3.25) i n place of (6.3.24), the same argument as above shows that Y satisfies (6.3.27). This means that Y given by (6.3.27) is the unique solution of (6.3.30), equivalently of (6.3.23). T h e relation (6.3.29) is clear from (6.3.27) and (6.3.30).
•
6.3 MUTUAL
INFORMATION
IN WHITE
GAUSSIAN
CHANNELS
(II)
2-15
R e m a r k . Equation (6.3.28) presents the non-feedback Gaussian channel corresponding t o the feedback channel (6.3.23). We have already seen that additive feedback does not change the mutual information and the filtering error (see (6.2.15) and (6.2.16)).
Clearly the invariance (6.3.29) under linear feedback is
a stronger property than (6.2.15) or (6.2.16), and we obtain equations £(f) = £[£(01 > » , u <
t] = £ [ £ ( 0 1 Y l u ) , a
f
u
Mi, a) dY„(s)
7
(6.3.31)
Jo
and I,{t,Y)
= I,(Z,Yo)
(6.3.32)
=
where k(t, s) is the solution of (6.3.2). One of the most important model of an additive feedback channel is presented by
f {£(«,«)-£>,«)}*< + *(*,«),
Y(t,*)=
(6.3.33)
Jo where £(ti) = £[£(u)l K ( s ) , s < u j . I t is shown that the feedback i n (6.3.33) is linear and that the signal energy ( = the variance of the input) is minimized by the feedback. T h e o r e m 6.3.7. Under the assumptions of Theorem 6.3.6, the stochastic equation (6.3.33) has a unique solution Y(t,u)
where
Y
0
= Yo(t,u)-
J
(f
h(v,a)dY (a,u)j
is given by (6.3.28) and
= {Y (t)} 0
du,
0
k(t,a)
(6.3.34)
is the solution of (6.3.2). The
filter of ((t) is
tO>j[
l(t,s)dY(s)
=
jf
(6.3.35)
h(t,s)dY (a), 0
where /((, a) is the resolvent kernel o f — h(t, s). Furthermore, for any white Gaussian channel (6.3.19) w i t h additive feedback, i t holds that Elm
- c(*)i ] > mm 2
- mn
o< *< r.
(6.3.36)
P r o o f . By Lemma 6.3.5, — h(t, s) is the resolvent kernel of i(r, a), and by Theorem 6.3.6, the solution of Y{t,v)
= Y (t,w)-jf 0
( T
l{u,s)dY(a w)^ t
du,
(6.3.37)
246
CONTINUOUS
GAUSSIAN
CHANNELS
is given by (6.3.34). Combining (6.3.31), (6.3.34) and (6.3.37), we have E[ttt)\Y(u),u
h(t,s)dY ( )=
f