Nonlinear Optimization
Nonlinear Optimization
Andrzej Ruszczynski ´
PRINCETON UNIVERSITY PRESS PRINCETON AND OXFORD...
161 downloads
3760 Views
2MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
Nonlinear Optimization
Nonlinear Optimization
Andrzej Ruszczynski ´
PRINCETON UNIVERSITY PRESS PRINCETON AND OXFORD
Copyright © 2006 by Princeton University Press
Published by Princeton University Press, 41 William Street, Princeton, New Jersey 08540 In the United Kingdom: Princeton University Press, 3 Market Place, Woodstock, Oxfordshire OX20 1SY
All Rights Reserved Library of Congress Cataloging-in-Publication Data Ruszczy´nski, Andrzej P. Nonlinear optimization / Andrzej Ruszczy´nski. p. cm. Includes bibliographical references and index. ISBN-13: 978-0-691-11915-1 (acid-free paper) ISBN-10: 0-691-11915-5 (acid-free paper) 1. Mathematical optimization. 2. Nonlinear theories. I. Title. QA402.5.R87 2006 519.6–dc22
2005049614
British Library Cataloging-in-Publication Data is available
The publisher would like to acknowledge the author of this volume for providing the camera-ready copy from which this book was printed. This book has been composed in Times Roman and MathTime using LATEX. Printed on acid-free paper. ∞ pup.princeton.edu Printed in the United States of America 10 9 8 7 6 5 4 3 2 1
to Darinka
Wiele˙z lat czeka´c trzeba, nim sie˛ przedmiot s´ wie˙zy Jak figa ucukruje, jak tytun´ ule˙zy? – Adam Mickiewicz, Dziady, Cze˛´sc´ III
In how few years are fresh themes matured, As figs grow candied or tobacco cured? – Adam Mickiewicz, Forefathers’ Eve, Part III
Contents
Preface
xi
Chapter 1. Introduction
1
PART 1. THEORY
15
Chapter 2. Elements of Convex Analysis
17
2.1 2.2 2.3 2.4 2.5 2.6
Convex Sets Cones Extreme Points Convex Functions Subdifferential Calculus Conjugate Duality
Chapter 3. Optimality Conditions 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8
Unconstrained Minima of Differentiable Functions Unconstrained Minima of Convex Functions Tangent Cones Optimality Conditions for Smooth Problems Optimality Conditions for Convex Problems Optimality Conditions for Smooth–Convex Problems Second Order Optimality Conditions Sensitivity
Chapter 4. Lagrangian Duality 4.1 4.2 4.3 4.4 4.5 4.6 4.7
The Dual Problem Duality Relations Conic Programming Decomposition Convex Relaxation of Nonconvex Problems The Optimal Value Function The Augmented Lagrangian
17 25 39 44 57 75 88 88 92 98 113 125 133 139 150 160 160 166 175 180 186 191 196
PART 2. METHODS
209
Chapter 5. Unconstrained Optimization of Differentiable Functions
211
5.1 Introduction to Iterative Algorithms 5.2 Line Search
211 213
x
CONTENTS
5.3 5.4 5.5 5.6 5.7 5.8
The Method of Steepest Descent Newton’s Method The Conjugate Gradient Method Quasi-Newton Methods Trust Region Methods Nongradient Methods
Chapter 6. Constrained Optimization of Differentiable Functions
218 233 240 257 266 275 286
Feasible Point Methods Penalty Methods The Basic Dual Method The Augmented Lagrangian Method Newton’s Method Barrier Methods
286 297 308 311 324 331
Chapter 7. Nondifferentiable Optimization
343
6.1 6.2 6.3 6.4 6.5 6.6
7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8
The Subgradient Method The Cutting Plane Method The Proximal Point Method The Bundle Method The Trust Region Method Constrained Problems Composite Optimization Nonconvex Constraints
Appendix A. Stability of Set-Constrained Systems A.1 Linear–Conic Systems A.2 Set-Constrained Linear Systems A.3 Set-Constrained Nonlinear Systems
343 357 366 372 384 389 397 406 411 411 415 418
Further Reading
427
Bibliography
431
Index
445
Preface Optimization is a mature and very successful area of applied mathematics. It deals with the analysis and solution of problems of finding the best element in a set. The structure of the set and the way the term “best” is understood determine different fields of the optimization theory: linear, integer, stochastic, nonsmooth optimization, optimal control, semi-infinite programming, etc. Within these fields, and across them, several directions of research are pursued: optimality conditions, duality theory, sensitivity analysis, and numerical methods. Applications of optimization are very broad and diverse. The most notable areas of application include engineering, statistics, economics, management sciences, computer science, and mathematics itself. Optimization problems arise in approximation theory, probability theory, structure design, chemical process control, routing in telecommunication networks, image reconstruction, experiment design, radiation therapy, asset valuation, portfolio management, supply chain management, facility location, among others. Many books discuss specific topics within the area of optimization, different groups of numerical methods, and particular application areas. My intention is to provide a book that will help to enter the area of optimization and to understand its modern ideas, principles, and methods within a reasonable time, but without sacrificing mathematical precision. Optimization problems arising in applications usually require additional analysis. It is my firm conviction, supported by many years of experience, that understanding of the mathematical foundations of optimization is the key to successful analysis of applied problems and to the choice or construction of an appropriate numerical method. Our way to achieve these objectives is to present both the theory and the methods in a unified and coherent way. Such an approach requires a careful selection of the material, in order to focus attention on the most important principles. The first part of the book is devoted to the theory of optimization. We present theoretical foundations of convex analysis, with the focus on the elements that are necessary for development of the optimization theory. Then we develop necessary and sufficient conditions of optimality and the duality theory for nonlinear optimization problems. In the second part of the book,
xii
PREFACE
we apply the theory to construct numerical methods of optimization, and to analyze their applicability and efficiency. While we focus on the most important ideas, we do not restrict our presentation to basic topics only. We cover rather modern concepts and techniques, such as methods for solving optimization problems involving nonsmooth functions. This is possible because of the unified treatment of the theory and methods, which clarifies and simplifies the description of the algorithmic ideas. This book grew out of my lecture notes developed during 25 years of teaching optimization at Warsaw University of Technology, Princeton University, University of Wisconsin-Madison, and Rutgers University, for students of engineering, applied mathematics, and management sciences. A one-semester course can be taught on the basis of the chapters on convex analysis (without the subdifferential calculus), optimality conditions and duality for smooth problems, and numerical methods for smooth problems. It is well-suited for graduate students in engineering, applied mathematics, or management sciences, who have had no prior classes on optimization. The inclusion of the material on nondifferentiable optimization is possible in a one-year course, or in a course for students who have already had contact with optimization on a basic level and who have a good background in multivariate calculus and linear algebra. The book can also be used for independent study. It is self-contained, it has many examples and figures illustrating the relevance and applications of the theoretical concepts, and it has a collection of carefully selected exercises. I hope that it will serve as a stepping stone for all those who want to pursue further studies of advanced topics in the area of optimization. I would like to thank Jacek Szymanowski and Andrzej Wierzbicki for introducing me to the area of nonlinear optimization. Many thanks also go to Robert Vanderbei for encouraging me to undertake this project and for his support. Ekkehard Sachs provided me with invaluable feedback, which led to significant improvements in the chapters on numerical methods. And special thanks are due to my wife and closest collaborator, Darinka Dentcheva, whose continuous encouragement, creative suggestions, and infinite patience helped me see this project to its conclusion. Princeton, New Jersey, July 2005
Andrzej Ruszczy´nski
Nonlinear Optimization
Chapter One Introduction A general optimization problem can be stated very simply as follows. We have a certain set X and a function f which assigns to every element of X a real number. The problem is to find a point xˆ ∈ X such that f (x) ˆ ≤ f (x) for all x ∈ X. The set X is usually called the feasible set, and the function f is called the objective function. At this level of generality, very little can be said about optimization problems. In this book, we are interested in problems in which X is a subset of a finite dimensional real space n , and the function f is sufficiently regular, for example, convex or differentiable. Frequently, the definition of X involves systems of equations and inequalities, which we call constraints. The simplest case is when X = n . Such a problem is called the unconstrained optimization problem. If X is a strict subset of the space n , we speak about the constrained problem. The most popular constrained optimization problem is the linear programming problem, in which f is a linear function and the set X is defined by finitely many linear equations and inequalities. If the objective function f or some of the equations or inequalities defining the feasible set X are nonlinear, the optimization problem is called the nonlinear optimization (or nonlinear programming) problem. In this case, the specific techniques and theoretical results of linear programming cannot be directly applied, and a more general approach is needed. Nonlinear optimization problems have attracted the attention of science since ancient times. We remember classical geometrical problems like the problem of finding the largest area rectangle inscribed in a given circle, or the problem of finding the point that has the smallest sum of distances to the vertices of a triangle. We know that optimization occurs in nature: many laws of physics are formulated as principles of minimum or maximum of some scalar characteristics of observed objects or systems, like energy or entropy. Interestingly, bee cells have the optimal shape that minimizes the average amount of wall material per cell. But the most important questions leading to optimization problems are associated with human activities. What is the cheapest way of reaching a
2
CHAPTER 1
certain goal? What is the maximum effect that can be achieved, given limited resources? Which model reproduces the results of observations in the best way? These, and many other similar questions arise in very different areas of science, and motivate the development of nonlinear optimization models, nonlinear optimization theory, and numerical methods of optimization. Let us examine several typical application problems. Of course, the versions of problems that we present here are drastically simplified for the purpose of illustrating their main features. Image Reconstruction One of the diagnostic methods used in modern medicine is computer tomography. Its idea is to send a beam through a part of the body at many angles, and to reconstruct a three-dimensional image of the internal organs from the results. Formally, this procedure can be described as follows. A part of the body (for example, the head) is represented as a set S ⊂ 3 . For the purpose of analysis we assume that S is composed of very small cells S i , i = 1, . . . , n (for example, cubes). We further assume that the density of the matter is constant within each cell, similar to the darkness of each pixel in a digital picture. We denote by x i the density within cell Si . These quantities are unknown and the purpose of the analysis is to establish their values.
L
Figure 1.1. Beam L intersects a subset of cells.
A beam L passing through the set S intersects some of the cells (see Figure 1.1). Let I (L) be this subset of cells. For every i ∈ I (L) we know the length ai (L) of the path of the beam within cell i. The attenuation P of the beam’s energy depends on the total amount of matter on its way, i∈I (L) ai (L)x i . For every beam, given the measurements of its energy at the exit point, we
3
INTRODUCTION
can estimate this sum by some quantity b(L). Thus, we obtain the linear relation X ai (L)x i = b(L). i∈I (L)
If this analysis is repeated for a collection of beams L j , j = 1, . . . , m, we obtain the system of linear equations X ai (L j )x i = b(L j ), j = 1, . . . , m. (1.1) i∈I (L j )
In addition, the physical nature of the problem dictates the inequalities: x i ≥ 0, i = 1, . . . , n.
(1.2)
Ideally, if all quantities were measured perfectly and the set S was actually built of small, uniform cells, the unknown density vector x would be a solution of this system. But several difficulties immediately arise. The number of beams, m, is not usually equal to the number of cells, n. The system of equations (1.1) may have too few equations to determine x in a unique way, or it may have too many or inconsistent equations. Its solution may violate inequalities (1.2), or it may have no solution at all. Moreover, the cell model is an approximation, the measurements have errors of their own, and it may be unreasonable to attempt to perfectly match their results. For all these reasons, it is better to abandon the idea of perfectly satisfying equations (1.1) and to use instead some measure of fit. It may be the sum of the squares f (x) =
m X j =1
b(L j ) −
X
ai (L j )x i
i∈I (L j )
2
.
The smaller the values of f , the better, and therefore we formulate the following problem: minimize
m X j =1
b(L j ) −
subject to x ≥ 0.
X
ai (L j )x i
i∈I (L j )
2
(1.3)
The above notation means that we want to find a point xˆ ≥ 0 such that f (x) ˆ ≤ f (x) for all x ≥ 0. This problem will always have an optimal solution, because the function f (x) → ∞ if any coordinate x i → ∞, and f (·) is a continuous function.
4
CHAPTER 1
Our optimization problem (1.3) may still have some flaws. If the number of cells is larger than the number of beams used, the problem may have many optimal solutions. We would prefer a model that selects one of them in some reasonable way. One possibility is to modify the function f (x) by adding to it the total amount of matter: f (x) =
m n 2 X X X b(L j ) − ai (L j )x i + δ xj , j =1
i∈I (L j )
j =1
where δ > 0 is a model parameter. The objective function of problem (1.3) and of the last modification is quadratic, and the constraints are linear. Such problems are called quadratic programming problems. There are numerous other ways to formulate a meaningful objective function in this problem. They may involve nonlinear terms representing our knowledge about the properties of the image. We may formulate a stochastic model of attenuation and an appropriate maximum likelihood estimator. There is no need to discuss all these forms here. Most important for us is that the optimization model provides the flexibility that the system of equations did not have. Portfolio Optimization We want to invest our capital K in n assets. If we invest an amount z i in asset i, after a fiscal quarter the value of this investment will become (1 + R i )z i , where Ri is the quarterly return rate of asset i. The return rate R i is a random variable. Assume that all capital K is invested: z 1 + z 2 + · · · + z n = K . Introducing the variables x i = z i /K , i = 1, . . . , n, we can express the quarterly return rate of the entire portfolio as 1 (1 + R1 )z 1 + (1 + R2 )z 2 + · · · + (1 + Rn )z n − 1 K = R1 x 1 + R2 x 2 + · · · + R n x n .
R(x) =
It is a random quantity depending on our asset allocations x i , i = 1, . . . , n. By the definition of the variables x i , they satisfy the equation x 1 + x 2 + · · · + x n = 1. We cannot sell assets that we do not own and thus we require that x i ≥ 0, i = 1, . . . , n. We denote by X the set of asset allocations defined by the last two relations. Our idea is to decide about the asset allocations by an optimization problem.
5
INTRODUCTION
The expected return rate of the portfolio has the form E R(x) = E R1 x 1 + E R2 x 2 + · · · + E Rn x n .
However, the problem
maximize E R(x) x∈X
∗ has a trivial and meaningless solution: invest everything in the asset j having the highest expected return rate E R j ∗ . In order to correctly model our preferences, we have to modify the problem. We have to introduce the concept of risk into the model and to express the aversion to risk in the objective function. One way to this is to consider the variance accomplish of the portfolio return rate, V R(x) , as the measure of risk. Then we can formulate a much more meaningful optimization problem maximize E R(x) − δV R(x) . x∈X
The parameter δ > 0 is fixed here. It represents the trade-off between the mean return rate and the risk. It is up to the modeler to select its value. It is convenient to introduce a notation for the mean return rates, µi = E Ri , i = 1, . . . , n,
for their covariances ci j = cov Ri , R j = E (Ri − µi )(R j − µ j ) , i, j = 1, . . . , n,
and for the covariance matrix
c11 c12 . . . c21 c22 . . . C = .. .. . . cn1 cn2 . . .
c1n c2n .. . .
cnn
Then we can express the mean return rate and the variance of the return rate as follows:
E R(x) = V R(x) =
n X
µi x i i=1 n X n X i=1 j =1
= hµ, xi,
ci j x i x j = hx, Cxi.
The optimization problem assumes the lucid form maximize hµ, xi − δhx, Cxi. x∈X
(1.4)
6
CHAPTER 1
The objective function of this problem is quadratic and the constraints are linear. Such an optimization problem is called a quadratic optimization problem. Our model can be modified in various ways. Most importantly, the variance is not a good measure of risk, as it penalizes the surplus return (over the mean) equally as the shortfall. One possibility is to measure risk by the central semideviation of order p: p 1/ p σp R(x) = E max 0, E[R(x)] − R(x) ,
with p ≥ 1. Then the optimization problem
maximize hµ, xi − δσp R(x) x∈X
becomes more difficult, because it involves a nonsmooth objective function. We can also introduce additional constraints on asset allocations. And for all these models, we can change the trade-off coefficient δ > 0 to analyze its influence on the solution to the problem. Signal Processing A transmitter sends in short time intervals I k harmonic signals sk (t) = ak cos(ωt), t ∈ I k , k = 1, 2, . . . . The pulsation ω is fixed. The signals are received by n receivers. Because of their different locations, each of the receivers sees the signal with a different amplitude, and with a different phase shift. The incoming signals are equal to u kj = c j ak cos(ωt − ψ j ),
j = 1, . . . , n, k = 1, 2, . . . ,
where c j and ψ j denote the amplitude multiplier and the phase shift associated with receiver j . It is convenient to represent the signal s k as a complex number Sk with the real and imaginary parts <(Sk ) = ak and =(Sk ) = 0. Similarly, each amplitude and phase shift are represented by the complex number Hj = c j e−iψj ,
j = 1, . . . , n.
Then the incoming signals can be represented in the complex domain as follows: Ukj = Hj Sk ,
j = 1, . . . , n, k = 1, 2, . . . .
7
INTRODUCTION
Because of various noises and interferences, the receivers actually receive the signals X kj = Ukj + Z kj ,
j = 1, . . . , n, k = 1, 2, . . . .
Here Z kj denotes the complex representation of the noise at receiver j in the kth interval. To facilitate the reconstruction of the transmitted signal {S k }, we want to combine the received signals with some complex weights W 1 , W2 , …, Wn to obtain the output signals: Yk =
n X j =1
W j∗ X kj , k = 1, 2, . . . .
In the formula above, W j∗ denotes the complex conjugate of W j . Introduce the notation: Z k1 W1 H1 .. .. .. Zk = . . W = . , H = . , Z kn Wn Hn We have
Yk =
n X j =1
W j∗ Hj Sk + Z kj = (W ∗ H )Sk + W ∗ Z k , k = 1, 2, . . . ,
where W ∗ = W1∗ W2∗ . . . Wn∗ . Our objective is to find the weights W j to maximize the useful part of the signal Yk , k = 1, 2, . . . , relative to the part due to the noise and interferences. To formalize this objective we consider the sequences {S k } and {Z k } as discrete-time stochastic processes with values in and n , respectively.† The noise process {Z k } is assumed to be uncorrelated with the signal {S k } and to have the covariance matrix 2 = E Z k Z k∗ .
The energy of the transmitted signal is denoted by σ 2 = E Sk Sk∗ .
The energy of the output signal {Yk } can be calculated as follows: E Yk Yk∗ = σ 2 kW ∗ H k2 + W ∗ 2W. † The symbol
denotes the space of complex numbers.
(1.5)
8
CHAPTER 1
Here the symbol kAk denotes the norm of the complex vector A ∈ kAk =
n X j =1
|A j |2
1/2
n
,
with |A j |2 = A j A∗j .
The first component on the right hand side of (1.5) represents the part of the output signal energy due to the transmitted signal {S k }, while the second component is the energy resulting from the noise. The basic problem of minimizing the noise-to-signal ratio can now be formalized as follows: minimize W ∗ 2W subject to kW ∗ H k ≥ 1.
(1.6)
Unfortunately, in practice the vector H is not exactly known. We would like to formulate a problem that allows H to take any value of form H0 + 1, where H0 is some nominal value and k1k ≤ δ, with the uncertainty radius δ > 0 given. Problem (1.6) becomes: minimize W ∗ 2W subject to kW ∗ (H0 + 1)k ≥ 1 for all k1k ≤ δ. It is a rather complex nonlinear optimization problem with a quadratic objective and with the feasible set defined by infinitely many nonlinear constraints. We can transform it to a simpler form as follows. First, we notice that the worst value of 1 for each W is 1 = −δW/kW k. Thus all constraints are equivalent to the inequality kW ∗ H k ≥ 1 + δkW k. Also, adding a phase shift to all components of W does not change the objective and the constraints, and therefore we can assume that the imaginary part of W ∗ H is zero. This yields the problem minimize W ∗ 2W subject to <(W ∗ H ) ≥ 1 + δkW k, =(W ∗ H ) = 0.
(1.7)
It has a convex objective function, one convex nonlinear constraint, and one linear constraint. Classification We have two sets: A = {a1 , . . . , ak }, and B = {b1 , . . . , bm } in n . Each vector ai may be, for example, the set of results of various measurements or tests performed on patients for whom a particular treatment was effective,
9
INTRODUCTION
while the vectors b j may represent results of the same tests for patients who did not respond to this treatment. Our goal is to find a classifier: ideally it would be a function ϕ : n → such that ϕ(ai ) > 0 for all i = 1, . . . , k, and ϕ(b j ) < 0 for all j = 1, . . . , m. When a new vector of results c ∈ n appears, we can assign it to sets A or B, depending on the sign of ϕ(c). This problem is usually modeled as follows. We first determine a family of functions ϕ(·) in which we look for a suitable classifier. Suppose we decided to use linear classifiers of the form ϕ(a) = hv, ai − γ . In this formula v ∈ n and γ ∈ are unknown. To find the “best” classifier in this family means to determine the values of v and γ .
v
Figure 1.2. Linear classifier. The white points belong to the set A and the dark points belong to the set B.
When a fixed family of classifiers is used, we cannot expect to be able to separate the sets A and B with a classifier from this family. There is always a possibility of misclassification, and we have to determine the way in which we shall decide which classifier is “better.” Figure 1.2 illustrates this issue for two sets and for a linear classifier. One way to measure the quality of classification is to consider for each point a i the classification error: e− (ai ; v, γ ) =
(
0 if hv, ai i − γ ≥ 0, γ − hv, ai i if hv, ai i − γ < 0.
10
CHAPTER 1
Similarly, for each point b j the classification error is e+ (b j ; v, γ ) =
(
0 if hv, b j i − γ ≤ 0, hv, b j i − γ if hv, b j i − γ > 0.
Then our problem can be formulated as an optimization problem minimize v,γ
k X i=1
e− (ai ; v, γ ) +
m X j =1
e+ (b j ; v, γ ).
(1.8)
We immediately notice a flaw in this model: setting γ = 0 and v = 0 we can make all classification errors zero, but the resulting classifier is useless. We need to restrict the set of parameters to ensure a discriminating power of the classifier. One possible condition is kvk = 1.
(1.9)
Problem (1.8)–(1.9) is a nonlinear optimization problem with a piecewise linear objective function and a nonlinear constraint. By introducing slack variables s ∈ k and z ∈ m , we can rewrite this problem as follows: minimize v,γ ,s,z
k X i=1
si +
m X
zj
j =1
subject to hv, ai i − γ + si ≥ 0, i = 1, . . . , k, hv, b j i − γ − z j ≤ 0, j = 1, . . . , m, s ≥ 0, z ≥ 0, kvk = 1.
(1.10)
Another way to guarantee a discriminating power of the classifier is to enforce a “buffer zone” between the two sets, by requiring that hv, ai i − γ + si ≥ 1, i = 1, . . . , k, hv, b j i − γ − z j ≤ −1, j = 1, . . . , m. The width of the buffer zone is equal to 2/kvk. The condition kvk = 1 is no longer needed, because v = 0 and γ = 0 are no longer attractive. However, we have to prevent kvk from becoming too large (otherwise the buffer zone has little meaning). One way to accomplish this is to add to the objective
11
INTRODUCTION
function a term which quickly grows with kvk, as in the problem below: minimize v,γ ,s,z
k X i=1
si +
m X j =1
z j + δkvk2 (1.11)
subject to hv, ai i − γ + si ≥ 1, i = 1, . . . , k, hv, b j i − γ − z j ≤ −1, j = 1, . . . , m, s ≥ 0, z ≥ 0.
Here δ > 0 is a fixed parameter of the problem. This quadratic optimization problem, under the name support vector machine, is commonly used in computer science for data classification. Optimal Control A robot has to move an object of mass M from position x ini to position x fin within time [0, T ] avoiding an obstacle (see Figure 1.3).
Y
x ini
x fin
Figure 1.3. The object has to be moved from x ini to x fin avoiding the obstacle Y .
One way to approach this problem is to split the time interval [0, T ] into n equal pieces h kT (k + 1)T , , k = 0, . . . , n − 1, Ik = n n
and to apply a constant force to the object within each interval. Denote by Fk the force used in interval k, and by v k the velocity of the object at the beginning of the interval k. Both Fk and vk are three-dimensional vectors. The velocity at the end of interval k equals vk+1 = vk +
T (Fk + G), k = 1, . . . , n. nM
(1.12)
Here G denotes the gravitation force. We ignore, for simplicity, the mass of the robot arm and the mechanics of the arm. Denoting by x k the three-dimensional vector representing the location of the object at the beginning of interval k, we can write the equation for the
12
CHAPTER 1
position at the end of interval k: x k+1 = x k +
T2 T vk + 2 (Fk + G), k = 1, . . . , n. n 2n M
(1.13)
Equations (1.12)–(1.13) are an example of state equations for a discrete-time dynamical system. The six-dimensional vector (x k , vk ) represents the state of the system at the beginning of the kth interval. The three-dimensional vector Fk is the control applied in the kth interval. The condition that the object cannot hit the obstacle can be modeled as follows. We describe the obstacle as a certain closed set Y ⊂ 3 and we introduce the distance function† M
d(x, Y ) = min kx − yk. y∈Y
Our requirement can now be formulated as follows: d(x k , Y ) ≥ δ, k = 0, 1, . . . , n, where δ > 0 is some minimum distance to the obstacle allowed. For the purpose of practical tractability we may replace this condition with a slightly weaker one, formulated for a sufficiently dense net of points y j ∈ Y , j = 1, . . . , J : kx k − y j k ≥ δ, k = 0, 1, . . . , n,
j = 1, . . . , J.
(1.14)
Other constraints are obvious from the problem formulation: (x 0 , v0 ) = (x ini , 0), (x n , vn ) = (x fin , 0), x k3 ≥ 0, k = 1, . . . , n − 1.
(1.15)
Finally, to formulate an optimization problem, we need to define a suitable objective function. Many reasonable formulations are possible. We can be concerned with the comfort of the travel by minimizing the variation of the force: f 1 (F) = kF0 + Gk + We can minimize the total work:
n−1 X k=1
kFk − Fk−1 k + kFk−1 + Gk.
n−1 X f 2 (x, F) = hFk , x k+1 − x k i, k=1
M † The symbol = means equal by definition.
13
INTRODUCTION
a combination of the two functions, or some other expression representing our preferences. The decision variables in the model are (x k , vk , Fk ), k = 1, . . . , n, and the state equations (1.12)–(1.13), as well as additional conditions (1.14)–(1.15), are constraints of the problem. Approximation of Functions We have a certain space of functions S defined on a domain D ⊂ m , and a function ψ ∈ S. The space S may be, for example, the space C(D) of continuous functions on a compact set D, or the space of integrable functions L p (D, B, µ), where p ∈ [1, ∞], D is a Lebesgue measurable set in m , µ is the Lebesgue measure, and B is a σ -algebra of measurable subsets of D. The space S is equipped with a metric dist(·, ·). We are also given a mapping A : n → S. For example, it may have the form A(x) =
n X
x j ϕj ,
j =1
where ϕ j , j = 1, . . . , n, are given functions in S. The last relation is understood as follows: A(x) is a function with values
A(x) (u) =
n X j =1
x j ϕ j (u), u ∈ D.
The approximation problem is simply the optimization problem: dist(ψ, A(x)). minimize n x∈
Several special cases are of interest. If S = C(D) and the distance between two continuous functions is measured by the maximum norm, we obtain the Chebyshev approximation problem minimize max ψ(u) − A(x) (u) . n x∈
u∈D
If S = L p (D, B, µ), with some p ∈ [1, ∞), then our problem has the form Z ψ(u) − A(x) (u) p dµ. minimize n x∈
D
A special case of practical importance arises when the domain D is finite: D = {u 1 , u 2 , . . . , u m }. Then the approximated function ψ can be viewed as a finite collection of data yi = ψ(u i ), i = 1, . . . , m. The approximation
14
CHAPTER 1
y
Data Model
u Figure 1.4. Chebyshev approximation.
problem is then the problem of constructing a model which is close to the data in the sense of the selected metric dist(·, ·). Figure 1.4 illustrates the solution of a Chebyshev approximation problem with a finite set of data. Optimization problems arise in engineering, economics, statistics, business, medicine, physics, chemistry, biology, and in mathematics itself. New optimization problems appear all the time, and scientists analyze their properties and look for their solutions. Frequently, as in the examples discussed above, the models have to be adjusted or modified, to reflect the intentions of their authors and the specificity of the application. Because of these reasons, almost no applied nonlinear optimization problem is exactly the same as models discussed in textbooks. Therefore, it is our intention to provide readers with tools that will help them to analyze their models, to choose the best solution methods, and to improve the models if the results are not appropriate. Such tools necessarily involve solid mathematical foundations.
PART 1
Theory
Chapter Two Elements of Convex Analysis 2.1 CONVEX SETS 2.1.1 Basic Properties
The notion of a convex set is central to optimization theory. A convex set is such that, for any two of its points, the entire segment joining these points is contained in the set (see Figure 2.1). C B
A
D
Figure 2.1. Convex sets (A, B) and nonconvex sets (C, D).
D EFINITION 2.1. A set X ⊂ x 2 ∈ X it contains all points
n
is called convex if for all x 1 ∈ X and
αx 1 + (1 − α)x 2 , 0 < α < 1. Convexity is preserved by the operation of intersection. L EMMA 2.2. Let I be an arbitrary index set. If the sets X i ⊂ T are convex, then the set X = i∈I X i is convex.
n
, i ∈ I,
Proof. If two points x 1 and x 2 are elements of X, they belong to each X i . Since X i ’s are convex, the segment joining these points is included in X i , for all i. It is, therefore, included in their intersection. Sets can be subjected to algebraic operations similarly to points in
n
.
18
CHAPTER 2
We can multiply a set X by a scalar c to get M
cX = {y ∈
n
: y = cx, x ∈ X}.
The Minkowski sum of two sets is defined as follows: M
X + Y = {z ∈
n
: z = x + y, x ∈ X, y ∈ Y }.
These operations preserve convexity. L EMMA 2.3. Let X and Y be convex sets in numbers. Then the set Z = cX + dY is convex.
n
and let c and d be real
Proof. If z 1 ∈ Z then z 1 = cx 1 + dy 1 with x 1 ∈ X and y 1 ∈ Y . Similarly, z 2 ∈ Z has the form z 2 = cx 2 + dy 2 with x 2 ∈ X and y 2 ∈ Y . Then, for every α ∈ [0, 1], αz 1 + (1 − α)z 2 = c(αx 1 + (1 − α)x 2 ) + d(αy 1 + (1 − α)y 2 ) ∈ Z,
as required. A point αx 1 + (1 − α)x 2 , where α ∈ [0, 1], appearing in Definition 2.1, belongs to the segment joining x 1 and x 2 . We can “join” more points by constructing their convex hull. D EFINITION 2.4. A point x is called a convex combination of points x 1 , . . . , x m if there exist α1 ≥ 0, . . . , αm ≥ 0 such that x = α 1 x 1 + α2 x 2 + · · · + α m x m
and α1 + α2 + · · · + αm = 1. D EFINITION 2.5. The convex hull of the set X (denoted by conv X) is the intersection of all convex sets containing X. The relation between these two concepts is the subject of the next lemma. L EMMA 2.6. The set conv X is the set of all convex combinations of points of X. Proof. Let us consider the set Y of all convex combinations of elements of X. If y 1 ∈ Y and y 2 ∈ Y , then y 1 = α1 x 1 + α2 x 2 + · · · + α m x m ,
y 2 = β1 z 1 + β2 z 2 + · · · + βl z l ,
19
ELEMENTS OF CONVEX ANALYSIS
where x 1 . . . , x m , z 1 , . . . , z l ∈ X, all α’s and β’s are nonnegative, and m X i=1
αi = 1,
l X i=1
βi = 1.
Therefore, for every λ ∈ (0, 1), the point 1
2
λy + (1 − λ)y =
m X i=1
l X λαi x + (1 − λ)βi z i i
i=1
is a convex combination of the points x 1 , . . . , x m , z 1 , . . . , z l . Consequently, the set Y is convex. Obviously, Y ⊃ X. Thus convX ⊂ Y. On the other hand, if y ∈ Y then y, as a convex combination of points of X, must be contained in every convex set containing X. Hence, convX ⊃ Y , which completes the proof. For example, the convex hull of three points in 2 is the triangle having these points as vertices. In the above result we considered all convex combinations, that is, for all numbers m of points and for arbitrary selections of these points. Both m and the points selected can be restricted. We now show that the number of points m need not be larger than n + 1. In Section 2.3 we further show that only some special points need to be considered, if the set X is compact. L EMMA 2.7. If X ⊂ n , then every element of conv X is a convex combination of at most n + 1 points of X. Proof. Let x be a convex combination of m > n + 1 points of X. We shall show that m can be reduced by one. If α j = 0 for some j , then we can delete the j th point and we are done. So, let all α i > 0. Since m > n + 1, one can find γ1 , γ2 , . . . , γm , not all equal 0, so that 1 2 m x x x γ1 + γ2 + · · · + γm = 0. (2.1) 1 1 1 Let τ = min αγii : γi > 0 . Note that τ is well defined, because some γ j > 0, if P their sum is zero. Let Pmα¯ i = αi i − τ γi , i = 1, 2, . . . , m. By (2.1) we still have m α ¯ = 1 and ¯ i x = x. By the definition of τ , at least i=1 i i=1 α one α¯ j = 0 and we can delete the j th point. Continuing in this way, we can reduce the number of points to n + 1.
20
CHAPTER 2
The last result is known as Carathéodory’s Lemma. We end this section with some basic relations of convexity and topological properties of a set. L EMMA 2.8. If X is convex, then its interior int X and its closure X are convex. Proof. Let B denote the unit ball. If x 1 ∈ int X, x 2 ∈ int X, then one can find ε > 0 such that x 1 +ε B ⊂ X and x 2 +ε B ⊂ X. Thus αx 1 +(1−α)x 2 +ε B ⊂ X for 0 < α < 1. Therefore αx 1 + (1 − α)x 2 ∈ int X. To prove the second part of the lemma, let x k → x and y k → y with x k ∈ X and y k ∈ X. Then the sequence of points αx k + (1 − α)y k is contained in X and converges to αx + (1 − α)y ∈ X. L EMMA 2.9. Assume that the set X ⊂ n is convex. Then int X = ∅ if and only if X is contained in a linear manifold of dimension smaller than n. Proof. Let x 0 ∈ X. Consider the collection of vectors x − x 0 for all x ∈ X. Let m be the maximum number of linearly independent vectors in this collection. Then all vectors x −x 0 , where x ∈ X, can be expressed as linear combinations of m selected vectors v1 , v2 , . . . , vm . Denoting by lin{v1 , v2 , . . . , vm } the subspace of all linear combinations of the vectors v 1 , . . . , vm we can write X ⊂ x 0 + lin{v1 , v2 , . . . , vm }.
If the set X has a nonempty interior, we can choose x 0 ∈ int X. Then a small ball about x 0 is included in X and we can choose exactly n independent vectors vi in the construction above. Therefore, m = n in this case. Conversely, suppose the set {x − x 0 : x ∈ X} contains n linearly independent vectors v1 , . . . , vn . By the convexity of X we have n n X X γi ≤ 1, γi ≥ 0, i = 1, . . . , n . X ⊃ x0 + γi vi : i=1
i=1
Due to linear independence of the vectors v i , the set on the right hand side is an n-dimensional simplex having a nonempty interior. 2.1.2 Projection
Let us consider a convex closed set V ⊂ n and a point x ∈ n . We call the point in V that is closest to x the projection of x on V and we denote it by 5V (x). Obviously, if x ∈ V then 5 V (x) = x, but the projection is always well defined, as the following result shows. T HEOREM 2.10. If the set V ⊂ n is nonempty, convex and closed, then for every x ∈ n there exists exactly one point z ∈ V that is closest to x.
21
ELEMENTS OF CONVEX ANALYSIS
Proof. Let µ = inf{kz − xk : z ∈ V }. Since V is nonempty, µ is finite. Let us consider a sequence of points z k ∈ V such that kz k − xk → µ, as k → ∞. It is bounded, so it must have a convergent subsequence, {z k }, k ∈ K. Denote the limit of this subsequence by z. We have kz − xk = lim kz k − xk = µ. k→∞ k∈K
Since V is closed, z ∈ V . This proves the existence. Suppose two different points z 1 ∈ V and z 2 ∈ V have distance µ from x. Consider the point z = (z 1 + z 2 )/2. It belongs to V , by convexity. Its distance to x can be calculated by the Pythagorean theorem: 1 kz − xk2 = µ2 − kz 1 − z 2 k2 < µ2 , 4 a contradiction. Thus the projection is unique. L EMMA 2.11. Assume that V ⊂ n is a closed convex set and let x ∈ Then z = 5V (x) if and only if z ∈ V and hv − z, x − zi ≤ 0 for all v ∈ V.
n
.
(2.2)
Proof. Let z = 5 V (x) and v ∈ V (see Figure 2.2). Consider points of the form w(α) = αv + (1 − α)z, 0 ≤ α ≤ 1.
By convexity, they are all in V and their distance to x cannot be smaller than kz − xk. We have kw(α) − xk2 = hz + α(v − z) − x, z + α(v − z) − xi
= kz − xk2 + 2αhz − x, v − zi + α 2 kv − zk2 .
Consider the above expression as a function of α ∈ [0, 1]. It is bounded below by kz − xk2 if and only if the linear term has a nonnegative coefficient. The last statement is identical to (2.2). Suppose (2.2) for some z ∈ V . Setting v = 5 V (x) in (2.2) we get h5V (x) − z, x − zi ≤ 0.
By the first part of our proof, as z ∈ V , inequality (2.2) has to be satisfied: hz − 5V (x), x − 5V (x)i ≤ 0.
Adding the last two inequalities we get
h5V (x) − z, 5V (x) − zi ≤ 0,
which is possible only if 5 V (x) = z.
22
CHAPTER 2
x
5V (x)
v
V
Figure 2.2. Projection.
In particular, if the set V is a linear manifold, for every v ∈ V we have w = 25V (x) − v ∈ V as well. Therefore the inequalities hv − 5V (x), x − 5V (x)i ≤ 0, hw − 5V (x), x − 5V (x)i ≤ 0, are equivalent to hv − 5V (x), x − 5V (x)i = 0. Consequently, x − 5 V (x) ⊥ V . Example 2.12. A model represents the output variable, y ∈ y=
n X
, as a linear function,
xi u i ,
i=1
of input variables u 1 , . . . , u n . The quantities x 1 , . . . , x n are unknown model coefficients. We have N observations of input and output variables: (u j , y j ), j = 1, . . . , N. One way to determine the values of the coefficients x 1 , . . . , x n is to minimize the sum of the squared errors: f (x) =
N X j =1
yj −
n X i=1
j 2
xi u i
.
23
ELEMENTS OF CONVEX ANALYSIS
Defining the matrix of input data
u 11 u2 1 U = . ..
u 1N
u 12 u 22 .. .
u 1n u 2n .. .
... ...
u 2N
u nN
...
and the set of all possible model outcomes V = {U x : x ∈
n
},
we see that this step of the estimation is equivalent to finding the orthogonal projection of the vector y = (y 1 , . . . , y N ) on V : yˆ = 5V (y).
If there exist more than one x such that U x = yˆ , the final step is to find in the set X = {x ∈
the minimum norm element x, ˆ that is,
n
: U x = yˆ }
xˆ = 5 X (0).
The projection operator is nonexpansive. T HEOREM 2.13. Assume that V ⊂ x ∈ n and y ∈ n we have
n
is a closed convex set. Then for all
k5V (x) − 5V (y)k ≤ kx − yk.
Proof. By Lemma 2.11,
h5V (y) − 5V (x), x − 5V (x)i ≤ 0, h5V (x) − 5V (y), y − 5V (y)i ≤ 0.
Adding both sides we get
k5V (x) − 5V (y)k2 + h5V (x) − 5V (y), y − xi ≤ 0.
We obviously have that
1 k5V (x) − 5V (y) + (y − x)k2 2 1 1 = k5V (x) − 5V (y)k2 + h5V (x) − 5V (y), y − xi + ky − xk2 . 2 2 Adding the last two inequalities we conclude that 0≤
1 1 k5V (x) − 5V (y)k2 ≤ ky − xk2 , 2 2 which was what we set out to prove.
24
CHAPTER 2
2.1.3 Separation Theorems
A convex closed set and a point outside of it can be separated by a plane. T HEOREM 2.14. Let X ⊂ n be a closed convex set and let x ∈ / X. Then there exist a nonzero y ∈ n and ε > 0 such that hy, vi ≤ hy, xi − ε for all v ∈ X. Proof. Let z be the point of X which is closest to x. It exists, since X is closed. By Lemma 2.11, hx − z, v − zi ≤ 0 for all v ∈ X. Define y = x − z. Note that y is nonzero, because x ∈ / X. For each v ∈ X the last inequality implies that hy, vi ≤ hy, zi = hy, xi + hy, z − xi = hy, xi − kyk 2 .
Setting ε = kyk2 we obtain the assertion of the theorem.
See Figure 2.3 for a geometrical interpretation of this property. y
x
5 X (x)
X
Figure 2.3. The main separation theorem.
If the set X is not closed, the point x ∈ / X may be a boundary point and we have a weaker version of the above result. / X. Then there T HEOREM 2.15. Let X ⊂ n be a convex set and let x ∈ exists a nonzero y ∈ n such that hy, vi ≤ hy, xi for all v ∈ X.
25
ELEMENTS OF CONVEX ANALYSIS
Proof. Let x k → x, x k ∈ / X. By Theorem 2.14 we can find y k 6 = 0 such that hy k , vi ≤ hy k , x k i for all v ∈ X.
(2.3)
Since y k 6 = 0, with no loss of generality we may assume that ky k k = 1, because we can divide both sides of (2.3) by ky k k. Let y be an accumulation point of the sequence {y k }. Passing to the limit (over a subsequence, if necessary) in (2.3) we obtain the required result. Separation is possible also when both objects being separated are convex sets. T HEOREM 2.16. Let X 1 and X 2 be convex sets in there exists a nonzero y ∈ n such that
n
. If X 1 ∩ X 2 = ∅, then
hy, x 1 i ≤ hy, x 2 i for all x 1 ∈ X 1 , x 2 ∈ X 2 .
Proof. Define X = X 1 − X 2 . Since x = 0 does not belong to X, we can use Theorem 2.15 to find y 6 = 0 separating 0 from X. This yields hy, vi ≤ 0 for all v of the form x 1 − x 2 with x 1 ∈ X 1 and x 2 ∈ X 2 . If one of the sets is bounded, strict separation is possible. T HEOREM 2.17. Let X 1 and X 2 be closed convex sets in bounded. If X 1 ∩ X 2 = ∅, then there exist a nonzero y ∈ that
n
n
and let X 1 be and ε > 0 such
hy, x 1 i ≤ hy, x 2 i − ε
for all x 1 ∈ X 1 and all x 2 ∈ X 2 .
Proof. By the closedness of X 1 and X 2 and boundedness of X 1 , the set X = X 1 − X 2 is closed. Since 0 ∈ / X, we can apply Theorem 2.14, similar to the proof of Theorem 2.16. The boundedness assumption in Theorem 2.16 is essential. The closed sets X 1 = {x ∈ 2 : x 2 ≤ 0} and X 2 = {x ∈ 2 : x 2 ≥ e−x1 } cannot be strictly separated. These geometrical facts have profound implications, as we shall see later in this chapter. 2.2 CONES 2.2.1 Basic Concepts
A particular class of convex sets, convex cones, play a significant role in optimization theory.
26
CHAPTER 2
D EFINITION 2.18. A set K ⊂ n is called a cone if for every x ∈ K and all α > 0 one has αx ∈ K . A convex cone is a cone that is a convex set.
We use only α > 0 in the cone definition to allow easy formulation of the cone separation theorem (Theorem 2.32), later in this chapter. A simple example of a convex cone in n is the nonnegative orthant: n +
= {x ∈
n
: x j ≥ 0, j = 1, . . . , n}.
For convex cones, positive combinations remain in the set, similar to convex combinations for convex sets. L EMMA 2.19. Let K be a convex cone. If x 1 ∈ K , x 2 ∈ K , …, x m ∈ K and α1 > 0, α2 > 0,…, αm > 0, then α1 x 1 + α2 x 2 + · · · + αm x m ∈ K .
Proof. By convexity,
α1 x 1 + α 2 x 2 + · · · + α m x m ∈ K. α1 + α 2 + · · · + α m
As K is a cone, we can multiply the point on the left hand side by α 1 +α2 +. . . + αm and stay in K . Let us consider two important examples of convex cones. L EMMA 2.20. Assume that X is a convex set. Then the set is a convex cone.
cone(X) = {γ x : x ∈ X, γ ≥ 0}
Proof. The set cone(X) is a cone, because for all its elements d = γ x and all β ≥ 0, we also have βd = (βγ )x ∈ cone(X). To prove that it is convex, let us consider d 1 = γ1 x 1 , x 1 ∈ X,
d 2 = γ2 x 2 , x 2 ∈ X,
d = αd 1 + (1 − α)d 2 ,
with α ∈ [0, 1]. If γ1 = 0 or γ2 = 0, then d is an element of cone(X) in a trivial way. We need to consider only the case with γ 1 > 0 and γ2 > 0. We have d = αγ1 x 1 + (1 − α)γ2 x 2 = (αγ1 + (1 − α)γ2 )
αγ1 x 1 + (1 − α)γ2 x 2 . αγ1 + (1 − α)γ2
The fraction on the right hand side, which we denote by x, is a convex combination of x 1 and x 2 and is therefore an element of X. Thus d = γ x with γ = αγ1 + (1 − α)γ2 , and therefore d ∈ cone(X).
27
ELEMENTS OF CONVEX ANALYSIS
The set cone(X) is called the cone generated by the set X. For a convex set X and a point x ∈ X the set M
KX (x) = cone(X − x) is called the cone of feasible directions of X at x (or the radial cone). It follows directly from the definition that it is a convex cone (see Figure 2.4).
x + KX (x)
X x Figure 2.4. The cone of feasible directions (translated to the point x). Example 2.21. Assume that the set X ⊂ n is a closed convex cone itself, and x ∈ X. Let us calculate the cone of feasible directions for X at x:
n
KX (x) = {d ∈
n
: d = τ (y − x), y ∈ X, τ ≥ 0}
= {d ∈ : d = h − τ x, h ∈ X, τ ≥ 0} = X − {τ x : τ ≥ 0} = X + {τ x : τ ∈ }.
In the last two equations we used the fact that X is a cone.
D EFINITION 2.22. Let X ⊂
m
be a convex set. The set
M
X ∞ = {d : X + d ⊂ X} is called the recession cone of X. We shall show that X ∞ is a convex cone. We first note that for each d ∈ X ∞ and for every m: X + md ⊂ X + (m − 1)d ⊂ · · · ⊂ X + d ⊂ X. Using convexity of X we infer that X + τ d ⊂ X for all τ ≥ 0. Hence τ d ∈ X ∞ for all τ ≥ 0. This means that X ∞ is a cone.
28
CHAPTER 2
The fact that X ∞ is convex can be verified directly from the definition. Indeed, if d 1 ∈ X ∞ and d 2 ∈ X ∞ , then x + (αd 1 + (1 − α)d 2 ) = α(x + d 1 ) + (1 − α)(x + d 2 ) ∈ X,
for all x ∈ X and all α ∈ (0, 1). D EFINITION 2.23. Let K be a cone in M
K ◦ = {y ∈
n
n
. The set
: hy, xi ≤ 0 for all x ∈ K }
is called the polar cone of K . For example, the set K = {x ∈ n : x i ≥ 0, i = 1, . . . , n} is a convex closed cone. Its polar is −K , as can be verified directly. Another example of a polar cone is shown in Figure 2.5. The negative of the polar cone, M K ∗ = −K ◦ , is called the dual cone. x K
y
K◦
Figure 2.5. The polar cone. Example 2.24. Let K 1 , . . . , K m be cones in n and let K = K 1 + · · · + K m . Clearly, K is a cone. We shall calculate its polar cone. If z ∈ K ◦ then for every x 1 ∈ K 1 , . . . , x m ∈ K m we have
hz, x 1 i + · · · + hz, x m i ≤ 0.
(2.4)
Let us choose j ∈ {1, . . . , n}. Setting all x i = 0 in (2.4), except for i = j , we conclude that hz, x j i ≤ 0 for all x j ∈ K j . Consequently, z ∈ K j◦ . As j was arbitrary,
K ◦ ⊂ K 1◦ ∩ · · · ∩ K m◦ .
29
ELEMENTS OF CONVEX ANALYSIS
On the other hand, for every element z of K 1◦ ∩ · · · ∩ K m◦ inequality (2.4) is satisfied, and thus z ∈ K ◦ . Therefore, (K 1 + · · · + K m )◦ = K 1◦ ∩ · · · ∩ K m◦ .
Directly from the definition we obtain the following properties. L EMMA 2.25. For every convex cone K ⊂
n
:
(i) The polar cone K ◦ is convex and closed;
(ii) K ◦ = (K )◦ .
We also have the following useful property.
L EMMA 2.26. Let K be a cone in n and let y ∈ n be such that the scalar product hy, xi is bounded from above for all x ∈ K . Then y ∈ K ◦ .
Proof. Suppose y ∈ / K ◦ . Then we can find x ∈ K such that hy, xi > 0. Since K is a cone, αx ∈ K for all α > 0. But then hy, αxi is not bounded from above, which contradicts our assumption. We use the notation K ◦◦ for the polar cone of the polar cone of K . The following result is known as the Bipolar Theorem. T HEOREM 2.27. If K ⊂
n
is a closed convex cone, then K ◦◦ = K.
Proof. If x ∈ K then hy, xi ≤ 0 for all y ∈ K ◦ , so x ∈ K ◦◦ . Hence K ⊂ K ◦◦ . We shall prove equality here. Suppose one can find z ∈ K ◦◦ \ K . Since K is closed, we can use Theorem 2.14 to strictly separate z and K . So, for some y 6 = 0 and ε > 0 we have hy, xi ≤ hy, zi − ε for all x ∈ K.
Thus the left hand side is bounded from above and Lemma 2.26 yields y ∈ K ◦ . On the other hand, setting x = 0 we get hy, zi ≥ ε, which contradicts our assumption that z ∈ K ◦◦ . Polar cones to pre-images of cones under linear transformations can be calculated in an explicit form. T HEOREM 2.28. Assume that C is a closed convex cone in m × n matrix. Let K = {x ∈
Then K is a closed convex cone and
n
: Ax ∈ C}.
K ◦ = {ATλ : λ ∈ C ◦ }.
m
and A is an
30
CHAPTER 2
Proof. K is a closed convex cone by construction. If z = A Tλ with λ ∈ C ◦ , then for every x ∈ K we have hx, zi = hx, ATλi = hAx, λi ≤ 0, because Ax ∈ C. Hence {ATλ : λ ∈ C ◦ } ⊂ K ◦ . The set on the left hand side is convex and closed. Suppose h ∈ K ◦ \ {ATλ : λ ∈ C ◦ }. Then we can strictly separate h and the set in question: there exist y 6 = 0 and ε > 0 such that hy, hi ≥ hy, ATλi + ε for all λ ∈ C ◦ .
(2.5)
Thus the product hy, ATλi = hAy, λi is bounded from above for all λ ∈ C ◦ . By Lemma 2.26, Ay ∈ C ◦◦ . Since C is closed, Theorem 2.27 implies that Ay ∈ C and therefore y ∈ K . Since h ∈ K ◦ , we obtain hy, hi ≤ 0. But then inequality (2.5) with λ = 0 yields a contradiction: 0 ≥ ε. Theorem 2.28 allows us to explicitly calculate polar cones of cones defined by systems of linear inequalities. Setting m
C = {y ∈
: yi ≤ 0, i = 1, . . . , m}
we obtain the following statement (known as Farkas Lemma). C OROLLARY 2.29. Let A be an m × n matrix and let K = {x ∈
n
: Ax ≤ 0}.
Then K ◦ = {y ∈
n
: y = ATλ, λ ∈
m
, λ ≥ 0}.
(2.6)
The above fact is frequently formulated as an alternative: exactly one of the following two systems has a solution, either (i) Ax ≤ 0 and hc, xi > 0; or (ii) c = ATλ, λ ≥ 0. Indeed, system (i) is not solvable if and only if c ∈ K ◦ , which is equivalent to (ii), by virtue of Corollary 2.29.
31
ELEMENTS OF CONVEX ANALYSIS
Example 2.30. Suppose we have n securities with present prices c 1 , . . . , cn . The securities can be either bought or sold short, that is, borrowed and sold for cash. Any amounts can be traded. At some future time one of m states may occur. The price of security j in state i will be equal to ai j , i = 1, . . . , m, j = 1, . . . , n. At this time we will liquidate our holding, that is, the securities held will be sold, and the short positions will be covered by purchasing and returning the amounts borrowed. An arbitrage, in the simplest version, is the existence of x¯ ∈ n such that hc, xi ¯ < 0 and A x¯ ≥ 0. Such a portfolio x¯ can be “purchased” with an immediate profit and then liquidated at a future point in time with no additional loss, whatever the future state. If arbitrage is present, considering portfolios M x, ¯ with M → ∞, we can increase the profits without any limits. Defining the convex cone
K = {x ∈
n
: Ax ≥ 0},
we see that the absence of arbitrage is equivalent to the fact that hc, xi ≥ 0 for all x ∈ K . The latter is nothing else but −c ∈ K ◦ . Corollary 2.29 implies that there exist state prices p ≥ 0 such that c = A T p.
(2.7)
The converse is obvious: if such state prices exist, then hc, xi = hA T p, xi = h p, Axi ≥ 0 for all x ∈
n
.
Suppose the first security (cash) has a constant unit price of 1 for all states: c 1 = 1 and ai1 = 1, i = 1, . . . , m. Then it follows from the first relation in (2.7) that m X i=1
pi = 1.
The vector p may be interpreted as a vector of implied probabilities of states 1, . . . , m. Similar considerations can be carried out for other definitions of arbitrage (see Example 2.33 below). Example 2.31. Consider the space Sn of symmetric matrices of dimension n × n. The set Sn+ of all positive semidefinite matrices is a cone in this space, as can be verified directly from the definition. In order to speak about the polar cone, we need to define the scalar product hA, BiS for symmetric matrices A and B. We set hA, BiS = tr(AB) =
n X n X
i=1 j =1
ai j bi j .
32
CHAPTER 2
This function is called the Frobenius inner product of matrices. We then have n ◦ S+ = {B ∈ Sn : tr(B A) ≤ 0, for all A ∈ Sn+ }. We shall prove that
Suppose B ∈ −Sn+ and A and let λ1 , . . . , λn be the
Sn+ .
n ◦ S+ = −Sn+ .
(2.8)
∈ Let z 1 , . . . , z n be eigenvectors of A having length 1, corresponding eigenvalues. We have the equation
n n X X tr(B A) = tr B λ j z j z Tj = λ j tr Bz j z Tj
=
n X j =1
j =1
j =1
λ j hz j , Bz j i.
Since the eigenvalues of A are nonnegative and B is negative semidefinite, we conclude that tr(B A) ≤ 0 for all A ∈ Sn+ . Thus n ◦ S+ ⊃ −Sn+ .
◦ To prove the converse inclusion, consider any B ∈ Sn+ . For an arbitrary z ∈ we can set A = zz T and get 0 ≥ tr(B A) = tr Bzz T = hz, Bzi.
n
Hence B is negative semidefinite and formula (2.8) holds true. 2.2.2 Separation of Cones
In Section 2.1.3 we considered separation of two disjoint convex sets. Clearly, convex cones can also be separated by the same principle. It follows from Theorem 2.16 that if K 1 and K 2 are convex cones, and K 1 ∩ K 2 = ∅, then there exists y 6 = 0 such that hy, x 1 i ≤ hy, x 2 i for all x 1 ∈ K 1 , x 2 ∈ K 2 . Using Lemma 2.26 we deduce that y ∈ K 1◦ and −y ∈ K 2◦ . Therefore Theorem 2.16 can be rephrased as follows: there exist y 1 ∈ K 1◦ and y 2 ∈ K 2◦ such that y 1 + y 2 = 0. In this algebraic formulation, the separation theorem for cones has a very useful generalization.
33
ELEMENTS OF CONVEX ANALYSIS
T HEOREM 2.32. Let K 1 , K 2 . . . , K m be convex cones in n . If K 1 ∩ K 2 ∩ · · · ∩ K m = ∅, then there exist y i ∈ K i◦ , i = 1, 2, . . . , m, not all equal 0, such that y 1 + y 2 + · · · + y m = 0. n
Proof. Let us define two cones in
×
n
× ··· ×
n
=
mn
:
C1 = {z = (z 1 , . . . , z m ) : z i ∈ K i , i = 1, 2, . . . , m}, C2 = {w = (x, . . . , x) : x ∈ n }.
T Since m i=1 K i = ∅, we have C 1 ∩ C 2 = ∅. Then by Theorem 2.16 we can find a nonzero y ∈ mn such that hy, wi ≥ hy, zi for all w = (x, . . . , x) and all z ∈ C 1 . Writing y = (y 1 , y 2 , . . . , y m ) we obtain hy 1 + y 2 + · · · + y m , xi ≥ hy 1 , z 1 i + hy 2 , z 2 i + · · · + hy m , z m i
(2.9)
for all x ∈ n and all z i ∈ K i , i = 1, 2, . . . , m. Setting x = 0, we see that the right hand side is bounded from above for all z i ∈ K i , i = 1, 2, . . . , m, which implies that each hy i , z i i is bounded from above for all z i ∈ K i . By Lemma 2.26, y i ∈ K i◦ . The left hand side of (2.9) is bounded from below for all x ∈ n , which is possible only when y 1 + y 2 + · · · + y m = 0. Example 2.33. Let us return to Example 2.30 and let us define a weak arbitrage as the existence of x¯ ∈ n such that hc, xi ¯ ≤ 0, and A x¯ ≥ 0, A x¯ 6= 0. In other words, portfolio x¯ can be purchased without any additional cash, and at a future point in time it can be sold with no loss, and with some profit for at least one state. Defining the convex cones
n
K 1 = {x ∈
K 2 = {x ∈
n
: hc, xi ≤ 0},
: Ax ≥ 0, Ax 6= 0},
we see that the absence of weak arbitrage is equivalent to the fact that K 1 ∩ K 2 = ∅. It follows that there exists y 6= 0 such that y ∈ K 1◦ and −y ∈ K 2◦ . The first polar cone is obvious: K 1◦ = {αc : α ≥ 0}. To calculate the second polar, we assume that K 2 6= ∅. Then K 2 = {x ∈
n
: Ax ≥ 0}.
34
CHAPTER 2
By Lemma 2.25, we have K 2◦ = (K 2 )◦ . Theorem 2.32 yields: (K 2 )◦ = {−A Tλ : λ ≥ 0}. Consequently, for some α ≥ 0 and a nonnegative vector λ we have y = αc = A Tλ.
Since y is nonzero, α > 0. Dividing by α and setting p = λ/α we conclude that there exists a vector of state prices p ∈ m , p ≥ 0, such that
c = A T p.
The converse statement can be proved as in Example 2.30. Again, if one of the securities is cash, then p may be interpreted as a vector of probabilities. Note that if K 2 = ∅ the existence of state prices is not guaranteed.
To use Theorem 2.32 we need to be able to calculate polar cones. The following technical lemma is useful. L EMMA 2.34. If x ∈ int K , then hy, xi < 0 for all nonzero y ∈ K ◦ .
Proof. Suppose hy, xi = 0 for some nonzero y ∈ K ◦ . Define z = x + εy. Since x ∈ int K , for sufficiently small ε > 0 we have z ∈ K and hy, zi > 0, a contradiction with y ∈ K ◦ . We are now in a position to present the general formula for a polar cone of an intersection. T HEOREM 2.35. Let K 1 , . . . , K m be convex cones in K 1 ∩ · · · ∩ K m . If K 1 ∩ int K 2 ∩ · · · ∩ int K m 6 = ∅, then
n
and let K =
K ◦ = K 1◦ + K 2◦ + · · · + K m◦ .
Proof. If y 1 ∈ K 1◦ , y 2 ∈ K 2◦ , …, y m ∈ K m◦ , then for all x ∈ K we get hy 1 + y 2 + · · · + y m , xi ≤ 0,
so K 1◦ + K 2◦ + · · · + K m◦ ⊂ K ◦ . We shall prove the inverse inclusion. Choose y ∈ K ◦ and define the cone C = {x : hx, yi > 0}. Clearly, C ∩ K = ∅, because hx, yi ≤ 0 for all x ∈ K . So, C ∩ K 1 ∩· · ·∩ K m = ∅ and by Theorem 2.32 we can find d ∈ C ◦ , y 1 ∈ K 1◦ , . . . , y m ∈ K m◦ , not all zero, such that d + y 1 + · · · + y m = 0.
(2.10)
35
ELEMENTS OF CONVEX ANALYSIS
Directly from the definition of C we see that d = −αy for some α ≥ 0. If α = 0, then d = 0. By assumption, there exists x ∈ K 1 ∩ int K 2 ∩ · · · ∩ int K m . Taking the scalar product of x and both sides of (2.10) we get hx, y 1 i + · · · + hx, y m i = 0.
All components on the left hand side are nonpositive, so hx, y i i = 0, i = 1, . . . , m. Since x ∈ int K i , i = 2, . . . , m, Lemma 2.34 implies that y i = 0, i = 2, . . . , m. Equation (2.10) then yields y 1 = 0, a contradiction. Thus α > 0. Dividing both sides of (2.10) by α and rearranging terms we get 1 1 1 1 y = − d = y 1 + y 2 + · · · + y m ∈ K 1◦ + K 2◦ + · · · + K m◦ . α α α α Recall that y was an arbitrary element of K ◦ . Therefore K ◦ ⊂ K 1◦ + K 2◦ + · · · + K m◦ , which completes the proof. If the cones K 1 , . . . , K m are polyhedral, the regularity assumption K 1 ∩ int K 2 ∩ · · · ∩ int K m 6 = ∅ is not needed; the result follows directly from Corollary 2.29 (see Exercise 2.7). Another application of Theorem 2.32 is the calculation of the polar to the cone K = {x ∈ K 1 : Ax ∈ K 2 },
(2.11)
which arises in a natural way in the analysis of optimality conditions (see Chapter 3). In (2.11) the sets K 1 ⊂ n and K 2 ⊂ m are closed convex cones and A is an m × n matrix. T HEOREM 2.36. Assume that K 1 and K 2 are closed convex cones, and K is defined by (2.11). If 0 ∈ int{Ax − y : x ∈ K 1 , y ∈ K 2 }, then K ◦ = K 1◦ + {ATλ : λ ∈ K 2◦ }. Proof. Define D = {x ∈
n
: Ax ∈ K 2 }. Clearly, K = K 1 ∩ D.
Hence K ◦ ⊃ K 1◦ + D ◦ .
(2.12)
36
CHAPTER 2
We shall show the equality here. Choose any v ∈ K ◦ and define the cone C = {x : hx, vi > 0}. We have C ∩ K = ∅, because hx, vi ≤ 0 for all x ∈ K . Thus, C ∩ K 1 ∩ D = ∅. By Theorem 2.32, we can find d ∈ C ◦ , w ∈ K 1◦ , z ∈ D ◦ , not all zero, such that d + w + z = 0.
(2.13)
Directly from the definition of C we see that d = −αv for some α ≥ 0. Suppose α = 0. Then (2.13) yields w = −z. Using Theorem 2.28 we see that z = ATλ, λ ∈ K 2◦ . It follows that −ATλ ∈ K 1◦ , where λ ∈ K 2◦ . Thus, for every x ∈ K 1 we obtain −hx, ATλi ≤ 0, that is, hAx, λi ≥ 0. As λ ∈ K 2◦ , we also have hy, λi ≤ 0 for all y ∈ K 2 . Hence hAx − y, λi ≥ 0 for all x ∈ K 1 and all y ∈ K 2 . Since the set of all Ax − y, where x ∈ K 1 and y ∈ K 2 , contains a neighborhood of 0 by assumption, we get λ = 0. This yields z = 0 and w = 0, a contradiction. Consequently, α > 0. Dividing (2.13) by α and rearranging terms we obtain v=
1 1 w + ATλ, w ∈ K 1◦ , λ ∈ K 2◦ . α α
Thus v ∈ K 1◦ + D ◦ , as required. Again, when all cones are polyhedral, the regularity assumption (2.12) is not needed.
37
ELEMENTS OF CONVEX ANALYSIS
2.2.3 Normal Cones
D EFINITION 2.37. Consider a convex closed set X ⊂ The set
n
and a point x ∈ X.
M
N X (x) = [cone(X − x)]◦ is called the normal cone to X at x. As a polar cone, the normal cone is closed and convex. It follows from the definition that v ∈ N X (x) if and only if hv, y − xi ≤ 0 for all y ∈ X.
(2.14)
Lemma 2.11 therefore implies the following characterization of the normal cone. L EMMA 2.38. Let X be a closed convex set and let x ∈ X. Then N X (x) = {v ∈
n
: 5 X (x + v) = x}.
This is illustrated in Figure 2.6.
x + N X (x)
X x
Figure 2.6. Normal cones (translated to the points at which they are calculated). Example 2.39. Suppose C is a closed convex cone in the projection x = 5C (z).
n and that z
Define y = z − x. It follows from Lemma 2.38 that ◦ y ∈ NC (x) = K C (x) . By Example 2.21, K C (x) = C + {τ x : τ ∈
n.
Consider
}. Example 2.24 then yields
NC (x) = C ◦ ∩ {y ∈
∈
n
: y ⊥ x}.
38
CHAPTER 2
We conclude that z = x + y,
y ∈ C ◦,
x ∈ C,
y ⊥ x.
By symmetry, y = 5C ◦ (z).
Normal cones to intersections can be calculated as sums of normal cones, but under an additional regularity assumption. L EMMA 2.40. Assume that X = X 1 ∩ · · · ∩ X m , where X i are closed convex sets, i = 1, . . . , m, and let x ∈ X. If X 1 ∩ int X 2 ∩ · · · ∩ int X m 6 = ∅, then N X (x) = N X 1 (x) + · · · + N X m (x). Proof. We have cone(X − x) = cone(X 1 − x) ∩ · · · ∩ cone(X m − x). By assumption, cone(X 1 − x) ∩ int cone(X 2 − x) ∩ · · · ∩ int cone(X m − x) 6 = ∅.
Applying Theorem 2.35, we obtain the required result.
The regularity assumption cannot be dropped, in general, as the following example shows. Example 2.41. Let X 1 = {x ∈
2
: kxk ≤ 1},
X 2 = {x ∈
2
: x 1 = 1},
and let x = (1, 0). We have N X 1 (x) = {v ∈
2
: v1 ≥ 0, v2 = 0},
N X 2 (x) = {v ∈
2
: v2 = 0}.
On the other hand X = X 1 ∩ X 2 contains just the point x, and N X (x) =
2
.
The operation x 7 → N X (x) is upper semicontinuous in the following sense.
L EMMA 2.42. Assume that X ⊂ n is a closed convex set and that a sequence {x k } of elements of X is convergent to a point x. ˆ Then for every convergent sequence of elements v k ∈ N X (x k ) its limit vˆ is an element of N X (x). ˆ
Proof. Take any y ∈ X. It follows from (2.14) at the point x k that hv k , y − x k i ≤ 0, k = 1, 2, . . . .
Passing to the limit in the above inequality we obtain (2.14) at x. ˆ
39
ELEMENTS OF CONVEX ANALYSIS
2.3 EXTREME POINTS
D EFINITION 2.43. A point x of a convex set X is called an extreme point of X if no other points x 1 ∈ X and x 2 ∈ X exist such that 1 1 1 2 x + x . 2 2 The notion of an extreme point is illustrated in Figure 2.7. x=
Figure 2.7. Extreme points of convex compact sets.
Extreme points of a compact convex set fully characterize the set. † T HEOREM 2.44. A convex and compact set in of the set of its extreme points.
n
is equal to the convex hull
Proof. The proof is by induction on the dimension of the space n. For n = 1 the theorem is true, because all compact convex sets in are bounded intervals [a, b] and their extreme points are the ends: a and b. Supposing that the theorem holds true for all k ≤ n we shall prove it for n + 1. Let X ⊂ n+1 be a convex, closed and bounded set. We may assume that int X 6 = ∅, because otherwise, by Lemma 2.9, we would be able to define a space of a smaller dimension containing X and our result would follow immediately. So, let p be an arbitrary point of X and let us take some x 0 ∈ int X which is different from p. Define d = p − x 0 and consider the line x 0 + τ d for τ ∈ . Since X is convex, closed and bounded, the line must cross the boundary of X at its two points: x 1 = x 0 + τ1 d and x 2 = x 0 + τ2 d with τ1 > 0 and τ2 < 0. Obviously, p is a convex combination of x 1 and x 2 . It suffices to show that both x 1 and x 2 are convex combinations of extreme points of X (see Figure 2.8). † Theorem 2.44 is known as Minkowski’s Theorem.
40
CHAPTER 2
S
x1 p
L x2
Figure 2.8. Construction of a convex combination of extreme points.
Let us focus on x 1 . Define the set S = {x = x 0 + τ d : τ > τ1 }. Clearly, X ∩ S = ∅. By Theorem 2.15, we can separate X and S by a plane defined by some y 6 = 0 such that hy, xi ≤ hy, x 0 + τ di for all x ∈ X and all τ > τ1 . Letting τ → τ1 we obtain hy, xi ≤ hy, x 1 i
(2.15)
for all x ∈ X. Let us define the hyperplane L = {x : hy, xi = hy, x 1 i}. Consider the set X 1 = X ∩L. It is a convex, closed and bounded set contained in a linear manifold L of dimension n and x 1 ∈ X 1 . So, by our assumption, x 1 is a convex combination of extreme points of X 1 . We shall show that extreme points of X 1 are also extreme points of X. Suppose it is not true and an extreme point v of X 1 is not an extreme point of X. Then v is a center of an interval with both ends in X. Since X lies on one side of L, the ends must be in L, hence in X 1 , a contradiction. Summing up, x 1 is a convex combination of extreme points of X. We treat 2 x in a similar way and conclude that our assertion is true for sets in n+1 . By induction, the result holds true for all n. The above result can be extended to unbounded convex sets with the use of the concept of the recession cone. L EMMA 2.45. A closed convex set X is bounded if and only if X ∞ = {0}.
41
ELEMENTS OF CONVEX ANALYSIS
Proof. If X is bounded, it cannot contain any rays, so X ∞ = {0}. It remains to prove the converse. Suppose X ∞ = {0}. If the set X is unbounded, there exists a sequence x 1 , x 2 , . . . of elements of X whose norms diverge. As the vectors y i = x i/kx i k have norm 1, they have an accumulation point y. Consider the point z = x + λy for some x ∈ X and some positive number λ. We construct the points λ λ xi. zi = 1 − i x + kx k kx i k
For i large enough, we have λ/kx i k ∈ (0, 1) and thus z i ∈ X. On the other hand, the points z i have an accumulation point z and the set X is closed. Thus, z ∈ X. This implies that y ∈ X ∞ , since the point x ∈ X and the number λ > 0 were arbitrary. We obtain a contradiction, which proves the statement. We are now ready to state our main result. We denote by E(X) the set of all extreme points of a set X. T HEOREM 2.46. A closed convex set X, which has at least one extreme point, can be represented as follows: X = conv E(X) + X ∞ . Proof. We first observe that the set X does not contain any line. Indeed, if it contains a line defined by some vector d, then d and −d belong to X ∞ . Then no point x can be extreme, because x = (x + d)/2 + (x − d)/2. We can now proceed as in the proof of Theorem 2.44 with the following modification. Taking a line through the point x 0 , we may obtain either two boundary points x 1 and x 2 , as before, or one boundary point (say x 1 ) and a vector (e.g., −d) from the recession cone X ∞ . This result can be further refined with the use of the concept of an extreme ray of the recession cone (see Exercise 2.3). Our results are important for linear programming. T HEOREM 2.47. Let A be an m × n matrix and let the set X ⊂ defined as X = {x ∈
n
n
be
: Ax = b, x ≥ 0}.
A point x is an extreme point of X if and only if the columns of A that correspond to positive components of x are linearly independent.
42
CHAPTER 2
Proof. Let J = { j : x j > 0} and let the columns a j , j ∈ J , be linearly dependent. Then one can find γ j , j ∈ J , not all equal to 0, such that P j 1 2 1 j ∈J γ j a = 0. Let us define two points x and x by setting x j = x j + τ γ j and x j2 = x j − τ γ j for j ∈ J , and x j1 = x j2 = x j for j ∈ / J . Then for 1 2 sufficiently small τ > 0 both x and x belong to X and x = 12 x 1 + 21 x 2 . Therefore x cannot be an extreme point of X. To prove the inverse implication, consider a point x which is not extreme. Then we can find two different points x 1 and x 2 in X such that x = 21 x 1 + 21 x 2 . Let J be the set of indices for which x j1 6 = x j2 . Clearly, J is nonempty and x j > 0 for j ∈ J . Since Ax 1 = Ax 2 = b, we have X (x j1 − x j2 )a j = 0, j ∈J
which proves that the columns a j , j ∈ J are linearly dependent. Solutions of the system Ax = b whose nonzero components correspond to linearly independent columns of A, are called in linear programming basic solutions. If they are nonnegative, they are called basic feasible solutions. Their role is now evident. T HEOREM 2.48. If the set X = {x ∈ n : Ax = b, x ≥ 0} is bounded, it is the convex hull of the set of basic feasible solutions. Proof. The result follows immediately from Theorems 2.44 and 2.47. We use this observation in Section 2.4. Extreme points play a crucial role in the analysis of the problem of moments in the theory of probability. We present a simplified version of this problem. Example 2.49. Consider the set S of all random variables with realizations in the set T = {t1 , t2 , . . . , tn } ⊂ . We know the first m moments of a certain random variable X ∈ S: µk = E X k , k = 1, . . . , m,
where m < n. We shall prove that there exists a random variable Y ∈ S with identical values of its first m moments and such that no more than m + 1 realizations from the set T have positive probability. Define the set of possible probability distributions on T : P = {p ∈
n
:
n X
j =1
p j = 1, p j ≥ 0, j = 1, . . . , n}.
43
ELEMENTS OF CONVEX ANALYSIS
Let
t1 t2 1 A= . ..
t2 t22 .. .
t1m
The set
t2m
tn tn2 .. . .
... ...
tnm
...
M = {Ap : p ∈ P} is the collection of all m-tuples of moments of random variables having realizations in T . The set M is convex and compact, and µ = (µ1 , . . . , µm ) is its element. Therefore there exist m + 1 extreme points z i of M such that µ=
m+1 X
αi z i ,
i=1
m+1 X i=1
αi = 1, αi ≥ 0, i = 1, . . . , m + 1.
We shall show that each z i can be represented as z i = Ae j (i) , where e j is the j th unit vector in n . Consider a fixed i . We have z i = Api , where pi ∈ P. Writing
pi =
n X
pij e j
j =1
we see that zi =
n X
pij Ae j ,
j =1
n X j =1
pij = 1,
pij ≥ 0,
j = 1, . . . , n.
It follows that z i is a convex combination of points Ae j , j = 1, . . . , m. Since z i is an extreme point of M, this may happen only if z i = Ae j whenever p ij > 0. We can, therefore, select one index j (i ) with p ij (i) > 0, and conclude that z i = Ae j (i). This can be done for all i and thus µ=
m+1 X i=1
αi Ae j (i) = A
m+1 X
αi e j (i) .
i=1
It follows that µ = A p, ˆ where the vector of probabilities pˆ is given by pˆ k =
X
{i: j (i)=k}
αi , k = 1, . . . , n.
There can be no more than m + 1 different indices j (i ), when i = 1, . . . , m + 1. Thus no more than m + 1 of probabilities pˆ k are positive.
44
CHAPTER 2
Another way to obtain this conclusion is to consider the set S of all vectors p ∈ such that
n
Ap = µ,
n X j =1
p j = 1, p ≥ 0.
By Theorem 2.48 its extreme points have no more than m + 1 positive components. 2.4 CONVEX FUNCTIONS 2.4.1 Basic Concepts
In our analysis it is convenient to consider functions which may take, in addition to real values, two special values: −∞ and +∞. The real line augmented with these special values will be denoted by . With every function f : n → we can associate two sets: the domain M
dom f = {x : f (x) < +∞} and the epigraph M
epi f = {(x, v) ∈
n
×
: v ≥ f (x)}.
D EFINITION 2.50. A function f is called convex if epi f is a convex set. An example of a convex function, x ln(x) − x if x > 0, f (x) = 0 if x = 0, +∞ if x < 0.
is shown in Figure 2.9.
D EFINITION 2.51. A function f is called concave if − f is convex.
D EFINITION 2.52. A function f is called proper if f (x) > −∞ for all x and f (x) < +∞ for at least one x. From now on we will always deal with proper convex functions.
L EMMA 2.53. A function f is convex if and only if for all x 1 and x 2 and for all 0 ≤ α ≤ 1 we have f (αx 1 + (1 − α)x 2 ) ≤ α f (x 1 ) + (1 − α) f (x 2 ).
(2.16)
45
ELEMENTS OF CONVEX ANALYSIS
4
epigraph
2
0
-2
0
2
4
Figure 2.9. Example of a convex function.
Proof. If x 1 ∈ / dom f or x 2 ∈ / dom f , the above inequality is satisfied trivially. It remains to consider the case when x 1 ∈ dom f and x 2 ∈ dom f . Then the points 1 2 x x and f (x 1 ) f (x 2 ) belong to epi f . If f is convex, then their convex combination is in the epigraph: αx 1 + (1 − α)x 2 ∈ epi f, (2.17) α f (x 1 ) + (1 − α) f (x 2 ) which implies (2.16). On the other hand, (2.16) entails (2.17), which in turn yields the convexity of the epigraph. Inequality (2.16) can be used as an alternative definition of proper convex functions. Example 2.54. The function
f (x) = kxk♦ ,
where k · k♦ is a norm in n , is a convex proper function. Indeed, using the triangle inequality for the norm, for all x, y and α ∈ [0, 1] we obtain
kαx + (1 − α)yk♦ ≤ kαxk♦ + k(1 − α)yk♦ = αkxk♦ + (1 − α)kyk♦ . In the last equation we also used the fact that a norm is positively homogeneous.
46
CHAPTER 2
Example 2.55. Assume Z is a closed convex set in
n.
The distance to Z ,
f (x) = min kx − zk♦ , z∈Z
where k · k♦ is a norm in n , is a convex proper function. Indeed, consider two points, x and y, and any α ∈ (0, 1). Define the points v ∈ Z and w ∈ Z such that
f (x) = kx − vk♦ ,
f (y) = ky − wk♦ .
They exist, because Z is a closed set. In the particular case when k · k ♦ is the Euclidean norm, v and w are the orthogonal projections of x and y on Z . By the convexity of Z , the convex combination of these points, αv + (1 − α)w, with α ∈ (0, 1), is an element of Z as well. Therefore, f (αx + (1 − α)y) = min kαx + (1 − α)y − zk♦ z∈Z
≤ kαx + (1 − α)y − [αv + (1 − α)w]k♦
≤ αkx − vk♦ + (1 − α)ky − wk♦ = α f (x) + (1 − α) f (y).
In the last inequality we used the properties of the norm. The convexity of the set Z is relevant here; the distance to a nonconvex set is not a convex function.
D EFINITION 2.56. A function f is called strictly convex if inequality (2.16) is strict for all x 1 6 = x 2 and for all 0 < α < 1. L EMMA 2.57. If f is convex, then dom f is a convex set. Proof. If x 1 ∈ dom f and x 2 ∈ dom f , then, by Lemma 2.53, f (αx 1 + (1 − α)x 2 ) < +∞. L EMMA 2.58. If f i , i ∈ I , is a family of convex functions, then f (x) = sup f i (x) i∈I
is convex. Proof. Noting that epi f =
\
epi f i
i∈I
we obtain the required result from Lemma 2.2.
47
ELEMENTS OF CONVEX ANALYSIS
Example 2.59. For a symmetric matrix A, its maximum eigenvalue λ max (A) is well defined. We can consider, therefore, the function f (A) = λmax (A), defined on the space Sn of symmetric matrices of dimension n × n. As λmax (A) = max hy, Ayi, kyk=1
and each function f y (A) = hy, Ayi is linear, the function λmax (·) is convex.
L EMMA 2.60. If f is a convex function, then for all x 1 , x 2 , . . . , x m and all α1 ≥ 0, α2 ≥ 0, . . . , αm ≥ 0 such that α1 + α2 + · · · + αm = 1, one has f (α1 x 1 + α2 x 2 + · · · + αm x m ) ≤ α1 f (x 1 ) + α2 f (x 2 ) + · · · + αm f (x m ). Proof. The result follows from the convexity of epi f : the points i x , i = 1, 2, . . . , m, f (x i ) belong to the epigraph, so their convex combination is in epi f , too. L EMMA 2.61. If the functions f i , i = 1, 2, . . . , m, are convex, then for all c1 ≥ 0, c2 ≥ 0, . . . , cm ≥ 0 the function f (x) = c1 f 1 (x) + c2 f 2 (x) + · · · + cm f m (x) is convex. Proof. Since (2.16) holds true for each f i , we can multiply these inequalities by ci ≥ 0 and sum up to get (2.16) for f . The epigraph of a function f : n → can be used to characterize continuity properties of the function, irrespective of its convexity. A function f : n → is called lower semicontinuous, if for every convergent sequence of points {x k } we have f lim x k ≤ lim inf f (x k ). k→∞
L EMMA 2.62. A function f : if its epigraph is a closed set.
k→∞
n
→
is lower semicontinuous if and only
48
CHAPTER 2
Proof. Consider a sequence {(x k , α k )} of points of epi f , and suppose x k → x and α k → α, as k → ∞. If f is lower semicontinuous, then f (x) ≤ lim inf f (x k ) ≤ lim α k = α, k→∞
k→∞
which proves that (x, α) ∈ epi f . Suppose the epigraph is closed, but f is not lower semicontinuous. Then there exists a sequence {x k } ⊂ n convergent to some point x ∈ n such that f (x) > lim f (x k ), k→∞
where the limit on the right hand side may be −∞, in general. Then there exists ε > 0 such that f (x k ) < f (x) − ε for all sufficiently large k. Hence (x k , f (x) − ε) ∈ epi f for all sufficiently large k. Since the epigraph is closed, the limit of these points, (x, f (x) − ε), is an element of the epigraph of f as well. This means that f (x) − ε ≥ f (x), a contradiction. Thus f must be lower semicontinuous. L EMMA 2.63. If f :
n
→
is convex, then for each β ∈
Mβ = {x : f (x) ≤ β}
the set (2.18)
is convex. If, in addition, f is lower semicontinuous, then the set M β is closed for all β. Proof. If x ∈ Mβ and y ∈ Mβ , then, by Lemma 2.53, f (αx + (1 − α)y) ≤ α f (x) + (1 − α) f (y) ≤ β, so αx + (1 − α)y ∈ Mβ . If f is lower semicontinuous, its epigraph is closed (Lemma 2.62). Consider the set in n × : Mβ × {β} = epi f ∩ {(x, β) : x ∈
n
}.
It is closed, because epi f is closed. Thus M β is closed. The set Mβ in the last lemma is called the level set of f . Not every √ function that has convex level sets is convex, as the example of f (x) = |x| clearly demonstrates.
49
ELEMENTS OF CONVEX ANALYSIS
L EMMA 2.64. Let X ⊂ n be a convex set and let f : n → be a convex function. Then the set Xˆ of solutions of the optimization problem minimize f (x) x∈X
(2.19)
is convex. Proof. If (2.19) has no solutions, Xˆ is convex, because it is empty. Let ˆ β = f (x). Xˆ 6 = ∅ and let xˆ ∈ X, ˆ Then Xˆ = X ∩ Mβ with Mβ defined by (2.18). By Lemma 2.2, Xˆ is convex. The maxima of convex functions in convex sets can be characterized as well. T HEOREM 2.65. Let f : n → be a convex function and let X ⊂ dom f be a convex, closed and bounded set. Then the set of solutions of the problem maximize f (x) x∈X
(2.20)
contains at least one extreme point of X. If, in addition, the function f (·) is affine, then the set of solutions of (2.20) is the convex hull of the set of the extreme points of X that are solutions of (2.20). Proof. Let xˆ be a solution of (2.20). By Theorem 2.44 we can find extreme points x 1 , x 2 , . . . , x m of X such that xˆ = α1 x 1 + α2 x 2 + · · · + αm x m with some α1 > 0, α2 > 0, . . . , αm > 0, α1 + α2 + · · · + αm = 1. By Lemma 2.60, f (x) ˆ ≤ This can be rewritten as m X i=1
m X
αi f (x i ).
i=1
αi f (x) ˆ − f (x i ) = 0.
(2.21)
Because xˆ is optimal, f (x) ˆ ≥ f (x i ), i = 1, . . . , m. Then it follows from i (2.21) that f (x) ˆ = f (x ), i = 1, . . . , m. All points x i are optimal as well. We see that the set of solutions of (2.20) is included in the convex hull of the extreme points that are solutions of (2.20).
50
CHAPTER 2
If the function f (·) is affine then − f (·) is convex. The set of solutions of problem (2.20) is the same as the set of minima of − f (x) over x ∈ X. It is convex, by virtue of Lemma 2.64. Therefore the convex hull of the the extreme points that are solutions of (2.20) is included in the set of all solutions of (2.20). We can now return to the linear programming problem, which we analyzed at the end of Section 2.3. T HEOREM 2.66. If the feasible set of the linear programming problem minimize hc, xi subject to Ax = b, x ≥ 0,
(2.22)
is bounded, then the set of optimal solutions is the convex hull of the set of optimal basic feasible solutions. Proof. Let us define f (x) = −hc, xi. It is an affine function and each solution of (2.22) solves (2.20) with X = {x : Ax = b, x ≥ 0}. The result follows directly from Theorem 2.65. The above fact is used by the simplex method for solving linear programming problems. It moves from one basic feasible solution to a better one, as long as progress is possible. The best basic feasible solution is guaranteed to be optimal. It can be found after finitely many steps if a solution exists. If the set is unbounded, we may discover a ray from the recession cone, along which the objective can be decreased without limits. In this case no optimal solution exists. 2.4.2 Smooth Convex Functions
We now formulate convexity criteria for smooth functions. We denote by ∇ f (x) the gradient of the function f at x, ∂ f (x) ∂x
1 ∂ f (x) M ∂ x2 ∇ f (x) = . . ..
∂ f (x) ∂ xn
Here x 1 , x 2 , . . . , x n denote the coordinates of the vector x.
51
ELEMENTS OF CONVEX ANALYSIS
If f is twice continuously differentiable, ∇ 2 f (x) denotes the Hessian of f at x, ∂ 2 f (x) ∂ 2 f (x) 2 · · · ∂∂ x1f∂(x) ∂ x1 ∂ x2 xn ∂ x 12 ∂ 2 f (x) ∂ 2 f (x) ∂ 2 f (x) · · · 2 M ∂x ∂x ∂ x2 ∂ xn ∂ x2 . ∇ 2 f (x) = 2. 1 .. .. .. . . . . . ∂ 2 f (x) ∂ 2 f (x) ∂ 2 f (x) · · · ∂ xn ∂ x1 ∂ xn ∂ x2 ∂x2 n
We use the symbol C 2 to denote the set of twice continuously differentiable real valued functions. For f ∈ C 2 the Hessian is a symmetric matrix, because the values of the mixed derivatives do not depend on the order of differentiation. T HEOREM 2.67. Assume that a function f is continuously differentiable. Then (i) f is convex if and only if for all x and y f (y) ≥ f (x) + h∇ f (x), y − xi;
(2.23)
(ii) f is strictly convex if and only if for all x 6 = y f (y) > f (x) + h∇ f (x), y − xi.
(2.24)
Proof. (i) Suppose f is convex, but for some x and y f (y) ≤ f (x) + h∇ f (x), y − xi − ε with ε > 0. Let us consider z = αy + (1 − α)x for 0 < α < 1. By Lemma 2.53, f (z) ≤ α f (y) + (1 − α) f (x) ≤ f (x) + αh∇ f (x), y − xi − αε. Rearranging terms and dividing by α we get f (z) − f (x) ≤ h∇ f (x), y − xi − ε. α
(2.25)
Let α ↓ 0. Since z = x +αd with d = y − x, the left side of (2.25) converges to the directional derivative at x in the direction d f 0 (x; d) = h∇ f (x), di and we obtain a contradiction in (2.25).
52
CHAPTER 2
To prove the opposite implication, let us assume (2.23). Let y and z be arbitrary points, y 6 = z, and let x = αy + (1 − α)z with α ∈ (0, 1). Then, by (2.23) we have f (y) ≥ f (x) + h∇ f (x), y − xi, f (z) ≥ f (x) + h∇ f (x), z − xi. Multiplying these inequalities by α and 1 − α, respectively, and adding them one gets α f (y) + (1 − α) f (z) ≥ f (x), which was what we set out to prove. (ii) If f is strictly convex, then f is convex and (2.23) is true. We shall prove that the inequality in (2.23) is strict, if y 6 = x and α ∈ (0, 1). Suppose equality in (2.23). Let z = 12 x + 12 y. Then by strict convexity of f and by equality in (2.23) f (z) <
1 1 1 f (x) + f (y) = f (x) + h∇ f (x), y − xi. 2 2 2
(2.26)
Let v = βx + (1 − β)z with 0 < β < 1. Then, by convexity and by (2.26) 1 f (v) < β f (x) + (1 − β) f (z) < f (x) + (1 − β)h∇ f (x), y − xi. 2
Since v − x = (1 − β)(z − x) = 12 (1 − β)(y − x), the last inequality reads f (v) < f (x) + h∇ f (x), v − xi, which contradicts (2.23). To prove that (2.24) implies strict convexity, let us simply note that the argument used in case (i) applies again, but with sharp inequalities. It follows from the proof of the above theorem that if f : n → convex and differentiable at the point x then (2.23) holds true for all y ∈ If f is strictly convex, (2.24) remains valid as well, for all y ∈ n . Example 2.68. Consider the function f :
n
→
is .
n
defined as the quadratic form,
f (x) = hx, Axi, where A is a symmetric matrix. The function f is convex if and only if A is a positive semidefinite matrix, and f is strictly convex if and only if A is a positive definite matrix. Indeed, ∇ f (x) = 2Ax,
53
ELEMENTS OF CONVEX ANALYSIS
and for all x and all y we have the equation f (y) − f (x) − h∇ f (x), y − xi = hy, Ayi − hx, Axi − 2hAx, y − xi = hy, Ayi + hx, Axi − 2hAx, yi = hy − x, A(y − x)i.
The expression on the right hand side is nonnegative for all x and y if and only if A is positive semidefinite. This expression is positive for all y 6= x if and only if A is positive definite. Examples of quadratic functions in 2 corresponding to a positive definite matrix and to an indefinite matrix are illustrated in Figure 2.10.
Figure 2.10. Convex and nonconvex quadratic functions.
The theorem below uses the observation of Example 2.68 in a more general setting. T HEOREM 2.69. Assume that f : tiable. Then
n
→
is twice continuously differen-
(i) f is convex if and only if its Hessian ∇ 2 f (x) is positive semidefinite for all x ∈ n ; (ii) if the Hessian ∇ 2 f (x) is positive definite for all x ∈ strictly convex.
n
then f is
Proof. If f ∈ C 2 , then for all x and all y we have
1 f (y) = f (x) + h∇ f (x), y − xi + hy − x, ∇ 2 f (x θ )(y − x)i, 2
(2.27)
where x θ = x + θ(y − x) with some 0 ≤ θ ≤ 1. If the Hessian ∇ 2 f (x) is positive semidefinite for all x, then the quadratic term in (2.27) is nonnegative
54
CHAPTER 2
and we obtain (2.23). If the Hessian is positive definite, then for y 6 = x the quadratic term is positive and we get (2.24). Suppose the Hessian is not positive semidefinite for some x. Then there exists d such that hd, ∇ 2 f (x)di < 0. Let y = x + εd for some ε > 0. If ε is small enough, then y and x θ are so close to x that hd, ∇ 2 f (x θ )di < 0, because the Hessian is continuous. But then the quadratic term in (2.27) is negative and we obtain a contradiction with (2.23). Note that statement (ii) of the theorem does not have the “only if” part, which appeared in the quadratic case. For example, the function f (x) = x 4 is strictly convex, but its second derivative vanishes at 0. 2.4.3 Directional Derivatives
So far we have discussed convex functions that are at least once continuously differentiable. In many applications one has to deal with functions that are nonsmooth. For example the Euclidean norm kxk =
n X j =1
x j2
1/2
is not differentiable at 0. In fact, no norm is differentiable at 0. Some, like kxk1 =
n X j =1
|x j | or kxk∞ = max |x j |, 1≤ j ≤n
are nondifferentiable at many points (see Figure 2.11). Nonsmooth functions
Figure 2.11. The norms k · k, k · k1 and k · k∞ .
55
ELEMENTS OF CONVEX ANALYSIS
are very common in optimization models. The concept of the gradient of a smooth function can be generalized to the case of nonsmooth functions, and in particular: convex nonsmooth functions. To understand this construction we must first prove several important properties of convex functions. be a convex function. For every x ∈ L EMMA 2.70. Let f : n → int dom f there exist δ > 0 and L such that | f (y) − f (x)| ≤ Lky − xk whenever ky − xk < δ. Proof. Since x ∈ int dom f , we can choose a small simplex S about x such that it is included in the domain of f . Every point y of the simplex is a convex combination of its vertices v 1 , . . . , v n+1 with some nonnegative coefficients α1 , . . . , αn+1 totaling 1. By the convexity of f , f (y) ≤
n+1 X
αi f (v i ) ≤ max f (v i ) = M. 1≤i≤n+1
i=1
Let ε > 0 be the lower bound on the distance from x to the boundary of S. Suppose ky − xk < ε/2. We can represent y as a convex combination of x and a point w at which the line passing through x and y intersects the boundary of S (we choose the intersection point that is closer to y). We have y = (1 − α)x + αw, with α= By the convexity of f
ky − xk ky − xk ≤ . kw − xk ε
f (y) − f (x) ≤ α[ f (w) − f (x)] ≤ α[M − f (x)] ≤
M − f (x) ky − xk. ε
(2.28)
We can also represent x as a convex combination of y and the point z at which the same line crosses the boundary of S on the other side: x = (1 − β)y + βz with β=
ky − xk ky − xk ≤ . ky − zk ε
56
CHAPTER 2
By the convexity of f , f (x) ≤ (1 − β) f (y) + β f (z) ≤ (1 − β) f (y) + β M. Hence f (y) − f (x) ≥
2( f (x) − M) ky − xk. ε
Combining the last inequality with (2.28) we conclude that | f (y) − f (x)| ≤
2(M − f (x)) ky − xk ε
whenever ky − xk < ε/2.
The required inequality is satisfied with L = 2(M − f (x))/ε. Let f : n → be a convex function and let x ∈ dom f . Then for every d ∈ n the quantity f 0 (x; d) = lim τ ↓0
f (x + τ d) − f (x) , τ
(2.29)
is called the directional derivative of f at x in the direction d. L EMMA 2.71. For every x ∈ dom f and every d ∈ n the limit in (2.29) exists (finite or infinite). If x ∈ int dom f , then f 0 (x; d) is finite for all d. Proof. Consider the differential quotient Q(τ ) =
f (x + τ d) − f (x) . τ
If f (x + τ d) = +∞ for all τ > 0, we have Q(τ ) = +∞ for all τ > 0 and f 0 (x; d) = +∞. If f (x + τ0 d) < +∞ for some τ0 > 0, the convexity of f implies that f (x + τ d) < +∞ for all 0 < τ < τ 0 , and Q(τ ) is well defined for these τ . Let 0 < τ1 < τ2 < τ0 . We have τ1 τ1 x + (x + τ2 d). x + τ1 d = 1 − τ2 τ2 The convexity of f (·) renders τ1 τ1 f (x) + f (x + τ2 d), f (x + τ1 d) ≤ 1 − τ2 τ2
which can be rewritten as
f (x + τ1 d) − f (x) ≤
τ1 [ f (x + τ2 d) − f (x)]. τ2
57
ELEMENTS OF CONVEX ANALYSIS
Dividing by τ1 we see that the differential quotients are monotone: Q(τ1 ) ≤ Q(τ2 ) for all 0 < τ1 ≤ τ2 .
(2.30)
Therefore the limit in (2.29) exists (finite or equal to −∞). The monotonicity of Q(·) implies that f 0 (x; d) ≤ Q(τ ) for all τ > 0.
(2.31)
If x ∈ int dom f , Lemma 2.70 implies that for all sufficiently small τ |Q(τ )| =
| f (x + τ d) − f (x)| ≤ Lkdk, τ
and therefore the limit of Q(τ ) for τ ↓ 0 must be finite. 2.5 SUBDIFFERENTIAL CALCULUS 2.5.1 Subgradients and Subdifferentials
For general convex functions the concept of the gradient is generalized to the notion of a subgradient. It is defined by an inequality similar to inequality (2.23) which characterizes smooth convex functions. D EFINITION 2.72. Let f : n → be a proper convex function and let x ∈ dom f . A vector g ∈ n such that f (y) ≥ f (x) + hg, y − xi for all y ∈
n
(2.32)
is called a subgradient of f at x. The set of all subgradients of f at x is called the subdifferential of f at x and is denoted by ∂ f (x). The notion of the subgradient has a useful geometrical interpretation. Suppose g ∈ ∂ f (x). Inequality (2.32) means that the epigraph of f is located on or above the graph of the affine function l(y) = f (x) + hg, y − xi. This is illustrated in Figure 2.12. For every point (y, v) ∈ epi f we have v ≥ f (y) ≥ f (x) + hg, y − xi, which can be rewritten as hg, y − xi + (−1)(v − f (x)) ≤ 0. Consequently, (g, −1) is an element of the normal cone N epi f (x, f (x)).
58
CHAPTER 2
4 3 2 1 0
(g, −1)
-1 -2 -3
-2
-1
0
1
2
4
3
Figure 2.12. A subgradient.
The reverse is also true, under some conditions. If a vector (u, γ ) ∈ Nepi f (x, f (x)) and γ 6 = 0, then g = −u/γ is a subgradient of f at x. The condition that γ 6 = 0 is essential here. Consider the convex function shown in Figure 2.9 on page 45. At the point (0,0) all normals to the epigraph have the form (u, 0), where u < 0, and f has no subgradient at x = 0. Subgradients, if they exist, provide lower bounds for directional derivatives. L EMMA 2.73. Assume that f : n → is a proper convex function and let x ∈ dom f . A vector g is a subgradient of f at x if and only if f 0 (x; d) ≥ hg, di for every d ∈
n
.
(2.33)
Proof. Suppose (2.33) is true. Then for every y, by (2.31), f (y) ≥ f (x) + f 0 (x; y − x) ≥ f (x) + hg, y − xi, which shows that g is a subgradient. To prove the converse, suppose g ∈ ∂ f (x). Then for every d and every τ > 0 the subgradient inequality (2.32) yields hg, τ di f (x + τ d) − f (x) ≥ = hg, di. τ τ Passing to the limit with τ ↓ 0 we get (2.33). We are now in a position to resolve the question of the existence of subgradients and to fully characterize the directional derivatives.
59
ELEMENTS OF CONVEX ANALYSIS
T HEOREM 2.74. Let f : n → be a convex function. Assume that x ∈ int dom f . Then ∂ f (x) is a nonempty, convex, closed and bounded set. Furthermore, for every direction d ∈ n one has f 0 (x; d) = max hg, di. g∈∂ f (x)
Proof. In view of Lemma 2.73, it is sufficient to show that for every d there exists g ∈ ∂ f (x) such that f 0 (x; d) = hg, di. Consider two sets in
n+1
(2.34)
:
E = {(y, v) : v > f (y)}, and L = {(x + τ d, f (x) + τ f 0 (x; d)) : τ ∈
}.
Since f (x + τ d) ≥ f (x) + τ f 0 (x; d) for all τ ∈
,
these sets have no common points. They are convex and therefore can be separated by a plane (see Figure 2.13).
Figure 2.13. The set E and line L can be separated.
By Theorem 2.15, there exists a nonzero u z= γ
60
CHAPTER 2
such that for every point (y, v) ∈ E and every τ ∈
we have
hu, yi + γ v ≥ hu, x + τ di + γ [ f (x) + τ f 0 (x; d)].
(2.35)
We deduce that γ ≥ 0, for otherwise letting v → ∞ leads to a contradiction. Suppose γ = 0. As x is an interior point of the domain, we can choose y from a small ball B about x in such a way that there exists v > f (y). Setting τ = 0 in (2.35), we get hu, yi ≥ hu, xi for all y ∈ B, which is possible only when u = 0 and contradicts the condition that z 6 = 0. Thus γ > 0. Dividing both sides of (2.35) by γ , setting g = −u/γ and letting v ↓ f (y) we obtain: f (y) − hg, yi ≥ f (x) + τ f 0 (x; d) − hg, x + τ di for all y ∈ int dom f and all τ ∈ . It follows from this inequality that τ f 0 (x; d) − hg, di ≤ f (y) − f (x) − hg, y − xi,
for all τ ∈ . This is possible only if the coefficient multiplied by τ is zero, and thus (2.34) holds true. It remains to show that g is a subgradient. Setting τ = 0 we conclude that f (y) ≥ f (x) + hg, y − xi, for all y ∈ int dom f.
(2.36)
By convexity, for every y ∈ dom f , f (y) − f (x) ≥ 2[ f ((x + y)/2) − f (x)], and (x + y)/2 ∈ int dom f . Applying (2.36) to f ((x + y)/2) we see that (2.32) holds true for all y ∈ dom f , and thus for all y ∈ n . Therefore g is a subgradient of f at x and the subdifferential is nonempty. Suppose g 1 ∈ ∂ f (x) and g 2 ∈ ∂ f (x). Then for all y f (y) ≥ f (x) + hg 1 , y − xi,
f (y) ≥ f (x) + hg 2 , y − xi.
Adding these inequalities multiplied by α and by 1−α, where α ∈ (0, 1), we conclude that αg 1 + (1 − α)g 2 ∈ ∂ f (x). Thus the subdifferential is convex. If g k ∈ ∂ f (x) and g k → g, then we can pass to the limit in the inequality f (y) ≥ f (x) + hg k , y − xi,
to conclude that g ∈ ∂ f (x). Therefore the subdifferential is closed. Let us prove the boundedness. Let g ∈ ∂ f (x). By Lemma 2.36 for x + τ d sufficiently close to x we have f (x + τ d) − f (x) ≤ τ Lkdk.
61
ELEMENTS OF CONVEX ANALYSIS
Hence f 0 (x; d) ≤ Lkdk, for all d. Using Lemma 2.73 we obtain: hg, di ≤ f 0 (x; d) ≤ Lkdk, for all d, which implies kgk ≤ L. It follows from the proof that the subdifferential ∂ f (x) is a convex and closed set at every point x at which f (·) has at least one subgradient (is subdifferentiable). The formula for the directional derivative can be generalized to boundary points at which f is subdifferentiable. L EMMA 2.75. If a convex function f : point x, then for every d
n
→
is subdifferentiable at a
f 0 (x; d) = sup hg, di. g∈∂ f (x)
Moreover, if f 0 (x; d) < ∞ then the supremum above is attained. Proof. By Lemma 2.71, the directional derivative f 0 (x; d) exists (finite or infinite). Also, from the definition of the subdifferential it follows that for each g ∈ ∂ f (x) we have f 0 (x; d) = lim τ ↓0
f (x + τ d) − f (x) ≥ hg, di. τ
Suppose d ∈ n and ε > 0 exist such that f 0 (x; d) > hg, di + ε for all g ∈ ∂ f (x). Then µ = sup hg, di + < ∞. g∈∂ f (x)
We can define the line L = {(x + τ d, f (x) + τ µ) : τ ∈
}.
Proceeding exactly as in the proof of Theorem 2.74 we can separate the line L and the set E = {(y, v) ∈
n+1
: v > f (y)}
and find a new subgradient g¯ such that µ = hg, ¯ di. This leads to a contradiction and proves the result.
62
CHAPTER 2
From Lemma 2.73 and Theorem 2.74 we can immediately draw the following conclusion. L EMMA 2.76. A convex function f : n → is differentiable at x if and only if the subdifferential ∂ f (x) has only one element, in which case it is the gradient of f at x. Proof. A function f is differentiable at x if and only if its directional derivative f 0 (x; d) is linear in d. Then f 0 (x; d) = h∇ f (x), di for all d. It follows from Lemma 2.73 that ∇ f (x) ∈ ∂ f (x). If there exists another subgradient g 6 = ∇ f (x), then we can set d = g − ∇ f (x) to obtain from Lemma 2.73 f 0 (x; d) ≥ hg, di = kgk2 − hg, ∇ f (x)i. By differentiability, f 0 (x; d) = h∇ f (x), di = −k∇ f (x)k2 + hg, ∇ f (x)i. Subtracting the last two relations we conclude that kgk2 − 2hg, ∇ f (x)i + k∇ f (x)k2 ≤ 0, which is possible only when g = ∇ f (x). To prove the opposite implication, suppose the subdifferential has only one element, g. Theorem 2.74 implies that f 0 (x; d) = hg, di for all d. Thus the directional derivative is linear in the direction, which is equivalent to the differentiability of f at x and to g = ∇ f (x). For a convex function f : n → R, which is subdifferentiable at a point x, we define the direction of steepest descent as the vector d ∈ n of length not exceeding 1 for which the directional derivative f 0 (x; d) is minimal. L EMMA 2.77. If f is subdifferentiable at x and 0 ∈ / ∂ f (x), then the direction of steepest descent of f at x has the form g , dˆ = − kgk where g is the minimum norm element of ∂ f (x).
63
ELEMENTS OF CONVEX ANALYSIS
Proof. Consider the minimum norm element g ∈ ∂ f (x). It is nonzero, because 0 ∈ / ∂ f (x) and the subdifferential is closed. As g is the projection of 0 on ∂ f (x), Lemma 2.11 yields the inequality h0 − g, s − gi ≤ 0 for all s ∈ ∂ f (x). This can be rewritten as follows: max h−g, si = −kgk2 .
s∈∂ f (x)
Dividing by kgk and using Lemma 2.75 we obtain
ˆ = max hd, ˆ si = max − g , s = −kgk. f 0 (x; d) s∈∂ f (x) s∈∂ f (x) kgk For every other d of length at most 1 we have
f 0 (x; d) = max hd, si ≥ hd, gi ≥ −kgk. s∈∂ f (x)
ˆ is the smallest possible. This proves that f 0 (x; d) Let us consider four examples of convex functions and their subdifferentials. Example 2.78. Let k · k be the Euclidean norm in
n,
and let
f (x) = kxk. Then
∂ f (0) = {g ∈
n
Indeed, for every direction d,
f 0 (0; d) = lim τ ↓0
: kgk ≤ 1}.
kτ dk = kdk. τ
By Lemma 2.73, a vector g ∈ ∂ f (0) if and only if hg, di ≤ kdk for all d, which is equivalent to kgk ≤ 1. Our result can be extended to any norm k · k♦ in we see that its subdifferential at 0 is the set ∂k0k♦ = {g ∈
n
n.
By an identical argument
: hg, di ≤ kdk♦ for all d}.
Defining the dual norm hg, di = sup hg, di kdk♦ =1 d6 =0 kdk♦
kgk∗ = sup
64
CHAPTER 2
we find that ∂k0k♦ = {g ∈
n
: kgk∗ ≤ 1}.
In particular, the subdifferential of k · k1 at 0 is the set {g : kgk∞ ≤ 1}, and the subdifferential of k · k∞ at 0 is the set {g : kgk1 ≤ 1}. Let us now consider an arbitrary nonzero point x. For every vector g ∈ ∂kxk ♦ we must have 2kxk♦ = kx + xk♦ ≥ kxk♦ + hg, xi, 0 = kx − xk♦ ≥ kxk♦ − hg, xi, which implies that hg, xi = kxk♦ .
Dividing both sides by kxk♦ (which is nonzero), we conclude that kgk∗ ≥ hg,
x i = 1. kxk♦
Suppose kgk∗ > 1. Then there exists d such that kdk♦ = 1 and hg, di > 1. We obtain kxk♦ + 1 = kxk♦ + kdk♦ ≥ kx + dk♦ ≥ kxk♦ + hg, di > kxk♦ + 1, a contradiction. We claim, therefore, that ∂kxk♦ = {g ∈
n
: kgk∗ ≤ 1, hg, xi = kxk♦ }.
(2.37)
Since we have already established that every subgradient is an element of the set on the right hand side, it remains to prove the converse. Let g be an element of the set on the right hand side. Since kgk∗ ≤ 1, for every y we have kyk♦ ≥ hg, yi = hg, xi + hg, y − xi = kxk♦ + hg, y − xi, so g is a subgradient of k · k♦ at x. Our formula for the subdifferential of the norm is now valid for y = 0 and for y 6= 0. Example 2.79. Let Z be a closed convex set in n and k · k♦ be a norm. Consider the function f (x) = min kx − zk♦ .
z∈Z
From Example 2.55 we know that f is a convex function. Let us calculate its subdifferential at a point x. Choose zˆ ∈ Z such that kx − zˆ k♦ = f (x). Such a point exists because Z is a closed set, but it does not have to be unique, if the norm k · k♦ is not strictly convex. For every y we have f (y) ≤ ky − zˆ k♦ ,
65
ELEMENTS OF CONVEX ANALYSIS
and at y = x the above inequality becomes an equation. Thus, for every g ∈ ∂ f (x) we obtain ky − zˆ k♦ ≥ f (y) ≥ hg, y − xi.
This implies that g ∈ ∂kx − zˆ k♦ . Therefore,
∂ f (x) ⊂ ∂kx − zˆ k♦ . For every g ∈ ∂ f (x) and for every z ∈ Z we must have 0 = f (z) ≥ f (x) + hg, z − xi = kx − zˆ k♦ + hg, zˆ − xi + hg, z − zˆ i. We already know that g ∈ ∂kx − zˆ k♦ , so we can use the results of the calculation in Example 2.78. We have two cases. Either x = zˆ , in which case the first two terms on the right hand side vanish, or x 6= zˆ . In the latter case, formula (2.37) yields kx − zˆ k♦ = hg, x − zˆ i. In both cases the sum of the first two terms on the right hand side is zero. The last displayed inequality simplifies to hg, z − zˆ i ≤ 0 for all z ∈ Z . Hence We claim that
g ∈ N Z (ˆz ). ∂ f (x) = N Z (ˆz ) ∩ ∂kx − zˆ k♦ .
(2.38)
Note that the point zˆ need not be unique, in general, so we also claim that the set on the right hand side is the same for all possible projections zˆ . We have already proved that for every zˆ each subgradient of f at x is an element of the set at the right hand side of (2.38). Let us prove the converse. Let g be an element of the set at the right hand side of (2.38). Since kgk∗ ≤ 1, for every y we have f (y) = inf ky − zk♦ ≥ inf hg, y − zi z∈Z
z∈Z
= hg, y − xi + hg, x − zˆ i + inf hg, zˆ − zi. z∈Z
As g ∈ ∂kx − zˆ k♦ we have hg, x − zˆ i = kx − zˆ k♦ = f (x). We also have in (2.38) that g ∈ N Z (ˆz ), so hg, zˆ − zi ≥ 0 for all z ∈ Z . Consequently, the last displayed inequality yields f (y) ≥ f (x) + hg, y − xi for all y,
and g is a subgradient. In particular, if k · k♦ is the Euclidean norm and x ∈ / Z , then the point zˆ is the orthogonal projection of x on Z . Therefore ∂ f (x) = N Z (5 Z (x)) ∩ ∂kx − 5 Z (x)k. But the subdifferential of the norm contains only one element: ∂kx − 5 Z (x)k = {x − 5 Z (x)},
66
CHAPTER 2
and
x − 5 Z (x) ∈ N Z (5 Z (x)).
Therefore the distance function is differentiable at every x ∈ / Z and its gradient is given by ∇ f (x) = x − 5 Z (x). Example 2.80. Let
f (x) = max f i (x), i∈I
where I is a finite set, and f i are convex and differentiable functions for each i ∈ I . Define Iˆ(x) = {i ∈ I : f i (x) = f (x)}. Then
∂ f (x) = conv{∇ f i (x) : i ∈ Iˆ(x)}.
Indeed, if s ∈ Iˆ(x) we have
f (y) ≥ f s (y) ≥ f s (x) + h∇ fs (x), y − xi = f (x) + h∇ f s (x), y − xi. Thus ∇ f s (x) ∈ ∂ f (x). Since the subdifferential is convex, we have therefore conv{∇ f i (x) : i ∈ Iˆ(x)} ⊂ ∂ f (x). To prove the equality of these sets, suppose g ∈ ∂ f (x) exists such that g∈ / conv{∇ f i (x) : i ∈ Iˆ(x)}. By Theorem 2.14 we can strictly separate g and the convex hull above: there exist d 6= 0 and ε > 0 such that hg, di ≥ max h∇ fi (x), di + ε. i∈ Iˆ(x)
Thus for all τ > 0 f (x + τ d) ≥ f (x) + τ hg, di ≥ f (x) + τ max h∇ fi (x), di + τ ε. i∈ Iˆ(x)
(2.39)
For a sufficiently small τ > 0 we can always find r ∈ Iˆ(x) such that f (x + τ d) = fr (x + τ d). By the differentiability of the f i ’s there exist functions oi (τ ) such that f i (x + τ d) = fi (x) + τ h∇ fi (x), di + oi (τ ) and oi (τ )/τ → 0 as τ ↓ 0. Thus, for small τ > 0 f (x + τ d) ≤ f (x) + τ max h∇ f i (x), di + max oi (τ ). i∈ Iˆ(x)
i∈ Iˆ(x)
67
ELEMENTS OF CONVEX ANALYSIS
Combining this inequality with (2.39) we obtain f (x) + τ max h∇ fi (x), di + max oi (τ ) ≥ f (x) + τ max h∇ f i (x), di + τ ε. i∈ Iˆ(x)
i∈ Iˆ(x)
i∈ Iˆ(x)
Simplifying, dividing by τ and letting τ ↓ 0 we obtain 0 ≥ ε, a contradiction. The function in Figure 2.12 on page 58 is the maximum of four smooth functions. Two of them are “active” at the selected point, and all convex combinations of their gradients are subgradients of f at this point. Example 2.81. Let C be a closed convex set in n and let us consider the indicator function of C ( 0 if x ∈ C δC (x) = +∞ otherwise.
It is convex. Let us calculate its subdifferential at a point x ∈ C. We have g ∈ ∂δ(x) if and only if δC (y) − δC (x) ≥ hg, y − xi for all y. This is trivially satisfied for y∈ / C, so the only meaningful case is when y ∈ C. Thus, for g to be a subgradient of δC (·) at x it is necessary and sufficient that hg, y − xi ≤ 0, for all y ∈ C. Hence
hg, di ≤ 0 for all d ∈ cone(C − x).
We have obtained, therefore, the representation of the subdifferential as the normal cone: ∂δC (x) = [cone(C − x)]◦ = NC (x).
In particular, if K is a closed convex cone, then
∂δ K (0) = K ◦ . 2.5.2 Chain Rules of Subdifferential Calculus
Many simple rules of calculus extend to subdifferentials. L EMMA 2.82. Assume that f : n → is a convex function, α > 0, and h(x) = α f (x). Then h is convex, and ∂h(x) = α∂ f (x), for every x. Proof. The relation follows directly from the definition. We have g ∈ ∂ f (x) if and only if for all y h(y) = α f (y) ≥ α[ f (x) + hg, y − xi] = h(x) + hαg, y − xi, which is equivalent to αg ∈ ∂h(x). L EMMA 2.83. Assume that f : m → is a convex function, A is an m ×n matrix, and h(x) = f (Ax). Then ∂h(x) = A T ∂ f (Ax), for every x.
68
CHAPTER 2
Proof. The relation follows directly from the definition. We have g ∈ ∂ f (Ax) if and only if h(y) = f (Ay) ≥ f (Ax) + hg, Ay − Axi = h(x) + hA T g, y − xi, which is equivalent to A Tg ∈ ∂h(x). Example 2.84. Consider the function h(x) =
p
hx, Axi,
where A is a symmetric positive definite matrix. Factorizing A = U T U , with some nonsingular U , we see that h(x) = kU xk.
Therefore we can use Example 2.78 and Lemma 2.83 to conclude that ∂h(0) = {U T g : kgk ≤ 1}
= {v : k(U T )−1 vk ≤ 1} = {v : hv, A −1 vi ≤ 1}.
p In fact, h(x) is a norm, whose dual norm is kvk A−1 = hv, A−1 vi. Our calculation re-establishes in this special case the general result about subdifferentials of norms presented in Example 2.78.
and T HEOREM 2.85. Assume that f = f 1 + f 2 , where f 1 : n → f 2 : n → are convex proper functions. If there exists a point x 0 ∈ dom f such that f 1 is continuous at x 0 , then ∂ f (x) = ∂ f 1(x) + ∂ f 2 (x), for all x ∈ dom f. Proof. Suppose g 1 ∈ ∂ f 1 (x) and g 2 ∈ ∂ f 2 (x). Then, for all y f (y) = f 1 (y) + f 2 (y) ≥ f 1 (x) + hg 1 , y − xi + f 2 (x) + hg 2 , y − xi = f (x) + hg 1 + g 2 , y − xi.
This proves that ∂ f (x) ⊃ ∂ f 1(x) + ∂ f 2 (x). We shall show the equality here. Suppose a subgradient g ∈ ∂ f (x) exists such that g ∈ / ∂ f 1 (x)+∂ f 2 (x). To apply the separation theorem (Theorem 2.14) we need to verify that ∂ f 1 (x)+ ∂ f 2 (x) is a closed convex set. Its convexity follows from the convexity of the subdifferentials ∂ f 1(x) and ∂ f 2 (x) (see Theorem 2.74 and the remark after its proof) and from Lemma 2.3.
69
ELEMENTS OF CONVEX ANALYSIS
Both subdifferentials are closed and the sum ∂ f 1 (x) + ∂ f 2 (x) is closed as well. We prove this technical property after the main line of the proof. If ∂ f 1(x) + ∂ f 2 (x) is closed, we can invoke Theorem 2.14 and separate g from the sum ∂ f 1(x) + ∂ f 2 (x). There exists a direction d ∈ n and ε > 0 such that hg, di ≥ hg 1 + g 2 , di + ε,
(2.40)
for all g 1 ∈ ∂ f 1(x) and all g 2 ∈ ∂ f 2 (x). It follows that hg 1 , di is bounded from above for all g 1 ∈ ∂ f 1(x). Thus, by Lemma 2.75, f 10 (x; d) = max hg 1 , di < ∞. g 1 ∈∂ f 1 (x)
Similarly, f 20 (x; d) = max hg 2 , di < ∞. g 2 ∈∂ f 2 (x)
Taking the supremum of the right hand side of (2.40) over g 1 ∈ ∂ f 1 (x) and g 2 ∈ ∂ f 2 (x) and using the last two relations, we obtain hg, di ≥ max hg 1 , di + max hg 2 , di + ε = f 10 (x; d) + f 20 (x; d) + ε. g 1 ∈∂ f 1 (x)
g 2 ∈∂ f 2 (x)
On the other hand, hg, di ≤ f 0 (x; d) and the last inequality yields f 0 (x; d) ≥ f 10 (x; d) + f 20 (x; d) + ε,
a contradiction. Therefore a subgradient g ∈ ∂ f (x) \ ∂ f 1 (x) + ∂ f 2 (x) cannot exist. Let us now return to the issue of the closedness of the sum ∂ f 1 (x)+∂ f 2 (x). If one of the functions f 1 or f 2 is continuous at x, then its subdifferential is compact, and the sum ∂ f 1(x) + ∂ f 2 (x) is closed, too. The only remaining case is when both subdifferentials are unbounded. Consider two sequences g1k ∈ ∂ f 1 (x) and g2k ∈ ∂ f 2(x) such that g1k + g2k → s, as k → ∞.
Suppose s ∈ / ∂ f 1 (x)+∂ f 2 (x). As mentioned above, kg1k k → ∞ and kg2k k → ∞, because all accumulation points of these sequences must be elements of the corresponding subdifferentials. Consider the sequence zk =
g1k . kg1k k
By choosing a subsequence, if necessary, we may assume that {z k } has limit z. We have f 1 (x 0 ) − f 1 (x) ≥ hg1k , x 0 − xi.
70
CHAPTER 2
Dividing by kg1k k and passing to the limit with k → ∞ we conclude that hz, x 0 − xi ≤ 0. We also have g2k = s − g1k , and thus f 2 (x 0 ) − f 2 (x) ≥ hs − g1k , x 0 − xi.
Dividing by kg1k k and passing to the limit we obtain hz, x 0 −xi ≥ 0. Therefore hz, x 0 − xi = 0. Since f 1 is continuous at x 0 , we can find ε > 0 such that f 1 (x 0 + εz) < ∞. We obtain f 1 (x 0 + εz) − f 1 (x) ≥ hg1k , x 0 + εz − xi.
Dividing by kg1k k and passing to the limit we conclude that 0 ≥ hz, x 0 − x + εzi = εkzk2 = ε,
a contradiction. Therefore the sum ∂ f 1(x) + ∂ f 2 (x) is closed. The last theorem is called the Moreau–Rockafellar Theorem. Example 2.86. Consider the set K = K1 ∩ K2 ∩ · · · ∩ Km , where each K i , i = 1, . . . , m is a convex cone. From Example 2.81 we know that ∂δ K (0) = K ◦ . On the other hand, δ K (x) = δ K 1 (x) + δ K 2 (x) + · · · + δ K m (x). If K 1 ∩ int K 2 ∩ · · · ∩ int K m 6= ∅, we can employ Theorem 2.85 to get ∂δ K (0) = ∂δ K 1 (0) + ∂δ K 2 (0) + · · · + ∂δ K m (0) = K 1◦ + K 2◦ + · · · + K m◦ . We have thus established again the result of Theorem 2.35. More generally, let X i , i = 1, . . . , m be closed convex sets, and let X = X1 ∩ X2 ∩ · · · ∩ Xm. We have
∂δ X (x) = N X (x).
If X 1 ∩ int X 2 ∩ · · · ∩ int X m 6= ∅, Theorem 2.85 yields again N X (x) = N X 1 (x) + N X 2 (x) + · · · + N X m (x), as previously established in Lemma 2.40.
71
ELEMENTS OF CONVEX ANALYSIS
2.5.3 The Subdifferential of the Maximum Function
Consider the function F(x) = sup f (x, y). y∈Y
We assume that f :
n
×Y →
satisfies the following conditions:
(i) f (·, y) is convex for all y ∈ Y ; (ii) f (x, ·) is upper semicontinuous for all x in a certain neighborhood of a point x 0 ; (iii) The set Y ⊂
m
is compact.
The maximum function F is convex by virtue of Lemma 2.58. It is proper, due to (ii). Our intention is to describe the subdifferential of F at the point x 0 . We have already considered a special case of this problem in Example 2.80, and we shall generalize the formula developed there. Our analysis follows similar lines, but with more technical complications. We denote by Yˆ (x) the set of y ∈ Y at which f (x, y) = F(x). Since f (x, ·) is upper semicontinuous and Y is compact, the set Yˆ (x) is nonempty and compact, for every x in a certain neighborhood of x 0 . We use the symbol ∂ x f (x 0 , y) to denote the subdifferential of the function f (·, y) at x 0 . T HEOREM 2.87. Assume conditions (i)–(iii). Then [ ∂ F(x 0 ) ⊃ conv ∂x f (x 0 , y) . y∈Yˆ (x 0 )
If, in addition, the function f (·, y) is continuous at x 0 for all y ∈ Y , then [ ∂ F(x 0 ) = conv ∂x f (x 0 , y) . (2.41) y∈Yˆ (x 0 )
Proof. Suppose g ∈ ∂ x f (x 0 , y0 ) for some y0 ∈ Yˆ (x 0 ). Then, for every x we have F(x) = sup f (x, y) ≥ f (x, y0 ) ≥ f (x 0 , y0 ) + hg, x − x 0 i. y∈Y
Therefore g ∈ ∂ F(x 0 ). Since the subdifferential is convex, our first assertion holds true. We now prove that the set on the right hand side of (2.41) is closed. Consider a convergent sequence of vectors s k ∈ ∂x f (x 0 , yk ), with yk ∈
72
CHAPTER 2
Yˆ (x 0 ), and let s ∗ = lim k→∞ sk . As the set Yˆ (x 0 ) is compact, by passing to a subsequence, if necessary, we may also assume that the sequence {y k } is convergent. Its limit, y ∗ , is an element of Yˆ (x 0 ). For every x in a small neighborhood of x 0 , the upper semicontinuity of f (x, ·) and the fact that s k is a subgradient imply f (x, y ∗ ) ≥ lim sup f (x, yk ) ≥ lim sup f (x 0 , yk ) + hsk , x − x 0 i k→∞
k→∞
= f (x 0 , y ∗ ) + hs ∗ , x − x 0 i.
In the last transformation we use the fact that f (x 0 , yk ) = F(x 0 ) = f (x 0 , y ∗ ). This proves that s ∗ ∈ ∂ f (x 0 , y ∗ ). Suppose our second assertion is false, that is, there exists g ∈ ∂ F(x 0 ) such that [ g∈ / conv ∂x f (x 0 , y) . y∈Yˆ (x 0 )
Since the set on the right hand side is convex and closed, Theorem 2.14 implies that there exist d 6 = 0 and ε > 0 such that hg, di ≥ hs, di + ε
(2.42)
for all s ∈ ∂ x f (x 0 , y) and all y ∈ Yˆ (x 0 ). Consider points of the form x 0 + τk d, where τk ↓ 0. By the convexity of F we have F(x 0 + τk d) − F(x 0 ) ≥ hg, di. τk Define the sets n o f (x 0 + τk d, y) − F(x 0 ) ≥ hg, di , k = 1, 2, . . . . Yk = y ∈ Y : τk
They are closed, because f (x, ·) is upper semicontinuous. They are nonempty, because Yˆ (x 0 + τk d) ⊂ Yk . Moreover, for every y ∈ Y the expression f (x 0 + τ d, y) − f (x 0 , y) f (x 0 , y) − F(x 0 ) f (x 0 + τ d, y) − F(x 0 ) = + τ τ τ defines an increasing function of τ . Indeed, the first fraction is the difference quotient of a convex function (see formula (2.30) on page 57), and the second fraction has a fixed nonpositive numerator. This implies that Y1 ⊃ Y 2 ⊃ Y 3 ⊃ · · · .
73
ELEMENTS OF CONVEX ANALYSIS
The sets Yk are compact and nonempty, and thus there exists a point y¯ which belongs to all of them. Then f (x 0 + τk d, y) ¯ − F(x 0 ) ≥ hg, di, k = 1, 2, . . . . τk Because f (x 0 + τk d, y¯ ) → f (x 0 , y¯ ) as k → ∞, we must have f (x 0 , y) ¯ = F(x 0 ), i.e., y¯ ∈ Yˆ (x 0 ). Passing to the limit in the last inequality with k → ∞ and employing Theorem 2.74 we conclude that lim
k→∞
f (x 0 + τk d, y¯ ) − F(x 0 ) = hs, di ≥ hg, di, τk
with some s ∈ ∂ x f (x 0 , y¯ ). This contradicts (2.42). Example 2.88. We have n continuous functions ϕi (t), i = 1, . . . , n, where t ∈ [a, b] ⊂ . We call them basic functions. Our objective is to approximate a continuous function ψ(t) by a linear combination of basic functions:
n X i=1
x i ϕi (t) ≈ ψ(t), t ∈ [a, b].
In the last formula x i , i = 1, . . . , n, are unknown coefficients. If we are interested in a uniform approximation in the interval [a, b], it is natural to measure the approximation error as follows: n X x i ϕi (t) − ψ(t) . F(x) = max a≤t≤b
i=1
Approximation based on this criterion is frequently referred to as Chebyshev approximation. Theorem 2.87 allows us to calculate the subdifferential of the error function F(x) at any point x. Define the set n X n o Tˆ (x) = t ∈ [a, b] : x i ϕi (t) − ψ(t) = F(x) . i=1
In the typical case, when F(x) > 0, we obtain
ϕ1 (t) n o n X ϕ2 (t) ∂ F(x) = conv sgn x i ϕi (t) − ψ(t) . : t ∈ Tˆ (x) . .. i=1
The symbol sgn(·) denotes the sign function.
ϕn (t)
74
CHAPTER 2
Example 2.89. Consider the maximum eigenvalue function λ max (·) defined on the space Sn of symmetric matrices of dimension n × n. It is convex (cf. Example 2.59). Since λmax (A) = max hy, Ayi, kyk=1
we can calculate its subdifferential by Theorem 2.87. The set Yˆ (A) = {y ∈
n
: hy, Ayi = λmax (A), kyk = 1},
is the set of all eigenvectors of A corresponding to the maximum eigenvalue and having length 1. As in Example 2.31, we use the Frobenius inner product, hA, H iS = tr(AH ) =
n X n X
ai j h i j ,
i=1 j =1
and we rewrite the function as follows: λmax (A) = max hyy T , AiS . kyk=1
For every y ∈ n the function f y (A) = hyy T , AiS is linear and its gradient equals ∇ f y (A) = yy T . Consequently
∂λmax (A) = conv{yy T : Ay = λmax (A)y, kyk = 1}.
Our final observation is that the set of matrices of the form W = yy T can be equivalently characterized as follows: W has rank one and W ∈ Sn+ . Then tr(W ) = kyk2
and
hA, W iS = hy, Ayi.
Therefore the requirement that y is an eigenvector of length 1 corresponding to the maximum eigenvalue is equivalent to the conditions tr(W ) = 1
and
hA, W iS = λmax (A).
We obtain ∂λmax (A) = conv{W ∈ Sn+ : hA, W iS = λmax (A), tr(W ) = 1, rank (W ) = 1}. We can omit the convex hull and the rank restriction in the last representation of the subdifferential: ∂λmax (A) = {W ∈ Sn+ : hA, W iS = λmax (A), tr(W ) = 1}. P Indeed, every element of this set has the form W = nj =1 µ j y j y jT , with orthogonal eigenvectors y j of W having norm 1, Pand with the corresponding eigenvalues µ j ≥ 0. The condition tr(W ) = 1 implies nj =1 µ j = 1. Hence hA, W iS =
n X j =1
µ j hy j , Ay j i ≤
n X j =1
µ j λmax (A) = λmax (A).
75
ELEMENTS OF CONVEX ANALYSIS
Equality occurs if and only if y j ’s corresponding to positive µ j ’s are eigenvectors of A corresponding to its maximum eigenvalue. Then W is a convex combination of rank one matrices y j y jT formed of these eigenvectors, with the weights µ j . 2.6 CONJUGATE DUALITY 2.6.1 The Conjugate Function
The domain of a proper convex function is a convex set. If it has a nonempty interior, then the function is subdifferentiable at all points of the interior (Theorem 2.74). If the domain has no interior, we may restrict our considerations to the linear manifold of the smallest dimension containing the domain. The function restricted to this manifold (after an appropriate change of variables) will be subdifferentiable in the corresponding space at interior points of its domain. It follows that for every proper convex function f there exists a point x 0 ∈ dom f such that f is subdifferentiable at x 0 . Then for every x ∈ n inequality (2.32) holds true: f (x) ≥ f (x 0 ) + hs0 , x − x 0 i, where s0 ∈ ∂ f (x 0 ). Let us consider the set of α ∈ function
, for which the affine
lα (x) = hs0 , xi − α
(2.43)
is a minorant of f , that is, it satisfies the inequality lα (x) ≤ f (x) for all x ∈
n
.
We call s0 the slope of the affine minorant l α (·). Since s0 ∈ ∂ f (x 0 ), the set of affine minorants having slope s0 is nonempty and has the maximum element, corresponding to α ∗ = hs0 , x 0 i − f (x 0 ). This motivates the following two questions: 1. For which vectors s ∈ n do affine minorants of the form (2.43) exist, and do they have a finite supremum? 2. Is it possible to represent f as a supremum of all possible affine minorants, for all s ∈ n ? In order to analyze these issues, we introduce the following concept.
76
CHAPTER 2
D EFINITION 2.90. Let f : n → be a proper convex function. The function f ∗ : n → defined by M f ∗ (s) = sup hs, xi − f (x) x
is called the conjugate function to f .
It follows directly from the definition that for every x ∈ s ∈ n we have: f (x) + f ∗ (s) ≥ hs, xi.
n
and every (2.44)
If f ∗ (s) < +∞ our first question has a positive answer: f has affine minorants with slope s. If f ∗ (s) = +∞, no affine minorants with slope s exist. In fact, we can define the conjugate function f ∗ for every extended real valued function f : n → . L EMMA 2.91. If the function f : n → is proper and has an affine minorant, then the conjugate function is proper, convex, and lower semicontinuous. Proof. By Definition 2.90, the epigraph of the function f ∗ has the form \ epi f ∗ = (s, v) : v ≥ hs, xi − f (x) . x∈dom f
As an intersection of closed convex sets, it is a closed convex set (Lemma 2.2). Thus f ∗ is a convex function. Moreover, by Lemma 2.62, f ∗ is a lower semicontinuous function. By assumption, an affine minorant of the form (2.43) exists. Because f is proper, there exists a point x¯ such that −∞ < f (x) ¯ < +∞. We obtain f ∗ (s0 ) = sup hs0 , xi − f (x) ≥ hs0 , xi ¯ − f (x). ¯ x
On the other hand, (2.43) is a minorant of f , and thus f ∗ (s0 ) = sup lα (x) + α − f (x) ≤ α. x
Hence f (s0 ) is finite. For every s ∈ n we have ∗
f ∗ (s) ≥ hs, x 0 i − f (x 0 ) = f ∗ (s0 ) + hs − s0 , x 0 i. Therefore f ∗ is proper.
77
ELEMENTS OF CONVEX ANALYSIS
If f is convex and proper then it always has an affine minorant, and thus f ∗ is convex, proper, and lower semicontinuous. Example 2.92. Consider a convex cone K ⊂ n and its indicator function ( 0 if x ∈ K δ K (x) = +∞ otherwise.
Let us calculate its conjugate δ ∗K . We have δ ∗K (s) = sup hs, xi − δ K (x) = sup hs, xi. x∈
n
x∈K
By Lemma 2.26, hs, xi has finite supremum over x ∈ K if and only if s ∈ K ◦ . Therefore ( 0 if s ∈ K ◦ ∗ δ K (s) = +∞ otherwise, that is, δ ∗K = δ K ◦ . Let us observe that the conjugate of δ K ◦ is δ K ◦◦ , which is equal to δ K again, if K is closed (Theorem 2.27). We discuss this in the next section. Example 2.93. Let k · k♦ be a norm in B = {x ∈
n
n.
Let us consider the unit ball,
: kxk♦ ≤ 1},
and its indicator function, δ B (x) =
(
0 if kxk♦ ≤ 1 +∞ otherwise.
The conjugate function has the form (δ B )∗ (s) = sup hs, xi − δ B (x) = sup hs, xi = ksk∗ , x∈
n
kxk♦ ≤1
where k · k∗ is the dual norm to k · k♦ .
Example 2.94. Let k·k♦ be a norm in n . The conjugate function to f (x) = kxk♦ has the form f ∗ (s) = sup hs, xi − kxk♦ .
x∈
n
Let k · k∗ be the dual norm. If ksk∗ > 1 we can find x¯ such that k xk ¯ ♦ = 1 and hs, x¯ i > 1. Multiplying x¯ by an arbitrary β > 0 we see that the supremum on the right hand side is +∞. On the other hand, if ksk∗ ≤ 1, then hs, xi ≤ kxk♦ . Thus the supremum on the right hand side is 0. Consequently, f ∗ (s) = δ B∗ (s),
78
CHAPTER 2
where B∗ = {s ∈
n
: ksk∗ ≤ 1}
is the unit ball with respect to the dual norm. Let us combine this result with Example 2.93 and note that k · k♦ is the dual norm to k · k∗ . It follows that k · k♦ and δ B∗ (·) are mutually conjugate. We explain this in the next section. 2.6.2 The Biconjugate Function
Let us now consider the conjugate function to the conjugate function f ∗ : M f ∗∗ (x) = sup hs, xi − f ∗ (s) .
(2.45)
s
We call it the biconjugate function. By Definition 2.90, each function l s :
n
→
given by
ls (x) = hs, xi − f ∗ (s) is the pointwise supremum of all affine minorants of f having slope s. Thus f ∗∗ (·) is the pointwise supremum of all affine minorants of f . The following result is known as the Fenchel–Moreau Theorem. T HEOREM 2.95. If an extended real valued function f : least one affine minorant, then
n
→
has at
epi f ∗∗ = conv(epi f ). In particular, if f is a proper, lower semicontinuous and convex function, then f ∗∗ = f. Proof. As we saw before stating the theorem, f ∗∗ ≤ f , and thus epi f ⊂ epi f ∗∗ . It follows from (2.45) that the epigraph of f ∗∗ is an intersection of closed halfspaces, and is therefore a closed convex set. Consequently, conv(epi f ) ⊂ epi f ∗∗ . Suppose a strict inclusion above. Then there exists a point (y, β) ∈ epi f ∗∗ \ conv(epi f ) .
79
ELEMENTS OF CONVEX ANALYSIS
Since the set conv(epi f ) is closed and convex, by virtue of Theorem 2.14 we can strictly separate (y, β) from this set. There exist s ∈ n , γ ∈ , and ε > 0 such that hs, yi + γβ ≤ hs, xi + γ α − ε, for all (x, α) ∈ conv(epi f ). Letting α → ∞ we deduce that γ ≥ 0. If γ > 0 we can divide both sides of the last inequality by γ . Setting α = f (x) we conclude that l(x) =
s
ε , y − x + β + ≤ f (x), α γ
for all x ∈ dom f . Thus l(·) is an affine minorant of f . It follows that f ∗∗ (y) ≥ l(y) = β + γε > β, which contradicts the inclusion (y, β) ∈ epi f ∗∗ . If γ = 0, the separating plane is vertical: hs, yi ≤ hs, xi − ε for all x ∈ dom f.
(2.46)
By assumption, f has an affine minorant: hz, xi + c ≤ f (x) for all x ∈
n
.
Adding (2.46) multiplied by M > 0 to the last displayed inequality and rearranging terms we obtain hz − Ms, xi + Mhs, yi + c + Mε ≤ f (x) for all x ∈
n
.
Consequently, the function on the left hand side is an affine minorant of f , and its value at y must not be greater than f ∗∗ (y). It follows that hz, yi + c + Mε ≤ f ∗∗ (y) for all M > 0. Letting M → ∞ we conclude that f ∗∗ (y) = +∞. This contradicts the inclusion (y, β) ∈ epi f ∗∗ . If f has no affine minorants, then f ∗ (s) = +∞ for all s ∈ n , and thus f (x) = −∞ for all x. Theorem 2.95 explains the phenomenon that is evident in Examples 2.92, 2.93, and 2.94. We can also use it to calculate conjugate functions. ∗∗
Example 2.96. Let Z be a set in n . Let us define the support function of Z as follows: σ Z (s) = sup hs, xi.
x∈Z
80
CHAPTER 2
We notice that σ Z (·) is the conjugate of the indicator function ( 0 if x ∈ Z δ Z (x) = +∞ otherwise. Therefore σ Z∗ = δ ∗∗ Z . By Theorem 2.95,
epiδ ∗∗ Z = conv(epiδ Z ) = epiδconv(Z ) .
It follows that
σ Z∗ (·) = δconv(Z )(·).
(2.47)
If Z is convex and closed, the functions σ Z (·) and δ Z (·) are mutually conjugate. If Z is not convex and closed, the biconjugate δ ∗∗ Z (·) is the indicator function of the closed convex hull of Z . In particular, a norm k·k♦ is the support function of the dual ball B∗, and Examples 2.93 and 2.94 illustrate the general formula (2.47). Example 2.97. Assume that an uncertain outcome may take one of n values: x 1 , . . . , x n . Define as the vector in n having all components equal to 1. A convex risk function is defined as a convex function f : n → satisfying the following conditions:
(i) If x ≤ y then f (x) ≤ f (y);
(ii) For all x ∈
n
and a ∈
we have f (x + a ) = f (x) + a;
(iii) f (·) is positively homogeneous.
We use Theorem 2.95 to derive an alternative representation of a risk function. As f (·) is convex and finite-valued, it is continuous. Therefore f = f ∗∗ . More specifically, for all x ∈ n , f (x) = sup h p, xi − f ∗ ( p) . (2.48)
p∈
n
We use the symbol p for the dual variable, in view of the following characterization of the domain of the conjugate function. We first show that dom f ∗ ⊂ n+ . Suppose we can find p¯ ∈ dom f ∗ which is not an element of the cone n+ . Then it can be strictly separated from n+ . There exist z ∈ n and ε > 0 such that:
hz, pi ≤ hz, pi ¯ − ε for all p ∈
n +.
It follows from Lemma 2.26 that z ≤ 0. Setting p = 0 we also infer that hz, pi ¯ ≥ ε > 0. Consider an arbitrary point x ∈ n and points y = x + tz, for t ≥ 0. As y ≤ x, condition (i) implies that f (y) ≤ f (x). From the definition of the conjugate function we deduce that f ∗ ( p) ¯ = sup h p, ¯ xi − f (x) = sup h p, ¯ x + tzi − f (x + tz)
n
x∈
x∈
n
≥ sup h p, ¯ xi − f (x) + th p, ¯ zi ≥ f ∗ ( p) ¯ + tε. x∈
n
81
ELEMENTS OF CONVEX ANALYSIS
As t may be an arbitrary positive number, we conclude that f ∗ ( p) ¯ = ∞, a contradiction. Consequently, dom f ∗ ⊂ n+ . Condition (ii) implies that for all a ∈ : f ∗ ( p) = sup h p, xi − f (x) = sup h p, x + a i − f (x + a )
n
x∈
x∈
n
= sup h p, xi − f (x) + a h p, i − 1 = f ∗ ( p) + a h p, i − 1 .
x∈
n
This means that f ∗ ( p) < ∞ only if h p, i = 1. We conclude that the domain of the conjugate function is included in the set of probability vectors
P= p∈
n +
:
n X
j =1
pj = 1 .
If the function f (·) satisfies condition (iii), for every t > 0 and every p ∈ dom f ∗ we have f ∗ ( p) = sup h p, xi − f (x) = sup h p, t xi − f (t x) n
x∈
x∈
n
∗
= sup th p, xi − t f (x) = t f ( p). x∈
n
This implies that f ∗ ( p) = 0. Consequently, f ∗ (·) is the indicator function of the certain set A ⊂ P. Formula (2.48) simplifies to f (x) = sup h p, xi, p∈A
which can be interpreted as the largest expected value of x,with respect to probability vectors from the set A. 2.6.3 Subgradients of Conjugate Functions
Let us recall that if a convex function f : n → is subdifferentiable at x and if s ∈ ∂ f (x), then for all y ∈ n the following inequality holds true: f (y) ≥ f (x) + hs, y − xi. At y = x the above inequality becomes an equation, and thus the right hand side is the largest affine minorant of f having slope s. Consequently, inequality (2.44) becomes an equation. In fact, this equation can be used as a characterization of subgradients of f and f ∗ . T HEOREM 2.98. Assume that f : n → is a proper convex function. Then the following two statements are equivalent: (i) s ∈ ∂ f (x); (ii) f (x) + f ∗ (s) = hs, xi.
82
CHAPTER 2
If, in addition, f is lower semicontinuous then both statements are equivalent to (iii) x ∈ ∂ f ∗ (s).
Proof. Suppose assertion (i) is true. If s ∈ ∂ f (x), then, as discussed before the theorem, inequality (2.44) becomes an equation. Thus (ii) holds true. By the definition of the conjugate function, for every y ∈ n we have f ∗ (s) ≥ hs, yi − f (y).
Suppose equation (ii) holds true. Manipulating the above inequality and using (ii) we obtain: f (y) ≥ hs, yi − f ∗ (s) = f (x) + hs, y − xi.
Thus assertion (i) is true. Consider the function g(x) = f ∗ (x). Since f is lower semicontinuous, Theorem 2.95 implies that g ∗ = f . Therefore assertion (ii) can be rewritten as follows: g ∗ (x) + g(s) = hs, xi.
We have proved that this is equivalent to (i) (with the roles of s and x reversed), which now reads This is (iii) for f , as required.
x ∈ ∂g(s).
Example 2.99. Let C be a closed convex set in n and let us consider the indicator function of C, ( 0 if x ∈ C δC (x) = +∞ otherwise.
Let us calculate its subdifferential at a point x ∈ C. From Example 2.96 we know that δC∗ (s) = σC (s), where σC (·) is the support function of S. Both functions are convex and lower semicontinuous. We can, therefore, use Theorem 2.98 to conclude that for x ∈ C ∂δC (x) = s ∈ n : σC (s) + δC (x) = hs, xi = s ∈ n : sup hs, yi = hs, xi .
y∈C
It follows that s ∈ ∂δC (x) if and only if
hs, y − xi ≤ 0 for all y ∈ C. This is equivalent to s ∈ NC (x), as already proved in Example 2.81.
83
ELEMENTS OF CONVEX ANALYSIS
Example 2.100. Let a1 , . . . , am be arbitrary vectors in linear function: f (x) = max hai , xi.
n.
Consider the piecewise
1≤i≤m
To calculate its conjugate function we define the set A = {a1 , . . . , am } and we notice that f is the support function of A: f (x) = max ha, xi = σ A (x). a∈A
By Theorem 2.95 and Example 2.96, f ∗ (s) = σ A∗ (s) = δ ∗∗ A (s) = δconv(A) (s), that is, f ∗ is the indicator function of the convex hull of the vectors a1 , . . . , am . The subdifferential of f at 0 is, by Theorem 2.98, equal to the set of s such that f (0) + δconv(A) (s) = hs, 0i. This simplifies to δconv(A) (s) = 0 and yields ∂ f (0) = conv{a1 , . . . , am }. This is the same as obtained in Example 2.80 on page 66 and in Section 2.5.3; however, our techniques there were more specific, and the results deeper. Example 2.101. Subdifferentials of norms and distances are very easy to calculate with the application of Theorem 2.98. Any norm k · k♦ in n is the support function of the dual ball: kxk♦ = sup hs, xi = σ B∗ (x).
s∈B∗
Its conjugate function, by Example 2.94, is the indicator function δ B∗ (·) of the dual ball. Therefore ∂kxk♦ = s ∈ n : hs, xi = kxk♦ + δ B∗ (s) = s ∈ B∗ : hs, xi = kxk♦ .
We have obtained this formula directly in Example 2.78.
Example 2.102. Consider a discrete random variable Y attaining values y 1 < y2 < · · · < ym with probabilities p1 , p2 , . . . , pm . For each target level x ∈ we can calculate the expected shortfall below x: X f (x) = E max(0, x − Y ) = pk (x − yk ).
{k:yk <x}
The function f is convex and nondecreasing. It is also piecewise linear with break points at y1 , . . . , ym and with slopes si =
i X k=1
pk , if yi < x < yi+1 , i = 1, . . . , m − 1.
84
CHAPTER 2
To the left of y1 the function f has value 0 and slope s0 = 0, and to the right of ym it has slope sm = 1. Let us derive its conjugate function. For every i = 0, 1, . . . , m − 1 we have s ∈ ∂ f (yi+1 ) ⇔ si ≤ s ≤ si+1 .
It follows from Theorem 2.98 that for s ∈ [si , si+1 ] the conjugate function has the form: f ∗ (s) = syi+1 − f (yi+1 ) = syi+1 −
i X k=1
pk (yi+1 − yk )
= (s − si )yi+1 + f ∗ (0)
f ∗ (1)
Pm
i X
p k yk .
k=1
Clearly, = 0 and = k=1 pk yk = E[Y ]. If s < 0 or s > 1, then f ∗ (s) = +∞. The function f ∗ (s) is convex, piecewise linear, and has break points at cumulative probabilities si , i = 0, 1, . . . , m. Its value at the point si is the contribution of the smallest i realizations of Y to the expected value of Y . In economics it is called the absolute Lorenz curve. EXERCISES 2.1. There are n mutual funds and m asset categories. Let ai j be the fraction of the capital of fund j invested in category i . We have initial capital C and we want to invest all or part of it in these funds. We denote by x j the amount invested in fund j = 1, . . . , n. No short selling is allowed, so x j has to be nonnegative. (a) Describe the set X ⊂ n of all possible amounts invested in these funds (fund portfolios). What are its extreme points?
(b) Describe the set Y ⊂ m of all possible amounts invested in this way in the m asset categories (asset portfolios).
(c) Show that if a point y is an extreme point of Y , it has the form y = Ax, where x is an extreme point of X. (d) Suppose y ∈ Y is an asset portfolio obtained by investing in some of the available funds. Prove that you can construct it by investing in no more than m + 1 funds. 2.2. Prove that x is an extreme point of a convex set X if and only if X \ {x} is convex. 2.3. Let K be a closed convex cone in n . For a point x ∈ K the set {αx : α ≥ 0} is called a ray. An extreme ray of K is a ray which cannot be expressed as a convex combination of two other rays.
85
ELEMENTS OF CONVEX ANALYSIS
(a) Prove that a ray T is extreme if and only if K \ T is convex.
(b) Prove that a closed convex cone which does not contain any line is the convex hull of the set of its extreme rays. 2.4. P Let S be the set of all n × n matrices P A such that ai j ≥ 0 for i, j = 1, . . . , n and ni=1 ai j = 1 for j = 1, . . . , n, nj =1 ai j = 1 for i = 1, . . . , n. Prove that S is convex and that its extreme points are permutation matrices, that is, matrices P ∈ S such that pi j ∈ {0, 1}, i, j = 1, . . . , n. 2.5. The cone K in
n
is defined as follows:
K = {x ∈
n
: 0 ≤ x 1 ≤ x 2 ≤ · · · ≤ x n }.
Find the polar cone K ◦ . 2.6. C is a closed convex cone in
n.
The cone K ∈
K = {y = (y 1 , y 2 , . . . , y m ) : y i ∈ 1
n
2
mn
is defined by
, i = 1, . . . , m,
y + y + · · · + y m ∈ C}.
Find the polar cone K ◦ . 2.7. Assume that the cones K 1 , . . . , K m in K i = {x ∈
n
n
are polyhedral, that is,
: Ai x ≤ 0},
with some matrices A i of dimension l i × n, i = 1, . . . , m. Prove that K ◦ = K 1◦ + K 2◦ + · · · + K m◦ . 2.8. Let K be a closed convex cone in n . Prove that every x ∈ represented as x = 5 K (x) + 5 K ◦ (x)
and
n
can be
h5 K (x), 5 K ◦ (x)i = 0.
2.9. Let K be a closed convex cone. Find the normal cone to K at x ∈ K . 2.10. Prove the generalized geometric–arithmetic mean inequality: for all x 1 > 0, x 2 > 0, . . . , x n > 0 and all α1 ≥ 0, α2 ≥ 0, . . . , αn ≥ 0 such that α1 + α2 + · · · + αn = 1 we have x 1α1 x 2α2 · · · x nαn ≤ α1 x 1 + α2 x 2 + · · · + αn x n . 2.11. Let ai be vectors in
n,
i = 1, . . . , m. Prove that the function
f (x) = ln is convex.
m X i=1
ehai ,xi
86
CHAPTER 2
2.12. Prove that if a convex function f has a nonempty and bounded level set Mβ = {x : f (x) ≤ β} for some β ∈ , then all level sets of f are bounded.
2.13. Prove that for a norm k · k♦ in that its polar cone has the form K ◦ = (λ, α) ∈
n
its epigraph K is a cone in
n
×
n+1 .
Prove
: α ≤ −kλk∗ ,
where k · k∗ is the dual norm. Use Example 2.94.
2.14. Prove that for a convex function f : n → its subdifferential is a monotone mapping, that is, hg 2 − g 1 , x 2 − x 1 i ≥ 0,
for all x 1 , x 2 ∈ dom f and all g 1 ∈ ∂ f (x 1 ), g 2 ∈ ∂ f (x 2 ). 2.15. A model represents the output variable, y ∈ y=
n X
, as a linear function,
xi u i ,
i=1
of input variables u 1 , . . . , u n . The quantities x 1 , . . . , x n are unknown model coefficients. We have N observations of input and output variables: (u j , y j ), j = 1, . . . , N. One way to determine the values of the coefficients x 1 , . . . , x n is to minimize the sum of the absolute errors: f (x) =
N n X j X j x i u i . y − j =1
i=1
Calculate the subdifferential of the function f at a point x. 2.16. For a vector x ∈
n
we define x [ j ], j = 1, . . . , n, as its ordered coordinates: x [1] ≥ x [2] ≥ · · · ≥ x [n] .
Prove that for any 1 ≤ k ≤ n the function f k (x) =
k X
j =1
x[ j ]
is convex and calculate its subdifferential. 2.17. Let X be a convex compact polyhedron in Prove that the function F : m → ,
n
F(λ) = max hλ, Axi x∈X
is convex. Calculate its subdifferential at a point λ.
and let A be an m × n matrix.
87
ELEMENTS OF CONVEX ANALYSIS
2.18. Assume that f : m → and let A be an m × n matrix. Derive the formula for the conjugate of the function g : n → given by g(x) = f (Ax).
2.19. Let A be a positive definite symmetric matrix of dimension n × n. Calculate the conjugate function to 1 f A (x) = hx, Axi. 2 Use this result to prove that if A and B are positive definite, and A − B is positive semidefinite, then B −1 − A−1 is positive semidefinite. 2.20. Let Z be a convex closed set in
n
and let
dist(x, Z ) = min kx − zk, z∈Z
where kx − zk is the Euclidean norm in
n.
Prove that the function
2 f (x) = kxk2 − dist(x, Z )
is convex. Hint: Represent f (x) using a conjugate of a convex function. 2.21. Derive Theorem 2.27 (the bipolar theorem) as a special case of the Fenchel– Moreau Theorem (Theorem 2.95). Hint: Use Example 2.96. 2.22. Let Z be a convex closed set in
n
and let
f (x) = dist(x, Z ) = min kx − zk♦ , z∈Z
where kx − zk♦ is a norm in
n.
Prove that the conjugate function f ∗ has the form
f ∗ (s) = σ Z (s) + δ B∗ (s), where σ Z (·) is the support function of Z , and δ B∗ (·) is the indicator function of the dual ball B∗ = {s ∈ n : ksk∗ ≤ 1}. Use this result to calculate the subdifferential of f .
Chapter Three Optimality Conditions 3.1 UNCONSTRAINED MINIMA OF DIFFERENTIABLE FUNCTIONS
In this chapter we analyze conditions that have to be satisfied at points which are local minima of optimization problems. Let f : n → , X ⊂ n , and let us consider the problem minimize f (x). x∈X
A point xˆ ∈ X is called a local minimum of this problem if there exists ε > 0 such that f (y) ≥ f (x) ˆ for all y ∈ X such that ky − xk ˆ ≤ ε. If f (y) ≥ f (x) ˆ for all y ∈ X, the point xˆ is called the global minimum. When X = n we discuss unconstrained local and global minima. From the classical multivariate calculus we know the following theorem. T HEOREM 3.1. Assume that f :
n
→
is differentiable at a point x. ˆ
(i) If f (·) attains its unconstrained local minimum at x, ˆ then ∇ f (x) ˆ = 0.
(3.1)
(ii) If f (·) is convex and (3.1) is satisfied, then xˆ is an unconstrained global minimum of f (·). Proof. (i) From the definition of the gradient, for all y ∈ f (y) = f (x) ˆ + h∇ f (x), ˆ y − xi + r(xˆ , y) where lim
y→ xˆ
r(x, ˆ y) = 0. ky − xk ˆ
If ∇ f (x) ˆ 6 = 0, then we consider the points y(τ ) = xˆ − τ ∇ f (x), ˆ τ > 0.
n
89
OPTIMALITY CONDITIONS
We obtain f (y(τ )) − f (x) ˆ = h∇ f (x), ˆ y(τ ) − xi ˆ + r(x, ˆ y(τ )) = −τ k∇ f (x)k ˆ 2 + r(x, ˆ y(τ )).
(3.2)
Since the rest r(xˆ , y(τ )) is infinitely smaller than ky(τ ) − xk, ˆ when τ → 0, we can find τ¯ > 0 such that for all τ ∈ (0, τ¯ ) 1 r(x, ˆ y(τ )) ˆ ≤ k∇ f (x)k. ky(τ ) − xk 2 ˆ
This can be equivalently expressed as r(x, ˆ y(τ )) ≤
1 τ k∇ f (x)k ˆ 2. 2
Substituting the last inequality into (3.2) we conclude that for all τ ∈ (0, τ¯ ) 1 ˆ 2 < 0. f (y(τ )) − f (x) ˆ ≤ − τ k∇ f (x)k 2
Therefore xˆ cannot be a local minimum. (ii) If f (·) is convex, then for all y ∈
n
we have
f (y) ≥ f (x) ˆ + h∇ f (x), ˆ y − xi = f (x). ˆ Therefore, xˆ is a global minimum. Points satisfying condition (3.1) are called stationary. Theorem 3.1 has many applications. Example 3.2. A real random variable X has the probability density function f (x, θ), where θ ∈ m is a vector of distribution parameters. We have collected n independent observations x 1 , . . . , x n of X. The maximum likelihood estimator θˆ of θ is obtained by maximizing (with respect to θ) the function
L(θ) = f (x 1 , θ) f (x 2 , θ) · · · f (x n , θ). Assume that the set of θ, for which all the factors above are positive, is nonempty. Since the function ln(·) is strictly increasing, θˆ is also the maximum of ln(L(θ)) =
n X
ln( f (x j , θ)).
j =1
ˆ we If the density function is differentiable with respect to the parameters θ at θ, conclude that n X ˆ ∇θ f (x j , θ) = 0. ˆ f (x j , θ) j =1
90
CHAPTER 3
For example, if f (·) is the density function of the normal distribution, f (x) =
2 1 − (x −µ) √ e 2σ 2 , σ 2π
and θ = (µ, σ ), we obtain − ln(L(θ)) =
n 1 X n ln(2π) + n ln(σ ) + (x j − µ)2 . 2 2σ 2 j =1
The minimum of this function is obtained at a point at which n ∂ − ln(L(θ)) 1 X =− 2 (x j − µ) = 0, ∂µ σ j =1
and
n ∂ − ln(L(θ)) n 1 X = − 3 (x j − µ)2 = 0. ∂σ σ σ j =1
These equations render v u X n u1 n 1X µˆ = x j , σˆ = t (x j − µ) ˆ 2. n n j =1
j =1
Example 3.3. A model represents the output variable, y ∈ y=
n X
, as a linear function,
xi u i ,
i=1
of input variables u 1 , . . . , u n . The quantities x 1 , . . . , x n are unknown model coefficients. We have N observations of the values of input and output variables: (u j , y j ), j = 1, . . . , N. One way to determine the values of the coefficients x 1 , . . . , x n is to minimize the sum of the squared errors: f (x) =
N X j =1
yj −
n X i=1
j 2
xi u i
.
Defining the matrix of input data
u 11 u2 1 U = . ..
u 1N
u 12 u 22 .. .
u 2N
··· ··· ···
u 1n u 2n .. , .
u nN
91
OPTIMALITY CONDITIONS
we obtain
f (x) = ky − U xk2 .
The gradient of f (·) has the form ∇ f (x) = 2U T (U x − y), and the best values of the coefficients can be calculated by solving the system of equations U T U x = U T y. If U has rank n, we obtain
xˆ = (U T U )−1 U T y.
If the rank of U is smaller than n, many minima exist. Example 3.4. Two points, a and b, are connected by a chain consisting of n mass points joined by extensible links (see Figure 3.1). Each point has mass m, and the spring constant of the links equals K . a
b
Figure 3.1. The chain with elastic links. To calculate the location of the points at equilibrium we use the principle of minimum energy. Let us denote by x i the location of the i th point, i = 1, . . . , n. Also, for the uniformity of the notation, we write x 0 = a, x n+1 = b. The potential energy of the link between the points i and i + 1 equals K2 kx i+1 − i x k2 . The potential energy of the i th mass point is equal to mghx i , ei, where e = (0, 1) and g is the gravitation constant. The total energy, therefore, has the form n n X K X i+1 kx − x i k2 + mg he, x i i. f (x 1 , . . . , x n ) = 2 i=1
i=1
92
CHAPTER 3
Differentiating with respect to x i we obtain: ∇x i f (x) = K (2x i − x i−1 − x i+1 ) + mge = 0, i = 1, . . . , n. This equation represents the equilibrium of forces. Setting z i = x i+1 − x i , i = 0, 1, . . . , n, we get i mge , i = 1, . . . , n. zi = z0 + K Therefore i (i − 1)mge , i = 1, . . . , n + 1, x i = a + i z0 + 2K and the value of z 0 can be easily found from the equation x n+1 = b. Finally, xi =
i i (n + 1 − i )mge n+1−i a+ b− , i = 1, . . . , m. n+1 n+1 2K
It follows that the horizontal positions of the mass points are equally spaced. As the second differences, z i − z i−1 , are constant, the mass points are located on a parabola. 3.2 UNCONSTRAINED MINIMA OF CONVEX FUNCTIONS
If a function f : n → is convex, all its local minima are in fact global minima. Indeed, suppose xˆ is a local minimum of f (·), and there exists a point y such that f (y) < f (x). ˆ Consider points on the segment connecting xˆ and y: z(τ ) = (1 − τ )xˆ + τ y, τ ∈ [0, 1]. The convexity of f (·) implies that f (z(τ )) < f (x) ˆ for τ ∈ (0, 1] and therefore xˆ cannot be a local minimum. It follows that conditions for a local minimum and a global minimum are identical. Using the concept of a subdifferential of f (·) we easily obtain a counterpart of Theorem 3.1 on page 88. T HEOREM 3.5. Let f : n → be a proper convex function. A point xˆ is a global minimum of f (·) if and only if 0 ∈ ∂ f (x). ˆ Proof. A point xˆ is a global minimum if and only if for all y f (y) ≥ f (x) ˆ + h0, y − xi, which is equivalent to the assertion.
93
OPTIMALITY CONDITIONS
This simple observation, together with our techniques for calculating subdifferentials, helps to analyze many applied problems involving distances and maxima of families of functions. Example 3.6. Let us return to Example 3.3, but with the sum of absolute errors as the measure of the quality of the model: f (x) = In more abstract terms,
N X n X j x i u i − y j . j =1 i=1
f (x) = kU x − yk1 . with
u 11 u2 1 U = . ..
u 1N
u 12 u 22 .. .
u 2N
··· ··· ···
1 T (u ) u 1n (u 2 )T u 2n .. = .. . .
u nN
(u N )T
.
The subdifferential of f (·) can be calculated using Lemma 2.83 and Example 2.78 (see also Exercise 2.15). In the typical case, when y 6= U x, ∂ f (x) = U T g : g ∈ ∂kz − yk, z = U x = U T g : kgk∞ = 1, hg, U x − yi = ky − U xk1 . At the optimal x, zero is an element of the subdifferential. We obtain 0 = U T g. ˆ
(3.3)
To express the conditions on gˆ introduce the sets J
+
n
j
= j:y > n
J− = j : y j < n
J0 = j : y j =
n X
o j xˆi u i ,
i=1 n X
o j xˆi u i .
i=1
n X
i=1
o j xˆi u i ,
It follows from the equation hg, U x − yi = ky − U xk1 that + −1 if j ∈ J , gˆ j = 1 if j ∈ J − , β j if j ∈ J 0 ,
94
CHAPTER 3
with β j ∈ [−1, 1]. This result can be transformed as follows. Using u j to denote the transpose of the j th row of U we can write conditions (3.3) as follows: X X X uj − uj = βj u j . j ∈J +
j ∈J −
j ∈J 0
Example 3.7. We have n continuous functions ϕi (t), i = 1, . . . , n, where t ∈ [a, b] ⊂ . Our objective is to approximate in a uniform way a continuous function ψ(t) by a linear combination of the basic functions,
n X i=1
x i ϕi (t) ≈ ψ(t), t ∈ [a, b],
where x i , i = 1, . . . , n, are unknown coefficients. The approximation error is defined as follows: n X F(x) = max x i ϕi (t) − ψ(t) . a≤t≤b
i=1
This approach is frequently referred to as Chebyshev approximation. In Example 2.88 on page 73 we calculated the subdifferential of F(·) at points x at which F(x) > 0: ϕ1 (t) n n X ϕ2 (t) o x i ϕi (t) − ψ(t) . , t ∈ Tˆ (x) , ∂ F(x) = conv sgn .. i=1
where
ϕn (t)
n X n o Tˆ (x) = t ∈ [a, b] : x i ϕi (t) − ψ(t) = F(x) i=1
and sgn(·) is the sign function,
−1 v < 0, sgn(v) = 0 v = 0, 1 v > 0.
Suppose xˆ is the best value of the coefficients x. Since the set of vectors of the form (ϕ1 (t), ϕ2 (t), . . . , ϕn (t)), t ∈ Tˆ (x), ˆ is closed, Theorem 3.5 implies that ϕ1 (t) n n X ϕ2 (t) o 0 ∈ conv sgn xˆi ϕi (t) − ψ(t) . , t ∈ Tˆ (x) ˆ . .. i=1 ϕn (t)
The set on the right hand side is convex and compact, and thus 0 is also a convex combination of at most n + 1 of its points (Lemma 2.7). It follows that there exist
95
OPTIMALITY CONDITIONS
points t1 , . . . , tm belonging to the set Tˆ (x), ˆ where m ≤ n + 1, and nonnegative coefficients αk , k = 1, . . . , m, totaling one, such that ϕ1 (tk ) m n X ϕ2 (tk ) X αk sgn xˆi ϕi (tk ) − ψ(tk ) . = 0. .. k=1
i=1
ϕn (tk )
Example 3.8. Let Y be a compact set in n . We want to find the smallest ball containing Y . Denoting by x the center of the ball, we see that our objective is to minimize the function f (x) = max kx − yk.
y∈Y
Let xˆ be the best location of the center. Define the set Yˆ = {y ∈ Y : kxˆ − yk = f (x)}. ˆ By Theorem 2.87,
[ ∂ f (x) ˆ = conv ∂kxˆ − yk . y∈Yˆ
We can calculate the subdifferentials of the norm by using Example 2.78 on page 63. In the only interesting case when f (x) ˆ > 0 (the set Y is not a singleton) the norm is differentiable and we obtain ∇kxˆ − yk = Hence
xˆ − y . kxˆ − yk
[ xˆ − y [ xˆ − y 0 ∈ conv = conv . kxˆ − yk kxˆ − yk y∈Yˆ
y∈Yˆ
By Theorem 2.44 and Lemma 2.7, we can choose no more than n + 1 of its extreme points such that 0 is their convex combination. It follows that there exist m ≤ n + 1 points y i ∈ Yˆ and nonnegative weights αi totaling 1, such that 0= Because kxˆ − the equation
yi k
m X i=1
αi
xˆ − y i . kxˆ − y i k
= f (x) ˆ > 0, we can multiply both sides by f (x) ˆ and we obtain xˆ =
m X
αi y i .
i=1
In other words, the sphere passing through the points y i encompasses Y , and the center of this sphere is an element of the m − 1 dimensional simplex spanned by these points.
96
CHAPTER 3
Example 3.9. We are given three convex closed sets Y1 , Y2 and Y3 in 2 . Our objective is to find a point x whose sum of distances to the given sets is minimal. Formally, we need to find the minimum of the function
f (x) = dist(x, Y1 ) + dist(x, Y2 ) + dist(x, Y3 ), where
dist(x, Y ) = min kx − yk. y∈Y
From Example 2.55 on page 46 we know that f (·) is convex, and Example 2.79 on page 64 provides us with the form of the subdifferential of each of the functions dist(x, Yi ). Suppose the optimal point xˆ is not an element of any of the given sets. Then each of the functions dist(x, Yi ) is differentiable with the gradient xˆ − 5Yi (x) ˆ . ∇ dist(x, ˆ Yi ) = kxˆ − 5Yi (x)k ˆ
The necessary condition of optimality reads 0=
3 X xˆ − 5Yi (x) ˆ . kxˆ − 5Yi (x)k ˆ i=1
Each of the three vectors in the sum above has length 1, and they sum to 0, which is possible only if they form an equilateral triangle. Consequently, the angles between the vectors 5Yi (x) ˆ − x, ˆ i = 1, 2, 3,
are all equal to 2π/3. If such a point does not exist, the optimal solution must be an element of one of the sets. Suppose xˆ ∈ Y1 , but xˆ ∈ / Y2 ∪ Y3 . Then, according to Example 2.79, ∂ dist(x, ˆ Y1 ) = NY1 (x) ˆ ∩ ∂kx − xk ˆ = NY1 (x) ˆ ∩ {g : kgk ≤ 1}. We obtain 0 ∈ ∂ f (x) ˆ = ∂ dist(x, ˆ Y1 ) + Equivalently,
and
xˆ − 5Y3 (x) ˆ xˆ − 5Y2 (x) ˆ + . kxˆ − 5Y2 (x)k ˆ kxˆ − 5Y3 (x)k ˆ
5Y2 (x) ˆ − xˆ 5Y3 (x) ˆ − xˆ + ∈ NY1 (x), ˆ k5Y2 (x) ˆ − xk k5Y3 (x) ˆ − xk ˆ ˆ
5 (x) 5Y3 (x) ˆ − xˆ
Y2 ˆ − xˆ
+
≤ 1. k5Y2 (x) ˆ − xk k5Y3 (x) ˆ − xk ˆ ˆ
Thus the angle between the vectors 5Y2 (x) ˆ − xˆ and 5Y3 (x) ˆ − xˆ is at least 2π/3, and its median is normal to Y1 . This is illustrated in Figure 3.2, which also shows
97
OPTIMALITY CONDITIONS
Y3
x Y1
Y2
Figure 3.2. The closest point to three sets. the geometrical interpretation of the optimality condition: x is the reflection point of a ray starting at a normal direction to Y2 , reflecting at Y1 and hitting Y3 from its normal cone. If no set Yi contains a point satisfying these conditions, the solution must be an element of an intersection of two of them. We leave to the reader the analysis of this case.
Optimality conditions for convex functions allow for an easy proof of a classical geometrical result, known as Helly’s Theorem. T HEOREM 3.10. Assume that I is a finite family of closed convex sets in n . If every n + 1 of them have a nonempty intersection, then they all have a nonempty intersection. Proof. Let Yi , i ∈ I , be the given sets and let dist(x, Yi ) = min kx − yk. y∈Yi
Consider the function f (x) = max dist(x, Yi ) . i∈I
It is convex, finite everywhere, and it converges to +∞, as kxk → ∞. Therefore f (·) has a minimal point x. ˆ Suppose the intersection of all sets is empty and thus f (x) ˆ > 0. Define Iˆ = {i ∈ I : dist(x, ˆ Yi ) = f (x)}. ˆ
98
CHAPTER 3
By Theorem 2.87 we have 0 ∈ ∂ f (x) ˆ = conv Since f (x) ˆ > 0, we obtain 0 ∈ conv
[ i∈ Iˆ
∂ dist(x, ˆ Yi ) ,
[ xˆ − 5 (x) Yi ˆ . kxˆ − 5Yi (x)k ˆ i∈ Iˆ
It follows from Lemma 2.7 that there exists a subfamily J ⊂ Iˆ of no more than n + 1 sets Yi such that [ xˆ − 5 (x) Yi ˆ 0 ∈ conv . kxˆ − 5Yi (x)k ˆ i∈J The right hand side of this expression is the subdifferential of the function f J (x) = max dist(x, Yi ) . i∈J
By virtue of Theorem 3.5 the point xˆ is its global minimizer. Since the subfamily J has cardinality at most n + 1, it must have a nonempty intersection, and thus \ xˆ ∈ Yi . i∈J
This contradicts the assumption that dist(x, ˆ Y i ) = f (x) ˆ > 0 for i ∈ Iˆ. For application of this theorem, see Exercise 3.18. 3.3 TANGENT CONES 3.3.1 Basic Concepts
If an optimization problem has constraints x ∈ X, the derivation of optimality conditions at a point xˆ must involve the analysis of ways to perturb xˆ while remaining in X. The fundamental concept in this analysis is that of a tangent direction. The most convenient way to define it is to reverse the perturbation operation and to consider sequences of points of X converging to x. ˆ
D EFINITION 3.11. A direction d is called tangent to the set X ⊂ n at the point x ∈ X if there exist sequences of points x k ∈ X and scalars τk > 0, k = 1, 2, . . . , such that τk ↓ 0 and d = lim
k→∞
xk − x . τk
99
OPTIMALITY CONDITIONS
It is implicit in this definition that x k → x, otherwise the limit above does not exists. L EMMA 3.12. Let X ⊂ n and let x ∈ X. The set TX (x) of all tangent directions for X at x is a closed cone. Proof. Suppose d ∈ TX (x). For every β > 0 we have βd = lim
k→∞
xk − x , (τk /β)
so the sequences {x k } and {τk /β} satisfy Definition 3.11 for the direction βd. Hence TX (x) is a cone. Let directions d j be tangent to X at x with the corresponding sequences {x j,k } and {τ j,k }, k = 1, 2, . . . , satisfying Definition 3.11, and let lim j →∞ d j = d. Since directions d j are tangent, for every j we can find k( j ) such that
Therefore
x j,k( j ) − x
− d j ≤ kd j − dk.
τ j,k( j )
x j,k( j ) − x
− d ≤ 2kd j − dk.
τ j,k( j )
It follows that the sequences {x j,k( j ) } and {τ j,k( j )}, j = 1, 2, . . . , satisfy Definition 3.11 for the direction d. Consequently, the cone TX (x) is closed. The notion of the tangent cone is illustrated in Figure 3.3. Tangent cones are crucial for developing optimality conditions for nonlinear optimization problems. In Theorem 3.24 on page 113 we establish that a local minimum xˆ of problem minimize f (x) x∈X
satisfies the relation ◦ −∇ f (x) ˆ ∈ TX (x) ˆ .
All optimality conditions, in one way or another, decipher this inclusion. This is the main motivation for the considerations of this section. In general, cones of tangent directions may be nonconvex, which makes the analysis of optimality conditions difficult. But we can still identify some
100
CHAPTER 3
x + TX (x)
X
x Figure 3.3. The tangent cone (translated to the point at which it is evaluated).
important cases when these cones are convex and we can provide their analytical description. Recall for convenience the definition of the cone of feasible directions at x ∈ X: n
KX (x) = {d ∈ L EMMA 3.13. Let X ⊂
n
: d = β(y − x), y ∈ X, β ≥ 0}.
be a convex set and let x ∈ X. Then TX (x) = KX (x).
Proof. Each d ∈ KX (x) is a tangent direction by definition. Moreover, KX (x) is a convex cone. Since the tangent cone is closed, KX (x) ⊂ TX (x). If these sets are not equal, we can find a direction h ∈ TX (x) \ KX (x). By the Separation Theorem 2.14 there exists y 6 = 0 such that hy, hi > 0 and hy, di ≤ 0 for all d ∈ KX (x). The direction h is tangent to X at x. If a sequence {x k } of points of X and a sequence τk ↓ 0 satisfy Definition 3.11 for the direction h, we obtain
1 1 hy, hi = y, lim (x k − x) = lim hy, x k − xi. k→∞ τk k→∞ τk
101
OPTIMALITY CONDITIONS
Each vector x k −x is an element of KX (x) and therefore hy, x k −xi ≤ 0. The last displayed equation renders hy, hi ≤ 0, which contradicts the postulated properties of the separating y. In applications, we encounter sets defined by an intersection: X = X1 ∩ X2 ∩ · · · ∩ Xm. For a point x ∈ X we always have TX (x) ⊂ TX 1 (x) ∩ TX 2 (x) ∩ · · · ∩ TX m (x) but an equality is not guaranteed. The main part of this section is devoted to conditions under which we obtain equality in this relation, for specific forms of the sets X i . Moreover, we calculate the polar cone [TX (x)]◦ in these cases. 3.3.2 Smooth Constraints. Metric Regularity
Feasible sets of nonlinear optimization problems are usually defined by systems of inequalities and equations: gi (x) ≤ 0, i = 1, . . . , m, h i (x) = 0, i = 1, . . . , p,
with gi : n → , i = 1, . . . , m, h i : n → , i = 1, . . . , p. Additionally, we may have abstract set constraints of the form x ∈ X 0 . In order to develop algebraic forms of tangent cones to such sets, it is convenient to consider an abstract system g(x) ∈ Y0 , x ∈ X 0.
(3.4)
Here g : n → m is continuously differentiable, Y 0 is a closed convex set in m , and X 0 is a closed convex set in n . For example, when Y0 = {y ∈ m : y ≤ 0}, system (3.4) has inequality constraints g i (x) ≤ 0, i = 1, . . . , m. When Y0 = {0} the system has equality constraints g i (x) = 0, i = 1, . . . , m. Combinations of equality and inequality constraints can be represented by defining Y0 as a product of half-lines and zeros. We use the symbol g 0 (x) to denote the Jacobian of the function g(·) at the point x: ∂g (x) ∂g (x) 1 1 . . . ∂g∂1x(x) ∂ x1 ∂ x2 n ∂g2 (x) ∂g2 (x) ∂x . . . ∂g∂2x(x) ∂ x 0 n 1 2 g (x) = .. .. .. . .. . . . . ∂gm (x) ∂gm (x) . . . ∂g∂mx(x) ∂ x1 ∂ x2 n
102
CHAPTER 3
Denote by X the set defined by system (3.4), X = {x ∈ X 0 : g(x) ∈ Y0 }, and consider a point x 0 ∈ X. For a direction d to be tangent to X at x 0 it is necessary that d ∈ TX 0 (x 0 ). Furthermore, when x 0 is perturbed in the direction d, then g(x 0 ) is perturbed (in first order terms) in the direction g 0 (x 0 )d. Thus it is also necessary that g 0 (x 0 )d ∈ TY0 (y0 ). Consequently, TX (x 0 ) ⊂ d ∈ n : d ∈ TX 0 (x 0 ), g 0 (x 0 )d ∈ TY0 (g(x 0 )) . We formalize this argument in Theorem 3.15 below. The last relation becomes an equation, if the sets X 0 and Y0 are convex closed polyhedra, and if the function g(·) is affine. Indeed, in this case the first order expansions are exact, and the tangent directions are feasible directions. In general, however, the equality in the relation above is not guaranteed, unless system (3.4) has an additional property, called metric regularity. Roughly speaking, it allows compensation of small perturbations in conditions (3.4) with small adjustments of x. D EFINITION 3.14. System (3.4) is called metrically regular at the point x 0 ∈ X if there exist ε > 0 and C such that for all e x and e u satisfying ke x − x 0 k ≤ ε and ke u k ≤ ε we can find x R ∈ X 0 satisfying the inclusion and such that
g(x R ) − e u ∈ Y0 ,
kx R − e x k ≤ C dist(e x , X 0 ) + dist(g(e x) − e u , Y0 ) .
(3.5)
The concept of metric regularity of a system g(x) = 0, x ∈ X 0 is illustrated in Figure 3.4. We prove in Theorem A.10 in the Appendix that metric regularity is equivalent to the following Robinson’s condition: 0 (3.6) g (x 0 )d − v : d ∈ K X 0 (x 0 ), v ∈ K Y0 (g(x 0 )) = m . Observe that the set on the left hand side of (3.6) is a cone, and thus an equivalent expression of Robinson’s condition is 0 ∈ int g 0 (x 0 )(x − x 0 ) − (y − g(x 0 )) : x ∈ X 0 , y ∈ Y0 .
The condition is clear, but the considerations associated with it are very technical. To avoid a long detour through the area of stability of systems of inclusions, we present these considerations in the Appendix. However,
103
OPTIMALITY CONDITIONS
x˜
xR g(x) = u
x0
g(x) = 0 X0
Figure 3.4. Metric regularity of the system. The distance of the perturbed point x˜ to a solution x R of the perturbed system is proportional to the violation of the two constraints.
it may be useful at this stage to notice that in the special case of equality constraints g(x) = 0, when X 0 = n and Y0 = {0}, metric regularity is equivalent to the linear independence of the gradients of the constraints ∇gi (x 0 ), i = 1, . . . , m. The relevance of the concept of metric regularity is demonstrated in the following theorem. T HEOREM 3.15. If system (3.4) is metrically regular, then TX (x 0 ) = d ∈ n : d ∈ TX 0 (x 0 ), g 0 (x 0 )d ∈ TY0 (g(x 0 )) .
(3.7)
Proof. Let us prove at first that every tangent direction d is a member of the set on the right hand side of (3.7). Since X ⊂ X 0 , the direction d is an element of TX 0 (x 0 ). By Definition 3.11, there exist points x k ∈ X and scalars τk ↓ 0 such that d = lim
k→∞
x k − x0 . τk
Set y k = g(x k ), y0 = g(x 0 ). We have y k = y0 + g 0 (x 0 )(x k − x 0 ) + ok ,
104
CHAPTER 3
with limk→∞ ok /τk = 0. Dividing the last displayed equation by τ k and passing to the limit, we obtain lim
k→∞
yk − y 0 = g 0 (x 0 )d. τk
As y k ∈ Y0 , we get g 0 (x 0 )d ∈ TY0 (y0 ). Summing up, the inclusion “⊂” is true in (3.7). We need to prove the opposite inclusion. Let d be a direction in the set on the right hand side of (3.7). Consider points of the form x(τ ) = x 0 + τ d, τ > 0. As d ∈ TX 0 (x 0 ), dist(x(τ ), X 0 ) = o1 (τ ),
(3.8)
with o1 (τ )/τ → 0, when τ ↓ 0. Also, g(x(τ )) = g(x 0 ) + τ g 0 (x 0 )d + o2 (τ ), with ko2 (τ )k/τ → 0, when τ ↓ 0. Since g 0 (x 0 )d ∈ TY0 (g(x 0 )), it follows that dist(g(x(τ )), Y0 ) ≤ ko2 (τ )k + dist(g(x 0 ) + τ g 0 (x 0 )d, Y0 ) = o3 (τ ), (3.9) with o3 (τ )/τ → 0, when τ ↓ 0. Consequently, the points x(τ ) “almost” belong to X 0 , and “almost” satisfy the constraint g(x) ∈ Y 0 . The errors are negligible with respect to τ . We can now use the property of metric regularity. Setting e x = x(τ ) and e u = 0 in Definition 3.14, we deduce that for sufficiently small τ > 0 we can find points x R (τ ) ∈ X such that kx R (τ ) − x(τ )k ≤ C dist(x(τ ), X 0 ) + dist(g(x(τ )), Y0 ) .
Using (3.8) and (3.9) we conclude that lim τ ↓0
kx R (τ ) − x 0 k = d. τ
Hence d is a tangent direction to X at x 0 . We can now easily develop algebraic forms of tangent cones to systems of equations and inequalities. Consider the system gi (x) ≤ 0, i = 1, . . . , m, h i (x) = 0, i = 1, . . . , p, x ∈ X 0,
(3.10)
105
OPTIMALITY CONDITIONS
with continuously differentiable functions g : n → m and h : n → p and with a closed convex set X 0 . We consider a point x 0 satisfying (3.10) and we define the set of active inequality constraints: I 0 (x 0 ) = {1 ≤ i ≤ m : gi (x 0 ) = 0}.
System (3.10) is a special case of system (3.4) with Y0 = {(y, 0) ∈
m
×
p
: yi ≤ 0, i ∈ I 0 (x 0 )}.
Robinson’s condition (3.6) takes on the form: 0 g (x 0 )d − v m 0 : d ∈ TX 0 (x 0 ), v ∈ , vi ≤ 0, i ∈ I (x 0 ) = h 0 (x 0 )d
m
×
p
.
(3.11) As the set on the left hand side is a cone, it is equivalent to require that 0 is an interior point of this set. A simpler sufficient condition of metric regularity can be obtained as follows. L EMMA 3.16. Assume that there exists a point x MF ∈ int X 0 such that h∇gi (x 0 ), x MF − x 0 i < 0, i ∈ I 0 (x 0 ), h∇h i (x 0 ), x MF − x 0 i = 0, i = 1, . . . , p,
(3.12)
and that the gradients ∇h i (x 0 ), i = 1, . . . , p, are linearly independent. Then system (3.10) is metrically regular at x 0 . Proof. As x MF is an interior point, there exists ε > 0 such that a ball about x MF of radius ε is also included in the interior of X 0 . Define B = {s ∈ n : ksk ≤ ε}. Then x MF − x 0 + B ⊂ K X 0 (x 0 ).
In view of (3.12) we can choose ε small enough (but positive) and a positive δ, so that h∇gi (x 0 ), x MF − x 0 + si < −δ, for all i ∈ I 0 (x 0 ), s ∈ B.
(3.13)
Since the gradients ∇h i (x 0 ), i = 1, . . . , p, are linearly independent, we have 0 ∈ int{h 0 (x 0 )s : s ∈ B}.
(3.14)
By choosing δ > 0 sufficiently small, we can ensure that a ball of radius δ is included in the set on the right hand side of (3.14). To formally verify Robinson’s condition (3.11), choose any (y, z) ∈ m × p such that k(y, z)k ≤ δ. By (3.14), we can find s ∈ B such that h 0 (x 0 )s = z. Then the second relation of (3.12) yields h 0 (x 0 )(x MF − x 0 + s) = z.
106
CHAPTER 3
Relation (3.13) implies that we can find v ≤ 0 such that g 0 (x 0 )(x MF − x 0 + s) − v = y.
This means that for the direction d = x MF − x 0 + s and for the selected v the corresponding element of the set at the left hand side of (3.11) equals (y, z). This can be done for all (y, z) such that k(y, z)k ≤ δ. Consequently, Robinson’s condition holds true and the system is metrically regular. We have to stress that the conditions of Lemma 3.16 are not equivalent to metric regularity, because we made a strong assumption that x MF is an interior point of X 0 . The assumptions of Lemma 3.16 for X 0 = n are known as the Mangasarian–Fromovitz constraint qualification condition. In this case it is equivalent to metric regularity. L EMMA 3.17. System (3.10) with X 0 = n satisfies the Mangasarian– Fromovitz constraint qualification condition at a point x 0 if and only if it is metrically regular at x 0 . Proof. In view of Lemma 3.10, it remains to prove that metric regularity implies the Mangasarian–Fromovitz condition. With no loss of generality we may assume that all inequality constraints are active at x 0 (otherwise, we could just ignore the inactive constraints). Metric regularity of (3.10) with X 0 = n is equivalent to Robinson’s condition: 0 g (x 0 )d − v n m , v∈ , v ≤ 0 = m × p. :d∈ h 0 (x 0 )d In particular, {h 0 (x 0 )d : d ∈ n } = p , and thus the gradients ∇h i (x 0 ) are linearly independent. Moreover, for each vector of the form (y, 0) ∈ m+ p it follows from Robinson’s condition that there exist d ∈ n and v ≤ 0 such that h∇gi (x 0 ), di = yi + vi , i = 1, . . . , m, h∇h i (x 0 ), di = 0, i = 1, . . . , p.
Choosing yi < 0, i = 1, . . . , m, we obtain the Mangasarian–Fromovitz condition. We end this section by calculating the polar of the tangent cone described in Theorem 3.15. Theorem 2.36 provides the formula: [TX (x 0 )]◦ = [TX 0 (x 0 )]◦ + [g 0 (x 0 )]T λ : λ ∈ [TY0 (g(x 0 ))]◦ . (3.15)
We use it in the analysis of optimality conditions for problems with feasible sets of the form (3.4).
107
OPTIMALITY CONDITIONS
3.3.3 Convex Nonsmooth Constraints
Tangent cones of sets defined by constraints involving nonsmooth convex functions can be described with the use of subdifferentials of the constraint functions. Consider the set X = {x ∈
n
: g(x) ≤ 0},
is a convex function. The key role is played here by where g : n → Slater’s condition: there exists x S such that g(x S ) < 0. T HEOREM 3.18. Assume that g(x 0 ) = 0, the function g is subdifferentiable at x 0 , and Slater’s condition is satisfied. Then [KX (x 0 )]◦ = cone(∂g(x 0 )). Proof. If d ∈ KX (x 0 ), then there exists τ¯ > 0 such that g(x 0 + τ d) ≤ 0 for all τ ∈ [0, τ¯ ]. Therefore g 0 (x 0 ; d) ≤ 0. By Lemma 2.75, g 0 (x 0 ; d) = max hs, di. s∈∂g(x 0 )
Hence hs, di ≤ 0 for all s ∈ cone(∂g(x 0 )). This is equivalent to d ∈ [cone(∂g(x 0 ))]◦ . Since d ∈ KX (x 0 ) was arbitrary, we have KX (x 0 ) ⊂ [cone(∂g(x 0 ))]◦ .
(3.16)
On the other hand, if d is an element of the set on the right hand side, then g 0 (x 0 ; d) = sup hs, di ≤ 0. s∈∂g(x 0 )
If g 0 (x 0 ; d) < 0, we have g(x 0 + τ d) < 0 for all sufficiently small τ > 0. Thus x 0 + τ d ∈ X for these τ and therefore d ∈ cone(X − x 0 ). If g 0 (x 0 ; d) = 0, we can represent d as the limit of directions d k = (1 − αk )d + αk (x S − x 0 ), αk ↓ 0,
where x S is the point satisfying Slater’s condition. We can estimate the directional derivative as follows: g 0 (x 0 ; d k ) = max hs, d k i s∈∂g(x 0 )
≤ (1 − αk ) max hs, di + αk max hs, x S − x 0 i ≤ (1 −
s∈∂g(x 0 ) αk )g 0 (x 0 ; d)
s∈∂g(x 0 )
+ αk g(x S ) < 0.
108
CHAPTER 3
It follows that g(x 0 + τ d k ) < 0 for all sufficiently small τ > 0 and thus d k is an element of KX (x 0 ). Consequently, d ∈ KX (x 0 ). Since d was an arbitrary element of the set on the right hand side of (3.23), we conclude that [cone(∂g(x 0 ))]◦ ⊂ KX (x 0 ).
(3.17)
Combining (3.16) and (3.17), we obtain KX (x 0 ) ⊂ [cone(∂g(x 0 ))]◦ ⊂ KX (x 0 ). Since the polar cone is always closed, taking the closure of all three sets above we conclude that [cone(∂g(x 0 ))]◦ = KX (x 0 ). We can now calculate the polar cone of both sides of this equation: [cone(∂g(x 0 ))]◦◦ = [KX (x 0 )]◦ = [KX (x 0 )]◦ . The set cone(∂g(x 0 )) is closed, and Theorem 2.27 implies that it is equal to the left side of the last equation. This yields the assertion of the theorem. Since KX (x 0 ) = TX (x 0 ), we obtain the following conclusion. C OROLLARY 3.19. Under the assumptions of Theorem 3.18 we have TX (x 0 ) = [cone(∂g(x 0 ))]◦ . These results can be used to analyze the case of finitely many convex inequalities and affine equations. Consider the set X = {x ∈ X 0 : gi (x) ≤ 0, i = 1, . . . , m, h i (x) = 0, i = 1, . . . , p}, where gi : n → , i = 1, . . . , m, are convex functions, h i : n → , i = 1, . . . , p, are affine functions, and X 0 is a closed convex set in n . Slater’s condition takes on the form: there exists a point x S ∈ X 0 such that gi (x S ) < 0, i = 1, . . . , m, h i (x S ) = 0, i = 1, . . . , p; additionally, x S ∈ int X 0 , if p > 0. We define the set of active inequality constraints I 0 (x 0 ) = {1 ≤ i ≤ m : gi (x 0 ) = 0}. In view of applications to optimality conditions, we are mainly interested in the polar to the tangent cone TX (x 0 ).
109
OPTIMALITY CONDITIONS
T HEOREM 3.20. Assume that Slater’s condition is satisfied. Then X [TX (x 0 )]◦ = [TX 0 (x 0 )]◦ + cone[∂gi (x 0 )] i∈I 0 (x 0 )
+ lin{∇h i (x 0 ), i = 1, . . . , p}.
(3.18)
Proof. Suppose p = 0 and consider the sets n
X i = {x ∈
: gi (x) ≤ 0}, i ∈ I 0 (x 0 ).
As the functions gi (·) are finite valued, they are continuous (Lemma 2.70 on page 55). Slater’s condition implies that x S ∈ int X i , i = 1, . . . , m. Hence \ x S ∈ X 0 ∩ int Xi . i∈I 0 (x 0 )
All sets X i , i = 0, 1, . . . , m, are closed and convex. We can thus use Lemma 2.40 to obtain X N X (x 0 ) = N X 0 (x 0 ) + N X i (x 0 ). i∈I 0 (x 0 )
By the definition of the normal cone on page 37 and by Lemma 3.13 we can rewrite this relation as X [TX (x 0 )]◦ = [TX 0 (x 0 )]◦ + [TX i (x 0 )]◦ . i∈I 0 (x 0 )
By virtue of Theorem 2.74 on page 59, the functions g i (·) are subdifferentiable everywhere. Applying Theorem 3.18 and Lemma 2.25(ii) on page 29 we conclude that [TX i (x 0 )]◦ = [K X i (x 0 )]◦ = cone[∂gi (x 0 )], i ∈ I 0 (x 0 ),
which yields the required result in this case. If equality constraints are present, we additionally define the set A = {x ∈
n
: h i (x) = 0, i = 1, . . . , p}.
In this case, Slater’s condition implies that \ x S ∈ int X 0 ∩ int X i ∩ A. i∈I 0 (x 0 )
We can employ Lemma 2.40 again to conclude that X N X (x 0 ) = N X 0 (x 0 ) + N X i (x 0 ) + N A (x 0 ). i∈I 0 (x 0 )
Using Lemma 3.13 again we obtain the assertion of the theorem.
110
CHAPTER 3
The tangent cone itself can be now calculated by using Theorem 2.35. C OROLLARY 3.21. Under the assumptions of Theorem 3.20, \ \ ◦ ◦ TX (x 0 ) = TX 0 (x 0 ) ∩ cone(∂gi (x 0 )) ∩ lin(∇h i (x 0 )) . i∈I 0 (x 0 )
1≤i≤ p
The interior of the set X 0 can be replaced by the relative interior, if necessary.
Example 3.22. Consider the set Sn− of symmetric negative semidefinite matrices of dimension n. It is a closed convex cone in the space Sn of all symmetric matrices of dimension n. We considered it in Example 2.31, where we developed the formula for its polar cone. Now it can be obtained in another way, which is slightly more complicated but quite instructive. The cone Sn− can be expressed algebraically as follows: Sn− = {A ∈ Sn : λmax (A) ≤ 0}, where λmax (A) denotes the maximum eigenvalue of the matrix A. As discussed in Example 2.59 on page 47, the maximum eigenvalue function λmax (·) is convex. The set Sn− satisfies Slater’s condition, because λmax (−I ) = −1. The subdifferential of the function λmax (·) has been calculated in Example 2.89 on page 74: ∂λmax (A) = conv{yy T : Ay = λmax (A)y, kyk = 1}. Suppose A is a boundary point of Sn− . Then λmax (A) = 0, and ∂λmax (A) = conv{yy T : Ay = 0, kyk = 1}. We can now apply Theorem 3.18 to obtain [K Sn− (A)]◦ = cone ∂λmax (A) = cone conv{yy T : Ay = 0, kyk = 1} .
The last set can be written in a more transparent way. By interchanging the operations of taking the convex hull and the conic extension we arrive at the expression [K Sn− (A)]◦ = conv cone{yy T : Ay = 0, kyk = 1} = conv{αyy T : Ay = 0, kyk = 1, α ≥ 0}.
Since αyy T =
√ √ T αy αy for α ≥ 0, we finally obtain
NSn− (A) = [K Sn− (A)]◦ = conv{yy T : Ay = 0}.
At A = 0 the normal cone is just the polar cone, ◦ NSn− (0) = Sn− ,
(3.19)
111
OPTIMALITY CONDITIONS
and thus
Sn−
◦
= conv{yy T : y ∈
n
(3.20)
}.
Each element of the set on the right hand side is a positive semidefinite matrix, because for all yi ∈ n and all αi ≥ 0, i = 1, . . . , k, we have
D for all d ∈
d,
k X
i=1
n.
αi (yi yiT )d
E
=
k X i=1
αi hyiT d, yiT di
=
k X i=1
αi kyiT dk2 ≥ 0,
On the other hand, every symmetric matrix Q can be represented as Q=
n X
λ j z j z Tj ,
j =1
where z 1 , . . . , z n are orthogonal eigenvectors of Q of length 1,and λ1 , . . . , λn are the corresponding eigenvalues. Q is positive semidefinite if and only if its eigenvalues are nonnegative. Therefore every positive semidefinite matrix Q is an element of the set on the right hand side of (3.20) (the fact that the eigenvalues do not total 1 does not matter here, because this set is a cone). Consequently, we obtain the following formula for the polar cone to the cone of negative semidefinite matrices: n ◦ S− = Sn+ . We have thus re-established the result of Example 2.31. Suppose A is not necessarily 0, but λmax (A) = 0. The normal cone (3.19) can also be derived as follows. We know from Example 2.21 on page 27 that TSn− (A) = Sn− − {τ A : τ ≥ 0}. It is a sum of cones, and thus its polar cone is the intersection of the corresponding polar cones (see Example 2.24 on page 28): ◦ ◦ ◦ NSn− (A) = TSn− (A) = Sn− ∩ {−τ A : τ ≥ 0} = Sn+ ∩ {H ∈ Sn : tr(H A) ≥ 0}.
In fact, for every H ∈ Sn+ we must have tr(H A) ≤ 0, and thus the last equation can also be written as NSn− (A) = Sn+ ∩ {H ∈ Sn : tr(H A) = 0}. This is the same as (3.19). By symmetry, at every A ∈ Sn+ , we also have the formula NSn+ (A) = Sn− ∩ {H ∈ Sn : tr(H A) = 0}. This formula is of importance for semidefinite programming, which we discuss later in Example 3.36.
112
CHAPTER 3
3.3.4 Smooth–Convex Constraints
Let us now consider the case when both smooth and nonsmooth (but convex) constraints are present: m − }.
X = {x ∈ X 0 : h(x) ∈ Y0 , g(x) ∈
(3.21)
We assume that the function h : n → p is continuously differentiable, the functions gi : n → , i = 1, . . . , m are convex, and that X 0 and Y0 are closed convex sets in n and p . The symbol m − denotes the nonpositive m orthant: {v ∈ : v j ≤ 0, j = 1, . . . , m}. Our objective is to describe the tangent cone to the set X defined by (3.21) at a point x 0 ∈ X in terms of the tangent cones to the sets X 0 and Y0 , the derivatives of h(·), and the subdifferentials of g i (·). Consider the set defined by the convex constraints, C = {x ∈ X 0 : gi (x) ≤ 0, i = 1, . . . , m}, and let I 0 (x) ˆ be the set of active constraints: I 0 (x) ˆ = {1 ≤ i ≤ m : gi (x) ˆ = 0}. T HEOREM 3.23. Assume that there exists x S ∈ X 0 such that gi (x S ) < 0, i = 1, . . . , m. Furthermore, let Robinson’s condition be satisfied: h 0 (x)K ˆ C (x) ˆ − K Y0 (h(x)) ˆ =
p
.
Then [TX (x)] ˆ ◦ = [TX 0 (x)] ˆ ◦ + [h 0 (x)] ˆ T λ : λ ∈ [TY0 (h(x))] ˆ ◦ X + cone(∂gi (x)). ˆ
(3.22)
i∈I 0 ( x) ˆ
Proof. By virtue of Theorem 3.20, TC (x) ˆ = TX 0 (x) ˆ ∩
\
[cone(∂gi (x))] ˆ ◦.
i∈I 0 ( x) ˆ
Under Robinson’s condition we obtain from Theorem 3.15 that TX (x) ˆ = d ∈ n : h 0 (x)d ˆ ∈ TY0 (h(x)), ˆ d ∈ TC (x) ˆ .
The polar cone can be calculated as follows. First, formula (3.15) yields: [TX (x)] ˆ ◦ = [TC (x)] ˆ ◦ + [h 0 (x)] ˆ T λ : λ ∈ [TY0 (h(x))] ˆ ◦ .
113
OPTIMALITY CONDITIONS
Second, formula (3.18) implies: [TC (x)] ˆ ◦ = [TX 0 (x)] ˆ ◦+
X
cone(∂gi (x)). ˆ
i∈I 0 ( x) ˆ
Combining the last two expressions we get the required expression for the polar of the tangent cone. Our system of constraints (3.21) encompasses, as special cases, all systems considered before. The representation of the polar to the tangent cone allows us to formulate necessary conditions of optimality for problems involving such combined constraints. Unfortunately, the verification of the constraint qualification conditions is very difficult in this case. Another way to derive the representation (3.22) of the polar to the tangent cone is to assume metric regularity of the system of constraints (3.21), as in the analysis of system (3.4). However, the constraint function (g, h) becomes nondifferentiable and nonconvex in this case. The derivation of the tangent cone then requires application of techniques of nonsmooth calculus, which are beyond the scope of this book. 3.4 OPTIMALITY CONDITIONS FOR SMOOTH PROBLEMS
Consider the constrained optimization problem minimize f (x), x∈X
(3.23)
and a set X ⊂ n . If its with a differentiable function f : n → solution xˆ is a boundary point of the feasible set X, the necessary condition of optimality formulated in Theorem 3.1 does not have to be satisfied. The main reason is that the perturbations of the point xˆ which take it out of the feasible set X are not allowed, and therefore they may correspond to a decrease of the objective function. In order to obtain necessary conditions of optimality, we restrict the set of possible perturbations to tangent directions at x. ˆ T HEOREM 3.24. Assume that xˆ is a local minimum of problem (3.23) and that f (·) is differentiable at x. ˆ Let TX (x) ˆ be the tangent cone to the set X at x. ˆ Then ◦ −∇ f (x) ˆ ∈ TX (x) ˆ . (3.24)
Conversely, if the function f (·) is convex, the set X is convex, and a point xˆ ∈ X satisfies relation (3.24), then xˆ is a global minimum of problem (3.23).
114
CHAPTER 3
Proof. Suppose our assertion is false: ◦ −∇ f (x) ˆ ∈ / TX (x) ˆ .
This means that there exists a direction d ∈ TX (x) ˆ such that h∇ f (x), ˆ di < 0.
(3.25)
As d is a tangent direction, there exists a sequence of points x k ∈ X convergent to xˆ and a sequence of scalars τ k ↓ 0 such that lim
k→∞
x k − xˆ = d. τk
(3.26)
Since f (·) is differentiable at x, ˆ f (x k ) − f (x) ˆ = h∇ f (x), ˆ x k − xi ˆ + αk , where αk /kx k − xk ˆ → 0, as k → ∞. Dividing both sides of the last equation by τk we obtain αk
x k − xˆ f (x k ) − f (x) ˆ = h∇ f (x), −d + . ˆ di + ∇ f (x), ˆ τk τk τk
Inequality (3.25) implies that kdk 6 = 0 and thus
αk kdk αk = lim = 0. k→∞ kx k − xk k→∞ τk ˆ lim
By virtue of (3.25) and (3.26) we conclude that lim
k→∞
f (x k ) − f (x) ˆ = h∇ f (x), ˆ di < 0. τk
On the other hand, all points x k are feasible and they approach x. ˆ Since xˆ is k a local minimum, f (x ) ≥ f (x) ˆ for all sufficiently large k. Hence lim inf k→∞
f (x k ) − f (x) ˆ ≥ 0, τk
a contradiction. Therefore, relation (3.24) is valid. Assume now that the function f (·) and set X are convex, and that (3.24) is satisfied at a point xˆ ∈ X. Since the set X is convex, for every y ∈ X the direction d = y − xˆ
115
OPTIMALITY CONDITIONS
is a tangent direction for X at xˆ (Lemma 3.13). Thus, condition (3.24) implies that h∇ f (x), ˆ y − xi ≥ 0. The function f (·) is convex and Theorem 2.67 yields f (y) ≥ f (x) ˆ + h∇ f (x), ˆ y − xi. Therefore f (y) ≥ f (x) ˆ for all y ∈ X, as required. If the function f (·) has only directional derivatives at x, ˆ then the necessary condition of optimality is f 0 (x; ˆ d) ≥ 0 for all d ∈ TX (x). ˆ
(3.27)
The proof of this condition can easily be obtained by following the argument of the proof of Theorem 3.24, after formula (3.26). We just need to use the directional derivative f 0 (x, ˆ d) instead of h∇ f (x), ˆ di. The development of necessary conditions of optimality for various classes of nonlinear optimization problems consists mainly in deciphering the fundamental relation (3.24) for different forms of feasible sets X. The formulas for the polar cone of the tangent cone to X at xˆ play the key role there. Let us consider the nonlinear optimization problem minimize f (x) subject to gi (x) ≤ 0, i = 1, . . . , m, h i (x) = 0, i = 1, . . . , p, x ∈ X 0.
(3.28)
We assume that the functions f : n → , gi : n → , i = 1, . . . , m, and h i : n → , i = 1, . . . , p, are continuously differentiable, and that the set X 0 ⊂ n is convex and closed. The feasible set of this problem is denoted by X. The optimality conditions of Theorem 3.24 for problem (3.28) involve the tangent cone to the feasible set X at the optimal point x. ˆ In Theorem 3.15 on page 103 we established that if the system of constraints of problem (3.28) is metrically regular at the point x, ˆ then the tangent cone to the feasible set X has the form TX (x) ˆ = d ∈ TX 0 (x) ˆ : h∇gi (x), ˆ di ≤ 0, i ∈ I 0 (x), ˆ (3.29) h∇h i (x), ˆ di = 0, i = 1, . . . , p .
As before, I 0 (x) ˆ is the set of active inequality constraints at x: ˆ I 0 (x) ˆ = {1 ≤ i ≤ m : gi (x) ˆ = 0}.
116
CHAPTER 3
Formula (3.29) is valid, if all constraint functions g i (·) and h i (·) are affine and the set X 0 is a convex polyhedron. In the nonlinear case, Robinson’s condition (3.11), which is equivalent to metric regularity, is sufficient for (3.29) to be true. For X 0 = n the Mangasarian–Fromovitz constraint qualification condition is sufficient (see Lemma 3.17 on page 106). Using Corollary 3.21 for smooth convex functions, we see that formula (3.29) is also valid if problem (3.28) satisfies Slater’s constraint qualification condition: the functions gi (·), i = 1, . . . , m are convex, the functions h i (·), i = 1, . . . , p are affine, there exists a feasible point x S such that gi (x S ) < 0, i = 1, . . . , m, and x S ∈ int X 0 if p > 0. If any of the sufficient conditions for (3.29) is satisfied, we say that problem (3.28) satisfies the constraint qualification condition. T HEOREM 3.25. Let xˆ be a local minimum of problem (3.28). Assume that at xˆ the constraint qualification condition is satisfied. Then there exist multipliers λˆ i ≥ 0, i = 1, . . . , m, and µˆ i ∈ , i = 1, . . . , p, such that 0 ∈ ∇ f (x) ˆ +
m X i=1
λˆ i ∇gi (x) ˆ +
p X i=1
µˆ i ∇h i (x) ˆ + N X 0 (x), ˆ
(3.30)
and λˆ i gi (x) ˆ = 0, i = 1, . . . , m.
(3.31)
Proof. By the constraint qualification condition, the cone TX (x) ˆ defined by (3.29) is the tangent cone to the feasible set X at x. ˆ Then, by virtue of Theorem 3.24, −∇ f (x) ˆ ∈ (TX (x)) ˆ ◦. It remains to describe the polar to the tangent cone. Assume for simplicity that I 0 (x) ˆ = {1, . . . , m 0 }. Robinson’s condition implies that the assumptions of Theorem 2.36 are satisfied with (∇g1 (x)) ˆ T .. . ˆ T (∇gm 0 (x)) p 0 A= K 1 = TX 0 (x), ˆ K2 = m , − × {0} . ˆ T (∇h 1 (x)) .. . (∇h p (x)) ˆ T
If the constraint functions gi (·) and h i (·) are affine and the set X 0 is a convex polyhedron, Theorem 2.36 holds true without the regularity assumption.
117
OPTIMALITY CONDITIONS
0 p Since K 2◦ = m , we deduce that there exist multipliers λˆ i ≥ 0, + × 0 i ∈ I (x), ˆ and µˆ i ∈ , i = 1, . . . , p, such that
−∇ f (x) ˆ ∈
X
λˆ i ∇gi (x) ˆ +
i∈I 0 ( x) ˆ
p X i=1
µˆ i ∇h i (x) ˆ + [TX 0 (x)] ˆ ◦.
For i ∈ / I 0 (x) ˆ we formally define λˆ i = 0. As [TX 0 (x)] ˆ ◦ = N X 0 (x), ˆ the last relation becomes identical with (3.30). The requirement that the multipliers λi are 0 for inactive constraints can be algebraically formulated as (3.31). Equations (3.30)–(3.31) generalize to the case of inequality constrained problems the classical Lagrange multiplier rule for problems with equality constraints only. The multipliers λˆ i , i = 1, . . . , m, and µˆ i , i = 1, . . . , p, in (3.30)–(3.31) are called Lagrange multipliers. Conditions (3.30)–(3.31) are called Karush–Kuhn–Tucker (KKT) conditions. ˆ x) L EMMA 3.26. Let xˆ be a local minimum of problem (3.28) and let Λ( ˆ be m p ˆ the set of Lagrange multipliers λ ∈ + and µˆ ∈ satisfying (3.30)–(3.31). ˆ x) (i) The set Λ( ˆ is convex and closed. ˆ x) (ii) If problem (3.28) satisfies Robinson’s condition at x, ˆ then the set Λ( ˆ is also bounded. ˆ µ) Proof. The set N X 0 (x) ˆ is convex and closed, and thus the set of ( λ, ˆ satisfying (3.30) is convex and closed. Conditions (3.31) are linear equations ˆ and thus assertion (i) is true. in λ, Assume that problem (3.28) satisfies Robinson’s condition (3.11) at xˆ and ˆ µ) ˆ x). that (λ, ˆ ∈ Λ( ˆ Then we can choose d ∈ K X 0 (x) ˆ such that h∇gi (x), ˆ di ≤ −1, i ∈ I 0 (x), ˆ ( −1 if µˆ i ≥ 0, h∇h i (x), ˆ di = 1 if µˆ i < 0,
i = 1, . . . , p.
It follows from (3.30) that there exists z ∈ N X 0 (x) ˆ such that 0 = ∇ f (x) ˆ +
m X i=1
λˆ i ∇gi (x) ˆ +
Multiplying both sides by d we find that 0 ≤ h∇ f (x), ˆ di −
X
λˆ i −
i∈I 0 ( x) ˆ
p X
µˆ i ∇h i (x) ˆ + z.
p X
|µi | + hz, di.
i=1
i=1
118
CHAPTER 3
As hz, di ≤ 0, X
λˆ i +
i∈I 0 ( x) ˆ
p X i=1
|µi | ≤ h∇ f (x), ˆ di.
This, combined with the nonnegativity condition λˆ ≥ 0, proves the boundˆ x). edness of the set of multipliers Λ( ˆ Let us introduce the Lagrangian associated with the constrained problem: L(x, λ, µ) = f (x) +
m X i=1
λi gi (x) +
p X
µi h i (x).
(3.32)
i=1
Using the Lagrangian, condition (3.30) can be written compactly as ˆ µ) −∇x L(x, ˆ λ, ˆ ∈ N X 0 (x). ˆ
(3.33)
Thus the necessary condition of Theorem 3.24 for a minimum of the function ˆ µ) L(x, λ, ˆ over x ∈ X 0 is satisfied at x. ˆ We explore this observation in Theorem 3.27 below and in Chapter 4. T HEOREM 3.27. Assume that the functions f (·) and g i (·), i = 1, . . . , m, are convex and the functions h i (·), i = 1, . . . , p, are affine. If the point xˆ ∈ X and multipliers λˆ i ≥ 0, i = 1, . . . , m, and µˆ i ∈ , i = 1, . . . , p, satisfy conditions (3.30)–(3.31), then xˆ is a global minimum of problem (3.28). ˆ µ) Proof. It follows from the assumptions that the Lagrangian L(x, λ, ˆ is convex with respect to x. By (3.33) and the second part of Theorem 3.24, ˆ µ) ˆ µ) L(x, ˆ λ, ˆ ≤ L(x, λ, ˆ for all x ∈ X 0 . At feasible points of problem (3.28) we have ˆ µ) L(x, λ, ˆ ≤ f (x), and at the point xˆ condition (3.31) implies that ˆ µ). f (x) ˆ = L(x, ˆ λ, ˆ Hence f (x) ˆ ≤ f (x) for all x ∈ X. Example 3.28. Let A be an m × n matrix of rank m and let b ∈ set of points in n satisfying the equation
Ax = b.
m.
Consider the
119
OPTIMALITY CONDITIONS
It is a linear manifold. Our objective is to calculate the projection of a point z ∈ n on this manifold (see Section 2.1.2 on page 20). The projection is the solution of the problem
minimize kx − zk2
subject to Ax − b = 0. It has form (3.28) with and
f (x) = kx − zk2
h i (x) = hai , xi − bi , i = 1, . . . , m,
where ai denotes the i th row of A. The constraints are affine and therefore the constraint qualification condition is satisfied. Introducing the vector of Lagrange multipliers µ ∈ m , we can write the necessary condition of optimality (3.30) as follows:
2(x − z) + A Tµ = 0.
(3.34)
Let us multiply both sides of this equation by A. Using the fact that Ax = b we obtain A A Tµ = 2(Az − b). The matrix A has row rank m and thus A A T is a nonsingular matrix. Therefore µ = 2(A A T )−1 (Az − b). Substitution into (3.34) renders x =z−
1 T A µ = z − A T (A A T )−1 (Az − b). 2
The reader can verify directly that Ax = b and that the vector z − x is perpendicular to the subspace {d : Ad = 0}. Thus the necessary and sufficient conditions for the projection, as stated in Lemma 2.11 on page 21, are satisfied. Example 3.29. Let Q be a symmetric matrix of dimension n. Consider the quadratic form f (x) = hx, Qxi
on the unit sphere
S = {x ∈
n
: kxk = 1}.
To find the minimum value of the quadratic form on the sphere, we formulate the problem minimize hx, Qxi
subject to 1 − kxk2 = 0.
120
CHAPTER 3
The gradient of the constraint function is equal to −2x. It is never zero for x ∈ S and therefore Robinson’s condition is satisfied. The optimal solution of the problem must satisfy the necessary conditions (3.30)–(3.31). Introducing the Lagrange multiplier µ associated with the equality constraint, we can write (3.30) as follows: 2Qx − 2µx = 0, or simply as Qx = µx. It follows that x is an eigenvector of Q having length 1, and µ is the corresponding eigenvalue. In order to find the best eigenvector, we calculate the value of the objective function: hx, Qxi = hx, µxi = µ. Therefore the optimal solution is an eigenvector of length 1 corresponding to the smallest eigenvalue of Q. This is illustrated in Figure 3.5.
S z1
z2
Figure 3.5. The minimum and maximum of a quadratic form on a sphere are attained at eigenvectors ±z 1 and ±z 2 . The largest value of the quadratic form hx, Qxi on the sphere can be calculated by minimizing hx, (−Q)xi in S. The solution is an eigenvector of Q corresponding to the largest eigenvalue, and the maximum value of the form is equal to the largest eigenvalue. Example 3.30. Two points, a and b, are connected by a chain consisting of n mass points joined by extensible links. Each point has mass m, and the spring constant of the links equals K . We considered this problem in Example 3.4 on page 91, where we calculated the location of the mass points at equilibrium.
121
OPTIMALITY CONDITIONS
Let us assume now that the point b is located on a horizontal board, which constitutes an obstacle for the chain, and the point a is above the board (a 2 > b2 ), as illustrated in Figure 3.6. No friction exists between the board and the mass points. a
b
Figure 3.6. The chain with elastic links and a horizontal obstacle. All mass points have to remain on or above the board, that is, their locations have to satisfy the inequalities x 2i ≥ b2 , i = 1, . . . , n. Our goal is to calculate the location of the mass points at equilibrium. In Example 3.4 we derived the expression for the potential energy of the mass points: n n X K X i+1 i 2 1 n kx − x k + mg he, x i i, f (x , . . . , x ) = 2 i=1
x0
i=1
x n+1
with the convention that = a and = b, and with e = (0, 1). The problem of minimizing the potential energy in the presence of an obstacle can be formulated as follows: minimize
n n X K X i+1 kx − x i k2 + mg he, x i i 2 i=1
i=1
subject to b2 − he, x i i ≤ 0, i = 1, . . . , n. The constraint functions gi (x) = b2 − he, x i i, i = 1, . . . , n,
are affine, and therefore the constraint qualification condition is satisfied. Let us introduce Lagrange multipliers λi ≥ 0, i = 1, . . . , n. Condition (3.30) has the form ∇ f (x) +
n X i=1
λi ∇gi (x) = 0.
122
CHAPTER 3
Observe that each constraint function gi (·) depends on x i only, and thus the last equation can be written for each mass point separately: ∇x i f (x) + λi ∇x i gi (x) = 0, i = 1, . . . , n. Substituting the formulas for f (·) and gi (·) and differentiating we obtain K (2x i − x i−1 − x i+1 ) + (mg − λi )e = 0, i = 1, . . . , n.
(3.35)
The complementarity conditions (3.31) have the form λi (b2 − he, x i i) = 0, i = 1, . . . , n.
(3.36)
In general, solving systems like the last one requires considering 2 n cases, for all possible sets I of active constraints. For each set I we require that λi = 0 for i ∈ / I and that the inequality constraints are satisfied as equations for i ∈ I . In this example we can drastically reduce the number of cases to be considered, by employing our understanding of the physical nature of the problem. Equation (3.35) for the first coordinate, x 1i , i = 1, . . . , n, reads x 1i+1 − x 1i = x 1i − x 1i−1 , i = 1, . . . , n. That is, the mass points are horizontally equally spaced. The vertical locations are more difficult to calculate. Common sense suggests that if a mass point j is on the board, then all mass points j +1, . . . , n should remain on the board as well. Formally, we can derive this as follows. Equation (3.35) for the second coordinate has the form: K x 2i+1 − x 2i = K x 2i − x 2i−1 + mg − λi , i = 1, . . . , n. (3.37) j +1
j
> b2 . From (3.36) we obtain λ j +1 = 0. The last Suppose x 2 = b2 , but x 2 equation for i = j + 1 implies: j +2
x2
j +1
− x2
j +1
= x2
j
− x2 +
mg > 0. K
This propagates to the following points and we get x 2i+1 > x 2i for all i = j, . . . , n. This contradicts the equation x 2n+1 = b2 . Consequently, all points i = j, . . . , n are located on the straight line between x j and b and are equally spaced. Let us denote by j ∗ the first mass point that lies on the board. It follows that i x 2 > b2 for i = 1, . . . , j ∗ − 1, and x 2i = b2 for i = j ∗ , . . . , n. Equation (3.37) for i = j ∗ implies that λ j∗ mg j∗ j ∗ −1 = ≥ 0. x2 − x2 + (3.38) K K Since λi = 0 for i = 1, . . . , j ∗ − 1, equation (3.37) for i = 1, . . . , j ∗ − 1 yields x 2i+1 − x 2i = x 21 − x 20 +
i mg . K
(3.39)
123
OPTIMALITY CONDITIONS
Combining the last relation for i = j ∗ − 1 with (3.38), after simple manipulations we conclude that j ∗mg . (3.40) x 21 − x 20 ≥ − K Summing equations (3.39) for i = 1, . . . , j ∗ − 1 we obtain ( j ∗ − 1) j ∗ mg . b2 − a2 = ( j ∗ − 1) x 21 − x 20 + 2K
Therefore, j ∗ is the smallest index for which the inequality
j ∗ ( j ∗ − 1)mg ≤ b2 − a2 ( j ∗ − 1) x 21 − x 20 + 2K
is satisfied. Dividing by j ∗ − 1 we obtain x 21 − x 20 ≤ Because of (3.40),
j ∗ mg b2 − a 2 − . ∗ j −1 2K
j ∗ mg a2 − b 2 ≤ . j∗ − 1 2K
The left hand side of the above inequality is a decreasing function of j ∗ , while the right hand side in an increasing function. The critical index j ∗ is the smallest integer in {1, . . . , n}, for which this inequality is satisfied. If such an integer does not exist, all mass points are above the board. If j ∗ can be found, the mass points j ∗ , . . . , n lie on the board and are equally spaced, and the mass points 1, . . . , j ∗ − 1 can be ∗ found from the unconstrained problem with ends at a and x j , as in Example 3.4. The solution, for the same data as in Example 3.4, is displayed in Figure 3.6. Let us observe that the active constraints in the constrained problem cannot be determined by verifying which constraints are violated by the unconstrained solution. The second mass point is below the board level in the unconstrained solution, but the corresponding constraint is not active at the optimal solution of the constrained problem. If the point b is located above the obstacle, we also have to determine the last mass point lying on the board. We leave to the reader the analysis of this case. If the obstacle is not horizontal, the solution method is similar, but more complicated. The main property, though, that there will be two key mass points – the first and the last one on the obstacle – remains valid. Example 3.31. Consider the unconstrained optimization problem minimize f (x), x∈
with the function f :
n
→
n
defined by f (x) = max
i=1,...,N
f i (x).
(3.41)
124
CHAPTER 3
We assume that each f i : n → , i = 1, . . . , N, is continuously differentiable. The necessary condition of optimality for this problem can be developed from optimality conditions for constrained optimization problems. Let us consider the problem minimize v (3.42) subject to f i (x) − v ≤ 0, i ∈ 1, . . . , N,
with decision variables x ∈ n and v ∈ . For every x ∈ n the minimum value of v is equal to f (x), and therefore this problem is equivalent to (3.41). Any feasible point (x, v) of problem (3.42) satisfies the Mangasarian–Fromovitz constraint qualification condition with the direction d = (0, 1) ∈ n × . Denote by (x, ˆ v) ˆ the optimal solution of (3.42). Optimality conditions (3.30)– (3.31) have the form
X N 0 ∇ f i (x) ˆ + λi = 0, 1 −1
(3.43)
i=1
λi ( fi (x) ˆ − v) ˆ = 0, λi ≥ 0, i = 1, . . . , N.
(3.44)
At the optimal solution we must have vˆ = f (x) ˆ and therefore condition (3.44) means that λi = 0, if f i (x) ˆ < f (x). ˆ The first n rows in equation (3.43) have the form 0= The last row reads
N X i=1
N X i=1
λi ∇ fi (x). ˆ
λi = 1.
Therefore, zero is a convex combination of the gradients ∇ f i (x) ˆ of these functions f i (·), for which f i (x) ˆ = f (x) ˆ (active pieces).
Example 3.32. We can now apply the results of Example 3.31 to a nonlinear version of the approximation problem of Example 3.3. A model represents the output variable, y ∈ , as a function
y = ϕ(u, x) of input variables u 1 , . . . , u m . Here ϕ is a function from m × n to . The quantities x 1 , . . . , x n are unknown model coefficients. We have N observations of values of input variables and the corresponding values of output variables: (u j , y j ), j = 1, . . . , N. One way to determine the values of the coefficients x 1 , . . . , x n is to minimize the largest deviation between the values observed and those predicted by the model: f (x) = max y j − ϕ(u j , x) .
1≤ j ≤N
125
OPTIMALITY CONDITIONS
Such uniform approximation (Chebyshev approximation) has already been discussed in Example 3.7 on page 94 (for a model linear in x and for a continuum of “observations”). Assume that for each u j the function ϕ(·, u j ) is smooth. Then we can represent f (x) as the maximum of 2N smooth functions f j (x) = y j − ϕ(u j , x),
f j (x) = − f j −N (x),
j = 1, . . . , N,
j = N + 1, . . . , 2N.
We focus on the most interesting case, when the smallest distance is positive (the perfect match is impossible). Then f (x) > 0 for all x. It is apparent that if f j (x) = f (x) then − f j (x) 6= f (x). Let us define the set Jˆ(x) ˆ = {1 ≤ j ≤ N : y j − ϕ(u j , x) = f (x)}. ˆ
Optimality conditions of Example 3.31 imply that there exist a subset A of no more than n + 1 indices from Jˆ(x) ˆ and nonnegative coefficients α j , j ∈ A, totaling one and such that X α j sgn y j − ϕ(u j , x) ∇x ϕ(u j , x) ˆ = 0. j ∈A
This condition extends to the nonlinear case the conditions obtained in Example 3.7. Of course, here we are comparing the model and the reference outcomes y j at finitely many points, while in Example 3.7 we considered the maximum over an interval. When the model is nonlinear in parameters x, the maximum function is nonconvex and nonsmooth, and our techniques only allow us to analyze the case of finitely many points. In Figure 1.4 on page 14 we illustrate the optimal approximation of 21 data points with a model having 5 parameters. As predicted, 6 data points determine the error of the approximation. 3.5 OPTIMALITY CONDITIONS FOR CONVEX PROBLEMS
If the objective function or the constraint functions in a nonlinear optimization problem are convex, but not necessarily differentiable, we use their subgradients in the necessary conditions of optimality. Although the results of this section can be derived as special cases of the results of the following one, we analyze the convex case separately for educational reasons, and because of wide applications. At first we establish the counterpart of the fundamental relation (3.24) from page 113 and of Theorem 3.5 from page 92. Consider the constrained optimization problem minimize f (x),
(3.45)
x∈X
with a convex function f :
n
→
and a convex set X ⊂
n
.
126
CHAPTER 3
T HEOREM 3.33. Assume that xˆ is a local minimum of problem (3.45) and that there exists a point x 0 ∈ X such that f is continuous at x 0 . Then there exists s ∈ ∂ f (x) ˆ such that −s ∈ N X (x). ˆ
(3.46)
Conversely, if a point xˆ ∈ X satisfies relation (3.46) for some s ∈ ∂ f (x), ˆ then xˆ is a global minimum of problem (3.45). Proof. Consider the indicator function ( 0 if x ∈ X, δ X (x) = +∞ otherwise. Problem (3.45) is equivalent to the unconstrained optimization problem f (x) + δ X (x). minimize n x∈
If f (·) is continuous at x 0 ∈ X then Theorem 2.85 on page 68 renders ∂ f (x) ˆ + δ X (x) ˆ = ∂ f (x) ˆ + ∂δ X (x). ˆ
By Theorem 3.5 on page 92, xˆ is a minimum if and only if 0 ∈ ∂ f (x) ˆ + δ X (x) ˆ .
Combining the last two relations we see that there must exist a subgradient s ∈ ∂ f (x) ˆ such that −s ∈ ∂δ X (x). ˆ Using Example 2.81 on page 67 we note that ∂δ X (x) ˆ = N X (x), ˆ and formula (3.46) holds true. To prove the opposite implication, suppose (3.46) is satisfied for some s ∈ ∂ f (x). ˆ Then, for every y ∈ X, f (y) ≥ f (x) ˆ + hs, y − xi. ˆ Since y − xˆ ∈ KX (x) ˆ and −s ∈ N X (x), ˆ hs, y − xi ˆ ≥ 0. Therefore f (y) ≥ f (x) ˆ for all y ∈ X.
127
OPTIMALITY CONDITIONS
This general fact helps us to obtain optimality conditions for the following problem: minimize f (x) subject to gi (x) ≤ 0, i = 1, . . . , m, h i (x) = 0, i = 1, . . . , p, x ∈ X 0.
(3.47)
We assume that the functions f : n → , and gi : n → , i = 1, . . . , m, are convex, and the functions h i : n → , i = 1, . . . , p, are affine. The set X 0 is convex and closed. Recall from page 109 that Slater’s constraint qualification condition for convex constraints takes on the form: There exists a point x S ∈ X 0 such that gi (x S ) < 0, i = 1, . . . , m, h i (x S ) = 0, i = 1, . . . , p, and x S ∈ int X 0 , if p > 0. In Theorem 3.20 on page 109 we established that under this condition N X (x) ˆ = [K X 0 (x)] ˆ + ◦
X
cone[∂gi (x)] ˆ +
i∈I 0 ( x) ˆ
p X
lin[∇h i (x)]. ˆ
i=1
(3.48)
Here I 0 (x) ˆ is the set of i ∈ {1, . . . , m} for which g i (x) ˆ = 0. This can be combined with Theorem 3.33 to obtain the following optimality conditions for problem (3.47). T HEOREM 3.34. Assume that xˆ is the minimum of problem (3.47), the function f (·) is continuous at some feasible point x 0 , and Slater’s condition is ˆ ∈ p such that satisfied. Then there exist λˆ ∈ m + and µ 0 ∈ ∂ f (x) ˆ + and
m X i=1
λˆ i ∂gi (x) ˆ +
p X i=1
µˆ i ∇h i (x) ˆ + N X 0 (x) ˆ
λˆ i gi (x) ˆ = 0, i = 1, . . . , m.
(3.49)
(3.50)
Conversely, if for some feasible point xˆ of (3.47) and some λˆ ∈ m + and µˆ ∈ p conditions (3.49)–(3.50) are satisfied, then xˆ is the global minimum of problem (3.47). Proof. Using Theorem 3.33, the formula (3.48) for the normal cone, and the equation [K X 0 (x)] ˆ ◦ = N X 0 (x), ˆ we see that there exists s ∈ ∂ f (x) ˆ such that −s ∈ N X 0 (x) ˆ +
X
cone[∂gi (x)] ˆ +
i∈I 0 ( x) ˆ
p X i=1
lin[∇h i (x)]. ˆ
128
CHAPTER 3
This means that there exist λˆ i ≥ 0, i ∈ I 0 (x), ˆ and µˆ ∈ X
−s ∈ N X 0 (x) ˆ +
λˆ i ∂gi (x) ˆ +
i∈I 0 ( x) ˆ
p X i=1
p
such that
µˆ i ∇h i (x). ˆ
Setting λˆ i = 0 for i ∈ / I 0 (x) ˆ we obtain (3.49)–(3.50). To prove the converse, assume (3.49)–(3.50). This implies that there exist subgradients s ∈ ∂ f (x), ˆ si ∈ ∂gi (x), ˆ i ∈ I 0 (x), ˆ and some v ∈ N X 0 (x) ˆ such that 0=s+
X
λˆ i si +
i∈I 0 ( x) ˆ
p X i=1
µˆ i ∇h i (x) ˆ + v.
Suppose y is a feasible point of problem (3.47). Evaluating the scalar product of y − xˆ with both sides of the last equation we obtain: 0 = hs, y − xi ˆ +
X
λˆ i hsi , y − xi ˆ +
i∈I 0 ( x) ˆ
p X i=1
µˆ i h∇h i (x), ˆ y − xi ˆ + hv, y − xi. ˆ
Since y is feasible, y − xˆ ∈ K X 0 (x). ˆ Thus hv, y − xi ˆ ≤ 0, because v ∈ N X 0 (x). ˆ Moreover, gi (y) ≤ 0 and therefore for every i ∈ I 0 (x) ˆ we have hsi , y − xi ˆ ≤ gi (y) − gi (x) ˆ ≤ 0. Finally, the feasibility of y implies that h∇h i (x), ˆ y − xi ˆ = 0, i = 1, . . . , p. The last three displayed equations yield the inequality hs, y − xi ˆ ≥ 0. This means that f (y) ≥ f (x), ˆ as postulated. Again, using the Lagrangian, L(x, λ, µ) = f (x) +
m X i=1
λi gi (x) +
p X
µi h i (x),
i=1
we can write the necessary condition of optimality (3.49) as ˆ µ) 0 ∈ ∂x L(x, ˆ λ, ˆ + N X 0 (x), ˆ which is nothing else but the necessary and sufficient condition of optimality ˆ µ) for the problem of minimizing L(x, λ, ˆ over x ∈ X 0 .
129
OPTIMALITY CONDITIONS
Example 3.35. Given a matrix A of dimension m × n and vectors c ∈ b ∈ m , the linear programming problem is formulated as follows:
n
and
minimize hc, xi subject to Ax = b, x ≥ 0.
We see that the problem has form (3.47) with f (x) = hc, xi h i (x) = bi − hai , xi, i = 1, . . . , m, X0 =
n +.
The constraints are affine, and therefore formula (3.48) for the normal cone to the feasible set remains valid. We can use Theorem 3.34. Condition (3.49) has (at the optimal solution x) ˆ the form c−
m X i=1
µi ai ∈ −[T
n +
(x)] ˆ ◦.
We know from Example 2.21 on page 27 that T
n +
(x) ˆ =
n +
+ {τ xˆ : τ ∈
}.
Thus, by virtue of Example 2.24 on page 28, [T
n +
(x)] ˆ ◦=N
n +
(x) ˆ =
n −
∩ {d ∈
Setting u = −d, we conclude that there exist µ ∈
n
m
: hd, xi ˆ = 0}. and u ∈
n +
such that
c − A Tµ = u, hu, xi ˆ = 0.
These relations can also be derived from direct linear algebraic considerations, but it is instructive to see them in a more general framework. Example 3.36. Given symmetric matrices A 1 , . . . , Am and C in Sn the semidefinite programming problem is formulated as follows: minimize tr(C X) subject to tr(Ai X) = bi , i = 1, . . . , m, X ∈ Sn+ . The last condition means that the decision variable X in this problem is a symmetric positive semidefinite matrix.
130
CHAPTER 3
We see that the problem has form (3.47) with f (X) = tr(C X)
h i (X) = bi − tr(Ai X), i = 1, . . . , m, X 0 = Sn+ . The functions f (·) and h i (·) are affine. Assume Slater’s constraint qualification condition (see page 109). In our terms it means that there exists a positive definite matrix X S satisfying the equality constraints. Under this condition we can use Theorem 3.34. Condition (3.49) has (at the optimal solution Xˆ ) the form C−
m X i=1
µi Ai ∈ −NSn+ ( Xˆ ).
We know from Example 3.22 on page 110 that NSn+ ( Xˆ ) = Sn− ∩ {D ∈ Sn : tr(D Xˆ ) = 0}. Setting U = −D, we conclude that there exists a vector µ ∈ matrix U of dimension n such that C−
m X i=1
µi Ai = U,
U ∈ Sn+ ,
m
and a quadratic
(3.51)
tr(U Xˆ ) = 0. These conditions are formally similar to the optimality conditions in linear programming, as presented in Example 3.35. However, their derivation requires application of the formulas for the tangent cone to the positive semidefinite cone S n+ and for its polar cone, and the use of the optimality conditions of Theorem 3.34. The optimality conditions cannot be obtained by imitating the linear programming case. In particular, the constraint qualification condition (the existence of a feasible positive definite X S ) is crucial in this case, while in linear programming the constraint qualification condition is always satisfied. Example 3.37. There are n investment opportunities, with random return rates R1 , . . . , Rn in the next year. We assume that the return rates have a joint normal probability distribution. We have a practically unlimited initial capital and our aim is to invest some of it in such a way that the expected value of our investment after a year is maximized, under the condition that the chance of losing more than some fixed amount b > 0 is smaller than α, where α ∈ (0, 1). Such a requirement is called the Value at Risk constraint. Let x 1 , . . . , x n be the amounts invested in the n opportunities. The netP increase of the value of our investment after a year is random and equals G(x, R) = ni=1 Ri x i .
131
OPTIMALITY CONDITIONS
Its expected value is linear in x: n X E G(x, R) = r¯i x i , i=1
with r¯i = E Ri . Our problem takes on the form: maximize
n X i=1
subject to P
r¯i x i
n nX i=1
x ≥ 0.
o Ri x i ≥ −b ≥ 1 − α,
(3.52)
We do not impose any constraint of the form x 1 + ... + x n = W0 , where W0 is the total invested amount. We also assume that r¯ 6= 0, otherwise no improvement over x = 0 is possible. Denote by C the covariance matrix of the joint distribution of the return rates. The distribution of the total profit (or loss) is normal, with the expected value h¯r , xi, and the variance n X 2 (Ri − r¯i )x i = hx, C xi. V G(x, R) = E i=1
We assume that C is positive definite. If x 6= 0 then the random variable G(x, R) − h¯r , xi √ hx, C xi
has the normal distribution with mean zero and variance one. Our probability constraint is therefore equivalent to the inequality b + h¯r , xi √ ≥ zα , hx, C xi
where z α is the (1 − α)-quantile of the standard normal random variable. If the risk level α ≤√1/2 then z α ≥ 0. Therefore the last constraint (after multiplying both sides by hx, C xi) is equivalent to a constraint involving a convex function: p z α hx, C xi − h¯r , xi ≤ b.
The convexity follows from the positive definiteness of C (see Example 2.84 on page 68). If x = 0 the last constraint is satisfied as well, because b > 0. Consequently, we obtain the following convex optimization problem equivalent to problem (3.52): minimize − h¯r , xi p subject to z α hx, C xi − h¯r , xi − b ≤ 0, x ≥ 0.
(3.53)
132
CHAPTER 3
It satisfies Slater’s condition with x S = 0. To formulate the necessary (and sufficient) conditions of optimality, let λ ≥ 0 be the Lagrange multiplier associated with the constraint. Condition (3.49) takes on the form λz α C x ∈ −N n+ (x). ˆ −(1 + λ)¯r + √ hx, C xi
Substituting the explicit form of the normal cone (as in Example 3.35) we conclude that λz α C x −(1 + λ)¯r + √ ≥ 0, hx, C xi (3.54) D λz α C x E = 0. x, (1 + λ)¯r − √ hx, C xi If we ignore the nonnegativity constraint on x we can solve this system analytically. The optimality condition becomes λz α C x (1 + λ)¯r − √ = 0. hx, C xi Since r¯ is nonzero, λ > 0 and thus the inequality constraint must be satisfied as equality. From the last equation we deduce that the vectors C x and r¯ are colinear: there exists a scalar t such that C x = t r¯ . Substitution to the constraint (which is active) yields p t = b/%(z α − %), λ = (z α /% − 1)−1 , with % = h¯r , C −1 ri. ¯
Note that C −1 is positive definite and hence h¯r , C −1r¯ i is positive. If % = z α , no solution to the system exists. If % > z α , the solution has λ < 0, and thus no optimal solution of the problem exists. In both cases the problem is unbounded. If % < z α , the vector b C −1r¯ xˆ = %(z α − %)
is the solution to the problem without sign restrictions on x. If, in addition, C −1r¯ ≥ 0, then the vector xˆ solves our original problem. If C −1r¯ 6≥ 0 the sign restrictions on x cannot be ignored. We have to find a subset I of decision variables, such that the problem restricted to this subset (with other variables set to zero) can be solved as above and its solution is nonnegative. Then we need to have −1 C(I ) r¯(I ) ≥ 0, with C(I ) denoting the quadratic submatrix of C corresponding to I , and r¯(I ) is the subvector of r¯ with components from I . Moreover, for the remaining variables the first relation of (3.54) must hold, and this requires tedious testing of many subsets I . Numerical methods of convex optimization, which we discuss in Chapter 6, avoid examining all possible subsets of assets.
133
OPTIMALITY CONDITIONS
3.6 OPTIMALITY CONDITIONS FOR SMOOTH–CONVEX PROBLEMS
Our techniques allow us to develop optimality conditions for problems involving both nondifferentiable and nonconvex smooth functions. We consider the following problem formulation: minimize f (x) subject to gi (x) ≤ 0, i = 1, . . . , m, h(x) ∈ Y0 , x ∈ X 0.
(3.55)
We assume that the functions gi : n → , i = 1, . . . , m, are convex, but possibly nonsmooth, and the functions f : n → and h : n → p are continuously differentiable. The sets X 0 ⊂ n and Y0 ⊂ p are convex and closed. The assumption that f (·) is smooth is not restrictive, for if f (·) is convex and nonsmooth, we can introduce an additional variable v ∈ , an additional constraint f (x) ≤ v, and transform (3.55) to a problem of minimizing v subject to the additional constraint as well as other constraints originally appearing in the problem. Example 3.39 at the end of this section illustrates this transformation. If the functions gi (·) are smooth, but nonconvex, we can define g1 (x) ... h new (x) = Y0new = m − × Y0 , gm (x) , h(x) and formally remove the inequality constraints from (3.55). Thus, from the theoretical point of view, it is sufficient to analyze the case when f (·) is smooth and the functions gi (·) are nonsmooth, but convex. Consider the set defined by the convex constraints, C = {x ∈ X 0 : gi (x) ≤ 0, i = 1, . . . , m},
and let I 0 (x) ˆ be the set of active constraints at the optimal point x: ˆ I 0 (x) ˆ = {i : gi (x) ˆ = 0}.
This renders the formulation minimize f (x) subject to h(x) ∈ Y0 , x ∈ C.
134
CHAPTER 3
Recall that Robinson’s condition (3.6) for this problem has the form h 0 (x)K ˆ C (x) ˆ − K Y0 (h(x)) ˆ =
p
.
(3.56)
The set K C (x), ˆ in general, is difficult to describe exactly, but under Slater’s condition we can find its nonempty subset. Indeed, if there exists x S ∈ X 0 such that gi (x S ) < 0, i = 1, . . . , m, then the set K C0 (x) ˆ of directions 0 0 d ∈ K C (x) ˆ such that gi (x; ˆ d) < 0, i ∈ I (x), ˆ is nonempty. It obviously contains the direction d = x S − x. ˆ We can, therefore, formulate a more explicit sufficient condition for Robinson’s condition by replacing K C (x) ˆ with K C0 (x). ˆ In the theorem below it does not constitute any additional restriction. T HEOREM 3.38. Assume that xˆ ∈ X 0 is a local minimum of problem (3.55). Assume that the functions gi (·), i ∈ I 0 (x), ˆ are subdifferentiable at xˆ and that there exists x S ∈ X 0 such that gi (x S ) < 0, i = 1, . . . , m. Furthermore, let Robinson’s condition be satisfied at x. ˆ Then there exist λˆ ∈ m + and ◦ µˆ ∈ [TY0 (h(x))] ˆ such that 0 ∈ ∇ f (x) ˆ + and
m X i=1
λˆ i ∂gi (x) ˆ + [h 0 (x)] ˆ Tµˆ + N X 0 (x) ˆ
λˆ i gi (x) ˆ = 0, i = 1, . . . , m.
(3.57)
(3.58)
ˆ x) ˆ µ) Furthermore, the set Λ( ˆ of multipliers (λ, ˆ satisfying the above conditions is convex and compact. Proof. Let X be the feasible set of problem (3.55). By Theorem 3.24 on page 113, −∇ f (x) ˆ ∈ [TX (x)] ˆ ◦. By Theorem 3.23 on page 112, the polar of the tangent cone to X has the form [TX (x)] ˆ ◦ = N X 0 (x) ˆ + [h 0 (x)] ˆ Tµ : µ ∈ [TY0 (h(x))] ˆ ◦ X + cone(∂gi (x)). ˆ i∈I 0 ( x) ˆ
It follows that there exist λˆ i ≥ 0 and µˆ ∈ [TY0 (h(x))] ˆ ◦ such that X 0 ∈ ∇ f (x) ˆ + λˆ i ∂gi (x) ˆ + [h 0 (x)] ˆ Tµˆ + N X 0 (x). ˆ i∈I 0 ( x) ˆ
135
OPTIMALITY CONDITIONS
This is equivalent to (3.57)–(3.58). To prove the compactness of the set of Lagrange multipliers, we can adapt the idea of the proof of Lemma 3.26 in a straightforward manner. We leave the details of this adaptation to the reader. If the objective function f (·) is convex and nonsmooth, the standard transformation discussed at the beginning of this section reduces it to the form analyzed in Theorem 3.38. The multiplier associated with the additional constraint f (x) − v ≤ 0 equals one, and the optimality conditions can be reduced back to (3.57)–(3.58), with the only difference that ∇ f (x) ˆ should be replaced by ∂ f (x) ˆ in (3.57). Theorem 3.38 encompasses as its special cases all formulations of nonlinear optimization problems discussed in the preceding sections. Clearly, the verification of its constraint qualification conditions is very difficult. A formal simplification of the optimality conditions can be achieved by using the concept of metric regularity of the entire system of constraints. Due to the possible nonconvexity and nondifferentiability of the constraints, the analysis of metric regularity requires special techniques of nonsmooth calculus. This leads to other technical complications and exceeds the scope of this book. However, the techniques that we have mastered so far allow us to analyze many important problems involving nonconvex and nonsmooth functions, as in the example below. Example 3.39. Let h : n → m be a continuously differentiable function. Consider the unconstrained minimization problem
minimize kh(x)k♦ ,
(3.59)
where k · k♦ is a norm in m . The Chebyshev approximation problem of Example 3.32 can be formulated in this way. Since the objective function is nonsmooth and nonconvex, the optimality conditions for unconstrained optimization of Section 3.1 cannot be directly applied here. However, we can use conditions for constrained problems, after an appropriate transformation. By introducing the variables y ∈ m and v ∈ we can transform problem (3.59) to an equivalent constrained problem
minimize v subject to kyk♦ − v ≤ 0, h(x) − y = 0.
(3.60)
Indeed, for every point x the triple: x, y = h(x), and v = kyk♦ , is feasible in problem (3.60), and the objective values of both problems coincide. Therefore the optimal value of (3.60) is less than or equal to the optimal value of (3.59). If the triple x, ˆ yˆ , and vˆ are a solution to problem (3.60) then we have yˆ = h(x) ˆ and
136
CHAPTER 3
vˆ = k yˆ k♦ = kh(x)k ˆ ♦ . Thus the optimal value of problem (3.59) is less than or equal to the optimal value of (3.60). This means that both values are the same, and the x-part of the optimal solution of (3.60) is an optimal solution of (3.59). Let us verify the constraint qualification conditions for problem (3.60). Slater’s condition is satisfied, because for each y we can find v > kyk♦ . Consider the set C = {(x, y, v) : kyk♦ − v ≤ 0}. The cone of feasible directions to C at (x, ˆ yˆ , v) ˆ satisfies the inclusion K C (x, ˆ yˆ , v) ˆ ⊃ {(d x , d y , dv ) ∈
n
×
m
×
: dv ≥ kd y k♦ }.
Indeed, for every τ > 0 and for dv ≥ kd y k♦ we have ky + τ d y k♦ − (v + τ dv ) ≤ kyk♦ + τ kd y k♦ − v − τ dv ≤ kyk♦ − v. If the pair (y, v) is feasible for C, the pair (y + τ d y , v + τ dv ) is feasible for all τ > 0, as claimed. Thus each d y ∈ m is feasible and
{h 0 (x)d ˆ x − d y : (dx , d y , dv ) ∈ K C (x, ˆ yˆ , v)} ˆ =
m
.
Consequently, Robinson’s condition is satisfied. The constraint kyk♦ − v ≤ 0 must be active at the solution. It follows from Theorem 3.38 that there exist λ ≥ 0 and µ ∈ m such that T 0 0 0 h 0 (x) ˆ 0 ∈ 0 + λ ∂k yˆ k♦ + −I µ. 0 1 −1 0
The first block of equations, associated with x, yields
T h 0 (x) ˆ µ = 0.
(3.61)
The third one implies that λ = 1. Then the middle block reads µ ∈ ∂k yˆ k♦ . The subdifferential of the norm has been calculated in Example 2.78 on page 63. We obtain ˆ = k yˆ k♦ }. ∂k yˆ k♦ = {s ∈ m : ksk∗ ≤ 1, hs, yi
Here k · k∗ is the dual norm to k · k♦ . We conclude that if a point xˆ is an optimal solution of problem (3.59) then there exists µ ∈ m such that
kµk∗ ≤ 1,
hµ, h(xˆ )i = kh(x)k ˆ ♦, and equation (3.61) is satisfied. In the typical case, when kh(x)k ˆ > 0, we actually have kµk∗ = 1.
137
OPTIMALITY CONDITIONS
In the special case when k·k♦ is the Euclidean norm k·k this system is equivalent to the equation 0 T h (x) ˆ h(x) ˆ = 0,
which can be obtained as the necessary condition of a minimum of the smooth function kh(x)k2 .
Example 3.40. We are given N points y 1 , . . . , y N on the plane and a convex, bounded and closed set X ⊂ 2 . Our objective is to find a point x ∈ X for which the distance to the nearest point y i is the largest possible. Formally, the problem is
maximize min kx − y i k. x∈X
Defining
1≤i≤N
f i (x) = −kx − y i k, i = 1, . . . , N,
we can equivalently formulate the problem as minimize max f i (x). x∈X
1≤i≤N
The “max” function can be dealt with by introducing an auxiliary variable v ∈ and formulating an equivalent constrained optimization problem
minimize v subject to f i (x) − v ≤ 0, i = 1, . . . , N, x ∈ X.
At the optimal solution (x, ˆ v) ˆ we have kxˆ − y i k > 0 for all i = 1, . . . , N and vˆ < 0. Therefore the functions f i (·) are continuously differentiable in a neighborhood of x. ˆ Our problem has form (3.55) with h i (x, v) = f i (x) − v, i = 1, . . . , N, and Y0 =
N −.
Define the set of active constraints Iˆ = {1 ≤ i ≤ N : kx − y i k = −v}, ˆ
and let m be its cardinality. Robinson’s condition at xˆ (written for active constraints only) has the form nh i o h∇ f i (x), ˆ d x i − d v − d yi ) : dx ∈ KX (x), ˆ dv ∈ , d y ≤ 0 = m . i∈ Iˆ
It is obviously satisfied, because by setting d x = 0 (which is always tangent to X) we can obtain each vector as a combination of d y ≤ 0 and a vector with all components equal to dv (which can be any real number).
138
CHAPTER 3
The necessary condition (3.57) for a local minimum has the form: there exist multipliers µi ≥ 0, i ∈ Iˆ, such that X " y i −xˆ # N X (x) ˆ 0 i − xk ky ˆ + . 0∈ + µi 0 1 −1 ˆ i∈ I
The first block (corresponding to x) means that there exists a normal vector s ∈ N X (x) ˆ such that X y i − xˆ µi i + s = 0. (3.62) ky − xk ˆ i∈ Iˆ
The second block is simply
X i∈ Iˆ
Define α= and let By construction,
P
γi = i∈ Iˆ γi
µi = 1.
X k∈ Iˆ
µk , − xk ˆ
ky k
µi , i ∈ Iˆ. αky i − xk ˆ
= 1. Dividing both sides of (3.62) by α we find that X i∈ Iˆ
s γi y i − xˆ + = 0, α
which can be rewritten as xˆ −
X s = γi y i . α i∈ Iˆ
If the point xˆ is an interior point of X, then s = 0. Hence xˆ is a convex combination of the points y i , i ∈ Iˆ. By Lemma 2.7, xˆ is a convex combination of at most three points y i with i ∈ Iˆ. Since xˆ is different than points y i , the number of points in the convex combination is at least two. Moreover, xˆ is the center of a circle passing through these points and not containing strictly any other points y j . If the local minimum is a boundary point of X then we may have s 6= 0. In this case, xˆ − s¯ is a convex combination of active y i , for some normal s¯ = s/α. Considering points xˆ −τ s¯ , τ ∈ [0, 1], we can choose the smallest τ for which xˆ −τ s¯ is a convex combination of active points. Then the number of active points in this combination need not be larger than 2. If there are two such points, the solution xˆ is equally distant from both of them. If one point y i is active, xˆ − y i is normal to X at x. ˆ All these cases are illustrated in Figure 3.7 for an example of three points y 1 , y 2 , and y 3 , with the set X represented by the shaded area. The point x 1 is a local minimum: it is a convex combination of equally distant y 1 , y 2 , and y 3 . The point
139
OPTIMALITY CONDITIONS
x6 x7 y2
x8
x9
x5
x4
y3
x3
x1
x2
y1 x 11 x 10
s Figure 3.7. Points satisfying necessary conditions of optimality. x 2 is a local minimum: x 2 − y 1 is normal to X at x 2 . The point x 10 is another local minimum: x 10 − s is a convex combination of y 1 and y 3 . In fact, x 10 is the global minimum. Other local minima are located at x 4 , x 6 , and x 8 . The remaining points: x 3 , x 5 , x 7 , x 9 , and x 11 , satisfy the optimality conditions but are not local minima.
3.7 SECOND ORDER OPTIMALITY CONDITIONS 3.7.1 Second Order Tangent Sets
Consider a set X ⊂ n and a point x 0 ∈ X. Our development of optimality conditions for problems involving the constraint x ∈ X was based on the analysis of the tangent set TX (x 0 ). This is the set of directions s such that the ray x 0 + τ s, τ > 0, is very close to X: dist(x 0 + τk s, X) = o(τk ),
140
CHAPTER 3
at some points τk ↓ 0. In order to develop second order optimality conditions, we consider parabolic trajectories of the form x(τ ) = x 0 + τ s +
τ2 w, τ ≥ 0, 2
(3.63)
where s is a tangent direction. Our idea is to have their distance to X infinitely smaller than τk2 . D EFINITION 3.41. A direction w is called second order tangent to the set X ⊂ n at the point x 0 ∈ X and in direction s, if there exist a sequence of points x k ∈ X, k = 1, 2, . . . , and a sequence of scalars τ k > 0, k = 1, 2, . . . , such that τk ↓ 0 and w = lim
k→∞
x k − x 0 − τk s . 1 (τ )2 2 k
Denoting by ok the difference between the left hand and right hand sides of the last equation, we can write x k = x 0 + τk s +
τ2 τ2 τk2 w − ok k = x(τk ) − ok k . 2 2 2
It follows from the definition that the points on the parabolic trajectory (3.63) are, for τ = τk and k → ∞, very close to the set X: dist(x(τk ), X) = o(τk2 ). It is implicit in Definition 3.41 that s is a (first order) tangent direction: s ∈ TX (x 0 ). Indeed, if w is a second order tangent direction, then for the limit in Definition 3.41 to exist, it is necessary that lim
k→∞
x k − x 0 − τk s = 0, τk
and thus (x k −x 0 )/τk → s. For s ∈ / TX (x 0 ) no second order tangent direction exists. The set of all second order tangent directions to X at (x 0 , s) is called the second order tangent set and denoted by T X2 (x 0 , s). L EMMA 3.42. Let X ⊂ and
n
, x 0 ∈ X, s ∈ TX (x 0 ). The set TX2 (x 0 , s) is closed
TX2 (x 0 , αs) = α 2 TX2 (x 0 , s) for all α > 0.
141
OPTIMALITY CONDITIONS
Proof. Let w ∈ TX2 (x 0 , s) and let {x k } and {τk } be the sequences satisfying Definition 3.41. For every α > 0 we obtain α 2 (x k − x 0 − τk s) 1 k→∞ (τ )2 2 k
α 2 w = lim
x k − x 0 − ταk (αs) . = lim 1 τk 2 k→∞ 2
α
Thus α 2 w satisfies Definition 3.41 for the direction αs, with the same se quence of points {x k } and with the corresponding sequence of scalars τk /α . The reverse argument goes the same way. To prove that the second order tangent set is closed, consider a sequence {w j } converging to some w, where each w j ∈ TX2 (x 0 , s). We need to show that w satisfies Definition 3.41 for (x 0 , s). Choose any sequence ε j ↓ 0. For each w j , Definition 3.41 is satisfied with some sequences of points x j,k and scalars τ j,k , k = 1, 2, . . . . We can then find x j,k( j ) and τ j,k( j ) such that
Therefore
x j,k( j ) − x − τ
0 j,k( j ) s j − w
≤ εj . 1 (τ )2 2 j,k( j )
x j,k( j ) − x − τ
0 j,k( j ) s − w ≤ ε j + kw j − wk,
1 2 (τ ) 2 j,k( j )
and w satisfies Definition 3.41 with sequences x j,k( j ) and τ j,k( j ) , j = 1, 2, . . . In general, the second order tangent set is not a cone, and it may not be convex, even for a convex set X.† But for the case of a polyhedral set X, the second order tangent set has a nice representation, and is in fact a convex cone. L EMMA 3.43. If X is a convex polyhedron then TX2 (x 0 , s) = TTX (x0 ) (s). Proof. Let us recall that if X is a convex polyhedron, then the tangent cone TX (x 0 ) is identical with the cone of feasible directions KX (x 0 ), and both cones are polyhedral: they are defined by finitely many linear inequalities. † There exist alternative definitions of second order tangent sets, which guarantee the
convexity in this case, but then other complications arise. The reader is referred to the literature cited at the end for further studies of this topic.
142
CHAPTER 3
Denoting by ai , i = 1, . . . , m, the vectors defining the inequalities active at x 0 , we can write TX (x 0 ) = {s ∈
n
: hai , si ≤ 0, i = 1, . . . , m}.
Consider a direction s ∈ TX (x 0 ) and a sequence x(τk ) = x 0 + τk s +
τk2 w 2
with some τk ↓ 0. The vector w is an element of the second order tangent set if and only if dist(x(τk ), X) = o(τk2 ). This is equivalent to hai , x(τk ) − x 0 i ≤ o(τk2 ), i = 1, . . . , m.
(3.64)
Two cases are possible for each i = 1, . . . , m. If ha i , si = 0 then (3.64) holds true if and only if hai , wi ≤ 0. If hai , si < 0 then (3.64) is satisfied for all w ∈ n . Therefore w ∈ TX2 (x 0 ) if and only if hai , wi ≤ 0 for all i such that hai , si = 0. This is equivalent to the inclusion w ∈ TTX (x0 ) (s). Consider now the set X of points x ∈
n
satisfying the system of relations
gi (x) ≤ 0, i = 1, . . . , m, h i (x) = 0, i = 1, . . . , p, x ∈ X 0.
(3.65)
Let xˆ ∈ X and let all functions gi (·) and h i (·) be twice continuously differentiable at x. ˆ Furthermore, let X 0 be a convex set. Under Robinson’s condition (3.11), the cone TX (x) ˆ = s ∈ TX 0 (x) ˆ : h∇gi (x), ˆ si ≤ 0, i ∈ I 0 (x), ˆ
h∇h i (x), ˆ si = 0, i = 1, . . . , p
(3.66)
is the tangent cone to the feasible set at x. ˆ We shall derive an algebraic description of the second order tangent set. L EMMA 3.44. Assume that Robinson’s condition is satisfied at xˆ ∈ X. Then for every s ∈ TX (x), ˆ TX2 (x, ˆ s) = w ∈ TX20 (x, ˆ s) : h∇gi (x), ˆ wi ≤ −hs, ∇ 2gi (x)si, ˆ i ∈ I 00 (x, ˆ s), 2 h∇h i (x), ˆ wi = −hs, ∇ h i (x)si, ˆ i = 1, . . . , p ,
(3.67)
143
OPTIMALITY CONDITIONS
with I 00 (x, ˆ s) = {i ∈ I 0 (x) ˆ : h∇gi (x), ˆ si = 0}. Proof. Let s ∈ TX (x) ˆ and let w be an element of the set on the right hand side of (3.67). Consider the parabolic trajectory (3.63). For each i ∈ I 0 (x) ˆ we can expand gi (x(τ )) as follows: gi (x(τ )) = gi (x) ˆ + h∇gi (x), ˆ x(τ ) − xi ˆ
τ2 hx(τ ) − x, ˆ ∇ 2gi (x)(x(τ ˆ ) − x)i ˆ + o(τ 2 ) (3.68) 2 τ2 τ2 ˆ wi + hs, ∇ 2gi (x)si = τ h∇gi (x), ˆ si + h∇gi (x), ˆ + o(τ 2 ). 2 2 +
We have used here the fact that gi (x) ˆ = 0 for i ∈ I 0 (x). ˆ Two cases may 00 occur. If i ∈ / I (x, ˆ s) then h∇gi (x), ˆ si < 0. It follows that for every w ∈ n we have gi (x(τ )) < 0 for all sufficiently small τ > 0. In the second case, when i ∈ I 00 (x, ˆ s), the first order term vanishes. But if w is an element of the set defined in (3.67), then the second order term is nonpositive. Similar arguments apply to equality constraints. We conclude that if w satisfies the conditions on the right hand side of (3.67), then gi (x(τ )) ≤ o(τ 2 ), i ∈ I 0 (x), ˆ 2 h i (x(τ )) ≤ o(τ ), i = 1, . . . , p,
(3.69)
dist(x(τ ), X 0 ) = o(τ 2 ).
The last relation follows from the fact that s ∈ TX 0 (x) ˆ and w ∈ TX20 (x, ˆ s). By Theorem A.10, it follows from Robinson’s condition that the system of constraints is metrically regular at x. ˆ Using Definition 3.14, we deduce that there exist C ≥ 0 and τ0 > 0 such that for all τ ∈ [0, τ0 ] we can find a point ϕ(τ ) ∈ X satisfying the inequality: kϕ(τ ) − x(τ )k ≤ C dist(x(τ ), X 0 ) +
X
p X h i (x(τ )) . max 0, gi (x(τ )) +
i∈I 0 ( x) ˆ
By virtue of (3.69),
i=1
kϕ(τ ) − x(τ )k ≤ o(τ 2 ). Therefore dist(x(τ ), X) ≤ o(τ 2 ), and indeed w ∈ TX2 (x, ˆ s).
144
CHAPTER 3
To prove the converse, assume that w ∈ TX2 (x, ˆ s). Then we have (3.69) for some sequence τk ↓ 0. Inspecting the expansions (3.68) we conclude that w must be an element of the right hand side of (3.67). It follows from the last two lemmas that for a polyhedral X 0 and under Robinson’s condition, the second order tangent set to the set defined by (3.65) is a closed convex cone. 3.7.2 Optimality Conditions
We are now ready to develop the second order counterpart of the fundamental optimality condition of Theorem 3.24. We consider the general problem formulation minimize f (x).
(3.70)
x∈X
We assume that X is a set in differentiable.
n
, and f :
n
→
is twice continuously
T HEOREM 3.45. Assume that xˆ is an optimal solution of problem (3.70). Then for every s ∈ TX (x) ˆ such that h∇ f (x), ˆ si = 0 we have h∇ f (x), ˆ wi + hs, ∇ 2 f (x)si ˆ ≥ 0 for all w ∈ TX2 (x, ˆ s).
(3.71)
Proof. Consider the function x(τ ) defined in (3.63) with x 0 = x. ˆ If s ∈ 2 k TX (x) ˆ and w ∈ TX (x, ˆ s), there exist sequences of points x ∈ X, x k → x, ˆ and scalars τk ↓ 0, such that 2
k
x − xˆ − τk s − τk w = o(τ 2 ). k 2
Since xˆ is a local minimum, f (x k ) ≥ f (x) ˆ for all sufficiently large k. Therefore f (x(τk )) − f (x) ˆ ≥ o(τk2 ). The second order expansion of f (x(τ )) at τ = 0 renders f (x(τk ))− f (x) ˆ = τk h∇ f (x), ˆ si+
τk2 τ2 2 h∇ f (x), ˆ wi+ k hs, ∇ 2 f (x)si+o(τ ˆ k ). 2 2
Hence τk2 τ2 h∇ f (x), ˆ wi + k hs, ∇ 2 f (x)si ˆ ≥ o(τk2 ). 2 2 By assumption, the first term on the left hand side equals 0. Dividing both sides by τk2 and letting τk ↓ 0, we obtain inequality (3.71). τk h∇ f (x), ˆ si +
145
OPTIMALITY CONDITIONS
This general result allows for the formulation of second order conditions for a variety of nonlinear optimization problems. Most important is the problem with constraints involving inequalities and equations: minimize f (x) subject to gi (x) ≤ 0, i = 1, . . . , m, h i (x) = 0, i = 1, . . . , p, x ∈ X 0.
(3.72)
We assume that all functions are twice continuously differentiable at a local minimum x. ˆ The set X 0 is convex and closed. The associated Lagrangian has the form L(x, λ, µ) = f (x) +
m X i=1
λi gi (x) +
p X
µi h i (x).
i=1
It follows from Lemma 3.44 and Theorem 3.45 that, under Robinson’s condition, for every s ∈ TX (x) ˆ such that h∇ f (x), ˆ si = 0, the optimal value of the problem minimize h∇ f (x), ˆ wi + hs, ∇ 2 f (x)si ˆ
subject to w ∈ TX20 (x, ˆ s),
h∇gi (x), ˆ wi ≤ −hs, ∇ 2gi (x)si, ˆ i ∈ I 00 (x, ˆ s),
(3.73)
h∇h i (x), ˆ wi = −hs, ∇ 2h i (x)si, ˆ i = 1, . . . , p, is nonnegative. In order to analyze this problem in more detail, we make an additional assumption that X 0 is a convex polyhedron. Optimality conditions for problem (3.73) provide the second order conditions for problem (3.72). We use X to denote the feasible set of this problem. T HEOREM 3.46. Assume that X 0 is a convex polyhedron and that a point xˆ ∈ X is a local minimum of problem (3.72) satisfying Robinson’s condition. Then for every s ∈ TX (x) ˆ such that h∇ f (x), ˆ si = 0 the following condition holds true:
max s, ∇x2x L(x, ˆ λ, µ)s ≥ 0, (3.74) ˆ x) (λ,µ)∈Λ( ˆ
ˆ x) ˆ µ) where Λ( ˆ is the set of optimal values of Lagrange multipliers ( λ, ˆ in problem (3.72). Proof. Before starting the proof, we note that under Robinson’s condition ˆ x) the set Λ( ˆ is compact (Lemma 3.26), and therefore the maximum in (3.74) exists.
146
CHAPTER 3
Consider problem (3.73). By virtue of Lemma 3.43, TX20 (x, ˆ s) = TTX 0 (x) ˆ (s). Furthermore, Example 2.21 yields that TTX 0 (x) ˆ ˆ (s) ⊃ TX 0 ( x). Therefore the following problem minimize h∇ f (x), ˆ wi + hs, ∇ 2 f (x)si ˆ subject to w ∈ TX 0 (x), ˆ
h∇gi (x), ˆ wi ≤ −hs, ∇ 2gi (x)si, ˆ i ∈ I 00 (x, ˆ s),
(3.75)
h∇h i (x), ˆ wi = −hs, ∇ 2h i (x)si, ˆ i = 1, . . . , p, is a restriction of problem (3.73), and its optimal value must be nonnegative as well. It follows from Robinson’s condition that the feasible set of problem (3.75) is nonempty. Problem (3.75) has a linear objective function and linear and polyhedral constraints. Therefore it has an optimal solution: w. ˆ Denote by λ¯ i , i ∈ I 00 (x, ˆ s) and µ¯ i , i = 1, . . . , p, the Lagrange multipliers associated with the constraints. The necessary and sufficient conditions of optimality of Theorem 3.25 from page 116 for problem (3.75) read −∇ f (x) ˆ −
X
λ¯ i ∇gi (x) ˆ −
i∈I 00 ( x,s) ˆ
p X i=1
◦ µ¯ i ∇h i (x) ˆ ∈ TTX 0 (x) ˆ (w) ,
λ¯ i h∇gi (x), ˆ wi ˆ + hs, ∇ 2gi (x)si ˆ = 0, i ∈ I 00 (x, ˆ s), λ¯ ≥ 0.
Therefore the set of optimal values of Lagrange multipliers for problem ˆ x) (3.75) is included in the set Λ( ˆ of optimal values of Lagrange multipliers of problem (3.72) (we formally define λ¯ i = 0 for i ∈ / I 00 (x, ˆ s)). Moreover, the optimal value of problem (3.75) is equal to the value of its Lagrangian: ¯ µ) L 1 (w, ˆ λ, ¯ = h∇ f (x), ˆ wi ˆ + hs, ∇ 2 f (x)si ˆ X ˆ wi ˆ + hs, ∇ 2gi (x)si ˆ λ¯ i h∇gi (x), + i∈I 00 ( x,s) ˆ p
+
X i=1
µ¯ i h∇h i (x), ˆ wi ˆ + hs, ∇ 2h i (x)si ˆ .
Recall from Example 2.21 on page 27 that
TTX 0 (x) ˆ + {αw : α ∈ ˆ (w) = TX 0 ( x)
}.
147
OPTIMALITY CONDITIONS
¯ µ) If the minimum of the Lagrangian L 1 (·, λ, ¯ in TX 0 (x) ˆ occurs at w, ˆ then D
∇ f (x) ˆ +
X
ˆ + λ¯ i ∇gi (x)
i∈I 00 ( x,s) ˆ
p X i=1
E µ¯ i ∇h i (x), ˆ wˆ = 0.
The first order terms of the Lagrangian disappear and we conclude that X ¯ µ) L 1 (w, ˆ λ, ¯ = hs, ∇ 2 f (x)si ˆ + λ¯ i hs, ∇ 2gi (x)si ˆ p
+
X i=1
i∈I 00 ( x,s) ˆ
¯ µ)s µ¯ i hs, ∇ 2h i (x)si ˆ = s, ∇x2x L(x, ˆ λ, ¯ .
The optimal value is nonnegative, and therefore inequality (3.74) is valid. Another way to prove this theorem is to apply to problem (3.75) the duality theory in conic programming, which we discuss in Section 4.3. We can mention here that the assumption that X 0 is a polyhedron allowed us to use the explicit form of its second order tangent set. It is also possible to develop more general second order conditions, for a convex X 0 , but they are more involved. One has to replace in problem (3.73) the second order tangent set by its convex subsets, and involve general convex duality to analyze problem (3.75). In the development of the second order sufficient conditions for problem (3.72) we do not assume explicitly any constraint qualification condition, but we assume the existence of optimal Lagrange multipliers. The set (3.66) may be larger than the tangent cone, but we still denote it by TX . We also do not assume any special structure of the set X 0 , except for its convexity and closedness. T HEOREM 3.47. Assume that a point xˆ satisfies the first order optimality ˆ x) conditions for problem (3.72), and let Λ( ˆ be the set of Lagrange multipliers ˆ (λ, µ) ˆ for this problem. Assume that for every nonzero s in the set (3.66) such that h∇ f (x), ˆ si = 0, we have
ˆ µ)s sup s, ∇x2x L(x, ˆ λ, ˆ > 0. (3.76) ˆ µ)∈ ˆ x) (λ, ˆ Λ( ˆ
Then xˆ is a local minimum of (3.72).
Proof. We argue by contradiction. Suppose xˆ is not a local minimum. Then there exists a sequence of feasible points y k such that y k → xˆ and f (y k ) < f (x) ˆ for all k.
(3.77)
148
CHAPTER 3
We define the sequence sk =
y k − xˆ ky k − xk ˆ
and denote its accumulation point by s. It exists, because {s k } is bounded. For i ∈ I 0 (x) ˆ we can expand the function gi (·) around x: ˆ 0 ≥ gi (y k ) = h∇gi (x), ˆ y k − xi ˆ + o(y k − x), ˆ
with o(z)/kzk → 0 for z → 0. Dividing both sides by ky k − xk ˆ and passing to the limit over the subsequence for which s k → s, we obtain h∇gi (x), ˆ si ≤ 0, i ∈ I 0 (x). ˆ
A similar analysis of equality constraints yields the relations h∇h i (x), ˆ si = 0, i = 1, . . . , p. Therefore s is an element of the cone (3.66). It follows from the necessary conditions of optimality that h∇ f (x), ˆ si ≥ 0. In view of (3.77), h∇ f (x), ˆ si = 0. By assumption (3.76), we can choose ˆ µ) ˆ x) (λ, ˆ ∈ Λ( ˆ such that
ˆ µ)s s, ∇x2x L(x, ˆ λ, ˆ > 0. (3.78)
By Taylor’s formula
1 ˆ ∇ 2 f (x)(y ˆ k − x)i ˆ + δ(y k − x), ˆ f (y k ) = f (x) ˆ + h∇ f (x), ˆ y k − xi ˆ + hy k − x, 2 (3.79) δ(y k − x) ˆ with ky k −xk → 0. Similarly, ˆ 2 gi (y k ) =gi (x) ˆ + h∇gi (x), ˆ y k − xi ˆ 1 + hy k − x, ˆ ∇ 2gi (x)(y ˆ k − x)i ˆ + γi (y k − x), ˆ 2 h i (y k ) =h i (x) ˆ + h∇h i (x), ˆ y k − xi ˆ 1 + hy k − x, ˆ ∇ 2h i (x)(y ˆ k − x)i ˆ + δi (y k − x), ˆ 2
(3.80)
where γi and δi are infinitely small with respect to ky k − xk ˆ 2 . Let us add (3.80) multiplied by λˆ i and µˆ i , respectively, to (3.79). The sum of the first order terms on the right hand side satisfies the first order optimality condition: D
∇ f (x) ˆ +
m X i=1
λˆ i ∇gi (x) ˆ +
m X i=1
E µˆ i ∇h i (x), ˆ y k − xˆ ≥ 0,
149
OPTIMALITY CONDITIONS
because y k ∈ X 0 . Therefore the sum of (3.79)–(3.80) renders the inequality f (y k ) ≥ f (x) ˆ +
1 k ˆ µ)(y ¯ k − x), y − x, ˆ ∇x2x L(x, ˆ λ, ˆ k − x) ˆ + δ(y ˆ 2
in which δ¯ (y k − x) ˆ is infinitely small with respect to ky k − xk ˆ 2 . Using (3.77) k 2 and dividing both sides by ky − xk ˆ , we obtain for large k the relation ¯ k − x) δ(y ˆ 1 k 2 ˆ µ)s s , ∇x x L(x, ˆ λ, ˆ k + k < 0. 2 ky − xk ˆ 2
When k → ∞ (over an appropriate subsequence), then s k → s. It follows that
ˆ µ)s s, ∇x2x L(x, ˆ λ, ˆ ≤ 0, which contradicts (3.78).
Theorem 3.47 can be used to obtain simpler versions of the second order sufficient conditions. For example, a point xˆ is said to satisfy the semi-strong ˆ µ) ˆ x) second order sufficient conditions, if there exist multipliers ( λ, ˆ ∈ Λ( ˆ such that
ˆ µ)s s, ∇x2x L(x, ˆ λ, ˆ > 0, (3.81) for all nonzero s included in the plane: ˆ x) D( ˆ = s ∈ R n : h∇gi (x), ˆ si = 0 for all i such that λˆ i > 0, h∇h i (x), ˆ si = 0, i = 1, . . . , p .
We shall show that these conditions imply the assumptions of Theorem 3.47. Suppose s is a nonzero element of TX (x) ˆ such that h∇ f (x), ˆ si = 0. The first order necessary conditions imply that D
∇ f (x) ˆ +
X
λˆ i ∇gi (x) ˆ +
i∈I 0 ( x) ˆ
p X i=1
E µˆ i ∇h i (x), ˆ s ≥ 0.
Since h∇ f (x), ˆ si = 0 and h∇h i (x), ˆ si = 0, we deduce that X λˆ i h∇gi (x), ˆ si ≥ 0. i∈I 0 ( x) ˆ
All terms on the left hand side are nonpositive for s ∈ TX (x), ˆ and therefore they must be equal to 0. This implies that h∇g i (x), ˆ si = 0 whenever λˆ i > 0. ˆ x). Thus s ∈ D( ˆ Therefore the assumptions of Theorem 3.47 are satisfied.
150
CHAPTER 3
Example 3.48. Let Q be a symmetric matrix of dimension n. Consider the problem of minimizing the quadratic form on the unit sphere S: minimize hx, Qxi
subject to 1 − kxk2 = 0.
We analyzed this problem in Example 3.29, where we found that the first order necessary conditions of optimality, Qx − µx = 0,
are satisfied by each eigenvector of Q, and the corresponding eigenvalue is the value of the Lagrange multiplier µ. We shall show that the second order conditions allow us to find the optimal solution. Let µ1 ≤ µ2 ≤ · · · ≤ µn be the eigenvalues of Q, and let z 1 , . . . , z n be the corresponding orthogonal eigenvectors of unit length. Suppose xˆ = z k and µˆ = µk . The tangent plane to the sphere S at xˆ has the form: ˆ x) D( ˆ = {d ∈ R n : hx, ˆ di = 0} = {d ∈ R n : hz k , di = 0}.
Therefore every tangent direction d is a combination of the remaining eigenvectors X d= αi z i . i6 =k
The Lagrangian has the form
L(x, µ) = hx, Qxi − µkxk2 = hx, (Q − µI )xi,
and its Hessian at z k and µk becomes
∇ 2L(x, µ) ˆ = Q − µk I.
The second order necessary condition of optimality of Theorem 3.46 requires that hd, (Q − µk I )di ≥ 0
for every tangent direction d. In particular, setting d = z i for i 6= k we obtain hz i , (Q − µk I )z i i = µi − µk ≥ 0.
Therefore the local minimum is an eigenvector corresponding to the smallest eigenvalue of Q. 3.8 SENSITIVITY
Let us now consider the problem with parameters b ∈
m
minimize f (x) subject to gi (x) ≤ bi , i = 1, . . . , m, h i (x) = ci , i = 1, . . . , p.
and c ∈
p
:
(3.82)
151
OPTIMALITY CONDITIONS
We assume that f : n → , gi : n → , i = 1, . . . , m, and h i : n → , i = 1, . . . , p, are twice continuously differentiable functions. We are interested in the dependence of the solution and of the optimal value of this problem, on the parameters b and c. Assume that x 0 is a local minimum of (3.82) for b = b 0 and c = c0 . As before, define the set of active inequality constraints I 0 (x 0 ) = {1 ≤ i ≤ m : gi (x 0 ) = bi0 }.
We assume that x 0 satisfies a constraint qualification condition and we denote the vector of Lagrange multipliers at x 0 by (λ0 , µ0 ). We can analyze the sensitivity of problem (3.82) by considering the system of the first order conditions of optimality, under the semi-strong second order sufficient condition (3.81) discussed on page 149. T HEOREM 3.49. Assume that the gradients of all active constraints are linearly independent at x 0 , and that λ0i > 0 for all i ∈ I 0 (x 0 ). Furthermore, let the semi-strong second order sufficient condition (3.81) be satisfied at x 0 . Then there exists a neighborhood U of (b 0 , c0 ) in which we can define functions x(b, ˆ c), λˆ i (b, c), i ∈ I 0 (x 0 ), and µˆ i (b, c), i = 1, . . . , p, such ˆ c), µ(b, that x(b, ˆ c) is a local minimum of problem (3.82), and ( λ(b, ˆ c)) is the corresponding vector of Lagrange multipliers. Moreover, the functions ˆ c), and µ(b, x(b, ˆ c), λ(b, ˆ c) are differentiable at (b 0 , c0 ) and d f (x(b ˆ 0 , c0 )) = −λ0 , db d f (x(b ˆ 0 , c0 )) = −µ0 . dc Proof. For simplicity of notation, we assume that p = 0 (no equality constraints) and that I 0 (x 0 ) = {1, . . . , m 0 }. Define the Jacobian of the active constraint functions: (∇g1 (x))T .. A(x) = . . T (∇gm 0 (x)) If we ignore the constraints that are not active at x 0 , the first order necessary condition reads: ∇ f (x) + A(x)T λ = 0.
(3.83)
gi (x) = bi , i = 1, . . . , m 0 .
(3.84)
We postulate that the inequalities that are active at x 0 be satisfied as equations for b close to b 0 :
152
CHAPTER 3
The Jacobian of the system of equations (3.83)–(3.84) at (x 0 , λ0 ) has the form 2 ∇x x L(x 0 , λ0 ) [A(x 0 )]T 0 J = . (3.85) A(x 0 ) 0
Owing to the strong second order sufficient condition and the linear independence condition, it is a nonsingular matrix. Indeed, suppose s J0 = 0. u Then A(x 0 )s = 0. Moreover, T 0 s T s u J = s T ∇x2x L(x 0 , λ0 )s + 2s T [A(x 0 )]T u u = s T ∇x2x L(x 0 , λ0 )s.
If s 6 = 0, then the second order condition implies the positivity of the last expression, a contradiction. Thus s = 0. But then s = [A(x 0 )]T u = 0, J0 u
and the linear independence of the rows of A(x 0 ) implies that u = 0. Since the Jacobian is nonsingular, system (3.83)–(3.84) defines in a neighˆ borhood of b 0 an implicit function (x(b), ˆ λ(b)) whose derivative at b 0 satisfies the equation: # " d x(b ˆ 0) 0 0 db J d λ(b ˆ 0) = I . db
Using (3.85) we obtain the relations: ˆ 0) d x(b ˆ 0) d λ(b + [A(x 0 )]T = 0, db db d x(b ˆ 0) A(x 0 ) = I. db
∇x2x L(x 0 , λ0 )
ˆ We can use these equations to calculate the derivatives of x(b) ˆ and λ(b) 0 at b . We leave these manipulations to the reader (Exercise 3.17) and we concentrate only on the second equation, which simply follows from the fact that gi (x(b)) ˆ = b, i = 1, . . . , m 0 . Multiplying the last displayed equation T from the left by λ0 we get 0 T d x(b ˆ 0 ) 0 T = λ . λ A(x 0 ) db
153
OPTIMALITY CONDITIONS
Using (3.83) at b 0 we obtain
∇ f (x 0 )
T d x(b ˆ 0)
T = − λ0 .
db The left hand side of this equation is the transpose of the derivative of the composite function f (x(b)) ˆ at b 0 and therefore T d x(b ˆ 0) d f (x(b ˆ 0 )) ∇ f (x 0 ) = −λ0 , = db db ˆ as required. As the implicit function (x(b), ˆ λ(b)) is continuous, we have gi (x(b)) ˆ < 0, i ∈ / I 0 (x 0 ), λˆ i (b) > 0, i ∈ I 0 (b0 ),
ˆ for all b sufficiently close to b 0 . Therefore the pair (x(b), ˆ λ(b)) is feasible and satisfies the first order optimality conditions. It remains to show that x(b) ˆ is indeed a local minimum. To this end we shall prove that x(b) ˆ satisfies the strong second order sufficient conditions, for all b sufficiently close to b 0 . Suppose the opposite: for every ε > 0 we can find b and s 6 = 0 such that kb − b 0 k < ε and
A(xˆ (b))s = 0,
ˆ s, ∇x2x L(x(b), ˆ λ(b))s
≤ 0.
(3.86)
We can always normalize s to get ksk = 1. Let us consider a sequence εk ↓ 0 and the corresponding sequences {b k } and {s k } satisfying the above conditions. By construction, we have b k → b0 . The continuity of the implicit ˆ k ) → λ0 . By choosing a subsequence function renders x(b ˆ k ) → x 0 and λ(b (if needed) we also have the convergence of the sequence {s k } to some limit s¯ of length 1. Passing to the limit in (3.86) we obtain a contradiction with the strong second order condition at x 0 . If the linear independence condition or the semi-strong second order sufficient condition are not satisfied, the value function f (x(b, ˆ c)) may be nondifferentiable. The sensitivity analysis then requires specialized tools of nonsmooth calculus, and exceeds the scope of this book. In Section 4.6 we return to the sensitivity analysis in the convex case, without the strong second order sufficient condition. EXERCISES 3.1. Let
X = {x ∈
n
: hai , xi ≤ bi , i = 1, . . . , m},
154
CHAPTER 3
with given ai ∈ n and bi ∈ , i = 1, . . . , m. The analytic center of X is defined as the point at which the function f (x) = b1 − ha1 , xi b2 − ha2 , xi · · · bm − ham , xi
attains its maximum. Find an algebraic characterization of the analytic center.
3.2. We have a discrete real random Pvariable Y with realizations y k , k = 1, . . . , n, attained with probabilities pk > 0, nk=1 pk = 1. (a) Find the minimum of the function
f (x) = E (Y − x)2 .
(b) Find the minimum of the function
f (x) = E |Y − x| .
(c) For α ∈ (0, 1) find the minimum of the function f (x) = E max α(Y − x), (1 − α)(x − Y ) .
3.3. Consider the problem:
minimize (x 1 − 2)2 + (x 2 − 1)2 subject to x 1 + x 2 ≤ b, x 2 ≥ 0.
How do the solution x(b) ˆ and the optimal value f (x(b)) ˆ depend on the parameter b? Calculate the derivative d f (x(b)) ˆ db and relate it to the value of the Lagrange multiplier. 3.4. Consider the problem: minimize − x 1 + x 2
subject to (x 1 − 2)3 + x 2 ≤ b, x 2 ≥ 0.
How do the solution x(b) ˆ and the optimal value f (x(b)) ˆ depend on the parameter b? Calculate the derivative d f (x(b)) ˆ db and relate it to the value of the Lagrange multiplier.
155
OPTIMALITY CONDITIONS
3.5. Consider the problem maximize x 1 x 2 subject to x 1 + 2x 2 = 4. Is the objective function convex or concave? Is the feasible set convex? Using necessary conditions of optimality find the optimal solution. Prove its optimality by second order sufficient conditions. 3.6. Describe the orthogonal projection on the set X = {x ∈
n
: kxk ≤ 1, x ≥ 0}.
3.7. A ship at sea is observed from three stations located on the coastline 500 meters apart. At each station i , the angle αi between the line to the ship and the normal to the coastline is measured (see Figure 3.8). The observed angles (which are subject
α3
α2
α1
Figure 3.8. Three stations observe the ship (Exercise 3.7). to measurement errors) are α˜ 1 = 0.083, α˜ 2 = 0.024, α˜ 3 = −0.017 (in radians). Correct these results in a consistent way, so that the lines given by the angles cross at one point, and the sum of the squares of the differences of the errors is minimized. Then determine the location of the ship. Use the approximation tan α ≈ α for small angles. Hint: Express the condition that the lines cross at one point as a linear constraint on the angles. 3.8. The output variable y ∈ of a system depends on the input variable x ∈ . We have collected observations y˜ i , i = 1, . . . , N, for N different settings x i of the input variable x, with x 1 < x 2 < · · · < x N .
156
CHAPTER 3
(a) We know that the dependence of y on x should be nondecreasing. Formulate the problem of finding new values yi such that y1 ≤ y2 ≤ · · · ≤ y N and the sum of the squares of adjustments is minimized. (b) We know that the dependence of y on x should be convex and nondecreasing. Formulate the problem of finding new values yi such that this condition is satisfied and the sum of the squares of adjustments is minimized. (c) Solve both problems for N = 4, input values x i equal to 1, 2, 5, and 10, and the corresponding observations y˜ i equal to 2, 1, 5, and 9. 3.9. We have three assets with uncorrelated returns Ri , i = 1, 2, 3. They all have variance 1, and the mean values are 1, 2, and 3, respectively. We consider nonnegative portfolio allocations x 1 , x 2 , x 3 totaling one, and the portfolio return R(x) = x 1 R1 + x 2 R2 + x 2 R3 as a function of x. Let r (x) denote its mean, and V (x) its variance. We considered this example in the Introduction (see (1.4)). (a) Find the minimum variance portfolio as the solution of the problem minimize V (x) subject to x ∈ X 0 , where X 0 is the set of possible allocations. Denote by m 0 its expected return. (b) The mean–variance efficient portfolios are computed by solving the problem minimize V (x) subject to r (x) = m, x ∈ X 0,
for all m ≥ m 0 for which this problem has a solution. Find the set of all mean–variance efficient portfolios (so-called efficient frontier). (c) Assume that in addition we have a risk-free asset with a guaranteed return of 0.5. Describe the efficient frontier in this case. 3.10. Let p > 1 and let kxk p =
n X
j =1
|x j | p
1
p
.
(a) Solve the problem minimize c T x subject to kxk p ≤ 1.
157
OPTIMALITY CONDITIONS
(b) Prove that the point xˆ found in (a) solves the problem minimize kxk p
subject to c T x ≤ c T x. ˆ 3.11. Derive the necessary conditions of optimality for the problem minimize c T x − ε subject to Ax = b,
n X
ln x j
j =1
where ε > 0, A is an m × n matrix of rank m, and c and b are given. 3.12. A dynamical system evolves in time according to the equation x(t + 1) =
1 x(t) + u(t), t = 0, 1, . . . , T − 1, 2
in which x(t) ∈ denotes the state value at time t and u(t) ∈ is the control value at time t. The initial state is x(0) = 1. Find the sequence of controls u(0), . . ., u(T −1) and the resulting sequence of states x(0), . . . , x(T ) that minimize the cost function of the system T −1 2 2 1 1 X F(x, u) = u(t) + x(T ) . 2 2
t=0
How will the solution will change if we additionally impose the requirement that x(T ) = 0?
3.13. The performance of a technical device is described by m quality measures f j : n → , j = 1, . . . , m. The smaller their values the better the performance is. The requirements of technical feasibility have been formulated in the form of s inequalities: gi (x) ≤ 0, i = 1, . . . , s.
The vector x ∈ n represents design decisions and gi : n → , i = 1, . . . , s. All functions f j (·) and gi (·) are continuously differentiable. The builder is considering three different approaches to the design problem.
Approach 1: She selects one of the performance measures, for example f 1 (x), to minimize, while keeping all the other measures f j (x) below some specified target levels b j , j = 2, . . . , m. Approach 2: The builder creates an aggregate objective function f (x) =
m X
w j f j (x),
j =1
where w j ≥ 0 are some selected weights, at least one of which is positive.
158
CHAPTER 3
Approach 3: She uses target values b j and weights w j to create an aggregate objective function F(x) = max w j f j (x) − b j 1≤ j ≤m
representing the worst weighted excess over the targets.
(a) For each of these three approaches formulate the resulting nonlinear optimization problem. (b) Assuming that constraint qualification is satisfied, formulate the first order necessary conditions of optimality for the corresponding problem. (c) Prove that for every solution obtained by Approach 2 we can find target levels b j such that the same solution is optimal in Approach 1 and in Approach 3. (d) Prove that if all functions are convex and a constraint qualification condition is satisfied, for every solution obtained by Approach 1 we can find weights w j such that the same solution is optimal in Approach 2 and Approach 3. (e) Prove that for every solution obtained by Approach 3 we can find a performance measure to minimize so that the same solution is optimal in Approach 1. Prove that if the functions are convex then this optimal solution is also optimal in Approach 2. 3.14. A large retailer wants to establish n distribution centers to serve m stores located at points s 1 , . . . , s m . The average monthly demand of the j th store is D j , j = 1, . . . , m, and the cost of supplying a store from a center is proportional to the distance between them, and to the amount supplied. For simplicity we do not distinguish different products. Formulate this problem as a nonlinear optimization problem with a smooth objective function and convex constraints. Write and analyze the necessary conditions of optimality. 3.15. Consider problem (1.7) of optimal processing the signal from n receivers. Formulate for it the necessary and sufficient conditions of optimality. Assume that the matrix 2 is positive definite. 3.16. Consider the nonlinear optimization problem minimize f (x, z) subject to gi (x, z) ≤ 0, i = 1, . . . , m, with the decision vector x ∈ n and some model parameters z ∈ l . We know that z ∈ Z for some compact set Z ⊂ l , but the exact value of z is not known. We assume that the functions f : n × l → and gi : n × l → , i = 1, . . . , m, are convex with respect to the first argument, for all values of the second argument z ∈ Z . The principle of guaranteed result requires determination of the values of decision variables that minimize the worst possible value of f (·), subject to the requirement
159
OPTIMALITY CONDITIONS
that the constraints are satisfied for all possible values of z in Z . Formulate this problem as a convex optimization problem, and formulate necessary conditions of optimality for it. 3.17. Consider problem (3.82) under the conditions of Theorem 3.49. Calculate the derivatives of the optimal solution and the Lagrange multipliers with respect to the parameters b and c. 3.18. Consider the convex programming problem minimize f (x), subject to x ∈
m \
Xi ,
i=1
where f : n → is a convex function, and the sets X i , i = 1, . . . , m are closed convex subsets of n . Suppose it has an optimal solution x. ˆ Prove that there exist at most n sets X i1 , . . . , X ik , k ≤ n, such that xˆ is also an optimal solution of the problem k \ minimize f (x), subject to x ∈ X ij .
j =1
Hint: Suppose the statement is not true, and apply Helly’s theorem (Theorem 3.10 on page 97) to obtain a contradiction.
Chapter Four Lagrangian Duality 4.1 THE DUAL PROBLEM
We start from the general nonlinear optimization problem minimize f (x) subject to gi (x) ≤ 0, i = 1, . . . , m, h i (x) = 0, i = 1, . . . , p, x ∈ X0,
(4.1)
with some functions f : n → , gi : n → , i = 1, . . . , m, h i : n → , i = 1, . . . , p, and with a set X 0 ⊂ n . At this point we do not make any convexity assumptions about the functions of this problem and about the set X 0 . The Lagrangian has the form: m X
L(x, λ, µ) = f (x) +
i=1
λi gi (x) +
p X
µi h i (x).
(4.2)
i=1
We view it as a function of both x ∈ X 0 and (λ, µ) ∈ Λ0 , where Λ0 =
m +
×
p
.
We define the primal function associated with problem (4.1) as M
LP (x) =
sup
L(x, λ, µ),
(4.3)
LD (λ, µ) = inf L(x, λ, µ).
(4.4)
(λ,µ)∈Λ0
and the dual function as M
x∈X 0
If the supremum in (4.3) is +∞ we set L P (x) = +∞. Similarly, if the infimum in (4.4) is −∞ we set L D (λ, µ) = −∞. Thus, we consider the primal function L P (·) and the dual function L D (·) as functions attaining values in the extended real line = ∪ {+∞} ∪ {−∞}.
161
LAGRANGIAN DUALITY
The primal problem is to find min LP (x),
(4.5)
max LD (λ, µ).
(4.6)
x∈X 0
and the dual problem is to find (λ,µ)∈Λ0
The duality theory investigates relations between the primal problem and the dual problem. We can easily calculate the primal function at each x ∈ X 0 . If x satisfies all constraints of problem (4.1) then the terms in (4.2) that depend on λ are nonpositive and the µ-terms are identically 0. Consequently, L P (x) = f (x). Suppose at least one constraint is violated, e.g., g j (x) > 0. Then increasing λ j in (4.2) we can obtain arbitrarily large values of λ j g j (x). Thus L P (x) = +∞ in this case. A similar argument can be made for a violated equality constraint, and thus ( f (x) if x is feasible for (4.1), LP (x) = (4.7) +∞ otherwise. We conclude that the primal problem (4.5) is equivalent to the original problem (4.1). The dual function, however, is more difficult to calculate in an explicit form. In some cases, like linear or quadratic programming, a closed form of the dual problem can be derived. Before proceeding to the analysis of the properties of the dual problem, we present several examples. Example 4.1. Consider the projection problem, 1 kx − zk2 2 subject to Ax = 0, minimize
where z ∈ n is fixed, and A is a matrix of dimension m × n. We considered this problem in Section 2.1.2. Introducing multipliers µ ∈ m we can write the Lagrangian:
L(x, µ) =
1 kx − zk2 + hµ, Axi. 2
Differentiation with respect to x yields the minimizer in the problem at the right hand side of (4.4): x(µ) ˆ = z − A Tµ.
162
CHAPTER 4
This can be substituted to the Lagrangian to obtain the dual function. The dual problem takes on the form: 1 maximize − kA Tµk2 + hµ, Azi. 2 It is an unconstrained quadratic optimization problem. If we add and subtract the constant 21 kzk2 to complement to the full square, we can write the dual problem as follows: 1 − kA Tµ − zk2 + kzk2 . maximize 2
Its solution, A Tµ, ˆ is the point of the range of A T that is closest to z: the projection of z onto the orthogonal subspace to the null space of A. We can also note that the optimal values of the original and the dual problems are equal, because of the Pythagorean Theorem: kzk2 = kxˆ − zk2 + kA Tµˆ − zk2 . The equality of the optimal values of both problems is not coincidental. We explain it in the next section. Example 4.2. Consider the linear programming problem minimize hc, xi subject to Ax ≥ b,
(4.8)
x ≥ 0,
with a matrix A of dimension m × n, c ∈ n and b ∈ m . In (4.1) we have f (x) = hc, xi, g(x) = b − Ax and X 0 = n+ . We formulate the Lagrangian:
L(x, λ) = hc, xi + hλ, b − Axi = hc − A Tλ, xi + hb, λi. The dual function has the form LD (λ) = hb, λi + inf hc − A Tλ, xi. x≥0
By Lemma 2.26 on page 145, the infimum above is finite if and only if c − A Tλ ≥ 0. In this case the infimum value is just zero. We conclude that ( hb, λi if A Tλ ≤ c, LD (λ) = −∞ otherwise. The dual problem can therefore be formulated as the linear programming problem maximize hb, λi
subject to A Tλ ≤ c, λ ≥ 0.
(4.9)
163
LAGRANGIAN DUALITY
A similar analysis can be carried out for a formulation of a linear programming problem with equality constraints. It is instructive to consider the dual problem to the dual problem (4.9). We change the sign of the objective function to obtain a minimization problem and we use u ∈ n+ to denote the Lagrange multipliers associated with the constraints. We define the Lagrangian:
l(λ, u) = −hb, λi + hu, A Tλ − ci = hAu − b, λi − hc, ui. The dual function can be calculated similarly to the previous case: ( −hc, ui if Au ≥ b, lD (u) = inf l(λ, u) = λ≥0 −∞ otherwise. We can thus write the dual problem as maximize − hc, ui subject to Au ≥ b, u ≥ 0.
Changing the sign of the objective function again, we arrive at the primal problem (4.8). It should be stressed that the fact that the dual problem to the dual problem coincides with the original problem is due to the bilinear form of the Lagrangian. This is true for linear programming and, more generally, for problems with linear objective, linear constraints and additional cone constraint, as in the next example. Example 4.3. Given symmetric matrices A 1 , . . . , Am and C of dimension n, the semidefinite programming problem is formulated as follows: minimize tr(C X) subject to tr(Ai X) = bi , i = 1, . . . , m,
(4.10)
X ∈ Sn+ .
The last condition means that the decision variable X in this problem is a symmetric positive semidefinite matrix of dimension n. We have considered this problem in Example 3.36 on page 129. We show now that its dual problem can be written explicitly. Our problem has form (4.1) with f (X) = hC, XiS
h i (X) = bi − hAi , XiS , i = 1, . . . , m, X 0 = Sn+ ,
with the symbol h·, ·iS denoting the Frobenius inner product (see page 31). The functions f (·) and h i (·) are affine. Assume Slater’s constraint qualification condition (see page 109). It means that there exists a positive definite matrix X S satisfying the equality constraints.
164
CHAPTER 4
The Lagrangian has the form L(X, µ) = hC, XiS + D
= C− Let us calculate the dual function " LD (µ) = infn
X∈S+
m X
m X i=1
µi bi − hAi , XiS
µi A i , X
i=1
E
S
+ hb, µi.
# m D E X C− µi Ai , X + hb, µi . S
i=1
Consider for a fixed µ the problem m E D X minimize µ A , X . C − i i n X∈S+
S
i=1
(4.11)
For the infimum P to be finite, it is necessary and sufficient that the Frobenius scalar product of C − m i=1 µi A i and X be bounded below for all X belonging to the cone Sn+ . By Lemma 2.26 on page 29, this is equivalent to C−
m X i=1
◦ µi Ai ∈ − Sn+ .
◦ The polar cone is calculated in Example 2.31: Sn+ = −Sn+ . Therefore LD (µ) > −∞ if and only if m X µi Ai ∈ Sn+ . (4.12) C− i=1
Under this condition the lower bound of the scalar product in (4.11) is zero, and it ˆ is attained at X(µ) = 0. Then LD (µ) = hb, µi. We have arrived at the following formulation of the dual problem: maximize hb, µi subject to C −
m X i=1
µi Ai ∈ Sn+ .
(4.13)
It is a semidefinite programming problem again.
Let us return to the general nonlinear optimization problem (4.1) and the associated dual function (4.4). The dual function has many remarkable properties facilitating the solution of the dual problem. L EMMA 4.4. The dual function L D (λ, µ) is concave.
165
LAGRANGIAN DUALITY
Proof. Since the Lagrangian L(x, λ, µ) is affine in (λ, µ) for every x ∈ X 0 , the dual function is an infimum of a family of affine functions. Thus −L D (·) is a supremum of a family of affine functions and is convex by Lemma 2.58 on page 46. Our second observation uses the affine function at which the minimum is achieved. L EMMA 4.5. Assume that for (λ0 , µ0 ) we can find x 0 ∈ X 0 such that LD (λ0 , µ0 ) = L(x 0 , λ0 , µ0 ). Then for all (λ, µ) we have LD (λ, µ) ≤ LD (λ0 , µ0 ) + hg(x 0 ), λ − λ0 i + hh(x 0 ), µ − µ0 i,
(4.14)
where g(x) and h(x) are the vectors with coordinates g i (x), i = 1, . . . , m, and h i (x), i = 1, . . . , p, respectively. Proof. By the definition of the dual function LD (λ, µ) ≤ L(x 0 , λ, µ) = L(x 0 , λ0 , µ0 )+hg(x 0 ), λ−λ0 i+hh(x 0 ), µ−µ0 i, which was what we set out to prove. We can define the subdifferential of the dual function at (λ 0 , µ0 ) as the set of vectors (sλ , sµ ) ∈ m+ p such that LD (λ, µ) ≤ LD (λ0 , µ0 )+hsλ , λ−λ0 i+hsµ , µ−µ0 i for all (λ, µ) ∈
m+ p
.
Note that LD (·) is concave, and therefore we appropriately modify Definition 2.72 of the subdifferential of a convex function. Our definition corresponds to the set −∂ − LD (λ0 , µ0 ) , and thus all properties of the subdifferential of a convex function can be easily translated to properties of a subdifferential of a concave function. We still use the notation ∂ L D (λ0 , µ0 ) for the subdifferential; it never leads to any confusion. If the set X 0 is compact, Theorem 2.87 provides the representation of the subdifferential: 0 [ g(x ) 0 0 ∂ LD (λ , µ ) = conv , (4.15) h(x 0 ) ˆ 0 ,µ0 ) x 0 ∈ X(λ
ˆ 0 , µ0 ) is the set of the minimizers of the Lagrangian: where X(λ ˆ 0 , µ0 ) = {x ∈ X 0 : L(x 0 , λ0 , µ0 ) = LD (λ0 , µ0 )}. X(λ
166
CHAPTER 4
If the set X 0 is not compact, we can still describe the entire subdifferential of the dual function by using the fact that it is defined as a minimum of affine functions, rather than just concave functions. We present these calculations at the end of Section 4.6. Formula (4.14) provides us with means to apply nonsmooth optimization methods to the solution of the dual problem. We discuss these methods in Chapter 7. But the main question is, of course: why should we solve the dual problem at all? In the next section we show that the solutions of the dual problem and the solutions of the primal problem are very closely related. 4.2 DUALITY RELATIONS
Let us return to the Lagrangian of problem (4.1): L(x, λ, µ) = f (x) +
m X i=1
λi gi (x) +
p X
µi h i (x).
i=1
Recall that we consider it as a function of both x ∈ X 0 and (λ, µ) ∈ Λ0 , p . The following concept of a saddle point is central where Λ0 = m + × to the duality theory. ˜ µ)) D EFINITION 4.6. A point (x, ˜ (λ, ˜ ∈ X 0 × Λ0 is called a saddle point of the Lagrangian, if for all x ∈ X 0 and all (λ, µ) ∈ Λ0 the following inequalities are satisfied: ˜ µ) ˜ µ). L(x, ˜ λ, µ) ≤ L(x, ˜ λ, ˜ ≤ L(x, λ, ˜
(4.16)
In other words, a saddle point is such a point at which the maximum of the Lagrangian with respect to (λ, µ) ∈ Λ 0 and the minimum with respect to x ∈ X 0 are attained: ˜ µ) ˜ µ). max L(x, ˜ λ, µ) = L(x, ˜ λ, ˜ = min L(x, λ, ˜
(λ,µ)∈Λ0
x∈X 0
For convex optimization problems the optimal solution and its Lagrange multipliers (if they exist) constitute such a saddle point. The theorem below uses the first order optimality conditions in the subdifferential form (Theorem 3.34). An identical result holds for problem (4.1) with smooth convex functions, analyzed in Theorem 3.25. T HEOREM 4.7. Assume that the functions f (·) and g i (·), i = 1, . . . , m, in problem (4.1) are convex, the functions h i (·), i = 1, . . . , p, are affine, and the set X 0 is convex. Then a point xˆ satisfies the first order optimality ˆ µ) conditions of Theorem 3.34 with Lagrange multipliers ( λ, ˆ if and only if ˆ (x, ˆ (λ, µ)) ˆ is a saddle point of the Lagrangian.
167
LAGRANGIAN DUALITY
Proof. We provide the proof for the convex nonsmooth case. Readers who prefer to deal with the smooth case of Theorem 3.25 can substitute the gradients ∇x L, ∇ f , and ∇gi for the subdifferentials ∂ x L, ∂ f , and ∂gi . Assume that xˆ is an optimal solution satisfying the optimality conditions ˆ µ). ˆ µ) with multipliers (λ, ˆ For fixed values of (λ, ˆ ∈ Λ0 , the Lagrangian is a convex function of x. By Theorem 2.85, its subdifferential at xˆ can be calculated as follows: ˆ µ) ∂x L(x, ˆ λ, ˆ = ∂ f (x) ˆ +
m X i=1
λˆ i ∂gi (x) ˆ +
p X i=1
µˆ i ∇h i (x). ˆ
The first order optimality condition of Theorem 3.33 has the form ˆ µ) 0 ∈ ∂x L(x, ˆ λ, ˆ + N X 0 (x). ˆ
(4.17)
It follows from Theorem 3.46 (or Theorem 3.24 in the smooth case) that ˆ µ) ˆ µ). L(x, ˆ λ, ˆ = min L(x, λ, ˆ x∈X 0
Thus the right inequality in (4.16) holds true for all x ∈ X 0 . We shall prove the left inequality. By the complementarity condition (3.50), we have ˆ µ) L(x, ˆ λ, ˆ = f (x). ˆ Since gi (x) ˆ ≤ 0, i = 1, . . . , m, and h i (x) ˆ = 0, i = 1, . . . , p, for every (λ, µ) ∈ Λ0 we have the inequality L(x, ˆ λ, µ) = f (x) ˆ +
m X i=1
λi gi (x) ˆ +
p X i=1
µi h i (x) ˆ ≤ f (x). ˆ
Combining the last two displayed relations we obtain the left inequality in ˆ µ)) (4.16) for all (λ, µ) ∈ Λ0 . Therefore the point (x, ˆ (λ, ˆ is a saddle point of the Lagrangian. ˆ µ)) Let us now prove the converse. Suppose (x, ˆ ( λ, ˆ is a saddle point of the Lagrangian. By virtue of Theorem 3.46, the right inequality in (4.16) is equivalent to (4.17). This is identical with (3.49). P The left of the saddle point conditions (4.16) implies that m ˆ i=1 λi gi ( x) is bounded from above for all λ ≥ 0. Hence g i (x) ˆ ≤ 0, i = 1, . . . , m. Similarly, h i (x) ˆ = 0,Pi = 1, . . . , p. Consequently, the point xˆ is feasible. ˆ we have λˆ i gi (x) As the maximum of m ˆ is attained at λ, ˆ = 0 for i=1 λi gi ( x) all i = 1, . . . , m. Thus, (3.50) holds true as well. The relation between saddle points and the primal and dual problems is straightforward.
168
CHAPTER 4
˜ µ), T HEOREM 4.8. If the Lagrangian has a saddle point (x, ˜ λ, ˜ then x˜ is a ˜ µ) solution of the primal problem, (λ, ˜ is a solution of the dual problem, and the duality relation holds true: min LP (x) = max LD (λ, µ).
x∈X 0
(λ,µ)∈Λ0
(4.18)
Proof. By the saddle point relations, for every x ∈ X 0 and every (λ, µ) ∈ Λ0 we have ˜ µ) ˜ µ). L(x, ˜ λ, µ) ≤ L(x, ˜ λ, ˜ ≤ L(x, λ, ˜
(4.19)
˜ µ). This means that L P (x) ˜ = LD (λ, ˜ The minimization of the left hand side of (4.19) in x and the maximization of the right hand side in (λ, µ) may only strengthen the inequalities, so ˜ µ) ˜ µ) LD (λ, µ) ≤ L(x, ˜ λ, µ) ≤ LD (λ, ˜ = LP (x) ˜ ≤ L(x, λ, ˜ ≤ LP (x), which completes the proof of the duality relation. Note that no explicit convexity assumptions are made in this theorem. The duality relation is illustrated in Figure 4.1. We can also state the converse implication to Theorem 4.8. T HEOREM 4.9. Assume that the duality relation (4.18) is satisfied with finite values of the primal and dual functions. Then for every solution xˆ of ˆ µ) the primal problem and for every solution ( λ, ˆ of the dual problem the ˆ µ) point (x, ˆ λ, ˆ is a saddle point of the Lagrangian. Proof. The definition of the primal function yields ˆ µ). ˆ λ, µ) ≥ L(x, ˆ λ, ˆ LP (x) ˆ = max L(x, (λ,µ)∈Λ0
Similarly, from the definition of the dual function we obtain ˆ µ) ˆ µ) ˆ µ). LD (λ, ˆ = min L(x, λ, ˆ ≤ L(x, ˆ λ, ˆ x∈X
Since by (4.18) ˆ µ), LP (x) ˆ = LD (λ, ˆ we must have ˆ µ) ˆ µ) ˆ µ) L(x, ˆ λ, µ) ≤ L P (x) ˆ = L(x, ˆ λ, ˆ = LD (λ, ˆ ≤ L(x, λ, ˆ ˆ µ) for all x ∈ X 0 and all (λ, µ) ∈ Λ0 . Thus (x, ˆ λ, ˆ is a saddle point.
(4.20)
169
LAGRANGIAN DUALITY 4
f (x)
3.5
λ=1
3 2.5
λ=2
2 1.5
λ=3
1 0.5
x
0 −0.5 −1
g(x) 0
0.5
1
1.5
2
2.5
Figure 4.1. Illustration of the duality relation for the problem of minimizing f (x) = x 2 subject to g(x) = 1 − x ≤ 0. The graphs represent the Lagrangian, as a function of x, for several fixed values of λ. The minimum of the Lagrangian, i.e., the dual function, is maximized when it is equal to the minimum of the constrained problem.
Based on these results we can look for a solution to the primal problem by ˆ µ) first solving the dual problem to get ( λ, ˆ and then determining the primal solution xˆ from the saddle point conditions. T HEOREM 4.10. Assume that the duality relation (4.18) holds true. If ¯ µ) (λ, ¯ ∈ Λ0 is a feasible point of the dual problem, then every point xˆ ∈ X 0 such that ¯ µ) ¯ µ); (i) L(x, ˆ λ, ¯ = min L(x, λ, ¯ x∈X 0
(ii) all constraints of (4.1) are satisfied at x; ˆ (iii) λ¯ i gi (x) ˆ = 0, i = 1, . . . , m, is a solution of problem (4.1).
170
CHAPTER 4
¯ µ) Proof. Conditions (i), (ii) and (iii) imply that (x, ˆ λ, ¯ is a saddle point of the Lagrangian, and the result follows from Theorem 4.8. In fact, the point ¯ µ) (λ, ¯ is then the optimal solution of the dual problem. The following example from game theory motivated the development of the duality theory. Example 4.11. A two-player game is defined as follows. We have a matrix A of dimension m × n. Player C chooses a column j of the matrix, that is, a number between 1 and n. Player R chooses a row of the matrix, a number i between 1 and m. None of them knows the opponent’s decision. Then the decisions are revealed and the matrix entry ai j in row i and column j is the amount that C pays R (if it is negative, R pays C). It is difficult to decide on the course of action in such a game. The key concept that radically clarifies the problem of choosing the best play is that of a mixed strategy. A mixed strategy of Player C is a probability distribution x on the set of columns {1, . . . , n}. In other words, we assume that Player C chooses her column at random, according to some probabilities x j , j = 1, . . . , n. The set of all possible mixed strategies is equal to X 0 = {x ∈
n
n X
:
j =1
x j = 1, x j ≥ 0, j = 1, . . . , n}.
If Player R chooses row i then the expected amount that C pays R equals n X
j =1
ai j x j = hai , xi,
with ai denoting the i th row of A. To minimize the worst possible outcome, Player C solves the following problem: minimize max hai , xi. x∈X 0
1≤i≤m
(4.21)
As usual in min-max problems, we can reformulate it as a constrained problem: minimize v v,x
subject to hai , xi − v ≤ 0, i = 1, . . . , m, x ∈ X0.
The set X 0 is a compact polyhedron and the constraints are affine. Therefore, problem (4.21) has an optimal solution and the first order necessary conditions of optimality are satisfied. Denote by y the vector of Lagrange multipliers associated with the constraints. We use the notation y instead of λ because of the specific interpretation of the
171
LAGRANGIAN DUALITY
multipliers in our context. The Lagrangian has the form L(v, x, y) = v +
m X i=1
m X
yi hai , xi − v = v 1 −
m X i=1
yi + hy, Axi.
Let us calculate the dual function: LD (y) = inf L(v, x, y) = inf v 1 − v∈ x ∈X 0
v∈
i=1
yi + min hy, Axi. x∈X 0
We see that the infimum with respect to v is finite if and only if y is an element of the set Y0 = {y ∈
m
:
n X i=1
Pn
i=1 yi
= 1. Thus
yi = 1, yi ≥ 0, i = 1, . . . , m}.
The dual function becomes LD (y) = min hy, Axi = min hA T y, xi. x∈X 0
x∈X 0
If we use a j to denote the j th column of A, we can rewrite the dual function as follows: n X LD (y) = min x j ha j , yi = min ha j , yi. x∈X 0
1≤ j ≤n
j =1
The dual problem takes on the form maximize min ha j , yi. y∈Y0
1≤ j ≤n
(4.22)
A comparison with (4.21) reveals that the dual problem is the problem of finding the best mixed strategy of Player R. By using the fact that max hai , xi = max hy, Axi,
1≤i≤m
y∈Y0
we can rewrite both problems as follows. The primal problem is minimize max hy, Axi, x∈X 0
y∈Y0
and the dual problem is maximize min hy, Axi. y∈Y0
x∈X 0
The Lagrangian for every x ∈ X 0 and y ∈ Y0 has the form L(x, y) = hy, Axi.
172
CHAPTER 4
From the duality theory it follows that the optimal values of both problems are identical and that their solutions, xˆ and yˆ , form a saddle point of the Lagrangian hy, A xi ˆ ≤ h yˆ , A xi ˆ ≤ h yˆ , Axi for all x ∈ X 0 , y ∈ Y0 . It is the equilibrium of the game: if both players follow the mixed strategies xˆ and yˆ , it is not profitable for any of them to deviate from their solutions.
Our second example is from the area of finance. Example 4.12. We have n securities with present prices c1 , . . . , cn . The securities can be either bought or sold short, that is, borrowed and sold for cash. Any amounts can be traded. At some future time one of m states may occur. The price of security j in state i will be equal to ai j , i = 1, . . . , m, j = 1, . . . , n. At this time we will liquidate our holdings: the securities held will be sold, and the short positions will be covered by purchasing and returning the amounts borrowed. ¯ < 0 and A x¯ ≥ 0. Such An arbitrage is the existence of x¯ ∈ n such that hc, xi a portfolio x¯ can be “purchased” with an immediate profit and then liquidated at a future point in time with no additional loss, whatever the state. We analyzed this problem in Example 2.30 on page 31. Now we can interpret the results in the context of the duality theory. Consider the linear programming problem
minimize hc, xi
subject to Ax ≥ 0. If this problem has a feasible solution x¯ with hc, xi ¯ < 0, then no optimal solution exists, because the value of the objective goes to −∞ at feasible points M x, ¯ with M → ∞ (this is the essence of the concept of arbitrage). Thus, an optimal solution exists if and only if the optimal value is 0. Using p ∈ n , p ≥ 0, to denote the Lagrange multipliers, we can write the Lagrangian in the form
L(x, p) = hc, xi − h p, Axi = hc − A T p, xi. The infimum of this function with respect to x ∈ n is −∞, unless c − A T p = 0, in which case the infimum is simply zero. It follows that ( 0 if A T p = c, LD ( p) = inf L(x, p) = x∈ n −∞ otherwise.
The dual problem, therefore, has the simple form of a system of equations A T p = c. This is the non-arbitrage condition of Example 2.30, and the Lagrange multipliers p are the state prices found there.
173
LAGRANGIAN DUALITY
Finally, we have an illustration of relations of duality theory to expected utility theory in economics. Example 4.13. Our decisions x ∈ X ⊂ n affect a certain random outcome F(x). We assume that the number of possible elementary events is finite and we denote it by m. For every elementary P event i ∈ {1, . . . , m}, we denote its probability by p i . Obviously, pi > 0 and m i=1 pi = 1. The realization of the random outcome F(x) in event i is denoted by f i (x). To focus attention, we assume that we prefer larger values of f i (x) over smaller values (for example, f i (x) may represent profit in event i ). We also assume that the functions f i : n → are concave, and the set X is convex. One way to formulate a meaningful optimization problem, in the presence of uncertainty and with risk aversion, is to consider a benchmark random outcome Y , with realizations yi , i = 1, . . . , m. We require that our random outcome F(x) has an expected shortfall below each target level t ∈ at most as large as that of Y : E max 0, t − F(x) ≤ E max 0, t − Y for all t ∈ . (4.23)
The symbol E[·] denotes the expected value. In the theory of stochastic optimization models, constraint (4.23) is called the second order stochastic dominance constraint. Observe that the functions t 7→ E max 0, t−F(x) and t 7→ E max 0, t−Y are convex. The second one is also piecewise linear, with break points at y 1 , . . . , ym . Thus it is sufficient to enforce constraint (4.23) at the break points t = y j : E max 0, y j − F(x) ≤ E max 0, y j − Y for all j = 1, . . . , m. We formulate the optimization problem maximize
m X
pi fi (x)
i=1
subject to
m X i=1
m X pi max 0, y j − fi (x) ≤ pi max 0, y j − yi , j = 1, . . . , m, i=1
x ∈ X.
(4.24) The objective function represents our desire to have the average value of the outcome F(x) large, while the constraint represents risk aversion: we want to perform better with respect to each target level than the benchmark Y . We can develop the dual problem to problem (4.24). Denoting by λ j , j = 1, . . . , m, the nonnegative Lagrange multipliers associated with the shortfall constraints, we write the Lagrangian: L(x, λ) =
m X i=1
pi f i (x) −
m X j =1
λj
m X i=1
pi max 0, y j − fi (x) − max 0, y j − yi .
(4.25)
174
CHAPTER 4
Define the function u :
→
as
u(t) = −
m X j =1
λ j max 0, y j − t .
(4.26)
By changing the order of summation in (4.25) and using (4.26) we obtain the identity −
m X j =1
λj
m X i=1
m m X X pi max 0, y j − f i (x) = − pi λ j max 0, y j − f i (x)
= Similarly, m X
j =1
λj
m X i=1
i=1 m X i=1
j =1
pi u f i (x) = E u(F(x)) .
m X pi max 0, y j − yi = − pi u yi = −E u(Y ) . i=1
The function u(·) is concave and piecewise linear with break points at y 1 , . . . , ym . It vanishes to the right of the largest realization of Y . Every nonnegative vector λ of Lagrange multipliers can be associated with a function u(·) defined in (4.26), and every function of such a form defines a vector of Lagrange multipliers λ ≥ 0. Thus we can rewrite the Lagrangian (4.25) as a function of x and u(·): L(x, u) = E F(x) + E u(F(x)) − E u(Y ) . (4.27) We keep the symbol L for the Lagrangian, although it is a different function now. Its arguments are x ∈ n and a concave function u(·). All duality results for the Lagrangian (4.25) can now be formulated in terms of the Lagrangian (4.27). The dual function has the form: n o D(u) = sup E F(x) + E u(F(x)) − E u(Y ) . (4.28)
x∈X
Observe that in this problem we do not maximize the expected value of the outcome, but rather the expected value of a certain utility function ψ(t) = t + u(t), E ψ(F(x)) = E F(x) + u(F(x)) . The dual problem has the form
minimize D(u), u(·)
where the function u(·) has to be chosen from the class of piecewise linear concave functions with break points at y1 , . . . , ym , and equal to zero for all sufficiently large values of the argument. Its solution, u(·), ˆ is such that a solution xˆ of (4.28) satisfies the stochastic dominance constraint, and E u(F( ˆ x)) ˆ = E u(Y ˆ ).
175
LAGRANGIAN DUALITY
4.3 CONIC PROGRAMMING
We now focus on the following problem structure: minimize hc, xi subject to Ax = b, x ∈ K,
(4.29)
where x ∈ n , c ∈ n , b ∈ m , and A is an m × n matrix. The set K is a closed convex cone in n . The linear programming problem of Example 4.2 fits into this model with K = n+ . Another special case is the semidefinite programming problem of Example 4.3, with x denoting a symmetric matrix of dimension d (so n = d 2 ), and the scalar product h·, ·i understood in the Frobenius sense: hc, xi = tr(cx). In this case A is a linear operator from the space of square matrices to m and therefore has the form tr(A1 x) tr(A2 x) Ax = ... , tr(Am x) with Ai , i = 1, . . . , m, being square matrices of dimension d. The cone K is the semidefinite cone Sn+ . The development of the dual problem follows the argument of Examples 4.2 and 4.3. We introduce the Lagrangian
and the dual function
L(x, µ) = hc, xi + hµ, b − Axi
LD (µ) = inf L(x, µ) = inf hc − A∗µ, xi + hµ, bi. x∈K
x∈K
(4.30)
Here A∗ denotes the adjoint operator to A and is defined by the equation hµ, Axi = hA∗µ, xi. The specific algebraic form of A ∗ depends on the scalar product employed. For example, in linear programming we have P A∗ = AT , and in semidefinite programming A∗µ = m µ A . i i i=1 By Lemma 2.26, the infimum in (4.30) is finite if and only if c − A ∗µ ∈ −K ◦ , and then LD (µ) = hµ, bi. This leads to the following formulation of the dual problem: maximize hb, µi subject to A∗µ − c ∈ K ◦ .
(4.31)
We are now ready to state the duality theorem for conic optimization problems.
176
CHAPTER 4
T HEOREM 4.14. Assume that there exists a point x S ∈ int K such that Ax S = b. If the primal problem (4.29) has an optimal solution then the dual problem (4.31) has an optimal solution and the optimal values of both problems coincide. Proof. Recall from page 109 that the assumed existence of the point x S is nothing but Slater’s constraint qualification condition for problem (4.29). Under this condition, Theorem 3.34 implies that the optimal solution satisfies the first order conditions of optimality. Then it follows from Theorem 4.7 that the Lagrangian has a saddle point, and from Theorem 4.8 that the dual problem (4.31) has an optimal solution, and that the optimal values of both problems are equal. We can also conclude that under Slater’s condition for every solution xˆ of the primal problem and every solution µˆ of the dual problem, the complementarity condition hx, ˆ A∗µˆ − ci = 0
(4.32)
holds true. Indeed, the constraint of the dual problem yields hA ∗µ − c, xˆ i ≤ 0. Hence hb, µi ˆ ≤ hb, µi ˆ − hA∗µˆ − c, xi ˆ = hc, xˆ i + hb − Axˆ , µi ˆ = hc, xˆ i, with the equality occurring if and only if (4.32) is satisfied. In linear programming the cone K is polyhedral, and we do not need Slater’s condition for Theorem 4.14 to hold true. If we develop the dual problem to the dual problem (4.31) we obtain the primal problem again (see Exercise 4.8). In linear programming, the existence of an optimal solution to the dual problem implies the duality relation and the existence of an optimal solution of the primal problem, by virtue of Theorem 4.14 (without Slater’s condition). In semidefinite programming and conic programming, in general, application of Theorem 4.14 to the dual problem requires the dualSlater’s condition: there exists µS ∈ m such that A∗µS − c ∈ int K ◦ . We thus obtain the stronger version of Theorem 4.14. T HEOREM 4.15. Assume that both primal and dual Slater’s conditions are satisfied. Then the primal problem (4.29) has an optimal solution if and only if the dual problem (4.31) has an optimal solution, in which case the optimal values of both problems coincide.
177
LAGRANGIAN DUALITY
Example 4.16. Let A be an m ×n matrix and b ∈ m . Consider the approximation problem: minimize kb − Axk♦ , (4.33)
where k · k♦ is a norm in m . If k · k♦ = k · k, the above problem is simply the projection problem of Section 2.1.2, while for k · k♦ = k · k∞ we obtain the Chebyshev approximation problem of Example 3.32. Introducing new variables y ∈ m and v ∈ , as in Example 3.39, we can equivalently formulate the problem as follows:
minimize v subject to Ax + y = b, kyk♦ ≤ v.
(4.34)
M Since K = epi k · k♦ is a convex cone, we can compactly write the last constraint as (y, v) ∈ K . It follows that problem (4.34) is a conic programming problem. Specializing the derivation of (4.31), we arrive at the dual problem maximize hb, µi
subject to A Tµ = 0, (µ, −1) ∈ K ◦ .
The polar cone to K has the form K ◦ = (λ, α) ∈
m
×
: α ≤ −kλk∗ ,
where k · k∗ is the dual norm (see Exercise 2.13). Therefore the dual problem can be equivalently written as follows: maximize hb, µi
subject to A Tµ = 0, kµk∗ ≤ 1.
(4.35)
Problem (4.34) satisfies the constraint qualification condition, because v can be arbitrarily large. Therefore Theorem 4.14 applies and we conclude that the dual problem (4.35) has a solution, and that the optimal values of both problems coincide. It might be instructive to notice that the optimality conditions for any of the problems (4.33) or (4.34) are special cases of the general conditions derived in Example 3.39 for smooth nonlinear mappings A. Example 4.17. Consider the semidefinite programming problem (4.10) discussed in Example 4.3. Suppose the constraints hA i , XiS = bi imply that tr(X) = k. This is the case, for example, when (4.10) is a semidefinite relaxation of a 0–1 quadratic optimization problem, as in Exercise 4.11.
178
CHAPTER 4
Problem (4.10) augmented with the redundant constraint has the form: minimize hC, XiS subject to hAi , XiS = bi , i = 1, . . . , m,
(4.36)
hI, XiS = k, X ∈ Sn+ .
Associating with its constraints multipliers µ ∈ following form of the dual problem (4.13):
m
and µ0 ∈
maximize hb, µi + kµ0 subject to C −
m X i=1
µi Ai − µ0 I ∈ Sn+ .
, we obtain the
(4.37)
If problem (4.36) satisfies Slater’s constraint qualification,we can use Theorem 4.14. The constraint tr(X) = k implies that every optimal solution Xˆ of problem (4.36) is nonzero. In view of the complementarity condition (4.32), every optimal solution of the dual problem must satisfy the equation D
C−
m X i=1
Since Xˆ 6= 0, the matrix C − semidefinite, we must have
µi Ai − µ0 I, Xˆ
Pm
i=1
E
S
= 0.
µi Ai − µ0 I is singular. As it is positive
m X λmin C − µi Ai − µ0 I = 0. i=1
We use the notation λmin (A) and λmax (A) to denote the smallest and largest eigenvalues of a symmetric matrix A. The last relation is equivalent to m X λmin C − µi A i = µ 0 .
(4.38)
i=1
It follows that the dual problem (4.37) is equivalent to m X µi A i . maximize hb, µi + kλmin C − µ∈
n
i=1
Since λmin (A) = −λmax (−A), we can write this problem equivalently as an unconstrained convex optimization problem: minimize kλmax µ∈
n
m X i=1
µi Ai − C − hb, µi.
(4.39)
179
LAGRANGIAN DUALITY
We know from Example 2.89 on page 74 how to calculate subgradients of the maximum eigenvalue function, and thus we can apply methods of convex nonsmooth optimization to this problem. By Lemma 2.83 on page 67, each subgradient of function (4.39) has the form
k A 1 W − b1 k A 2 W − b2