Asymptotic Cones and Functions in Optimization and Variational Inequalities
Alfred Auslender Marc Teboulle
Springer
...
27 downloads
1204 Views
1MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
Asymptotic Cones and Functions in Optimization and Variational Inequalities
Alfred Auslender Marc Teboulle
Springer
This book is dedicated to Martine, Fran¸cois, and J´erˆ ome Rachel, Yoav, Yael, and Keren
This page intentionally left blank
Preface
Nonlinear applied analysis and in particular the related fields of continuous optimization and variational inequality problems have gone through major developments over the last three decades and have reached maturity. A pivotal role in these developments has been played by convex analysis, a rich area covering a broad range of problems in mathematical sciences and its applications. Separation of convex sets and the Legendre–Fenchel conjugate transforms are fundamental notions that have laid the ground for these fruitful developments. Two other fundamental notions that have contributed to making convex analysis a powerful analytical tool and that have often been hidden in these developments are the notions of asymptotic sets and functions. The purpose of this book is to provide a systematic and comprehensive account of asymptotic sets and functions, from which a broad and useful theory emerges in the areas of optimization and variational inequalities. There is a variety of motivations that led mathematicians to study questions revolving around attaintment of the infimum in a minimization problem and its stability, duality and minmax theorems, convexification of sets and functions, and maximal monotone maps. In all these topics we are faced with the central problem of handling unbounded situations. This is particularly true when standard compactness hypotheses are not present. The appropriate concepts and tools needed to study such kinds of problems are vital not only in theory but also within the development of numerical methods. For the latter, we need not only to prove that a sequence generated by a given algorithm is well defined, namely an existence
viii
result, but also to establish that the produced sequence remains bounded. One can seldom directly apply theorems of classical analysis to answer to such questions. The notions of asymptotic cones and associated asymptotic functions provide a natural and unifying framework to resolve these types of problems. These notions have been used mostly and traditionally in convex analysis, with many results scattered in the literature. Yet these concepts also have a prominent and independent role to play in both convex and nonconvex analysis. This book presents the material reflecting this last point with many parts, including new results and covering convex and nonconvex problems. In particular, our aim is to demonstrate not only the interplay between classical convex-analytic results and the asymptotic machinery, but also the wide potential of the latter in analyzing variational problems. We expect that this book will be useful to graduate students at an advanced level as well as to researchers and practitioners in the fields of optimization theory, nonlinear programming, and applied mathematical sciences. We decided to use a style with detailed and often transparent proofs. This might sometimes bore the more advanced reader, but should at least make the reading of the book easier and hopefully even enjoyable. The material is presented within the finite-dimensional setting. Our motivation for this choice was to eliminate the obvious complications that would have emerged within a more general topological setting and would have obscured the stream of the main ideas and results. For the more advanced reader, it is noteworthy to realize that most of the notions and properties developed here can be easily extended to reflexive Banach Spaces, assuming a supplementary condition with respect to weak convergence. The extension to more general arbitrary topological spaces is certainly not obvious, but the finite-dimensional setting is rich enough to motivate the interested reader toward the development of corresponding results needed in areas such as partial differential equations and probability analysis. Structure of the Book In Chapter 1 we recall the basic mathematical background: elementary convex analysis and set-valued maps. The results are presented without proofs. This material is classical and can be skipped by anyone who has had a standard course in convex analysis. None of this chapter’s results rely on any asymptotic notions. Chapter 2 is the heart of the book and gives the fundamental results on asymptotic cones and functions. The interplay between geometry and analysis is emphasized and will be followed consistently in the remaining chapters. Building on the concept of asymptotic cone of the epigraph of a function, the notion of asymptotic function emerges, and calculus at infinity can be developed. The role of asymptotic functions in formulating general optimization problems is described. Chapter 3 studies the existence of optimal solutions for general optimization problems and related stability results, and also demonstrates the power of the asymptotic
ix
results developed in Chapter 2. Standard results under coercivity and weak coercivity assumptions imply that the solution set is a nonempty compact set and the sum of a compact set with a linear space, respectively. Here we develop many new properties for the noncoercive and weakly coercive cases through the use of asymptotic sets to derive more general existence results with applications leading to some new theorems “`a la Helly” and for the convex feasibility problems. In Chapter 4 we study the subject of minimizing stationary sequences and error bounds. Both topics are central in the study of numerical methods. The concept of well-behaved asymptotic functions and the properties of such functions, which in turn is linked to the problems of error bounds associated with a given subset of a Euclidean space, are introduced. A general framework is developed around these two themes to characterize asymptotic optimality and error bounds for convex inequality systems. Duality theory plays a fundamental role in optimization and is developed in Chapter 5. The abstract perturbational scheme, valid for any optimization problem, is the starting point of the analysis. Under a minimal set of assumptions and thanks to asymptotic calculus, we derive key duality results, which are then applied to cover the classical Lagrange and Fenchel duality as well as minimax theorems, in a simple and unified way. Chapter 6 provides a self-contained introduction to maximal monotone maps and variational inequalities. Solving a convex optimization problem is reduced to solving a generalized equation associated with the subdifferential map. In many areas of applied mathematics, game theory, and equilibrium problems in economy, generalized equations arise and are described in terms of more general maps, in particular maximal monotone maps. The chapter covers the classical material together with some more recent results, streamlining the role of asymptotic functions. Each chapter ends with some bibliographical notes and references. We did not attempt to give a complete bibliography on the covered topics, which is rather large, and we apologize in advance for any omission in the cited references. Yet, we have tried to cite all the sources that have been used in this book as well as some significant original historical developments, together with more recent references in the field that should help to guide researchers for further reading. The book can be used as a complementary text to graduate courses in applied analysis and optimization theory. It can also serve as a text for a topics course at the graduate level, based, for example, on Chapters 2, 3, and 5, or as an introduction to variational inequality problems through Chapter 6, which is essentially self-contained. Alfred Auslender Marc Teboulle
Lyon, France Tel-Aviv, Israel
This page intentionally left blank
Contents
Preface 1 Convex Analysis and Set-Valued Maps: A Review 1.1 Convex Sets . . . . . . . . . . . . . . . . . . . . . . . 1.2 Convex Functions . . . . . . . . . . . . . . . . . . . . 1.3 Support Functions . . . . . . . . . . . . . . . . . . . 1.4 Set-Valued Maps . . . . . . . . . . . . . . . . . . . . 1.5 Notes and References . . . . . . . . . . . . . . . . . .
vii
. . . . .
. . . . .
. . . . .
. . . . .
1 1 9 17 20 23
2 Asymptotic Cones and Functions 2.1 Definitions of Asymptotic Cones . . . . . . . . . . . . . . . 2.2 Dual Characterization of Asymptotic Cones . . . . . . . . . 2.3 Closedness Criteria . . . . . . . . . . . . . . . . . . . . . . . 2.4 Continuous Convex Sets . . . . . . . . . . . . . . . . . . . . 2.5 Asymptotic Functions . . . . . . . . . . . . . . . . . . . . . 2.6 Differential Calculus at Infinity . . . . . . . . . . . . . . . . 2.7 Application I: Semidefinite Optimization . . . . . . . . . . . 2.8 Application II: Modeling and Smoothing Optimization Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9 Notes and References . . . . . . . . . . . . . . . . . . . . . .
25 25 31 32 44 47 60 66
3 Existence and Stability in Optimization Problems 3.1 Coercive Problems . . . . . . . . . . . . . . . . . . . . . . . 3.2 Weak Coercivity . . . . . . . . . . . . . . . . . . . . . . . .
81 81 85
72 78
xii
Contents
3.3 3.4 3.5 3.6 3.7 3.8 3.9
Asymptotically Level Stable Functions . . . . Existence of Optimal Solutions . . . . . . . . Stability for Constrained Problems . . . . . . Dual Operations and Subdifferential Calculus Additional Results in the Convex Case . . . . The Feasibility Problem. . . . . . . . . . . . . Notes and References . . . . . . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
93 96 100 107 112 116 118
4 Minimizing and Stationary Sequences 4.1 Optimality Conditions in Convex Minimization . . 4.2 Asymptotically Well-Behaved Functions . . . . . . 4.3 Error Bounds for Convex Inequality Systems . . . 4.4 Stationary Sequences in Constrained Minimization 4.5 Notes and References . . . . . . . . . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
119 119 124 133 140 143
5 Duality in Optimization Problems 5.1 Perturbational-Conjugate Duality . . 5.2 Fenchel Duality . . . . . . . . . . . . 5.3 Lagrangian Duality . . . . . . . . . . 5.4 Zero Duality Gap for Special Convex 5.5 Duality and Asymptotic Functions . 5.6 Lagrangians and Minimax Theory . 5.7 Duality and Stationary Sequences . . 5.8 Notes and References . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
145 145 154 157 162 166 170 178 181
6 Maximal Monotone Maps and Variational Inequalities 6.1 Maximal Monotone Maps . . . . . . . . . . . . . . . . . . 6.2 Minty Theorem . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Convex Functionals and Maximal Monotonicity . . . . . . 6.4 Domains and Ranges of Maximal Monotone Maps . . . . 6.5 Asymptotic Functionals of Maximal Monotone Maps . . . 6.6 Further Properties of Maximal Monotone Maps . . . . . . 6.7 Variational Inequalities Problems . . . . . . . . . . . . . . 6.8 Existence Results for Variational Inequalities . . . . . . . 6.9 Duality for Variational Inequalities . . . . . . . . . . . . . 6.10 Notes and References . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
183 183 186 191 195 197 206 212 214 221 230
. . . . . . . . . . . . . . . . . . Programs . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . .
. . . . . . . .
. . . . . . . .
References
233
Index of Notation
243
Index
245
1 Convex Analysis and Set-Valued Maps: A Review
Convexity plays a fundamental role in the analysis of optimization problems. In this very brief chapter we review some basic concepts on convex sets and convex functions. The material is presented quickly with no proofs and should be familiar to a reader who has had a standard convex analysis course. Some of the topics that might be less familiar and particularly relevant to this monograph will be given in greater details. Similarly, we review the basic definitions and properties of set-valued maps.
1.1 Convex Sets Throughout this book we will consider only finite-dimensional vector spaces. The n-dimensional real Euclidean vector space will be denoted by Rn . For ) ∈ Rn , y ∈ Rn , the inner product between x and y vectors x = (x1 , . . . , xn n is defined by x, y := i=1 xi yi . The norm arising from the inner product will be denoted by x := x, x1/2 . Definition 1.1.1 A subset C of Rn is convex if tx + (1 − t)y ∈ C, ∀x, y ∈ C, ∀t ∈ [0, 1]. Using the algebraic sum of two convex sets, αC1 + βC2 := {x | x = αc1 + βc2 , c1 ∈ C1 , c2 ∈ C2 }, ∀α, β ∈ R,
2
1. Convex Analysis and Set-Valued Maps: A Review
the above definition can also be written in the useful and compact notation C ⊂ Rn is convex ⇐⇒ tC + (1 − t)C ⊂ C, ∀t ∈ [0, 1]. The unit simplex of Rn is the convex set defined by n ti = 1, ti ≥ 0, i = 1, . . . , n . ∆n := t ∈ Rn | i=1
Hyperplanes, Convex and Affine Hulls A hyperplane is a set H := {x ∈ Rn |x, a = α}, where a ∈ Rn , a = 0 and α ∈ R. A hyperplane divides the space Rn into two closed half-spaces H + = {x ∈ Rn |x, a ≥ α} and H − = {x ∈ Rn |x, a ≤ α}. Clearly, a hyperplane and its associated half-spaces are convex sets. A set C is an affine manifold (or affine subspace) if x, y ∈ C ⇐⇒ tx + (1 − t)y ∈ C, ∀t ∈ R. The space Rn itself, points, lines, and hyperplanes in Rn are affine manifolds. An affine manifold is closed and convex. Proposition 1.1.1 For a nonempty set C ⊂ Rn the following properties are equivalent: (a) C is an affine manifold; (b) C = x+M = {y |y −x ∈ M }, where M is a subspace called the subspace parallel to M ; (c) C = {x |Ax = b} for some A ∈ Rm×n , b ∈ Rm . For a set C ⊂ Rn , the affine hull of C, denoted by aff C is the intersection of all affine manifolds containing C; the convex hull of C, denoted by conv C, is the intersection of all convex sets containing C. Proposition 1.1.2 For a set C ⊂ Rn , the following properties hold: (a) conv C is the set of all convex combinations of elements of C, i.e., m m ti xi |xi ∈ C, ti ≥ 0, ti = 1 . conv C = i=1
i=1
(b) aff C is itself a linear manifold, and conv C ⊂ aff C. (c) aff C = aff(conv C). Some elementary operations that preserve the convexity of a set are collected in the following proposition. Proposition 1.1.3 For any collection {Ci |i ∈ I} of convex sets Ci ⊂ Rni we have: (a) C1 × · · · × Cm is convex in Rn1 × . . . × Rnm . (b) ∩i∈I Ci is convex, here with ni = n for all i.
1.1 Convex Sets
3
m (c) The finite sum i=1 Ci is convex, with ni = n for all i. (d) The image of a convex set under a linear mapping is convex. A fundamental characterization of convex sets is provided by Carath´eodory’s theorem. Theorem 1.1.1 For any C ⊂ Rn , any element of conv C can be represented as a convex combination of no more than (n + 1) elements of C. Topological Properties of Convex Sets We now recall some basic topological concepts associated with convex sets. The closed unit ball in the n dimensional Euclidean space Rn will be denoted by: B = {x ∈ Rn | x ≤ 1}. The ball with center x0 and radius δ can thus be written as B(x0 , δ) := x0 + δB. Let C ⊂ Rn be a convex set. The interior and closure of C are also convex sets defined respectively by int C cl C
:= {x ∈ Rn | ∃ε > 0 such that x + εB ⊂ C}, (C + εB). := ε>0
The boundary of a set C ⊂ Rn is bd C := cl C \ int C. A point z belongs to the boundary of C if and only if for any ε > 0 the ball B(z, ε) contains a point of C as well as a point that is not in C. For a nonempty convex set of Rn with int C = ∅ we have aff C = Rn . The interior of a convex set C relative to its affine hull is called the relative interior of C and is defined by ri C := {x ∈ aff C | ∃ε > 0 such that (x + εB) ∩ aff C ⊂ C}. The relative interior clearly coincides with the interior when the affine hull is the whole space Rn . However, while for a nonempty convex set C we may have int C = ∅, in contrast, the relative interior ri C is not equal to ∅. This is an important property of the relative interior, which thus should be used in place of the interior when int C = ∅. The difference set cl C \ ri C is called the relative boundary of C and is denoted by rbd C. The most important properties of the relative interior, including the line segment principle, are summarized in the following results. Proposition 1.1.4 For C ⊂ Rn nonempty and convex, the set ri C is nonempty and has the same affine hull as C. Moreover, when x ∈ ri C, y ∈ cl C, one has tx + (1 − t)y ∈ ri C, ∀t ∈ (0, 1],
4
1. Convex Analysis and Set-Valued Maps: A Review
and thus ri C is convex. Furthermore, cl C = cl(ri C), ri C = ri(cl C). Proposition 1.1.5 Let C ⊂ Rn be nonempty and convex. Then x ∈ ri C if and only if for every y ∈ C there exists ε > 0 such that x + ε(y − x) ∈ C. A useful consequence of this characterization is the following. Corollary 1.1.1 Let C ⊂ Rn be convex. Then x ∈ int C if and only if for every d ∈ Rn there exists ε > 0 such that x + εd ∈ C. Proposition 1.1.6 For a nonempty convex set C ⊂ Rn one has (a) ri C ⊂ C ⊂ cl C, (b) cl cl C = cl C; ri(ri C)) = ri C, (c) A(cl C) ⊂ cl A(C) and ri A(C) = A(ri C), where A : Rn → Rm is a linear mapping. Moreover, let A−1 (S) := {x ∈ Rn | A(x) ∈ S} be the inverse image of A for any set S ⊂ Rn . Then, if A−1 (ri C) = ∅, one has ri(A−1 C) = A−1 (ri C), cl(A−1 C) = A−1 (cl C). Proposition 1.1.7 For a family of convex sets Ci ⊂ Rn indexed by i ∈ I and such that ∩i∈I ri Ci = ∅, one has cl Ci = clCi . i∈I
i∈I
Furthermore, if I is a finite index set, then Ci = riCi . ri i∈I
i∈I
As applications of these results one can obtain several other important rules involving relative interiors. The next one is frequently used. Proposition 1.1.8 For two convex sets C, D ⊂ Rn and for any scalars α, β ∈ R, one has ri(αC + βD) = α ri C + β ri D. Thus, in particular with α = −β = 1, we have 0 ∈ ri(C − D) ⇐⇒ ri C ∩ ri D = ∅. Proposition 1.1.9 Let C be a convex set in Rm+p . For each y ∈ Rm , let Cy := {z ∈ Rp | (y, z) ∈ C} and let D := {y | Cy = ∅}. Then (y, z) ∈ ri C ⇐⇒ y ∈ ri D and z ∈ ri Cy .
1.1 Convex Sets
5
Separation of Convex Sets Two fundamental results based on properties of closed convex sets and their interiors are the supporting hyperplane theorem and the separation principle. A hyperplane H := {x ∈ Rn | x, a = α}, with a ∈ Rn , a = 0, α ∈ R, is said to separate two sets C1 , C2 in Rn if C1 is contained in the closed half-space H + = {x ∈ Rn | x, a ≥ α}, while C2 is contained in the other. The separation is called proper if the hyperplane itself does not actually include both C1 and C2 . When the sets C1 , C2 lie in different half-spaces {x ∈ Rn | x, a ≥ α2 }, {x ∈ Rn | x, a ≤ α1 }, α2 > α1 , then one says that one has strong separation between C1 and C2 . Proposition 1.1.10 Let C1 and C2 be nonempty sets in Rn . Then there exists a hyperplane properly separating C1 and C2 if and only if there exists a nonzero vector a ∈ Rn such that inf{a, x | x ∈ C1 } ≥ sup{a, x | x ∈ C2 } and sup{a, x |x ∈ C1 } > inf{a, x | x ∈ C2 }. The next two theorems provide the main conditions that guarantee separation between two convex sets. Proposition 1.1.11 (Proper separation) Let C1 and C2 be nonempty convex sets in Rn . Then there exists a hyperplane that separates them properly if and only if ri C1 and ri C2 have no point in common. Proposition 1.1.12 (Strong separation) For two nonempty convex sets C1 and C2 in Rn such that C1 ∩C2 = ∅, with C1 closed and C2 compact, there exists a hyperplane that strongly separates them; i.e., there exist a vector 0 = a ∈ Rn and a scalar α ∈ R such that a, x1 ≤ α < a, x2 ∀x1 ∈ C1 , x2 ∈ C2 . Cones and Polyhedral Sets Cones are fundamental geometric objects associated with sets. They play a key role in several aspects of mathematics, and will be used extensively throughout this book. Here we recall some elementary properties, well known results, and examples. Further properties and examples will be given in Section 2.7, dealing with real symmetric matrices and related semidefinite optimization problems. Definition 1.1.2 A set K ⊂ Rn is called a cone if tx ∈ K for all x ∈ K and for all t ≥ 0.
6
1. Convex Analysis and Set-Valued Maps: A Review
Examples of convex cones include linear subspaces of Rn and the nonnegative orthant Rn+ := {x |xi ≥ 0, i = 1, . . . , n}. Other cones playing an important role in convex optimization problems are the cone of symmetric real positive semidefinite matrices of order n and the cone of Lorentz. More details are given in Chapter 2, Section 2.7. Proposition 1.1.13 For K ⊂ Rn the following are equivalent: (a) K is a convex cone. (b) K is a cone such that K + K ⊂ K. Definition 1.1.3 A cone K ⊂ Rn is pointed if the equation x1 +· · ·+xp = 0 has no solution with xi ∈ K unless xi = 0 for all i. Pointedness of convex cones can be checked via the following test. Let K ⊂ Rn be a convex cone. Then K is pointed ⇐⇒ K ∩ (−K) = {0}. Given a cone K ⊂ Rn , the polar of K is the cone defined by K ∗ := {y ∈ Rn | y, x ≤ 0, ∀x ∈ K}. Orthogonality of subspaces is a special case of polarity of cones. If M is a subspace of Rn , one has M ∗ = M ⊥ = {y ∈ Rn | y, x = 0, ∀x ∈ M }. The bipolar is the cone K ∗∗ := (K ∗ )∗ . Proposition 1.1.14 For a cone K ⊂ Rn , the polar cone is closed and convex, and K ∗∗ = cl(conv K). If K is also closed and convex, one then has K ∗∗ = K. Proposition 1.1.15 Let K ⊂ Rn be a convex cone. Then int K = ∅ ⇐⇒ K ∗ is pointed. In fact,
x ∈ int K ⇐⇒ x, y < 0, ∀ 0 = y ∈ K ∗ .
An important and useful object in variational problems is the normal cone to a given convex set. Definition 1.1.4 Let C ⊂ Rn be a nonempty convex set. The normal cone x) to C at x ¯ ∈ C is defined by NC (¯ NC (¯ x)
:= {v ∈ Rn | v, x − x ¯ ≤ 0 ∀x ∈ C} =
{v ∈ Rn | v, x ¯ = sup{v, x | x ∈ C}}.
1.1 Convex Sets
7
For all x ∈ C one has NC (x) := ∅ and x ∈ int C ⇐⇒ NC (x) = {0}. x) is pointed if and only if int C = ∅. Moreover, NC (¯ Furthermore, for a closed set C ⊂ Rn , a boundary point of the set C can be characterized through the normal cone of C. In fact, one has x ∈ bd C ⇐⇒ ∃v = 0, v ∈ NC (x). Another geometrical object associated with a convex set C ⊂ Rn , and closely related to the normal cone of C at x ¯, is the tangent cone of C at x ¯, given by x) = cl{d ∈ Rn | ∃t > 0 with x ¯ + td ∈ C}. TC (¯ x) and TC (¯ x) are polar to each other. For any x ¯ ∈ C, the cones NC (¯ Some useful operations on polar cones are summarized below. Proposition 1.1.16 For cones Ki of Rn , i = 1, 2, one has: (a) K1 ⊂ K2 =⇒ K2∗ ⊂ K1∗ and K1∗∗ ⊂ K2∗∗ . (b) K = K1 + K2 =⇒ K ∗ = K1∗ ∩ K2∗ . (c) K = K1 ∩ K2 with Ki closed =⇒ K ∗ = cl(K1∗ + K2∗ ). The closure operation can be removed if 0 ∈ int(K1 − K2 ). (d) For a family of cones {Ki | i ∈ I} in Rn , K = ∪i∈I Ki =⇒ K ∗ = ∩i∈I Ki∗ . (e) Let A : Rn → Rm be a linear mapping and K a closed convex cone of Rm . Then {x | Ax ∈ K}∗ = cl{AT y | y ∈ K}. The closure operation can be removed when 0 ∈ int(K − rge A), where rge A := {Ax |x ∈ Rn }. A cone K ⊂ Rn is said to be finitely generated if it can be written as p n tp ap | ai ∈ R , ti ≥ 0, i = 1, . . . , p . K= i=1
For a given nonempty set C ⊂ Rn , the smallest cone containing the set C is called the positive hull (or conical hull) of C. It is the smallest cone containing the set C and is given by pos C := {λx | x ∈ C, λ > 0} ∪ {0}. The positive hull, pos C, is also said to be the cone generated by C. Clearly C is itself a cone if and only if pos C = C. A set P ⊂ Rn is called polyhedral if it has the form P = {x ∈ Rn | ai , x ≤ bi , i = 1, . . . , p}, where ai ∈ Rn , bi ∈ R, i = 1, . . . , p. When bi = 0, ∀i, then P is called a polyhedral cone. The next result is a useful separation theorem involving the mixture of a polyhedral set with a convex set.
8
1. Convex Analysis and Set-Valued Maps: A Review
Proposition 1.1.17 Let C1 and C2 be nonempty convex sets in Rn , with C1 being a polyhedral set. Then C1 ∩ ri C2 = ∅ if and only there exists a hyperplane separating C1 and C2 properly and not containing C2 . Theorem 1.1.2 (Minkowski–Weyl Theorem) A cone K is polyhedral if and only if it is finitely generated. An important implication of this theorem is the following decomposition formula expressing a polyhedral set. Proposition 1.1.18 A set P is polyhedral if and only if there exist a nonempty and finite collection of points {a1 , . . . , ap } and a finitely generated cone K such that p P = x |x = y + ti ai , y ∈ K, t ∈ ∆p , i=1
where ∆p is the unit simplex of Rp . This kind of representation by convex hulls of minimal sets can be extended to general closed convex sets but requires the notion of extreme point and extreme ray. For any two points x and y, the closed (open) line segment joining x and y defined for any t ∈ [0, 1] (t ∈ (0, 1)) by tx + (1 − t)y is denoted by [x, y] (]x, y[). Let C ⊂ Rn be nonempty and convex. A nonempty subset F ⊂ C is said to be a face of C if ∀x, y ∈ C with F ∩ ]x, y[ = ∅ =⇒ [x, y] ⊂ F. A point z ∈ C is an extreme point of C if {z} is a face, namely, z cannot be written in the form z = λx + (1 − λ)y with x, y ∈ C, x = y, and λ ∈ (0, 1).The set of extreme points of C is denoted by ext C. An extreme ray of C is the direction of a half-line that is a face of C. We denote by extray C the union of extreme rays of C. Theorem 1.1.3 (Krein–Milman) Let C be a nonempty closed convex set containing no lines. Then C = conv(ext C ∪ extray C). When C is a compact set, one has ext C = ∅, and the Krein–Milman theorem implies the following, Theorem 1.1.4 (Minkowski) Let C be a nonempty compact convex set in Rn . Then C = conv ext C.
1.2 Convex Functions
9
1.2 Convex Functions We denote by R := [−∞, +∞] the whole extended real line. It is most convenient, in particular in the context of optimization problems, to work with extended real-valued functions, i.e., functions that take values in R ∪ {+∞} = (−∞, +∞], instead of just finite-valued functions, i.e., those taking values in R = (−∞, +∞). Rules of arithmetic are thus extended to include ∞ + ∞ = ∞, α · ∞ = ∞, ∀α ≥ 0, inf ∅ = ∞, sup ∅ = −∞. Let f : Rn → R. The effective domain of f is the set dom f := {x ∈ Rn |f (x) < +∞}. A function is called proper if f (x) < ∞ for at least one x ∈ Rn and f (x) > −∞, ∀x ∈ Rn otherwise, the function is called improper. Two important and useful geometrical objects associated with a function f are the epigraph and level set of f , defined, respectively by epi f lev(f, α)
:= :=
{(x, α) ∈ Rn × R |α ≥ f (x)}, {x ∈ Rn |f (x) ≤ α}.
The epigraph is thus a subset of Rn+1 that consists of all points of Rn+1 lying on or above the graph of f . From the above definitions one has (x, α) ∈ epi f ⇐⇒ x ∈ lev(f, α). For f : Rn → R, we write inf f arg min f
:= inf{f (x) | x ∈ Rn }, = arg min{f (x) | x ∈ Rn } := {x ∈ Rn | f (x) = inf f }.
An optimization problem can thus be expressed equivalently in terms of its epigraph and level set as inf f = inf{α | (x, α) ∈ epi f }, while the set of optimal solutions is arg min f = lev(f, inf f ). Some useful operations involving epigraphs and level sets are collected in the next proposition.
10
1. Convex Analysis and Set-Valued Maps: A Review
Proposition 1.2.1 Let f : R → R and let L : (x, α) → x be the projection maps mapping from Rn+1 to Rn . The following hold: (a) dom f = L(epi f ). (b) lev(f, α) = L(epi f ∩ {(x, α) |x ∈ Rn }). (c) For a collection of extended real-valued functions {fi | i ∈ I}, with I an arbitrary index sets one has epi fi , epi sup fi = epi fi . epi inf fi = i∈I
i∈I
i∈I
i∈I
Recall that lim inf f (x) := sup x→y
inf
r>0 x∈B(y,r)
f (x),
and lower limits are characterized via lim inf f (x) = min{α ∈ R | ∃xn → y with f (xn ) → α}. x→y
Note that one always has lim inf x→y f (x) ≤ f (y). Definition 1.2.1 The function f : Rn → R is lower semicontinuous (lsc) at x if f (x) = lim inf f (y), y→x
and lower semicontinuous on R if this holds for every x ∈ Rn . n
Theorem 1.2.1 Let f : Rn → R. The following statements are equivalent: (a) f is lsc at x. (b) lim inf n→∞ f (xn ) ≥ f (x), ∀xn → x. (c) For each α such that f (x) > α ∃δ > 0 such that f (y) > α, ∀y ∈ B(x, δ). Lower semicontinuity on Rn of a function can be characterized through its level set and epigraph. Theorem 1.2.2 Let f : Rn → R. The following statements are equivalent: (a) f is lsc on Rn . (b) The epigraph epi f is closed on Rn × R. (c) The level sets lev(f, α) are closed in Rn . When f is not lower semicontinuous, then its epigraph is not closed, but cl(epi f ) is closed and leads to the lower closure of f , denoted by cl f , and defined such that epi(cl f ) = cl(epi f ). In terms of f one then has (cl f )(x) := lim inf f (y), y→x
and it holds that cl f ≤ f . This function is the greatest of all the lsc functions g such that g ≤ f .
1.2 Convex Functions
11
Definition 1.2.2 A function f : Rn → R with f ≡ ∞ is called convex if epi f is a nonempty convex set. As a consequence, since the effective domain dom f is the projection of epi f , it follows that dom f is convex when f is convex. Alternatively, for an extended real-valued function f : Rn → R ∪ {+∞}, f is convex if and only if f (tx + (1 − t)y) ≤ tf (x) + (1 − t)f (y), ∀x, y ∈ Rn , ∀t ∈ (0, 1). The function is called strictly convex if the above inequality is strict for all x, y ∈ Rn with x = y and t ∈ (0, 1). A function f is called concave whenever −f is convex. Convexity of f can also be defined through Jensen’s inequality, namely, f is convex if and only if
p p ti xi ≤ ti f (xi ), ∀xi ∈ Rn , i = 1, . . . , p, t ∈ ∆p . f i=1
i=1
If f is convex, then lev(f, α) is a convex set for all α ∈ R (the converse statement does not hold). The indicator function of a set C of Rn , denoted by δC , is defined by 0 if x ∈ C, δC (x) = +∞ if x ∈ C. If C is a nonempty closed convex subset of Rn , then δC is a proper, lsc, and convex function. The indicator function allows for recovering the definition of a convex realvalued function defined on a convex set C. Indeed, given g a real-valued function on a convex set C, then with f := g + δC , Definition 1.2.2 becomes g(tx + (1 − t)y) ≤ tg(x) + (1 − t)g(y), ∀x, y ∈ C, ∀t ∈ [0, 1]. The following inf-projection operation of a convex function is particularly useful. Proposition 1.2.2 Let Φ : Rn ×Rm → R∪{+∞} be convex. Then, ϕ(u) = inf x Φ(x, u) is convex on Rm , and the set S(u) = arg minx Φ(x, u) is convex for each u. For a proper convex function, closedness is the same as lower semicontinuty. Furthermore, one has the following result. Proposition 1.2.3 A convex function f : Rn → [−∞, +∞] is lsc at a point x where it is finite if and only if f (x) = (cl f )(x), and in this case f is proper. Proposition 1.2.4 Let f : Rn → R ∪ {+∞} be an improper convex function. Then f (x) = −∞ for every x ∈ ri(dom f ).
12
1. Convex Analysis and Set-Valued Maps: A Review
Proposition 1.2.5 Let f : Rn → R ∪ {+∞} be a proper convex function. Then cl f is a lsc proper convex function, and dom(cl f ) and dom f have the same closure and relative interior, as well as the same dimension. Furthermore, f = cl f on ri dom f . It is thus important to remark that a convex function that is lsc at some point and takes the value −∞ cannot take values other than +∞ and −∞. Proposition 1.2.6 Let f : Rn → R ∪ {+∞} be a proper convex function. Then (cl f )(x) = lim+ f (x + t(y − x)), ∀x ∈ Rn , ∀y ∈ ri(dom f ). t→0
If in addition f is assumed lsc, then f (x) = lim+ f (x + t(y − x)), ∀x, ∀y ∈ dom f. t→0
Proposition 1.2.7 Let fi : Rn → R ∪ {+∞}, i = 1, . . . , p, be a collection of proper convex functions. If ∩pi=1 ri dom fi = ∅, then cl(f1 + · · · + fp ) = cl f1 + · · · + cl fp . Proposition 1.2.8 Let f : Rn → R ∪ {+∞} be a proper convex function with level sets lev(f, α). Then for any α ∈ R with α > inf f , one has ri(lev(f, α)) = {x ∈ ri(dom f ) | f (x) < α}. Theorem 1.2.3 Let f : Rn → R ∪ {+∞} be a convex function. Then f is relatively continuous on any relatively open convex set C contained in dom f , in particular on ri(dom f ). Recall that a function f : Rn → R∪{+∞} is called Lipschitz with constant L on a subset C of dom f if there exists L ≥ 0 such that |f (x) − f (y)| ≤ Lx − y, ∀x, y ∈ C, and f is called locally Lipschitz on C if for every c ∈ C there exists a neighborhood V of c such that f is Lipschitz on V ∩ C. Theorem 1.2.4 Let f : Rn → R ∪ {+∞} be a proper convex function and let C be any compact subset of ri(dom f ).Then f is Lipschitz relative to C. An important and useful class of convex functions is the class of polyhedral convex functions. Definition 1.2.3 A polyhedral convex function is a convex function whose epigraph is polyhedral, which holds, if and only if it is finitely generated. Such a function, if proper, is lsc.
1.2 Convex Functions
13
For a nonconvex function f there is a natural procedure to convexify it via the use of the convex hull of the epigraph of f . Definition 1.2.4 For a function f : Rn → R ∪ {+∞}, the convex hull of f is denoted by conv f and is defined for each x ∈ Rn by conv f (x)
:= =
inf {r ∈ R | (x, r) ∈ conv(epi f )} n+1 n+1 ti f (xi ) | ti xi = x, xi ∈ dom f, t ∈ ∆n+1 . inf i=1
i=1
Equivalently, it is the greatest convex function majorized by f . Similarly, for a collection of functions {fi |i ∈ I} on Rn , where I is an arbitrary index set, the convex hull is denoted by conv{fi | i ∈ I} and is the greatest convex function h such that h(x) ≤ fi (x) for every x ∈ Rn and every i ∈ I. Whenever the index set I := {1, . . . , p} is finite, the convex hull h of the finite family fi is then given by h(x)
:= =
inf {r ∈ R | (x, r) ∈ conv(∪pi=1 epi fi )} p p ti fi (xi ) | ti xi = x, xi ∈ dom fi , t ∈ ∆p . inf i=1
i=1
Proposition 1.2.9 Let fi : Rn → R ∪ {+∞}, i ∈ I, be an arbitrary collection of proper convex functions. Then for any x ∈ Rn , conv{fi | i ∈ I}(x) = inf ti fi (x) | ti xi = x , i∈I
i∈I
where the infimum is taken over all expressions of x as a convex combination in which at most (n + 1) of the ti are positive. Legendre–Fenchel Transform and Conjugate Functions Duality plays a fundamental role in optimization problems, convex and nonconvex ones. A key player in any duality framework is the Legendre– Fenchel transform, also called the conjugate of a given function. Definition 1.2.5 For any function f : Rn → R ∪ {+∞}, the function f ∗ : Rn → R ∪ {+∞} defined by f ∗ (y) := sup{x, y − f (x)} x
is called the conjugate to f . The biconjugate of f is defined by f ∗∗ (x) := sup{x, y − f ∗ (y)}. y
14
1. Convex Analysis and Set-Valued Maps: A Review
Whenever conv f is proper, one always has that both f ∗ and f ∗∗ are proper, lsc, and convex, and the following relations hold: f ∗∗ = cl(conv f ) and f ∗∗ ≤ cl f.
Theorem 1.2.5 (Fenchel–Moreau) Let f : Rn → R ∪ {+∞} be convex. The conjugate function f ∗ is proper, lsc, and convex if and only if f is proper. Moreover, (cl f )∗ = f ∗ and f ∗∗ = cl f . From the definition of the conjugate function, we immediately obtain Fenchel’s inequality: x, y ≤ f (x) + f ∗ (y),
∀x ∈ dom f, y ∈ dom f ∗ .
Definition 1.2.6 Let f, g : Rn → R ∪ {+∞} be proper functions. The infimal convolution of the function f with g is the function h defined by h(x) := (f 2g)(x) = inf{f (x1 ) + g(x2 ) | x1 + x2 = x}, ∀x ∈ Rn . The above definition can be extended as well for a finite collection {fi | i = 1, . . . , p} of proper functions, i.e., (f1 2 · · · 2fp )(x) := inf{f1 (x1 ) + · · · + fp (xp ) | x1 + · · · xp = x}. The next proposition gives some important results relating conjugates and their infimal convolutions. Proposition 1.2.10 Let fi : Rn → R∪{+∞}, i = 1, . . . , p, be a collection of proper and convex functions. Then: (a) (f1 2 · · · 2fp )∗ = f1∗ + · · · + fp∗ , (b) (cl f1 + · · · + cl fp )∗ = cl(f1∗ 2 · · · 2fp∗ ). Polyhedrality is preserved under the conjugacy operation. Proposition 1.2.11 Let f, g be proper polyhedral convex. Then: (a) f + g is polyhedral. (c) The conjugate of f is polyhedral. Proposition 1.2.12 Let fi : Rn → R ∪ {+∞}, i ∈ I, be an arbitrary collection of functions with conjugates fi∗ . Then: (a) (inf{fi |i ∈ I})∗ = sup{fi∗ | i ∈ I}. (b) conv{fi | i ∈ I}∗ = sup{fi∗ | i ∈ I}, and sup{clfi | i ∈ I})∗ = cl(conv{fi∗ | i ∈ I}), whenever the functions fi are proper convex.
1.2 Convex Functions
15
Differentiability and Subdifferentiability of Convex Functions Under differentiability assumptions, one can check the convexity of a function via the following useful tests. For a function f defined on Rn and sufficiently smooth, the gradient of f at x, denoted by ∇f (x), is a vector in Rn whose i-th component is the partial derivative of f with respect to xi , while the Hessian of f at x, when f is twice differentiable, is a symmetric n × n matrix, whose (i, j) element is the second-order partial derivative ∂ 2 f (x)/∂xi ∂xj , and is denoted by ∇2 f (x). Theorem 1.2.6 Let f be a differentiable function on an open convex subset S of Rn . Each of the following conditions is necessary and sufficient for f to be convex on S: (a) f (x) − f (y) ≥ x − y, ∇f (y), ∀x, y ∈ S. (b) ∇f (x) − ∇f (y), x − y ≥ 0, ∀x, y ∈ S. (c) ∇2 f (x) is positive semidefinite for all x ∈ S whenever f is twice differentiable on S. For nondifferentiable functions, convexity plays a fundamental role that allows one to define the concept of subgradients and subdifferential maps. A subgradient of an extended function of Rn at a point x with f (x) finite is any vector g ∈ Rn satisfying f (y) ≥ f (x) + g, y − x, ∀y. The set of all subgradients of f at x ∈ dom f is called the subdifferential of the function f at x and is denoted by ∂f (x); that is, ∂f (x) := {g | f (y) ≥ f (x) + g, y − x, ∀y ∈ Rn }. When x ∈ dom f we set ∂f (x) = ∅. The subdifferential of f at x can be viewed as a set-valued map ∂f that assigns to each point x ∈ Rn a certain subset ∂f (x) of Rn , see Section 1.4 below for more on set-valued maps. In general, ∂f (x) may be empty. When ∂f (x) = ∅, the function f is then called subdifferentiable at x. A remarkable and evident property is that ∂f (x) is a closed convex set. Indeed, from its definition, the subdifferential of f at x is nothing but an infinite intersection of closed half-spaces. Let f : Rn → R ∪ {+∞} be a convex function, and let x be any point where f is finite and d ∈ Rn . Then the function f (x + τ d) − f (x) τ is nondecreasing in τ > 0. As a consequence, the limit τ→
f (x; d) := lim
τ →0+
f (x + τ d) − f (x) τ
exists (finite or infinite) for all d ∈ Rn and is called the directional derivative of f at x.
16
1. Convex Analysis and Set-Valued Maps: A Review
Proposition 1.2.13 Let f : Rn → R∪{+∞} be convex and let x ∈ dom f . Then g is a subgradient of f at x if and only if f (x; d) ≥ g, d, ∀d ∈ Rn . Proposition 1.2.14 Let f : Rn → R ∪ {+∞} be a proper convex function and let x ∈ dom f be such that there exists a nonzero vector of the form d := y − x with y ∈ ri(dom f ) and such that f (x; d) > −∞. Then ∂f (x) = ∅. In particular, for any x ∈ ri(dom f ) one has ∂f (x) = ∅. Given subgradients of f one might identify the directional derivative. In fact, the directional derivative and the subdifferential of f are related by the following fundamental max-formula. Proposition 1.2.15 Let f : Rn → R ∪ {+∞} be a proper convex function. Then for any x ∈ ri(dom f ) and any direction d ∈ Rn one has d → f (x; d) is lsc and proper as a function of d, and f (x; d) = max{g, d |g ∈ ∂f (x)}. We have already observed that ∂f (x) is a closed convex set. The next result provides a simple criterion to determine when this set is nonempty and bounded, and hence compact. Proposition 1.2.16 Let f : Rn → R ∪ {+∞} be a proper convex function. Then ∂f (x) is nonempty and bounded if and only if x ∈ int dom f . Let dom ∂f := {x | ∂f (x) = ∅} denote the domain of ∂f . This set is not necessarily convex. However, from the above results it follows that for any proper convex function f : Rn → R ∪ {+∞}, one has the relation ri(dom f ) ⊂ dom ∂f ⊂ dom f.
(1.1)
A proper convex function f is differentiable at x ∈ dom f if and only if the subdifferential reduces to a singleton, which is the gradient of f , i.e., one has ∂f (x) = {∇f (x)}. In that case, one has f (x; d) = ∇f (x), d, ∀d ∈ Rn . There exists an interesting and fundamental interplay between conjugate functions and subdifferentials of convex functions that is quite at the heart of many significant results in convex optimization problems and that will be used all along in this monograph. We begin with a simple but most significant result played by subgradients in convex optimization, which follows immediately from the definition of the subdifferential. Proposition 1.2.17 Let f : Rn → R ∪ {+∞} be a proper convex function. A point x ∈ Rn is a global minimizer of f if and only if 0 ∈ ∂f (x). Using the definitions of the subdifferential and of the conjugate function, it is possible to characterize subdifferentiabililty in the following useful and important way.
1.3 Support Functions
17
Proposition 1.2.18 Let f : Rn → R ∪ {+∞} be a proper function. Then f (x) + f ∗ (y) ≥ x, y, ∀x, y, and equality holds if and only if y ∈ ∂f (x), or equivalently, if and only if x ∈ ∂f ∗ (y) whenever f is in addition lsc and convex. Proposition 1.2.19 Let f : Rn → R ∪ {+∞} and g : Rn → R ∪ {+∞} be proper convex functions on Rn . Then for any x ∈ Rn , ∂(f + g)(x) ⊃ ∂f (x) + ∂g(x). Proposition 1.2.20 Let f : Rn → R, and x with f (x) finite. Then: (a) ∂f (x) = ∅ =⇒ f (x) = f ∗∗ (x). (b)f (x) = f ∗∗ (x) =⇒ ∂f (x) = ∂f ∗∗ (x). (c) y ∈ ∂f (x) =⇒ x ∈ ∂f ∗ (y). Let C ⊂ Rn be a convex set with normal cone NC as given in Definition 1.1.4. Then, using the subgradient inequality, one immediately sees that the normal cone reduces to the subdifferential of the indicator function of the set C; i.e., (1.2) NC (x) = ∂δC (x), ∀x ∈ Rn . A useful representation for the normal cone of a polyhedral set is given by the following proposition. Proposition 1.2.21 Let (ai , αi ) ∈ Rn × R for i = 1, . . . , p, with ai = 0, n | ai , x ≤ αi }, C = ∩pi=1 Ci . Then for any x ∈ C one and Ci = {x ∈ R p has NC (x) = i=1 NCi (x), with R+ ai if ai , x = αi , NCi (x) = {0} otherwise. The normal cone of the level set of a convex function can be written explicitly: Proposition 1.2.22 Let f : Rn → R ∪ {+∞} be a proper convex function and let x ∈ ri(dom f ) be such that f (x) is not the minimum. Then for each α ∈ R one has R+ ∂f (x) if f (x) = α, Nlev(f,α) (x) = {0} if f (x) < α.
1.3 Support Functions Support functions play an important role in convex analysis and optimization. In fact they allow a characterization of closed convex sets and of the
18
1. Convex Analysis and Set-Valued Maps: A Review
position of a point relative to a set with powerful analytical tools. This permits one to translate geometrical questions on convex sets into questions in terms of convex functions, which often facilitate the way of handling these sets. Definition 1.3.1 Given a nonempty set C of Rn , the function σC : Rn → R ∪ {+∞} defined by σC (d) := sup{x, d | x ∈ C} is called the support function of C. The domain of σC defined by dom σC := {x ∈ Rn | σC (x) < +∞} is called the barrier cone of C and is denoted by b(C). One thus has σC (d) ≤ α ⇐⇒ C ⊂ {x | x, d ≤ α}, characterizing the closed half-space that contains C. Definition 1.3.2 A function π : Rn → R ∪ {+∞} is positively homogeneous if 0 ∈ dom π and π(λx) = λπ(x) for all x and all λ > 0. It is sublinear if in addition, π(x + y) ≤ π(x) + π(y), ∀x, y ∈ Rn . Examples of positively homogeneous functions that are also sublinear include norms and linear functions. The indicator function δC of a set C is positively homogeneous if and only if C is a cone. Proposition 1.3.1 A function π : Rn → R ∪ {+∞} is convex and positively homogeneous if and only if epi π is a nonempty convex cone in Rn ×R. From the definition of a support function we immediately obtain the following basic properties. Proposition 1.3.2 For any nonempty set C of Rn , the support function σC satisfies the following: (a) σC is an lsc convex positively homogeneous function. (b) The barrier cone dom σC is a convex cone (not necessarily closed). (c) σC = σcl C = σconv C = σcl conv C . (d) σC < +∞ if and only if C is bounded. (e) If C is convex, then σC = σri C . Example 1.3.1 (Examples of particular support functions) (i) σRn = δ{0} , σ∅ = −∞, σ{0} ≡ 0. (ii) σconv{a1 ,...,ap } (d) = max1≤i≤p d, ai .
m (iii) max1≤j≤m dj = sup d, x | x ≥ 0, j=1 xj = 1 . (iv) For any cone K ⊂ Rn , σK = δK ∗ .
1.3 Support Functions
19
The support functions possess a structure allowing the development of calculus rules. We list below some important formulas for support functions and barrier cones associated with arbitrary (not necessarily convex) nonempty sets of Rn . Proposition 1.3.3 Let C, D be nonempty sets of Rn . (a) C ⊂ D =⇒ dom σD ⊂ dom σC and σC (d) ≤ σD (d), ∀d ∈ Rn . (b) σC+D (d) = σC (d) + σD (d), and dom σC+D = dom σC ∩ dom σD . If D is a cone, then the formula above reduces to σC (d) if d ∈ D∗ , σC+D (d) = +∞ otherwise, and dom σC+D = dom σC ∩ D∗ . (c) For any point y ∈ Rn , σC+y (d) = σC (d) + d, y. (d) For a family of nonempty sets {C}i∈I with C := cl conv(∪i∈I Ci ) one has σC (d) = sup σCi (d) and dom C = ∩i∈I dom σCi . i∈I
(e) Let A : R → R be a linear mapping. For any nonempty set C of Rn we have σA(C) (d) = σcl A(C) (d) = σC (AT d), ∀d ∈ Rn . n
m
(f ) For any two closed convex sets C1 , C2 in Rn such that C1 ∩ C2 = ∅ one has σC1 ∩C2 (d) = cl(σC1 2σC2 )(d), ∀d. Turning to nonempty closed convex sets, we now state the fundamental correspondence between a closed convex set and its support function, namely the analytic version of the so called Hahn–Banach theorem. Theorem 1.3.1 Let C ⊂ Rn . The closed convex hull of C is given by cl conv C = {x ∈ Rn | x, d ≤ σC (d), ∀d ∈ Rn }. As a result of this theorem a closed convex set is completely determined by its support function. Key properties can then be usefully expressed via support functions. Theorem 1.3.2 Let C be a nonempty convex set in Rn and define B(d) := σC (d) + σC (−d), d ∈ Rn . Then B(d) ≥ 0 and: (a) x ∈ aff C if and only if x, d = σC (d), ∀d with B(d) = 0. (b) x ∈ ri C if and only if x, d < σC (d), ∀d with B(d) > 0. (c) x ∈ int C if and only if x, d < σC (d), ∀d = 0. (d) x ∈ cl C if and only if x, d ≤ σC (d), ∀d.
20
1. Convex Analysis and Set-Valued Maps: A Review
Corollary 1.3.1 For any convex sets C, D in Rn one has cl C ⊂ cl D if and only if σC ≤ σD . By its definition, the support function of a set C ⊂ Rn is in fact the conjugate function of the corresponding indicator function of that set, σC = ∗ . Therefore, one has the relation δC ∗ ∗ (·) = (δC (·))∗ = cl(δC (·)) = δcl C (·), σC
and when C is a closed convex set, this reduces to ∗ ∗∗ (·) = δC (·) = δC (·). σC
1.4 Set-Valued Maps Let X and Y be two sets of Rn . A set-valued map S from X to Y is a map that assigns to each point x ∈ X a subset S(x) of Y . Actually, it is convenient to characterize a set-valued map by its graph. The graph of S is the subset of X × Y defined by gph S := {(x, y) | y ∈ S(x)}. Conversely, every nonempty subset G of X ×Y is the graph of the set-valued map S uniquely determined by S(x) = {y ∈ Y | (x, y) ∈ G}. We use the notation S : X ⇒ Y to quantify the multivalued aspect of S whenever S(x) is a set containing more than one element. When S is single-valued, we mean that S(x) = {y} or simply S(x) = y, and in that case we use the notation S : X → Y . We shall say that a set-valued map S is convex-valued or closed-valued, etc., when S(x) is convex or closed, etc. The domain and range of S : X ⇒ Y are defined by the sets dom S
:=
{x ∈ X | S(x) = ∅},
rge S
:=
{y | ∃x such that y ∈ S(x)} =
S(x).
x∈X
Thus, the domain and range of S are respectively the images of gph S under the projections (x, y) → x and (x, y) → y. The inverse of any set-valued map always exists and is defined by S −1 (y) := {x ∈ X | y ∈ S(x)},
1.4 Set-Valued Maps
21
or equivalently, x ∈ S −1 (y) ⇐⇒ (x, y) ∈ gph S. The image of a set C is defined by S(C) =
S(x) = {u | S −1 (u) ∩ C = ∅}.
x∈C
One has the following relations: dom S −1 = rge S, rge S −1 = dom S, (S −1 )−1 = S. In this book we will work with X, Y that are subsets of the finite-dimensional spaces Rn and Rm . An easy way to extend a set-valued map to the whole space Rn is formally achieved by defining the map S(x) if x ∈ X, ˆ S(x) = ∅ if x ∈ X. We will consider here set-valued maps S : Rn ⇒ Rm with the understanding that S(x) = ∅ for any x ∈ dom S. A single-valued map F : X → Rm given on X ⊂ Rn can be treated in terms of the set-valued map S : Rn → Rm by setting {F (x)} if x ∈ X, S(x) = ∅ if x ∈ X, so that dom S = X. Basic operations with set-valued maps such as Minkowski addition and scalar multiplication and composition can be defined to generate new maps as follows: (S + T )(x) := S(x) + T (x), (λS)(x) := λS(x), λ ∈ R, and the composition of S : Rn → Rm with T : Rm ⇒ Rp by (T ◦ S)(x) := T (S(x)) =
T (y) = {z | S(x) ∩ T −1 (z) = ∅},
y∈S(x)
to get the mapping T ◦ S : Rn → Rp . Semicontinuity and Continuity of Set-Valued Maps Definition 1.4.1 A set-valued map S : Rn ⇒ Rm is upper semicontinuous (usc) at x ¯ if for each open set N containing S(¯ x) there exists a neighborhood U (¯ x) such that x ∈ U (¯ x) =⇒ S(x) ⊂ N.
22
1. Convex Analysis and Set-Valued Maps: A Review
Alternatively, upper semicontinuity of the set-valued map S at x ¯ can be expressed as x)), ∀x ∈ Bδ (¯ x), ∀ε > 0, ∃δ > 0 such that S(x) ⊂ Bε (S(¯ where we define for any set C, Bε (C) := {z | dist(z, C) < ε}. Clearly, when S is a single-valued map on Rn , i.e., a function, then for any x ∈ Rn , S(x) reduces to a singleton, and thus the above formulation coincides with the definition of continuous functions. The latter formulation also leads to the translation of upper semicontinuity in terms of infinite ¯. Then, sequences. Let {xk } ∈ dom S be a convergent sequence with limit x the set-valued map is upper semicontinuous at x ¯ if x)) = 0, lim dist(yk , S(¯
k→∞
∀ yk ∈ S(xk ).
Definition 1.4.2 A set-valued map S : Rn ⇒ Rm is lower semicontinuous (lsc) at x ¯ if for each open set N with N ∩ S(¯ x) = ∅, there exists a neighborhood U (¯ x) such that x ∈ U (¯ x) =⇒ S(x) ∩ N = ∅. S is called continuous at x ¯ if it is both lsc and usc at x ¯. We say that S is ¯ ∈ Rn , and continuous in lsc (usc) in Rn if it is lsc (usc) at every point x Rn if S is both lsc and usc in Rn . Proposition 1.4.1 A set-valued map S : Rn ⇒ Rm is lsc at x ¯ if and only ¯, and any y¯ ∈ S(¯ x), there if for any sequence {xk } ∈ Rn converging to x exist k0 ∈ N and a sequence {yk } ∈ S(xk ) for k ≥ k0 that converges to y¯. Proposition 1.4.2 Let Φ : Rn ×Rm → R∪{+∞} and let S be a set-valued map S : Rn ⇒ Rm . Define the function ϕ(x) = inf{Φ(x, u) |u ∈ S(x)}. If S is lsc at x ¯ and Φ is usc on {¯ x} × S(¯ x), then the function ϕ is usc at x ¯. Definition 1.4.3 A set-valued map S : Rn ⇒ Rm is closed at x ¯ if for any sequence {xk } ∈ Rn and any sequence {yk } ∈ Rm one has ¯, yk → y¯, yk ∈ S(xk ) =⇒ y¯ ∈ S(¯ x). xk → x From this definition one can verify the following useful equivalence: ¯ if ∀¯ y ∈ S(¯ x) there exist two neighborhoods S : Rn ⇒ Rn is closed at x U (¯ x), V (¯ y ) such that x ∈ U (¯ x) =⇒ S(x) ∩ V (¯ y ) = ∅. Definition 1.4.4 A set-valued map S: Rn ⇒ Rm is locally bounded at x ¯ if for some neighborhood N (¯ x) the set x∈N (¯x) S(x) is bounded. S is called ¯ ∈ Rn . It is bounded on Rn locally bounded on Rn if this holds for every x m if rge S is a bounded subset of R .
1.5 Notes and References
23
From the above definition one clearly has that if S is locally bounded at x ¯, then the set S(¯ x) is bounded, and in particular that S is locally bounded at any point x ¯ ∈ cl dom S. Furthermore, in terms of sequences one has that S : Rn ⇒ Rm is locally bounded if {xk } is a bounded sequence, uk ∈ S(xk ) =⇒ {uk } is a bounded sequence. The next results give some important relations between upper semicontinuous and locally bounded set-valued maps. Theorem 1.4.1 Suppose that the set-valued map S : Rn ⇒ Rm is usc at x ¯ and S(¯ x) is closed. Then S is closed at x ¯. Conversely, if S is locally bounded and closed at x ¯ then S is usc at that point. Theorem 1.4.2 Let S be a set-valued map S : Rn ⇒ Rm that is usc, and let C be a compact set in Rn . Then S(C) is compact. Corollary 1.4.1 Let S be a closed set-valued map S : Rn → Rm . Then S is locally bounded at x if and only if S(x) is bounded. As already mentioned, the subdifferential map of a convex function can be seen as a set-valued map that enjoys the following important properties. Proposition 1.4.3 Let f : Rn → R ∪ {+∞} be a proper convex function. Then ∂f is upper semicontinuous and locally bounded at every point x ∈ int(dom f ). Moreover, if f is assumed lsc, then ∂f is closed. We end this section by recalling some useful and basic operations preserving lower/upper semicontinuity of set-valued maps. Proposition 1.4.4 Let {Si |i ∈ I} be a family of set-valued maps defined in a finite-dimensional vector space with appropriate dimensions. Then the following properties hold: (a) The composition map S1 ◦ S2 of lsc (usc) maps Si , i = 1, 2 is lsc (usc). (b) The union of lsc maps ∪i∈I Si is an lsc map. (c) The intersection of usc maps ∩i∈I Si is a usc map. (d) The Cartesian product of a finite number of lsc (usc) maps Πm i=1 Si is an lsc (usc) map.
1.5 Notes and References Convex analysis has emerged as the most powerful and elegant theory underlying the development of optimization and variational analysis. The number of references including research papers, books, and monograph dealing with various aspects of convex analysis and its applications is today very large. The results briefly outlined in this chapter are far from reflecting
24
1. Convex Analysis and Set-Valued Maps: A Review
all the convex-analytic machinery. Thus this chapter is not an introduction to convex analysis. Rather, the writing of this chapter is intended to provide to the reader a concise review of the elements of convex analysis that will be needed and used throughout this monograph. For details and proofs of all the results summarized in this chapter, the reader is referred to the following short bibliography. Standard reference on convex analysis include the books and monographs by J. J. Moreau,[103], which study mostly convex functions and their properties on arbitrary topological vector spaces. R. T. Rockafellar [119], is a comprehensive book on all aspects of convex analysis in Rn . Laurent [85], which is a book with fundamental results on the theory and applications of convex analysis to optimization and approximation theory within the framework of abstract spaces. Ekeland and Temam [67], which deals with variational and infinite-dimensional problems. A recent concise and accessible account of convex analysis is given in Borwein and Lewis [38]. Other classics and useful books studying mainly properties of convex sets are H. G. Eggleston [65] and F. A. Valentine [129].
2 Asymptotic Cones and Functions
This chapter provides the fundamental tools used throughout this monograph. For a given subset of Rn we are interested in studying its behavior at infinity. This leads to the concepts of asymptotic cone and asymptotic function through its epigraph. Using elementary real analysis and geometrical concepts we develop a complete mathematical treatment to handle the asymptotic behavior of sets, functions, and other induced functional operations.
2.1 Definitions of Asymptotic Cones The set of natural numbers is denoted by N, so that k ∈ N means k = 1, 2, . . .. A sequence {xk }k∈N or simply {xk } in Rn is said to converge to x if xk − x → 0 as k → ∞, and this will be indicated by the notation xk → x or x = limk→∞ xk . We say that x is a cluster point of {xk } if some subsequence converges to x. Recall that every bounded sequence in Rn has at least one cluster point. A sequence in Rn converges to x if and only if it is bounded and has x as its unique cluster point. Let {xk } be a sequence in Rn . We are interested in knowing how to handle situations when the sequence {xk } ⊂ Rn is unbounded. To derive some convergence properties, we are led to consider directions dk := xk xk −1 with xk = 0, k ∈ N. From classical analysis, the Bolzano–Weierstrass theorem implies that we can extract a convergent subsequence d = limk∈K dk , K ⊂ N, with d = 0. Now suppose that the sequence {xk } ⊂ Rn is such that
26
2. Asymptotic Cones and Functions
xk → +∞. Then ∃ tk := xk , k ∈ K ⊂ N, such that lim tk = +∞ and lim k∈K
k∈K
xk = d. tk
This leads us to introduce the following concepts. Definition 2.1.1 A sequence {xk } ⊂ Rn is said to converge to a direction d ∈ Rn if xk ∃{tk }, with tk → +∞ such that lim = d. k→∞ tk Definition 2.1.2 Let C be a nonempty set in Rn . Then the asymptotic cone of the set C, denoted by C∞ , is the set of vectors d ∈ Rn that are limits in direction of the sequences {xk } ⊂ C, namely xk =d . C∞ = d ∈ Rn | ∃tk → +∞, ∃xk ∈ C with lim k→∞ tk From the definition we immediately deduce the following elementary facts. Proposition 2.1.1 Let C ⊂ Rn be nonempty. Then: (a) C∞ is a closed cone. (b) (cl C)∞ = C∞ . (c) If C is a cone, then C∞ = cl C. The importance of the asymptotic cone is revealed by the following key property, which is an immediate consequence of its definition. Proposition 2.1.2 A set C ⊂ Rn is bounded if and only if C∞ = {0}. Proof. It is clear that C∞ cannot contain any nonzero direction if C is bounded. Conversely, if C is unbounded, then there exists a sequence {xk } ⊂ C with xk = 0, ∀k ∈ N, such that tk := xk → ∞ and thus the vectors dk = t−1 k xk ∈ {d : d = 1}. Therefore, we can extract a subsequence of {dk } such that limk∈K dk = d, K ⊂ N, and with d = 1. This nonzero vector d is an element of C∞ by Definition 2.1.2, a contradiction. 2 Associated with the asymptotic cone C∞ is the following related concept, which will help us in simplifying the definition of C∞ in the particular case where C ⊂ Rn is assumed convex. Definition 2.1.3 Let C ⊂ Rn be nonempty and define xk 1 n C∞ := d ∈ R | ∀tk → +∞, ∃xk ∈ C with lim =d . k→∞ tk 1 . We say that C is asymptotically regular if C∞ = C∞
2.1 Definitions of Asymptotic Cones
27
Proposition 2.1.3 Let C be a nonempty convex set in Rn . Then C is asymptotically regular. 1 1 ⊂ C∞ clearly holds from the definitions of C∞ Proof. The inclusion C∞ and C∞ , respectively. Let d ∈ C∞ . Then ∃{xk } ∈ C, ∃sk → ∞ such that −1 d = limk→∞ s−1 k xk . Let x ∈ C and define dk = sk (xk − x). Then we have
d = lim dk , x + sk dk ∈ C. k→∞
Now let {tk } be an arbitrary sequence such that limk→∞ tk = +∞. For any fixed m ∈ N , there exists k(m) with limm→∞ k(m) = +∞ such that tm ≤ sk(m) , and since C is convex, we have xm := x + tm dk(m) ∈ C. Hence, 1 2 d = limm→∞ t−1 m xm , showing that d ∈ C∞ . We note that a set can be nonconvex, yet asymptotically regular. Indeed, consider, for example, sets defined by C := S + K, with S compact and K a closed convex cone. Then clearly C is not necessarily convex, but it can 1 . be easily seen that C∞ = C∞ 1 are related to Remark 2.1.1 Note that the definitions of C∞ and C∞ the theory of set convergence of Painlev´e–Kuratowski. Indeed, for a family {Ct }t>0 of susbsets of Rn , the outer limit as t → +∞ is the set lim sup Ct = x | lim inf d(x, Ct ) = 0 , t→+∞
t→+∞
while the inner limit as t → +∞ is the set lim inf Ct = x | lim sup d(x, Ct ) = 0 . t→+∞
t→+∞
It can then be verified that the corresponding asymptotic cones can be written as 1 = lim inf t−1 C. C∞ = lim sup t−1 C, C∞ t→+∞
t→+∞
Proposition 2.1.4 Let C ⊂ Rn be nonempty and define the normalized set xk n CN := d ∈ R | ∃{xk } ∈ C, xk → +∞ with d = lim . k→∞ xk Then C∞ = pos CN , where for any set C, pos C = {λx | x ∈ C, λ ≥ 0}. Proof. Clearly, one always has pos CN ⊂ C∞ . Conversely, let 0 = d ∈ C∞ . Then there exists tk → ∞, xk ∈ C such that −1 d = lim t−1 k xk = lim tk xk k→∞
k→∞
xk , with xk → ∞. xk
28
2. Asymptotic Cones and Functions
Thus the sequence {t−1 k xk } is a nonnegative bounded sequence, and by the Bolzano–Weierstrass theorem, there exists a subsequence {t−1 k xk }k∈K with K ⊂ N such that limk∈K t−1 k xk = λ ≥ 0, which means that d = λdN with dN ∈ CN , namely d ∈ pos CN . 2 We now turn to some useful formulations of the asymptotic cone for convex sets. Proposition 2.1.5 Let C be a nonempty convex set in Rn . Then the asymptotic cone C∞ is a closed convex cone. Moreover, define the following sets: D(x) E
:= :=
{d ∈ Rn | x + td ∈ cl C, ∀t > 0} ∀x ∈ C, {d ∈ Rn | ∃x ∈ C such that x + td ∈ cl C, ∀t > 0},
F
:=
{d ∈ Rn | d + cl C ⊂ cl C}.
Then D(x) is in fact independent of x, which is thus now denoted by D, and C∞ = D = E = F . Proof. We already know that C∞ is a closed cone. The convexity property 1 = C∞ . We simply follows from Proposition 2.1.3, which ensures that C∞ now prove the three equivalent formulations. Let d ∈ C∞ , x ∈ C, t > 0. From Definition 2.1.2, ∃tk → +∞ ∃dk → d, with x + tk dk ∈ C. Take k sufficiently large such that t ≤ tk , then since C is convex, we have −1 x + tdk = (1 − t−1 k t)x + tk t(x + tk dk ) ∈ C.
Passing to the limit, we thus have x + td ∈ cl C, ∀t > 0, thus proving the inclusion C∞ ⊂ D(x), ∀x ∈ C. Clearly, we also have the inclusion D(x) ⊂ E, ∀x ∈ C. We now show that E ⊂ C∞ . For that, let d ∈ E and x ∈ C be such that x(t) := x + td ∈ cl C, ∀t > 0. Then, since d = limt→∞ t−1 x(t) and x(t) ∈ cl C, we have d ∈ (cl C)∞ = C∞ (by Proposition 2.1.1), which also proves that D(x) is in fact independent of x, and we can write D(x) = D = C∞ = E. Finally, let d ∈ C∞ . Using the representation C∞ = D we thus have x + d ∈ cl C, ∀x ∈ C, and hence d + cl C ⊂ cl C, proving the inclusion C∞ ⊂ F . Now, if d ∈ F , then d + cl C ⊂ cl C, and thus 2d + cl C = d + (d + cl C) ⊂ d + cl C ⊂ cl C, and by induction cl C + md ⊂ cl C, ∀m ∈ N. Therefore, ∀x ∈ cl C, dm := x + md ∈ cl C and d = limm→∞ m−1 dm , namely d ∈ C∞ , showing that F ⊂ C∞ , and hence the proof of the three equivalent formulations for C∞ is completed. 2
2.1 Definitions of Asymptotic Cones
29
Remark 2.1.2 When C is a closed convex set the asymptotic cone is also called the recession cone. However, we will keep the terminology asymptotic cone throughout this book. From Proposition 2.1.5, using the set D as a representation of C∞ , an alternative useful representation of the asymptotic cone of a closed convex set is t−1 (C − x), ∀x ∈ C. C∞ = t>0
It is important to note that in the definition of the set D given above, the closure assumption on the convex set C ⊂ Rn is crucial and cannot be removed. Indeed, consider the convex set C = {x ∈ R2 : x1 > 0, x2 > 0} ∪ {(0, 0)}. Then from Definition 2.1.2 we have C∞ = R2+ , while any of the three formulations given in Proposition 2.1.5 would lead to the wrong result C∞ = C. Proposition 2.1.6 For any nonempty closed convex set C ⊂ Rn , one has C = C + C∞ . Proof. The inclusion C ⊂ C + C∞ is clear. Let x ∈ C + C∞ . Then there exists c ∈ C, d ∈ C∞ with x = c + d. Thus, there exists dk ∈ C, tk → ∞ −1 such that t−1 k dk → d, which implies that for k sufficiently large (1 − tk )c + −1 −1 −1 2 tk dk ∈ C and (1 − tk )c + tk dk → x ∈ C. Proposition 2.1.7 For any nonempty closed convex set C ⊂ Rn that contains no lines one has C = conv(ext C) + C∞ . Proof. Thanks to Proposition 2.1.6, we have only to prove that C ⊂ conv(ext C) + C∞ . Let x ∈ C. Then from Theorem 1.1.3, we can write x=
k i=1
λi xi +
m i=k+1
λ i di ,
m
λi = 1, λi ≥ 0, i = 1, . . . , m,
i=1
with xi ∈ ext C, di ∈ extray C, i = 1, . . . , m. Thus di ∈ extray C implies that di = ei + vi , where ei , the endpoint of the ray, is an extreme point of 2 C and vi ∈ C∞ , from which it follows that x ∈ conv(ext C) + C∞ . We now prove the following useful result, which as we shall see is often used as a technical device in analyzing closedness properties involving operations on closed sets in Rn .
30
2. Asymptotic Cones and Functions
Lemma 2.1.1 Let C be a nonempty closed set of Rn , and K the cone in Rn+1 generated by {(1, x) |x ∈ C}, i.e., K = pos{(1, x) |x ∈ C}. Define D := {(0, x) | x ∈ C∞ }. Then cl K = K ∪ D. Proof. The cone K ⊂ Rn+1 generated by {(1, x) |x ∈ C} can be written as K = {λ(1, x) |λ ≥ 0, x ∈ C} = {λ(1, x) |λ > 0, x ∈ C} ∪ {0}. Let y = (t, x) ∈ cl K. Then ∃tk ≥ 0, tk → t, xk ∈ C such that tk (1, xk ) → y, tk xk → x. If t = 0, x ∈ C∞ , and thus y ∈ D. Otherwise, t−1 x ∈ C and y = t(1, t−1 x) ∈ K, showing that cl K ⊂ K ∪ D. Conversely, any y ∈ D can be written as y = limk→∞ tk (1, xk ), with tk → 0, xk ∈ C, so that D ⊂ cl K, and hence since K ⊂ cl K, that D ∪ K ⊂ cl K. 2 Some further elementary operations on asymptotic cones of nonempty convex sets are collected below. Proposition 2.1.8 Let C ⊂ Rn be nonempty and convex. Then: (a) (cl C)∞ = (ri C)∞ = C∞ . (b) For any x ∈ ri C, one has d ∈ (cl C)∞ ⇐⇒ x + td ∈ ri C, ∀t > 0. (c) C ⊂ C∞ =⇒ ri C∞ = ri pos C. (d) C is closed with 0 ∈ C =⇒ C∞ = {d : t−1 d ∈ C, , ∀t > 0}. Proof. Since C is nonempty and convex, the relative interior of C is nonempty, and thus applying Proposition 2.1.1(b) to ri C we obtain (ri C)∞ = (cl ri C)∞ = (cl C)∞ = C∞ , where in the second equation we use Proposition 1.1.4, and (a) is proved, while (b) follows from (a) and the line segment principle, cf. Proposition 1.1.4. To prove (c), we first note that pos C is convex. Then under the assumption C ⊂ C∞ one has pos C ⊂ pos C∞ = C∞ and hence cl pos C ⊂ cl C∞ = C∞ , which together with the fact C∞ ⊂ cl pos C gives that cl pos C = C∞ = cl C∞ , which is equivalent to ri pos C = ri C∞ . In part (d), since 0 ∈ C, the representation of C∞ follows from the representation of C∞ via the set D given in Proposition 2.1.5. 2 Some useful operations with asymptotic cones are now given for arbitrary sets in Rn . Proposition 2.1.9 Let Ci ⊂ Rn , i ∈ I, be an arbitrary index set. Then: (a) (∩i∈I Ci )∞ ⊂ ∩i∈I (Ci )∞ , whenever ∩i∈I Ci is nonempty. (b) (∪i∈I Ci )∞ ⊃ ∪i∈I (Ci )∞ . The inclusion in (a) holds as an equation for closed convex sets Ci having a nonempty intersection. The inclusion (b) holds as an equation when I is a finite index set.
2.2 Dual Characterization of Asymptotic Cones
31
Proof. Let d be any point in the set C := (∩i∈I Ci )∞ , which is closed by definition of the asymptotic cone and satisfies ∃tk → ∞, ∃xk ∈ ∩i∈I Ci such that
xk → d. tk
Therefore, ∃tk → ∞, ∃xk ∈ Ci , ∀i ∈ I such that t−1 k xk → d, implying that d ∈ (Ci )∞ for all i ∈ I and proving (a). The relation (b) is proved in a similar way through the definition of the corresponding asymptotic cone. Finally, the special result for the convex case follows by applying the characterization given in Proposition 2.1.5. 2 Proposition 2.1.10 For any sets Ci ⊂ Rn , i = 1, . . . , m, one has (C1 × · · · × Cm )∞ ⊂ (C1 )∞ × · · · × (Cm )∞ . The inclusion holds as an equation if every Ci is nonempty and convex. Proof. The inclusion is proved by direct use of the definition of the asymptotic cone, and in the convex case using the characterization provided in Proposition 2.1.5. 2 Proposition 2.1.11 Let A : Rn → Rm be a linear mapping and C a closed convex set in Rn such that the inverse image of C is nonempty. Then (A−1 (C))∞ = A−1 (C∞ ). Proof. Since A is continuous and C is closed and convex, A−1 (C) is closed and convex. Let x ∈ A−1 (C). Then d ∈ (A−1 (C))∞ if and only if A(x + td) = Ax + tAd ⊂ C, ∀t ≥ 0, but the latter means that Ad ∈ C∞ , namely 2 d ∈ A−1 (C∞ ). We close the section by giving explicit formulas of asymptotic cones for some other important sets. Example 2.1.1 (i) Let C be a cone in Rn . Then C∞ = cl C. (ii) Let C be an affine set. Then C∞ is the linear subspace parallel to C. (iii) Let C be a polyhedral convex set C := {x ∈ Rn | Ax ≤ b}, where A is an m × n matrix and b ∈ Rm . Then C∞ = {d ∈ Rn | Ad ≤ 0}.
2.2 Dual Characterization of Asymptotic Cones There exists a close connection between the support function of a set and its asymptotic cone. We show below that for a closed convex set C ⊂ Rn ,
32
2. Asymptotic Cones and Functions
one can in fact give a dual characterization of the asymptotic cone, via the barrier cone of C. ∗ denote the polar Theorem 2.2.1 Let C ⊂ Rn be nonempty and let C∞ cone of C∞ . Then the following relations hold: ∗ . (a) dom σC ⊂ C∞ ∗ ∗ ⊂ dom σC . (b) If int C∞ = ∅, then int C∞ (c) If C is convex, then (dom σC )∗ = C∞ . ∗ . Then ∃ 0 = d ∈ C∞ such that d, y > 0. Since Proof. (a) Let y ∈ C∞ d ∈ C∞ , ∃tk → ∞, ∃xk ∈ C with t−1 k xk → d, and with d, y > 0, it follows that xk , y → +∞, proving that y ∈ dom σC . (b) Let y ∈ dom σC . Then ∃xk ∈ C with xk , y → +∞. Passing to subsequences if necessary we can assume without loss of generality that xk −1 xk → d = 0, with d ∈ C∞ , and hence xk −1 xk , y ≥ 0. Therefore, ∗ , i.e., ∀ε > 0 we have d, y + εd ≥ εd2 > 0, showing that y + εd ∈ C∞ ∗ . y ∈ int C∞ ∗∗ = (c) Since C is assumed convex, then C∞ is a closed convex cone and C∞ C∞ . Using (a) we thus have ∗ ∗∗ dom σC ⊂ C∞ =⇒ C∞ = C∞ ⊂ (dom σC )∗ .
We now prove the reverse inclusion. Let d ∈ (dom σC )∗ , t > 0, and let x ¯ be any point in C. Since td ∈ (dom σC )∗ , we have for any y ∈ dom σC , ¯ x + td, y
= ¯ x, y + td, y ≤ ¯ x, y ≤
sup x, y = σC (y).
x∈C
Since for any y ∈ dom σC we have σC (y) = +∞, the above inequality remains valid for any y ∈ Rn . Therefore, invoking Theorem 1.3.1 we obtain for all t > 0, x ¯ + td ∈ cl C, which by Proposition 2.1.5 proves that d ∈ C∞ . 2
2.3 Closedness Criteria Given a nonempty closed set of Rn , we are interested in answering the following question: Under what conditions does the image of a closed set under a linear mapping remain closed. This type of result is of fundamental importance and is at the root of several closedness criteria that are particularly useful for analyzing the existence of solutions of extremum problems. Asymptotic cones play a key role in the derivation of such results. Let C ⊂ Rn be a nonempty closed set and let A : Rn → Rm be a linear mapping. Let ker A := {x : Ax = 0} := A−1 (0). As we shall see, a basic
2.3 Closedness Criteria
33
sufficient condition that guarantees that A(C) is closed is ker A ∩ C∞ = {0}.
(2.1)
Moreover, under the condition (2.1) we also have A(C∞ ) = (A(C))∞ . For a convex set, a weaker sufficient condition to guarantee closedness of A(C) is (2.2) ker A ∩ C∞ is a linear subspace. Unfortunately, even in the convex case, the above condition can fail, as shown in the following two examples. Example 2.3.1 Let C = {x ∈ R2 | x2 ≥ x21 } and let A be the linear mapping A : R2 → R2 given by A(x1 , x2 ) = (x1 , 0), i.e., the projection onto the x1 -axis. Then, one has C∞
= {d ∈ R2 | d1 = 0, d2 ≥ 0},
A(C) = {x ∈ R2 | x2 = 0} = x1 -axis, ker A = {x ∈ R2 | x1 = 0} = x2 -axis, and condition (2.2) fails, yet A(C) is a closed set. Example 2.3.2 Let D = {x ∈ R2+ | x1 + x2 ≥ 1} and A be the linear mapping as in Example 2.3.1. One easily see that ker A ∩ D∞ = {x ∈ R2 | x1 = 0, x2 ≥ 0}, and thus condition 2.2 fails; yet A(D) is a closed set. As we shall see, the sets C and D are quite different in nature. The set C belongs to a class of sets called continuous sets, while the set D is in the class of asymptotically linear sets. These two classes of sets are quite general and are very different one from the other and will be introduced later on. We thus need conditions that can handle the situations just described. In fact, the next general result will establish a necessary and sufficient condition for preserving closedness of A(C). Theorem 2.3.1 Let C be any nonempty closed set of Rn and A : Rn → Rm a linear mapping. Let {yk } be any sequence in A(C) converging to y and define the set xk →x ¯ . S := {xk } | xk ∈ C : xk → ∞, Axk − y ≤ y − yk , xk (2.3) Then a necessary and sufficient condition for the set A(C) to be closed is that for each y and each sequence {yk } ∈ A(C) converging to y, (a) either S is empty or (b) for each sequence {xk } ∈ S there exists a sequence {zk , ρk } ⊂ Rn × R++ such that for k sufficiently large, xk − ρk zk ∈ C, A(xk − ρk zk ) − y ≤ y − yk , ρk ∈ (0, xk ], and zk → z with z − x ¯ < 1.
(2.4)
34
2. Asymptotic Cones and Functions
Proof. First, suppose that yk ∈ A(C), yk → y. Then there exists xk ∈ C such that yk = Axk , which implies Axk − y = yk − y. Thus, S = ∅ means that a subsequence of {xk } is bounded and has a cluster point, say x∗ ∈ C. Passing to the limit in the latter relation implies that Ax∗ −y = 0, showing that A(C) is closed. We now prove the second case, described in (b). Suppose that S = ∅, and (2.4) holds, and let yk → y with yk = Axk , xk ∈ C. Define the set Sk := {x ∈ C | Ax − y ≤ y − yk } and consider the optimization problem inf{x | x ∈ Sk }. By (2.3), we clearly have xk ∈ Sk , and hence Sk is nonempty. Furthermore, Sk is closed, and then the existence of xk ∈ argmin{x : x ∈ Sk } is guaranteed. If we can show that the sequence {xk } is bounded, we are done. Indeed, in that case there would exist a subsequence {xk }k∈K , K ⊂ N, such that limk∈K xk = x ∈ C and with Ax = y, proving the closedness of A(C). To prove this, suppose that {xk } is unbounded. Passing to subsequences if necessary, we can suppose without loss of generality that xk −1 xk → x ¯ = 0, and then using (2.3) there exists {zk , ρk } satisfying (2.4). As a consequence, xk − ρk zk ∈ Sk , and since by definition of xk we have xk ≤ x, ∀x ∈ Sk , we obtain in particular that xk ≤ xk − ρk zk . Now, xk − ρk zk = (1 − ||xk ||−1 ρk )xk + ρk (||xk ||−1 xk − zk ), ≤ (1 − ||xk ||−1 ρk )xk + ρk ||xk ||−1 xk − zk , = xk + ρk ( ||xk ||−1 xk − zk − 1), x − z ≥ 1, and hence ||xk ||−1 xk − zk ≥ 1. Passing to the limit we get ¯ contradicting assumption (2.4). Conversely, suppose that A(C) is closed and let yk → y with yk ∈ A(C). Then, since A(C) is closed, there exists x ∈ C with Ax = y. Set zk = ||xk ||−1 (xk − x), ρk = xk , and then (2.4) is clearly satisfied. 2 In general, it is easier and more convenient to choose in Theorem 2.3.1 ¯ or more generally zk = α¯ x, α ∈ (0, 2), in assumption (2.4). In this zk = x case with α = 1 we can say more about the image of C as shown in the theorem given below. As an application of that theorem we will also show how situations previously discussed, illustrating the failure of condition (2.2) to characterize closedness, can now be adequately handled. For any nonempty closed set C of Rn and any linear map A : Rn → Rm , it is immediate, using the definition of the asymptotic cone, to verify the inclusion A(C∞ ) ⊂ A(C)∞ . However, the reverse inclusion requires a much finer analysis. Theorem 2.3.2 Let C be any nonempty closed set of Rn and A : Rn → Rm a linear mapping. Let {yk } be any sequence in A(C) converging to y and let S be the set defined in (2.3). Suppose that for each y and any sequence yk ∈ A(C) converging to y, either S is empty or for any sequence {xk } ∈ S there exists a sequence {ρk } ⊂ R++ such that for k sufficiently
2.3 Closedness Criteria
35
large, xk − ρk x ¯ ∈ C, ρk ≤ xk . Then A(C) is closed, and we have A(C∞ ) = (A(C))∞ .
(2.5)
Proof. The closedness of A(C) follows as an immediate consequence of the ¯, it follows from (2.3) that previous theorem. Indeed, if we take zk = z = x A¯ x = 0, and (2.4) holds. Now, to prove (2.5) we first note that the inclusion A(C)∞ ⊃ A(C∞ ) follows immediately from the definition of the asymptotic cone. Thus it remains to prove the reverse inclusion (A(C))∞ ⊂ A(C∞ ). Let y ∈ (A(C))∞ and let uk ∈ C, tk → +∞, and yk = Auk with t−1 k yk → y. Define Sk = {x ∈ C | Ax = yk }. Then, Sk is a nonempty closed set, and the existence of xk ∈ argmin{x | x ∈ Sk } is guaranteed. We prove now that we cannot have limk∞ t−1 k xk = +∞. Indeed, in the contrary case, we would have limk∞ xk = +∞. Without loss of generality, we can suppose ¯ with ¯ x = 1. We thus obtain that xk −1 xk → x A¯ x = lim A k→∞
yk y k tk xk = lim = lim = 0. xk k→∞ xk k→∞ tk xk
Since (2.3) holds, there exists ρk ∈ (0, xk ] such that xk − ρk x ¯ ∈ C and ¯ ∈ Sk . By definition of xk we have since A¯ x = 0, it follows that xk − ρk x ¯, and as in the proof of Theorem 2.3.1 it follows that xk ≤ xk − ρk x ¯ ≥ 1 and passing to the limit, we get a contradiction. xk −1 xk − x Finally, since the sequence {t−1 k xk } is bounded, passing to subsequences if necessary, we can conclude that {t−1 k xk } converges to some x∞ . Since xk ∈ Sk , we have x∞ ∈ C∞ and t−1 k Axk → y = Ax∞ , proving the desired result. 2 We consider now the situation in which Theorem 2.3.2 is satisfied and give further applications. Corollary 2.3.1 Let C be a nonempty closed set of Rn and A : Rn → Rm a linear mapping. Let L(C) = C∞ ∩ −C∞ and L = L(C) ∩ ker A and suppose that the following two conditions hold: (a) For k sufficiently large, Ck + L ⊂ C, with Ck = {x ∈ C : x ≥ k}. (b) z ∈ ker A ∩ C∞ implies z ∈ −C∞ . Then A(C) is closed and (A(C))∞ = A(C∞ ). Proof. Let {yk } ⊂ A(C) be a sequence converging to y and let {xk } ∈ S be an unbounded sequence satisfying (2.3). Then clearly, x ¯ ∈ ker A ∩ C∞ , and from assumption (b) it follows that x ¯ ∈ −C∞ , which together with x ∈ L. Let ρ > 0. Then by assumption (a), for k x ¯ ∈ C∞ implies that −¯ x ∈ C, and thus we can apply Theorem sufficiently large we have xk − ρ¯ 2.3.2. 2
36
2. Asymptotic Cones and Functions
Remark 2.3.1 It is possible to replace assumption (a) in Corollary 2.3.1 by the somewhat stronger assumption C + L ⊂ C. Corollary 2.3.2 Let C be a nonempty closed set of Rn and A : Rn → Rm a linear mapping. Then, A(C) is closed and A(C∞ ) = (A(C))∞ under either of the following conditions: (a) ker A ∩ C∞ = {0}, (b) C is convex and ker A ∩ C∞ is a linear subspace. Proof. Assume (a) holds, and in order to use Theorem 2.3.2 let {yk } ⊂ A(C) converge to y. Then there cannot exist any sequence xk ∈ C satisfying (2.3). Indeed, in the contrary case, we would have xk yk y − yk A xk − xk ≤ xk , and passing to the limit we obtain A¯ x = 0. Furthermore, since xk ∈ C, x = 1, in contradiction to the assumption it follows that x ¯ ∈ C∞ with ¯ (a), asserting ker A ∩ C∞ = {0}. Now assume (b). Then since C is convex, assumption (a) in Corollary 2.3.1 is satisfied by Proposition 2.1.5. Furthermore, since ker A ∩ C∞ is a linear subspace, the assumption z ∈ ker A ∩ C∞ implies that z ∈ −C∞ , and assumption (b) in Corollary 2.3.1 holds. 2
Corollary 2.3.3 Let S ⊂ Rn be closed with 0 ∈ S. Then cl pos S = pos S ∪ S∞ . Furthermore, if S is bounded, then pos S is closed. Proof. Let K be the cone in Rn+1 generated by {(1, x) | x ∈ S}, and let A : (α, x) → x. Then pos S = A(K). By Lemma 2.1.1 one has cl K = K ∪ {(0, x) | x ∈ S∞ } and A(cl K) = pos S ∪ S∞ . Invoking Corollary 2.3.2 (a) with C := cl K and since 0 ∈ S noting that in that case ker A ∩ C∞ = {0}, it follows that pos S ∪ S∞ is closed. Now, since pos S ⊂ pos S ∪ S∞ and S∞ ⊂ cl pos S, then cl pos S ⊂ cl(pos S ∪ S∞ ) = pos S ∪ S∞ ⊂ pos S ∪ cl pos S ⊂ cl pos S, and hence pos S ∪ S∞ = cl pos S. Finally, when S is bounded S∞ = {0}, and this proves the last statement. 2 Asymptotic Linear Sets We introduce now the class of asymptotically linear sets. As we shall see, this notion is of great interest. We begin with the basic definition of this concept.
2.3 Closedness Criteria
37
Definition 2.3.1 Let C be a nonempty closed set of Rn . Then C is said to be an asymptotically linear set if for each ρ > 0 and each sequence {xk } satisfying ¯, (2.6) xk ∈ C, xk → +∞, xk xk −1 → x there exists k0 ∈ N such that x ∈ C ∀k ≥ k0 . xk − ρ¯
(2.7)
x ∈ C. This justifies the If C is a closed convex set, we also have xk + ρ¯ terminology “asymptotically linear”. As a consequence of Definition 2.3.1 a set that is the intersection of a finite number of sets, each of them being the union of a finite number of asymptotically linear sets, is also an asymptotically linear set. Definition 2.3.2 A set C ⊂ Rn is said to be a simple asymptotically polyhedral set if there exists a nonnegative integer l for which the set Cl := C ∩ {x : x ≥ l} admits the decomposition Cl = K + M, with K compact and M a polyhedral cone. The set C is said to be asymptotically polyhedral if it is the intersection of a finite number of sets, each of them being the union of a finite number of simple asymptotically polyhedral sets. When l = 0, the simplest set of this type is clearly a polyhedral set. Recall that when C is convex without lines, the Krein–Milman theorem (cf. Theorem 1.1.3), always permits the decomposition Cl = K + M , where K is the convex hull of the extreme points of C, M = C∞ , and l = 0. In this case when C∞ is polyhedral, and when the set of extreme points is bounded (which is the case if C is a polyhedral set), then C is asymptotically polyhedral. It is also easy to note that the extension to sets for which l > 0 gives a wider class of sets. Proposition 2.3.1 An asymptotically polyhedral set C of Rn is asymptotically linear. Proof. Obviously the proposition holds if it holds for simple asymptotically polyhedral sets. Thus, we suppose that C is a simple asymptotically polyhedral set and let {xk } be a sequence satisfying (2.6), and ρ > 0. If x} ∈ C. Without (2.7) is not satisfied, there exists a subsequence of {xk − ρ¯ x ∈ C, and that xk ≥ l for loss of generality we can suppose that xk − ρ¯ each k. Since M is a polyhedral cone, M is finitely generated (cf. Chapter 1) and there exist rays di , i = 1, . . . , r, such that r M = y | ∃λi ≥ 0 i = 1 . . . r such that y = λ i di . i=1
38
2. Asymptotic Cones and Functions
Then for each k, since xk ∈ Cl , there exist yk ∈ K, a subset Ik ⊂ {1, . . . , m}, and λki ≥ 0 for i ∈ Ik such that xk = yk + zk , zk = λki di , i∈Ik
and we can assume that the vectors {di }, i ∈ Ik , are linearly independent. Indeed it can be easily seen that M is a finite union of sets of the same type but with the vectors di linearly independent. Since {yk } is bounded, it follows that limk→∞ zk = +∞. As a consequence, there exist a subsequence zkm and a nonempty set of indices I such that the following hold: lim λki m = +∞ ,
m→∞
λki m → µi ≥ 0 ∀i ∈ I, km i∈I λi
λki m ≥ 0 ∀i ∈ I,
and the sequence {λki m } is the vectors di , i ∈ I, are linearly independent, k bounded for i ∈ I. Now let wk = i∈I λi di , w = i∈I µi di . Then w = 0 and wk w , ¯= lim m km = w, x m→∞ w i∈I λi and we thus have xkm − ρ¯ x = ykm +
i∈I
λki m
µi −ρ w
di +
λki m di .
i ∈I
µi ≥ 0 for i ∈ I, it follows that Since for m sufficiently large λki m − ρ w xkm − ρ¯ x ∈ C for m sufficiently large, which is impossible. 2
As we shall see in the next chapter, where we introduce the class of asymptotically stable functions, which contains in particular convex polynomial functions, we can enlarge considerably the class of asymptotically linear sets. Theorem 2.3.3 Let C be asymptotically linear. Then A(C) is closed and A(C∞ ) = (A(C))∞ . Proof. Let {yk } ⊂ A(C) converge to y and let {xk } be any sequence satisfying (2.3). Then, since C is asymptotically linear, by definition there ¯ ∈ C, ρk ≤ xk , exists ρk > 0 such that for k sufficiently large xk − ρk x and we can thus invoke Theorem 2.3.2, which implies the results. 2
2.3 Closedness Criteria
39
Closure of the Sum of Closed Convex Sets The sum of closed convex sets might not be closed even when the sets are themselves closed. It is thus important to find conditions under which closedness is preserved. Obviously, if C1 , C2 are closed and one of the two sets is also bounded, then C1 + C2 is also closed. As another application of Theorem 2.3.2, we now derive weaker and general conditions preserving closedness under appropriate assumptions on the sets involved. Definition 2.3.3 Let Ci ⊂ Rn , i = 1, . . . , m, be a collection m of nonempty closed sets such that for any zi ∈ (Ci )∞ , i = 1, . . . , m, i=1 zi = 0. Then: (a) If one has zi = 0 for all i = 1, . . . , m, the collection Ci is said to be in general position. (b) If for any i = 1, . . . , m one has (i) zi ∈ −(Ci )∞ , (ii) zi + Ci ⊂ Ci , the collection Ci is said to be in relative general position. Remark 2.3.2 Inclusion (ii) is in fact equivalent to R+ zi + Ci ⊂ Ci . The condition (a) imposed on the set Ci in the definition is stronger than the second one, asking for relative general position of the sets Ci , but is often easier to use for proving closedness of the sum of sets; see Corollary 2.3.4 below. Theorem 2.3.4 Let Ci ⊂ Rn , i = 1, . . . , m, be nonempty closed sets, which m are supposed to be in relative general position. Then i=1 Ci is a closed set, and we have (C1 + · · · + Cm )∞ ⊂ (C1 )∞ + · · · + (Cm )∞ , where the inclusion holds as an equation if in addition the Ci are all convex. Proof. Let C := C1 ×· · ·×Cm and define the linear map A : (x1 , . . . , xm ) → x1 + · · · + xm , xi ∈ Rn . Then A(C) = C1 + . . . + Cm . Since by Proposition 2.1.10 one has C∞ ⊂ (C1 )∞ × . . . × (Cm )∞ , then ∀z ∈ ker A ∩ C∞ it follows from hypothesis (i) and (ii) of Definition 2.3.3 that −z + C ⊂ C. Therefore, −2z + C = −z + (−z + C) ⊂ C, and by recurrence, −kz + C ⊂ C, ∀k ∈ N∗ . Thus, ∀x ∈ C, xk = x − kz ∈ C and limk→∞ xkk = −z, implying −z ∈ C∞ . Invoking Corollary 2.3.1 (with hypothesis (a) of that theorem replaced by (ii) of Definition 2.3.3, cf. Remark 2.3.2, the result concerning the inclusion is proved. If the Ci are in addition convex, the reverse inclusion is an immediate consequence of Proposition 2.1.5. 2 As another application of Theorem 2.3.4 we obtain the following simple test to check the closedness of the sum of nonempty closed sets.
40
2. Asymptotic Cones and Functions
Corollary 2.3.4 Let Ci ⊂ Rn , i = 1, . . . , m, be nonempty closed sets. Supm pose that i=1 zi = 0 implies that zi = 0 ∀i = 1, . . . , m. mzi ∈ (Ci )∞ with Then i=1 Ci is a closed set The next application of Theorem 2.3.2 concerns a closedness criterion of the set of convex combinations of a finite number of nonempty closed sets Ci , i = 1, . . . , m, of Rn . Define the following two sets: m S := x ∈ Rn | x = λi xi , λ ∈ ∆m , xi ∈ Ci , i = 1, . . . , m (2.8) i=1
T
:=
x∈R |x= n
m
λi ∗ xi , λ ∈ ∆m
(2.9)
i=1
where λi ∗ xi :=
λi xi , xi ,
with xi ∈ Ci if λi > 0, with xi ∈ (Ci )∞ if λi = 0,
and ∆m denotes the simplex in Rm . Theorem 2.3.5 For a finite collection of nonempty closed sets Ci ⊂ Rn , i = 1, . . . , m, that are in relative general position, one has cl S = T . Proof. Let Ki be a cone in Rn+1 generated by {(1, xi ) |xi ∈ Ci } and let Di := {(0, xi ) |xi ∈ (Ci )∞ }. Applying Lemma 2.1.1 for each i = 1, . . . , m we have cl Ki = Ki ∪ Di . (2.10) Furthermore, since Ki is a cone, we also have cl Ki = (Ki )∞ = (cl Ki )∞ = Ki ∪ Di .
(2.11)
We now want to apply Theorem 2.3.4 to the sets cl Ki . Let z1 , . . . , zm be vectors such that zi := (ti , di ) ∈ (cl Ki )∞ , ∀i ∈ [1, m],
m
zi = 0.
i=1
By (2.11), zi = (ti , di ) ∈ (cl Ki )∞ implies that zmi = (ti , di ) ∈ Ki ∪Di , which in turns implies ti = 0, di ∈ (Ci )∞ , and i=1 di = 0, i.e., zi = (0, di ) with di ∈ (Ci )∞ , and therefore, since hypothesis (i) of Definition 2.3.3(b) is satisfied for the sets Ci , it is also satisfied for the sets cl Ki , namely −zi ∈ cl Ki , ∀i = 1, . . . , m. Now, we need to verify that hypothesis (ii) of Definition 2.3.3(b) holds for the sets cl Ki , given that it holds for Ci ; i.e., we have to prove that zi + cl Ki ⊂ cl Ki , ∀i = 1, . . . , m.
(2.12)
2.3 Closedness Criteria
41
Let zi = (0, di ), di ∈ (Ci )∞ and take any 0 = yi ∈ cl Ki . Then by (2.10), yi ∈ Ki or yi ∈ Di . If yi ∈ Ki , one has yi = λi (1, y¯i ), λi > 0, with y¯i ∈ Ci and di yi + zi = (λi , λi y¯i ) + (0, di ) = λi 1, y¯i + . λi Since y¯i ∈ Ci , λ−1 i di ∈ (Ci )∞ , and the sets Ci satisfy hypothesis (ii) of Definition 2.3.3(b), then ui := y¯i + λ−1 i di ∈ Ci , and hence yi + zi = λi (1, ui ) ∈ Ki . Otherwise, if yi ∈ Di , one has yi = (0, d¯i ), d¯i ∈ (Ci )∞ , and yi + zi = (0, d¯i + di ). But since d¯i ∈ (Ci )∞ , then for all i = 1, . . . , m, ∃tki > 0, tki → 0+ , ∃xki ∈ Ci such that d¯i = lim tki xki , k→∞
so that
di di d¯i + di = lim tki xki + k , with xki + k ∈ Ci , k→∞ ti ti
by hypothesis (b)(ii) of Definition 2.3.3 on the sets Ci . This implies that d¯i + di ∈ (Ci )∞ and therefore yi + zi ∈ Di , proving (2.12). We can thus apply Theorem 2.3.4 to the sets cl Ki to conclude that cl K1 + · · · + cl Km is closed. But since one always has cl(K1 + · · · + Km ) = cl(cl K1 + · · · + cl Km ), it follows that cl(K1 + · · · + Km ) = cl K1 + · · · + cl Km .
(2.13)
Now consider the hyperplane H := {(1, x) |x ∈ Rn } and let E F
:= (cl K1 + · · · + cl Km ) ∩ H, := cl(K1 + · · · + Km ) ∩ H.
Since by (2.11) one has cl Ki = Ki ∪ Di , one can verify that in fact E = (1, T ), with T defined in (2.9). Now y := (t, x) ∈ F if and only if y = limk→∞ y k with yk =
λki (1, xki ), λki ≥ 0, xki ∈ Ci and
lim
k→∞
m
λki = 1.
i=1
m Define µki :=λki ( i=1 λki )−1 . Then y ∈ F if and only if y = (1, x) with m x = limk→∞ i=1 µki xki with xki ∈ Ci . In other words, one has F = (1, cl S). Finally, by the definition of E and F and from (2.13) it follows that T = cl S. 2 The convex case is much simpler and follows as an immediate consequence of Theorem 2.3.5.
42
2. Asymptotic Cones and Functions
Corollary 2.3.5 Let Ci ⊂ Rn , i = 1, . . . , m, be nonempty m closed convex sets such that for any zi ∈ (Ci )∞ ∀i = 1, . . . , m, and i=1 zi = 0 one has zi ∈ −(Ci )∞ for each i = 1, . . . , m. Then T = cl S = cl(conv ∪m i=1 Ci ), where S, T are as defined in (2.8)–(2.9). Proof. Note that when Ci is convex for each i, then one has S = conv(∪m i=1 Ci ), and hypothesis (ii) of Definition 2.3.3(b) holds. Applying Theorem 2.3.5, the result follows. 2 As another consequence of Theorem 2.3.5 we obtain a decomposition formula for the closed convex hull of an arbitrary closed set in Rn , in terms of its convex hull and the associated asymptotic cone. This leads to the following notions. Definition 2.3.4 Let C be a nonempty set of Rn and suppose that for any n+1 vectors c1 , . . . , cn+1 such that i=1 ci = 0, ci ∈ C∞ , i = 1, . . . , n + 1, one has ci ∈ −C∞ , ci + C ⊂ C, ∀i = 1, . . . , n + 1. Then C is said to be weakly semibounded. If instead of the above inclusion we have the stronger assumption ci = 0
∀i = 1 . . . m,
C is said to be semibounded. Another way to say that a set C is semibounded is to suppose that C∞ is a pointed cone; cf. Definition 1.1.3. Corollary 2.3.6 Let C be a closed weakly semibounded set. Then cl conv C = conv C + conv C∞ . Proof. Apply Theorem 2.3.5 with m = n + 1 and Ci = C ∀i.
2
Lemma 2.3.1 Let K be a closed pointed cone in Rn . Then: n+1 (a) ∃θ > 0 such that xi ≤ θ i=1 xi , ∀xi ∈ K, i = 1, . . . , n + 1. (b) conv K is closed pointed cone. Proof. (a) The proof is by contradiction. Then for all j ∈ N, there exist n+1 for i = 1, . . . , n + 1, uji ∈ K such that uji > j i=1 uji . Since K is n+1 j pointed, it follows that i=1 ui = 0, and then if we set n+1 j uji j xi = n+1 j , y j = xi , i=1 ui i=1
2.3 Closedness Criteria
we have
∀i
xji
∈ K, y = 1, λ := j
j
43
−1
max
1≤i≤n+1
xji )
→ 0.
Since λj xji ≤ 1, there exists a subsequence {λjl , xji l , i = 1, . . . , n + 1} such that for each i, λjl xji l → zi . Now since K is a closed cone, zi ∈ K and n+1 n+1 since i=1 λjl xji l = λjl , it follows that i=1 zi = 0. Furthermore, since n+1 j j one has 1 ≤ i=1 λ xi ≤ n + 1, then at least one zi is nonzero, which contradicts the hypothesis that K is pointed. (b) The fact that conv K is pointed is obvious. Now let us prove that it Theorem 1.1.1 there is closed, and let xj ∈ conv K with xj → x. Then by n+1 exist xji ∈ K with i = 1, . . . , n+1 and such that xj = i=1 xji . By part (a) j the sequences {xi }j∈N are bounded. Passing to subsequences if necessary, n+1 it follows that there exist {xji } with xji → yi . Then, x = i=1 yi ∈ conv K, and we can conclude that conv K is closed. 2 Lemma 2.3.2 Let C be a nonempty set of Rn . Then: (a) (conv C)∞ = conv C∞ . (b) If in addition C is semibounded then conv C is semibounded. Proof. (a) Clearly, one has that (conv C)∞ is a convex set. Therefore, the inclusion C∞ ⊂ (conv C)∞ implies that conv C∞ ⊂ (conv C)∞ . To prove the reverse inclusion (conv C)∞ ⊂ conv C∞ , let d ∈ (conv C)∞ . Then there exist xk ∈ conv C, tk → +∞ such that t−1 k xk → d. Every xk ∈ conv C can be written as n n λik xik , with xik ∈ C, λik ≥ 0, λik = 1. xk = i=0
i=0
i i {t−1 k λk xk }
is a bounded sequence, passing to subsequences Since nif necesi i i i λ x → d ∈ C and therefore d = sary, it follows that t−1 ∞ k k i=0 d , so k n that d ∈ conv C∞ . To prove (b), note that C ⊂ R being semi-bounded means that C∞ is pointed. Invoking Lemma 2.3.1(b), the latter implies that conv C∞ is pointed, which in turn implies that (conv C)∞ is pointed and hence conv C is semibounded. 2 Lemma 2.3.3 Let C be a nonempty convex and semibounded set of Rn . Then int C∞ = ∅ ⇐⇒ cl dom σC is pointed. Proof. Applying Theorem 2.2.1 to the closed convex set cl C one has C∞ = (cl C)∞ = (cl dom σC )∗ , and the desired result follows from Proposition 1.1.15. 2
44
2. Asymptotic Cones and Functions
2.4 Continuous Convex Sets Convex compact sets in Rn enjoy many important properties that are not shared by closed convex sets in general. For example, if C and D are compact and convex, then the sets C + D and conv(C ∪ D) enjoy the same property. Moreover, if C ∩ D = ∅, then C and D can be strongly separated by a hyperplane and δ(C, D) = inf{c − d | c ∈ C, d ∈ D} > 0. In general, none of these properties remain true for arbitrary closed convex sets in Rn . However, there is a natural class of closed convex sets in Rn that are not necessarily compact, yet which enjoy these properties. This is the class of continuous sets. Definition 2.4.1 A nonempty closed convex set of Rn is called continuous if its support function x → σC (x) is continuous for any x = 0. We recall that if σC (x) = +∞, the continuity of σC at x means that there exists a neighborhood V of x such that σC (y) = +∞ for all y ∈ V . Continuous convex sets can be characterized in a more geometric sense using the concepts of boundary rays and asymptotes. Definition 2.4.2 Let C be a nonempty closed set of Rn . A boundary ray of the set C is a half-line that is contained in the boundary of C. An asymptote of C is a half-line ρ contained in Rn \ C such that δ(ρ, C) := inf{x − y | x ∈ ρ, y ∈ C} = 0. Proposition 2.4.1 Let C be a nonempty closed convex set of Rn , x ∈ C, and suppose that the half-line {y + tρ : t ≥ 0} with ρ ∈ C∞ and y ∈ Rn has an empty intersection with C. For λ ∈ R, let x(λ) = (1 − λ)x + λy. Then there exist λ ∈ [0, 1], s ≥ 0 such that the half-line D(λ, s) = {x(λ) + tρ | t ≥ s} is a boundary ray or an asymptote of C. Proof. For any λ ∈ R and any t ≥ 0, define G(λ, t) = dist(x(λ) + tρ, C) and g(λ) = inf{G(λ, t) | t ≥ 0}. Then, since C is a closed convex set, the function g is finite and convex on R, and as a consequence, is continuous. Let E := {λ ∈ [0, 1] | g(λ) = 0}. Since x ∈ C and ρ ∈ C∞ , one has 0 ∈ E and since g is continuous, it follows that E is a nonempty compact set. Therefore, there exists λ∗ ∈ [0, 1] that maximizes λ on E. If λ∗ = 1, then by definition D(1, 0) is an asymptote of C; otherwise λ∗ < 1. Now suppose that D(λ∗ , 0) is not an asymptote. Then D(λ∗ , 0) ∩ C = ∅. Take u ∈ D(λ∗ , 0)∩C. Then D := {u+tρ | t ≥ 0} ⊂ C. Without loss of generality we can suppose that int C = ∅ (otherwise, D would be a boundary ray),
2.4 Continuous Convex Sets
45
and if D were not a boundary ray, there would exist s ≥ 0 such that z := u + sρ ∈ int C. As a consequence, there would exist ∆λ > 0 such that {x(λ∗ + ∆λ) + tρ | t ≥ 0} ∩ C = ∅, a contradiction with the fact that λ∗ is the maximum of λ on E. 2 The next result gives an important characterization of continuous convex sets. Theorem 2.4.1 A nonempty closed convex subset C of Rn is continuous if and only if C has no boundary ray or asymptote. Proof. Assume first that C has no boundary ray or asymptote and suppose that C is not continuous at some point u = 0. Then there exists a sequence {uk } converging to u such that σC (uk ) does not converge to σC (u). Since σC is lsc, it follows that α := lim supk→∞ σC (uk ) > σC (u). Furthermore, there exist β ∈ R with σC (u) < β ≤ α and a sequence {xkj } ⊂ C such that lim ukj , xkj ≥ β.
j→∞
(2.14)
When α ∈ R, taking β = α, the latter inequality follows from the definition of the support functional, and when α = +∞, it follows for the same reason. Let us prove now that the sequence {xkj } cannot be bounded. Assume the contrary. Then with {xkj } bounded, without loss of generality we can suppose that xkj → x ∈ C, and taking the limit in (2.14) it follows that σC (u) < β ≤ u, x, which is impossible. Thus, without loss of generality ¯. Dividing both we can suppose that xkj → +∞ and xkj xkj −1 → x members of (2.14) by xkj and passing to the limit when j → +∞, it follows that u, x ¯ ≥ 0. Now let y ∈ C, with y, u > σC (u); as a consequence it follows that y + λ¯ x, u > σC (u), ∀λ > 0, and then invoking Theorem 1.3.2, this implies that y+λ¯ x ∈ C, ∀λ > 0. Therefore, by Proposition 2.4.1, there exists some ray of direction x ¯, which is a boundary ray or an asymptote of C, in contradiction to the hypothesis of the theorem. We now prove the reverse statement. Suppose that σC is continuous, and there exist some ray y and a direction x ¯ such that the set D = {¯ x +λy| λ ≥ 0} is a boundary ray or an asymptote. In both cases, invoking Proposition 1.1.11, it can be separated from C by a hyperplane; i.e., there exist 0 = a ∈ Rn , b ∈ R such that (2.15) inf a, z ≥ b, sup a, x ≤ b. z∈D
x∈C
Let us prove that σC is not continuous at a. Since for z ∈ D one has a, z = a, x ¯ + λa, y, letting λ → +∞, it follows from the first inequality in (2.15) that a, y ≥ 0. Suppose that D is a boundary ray. Then from the second inequality in (2.15) it follows that λa, y ≤ b − a, x ¯, and letting λ → +∞, this implies a, y ≤ 0, so that we have proved that a, y = 0. The same holds if D is an asymptote. Indeed, by definition there exist
46
2. Asymptotic Cones and Functions
zk = x ¯ + λk y, with λk → +∞, xk ∈ C such that xk − zk → 0. Then using the second inequality in (2.15) we obtain ¯ + λk a, y + a, xk − zk ≤ b, a, xk = a, x and passing to the limit we obtain a, y ≤ 0, and hence a, y = 0. Now from the second inequality in (2.15) one has that σC (a) is finite. Let ε > 0 and let us prove that σC (a + εy) = +∞. This will prove that σC is not continuous at a. We consider two cases. If D is a boundary ray, then for all λ > 0, ¯ + λy = a, x ¯ + εy, x ¯ + ελy2 + λa, y, σC (a + εy) ≥ a + εy, x and letting λ → +∞, it follows that σC (a+εy) = +∞. If D is an asymptote, σC (a + εy) ≥ a + εy, xk = a + εy, x ¯ + a + εy, xk − zk + λk a + εy, y. Since a + εy, xk − zk → 0 and since λk a + εy, y = ελk y2 , it follows 2 that σC (a + εy) = +∞. As previously discussed in the beginning of Section 2.3, a key question in convex analysis is to know when the image of a closed convex set under a linear map remains closed. As demonstrated Example 2.3.1, where the set is continuous, the standard sufficient condition (2.2) fails, and in Theorem 2.3.2 we gave a necessary and sufficient condition for preserving closedness of the image of a closed convex set. However, thanks to this theorem, the next result shows that for continuous convex sets, the closedness property is guaranteed. Proposition 2.4.2 Let C ⊂ Rn be a nonempty closed convex and continuous set and let A : Rn → Rm be a linear map. Then A(C) is closed. Furthermore, if A is a surjective map, then A(C) is a continuous set. Proof. In order to prove that A(C) is closed we can suppose that aff A(C) = A(C). Otherwise, there is nothing to prove. Then there exists u ∈ bd(cl AC) with u ∈ ri A(C), and invoking the separation Proposition 1.1.11 there exists 0 = a ∈ Rn such that the hyperplane H = {x ∈ Rn | a, x = β} separates properly {u} and cl A(C), namely, one has a, u = β, a, z ≤ β, ∀z ∈ cl A(C),
inf
z∈cl A(C)
so that AT a, x ≤ β, ∀x ∈ C. Now take d and x satisfying AT a, d = β, AT a, x < β,
a, z < β,
2.5 Asymptotic Functions
47
and set z = 2d − x. In order to prove that A(C) is closed, let {yk } be a sequence converging to y. Then invoking Theorem 2.3.2 it is sufficient to prove only that there cannot exist any sequence {xk } satisfying (2.3). x = 0, Suppose the contrary, and let {xk } be such a sequence. Then since A¯ for each λ ≥ 0 one obtains AT a, z + λ¯ x
= =
2AT a, d − AT a, x + λAT a, x ¯ T 2β − A a, x > β,
showing that the ray ∆ = {z + λ¯ x : λ ≥ 0} does not meet C. Therefore, since x ¯ ∈ C∞ , it follows from Proposition 2.4.1 that C is not a continuous set, leading to the desired contradiction. To prove the last part of the proposition, assume now that the linear map is surjective. Since (ker(AT ))⊥ = A(Rn ) = Rm , it follows that ker(AT ) = {0}. Furthermore, from Proposition 1.3.3(e) we have σA(C) (z) = σC (AT z). Since AT z = 0, ∀z = 0, and since C is continuous, this implies from Definition 2.4.1 that σA(C) is continuous on Rm \ {0}, and hence A(C) is continuous. 2
Proposition 2.4.3 For any nonempty closed convex and unbounded set C ⊂ Rn , C ≡ Rn , the following statements are equivalent: (a) C is continuous. (b) dom σC \ {0} = int(dom σC ) = ∅. (c) arg sup σC (d) is a nonempty compact set for all d ∈ dom σC \ {0}. Proof. We first show that (a) =⇒ (b). If C is continuous, since C is assumed unbounded then dom σC = Rn and int dom σC ⊂ dom σC \ {0}. Furthermore, since C ≡ Rn , one has dom σC \ {0} = ∅. Now let 0 = d ∈ dom σC . Then d ∈ int dom σC . Indeed, in the contrary case there would exist a sequence {dk } converging to d and such that σC (dk ) = +∞, which is impossible, since σC (d) = limk→∞ σC (dk ). To prove the reverse implication (b) =⇒ (a), note that by Theorem 1.2.3, σC is continuous on dom σC \ {0}, hence everywhere on Rn \{0}, implying that the set C is continuous. Finally, by Proposition 1.2.18 one has arg sup σC (d) = ∂σC (d). But, by Proposition 1.2.16 the subdifferential of σC at d is nonempty and compact if and only if d ∈ int dom σC , and this proves the equivalence between (b) and (c). 2
2.5 Asymptotic Functions Building on the concept of asymptotic cone, we are now interested in understanding the behavior in the large of functions f : Rn → R ∪ {+∞}.
48
2. Asymptotic Cones and Functions
The epigraph plays a key role. Let F be a nonempty closed set in Rn+1 satisfying the property (x, µ) ∈ F =⇒ (x, µ ) ∈ F, ∀µ > µ.
(2.16)
Then clearly, there exists one and only one function g such that epi g = F , defined by g(x) := inf{µ | (x, µ) ∈ F }. Now let f : Rn → R ∪ {+∞} be proper. Then epi f is nonempty, and therefore (epi f )∞ is a nonempty closed cone in Rn+1 satisfying (2.16); i.e., (x, µ) ∈ (epi f )∞ =⇒ (x, µ ) ∈ (epi f )∞ , ∀µ > µ. Indeed, from the definition of the asymptotic cone given in Definition 2.1.2 and applied to the set epi f we have ∃(xk , µk ) ∈ epi f, ∃tk → ∞ : t−1 k (xk , µk ) → (x, µ).
Set µk := µk + (µ − µ)tk . Since µ > µ, we have µk > µk ≥ f (xk ) and t−1 k (xk , µk ) → (x, µ ), showing that (x, µ ) ∈ (epi f )∞ , ∀µ > µ. The above discussion leads us to introduce the following concept. Definition 2.5.1 For any proper function f : Rn → R ∪ {+∞}, there exists a unique function f∞ : Rn → R ∪ {+∞} associated with f , called the asymptotic function, such that epi f∞ = (epi f )∞ . This definition indicates that the epigraph of an asymptotic function is in fact a closed cone. This can be further elaborated through the use of lower semicontinuity and positively homogeneous functions; cf. Definition 1.3.2. The asymptotic function f∞ enjoys the following basic properties. Proposition 2.5.1 For any proper function f : Rn → R∪{+∞}, we have: (a) f∞ is lsc and positively homogeneous. (b) f∞ (0) = 0 or f∞ (0) = −∞. (c) If f∞ (0) = 0, then f∞ is proper. Proof. (a) Since by Definition (2.5.1) epi f∞ = (epi f )∞ and (epi f )∞ is a closed set by definition, it follows that f∞ is lsc. First, note that 0 ∈ dom f∞ . So, let x ∈ dom f∞ . Since epi f∞ is a cone, we have (x, f∞ (x)) ∈ epi f∞ =⇒ (λx, λf∞ (x)) ∈ epi f∞ , ∀λ > 0, i.e., f∞ (λx) ≤ λf∞ (x). Likewise, one has (λx, f∞ (λx)) ∈ epi f∞ , ∀x ∈ dom f∞ , ∀λ > 0, and hence (x, λ−1 f∞ (λx)) ∈ epi f∞ by the cone property. Therefore, λf∞ (x) ≤ f∞ (λx), and (a) is proved whenever x ∈ dom f . Finally, if x ∈ dom f∞ , then λx ∈ dom f∞ , ∀λ > 0, and hence f∞ (λx) = λf∞ (x) = +∞.
2.5 Asymptotic Functions
49
(b) Since f is proper, then epi f is nonempty, and hence either f∞ (0) is finite or f∞ (0) = −∞. If f∞ (0) is finite, then by (a), f∞ (0) = λf∞ (0), ∀λ > 0, and hence f∞ (0) = 0. (c) Suppose f∞ is not a proper function. Then there exists x such that f∞ (x) = −∞. Now let {λk } ⊂ R++ be a positive sequence converging to 0. Then λk x → 0, and under our assumption f∞ (0) = 0, the lower semicontinuity of f∞ , and property (a), we obtain 0 = f∞ (0) ≤ lim inf f∞ (λk x) = lim inf λk f∞ (x) = −∞, k→∞
k→∞
2
leading to a contradiction.
We can now give a fundamental analytic representation of the asymptotic function f∞ . Theorem 2.5.1 For any proper function f : Rn → R ∪ {+∞} the asymptotic function f∞ is given by f∞ (d) = lim inf d →d
t→+∞
f (td ) , t
(2.17)
or equivalently f (tk dk ) f∞ (d) = inf lim inf | tk → +∞, dk → d , k→∞ tk
(2.18)
where {tk } and {dk } are sequences in R and Rn , respectively. Proof. The equivalence of the two formulas above is clear. Let g(d) denote the right-hand side of (2.18). We will first show that (epi f )∞ ⊂ epi g. Let (d, µ) ∈ (epi f )∞ . Then by Definition 2.1.2, ∃tk → ∞, (dk , µk ) ∈ epi f such that t−1 k (dk , µk ) → (d, µ). As a consequence, since f (dk ) ≤ µk can −1 −1 be written as t−1 k f (tk dk · tk ) ≤ tk µk , passing to the limit, it follows that g(d) ≤ µ and hence (d, µ) ∈ epi g. Conversely, let (d, µ) ∈ epi g. By definition of g, there exist sequences {dk } ⊂ Rn and {tk } ∈ R such that g(d) = lim
k→∞
f (tk dk ) , tk → ∞, dk → d, tk
and since (d, µ) ∈ epi g, it follows from the definition of the limit that ∀ε > 0 and all k ∈ N sufficiently large, we have f (tk dk ) ≤ (µ + ε)tk and hence zk := tk (dk , µ + ε) ∈ epi f . Since t−1 k zk → (d, µ + ε), it follows that (d, µ + ε) ∈ (epi f )∞ , and therefore since (epi f )∞ is a closed set and ε > 0 2 was arbitrary, we also have (d, µ) ∈ (epi f )∞ . Corollary 2.5.1 For a nonempty set C ⊂ Rn one has (δC )∞ = δC∞ .
50
2. Asymptotic Cones and Functions
Proof. This is an immediate consequence of the definition of the asymptotic cone and the previous theorem. Corollary 2.5.2 Let f : Rn → R ∪ {+∞} be a proper function such that dom f ∗ is nonempty. Then f∞ (0) = 0. Proof. Indeed, in the contrary case f∞ (0) = −∞, but by hypothesis there exist v ∈ dom f ∗ , α ∈ R such that f (x) ≥ v, x − α
∀x.
−1 Then from the above inequality we get t−1 k f (tk dk ) ≥ v, dk −αtk . Passing to the limit with dk → 0, tk → ∞, it follows from Theorem 2.5.1 that f∞ (0) ≥ 0, which is impossible. When f is a convex function, there are simpler ways to express the asymptotic function, which is also called in this case the recession function. We will keep the terminology asymptotic function in both the convex and nonconvex cases.
Proposition 2.5.2 Let f : Rn → R ∪ {+∞} be a proper, lsc, convex function. The asymptotic function is a positively homogeneous, lsc, proper convex function, and for any d ∈ Rn , one has f∞ (d) = sup{f (x + d) − f (x) | x ∈ dom f }
(2.19)
and f (x + td) − f (x) f (x + td) − f (x) = sup , ∀x ∈ dom f. t→+∞ t t t>0 (2.20)
f∞ (d) = lim
Proof. By Proposition 2.5.1(a), the asymptotic function f∞ is lsc and positively homogeneous, while the convexity of f∞ follows from the convexity of f . Since f∞ is lsc, that f∞ is proper follows from (2.19), which we now prove. By definition, the asymptotic function f∞ associated with f is uniquely determined via epi f∞ = (epi f )∞ . Using Proposition 2.1.5 via the representation F of the asymptotic cone, one has (d, µ) ∈ (epi f )∞ if and only if for all (x, α) ∈ epi f it holds that (x, α) + (d, µ) ∈ epi f , namely if and only if f (x + d) ≤ α + µ. The latter inequality is clearly equivalent to f (x + d) − f (x) ≤ µ, ∀x ∈ dom f , proving formula (2.19). To verify (2.20), let x ∈ dom f . Then again using Proposition 2.1.5 but via the representation D of the asymptotic cone, one has for any x ∈ dom f , (epi f )∞ = {(d, µ) ∈ Rn × R| (x, f (x)) + t(d, µ) ∈ epi f, ∀t > 0}, and hence (d, µ) ∈ (epi f )∞ if and only if for any x ∈ dom f we have f (x + td) ≤ f (x) + tµ, ∀t > 0,
2.5 Asymptotic Functions
51
which means exactly that g(d) := sup t>0
f (x + td) − f (x) ≤ µ, t
and hence (epi f )∞ = epi g, ∀x ∈ dom f , proving the formula (2.20). That the limit in t in the first part of formula (2.20) coincides with the supremum in t > 0 simply follows by recalling that the convexity of f implies that for fixed x, d ∈ Rn , for any τ > 0, the function τ → (τ )−1 (f (x + τ d) − f (x)) is nondecreasing. 2 The next result gives a useful simplification of the formula for f∞ . Corollary 2.5.3 For any lsc, proper, and convex function f : Rn → R ∪ {+∞}, one has f∞ (d) = lim+ tf (t−1 d), ∀d ∈ dom f. t→0
If 0 ∈ dom f , the formula holds for every d ∈ Rn . Proof. When 0 ∈ dom f one has f (0) < ∞, and the formula above is immediate from (2.20). Consider now the case where 0 ∈ dom f . Using (2.20) with x := d ∈ dom f , one has for any d ∈ dom f , f∞ (d) = lim t−1 (f ((1 + t)d) − f (d)), t→+∞
from which the desired formula for f∞ follows.
2
Example 2.5.1 (Some useful asymptotic functions.) (a) Let Q be a sym1 metric n × n positive semidefinite matrix and f (x) := (1 + x, Qx) 2 . Then 1 f∞ (d) = d, Qd 2 . (b) For the quadratic function associated with Q positive semidefinite and given by f (x) := 12 x, Qx + c, x + s, c ∈ Rn , s ∈ R, one has c, d if Qd = 0, f∞ (d) = +∞ if Qd = 0. For a nonconvex quadratic function, that is, with Q not positive semidefinite, one needs to apply (2.17), and in that case one obtains −∞ for d with dT Qd ≤ 0, f∞ (d) = +∞ for d with dT Qd > 0. Thus, the asymptotic function of a proper function, and in that case even of a finite function, n can be improper. (c) Let f (x) = j=1 exj . Then f∞ (d) = δRn− . n (d) Let f (x) = log j=1 exj , n > 1. Then f∞ (d) = max1≤j≤n dj .
52
2. Asymptotic Cones and Functions
Further examples of asymptotic functions will be given in Sections 2.6–2.7. Proposition 2.5.3 For any proper function f : Rn → R ∪ {+∞} and any α ∈ R such that lev(f, α) = ∅ one has (lev(f, α))∞ ⊂ lev(f∞ , α), i.e., {x | f (x) ≤ α}∞ ⊂ {d | f∞ (d) ≤ 0}. Equality holds in the inclusion when f is lsc, proper, and convex. Proof. Since lev(f, α) = ∅, take d ∈ (lev(f, α))∞ . We have to show that f∞ (d) ≤ 0, ∀d. By definition, d ∈ (lev(f, α))∞ means that ∃xk ∈ lev(f, α), ∃tk → +∞ : lim t−1 k xk = d. k→∞
−1 Set dk = t−1 k xk . Then dk → d, and since xk ∈ lev(f, α), we have tk f (tk dk ) = −1 −1 tk f (xk ) ≤ tk α → 0, which implies by the fundamental formula (2.18) of an asymptotic function that f∞ (d) ≤ 0, ∀d. In the case that f is as-
sumed proper lsc convex, to prove the reverse inclusion, let d be such that f∞ (d) ≤ 0. Then for each x ∈ lev(f, α) and λ > 0 one has f (x+λd)−f (x) ≤ λf∞ (d) ≤ 0, so that x + λd ∈ lev(f, α), which by Proposition 2.1.5 means that d ∈ (lev(f, α))∞ . 2 Corollary 2.5.4 Let fi : Rn → R ∪ {+∞}, i ∈ I a collection of proper functions and, let S ⊂ Rn with S = ∅. Define C := {x ∈ S | fi (x) ≤ 0, ∀i ∈ I}. Then C∞ ⊂ {d ∈ S∞ | (fi )∞ (d) ≤ 0, ∀i ∈ I}. The inclusion holds as an equation when C = ∅, S is closed convex in Rn , and each fi is lsc convex. Proof. Let Ci := {x ∈ Rn : fi (x) ≤ 0}, so that C = S ∩ ∩i∈I Ci . Invoking Proposition 2.1.9, one has C∞ ⊂ S∞ ∩i∈I (Ci )∞ , and thus using Proposition 2.5.3, the inclusion statement of the theorem is proved. Furthermore, under the additional assumptions given here, by the same Proposition 2.1.9, the previous inclusion holds as an equation. 2 Example 2.5.2 (a) Let Q be an n × n symmetric positive semidefinite matrix, a ∈ Rn , and β ∈ R. Let C be the set defined by C = {x ∈ Rn | (x − a)T Q(x − a) ≤ β}. Then using the examples given in (2.5.1) one has C∞ = {d ∈ Rn | dT Qa ≤ 0, Qd = 0}.
2.5 Asymptotic Functions
53
(b) Let f : R++ → R defined by f (t) = t sin t−1 and consider the set Gr(f ) = {(x1 , x2 ) ∈ R2 | x2 = f (x1 ), x1 > 0}. Then (Gr(f ))∞ = {(d1 , d2 ) ∈ R2 | 0 ≤ d2 ≤ d1 }. The next result gives a useful relation between the asymptotic cone of the subdifferential of a convex function and the normal cone of its domain. Proposition 2.5.4 Let f : Rn → R ∪ {+∞} be proper and convex. Then for any z ∈ dom f such that ∂f (z) = ∅ one has (∂f (z))∞ = Ndom f (z). Proof. Let z ∈ dom f be such that ∂f (z) = ∅. Then the subdifferential of f is a nonempty closed convex set that can be described as the infinite intersection of closed half-spaces. More precisely, one has {u ∈ Rn | u, y − z ≤ f (y) − f (z)} ≡ Cy . ∂f (z) = y∈dom f
y∈dom f
Applying Proposition 2.1.9(a) to the closed convex sets Cy one obtains (∂f (z))∞ =
(Cy )∞ .
y∈dom f
But by Corollary 2.5.4 one obtains immediately that (Cy )∞ = {u | u, y − z ≤ 0}, and thus it follows that (∂f (z))∞ = {u | u, y − z ≤ 0} = Ndom f (z), y∈dom f
where the last equality follows from the definition of the normal cone of the nonempty convex set dom f at z. 2
Proposition 2.5.5 Let f : Rn → R ∪ {+∞} be lsc, proper, and convex. Then f is Lipschitz on Rn , i.e., |f (x) − f (y)| ≤ Lx − y, ∀x, y ∈ Rn , if and only if f∞ is finite everywhere on Rn . The Lipschitz constant L is given by L = sup{f∞ (d) | d = 1}. Proof. The asymptotic function f∞ , being finite everywhere on Rn , is continuous, and therefore sup{f∞ (d) | d = 1} = L < ∞. Since f∞ is
54
2. Asymptotic Cones and Functions
positively homogeneous, f∞ (d) ≤ Ld, ∀d. Invoking the formula (2.19) one thus has f (x + d) − f (x) ≤ Ld, ∀x ∈ dom f, d ∈ Rn , and since f (x + d) < ∞, then dom f = Rn and L is the Lipschitz constant for f . Conversely, let f be Lipschitz on Rn and such that x ∈ Rn . Then f (x + td) − f (x) ≤ tLd, ∀t > 0, d ∈ Rn , which implies by (2.20) that f∞ (d) ≤ Ld, ∀d, so that dom f∞ = Rn and 2 sup{f∞ (d) | d = 1} = L. To prove the next result, we need the following obvious technical fact on one-dimensional convex functions, the proof of which is left to the reader. Lemma 2.5.1 Let ψ : R → R∪{+∞} be a proper, lsc, convex function and let S denote the set of optimal solutions of the problem inf{ψ(t) : t ∈ R}. Then only one of the following two statements holds: (a) S is nonempty and compact. (b) The function ψ is a monotone function on R. Theorem 2.5.2 Let f : Rn → R∪{+∞} be a proper, lsc, convex function, and for any x ∈ dom f and 0 = d ∈ Rn let ψ(t) := f (x + td). (a) If f∞ (d) ≤ 0, then lim supt→+∞ ψ(t) < +∞. (b) If there exists some z ∈ dom f such that lim inf t→+∞ f (z + td) < +∞, then ψ is decreasing on R, which is equivalent to saying that f∞ (d) ≤ 0. Proof. From (2.20) we have t−1 (f (x + td) − f (x)) ≤ f∞ (d) ≤ 0, which implies that f (x + td) ≤ f (x), ∀t > 0, proving (a). To prove (b), let z ∈ dom f satisfy the hypothesis of the theorem. Then one has f∞ (d) ≤ 0. Indeed, suppose the contrary. Then again using (2.20) there would exist t0 > 0 and α > 0 such that f (z + td) ≥ f (z) + tα, ∀t ≥ t0 . Taking the limit as t → +∞ in the latter inequality implies that limt→+∞ f (z + td) = +∞, which contradicts the assumption made in (b). Since for each x ∈ dom f , one has ψ∞ (1) = f∞ (d) ≤ 0, invoking Lemma 2.5.1, it follows that the optimal solution set of the problem minimizing ψ is noncompact, and thus ψ is monotone. We can suppose that ψ is nonconstant. Then ψ is decreasing. Indeed, in the opposite case there would exist t0 ∈ dom ψ such that ψ(t) ≤ ψ(t0 ), ∀t ≤ t0 , and then it would follow that ψ∞ (−1) ≤ 0. Since ψ∞ (0) = 0, it follows from the convexity of ψ∞ that ψ∞ (−1) = ψ∞ (1) = 0 and then that ψ is constant. Finally, if ψ is decreasing on R, then f∞ (d) ≤ 0 follows from (2.20). 2
2.5 Asymptotic Functions
55
Definition 2.5.2 Given f : Rn → R ∪ {+∞} a proper convex function, we define: (a) The asymptotic cone of f by Kf = {d ∈ Rn | f∞ (d) ≤ 0} = (epi f )∞ ∩ {(d, 0) | d ∈ Rn }. This is a closed convex cone containing zero. A vector d ∈ Kf is called an asymptotic direction of f . (b) The constancy space of f by Cf = {d ∈ Rn | f∞ (d) = f∞ (−d) = 0} = (−Kf ) ∩ Kf . (c) The lineality space of f by Lf = {d ∈ Rn | f∞ (−d) = −f∞ (d)}. Theorem 2.5.3 Let f : Rn → R∪{+∞} be a proper, lsc, convex function. The following conditions on d ∈ Rn and µ ∈ R are equivalent: (a) f (x + td) = f (x) + tµ, ∀x ∈ dom f, t ∈ R. (b) (d, µ) ∈ −(epi f )∞ ∩ (epi f )∞ . (c) −f∞ (−d) = f∞ (d) = µ. (d) Furthermore, the vector d satisfies (a)–(c) with µ = f∞ (d) if there exists one x ∈ dom f such that f (x + td) is an affine function of t. (e) d ∈ Cf if and only if f is constant along the direction d. Proof. Suppose (a) holds, i.e., f (x + td) = f (x) + tµ, ∀x ∈ dom f, ∀t ∈ R. Thus in particular, using formula (2.19) one has µ = f∞ (d) and −µ = f∞ (−d), proving the implication (a) =⇒ (c). Now if (c) holds, this means that both (d, µ) and (−d, −µ) are in epi f∞ , and hence (b) holds. We end the proof by showing that (b) =⇒ (a). If (b) holds, then for any t ∈ R one has epi f = epi f − t(µ, d). Take x ∈ dom f . Then (x, f (x)) ∈ epi f , and if we define h(x) = f (x + td) − tµ, then epi h = epi f − t(µ, d), and hence (a) holds. Furthermore, (d) follows from the formula (2.20), and finally, since f∞ (0) = 0 ≤ f∞ (d) + f∞ (−d), it follows that f∞ (d) = f∞ (−d) = 0 if and only if d ∈ Cf , i.e., if and only if f is constant on the direction d by using the equivalence between (a) and (c). 2 There exists an interesting interplay between a proper, lsc, convex function and the convex sets associated with it, together with its conjugate and the corresponding asymptotic function. Theorem 2.5.4 Let f : Rn → R ∪ {+∞} be a proper convex function, and f ∗ its conjugate. The following relations hold: (a) (f ∗ )∞ = σdom f . (b) If f is also assumed lsc, then f∞ = σdom f ∗ , (f∞ )∗ = δcl dom f ∗ .
56
2. Asymptotic Cones and Functions
Proof. (a) Using the definition of f∞ given by the formula (2.19) and that of the conjugate function, one has (f ∗ )∞ (d)
= = = =
sup{f ∗ (x + d) − f ∗ (x) : x ∈ dom f ∗ } sup
{ sup {u, x + d − f (u)} − f ∗ (x)}}
x∈dom f ∗ u∈dom f
sup {
sup
u∈dom f x∈dom f ∗
{u, x − f ∗ (x)} + u, d − f (u)}}
sup {u, d + f ∗∗ (u) − f (u)}, u∈dom f
≤ σdom f (d), since f ∗∗ ≤ f. We now prove the reverse inequality (f ∗ )∞ (d) ≥ σdom f (d). Clearly, for any d ∈ (dom f ∗ )∞ there is nothing to prove. Thus, let any d ∈ (dom f ∗ )∞ and take µ = (f ∗ )∞ (d). Then one has f ∗ (z + td) − f ∗ (z) ≤ µt, ∀z ∈ dom f ∗ , ∀t > 0. On the other hand, by the definition of the conjugate, one has f (x) + f ∗ (z + td) ≥ x, z + tx, d, ∀t > 0, which combined with the inequality above implies f (x) ≥ x, z − f ∗ (z) + t(x, d − µ), ∀t > 0. Therefore, as t → +∞ it follows that x, d ≤ µ, ∀x ∈ dom f , and hence sup x, d ≤ µ and thus σdom f (d) ≤ (f ∗ )∞ (d).
x∈dom f
(b) We now assume that f is lsc, which implies that f ∗∗ = f . Then replacing f by f ∗ in the formula proven in (a) yields (f ∗∗ )∞ = σdom f ∗ = f∞ , proving the first formula of (b). Taking conjugates in that formula, and recalling ∗ we obtain that for any set C ⊂ Rn , σC = σcl C and σC = δC ∗∗ (f∞ )∗ = (σcl dom f ∗ )∗ = δcl dom f ∗ = δcl dom f ∗ ,
proving the second formula in (b).
2
Corollary 2.5.5 Let f : Rn → R ∪ {+∞} be a proper convex function and suppose that 0 ∈ E := aff dom f , i.e., E is a subspace. Then Cf ∗ = Lf ∗ = E⊥. Proof. From the definition of the lineality space and from Theorem 2.5.3 one has d ∈ Lf ∗ if and only if the function ·, d is constant on dom f , or equivalently on E = aff dom f , and since 0 ∈ E, this constant is 0, and one 2 has Lf ∗ = Cf ∗ and d ∈ E ⊥ if and only if d ∈ Cf ∗ .
2.5 Asymptotic Functions
57
The characterization of the asymptotic function of a convex function given in terms of the support function of its conjugate in the previous theorem is of fundamental importance, and as we shall see, will be used very often in the following chapters. An interesting immediate application of this formula is the following dual characterization of the constancy subspace Cf = {d ∈ Rn | f∞ (d) = f∞ (−d) = 0}, which by replacing f∞ by σdom f ∗ (when f is lsc) can thus be simply written as Cf = {d ∈ Rn | d, v = 0, ∀v ∈ dom f ∗ }.
(2.21)
Lemma 2.5.2 Let f : Rn → R ∪ {+∞} be a proper convex function. Then for all y ∈ Rn , f ∗ (y) = σepi f (y, −1) = sup{y, x − r | (x, r) ∈ epi f }. As a consequence the following formula holds ∗ y tf ( t ) (f ∗ )∞ (y) σepi f (y, −t) = +∞
for the epigraph of f : if t > 0, if t = 0, if t < 0.
Proof. The first formula in the theorem follows from the definition of the epigraph and conjugate of f . In fact, one has σepi f (y, −1) = sup {x, y − r} = f (x)≤r
=
sup sup {x, y − r} x r≥f (x)
sup{x, y − f (x)} = f ∗ (y). x
We now prove the second formula. First, if t < 0, one has σepi f (y, −t) = +∞. Suppose that t > 0. Then using the first formula just proved, we have σepi f (y, −t) = t sup {x, y/t − r} = tσepi f (y/t, −1) = tf ∗ (y/t). f (x)≤r
Finally, if t = 0, then σepi f (y, 0)
=
sup {x, y − r × 0}, f (x)≤r
=
sup{x, y | (x, r) ∈ epi f for some r},
= σdom f (y) = (f ∗ )∞ (y), where the last equation follows from Theorem 2.5.4(a).
2
58
2. Asymptotic Cones and Functions
For each function f : Rn → R ∪ {+∞} that is lsc and proper with 0 < f (0) < +∞, the positive hull, denoted by pos f , is defined through its epigraph via epi(pos f ) = pos(epi f ), which is equivalent to (since f (0) > 0, i.e., (0, 0) ∈ epi f ) (pos f )(x) = inf λf (λ−1 x). λ>0
(2.22)
Proposition 2.5.6 Let f : Rn → R ∪ {+∞} be lsc, and proper, with 0 < f (0) < +∞. Then: (a) cl(pos f )(x) = min{(pos f )(x), f∞ (x)} = min{ inf λf (λ−1 x), f∞ (x)}. λ>0
(b) If in addition f is assumed convex, then pos f is lsc, convex, proper, and positively homogeneous. Proof. (a) By the definition of the closure and the positive hull of a function, one has epi(cl(pos f )) = cl(epi(pos f )) = cl(pos(epi f )). Using Corollary 2.3.3 one thus obtains cl(pos(epi f ))
= pos(epi f ) ∪ (epi f )∞ = epi(pos f ) ∪ epi f∞ = epi(min{pos f, f∞ }) (using Proposition 1.2.1),
which establishes the first formula in (a), while the second formula follows from (2.22). (b) Since f is convex and 0 ∈ dom f , one has f∞ (x) = limλ→0+ λf (λ−1 x). Furthermore, since λf (λ−1 x) ≤ f∞ (x) − λf (0) and f (0) > 0, it follows that pos f (x) = inf λ>0 λf (λ−1 x) ≤ f∞ (x), and by part (a) we obtain cl pos f = pos f , which proves that pos f is lsc. Since epi f is convex, it follows that pos(epi f ) is convex, and then pos f is convex. Furthermore, since (pos f )(0)=inf λ>0 λf (0) = 0, one thus has that pos f , which is lsc, is also proper. Finally, since epi(pos f ) is a cone, one thus has that pos f is positively homogeneous. 2
Remark 2.5.1 In the convex case, from part (b) of Proposition 2.5.6 we always have inf λ>0 λf (λ−1 d) ≤ f∞ (d). This inequality can be strict. For example, with f (x) := x2 + 1 we obtain for d = 1, inf λ>0 λf (λ−1 ) < +∞ and f∞ (1) = +∞. When f is lsc, proper, and convex, the corresponding asymptotic function can be viewed as the closure of another function associated with f .
2.5 Asymptotic Functions
59
Definition 2.5.3 For any proper function f : Rn → R ∪ {+∞}, we associate the function pf defined by tf (t−1 x) if t > 0, 0 if t = 0, x = 0, p(t, x) = (2.23) +∞ otherwise, with dom p = {(0, 0} ∪ {t(1, x) | t > 0, f (x) < ∞} = {(0, 0)} ∪ R+ ({1} × dom f ). Proposition 2.5.7 Let f : Rn → R ∪ {+∞} be lsc, proper, and convex. Then the function p(t, x) given in (2.23) is proper and jointly convex in (t, x), and for any d ∈ Rn one has tf (t−1 x) if t > 0, if t = 0, f∞ (d) (clp)(t, d) = σepi f ∗ (y, −t) = +∞ if t < 0. Proof. The joint convexity of p(t, x) follows directly from applying the definition of the convexity of f by observing that for any α ∈ [0, 1], β = 1 − α, and any t1 , t2 > 0, x1 , x2 ∈ Rn one has αt1 x1 βt2 x2 + p(αt1 + βt2 , αx1 + βx2 ) = (αt1 + βt2 )f (αt1 + βt2 )t1 (αt1 + βt2 )t2 x1 x2 ≤ αt1 f + βt2 f . t1 t2 Now define the function g(s, x) =
f (x) if s = 1, +∞ if s = 1.
Clearly, g is a proper lsc convex function, and since for any x, g(0, x) = +∞, using (2.22) one has sf (s−1 x) if s > 0, −1 −1 +∞ if s = 0, x = 0, (pos g)(s, x) = inf λg(λ s, λ x) = λ>0 +∞ if s < 0. The point (0, 0) is not in the relative interior of p and pos g, respectively, and therefore the functions p and pos g coincide on their relative interiors, and hence their closures coincide. Applying Proposition 2.5.6(a) one thus obtains cl p(t, x) = cl(pos g)(t, x) = min{(pos g)(t, x), g∞ (t, x)}, and since here g∞ (t, d) = f∞ (d) for t = 0 and is +∞ whenever t = 0, the desired formula for cl p(t, x) follows using Lemma 2.5.2. 2
60
2. Asymptotic Cones and Functions
Corollary 2.5.6 Let C ⊂ Rn be a nonempty closed convex set with 0 ∈ C. Then the gauge function of C, denoted by γC and defined by γC = inf{λ ≥ 0 | x ∈ λC} is nonnegative, lsc, and positively homogeneous. Furthermore, one has: (a) C = {x | γC (x) ≤ 1}. (b) C∞ = {x | γC (x) = 0}. (c) pos C = dom γC . Proof. Let h(x) := δ(x|C) + 1. Then 1 = h(0) > 0 and λh(λ−1 x) = δ(x|λC) + λ, ∀λ > 0. Therefore, by definition one has γC (x) = pos h(x) and γC ≥ 0. Hence, by Proposition 2.5.6(b) it follows that γC is lsc, convex, proper, and positively homogeneous. Moreover, (a) is satisfied, and then C∞ = {d | (γC )∞ (d) ≤ 0}. Since γC is positively homogeneous, lsc convex, we have (γC )∞ = γC , and since γC ≥ 0, we obtain formula (b). Finally, using (a) and once more the fact that γC is positively homogeneous, we 2 obtain pos C = dom γC . Domains of proper, lsc, convex functions can be usefully characterized via the use of asymptotic functions. Proposition 2.5.8 Let h : Rn → R ∪ {+∞} be lsc, proper, and convex. Let z be a fixed vector and let f (x) = h(x) − x, z. Then: (a) z ∈ aff(dom h∗ ) ⇐⇒ f∞ (d) = 0, ∀d such that −f∞ (−d) = f∞ (d). (b) z ∈ ri(dom h∗ ) ⇐⇒ f∞ (d) > 0, ∀d except those satisfying −f∞ (−d) = f∞ (d) = 0. (c) z ∈ int(dom h∗ ) ⇐⇒ f∞ (d) > 0, ∀d = 0. (d) z ∈ cl(dom h∗ ) ⇐⇒ f∞ (d) ≥ 0, ∀d. Proof. We have f ∗ (y) = supx {x, y + z − h(x)} = h∗ (y + z), so that dom f ∗ = dom h∗ − z. Thus, z ∈ cl dom h∗ is equivalent to 0 ∈ cl dom f ∗ , and similarly for each of the other statements in the theorem. By Theorem 2.5.4(b), one has σdom f ∗ = f∞ . Therefore, statements (a)–(d) correspond exactly to those of Theorem 1.3.2 stated for the support function of dom f ∗ . 2
2.6 Differential Calculus at Infinity We develop here some useful formulas for computing asymptotic functions. n Proposition 2.6.1 Let fi : R → R ∪ {+∞}, i = 1, . . . , p, be a collecp tion of proper functions, f := i=1 fi , and suppose that f is proper; i.e.,
2.6 Differential Calculus at Infinity
61
dom f = ∩pi=1 dom fi = ∅. Then: (a) If the functions are all lsc, f is also lsc. p (b) f∞ (d) ≥ i=1 (fi )∞ (d) for all d satisfying the following condition: if (fi )∞ (d) = +∞ (respectively −∞) for some i, then (fj )∞ (d) > −∞ respectively (< +∞) for j = i. If in addition all the functions are lsc and convex, then equality holds in the inequality. Proof. For arbitrary extended real valued functions gi , one has for y ∈ Rn , lim inf (g1 (x) + · · · + gp (x)) ≥ lim inf g1 (x) + · · · + lim inf gp (x), x→y
x→y
x→y
as long as the right side of the inequality has a meaning, and hence (a) follows immediately by using the definition of lower semicontinuity. The inequality of assertion (b) follows also from the above inequality by using the definition of the asymptotic function (2.18), while equality in the case of lsc convex functions follows from formula (2.20). 2
Proposition 2.6.2 For a collection {fi }i∈I of proper functions fi : Rn → R ∪ {+∞} one has ≥ sup(fi )∞ and inf fi ≤ inf (fi )∞ . sup fi i∈I
∞
i∈I
i∈I
∞
i∈I
The first relation is an equality when fi are lsc, proper, and convex. The second relation is an equality for a finite index set I. Proof. Apply Proposition 2.1.9 to the corresponding epigraphs, recalling the epigraph algebraic properties of Proposition 1.2.1. 2
Proposition 2.6.3 Let g : Rm → R ∪ {+∞} be a proper function, let A be a linear map from Rn to Rm with A(Rn ) ∩ dom g = ∅, and let f (x) = g(Ax) be the proper composite function. The following properties hold for the function f : (a) If g is convex and there exists some y such that Ay ∈ ri dom g, then for any x, cl f (x) = cl g(Ax). (b) If g is lsc, then f is lsc and one has f∞ (d) ≥ g∞ (Ad)
∀d.
If in addition g is convex (so is f ), this holds also as an equality. Proof. (a) Since dom f = A−1 dom g, using Proposition 1.1.6(c) we get ri dom f = A−1 (ri dom g), which is equivalent to saying that y ∈ ri dom f if
62
2. Asymptotic Cones and Functions
and only if Ay ∈ ri dom g. Let y be such that Ay ∈ ri dom g (which exists by hypothesis). Then, since y ∈ ri dom f , by Proposition 1.2.6 we have cl f (x) = lim+ f (x + t(y − x)) = lim+ g(Ax + t(Ay − Ax)). t→0
t→0
By the same proposition, we also have cl g(Ax) = lim+ g(Ax + t(Ay − Ax)), t→0
so that cl f (x) = cl g(Ax). (b) If g is lsc, obviously f is also lsc. Furthermore, using the representation of the asymptotic function we obtain for each d f∞ (d) =
inf
dk →d tk →+∞
−1 lim inf t−1 g(td ) = g∞ (Ad). k g(tk Adk ) ≥ lim inf t k→∞
d →Ad
t→∞
If in addition g is convex, then obviously f is also convex. Let x0 ∈ dom f . Then since f is lsc, proper, and convex, we have f∞ (d) = lim
λ→∞
g(Ax0 + λAd) − g(Axo ) = g∞ (Ad). λ 2
It is interesting and often useful to have a formula for the asymptotic function defined as a composition of a nondecreasing function with a convex function. Proposition 2.6.4 (Composition rule) Let f : Rn → R ∪ {+∞} be lsc, proper, and convex, and let ψ : (−∞, b) → R with 0 ≤ b ≤ +∞ be a convex nondecreasing function with ψ∞ (1) > 0, and with dom ψ ∩ f (Rn ) = ∅. Consider the composite function ψ(f (x)) if x ∈ dom f, g(x) = +∞ otherwise. Then g is a proper, lsc, convex function, and one has ψ∞ (f∞ (d)) if d ∈ dom f∞ , g∞ (d) = +∞ otherwise. Proof. Clearly, the composite function g is an lsc convex function. Let x ∈ dom f be such that f (x) ∈ dom ψ. For every s < f∞ (d) there exists τ such that f (x + td) ≥ f (x) + ts, for t ≥ τ , and since ψ is nondecreasing, we obtain g(x + td) − g(x) ψ(f (x) + ts) − ψ(f (x)) ≥ , t t
2.6 Differential Calculus at Infinity
63
and passing to the limit with t → +∞ we deduce g∞ (d) ≥ ψ∞ (s). If f∞ (d) = +∞, using ψ(1) > 0 and letting s → +∞ we get g∞ (d) = +∞. In the other case, f∞ (d) < +∞, letting s → f∞ (d) and since ψ∞ is lsc, we deduce g∞ (d) ≥ ψ∞ (f∞ (d)). On the other hand, since f (x + td) ≤ f (x) + tf∞ (d), using the monotonicity of ψ we also get g∞ (d)
g(x + td) − g(x) t ψ(f (x) + tf∞ (d)) − ψ(f (x)) = ψ∞ (f∞ (d)). ≤ lim t→+∞ t =
lim
t→+∞
2 The next set of results concerns arbitrary proper functions. Proposition 2.6.5 Let fi : Rn → R ∪ {+∞} be lsc proper functions, i = 1, . . . , m, such that closed sets {epi fi | i = 1, . . . , m} are supposed to be in relative general position (see Definition 2.3.3) and let h := conv{fi | i = 1, . . . , m}. Then for all x ∈ dom cl h, ∃xi ∈ dom fi , yj ∈ dom(fj )∞ , λ ∈ int ∆p , i ∈ [1, p], j ∈ [1, q], with p + q ≤ n + 1, p ≥ 1, such that cl h(x) =
p i=1
with x =
p i=1
λi xi +
q j=1
λi fi (x) +
q
(fj )∞ (yj ),
j=1
yj .
Proof. Apply Theorem 2.3.5 to the corresponding epigraphs.
2
As in the case for sets where from Theorem 2.3.5 we have obtained an interesting corollary concerning the convex hull of a set and its closure we will now show how the convex hull of a function and its closure are related. For that purpose we need the following definition. Definition 2.6.1 Let f : Rn → R ∪ {+∞} be a proper function. Then f is said to be semibounded if its epigraph is semibounded, i.e., if epi f∞ is pointed. Proposition 2.6.6 Let f : Rn → R ∪ {+∞} be a proper, lsc, semibounded function. Then for all x ∈ dom(cl conv f ), ∃xi ∈ dom f, yj ∈ dom(f )∞ , λ ∈ int ∆p , i ∈ [1, p], j ∈ [1, q],
64
2. Asymptotic Cones and Functions
with p + q ≤ n + 1, p ≥ 1 such that cl conv f (x) =
p
λi f (xi ) +
i=1
with x =
p i=1
λi xi +
q j=1
q
f∞ (yj ),
j=1
yj .
Proof. Use Proposition 2.6.5 with m = n + 1, f = fi ∀i = 1, . . . , p.
2
An example of a function where Proposition 2.6.6 becomes simpler concerns cofinite functions. Definition 2.6.2 Let f : Rn → R ∪ {+∞} be a proper lsc function. Then f is said to be cofinite if f∞ (0) = 0, and f∞ (y) = +∞
∀y = 0.
Corollary 2.6.1 Let f be a cofinite function. Then cl conv f = conv f and ∀x ∈ dom(conv f ), ∃xi ∈ dom f, λ ∈ ∆n+1 , i ∈ [1, n + 1], such that conv f (x) = with x =
n+1 i=1
n+1
λi f (xi ),
i=1
λi xi .
Proof. As an immediate consequence of Corollary 2.3.6, since epi f∞ = {0}, we obtain cl conv f = conv f . From the definition of a cofinite function it follows that f is semibounded, and the claimed formula is obtained by using Proposition 2.6.6. 2 Proposition 2.6.7 Let f : Rn → R∪{+∞} be a proper lsc function. Then f is cofinite if and only if dom f ∗ = Rn . Proof. Suppose that f is cofinite and there exists some v ∈ / dom f ∗ , i.e., f ∗ (v) = +∞. Then we have inf{f (x) − v, x |x ∈ Rn } = −∞, and there exists a minimizing sequence {xk } such that f (xk ) − v, xk → −∞. Let j be an arbitrary integer. Then for k sufficiently large we have f (xk ) − v, xk ≤ −j.
(2.24)
Thus there are two cases: (i) The sequence {xk } is bounded. Therefore, it has at least one cluster
2.6 Differential Calculus at Infinity
65
point, and for each cluster point x, since f is lsc, passing to the limit in the last inequality it follows that f (x) − v, x ≤ −j, and taking j → ∞ we get a contradiction. (ii) Suppose now that {xk } is unbounded and without loss of generality suppose that xk xk → ∞, → y = 0. xk Then dividing both members of formula (2.24) by xk we obtain xk −1 xk f xk − v, xk xk −1 ≤ −jxk −1 , xk and passing to the limit we obtain f∞ (y) − v, y ≤ 0, a contradiction to the fact that f is cofinite. We prove now the reverse statement. Suppose that dom f ∗ = Rn . Then from Corollary 2.5.2 f∞ (0) = 0, and f∞ is proper. Suppose that f is not cofinite. Then there exists some y = 0 with f∞ (y) := α ∈ R such that we can find two sequences {yk } → y and {tk } → ∞ for which lim
k→∞
f (tk yk ) → α. tk
−1 ∗ Let s ∈ Rn . Then we have t−1 k (s, tk yk − f (tk yk )) ≤ tk f (s), and passing to the limit, since f ∗ (s) ∈ R, we get s, y − α ≤ 0 ∀s, which is obviously impossible. 2
We end this section with some interesting facts and properties of onedimensional convex functions and their relations with the corresponding asymptotic function. Lemma 2.6.1 Let ψ : (−∞, b) → R with 0 ≤ b ≤ +∞ be a convex nondecreasing function with ψ∞ (1) > 0. Then ψ −1 , the inverse of ψ exists and is a concave function, and the following relations hold: (a) sψ ∗ (s−1 ) if s > 0, (−ψ −1 )∗ (−s) = +∞ otherwise . (b) s ∈ dom ψ ∗ ⇐⇒ −s−1 ∈ dom(−ψ −1 )∗ and dom ψ ∗ ⊂ R+ . (c) ψ −1 (t) = inf s>0 {st + sψ(s−1 )}. (d) (−ψ −1 )∞ (t) = − inf{ts−1 : s ∈ R++ ∩ dom ψ ∗ }.
66
2. Asymptotic Cones and Functions
Proof. The strict monotonicity and continuity of ψ implies that ψ −1 exists, and thus from the definition of the convexity of ψ it follows that ψ −1 is concave. By definition of the conjugate of the convex function −ψ −1 one has (−ψ −1 )∗ (−s)
=
sup{−ts + ψ −1 (t) : t ∈ dom ψ −1 } = sup{−sψ(u) + u} u
= sup{s(us−1 − ψ(u)} u s supu {us−1 − ψ(u)} = +∞
if s > 0, otherwise,
proving the desired formula in (a). The relation between the domains in (b) follows as a direct consequence. Furthermore, since ri dom ψ ∗ ⊂ rge ∂ψ ⊂ dom ψ ∗ , and we assumed that ψ is nondecreasing, it follows that rge ⊂ ∂ψR++ , so that ri dom ψ ∗ ⊂ R++ , and hence dom ψ ∗ ⊂ R+ , as stated. To prove the infimal representation of the inverse function ψ in (c) we use the fact that −ψ −1 is lsc proper convex, so that −ψ −1 (t) = (−ψ −1 )∗∗ (t), and with the help of the conjugate formula derived in (a) we then have (−ψ −1 )∗∗ (t) = −ψ −1 (t)
=
sup{st − (−ψ −1 )∗ (s)}
=
sup{st + sψ ∗ (−s−1 )} = − inf {st + sψ ∗ (s−1 )}.
s<0 s<0
s>0
Finally, using the first formula in (b) of the same theorem, and (b) proven above, we have (−ψ −1 )∞ (t)
=
sup{st : s ∈ dom(−ψ −1 )∗ }
= − inf{st : −s ∈ dom(−ψ −1 )∗ } = − inf{ts−1 : s ∈ R++ ∩ dom ψ ∗ }, 2
proving (d).
Note that if in the above lemma we suppose in addition that ψ∞ (1) = +∞, ψ∞ (−1) = 0, then the function t → −ψ −1 (−t) is convex and nondecreasing with domain the interval (−∞, b) and such that −ψ −1 (1) = 0, −ψ −1 (−1) = +∞.
2.7 Application I: Semidefinite Optimization For an arbitrary closed convex cone K ⊂ Rn we define a partial ordering for vectors x, y ∈ Rn via x y ⇐⇒ x − y ∈ K.
2.7 Application I: Semidefinite Optimization
67
The partial ordering satisfies the following conditions: (i) x x, ∀x. (ii) x y =⇒ −y −x. (iii) x y =⇒ λx λy, ∀λ ≥ 0. (iv) x y and x y =⇒ x + x ≥ y + y . (v) xk yk , xk → x, yk → y =⇒ x y. The additional antisymmetric property (vi) x y and y x =⇒ x = y holds if and only if K is in addition a pointed cone. The standard case, already mentioned in Chapter 1, is for vector inequalities when K = Rn+ , the nonnegative orthant, which is a closed convex pointed cone. In that case reduces simply to the usual coordinatewise inequalities x y ⇐⇒ xj − yj ≥ 0, ∀j = 1, . . . , n. Another interesting and important partial ordering that fit the conditions (i) − (v) is for the space of square real symmetric matrices. Let Mn be the space of real square matrices of order n equipped with the inner product A, B :=
n
aij bij = tr AB,
i,j=1
where tr stands for the trace operator of a matrix C ∈ Mn , which is the sum of the diagonal elements of C. Of particular importance is the space of symmetric real matrices of order n, denotes by Sn . This in fact can be viewed as a linear subspace of Mn with dimension n(n + 1)/2 and can be treated as a Euclidean vector space with the trace inner product. Let K ⊂ Sn be a given cone. Then as in the vector case we may define the dual cone K ◦ = {Y ∈ Sn | Y, X ≥ 0, ∀X ∈ K}. Among all the cones in Sn , one of particular interest is the cone of positive semidefinite matrices, denoted by Sn+ and defined by Sn+ := {A ∈ Sn | xT Ax ≥ 0, ∀x ∈ Rn }. It is easy to verify that Sn+ is a closed convex pointed cone. The partial ordering associated with Sn+ is quantified by AB
⇐⇒
A − B positive semidefinite
and obeys the rules (i) − (vi). The cone Sn+ is self dual, i.e., (Sn+ )◦ = Sn+ . The interior of Sn+ consists of positive definite matrices, namely Sn++ := {A ∈ Sn | xT Ax > 0, ∀x = 0}.
68
2. Asymptotic Cones and Functions
Another particularly interesting partial ordering is the Lorentz cone (also called the second-order or ice-cream cone), defined by L := {x ∈ Rn+1 | xn+1 ≥ x}. Spectrally Defined Matrix Functions Spectrally defined functions arise in various areas of mathematical sciences and have recently gained strong interest within the area of matrix optimization problems, called semidefinite programming. Recall that a function f on Rn is called symmetric if f (P x) = f (x) for all n × n permutation matrices P . Definition 2.7.1 The function Φ : Sn → R∪{+∞} is said to be spectrally defined if there exists a symmetric function f : Rn → R ∪ {+∞} such that Φ(X) = Φf (X) := f (λ(X)), ∀X ∈ Sn , where λ(X) := (λ1 (X), . . . , λn (X))T is the vector of eigenvalues of X in nondecreasing order. The function Φ is spectrally defined if and only if Φ is orthonormally invariant, that is, Φ(U T AU ) = Φ(A), ∀U ∈ Un , with Un := the set of n × n orthogonal matrices. The symmetric function f in the definition is then given by f (x) = Φ(diag x), ∀x ∈ Rn , where diag x is the diagonal matrix with diagonal elements x1 , . . . xn . Of particular interest is the class of spectrally defined functions associated with symmetric functions f that are proper convex and lsc, leading to convex matrix functions Φf . Most of these are in fact represented via tr fs (A), for a particular choice of f . We recall that for a smooth scalar real-valued function f and a symmetric matrix A with real eigenvalues λi (A), i = 1, . . . , n, one defines fs (A) := V T diag(f (λ1 (A)), . . . , f (λn (A)))V , with λi (A) ∈ dom f, i = 1, . . . , n, arranged in nondecreasing order, and the trace of fs (A) is thus given by tr fs (A) =
n i=1
f (λi (A)).
(2.25)
2.7 Application I: Semidefinite Optimization
69
Example 2.7.1 (i) Let f (λ) =
n − i=1 log λi +∞
if λ > 0, otherwise.
Then for X ∈ Sn one has Φf (X) =
− log det X +∞,
if X 0, otherwise,
where det X is the determinant of the matrix X. (ii) Let f (λ) = max1≤i≤n λi . Then Φf (X) = λmax (X), the largest eigenvalue of X. (iii) Let n 1 − i=1 λi if λ > 0, f (λ) = +∞ otherwise. Then for X ∈ Sn one has
− tr(X −1 ) +∞,
if X 0, otherwise.
−(Πni=1 λi )1/n +∞
if λ > 0, otherwise.
Φf (X) = (iv) Let
f (λ) =
Then for X ∈ Sn one has Φf (X) =
−(det X)1/n +∞,
if X 0, otherwise.
Convex analysis tools in Rn can be translated to Sn . Most of the interesting properties of Φf can be deduced directly from those of f , and therefore convex analysis of spectrally defined convex functions can be easily developed. The effective domain of Φ is defined by dom Φ := {X ∈ Sn | Φ(X) < ∞}. The conjugate Φ∗ of Φ is defined by Φ∗ (Y ) := sup{X, Y − Φ(X) | X ∈ Sn }, ∀Y ∈ Sn , and the subdifferential of Φ at X ∈ dom Φ is ∂Φ(X)
= {Y ∈ Sn | Φ(Z) ≥ Φ(X) + Z − X, Y , ∀Z ∈ Sn } = {Y ∈ Sn | Φ∗ (Y ) + Φ(X) = X, Y }.
70
2. Asymptotic Cones and Functions
An important and useful inequality is the trace inequality X, Y ≤ λ(X), λ(Y ), ∀X, Y ∈ Sn ,
(2.26)
and equality holds if the matrices X, Y are simultaneously diagonalizable, i.e., if and only if there exists a matrix U ∈ Un such that U T XU = diag λ(X) and U T Y U = diag λ(Y ). Theorem 2.7.1 Let f : Rn → R ∪ {+∞} be a symmetric proper convex and lsc function and Φf : Sn → R ∪ {+∞} the induced convex spectral function. Then f ∗ is symmetric, and the following relations hold: (a) Φ∗f = Φf ∗ . (b) Φf is a proper, lsc, convex function. (c) Φ∗f (Y ) + Φf (X) ≥ λ(X), λ(Y ), ∀X, Y ∈ Sn . (d) Y ∈ ∂Φf (X) ⇐⇒ λ(X), λ(Y ) = X, Y and λ(Y ) ∈ ∂f (λ(X)). Furthermore, X and Y are simultaneously diagonalizable. Proof. The symmetry of f ∗ follows immediately from the fact that for any invertible matrix A, with h(x) = f (Ax) one obtains h∗ (y) = f ∗ ((AT )−1 y), and thus if A is an orthogonal matrix (in particular a permutation matrix) one has A−1 = AT and hence f ∗ (y) = f ∗ (Ay). To prove (a), clearly, we have Φ∗f (Y ) = Φ∗ (diag λ(Y )). Using the definition of the conjugate function one thus has Φ∗f (Y ) = Φ∗ (diag λ(Y ))
=
sup {X, diag λ(Y ) − Φf (X)},
X∈Sn
≥ =
sup {diag x, diag λ(Y ) − Φf (diag x))},
x∈Rn
sup{x, λ(Y ) − f (x)} = f ∗ (λ(Y )) = Φf ∗ (Y ).
The proof of the reverse inequality Φ∗f (Y ) ≤ Φf ∗ (Y ) is simply obtained by combining the Fenchel–Young inequality f ∗ (λ(Y ))+f (λ(X)) ≥ λ(X), λ(Y ) with the trace inequality (2.26). The statements (b), (c), (d) follow as easy consequences of the formula proven in (a), noting that in (d) the last statement follows from the trace equality condition given after (2.26). 2 As a consequence of Theorem 2.7.1, the optimization problems min{Φf (X) | X ∈ Sn } and min{f (x) | x ∈ Rn } are equivalent. In fact, one has inf Φf (X) = −Φ∗f (0) = −Φf ∗ (0) = −f ∗ (0) = inf f.
X∈Sn
Moreover, X ∈ argmin Φf ⇐⇒ λ(X) ∈ argmin f, with argmin Φ := {X ∈ Sn | Φ(X) = inf Φ}.
2.7 Application I: Semidefinite Optimization
71
Following Proposition 2.5.2, the asymptotic function of the proper convex lsc function Φ : Sn → R ∪ {+∞} is defined by Φ∞ (D) = sup t>0
Φ(A + tD) − Φ(A) ∀D ∈ Sn , t
where A ∈ dom Φ. Using Theorem 2.5.4 one has, in fact, Φ∞ (D) = sup{B, D, : B ∈ dom Φ∗ } ∀D ∈ Sn . We remark that if f is symmetric, lsc, convex, and proper, by using Proposition 2.5.2 so is f∞ . Theorem 2.7.2 Let f : Rn → R ∪ {+∞} be symmetric, lsc, proper, and convex with induced spectral function Φf . Then, (Φf )∞ (D) = Φf∞ (D) = f∞ (λ(D)). Proof. For any D ∈ Sn , one has (Φf )∞ (D)
= =
sup{Z, D | Z ∈ dom Φ∗f } sup{Z, D | Z ∈ dom Φf ∗ }
=
sup{Z, D | Z ∈ cl dom Φf ∗ },
where in the second equality we invoke Theorem 2.7.1(a). On the other hand, we also have Φf∞ (D)
= = =
sup{Z, D | Z ∈ dom Φ∗f∞ } sup{Z, D | Z ∈ dom Φ(f∞ )∗ }
sup{Z, D | Z ∈ cl dom Φ(f∞ )∗ }.
To prove the desired statement, it thus suffices to verify that cl dom Φ∗f = cl dom Φ(f∞ )∗ . Using again Theorem 2.7.1(a) one has dom Φf ∗ = {U diag x U T | U ∈ Un , x ∈ dom f ∗ }, and therefore cl dom Φf ∗
= {U diag x U T | U ∈ Un , x ∈ cl dom f ∗ } = {U diag x U T | U ∈ Un , x ∈ dom(f∞ )∗ } = dom Φ(f∞ )∗ ,
where in the second equality we use the fact dom(f∞ )∗ = cl dom f ∗ , which follows from Theorem 2.5.4(b). 2
72
2. Asymptotic Cones and Functions
2.8 Application II: Modeling and Smoothing Optimization Problems Asymptotic functions are essentially built-in within most optimization problems. This can be realized by considering the following general composite optimization model, inf{f0 (x) + p(F (x)) | F (x) ∈ dom p}, where F (x) = (f1 (x), . . . , fm (x))T with all fi real-valued, and p : Rm → R ∪ {+∞} is some given function. The composite model is rich enough to encompass most of the interesting formulations of nonlinear optimization problems and allows for recovering a generic class of both smooth and nonsmooth problems. In fact, as illustrated below, a viable choice for the function p is simply to pick the support function of some given subset Y ⊂ Rm , i.e., to set p(u) := supy∈Y y, u. Example 2.8.1 (i) l1 -norm approximation/optimization problems p(u) =
p
|ui |, Y = {y : y∞ ≤ 1}.
i=1
(ii) Discrete minimax problems p(u) = max ui , Y = ∆m = i=1,...,m
y∈R
m
|
m
yi = 1, y ≥ 0 .
i=1
(iii) l2 -norm nonsmooth problems p(u) = u, Y = {y | y ≤ 1}. Now suppose that there exists an lsc proper convex function H with conjugate H ∗ and such that Y = dom H ∗ . Then by Theorem 2.5.4 one has σY = σdom H ∗ = H∞ , and the composite model can be represented in terms of asymptotic functions and takes the form (CM) with
φ(x) =
v = inf{φ(x) | x ∈ Rn }
m f0 (x) + H∞ (f1 (x), . . . , fm (x)) if x ∈ i=1 dom fi , +∞ otherwise.
Now there are two key observations that can be made: (i) The model above is in fact fairly generic, since it contains all usual (u), problem constrained optimization problems. Indeed, for p(u) = δRm − (CM) is equivalent to inf{f0 (x) | fi (x) ≤ 0, i = 1, . . . , m}.
2.8 Application II: Modeling and Smoothing Optimization Problems
73
(ii) By Corollary 2.5.3 the asymptotic function H∞ of a given lsc proper convex function H can be approximated via H∞ (y) = lim {Hr (y) := rH(r−1 y)}, ∀y ∈ dom H, r→0+
and thus leads one naturally to consider as an approximate problem for (CM) the problem (CM)r vr = inf{φr (x) | x ∈ Rn } with
φr (x) =
f0 (x) + Hr (f1 (x), . . . , fm (x)) +∞
m if x ∈ i=1 dom fi , otherwise.
Thus, with H smooth enough, the above mechanism might lead to generic smooth approximation–type algorithms, such as penalty–barrier methods, which are often used as basic algorithms in the numerical treatment of optimization problems. Thus these methods can be analyzed within a unified framework based on the properties of asymptotic functions and where here r > 0 plays the role of smoothing parameter. The natural questions which thus arise are: Does there exist a solution xr of problem (CM)r ? If it exists, do we have xr → x as a solution of (CM) and vr → v? This type of questions can be efficiently handled via calculus rules of asymptotic functions, as illustrated below and in Chapter 5 later on. Before formalizing the above approach, let us go back first to some of the examples above. For the corresponding function p = H∞ in the three examples, we can easily verify that the corresponding functions H are respectively given by the smooth functions H(y) =
m i=1
1 + yi2 , H(y) = log
m
eyi , H(y) =
y2 + 1.
i=1
In the general case of constrained optimization problems it turns out that . there exists a wide class of functions H such that H∞ = δRm − Proposition 2.8.1 Let θ : R → R ∪ {+∞} be an lsc, proper convex and nondecreasing function with dom θm= (−∞, b) and b ∈ [0, ∞), θ∞ (−1) = 0, θ∞ (1) = +∞ and set H(y) = i=1 θ(yi ). Then H∞ (d) = δRm (d), ∀d ∈ Rm . − m Proof. By Proposition 2.6.1 one has H∞ (d) = i=1 θ∞ (di ), ∀d ∈ Rm . Under the hypothesis on the function θ we have sθ∞ (1) if s > 0, θ∞ (s) = −sθ∞ (−1) if s ≤ 0,
74
2. Asymptotic Cones and Functions
which proves the desired formula for H∞ .
2
Particularly interesting choices for the functions θ are θ1 (u) θ2 (u) θ3 (u) θ4 (u) θ5 (u)
= exp (u), dom θ = R, = − log(1 − u), dom θ = (−∞, 1), u , dom θ = (−∞, 1), = 1−u = − log(−u), dom θ = (−∞, 0), = −u−1 , dom θ = (−∞, 0).
Another interesting example is provided by the class of semidefinite optimization problems introduced in the previous section. Example 2.8.2 Semidefinite programming Consider the following semidefinite optimization problem (SDP) inf cT x : subject to B(x) 0, m with B(x) = B0 + i=1 xi Bi . The problem data are the vector c ∈ Rm and the (m + 1) symmetric matrices B0 , B1 , . . . , Bm of order n × n. This special class of convex problems can also be handled through the general composite model by a simple adjustment of the involved finite-dimensional setting. Let us replace the set Y ⊂ Rm with Y ⊂ Sn , the space of n × n symmetric matrices. More precisely, recalling that the indicator of the negative cone in Rn is a symmetric function, (cf. Section 2.7), we can associate to it the spectral function p : Sn → R ∪ {+∞} defined by p(U ) = δRn− (λ(U )), where λ(U ) is the vector of eigenvalues of the symmetric matrix U in nondecreasing order. For any symmetric matrix U one has δSn− (U ) = δRn− (λ(U )), where Sn− denotes the space of symmetric negative definite matrices, and thus one can write p(U ) = δSn− (U ) = sup{Z, U | Z ∈ Sn+ }. Therefore, problem (SDP) can be written as the composite optimization model inf{cT x + p(B(x)) | B(x) ∈ dom p}. n From here, we can thus proceed as in Example 2.8.1. Let h(y) = i=1 θ(yi ) with θ : R → R ∪ {+∞} a function satisfying Proposition 2.8.1. Then one has h∞ = δRn− . Invoking Theorem 2.7.2 (once again recall that the indicator of the negative cone in Rn is a symmetric function), it follows that for any D ∈ Sn one has H∞ (D) = δSn− (D) with H(D) = h(λ(D)). Thus problem (SDP) can be written in the form of the composite model (CM) inf {cT x + H∞ (B(x))},
x∈Rm
2.8 Application II: Modeling and Smoothing Optimization Problems
75
with corresponding approximate model given by (SDPr )
inf {cT x + Hr (B(x))},
x∈Rm
where Hr (D) = r−1 H(rD) = r−1 h(rλ(D)). Using for example the functions θ given above we thus obtain for any symmetric matrix D the following corresponding functions H H1 (D) = tr(exp D), H2 (D) = − log(det(I − D)) for D ≺ I, +∞ otherwise, H3 (D) = tr((I − D)−1 D) for D ≺ I, +∞ otherwise, H4 (D) = − log(det(−D)) for D ≺ 0, +∞ otherwise, H5 (D) = tr(−D−1 ) for D ≺ 0, +∞ otherwise. We note that for these examples each H is C ∞ on the interior of its domain. To formalize the above approach we need to make some minimal hypothesis on functionals involved in the structural representation of the problem (CM) inf{φ(x) | x ∈ Rn } and its corresponding approximation defined in (CM)r for every r > 0. Definition 2.8.1 A function H : Rm → R ∪ {+∞} that is a proper, lsc, convex function with asymptotic function H∞ is called a generating asymptotic approximation kernel if the following conditions hold: (a) H∞ is isotone, i.e., y i ≤ zi
∀i = 1, 2 . . . , m =⇒ H∞ (y) ≤ H∞ (z).
(b) limr→0+ Hr (y) = H∞ (y), ∀y ∈ ri dom H∞ . (c) ri dom H∞ ⊂ ri dom Hr , ∀r > 0. (d) The constancy space of H∞ satisfies CH∞ = {0}. It can be verified that the functions H defined through Proposition 2.8.1, and based on the choice of the functions θ outlined above, are generating asymptotic approximation kernels. In the case of semidefinite programming given in Example 2.8.2, a similar analysis can be developed to verify that claim. This can be done by either identifying the space Sn with R(n+1)n/2 or in a direct way. We refer the reader to the notes and references for details. Lemma 2.8.1 For any generating asymptotic approximation kernel H one has lim H∞ (a1 , . . . , ai−1 , ai + λ, ai+1 , . . . , am ) = +∞, ∀a ∈ Rm .
λ→+∞
Proof. By definition, H∞ is convex and isotone. Suppose the result does not hold. Then one has lim inf ψ(λ) := H∞ (a1 , . . . , ai−1 , ai + λ, ai+1 , . . . , am ) < +∞, λ→+∞
76
2. Asymptotic Cones and Functions
and by Theorem 2.5.2, the function ψ(λ) is decreasing, and since H∞ is assumed isotone, it follows that H∞ is constant along each direction d = ei , where ei , i = 1, . . . , m, denotes the canonical basis of Rm . This, however, contradicts the condition CH∞ = {0} (cf. Definition 2.8.1) imposed on a generating asymptotic kernel. 2 In the remainder of this section, for the problem’s data (CM) we now make the following minimal assumptions, which are needed to guarantee that (CM) is well-defined. (i) The functions fi : Rn → R ∪ {+∞} , i = 0, 1, . . . , m, are closed, proper and satisfy (fi )∞ (d) > −∞ ∀d. (ii) There exists x0 ∈ dom f0 such that F (x0 ) ∈ dom H∞ , where F (x) := / dom H we suppose in addition that (f1 (x), f2 (x), . . . , fm (x)). When 0 ∈ F (x0 ) ∈ ri dom H∞ . Note that the assumption (i) on fi is always satisfied when fi is convex. Note also that whenever H is isotone, then the isotonicity of H∞ follows at once, and thus if f0 is in addition convex, then φ and φr are convex, and (CM) and (CM)r are convex problems. Lemma 2.8.2 Consider problems (CM ) and (CM )r with a generating asymptotic approximation kernel H. Then: (a) The function φ is lsc, proper, and the same holds for any φr with r > 0 if H is isotone. (b) If 0 ∈ dom H, one has x ∈ dom φ =⇒ x ∈ dom φr . Proof. (a) We prove only that φ is lsc, since with the same arguments we can prove that φr is also lsc. Let x be arbitrary and let {xn } be a sequence converging to x. We have to prove that lim inf n→∞ φ(xn ) ≥ φ(x). As a consequence we have only to consider the case where xn ∈ dom φ. Let > 0 be arbitrary. Since the functions fi are lsc, then for n sufficiently large we have fi (xn ) ≥ fi (x) − , ∀i = 0, 1 . . . , m. Now, since H∞ is isotone, it follows for n sufficiently large that φ(xn ) ≥ f0 (x) + H∞ (f1 (x) − , . . . , fm (x) − ) − . Passing to the limit as n → ∞ we obtain lim infφ(xn ) ≥ f0 (x) + H∞ (f1 (x) − , . . . , fm (x) − ) − .
n→∞
Now let → 0+ . Since H∞ is lsc, we get φ(x) ≤ lim inf φ(xn ), n→∞
2.8 Application II: Modeling and Smoothing Optimization Problems
77
and it follows that φ is lsc. Since f0 , H∞ , H are proper, it follows that φ and φr never take the value −∞. The hypothesis on the problem’s data that there exists x0 ∈ dom f0 with F (x0 ) ∈ dom H∞ implies that φ(x0 ) is finite. If 0 ∈ dom H, then from Proposition 2.5.2, it follows that φr (x0 ) is also finite. In the other case, since ri dom H∞ is a cone, it follows that r−1 F (x0 ) ∈ ri dom H∞ , and then from Definition 2.8.1(c) it follows that φr (x0 ) is finite, proving that φ and φr are proper. To prove (b) we suppose now that 0 ∈ dom H and let x ∈ dom φ. Then F (x) ∈ dom H∞ , f0 (x) is finite, and from Corollary 2.5.2 2 it follows that φr (x) is finite. We arrive now with a key asymptotic formula that can be used in the analysis of the approximation problem (CM)r . Proposition 2.8.2 Consider the problem (CM) with a given generating asymptotic approximation kernel H. Let φ∞ (d) = (f0 )∞ (d) + H∞ (F∞ (d)) if d ∈
m
dom(fi )∞ , +∞ otherwise,
i=1
where F∞ (d) = ((f1 )∞ (d), . . . , (fm )∞ (d)). Then φ∞ (d) ≥ φ∞ (d)
∀d ∈ Rn ,
with equality when the functions fi are convex. Proof. Let ai < (fi )∞ (d) for i = 0, 1, . . . , m and dn → d, tn → +∞ with φ∞ (d) = lim inf n→+∞ t−1 n φ(tn dn ). Then using the fundamental analytical formula of an asymptotic function (cf. Theorem 2.5.1), for n sufficiently large we have for each i fi (tn dn ) ≥ ai tn ⇐⇒ F (tn dn ) ≥ atn with a = (a1 , . . . , am ). Furthermore, since asymptotic functions are positively homogeneous, it follows that F (tn dn ) f0 (tn dn ) φ(tn dn ) = + H∞ . tn tn tn Since H∞ is assumed isotone, using the inequalities above we get φ(tn dn ) ≥ a0 + H∞ (a). tn Now let ai → (fi )∞ (d). Then since H∞ is lsc, using the definition of the asymptotic function and passing to the limit in the above formula, we obtain φ∞ (d) ≥ (f0 )∞ (d) + H∞ (F∞ (d)) if d ∈
m i=1
dom(fi )∞ ,
78
2. Asymptotic Cones and Functions
proving the result for such a direction. Otherwise, the result remains valid by invoking Lemma 2.8.1. Now, if we suppose that for each i = 0, 1, . . . , m the functions fi are convex, since H∞ is sublinear and assumed isotone, it follows that m φ(x + λd) − φ(x) ≤ (f0 )∞ (d) + H∞ (F∞ (d)) if d ∈ dom(fi )∞ . λ i=1
Passing to the limit as λ → +∞ we get φ∞ (d) ≤ φ∞ (d), and thus the desired equality in the convex case follows. 2 We end this section by indicating that other approximating schemes can be naturally generated via the analytical asymptotic function formula. Indeed, consider a scalar function α : R+ → R satisfying α(r) > 0, ∀r > 0, lim α(r) = 0, lim r−1 α(r) = +∞. r→0+
r→0+
Consider now the class of functions θ : R → R+ , that are nondecreasing, convex, and such that lim θ(u) = 0, 0 < θ∞ (1) < +∞.
u→−∞
Since here θ∞ (1) = +∞, we no longer have the representation δRm (u) = − limr→0+ rθ(r−1 u). However, it is possible to write now that δRm (u) = − −1 limr→0+ α(r)θ(r u). Thus, with the functions α and θ as above, we can −1 now construct m the approximation kernel Hr (y) = α(r)H(r y), where m H(y) = i=1 θ(yi ), and we have limr→0+ Hr (y) = δR− (y). Within this class we then construct the approximate models (CM)r and (SDP)r as before. Two particular examples of functions θ that can be used to generate the approximate models are θ6 (u) = log(1 + eu ), θ7 (u) = 2−1 (u + u2 + 4). For any symmetric matrix D, the corresponding functions H used for approximating the semidefinite program (SDP) then take the forms H6 (D) = log det(I + exp(D)), H7 (D) = 2−1 tr D + D2 + 4I .
2.9 Notes and References The concept of asymptotic cone and many of its properties seem to have appeared first in the literature in the work of Steinitz in 1913 [126]. Properties of unbounded convex point sets have been also studied in 1940 by
2.9 Notes and References
79
Stoker [127]. These concepts can also be found in Bourbaki [39] and Choquet [47]. For convex sets, the importance of asymptotic cones were already realized in the work of Dieudonn´e in 1966 [60] and for closed convex sets, in Rockafellar [119], who uses the terminology of recession cones. The generalization of the concept of asymptotic cone of convex sets to closed sets in an arbitrary topological vector space was considered in the works of Dedieu [58], [59]. Section 2.1 includes the fundamental results on asymptotic cones in both the convex and nonconvex cases, most of which can be found in the book of Rockafellar [119], while in the nonconvex case several results outlined here appear in the references cited above, but in greater details in the recent book of Rockafellar–Wets [123]. The dual characterization given in Section 2.2 can be found in the book of Aubin–Ekeland [4]. Many results on closedness criteria have been scattered in the literature, and our intent here was to present some of the most useful criteria. Most results were originally concerned with sufficient conditions that ensure the closedness of the image of a closed set. The conditions are expressed in terms of some properties shared by the map and the asymptotic cone of the set in question. In the convex case, such results can be found in Fenchel [72], Choquet [47], and Rockafellar [117]. The weaky coercive case described by Corollary 2.3.2(b) can be found in [119]. It seems that Dedieu [57] was the first to give the corresponding result to Corollary 2.3.2(a) in the nonconvex coercive case. The necessary and sufficient condition that ensures that the image of a closed set under a linear map remains closed and given in Theorem 2.3.1 is due to Auslender [13]. The concept of asymptotic linear sets and their properties were introduced by Auslender [18]. Theorem 2.3.2, stating that the image of the asymptotic cone of a set coincides with the asymptotic cone of its image, is new. However, such a result was proven earlier in the coercive case by Zalinescu [135] and can be found for the convex case in Rockafellar’s book [119]. Specific results ensuring the closedness of the sum of closed sets and the set of convex combinations of a finite collection of closed sets have been given for the convex case in [119] and extended to the nonconvex case in [13]. Results concerning semibounded sets can be found in an article of Fadeev and Fadeva published in the collected works [100]. Continuous convex sets originated in the work of Gale and Klee [75], and the results given here follow this work. Propositions 2.4.2 and 2.4.3 are due to Auslender–Coutat [11]. Building on the concept of asymptotic cone, one can analyze the behavior in the large of real-valued functions through their epigraphs. This has been done by Rockafellar [119] for convex functions, who gave the representation formula of Proposition 2.5.2. For the nonconvex case, the notion of asymptotic function and the representation formula given in Theorem 2.5.1 were first given by Dedieu [58] and later on by Biaocchi, Butazzo, Gastaldi, and Tomeralli [22]. Most of the results presented in Section 2.5 and Section 2.6 are classical in the convex case and can be found in [119] as well as in the more recent book of Rockafellar and Wets [123], which also includes results for nonconvex functions. Some
80
2. Asymptotic Cones and Functions
other less classical or not so well known results include Proposition 2.6.6, due to Benoist and Hirriart-Urruty [26]; Corollary 2.6.1, due to Valadier [128]; Proposition 2.6.5 from Auslender [13]. Proposition 2.6.7, (well known is convex analysis) is new, and the formula on the composite of a convex function in Proposition 2.6.4 given in Auslender, Cominetti, and Haddou [14], while the results given in Lemma 2.6.1 on one-dimensional convex functions is from Ben-Tal, Ben-Israel, and Teboulle [27]. Systems of matrix inequalities and related optimization problems, today called Semidefinite Optimization originated in 1963 with the work of Bellman and Fan [24]. This class of problems today forms an active research area; see for example, the recent handbook of semidefinite programming [134], which includes about one thousand references. We gave in Section 2.7 some basic notions on convex functions of matrices first characterized via their eigenvalues by Davis [55] and further studied and extended by Lewis in [87] and Seeger [124], on which our presentation is based. The formula for the associated asymptotic function given in Theorem 2.7.2 can be found in Seeger [124], with a slightly different proof given here. Section 2.8 considers smoothing and approximation of optimization problems. Initial studies on that topic can be found in Bertsekas in [33] and in [34], who proposed and studied in particular the logarithmic sum of exponentials. Related numerical algorithms based on this approach such as penalty–barrier and Lagrangians methods can be found in [34] and the more recent book of Berstekas [36], which also includes recent developments in the field together with the relevant references. The idea of smoothing optimization via asymptotic functions was originally proposed by Ben-Tal and Teboulle [28] for finite convex functions, with some applications that can be found in Ben-Tal, Teboulle, and Hang [29]. More recently, Auslender [17] proposed a unified framework within asymptotic functional representation of optimization problems to handle and analyze the more important case of extended real-valued functions, thus allowing the modeling of general problems such as nonlinear constrained optimization and semidefinite programs via the use of Propositions 2.8.1 and 2.8.2 proven in [17]. The presentation here follows the work developed in [17].
3 Existence and Stability in Optimization Problems
A central question in optimization is to know whether an optimal solution exists. Associated with this question is the stability problem. A classical result states that a lower semicontinuous function attains its minimum over a compact set. The compactness hypothesis is often violated in the context of extremum problems, and thus the need for weaker and more realistic assumptions. This chapter develops several fundamental concepts and tools revolving around asymptotic functions to derive existence and stability results for general and convex minimization problems.
3.1 Coercive Problems A central question in minimization problems is to find appropriate conditions that will ensure the existence of a minimizer, namely that f : Rn → R ∪ {+∞} will attain its minimum over Rn . A classical calculus result providing an answer to this question is the well-known Weierstrass theorem, which guarantees that a continuous function defined on a compact subset of Rn attains its minimum (as well as its maximum). Both conditions required in that theorem, compactness and continuity, are rather stringent and often not present in the context of a given minimization problem. Nevertheless, the Weierstrass theorem provides the underlying idea for deriving existence results under weaker hypotheses on the problem’s data. The standard starting point is to realize that solving the general optimization problem (P)
inf{f (x) | x ∈ Rn }
82
3. Existence and Stability in Optimization Problems
in equivalent to minimizing f over one of its level sets lev(f, λ) = {x | f (x) ≤ λ}, for λ > inf f , and thus one should look at “Weierstrass-type” theorems on that set. Let us begin by first recalling the following basic result, which reveals that continuity can be replaced by lower semicontinuity. Proposition 3.1.1 Let f : Rn → R ∪ {+∞} be an lsc, proper function and let C be a nonempty compact set in Rn . Then the optimal solution set of the optimization problem m := inf{f (x) | x ∈ C} is nonempty and compact. Proof. Clearly, if f (x) = ∞ for every x ∈ C, then any x ∈ C attains the minimum of f over C. We can thus assume that inf C f < ∞. Let {xk } ∈ C be a minimizing sequence, i.e., lim f (xk ) = inf{f (x) | x ∈ C}.
k→∞
Since C is compact, there exists a subsequence {xkl } converging to some x ∈ C, and since f is lsc, it follows that f (x) ≤ lim inf l→∞ f (xkl ) = m, and the theorem is proved, since f (x) ≥ m. 2 A similar result for the existence of maxima of f over C can be stated by assuming that f is upper semicontinuous on the compact set C. Going back to the general optimization problem (P), since the optimal solution set of (P) is the same as that of minimizing f on a level set of f , we obtain as an immediate consequence of Proposition 3.1.1 the following result. Corollary 3.1.1 Let f : Rn → R ∪ {+∞} be lsc, and proper, and suppose that lev(f, λ) is bounded for some λ > inf f . Then the optimal solution set of problem (P) is nonempty and compact. Proof. Since f is proper, inf f < ∞. Thus for any λ > inf f , the level set lev(f, λ) is nonempty and closed, since f is lsc, hence compact, by assumption. We can thus apply Proposition 3.1.1 with C := lev(f, λ). 2 The fundamental role played by the asymptotic function in existence questions for minimization problems then comes naturally into action, since the boundedness requirement of a given set can be translated geometrically as asking for its associated asymptotic cone to reduce to the singleton {0} (cf. Proposition 2.1.2). The set in question is here the level set of f , and thus for any λ > inf f , we have that lev(f, λ) is nonempty and bounded if and only if lev(f, λ)∞ is reduced to {0}. Furthermore, as a consequence of
3.1 Coercive Problems
83
the definition of the asymptotic function f∞ we know that (cf. Proposition 2.5.3) (3.1) lev(f, λ)∞ ⊂ {d | f∞ (d) ≤ 0}, with equality replacing the inclusion when f is also assumed convex. This naturally leads one to define the following concepts. Definition 3.1.1 The function f : Rn → R ∪ {+∞} is called (a) level bounded if for each λ > inf f , the level set lev(f, λ) is bounded, (b) coercive if f∞ (d) > 0 ∀d = 0. As an immediate consequence of the definition we remark that f is level bounded if and only if (3.2) lim f (x) = +∞, x→∞
which means that the values of f (x) cannot remain bounded on any subset of Rn that is not bounded. This level boundedness property as given above is widely used in the literature (often called coercivity in classical analysis textbooks). Note that a coercive function is always level bounded, i.e., its level set lev(f, λ) is bounded and nonempty for any λ > inf f . The following proposition gives a useful criterion for recognizing a coercive function. Proposition 3.1.2 Let f : Rn → R ∪ {+∞} be lsc and proper, and define f (x) , β = inf {f∞ (d) | d = 1}. x→∞ x
α = lim inf
(3.3)
Then the infimum β is attained and α = β. Furthermore, f is coercive if and only if there exist γ > 0, δ ∈ R such that f (x) ≥ γx + δ,
∀x.
(3.4)
Proof. Since for any proper function f , its asymptotic function f∞ is lsc, using (3.3) there exists x such that x = 1, f∞ (x) = β.
(3.5)
Then from the analytic representation of f∞ (see Theorem 2.5.1) there exists a sequence dk converging to d, and a sequence tk converging to +∞ such that limtk →+∞ t−1 k f (tk dk ) = β. As a consequence we get α ≤ β. Now let us prove that α ≥ β. From the definition of α there exists a sequence {xk } such that f (xk ||xk ||−1 ||xk ||) = α. k→∞ ||xk ||
||xk || → ∞, xk ||xk ||−1 → x ¯, ¯ x = 1, lim
Once again, using the analytic representation of f∞ (cf. (2.18)), and passing x) ≤ α, which implies that α ≥ β, thus to the limit, it follows that f∞ (¯
84
3. Existence and Stability in Optimization Problems
proving the first part of the proposition. Suppose now that (3.4) holds. Then it obviously follows that α ≥ γ > 0. Since α = β and since f∞ (·) is positively homogeneous, this implies coercivity. Conversely, suppose that f is coercive. Then α = β > 0, and suppose that (3.4) does not hold. Then, for any δ ∈ R, γ > 0, there exists x such that f (x) < γx + δ. Pick δ = −k, γ = k −1 , k ∈ N. Then there exists xk such that f (xk ) < k −1 xk − k. If xk is bounded, it follows that f (xk ) → −∞, which is impossible. If xk is unbounded, then from the last inequality it follows that lim inf
x→+∞
f (x) f (xk ) ≤ lim inf ≤ 0, k→+∞ xk x
which contradicts the coercivity of f .
2
Definition 3.1.2 For any function f : Rn → R∪{+∞}, directions d ∈ Rn such that f∞ (d) ≤ 0 are called asymptotic directions, or recession directions in the case of f convex. In the convex case, the above concepts and relations are in fact all equivalent. Proposition 3.1.3 Let f : Rn → R ∪ {+∞} be lsc and proper. If f is coercive, then it is level bounded. Furthermore, if f is also convex, then the following statements are equivalent: (a) f is coercive. (b) f is level bounded. (c) The optimal set {x ∈ Rn | f (x) = inf f } is nonempty and compact. (d) 0 ∈ int dom f ∗ . Proof. Since lev(f, λ)∞ ⊂ {d | f∞ (d) ≤ 0}, then whenever f is coercive, namely f∞ (d) > 0, ∀0 = d, it follows that f is level bounded. When f is convex, the inclusion above becomes an equality, and since lev(f, λ) is compact, it follows that {d | f∞ (d) ≤ 0} is reduced to 0, so that f is coercive, and this proves the equivalence between (a) and (b). Moreover, by Theorem 1.3.2(c) one has 0 ∈ int dom f ∗ if and only if σdom f ∗ (d) > 0, ∀d = 0, and since by Theorem 2.5.4(a) σdom f ∗ = f∞ , the statements (a) and (d) are equivalent. Finally, since the set of optimal solutions can be written as {x ∈ Rn | f (x) = inf f } = ∂f ∗ (0), then from Proposition 1.2.16, (c) and (d) are equivalent. 2 Using asymptotic rules of calculus, the coercivity condition can translate into useful tests for optimization problems described via explicit constraints.
3.2 Weak Coercivity
85
Corollary 3.1.2 Let fi : Rn −→ R ∪ {+∞} i = 0, 1 . . . , m be lsc, proper functions. Let C := {x | fi (x) ≤ 0 i = 1, . . . , m}, f = f0 + δC (·), and consider the constrained optimization problem (P)
inf{f0 (x) | x ∈ C},
with dom f0 ∩ C = ∅. Suppose that (f0 )∞ (d) > −∞ ∀d = 0. If the functions fi , i = 0, 1, . . . , m have no common nonzero asymptotic direction, i.e., (fi )∞ (d) ≤ 0 ∀i = 0, . . . , m =⇒ d = 0, then the objective function f is coercive (and thus the optimal set of (P) is nonempty and compact). When the the functions fi , i = 0, 1, . . . , m are convex, the converse statement holds. Proof. By asymptotic calculus rules one has f∞ (d) ≥ (f0 )∞ (d) + δC∞ (d), which implies f∞ (d) ≥ (f0 )∞ (d), ∀d ∈ C∞ . Since C∞ ⊂ ∩m i=1 Kfi , the asymptotic assumption implies that f is coercive. Conversely, if the functions {fi , i = 0, . . . , m} are assumed convex, then the above inequalities and the inclusion become equalities thanks to Proposition 2.1.9(a) and Proposition 2.6.1, respectively. As a result, the functions fi , i = 0, . . . , m, have no common nonzero asymptotic directions. 2 Level-boundedness is also linked with the so-called Palais–Smale condition. Definition 3.1.3 Suppose that f : Rn → R is a C 1 function (continuously differentiable). Then: (a) A sequence {xk } such that ∇f (xk ) → 0 and {f (xk )} is bounded is said to be a Palais Smale [PS] sequence for f. (b) f is said to satisfy the Palais–Smale condition if each [PS] sequence for f is bounded. Clearly, if f is level bounded, then f satisfies the Palais–Smale condition. The converse holds if in addition f is assumed bounded below, and this will be proved as a corollary of Ekeland’s variational principle given in Section 4.1.
3.2
Weak Coercivity
For many theoretical purposes in convex analysis, coercivity is a notion that is often very stringent and thus replaced by weak coercivity. In fact, weak coercivity is a property that emerges from a wide class of functions called asymptotically directionally constant.
86
3. Existence and Stability in Optimization Problems
Definition 3.2.1 Let f : Rn → R ∪ {+∞} be a proper and lsc function. Then the function f is said to be: (a) asymptotically directionally constant (adc) if f∞ (d) = 0 =⇒ f (x + ρd) = f (x), ∀x ∈ dom f, ∀ρ ∈ R,
(3.6)
(b) weakly coercive if in addition to (3.6) one has f∞ (d) ≥ 0 ∀d = 0.
(3.7)
Note that a coercive function is obviously adc. The next example shows that there is a wide class of (not necessarly convex) functions that are adc and weakly coercive. Example 3.2.1 Let g : Rm → R ∪ {+∞} be a proper, lsc, and coercive function, and let A be an m × n real matrix with ker A = {0}, and b ∈ − dom g. Consider the function x → f (x) := g(Ax − b). Clearly, this function has unbounded level sets. Indeed, lim f (tk d) = g(−b), for 0 = d ∈ ker A, with
k→∞
lim tk = +∞.
k→∞
Furthermore, for such d since limk→∞ t−1 k f (tk dk ) = 0, it follows by the analytic representation of f∞ (d) that f∞ (d) ≤ 0. Now using again the analytical representation of the asymptotic function we obtain f∞ (d) ≥ inf g(t d ) = g∞ (Ad). lim inf t−1 k k k dk →Ad tk →∞
k→∞
Then if Ad = 0, it follows from the coercivity of g that f∞ (d) > 0, and we conclude that f∞ (d) ≤ 0 if and only if d ∈ ker A, so that f is adc. Furthermore, if g∞ (0) = 0, which is the case when g is convex, then (3.7) is satisfied and f is weakly coercive. In the convex case there are several classes of functions that are adc. Proposition 3.2.1 Let f : Rn → R ∪ {+∞} be a proper, lsc, and convex function such that f is affine on each line for which the direction is an asymptotic direction. Then f is adc. Quadratic convex functions, polynomial convex functions, and functions for which dom f ∗ is an affine set belong to this class. Proof. By hypothesis, f is affine on each line for which the direction d is an asymptotic direction, i.e., satisfying f∞ (d) ≤ 0. Invoking Theorem 2.5.3 we get f∞ (d) = −f∞ (−d). Then assuming that f∞ (d) = 0, it follows by parts (a)–(c) of the same theorem that f (x + ρd) = f (x), ∀x ∈ dom f, ρ ∈ R. Thus f is constant on such a direction, and f is adc. Consider now the class of convex quadratic functions. Using the formula of the corresponding
3.2 Weak Coercivity
87
asymptotic function f∞ given in Example 2.5.1(b), one easily see that f is adc for such functions. More generally, if f is a convex polynomial, then for each x and y the function h : λ → f (x + λy) is a convex polynomial of the single variable λ ∈ R. Since the coefficient associated with the term of highest degree is positive for a nonaffine convex polynomial, then it follows that h is affine or level bounded, so that f is adc. Finally, we consider the class of functions for which dom f ∗ is an affine set. By Theorem 2.5.4(b), f∞ is the support functional of dom f ∗ . Then if d is an asymptotic direction of f , it follows that the linear functional u → d, u is bounded above by 0 on dom f ∗ . Since every functional that is bounded above on an affine set is constant, it follows that −σdom f ∗ (−d) = σdom f ∗ (d), which is equivalent to −f∞ (−d) = f∞ (d). By Theorem 2.5.3 this means that f is affine along the direction d, and thus f is adc. 2 We give now further important equivalent ways for characterizing weak coercivity of convex functions. Theorem 3.2.1 Let f : Rn → R ∪ {+∞} be an lsc, proper, convex function. Then f∞ (d) ≥ 0 ∀0 = d ⇐⇒ 0 ∈ cl dom f ∗ . (3.8) Furthermore, the following statements are equivalent: (a) f is weakly coercive. (b) f∞ (d) ≥ 0 ∀ 0 = d and {d | f∞ (d) = 0} is a subspace, i.e., f∞ (d) = 0 =⇒ f∞ (−d) = 0. (c) f∞ (d) > 0, ∀d = 0, d ∈ Cf⊥ = {d | f∞ (d) = f∞ (−d) = 0}⊥ . (d) 0 ∈ ri dom f ∗ . Proof. By Theorem 2.5.4(b) we have f∞ = σdom f ∗ = σcl dom f ∗ , and then by Theorem 1.3.2, one has 0 ∈ cl dom f ∗ if and only if σcl dom f ∗ (d) ≥ 0, for every d, thus proving the relation (3.8). The equivalence between (a) and (b) is a direct consequence of Theorem 2.5.3. The equivalence between (c) and (d) is a consequence of Theorem 2.5.4 and Proposition 2.5.8(b). Indeed, by the latter proposition one has 0 ∈ ri dom f ∗ if and only if f∞ (d) > 0 for all d ∈ Cf , so that (d) clearly implies (c). Conversely, let PC be the projection operator on C. Then any d ∈ Rn can be decomposed as d = d1 + d2 with d1 = PCf (d), d2 = PCf⊥ (d). But by Theorem 2.5.4 one obtains f∞ (d) = f∞ (d1 + d2 ) = =
sup
{u, d1 + d2 , u}
sup
d2 , u = f∞ (d2 ),
u∈dom f ∗ u∈dom f ∗
where in the third equality we use the fact (cf. (2.21)) d1 ∈ Cf = {y : y, v = 0, ∀v ∈ dom f ∗ }. Therefore, we have proved that for any d ∈ Cf , d2 = 0, f∞ (d) = f∞ (d2 ) > 0, proving that assertion (c) implies (d).
88
3. Existence and Stability in Optimization Problems
Finally, that (b) implies (c) is obvious, while (d) implies (b) follows at once from Proposition 2.5.8(b) and the fact that f∞ (d) = f∞ (d2 ). 2 The following definition used for characterizing general weakly coercive functions will be of particular interest in the next section, devoted to the existence of optimal solutions. Definition 3.2.2 Let f : Rn → R ∪ {+∞} be any function and define by Sf the set of sequences {xk } satisfying ¯ ∈ ker f∞ . xk → +∞, {f (xk )} is bounded above and xk xk −1 → x (3.9) Then f is said to satisfy Assumption (A) if either one of the following two conditions is satisfied: (a) Sf is empty, (b) for each sequence {xk } ∈ Sf there exist ε ∈ (0, 12 ), ρk ∈ ((1+ε)xk , (2− ε)xk ) such that for k sufficiently large we have ¯) ≤ f (xk ). f (xk − ρk x
(3.10)
Proposition 3.2.2 Let f : Rn → R ∪ {+∞} be an lsc proper convex function satisfying f∞ (d) ≥ 0, ∀d = 0. Then f is weakly coercive if and only if f satisfies Assumption (A). Proof. We have only to prove the equivalence between Assumption (A) of Definition 3.2.2 and the relation (3.6). Suppose that relation (3.6) holds for some x ¯. Then from the equivalence of (a) and (b) in Theorem 3.2.1 it follows ¯) = f (xk ) for each ρk and (3.10) is satisfied. Conversely, that f (xk − ρk x suppose that f satisfies assumption (A) and let x ¯ be such that ¯ x = 1 x) = 0. We have only to prove that f∞ (−¯ x) = 0. Let x ∈ dom f , and f∞ (¯ x. Since k −1 (f (xk ) − f (x)) ≤ f∞ (¯ x) = 0, it follows that and set xk = x + k¯ {f (xk )} is bounded above by f (x), and then from Assumption (A) there exists ρk ∈ ((1 + ε)xk , (2 − ε)xk ) satisfying (3.10) for k sufficiently x) ≥ 0, we obtain limk→∞ xk −1 f (xk ) = 0. large. Furthermore, since f∞ (¯ Without loss of generality we may suppose that ρk xk −1 → α > 1. Let ¯), then yk → (1 − α)¯ x. Using (3.10) we obtain yk = xk −1 (xk − ρk x f (xk ) f (xk yk ) ≥ lim inf k→∞ xk k→∞ xk
0 = lim
≥ f∞ ((α − 1)(−¯ x)) =
and it follows that f∞ (−¯ x) = 0.
(α − 1)f∞ (−¯ x) ≥ 0, 2
Very often, the functions that are to be used appear as a result of applying different operations to other simpler functions, for example, the infimal
3.2 Weak Coercivity
89
convolution operation. It is therefore important to give criteria for verifying when the resulting function is weakly coercive, and this will be done by characterizing weakly coercive functions via images of coercive functions under linear transformations. To achieve this task, we first introduce a useful device as follows. To establish existence results we encounter conditions requiring that a point belong to the interior of the domain of a convex function. When the domain is of full dimensionality, the interior is guaranteed to be nonempty. However, it often occurs that the domain of the function under study is not fully dimensional, in which case the interior might be empty. It turns out that we can easily manage this kind of situation by employing a simple and useful dimension-reducing argument. Let h : Rn → R ∪ {+∞} be a convex function with 0 ∈ dom h and let E be the affine hull of its domain, i.e., E = aff(dom h). Note that since 0 ∈ dom h, then E is in fact a vector subspace of Rn . Given h, we associate the function hE defined by hE (x) = (h ◦ PE )(x) = h(PE (x)), where PE denotes the orthogonal projection operator from Rn onto E. Proposition 3.2.3 Let h : Rn → R ∪ {+∞} be a proper convex function with 0 ∈ dom h and E = aff(dom h). The following properties hold: (a) dom hE = dom h + E ⊥ . (b) int dom hE = ri dom h + E ⊥ . (c) ∂hE (x) ⊂ E and ∂hE (x) = ∂hE (y) if y − x ∈ E ⊥ . (d) h∗ (y) = h∗E (PE (y)). (e) ∂h(x) = ∂hE (x) + E ⊥ if x ∈ E; otherwise, ∂h(x) = ∅. (f ) ∂hE (x) = ∂h(PE (x)) ∩ E. Proof. (a) Follows from the definition of hE . To verify (b), use Proposition 1.1.8 and recall that for any subspace M one has ri M = M . Then since here ri dom hE = int dom hE , (b) follows from (a). To prove (c), consider w ∈ ∂hE (x) and take v ∈ E ⊥ . By (a), x + λv ∈ dom hE for each λ ∈ R, and we have hE (x) = hE (x + λv) ≥ hE (x) + λw, v, so that w ∈ E. Furthermore, hE (x) = hE (y) when x−y ∈ E ⊥ , by definition of hE . Then w ∈ ∂hE (x) if and only if for each z ∈ Rn we have hE (z) ≥ hE (x) + w, z − x = hE (y) + w, z − y + w, y − x, but w ∈ E, so that w, y − x = 0, and the second part of (c) is proved. To prove (d), using the definition of the conjugate we obtain h∗ (y)
=
sup {x, y − h(x)} = sup {y, PE (x) − h(PE (x))}
x∈E
x∈Rn
90
3. Existence and Stability in Optimization Problems
=
sup {x, PE (y) − hE (x)} = h∗E (PE (y)).
x∈Rn
We now prove (e) and (f). Let y = u + v with u ∈ ∂hE (x), v ∈ E ⊥ , and let us prove that y ∈ ∂h(x), i.e., h(z) − h(x) ≥ y, z − x, ∀z ∈ dom h. But since h(z) = hE (z), h(x) = hE (x), the above inequality follows from hE (z) − hE (x) ≥ u, z − x = u + v, z − x. Conversely, suppose that y ∈ ∂h(x) and let us prove that PE y ∈ ∂hE (x) so that (e) will hold. For any z ∈ Rn we have h(z) ≥ hE (x) + y, z − x, and then for any z ∈ Rn we get h(PE (z))
= hE (z) ≥ hE (x) + y, PE (z) − x = hE (x) + PE (y), PE (z) − x = hE (x) + PE (y), z − x.
From (c) and (e) we obtain (f) via the relations ∂hE (x) = [∂hE (x) + E ⊥ ] ∩ E
=
[∂hE (PE (x)) + E ⊥ ] ∩ E
=
∂h(PE (x)) ∩ E. 2
Armed with Proposition 3.2.3, we are then able to analyze existence results by working with the function hE , instead of h, since hE is continuous on the interior of its domain, which is always nonempty. The following result shows that weakly coercive functions can be characterized as images of coercive functions under linear transformations. Corollary 3.2.1 Let f : Rn → R ∪ {+∞} be lsc, proper, and convex with inf f > −∞. Then f (x) = (f ∗ ◦ PE )∗ (PE (x)), with E = aff(dom f ∗ ),
(3.11)
and f is weakly coercive if and only if (f ∗ ◦ PE )∗ is coercive. Proof. Since −f ∗ (0) = inf f , then 0 ∈ dom f ∗ , and we can apply Proposition 3.2.3 with h := f ∗ . Then formula (3.11) is nothing else than (d) in Proposition 3.2.3. By Theorem 3.2.1(d), f is weakly coercive if and only if 0 ∈ ri dom f ∗ , which by Proposition 3.2.3(b) is equivalent to saying that ∗ 0 ∈ int dom(f ∗ ◦ PE ), which in turn is equivalent to saying that (f ∗ ◦ PE ) is coercive by Proposition 3.1.3(d). 2 ,
3.2 Weak Coercivity
91
Proposition 3.2.4 Let f, g : Rn → R ∪ {+∞} be lsc, proper, and convex with g weakly coercive, and let h(x) = (f 2g)(x) = inf u {f (u) + g(x − u)} be their infimal convolution. The following hold: (a) If f is weakly coercive, then so is h. (b) Conversely, if g is cofinite, i.e., dom g ∗ = Rn and h is weakly coercive, then f is weakly coercive. (c) The constancy space of h satisfies Ch ⊃ Cf + Cg . Proof. From the equality h∗ = f ∗ + g ∗ (cf. Proposition 1.2.10(a)), we get dom h∗ = dom f ∗ ∩ dom g ∗ , and therefore using Proposition 1.1.7 one has ri dom h∗ = ri dom f ∗ ∩ ri dom g ∗ , from which the first two claims (a)–(b) of the proposition follow from Theorem 3.2.1. Property (c) follows directly from this last formula and from the characterization given in (2.21) for Ch = {d | d, v = 0, ∀v ∈ dom h∗ }, as well as the characterization for 2 Cf and Cg .
Corollary 3.2.2 Let f : Rn → R ∪ {+∞} be lsc, proper, convex, and let E = aff(dom f ∗ ). Then the optimization problem inf{f (x) | x ∈ Rn } has a nonempty optimal set of the form S = K + E ⊥ with K = ∂(f ∗ )E (0) a nonempty convex compact set if and only if f is weakly coercive. Proof. We are in the case where 0 ∈ dom f ∗ . Since S = ∂f ∗ (0), it follows from Proposition 3.2.3 (e) that ∂f ∗ (0) = ∂(f ∗ )E (0) + E ⊥ , and by Proposition 3.1.3 we have that ∂(f ∗ )E (0) is nonempty and compact if and only if 0 ∈ int dom(f ∗ )E , i.e., from Proposition 3.2.3(b) if and only if 0 ∈ ri dom f ∗ . 2
Proposition 3.2.5 Let fi : Rn −→ R ∪ {+∞}, i = 0, 1 . . . , m be lsc, proper, convex functions. Let C := {x | fi (x) ≤ 0 i = 1, 2 . . . , m}, f = f0 + δC (·), and consider the constrained convex optimization problem (P)
inf{f0 (x) | x ∈ C},
with dom f0 ∩ C = ∅. Then f is weakly coercive if and only if (f0 )∞ (d) > 0, ∀0 = d satisfying (fi )∞ (d) ≤ 0 ∀i = 1, . . . , m, d ∈ L⊥ , with L = {d : (fi )∞ (d) = (fi )∞ (−d) = 0, i = 1, . . . , m}. Furthermore, L = Cf , the constancy space of f . Proof. Since f∞ (d) = (f0 )∞ (d) if d ∈ C∞ and f∞ (d) = +∞ otherwise, then f∞ (−d) = (f0 )∞ (−d) whenever −d ∈ C∞ . Thus, it follows that d ∈ Cf , the constancy space of f , if and only if d ∈ Cf0 and m d ∈ C∞ ∩ −C∞ = ∩m i=1 (Kfi ∩ −Kfi ) = ∩i=1 Cfi .
92
3. Existence and Stability in Optimization Problems
To end the proof it then suffices to remark that C∞ = {d | (fi )∞ (d) ≤ 0, i = 1, . . . , m} and use Theorem 3.2.1 together with the first statement proved at the beginning. 2 The following proposition proves that in some cases weak coercivity of a function is a kind of stability property with respect to sequences of functions. Proposition 3.2.6 For a nondecreasing sequence {fk } of lsc, proper and convex functions on Rn with f := supk fk the following properties hold: (a) The sequence {fk∗ } is nonincreasing and f ∗ = cl(inf k fk∗ ). Moreover, the linear manifolds Ek = aff(dom fk∗ ) coincide for k sufficiently large with E = aff(dom f ∗ ). (b) If f is weakly coercive, then fk is weakly coercive for k sufficiently large. Proof. (a) Since Fenchel conjugacy reverses inequalities (g ≤ h ⇒ g ∗ ≥ h∗ ), it follows at once that fk∗ is nonincreasing. Now by direct computation one has ∗ = sup fk∗∗ = sup fk = f, inf fk∗ k
k
k
so that by taking the Fenchel conjugate we deduce f ∗ = cl(inf k fk∗ ). From this characterization and noting that a convex function and its closure have the same affine hull of their corresponding domains, we deduce E = aff(dom(inf fk∗ )) = aff(∪k dom fk∗ ) = ∪k aff(dom fk∗ ) = ∪k Ek . k
Since fk∗ is nonincreasing, the sets Ek form a nondecreasing sequence of linear manifolds contained in E and whose union gives the entire set E. The conclusion then follows easily: All the sets Ek must coincide with E for k large enough. (b) Since a convex function and its closure have the same relative interior of their respective domains, we deduce from (a), ri dom f ∗ = ri(∪k dom fk∗ ). Thus, f weakly coercive implies 0 ∈ ri(∪k dom fk∗ ), which can also be written as E = R+ (∪k dom fk∗ ) = ∪k R+ (dom fk∗ ). Again, the linear manifold E has been expressed as a monotone union of the family of convex cones R+ (dom fk∗ ), so that for k large enough we must have the equality R+ (dom fk∗ ) = E = Ek , which amounts precisely 2 to 0 ∈ ri dom fk∗ , that is, fk is weakly coercive.
3.3 Asymptotically Level Stable Functions
93
3.3 Asymptotically Level Stable Functions As will be seen in the next section, coercivity and weak coercivity will ensure the existence of optimal solutions. We want now to exhibit the widest class of functions for which existence of optimal solutions is ensured. Such functions will be called asymptotically level stable functions. Coercive, weakly coercive, and asymptotically directionally constant functions belong to this class, and our purpose here is to show that this class is very large. In fact, the importance of this class goes beyond the questions of existence of optimal solutions. Indeed, we will see in Section 3.5 that if we suppose in addition convexity, the alternative theorems and closedness criteria can be obtained with such functions under minimal assumptions. In order to prove that this class covers a large spectrum we now introduce different kinds of functions that rely on the concept of asymptotic linear sets introduced in Chapter 2, in Definition 2.3.1. Recall that a nonempty closed set C of Rn is asymptotically linear (als) if for each ρ > 0 and each sequence {xk } satisfying {xk } ∈ C,
||xk || → ∞ xk ||xk ||−1 → x ¯,
(3.12)
there exists k0 ∈ N such that x∈C xk − ρ¯
∀k ≥ k0 .
Proposition 3.3.1 Let D be a nonempty closed convex set of Rn supposed asymptotically linear. For i = 1, . . . , m, let fi : Rn → R ∪ {+∞} be convex asymptotically directionally constant, and let λi ∈ R. Consider the set C=D
m
lev(fi , λi ),
i=1
supposed nonempty. For each i = 1, . . . , m define Ci = D
m
lev(fj , λj ),
j =i
and suppose that fi is bounded below on Ci . Then C is asymptotically linear. Proof. Since all the sets are closed convex sets, we have C∞ = D∞
m i=1
lev(fi , λi )∞ , (Ci )∞ = D∞
(lev(fj , λj ))∞ , ∀i = 1, . . . , m.
j =i
Since fi is bounded below on Ci , it follows that y ∈ D∞ , (fj )∞ (y) ≤ 0 ∀j = i =⇒ (fi )∞ (y) ≥ 0.
(3.13)
94
3. Existence and Stability in Optimization Problems
Now let {xk } be a sequence satisfying (3.12). Then x ¯ ∈ D∞ , and since xk xk xk −1 fj ≤ xk −1 λj , xk passing to the limit, it follows from the formula of the asymptotic function of a convex function given in (2.18) that (fj )∞ (¯ x) ≤ 0 ∀j = 1, . . . , m. x) = 0, and since fj is adc, we have Then from (3.13) it follows that (fj )∞ (¯ x) = fj (xk ) ≤ λj ∀j = 1, . . . , m. fj (xk − ρ¯ Furthermore, since D is asymptotically linear, it follows that there exists x ∈ D for each k ≥ k0 , so that xk − ρ¯ x ∈ C for such k, k0 such that xk − ρ¯ and then C is asymptotically linear. 2 Note that Proposition 3.3.1 is particularly useful for m = 1. Definition 3.3.1 Let f : Rn → R ∪ {+∞} be lsc, and proper. Then f is said to be asymptotically linear if epi f is an asymptotically linear set. Proposition 3.3.2 Let f be asymptotically linear, then for each λ such that lev(f, λ) is nonempty, lev(f, λ) is asymptotically linear. Proof. Set C = lev(f, λ) =
∅. Let ρ > 0, and let {xk } be a sequence satisfying ¯. f (xk ) ≤ λ, ||xk || → ∞, xk ||xk ||−1 → x (xk ,λ) Then (xk , λ) ∈ epi f , and (x → (¯ x, 0), with (¯ x, 0) ∈ (epi f )∞ . Since k ,λ) epi f is asymptotically linear, it follows that for k sufficiently large (xk − x) ≤ λ, proving that lev(f, λ) ρ¯ x, λ − ρ · 0) ∈ epi f is equivalent to f (xk − ρ¯ is als. 2
Now we introduce the fundamental class of asymptotically level stable (als) functions. Definition 3.3.2 Let f : Rn → R∪{+∞} be an lsc, proper function. Then f is said asymptotically level stable (als) if for each ρ > 0, each bounded sequence of reals {λk }, and each sequence {xk } ∈ Rn satisfying ¯ ∈ ker f∞ , xk ∈ lev(f, λk ), xk → +∞ , xk xk −1 → x
(3.14)
there exists k0 such that x ∈ lev(f, λk ) xk − ρ¯
∀k ≥ k0 .
(3.15)
3.3 Asymptotically Level Stable Functions
95
We remark that if for each bounded sequence {λk } there exists no sequence {xk } satisfying (3.14), then automatically f is als. As a consequence coercive functions are als. In the next proposition we shall prove that asymptotically linear functions, as well as piecewise linear quadratic convex functions are, als. The latter class of functions is particularly wide and enjoys many interesting properties. It is often met in various fields such as stochastic programming and optimal control. Definition 3.3.3 A function f : Rn → R∪{+∞} is called piecewise linear quadratic (plq) if dom f can be represented as the union of finitely many polyhedral sets, relative to each of which f (x) is given by an expression of the form x, Ax + 2a, x + α, where α ∈ R, a ∈ Rn , and A ∈ Rn×n is a symmetric matrix. It can be proved (see Notes and References) that the following operations preserve the plq property: (i) The sum of a finite number of plq functions is plq. (ii) If g : Rm → R is plq and f (x) = g(Ax+a) for some A ∈ Rm×n , a ∈ Rm , then f is plq. (iii) If ϕ(x) = f (x, u ¯) with f : Rn × Rm → R with f plq, then ϕ is plq. n (iv) If f : R → R ∪ {+∞} is lsc, proper, and convex, then f is plq if and only if its conjugate f ∗ is plq. (v) For a nonempty polyhedral set Y ⊂ Rm and a symmetric positive semidefinite matrix B ∈ Rm×m the function θY,B : R → R defined by θY,B (u) := sup {y, u − y, By} y∈Y
is lsc, proper, convex, and plq on Rm . Proposition 3.3.3 Let f : Rn → R ∪ {+∞} be lsc and proper. (a) If f is asymptotically linear, then f is als. (b) If f is asymptotically directionally constant, then f is als. (c) Let g : Rn → R ∪ {+∞} be als and let C be an asymptotically linear set in Rn such that dom g ∩ C = ∅ and g∞ (d) ≥ 0, ∀d = 0 d ∈ C∞ . Then f := g + δC is als. (d) Let f : Rn → R ∪ {+∞} be lsc, proper, and convex such that dom f can be represented as the union of finitely many asymptotically linear convex sets Dl , l = 1, . . . , m, relative to each of which f = fl , with fl convex and als. Then f is als. In particular, if f is plq and convex, then f is als. Proof. (a) Suppose that f is asymptotically linear and let {xk } be a sequence satisfying (3.14). Then since {λk } is bounded, we have (xk , λk ) , (¯ x, 0) ∈ (epi f )∞ . k→∞ (xk , λk )
x, 0) = lim (xk , λk ) ∈ epi f , (¯
96
3. Existence and Stability in Optimization Problems
Since epi f is asymptotically linear, it follows that for k sufficiently large (xk − ρ¯ x, λk − ρ · 0) ∈ epi f , which is equivalent to f (xk − ρ¯ x) ≤ λk , i.e., x ∈ lev(f, λk ), and by Definition 3.3.2, that f is als. xk − ρ¯ (b) Let f be asymptotically directionally constant and {xk } be a sequence x) = 0, and since f is adc, it follows that f (xk − satisfying (3.14). Then f∞ (¯ ρ¯ x) = f (xk ) for each k, and therefore (3.15) is satisfied. (c) Let {xk } be a sequence satisfying (3.14). Then for k sufficiently large we have {xk } ∈ C, g(xk ) ≤ λk . As a consequence it follows that x ¯ ∈ C∞ and g∞ (¯ x) ≤ 0, which in turn x) = 0. Then since C is asymptotically linear and g is als, implies that g∞ (¯ for k sufficiently large we have x ∈ C, xk − ρ¯
g(xk − ρ¯ x) ≤ λk ,
which is equivalent to saying that f is als. (d) Let {xk } be a sequence satisfying (3.14). For each l = 1, . . . , m define S(l) = {k | xk ∈ Dl } and I = {l | S(l) is nonfinite}. Now let l ∈ I and yl ∈ dom f ∩ Dl . Since limk→∞,k∈S(l) xk −1 xk = x ¯, it x ∈ Dl for each λ > 0 and every follows that x ¯ ∈ (Dl )∞ , and then yl + λ¯ l ∈ I. As a consequence we have f (yl + t¯ fl (yl + t¯ x) − f (yl ) x) − fl (yl ) = . t t Passing to the limit as t → +∞ it follows that 0 = f∞ (¯ x) = (fl )∞ (¯ x) ∀l ∈ I. Since fl are als and Dl are asymptotically linear, it follows that for each x ∈ Dl , fl (xk − ρ¯ x) ≤ λk ∀k ≥ l ∈ I, there exists k0 (l) such that xk − ρ¯ k0 (l), k ∈ S(l). As a consequence we have f (xk − ρ¯ x) ≤ λk
∀k ≥ k0 (l), k ∈ S(l). Let k ∗ ∈ N be such that N \ [1, k ∗ ] ⊂ l∈I S(l) and k0 = maxl∈I (k ∗ , k0 (l)). x ∈ lev(f, λk ), which proves that f Then for k > k0 it follows that xk − ρ¯ is als. Finally, the last assertion is immediate, by definition of convex plq functions. 2
3.4 Existence of Optimal Solutions The results developed in the previous sections will now be applied to derived general necessary and sufficient conditions for the existence of minimizers.
3.4 Existence of Optimal Solutions
97
Let f : Rn → R ∪ {+∞} be an lsc, proper function and consider the optimization problem (P)
inf{f (x) | x ∈ Rn }.
We introduce two basic hypotheses under which the optimal set S of (P) is nonempty. The first hypothesis is simply expressed in terms of the asymptotic function H0 f∞ (d) ≥ 0 ∀d = 0 ∈ Rn . If inf f > −∞, by the analytic representation of f∞ it follows that H0 holds and appears as a natural necessary condition for the existence of optimal solutions. The second hypothesis is more delicate and is as follows. For any f : Rn → R ∪ {+∞} lsc, and proper, recall (cf. (3.9)) that Sf denotes the set of sequences {xk } satisfying ||xk || → ∞, {f (xk )}bounded above and xk ||xk ||−1 → x ¯ ∈ ker f∞ . (3.16) Then we say that H1 holds if (a) either Sf is empty, or (b) for each sequence {xk } ∈ Sf , there exists zk , ρk ∈ (0, ||xk ||] such that for k sufficiently large one has f (xk − ρk zk ) ≤ f (xk )
and zk → z with ¯ x − z < 1.
(3.17)
Remark 3.4.1 A coercive function obviously satisfies H0 and H1 , because in this case Sf is always empty. Indeed, suppose the contrary. Then there ¯ such that would exist a sequence {xk } and a point x lim ||xk || = +∞,
k→∞
lim xk ||xk ||−1 = x ¯,
k→∞
f∞ (¯ x) = 0.
But since ¯ x = 1 = 0, this is impossible. Theorem 3.4.1 Let f : Rn → R ∪ {+∞} be an lsc, proper function. Then a necessary and sufficient condition for the optimal set S to be nonempty is that H0 and H1 are satisfied. Proof. Suppose first that H0 and H1 hold and let {k } be a nonincreasing sequence of positive reals converging to 0. For each k ∈ N, set fk (x) = f (x) + k g(x), g(x) = x2 , and consider the optimization problem (Pk )
inf{fk (x) | x ∈ Rn }.
Since (fk )∞ (d) ≥ f∞ (d)+k g∞ (d), then from H0 it follows that (fk )∞ (d) > 0 ∀d = 0. As a consequence f is coercive, and from Corollary 3.1.1 there
98
3. Existence and Stability in Optimization Problems
exists at least an optimal solution xk of (Pk ). Let us prove now that the sequence {xk } is bounded. Suppose the contrary. Then without loss of gen¯, and since erality we can suppose that ||xk || → ∞ and xk ||xk ||−1 → x {f (xk )} is bounded above by f (x) + 0 x2 with x ∈ dom f , dividing x) ≤ 0. By Hyf (xk ) by xk and passing to the limit, it follows that f∞ (¯ ¯ ∈ kerf∞ , and then by Hypothesis H1 there pothesis H0 , this implies that x exist zk , ρk ∈ (0, xk ] satisfying (3.17) for k sufficiently large. Therefore, since xk ∈ argmin {fk (x) | x ∈ Rn }, we obtain f (xk ) + k xk 2 ≤ f (xk − ρk zk ) + k xk − ρk zk 2 ≤ f (xk ) + k xk − ρk zk 2 , (3.18) and since εk > 0, it follows that xk − ρk zk ≥ xk . Thus, if we set xk = xk ||xk ||−1 , we have xk − ρk zk = (1 − ρk xk −1 )xk + ρk (xk − zk ) ≤ xk (1 − ρk xk −1 ) + ρk xk − zk ≤
xk + ρk (xk − zk − 1).
(3.19)
It follows that xk − zk ≥ 1, and passing to the limit we get ¯ x − z ≥ 1, which is impossible, and therefore the sequence {xk } is bounded. Now let x ˜ be a limit point of the sequence {xk }. Since f (xk ) + k xk 2 ≤ f (x) + k x2 ∀x ∈ Rn , passing to the limit it follows that x ˜ ∈ S. To prove the converse, let x ˜ ∈ S; then writing x ˜ = xk − xk (xk − x ˜)xk −1 , the hypothesis H1 holds with ˜)xk −1 . 2 ρk = xk , zk = (xk − x A useful criterion for choosing zk in (3.17) consists in taking it constant and equal to α¯ x with α ∈ (0, 2). Then for this specific choice, the hypothesis H1 can be written as follows: (a) Either Sf defined by (3.16) is empty, or (b) for each sequence {xk } ∈ Sf there exists ε ∈ (0, 2) such that for k sufficiently large, there exists ρk ∈ R satisfying ρk ∈ (0, (2 − ε)||xk || ],
f (xk − ρk x ¯) ≤ f (xk ).
(3.20)
This reformulation of H1 will be denoted H1 . As a consequence, we immediately obtain the following corollary. Corollary 3.4.1 Let f : Rn → R ∪ {+∞} be an lsc, proper function such that H0 and H1 hold. Then the optimal set S of (P) is nonempty. Example 3.4.1 Let g : [0.5, 1] → R be continuous and satisfy g(0.5) = e−0.5 , g(1) = e−1 , min{g(x) | x ∈ [0.5, 1]} < 0.
3.4 Existence of Optimal Solutions
Consider the continuous function 2 e−x f (x) = g(x)
99
if x ∈ [0.5, 1] otherwise.
Then we have f∞ (−1) = f∞ (1) = 0, so that H0 holds and H 1 is obviously satisfied. Remark 3.4.2 If in addition to hypothesis H0 and H1 the function f is convex, and in (3.20) we suppose that ε ∈ (0, 12 ) and ρk ≥ (1 + ε)||xk ||, then not only is the optimal set S nonempty, but by Proposition 3.2.2, f is weakly coercive and S is equal to the sum of a compact set and a linear space. Corollary 3.4.2 Let f : Rn → R ∪ {+∞} be asymptotically level stable with inf f > −∞. Then the optimal set of (P) is nonempty. Proof. Since inf f > −∞, it follows that H0 holds. Furthermore, let {xk } ∈ Sf . Then by definition of Sf the sequence {λk = f (xk )} is bounded, and then if ρ > 0, by definition of an asymptotically level stable function it follows that for k sufficiently large f (xk − ρ¯ x) ≤ f (xk ), and H1 is satisfied.
2
Corollary 3.4.3 Let g : Rn → R ∪ {+∞} be lsc and proper and let C be a subset of Rn that is asymptotically linear. Consider the optimization problem m = inf{g(x) | x ∈ C}. (PC ) Suppose that C ∩ dom g = ∅ and that g∞ (d) ≥ 0 ∀d ∈ C∞ , d = 0.
(3.21)
Suppose also that g is adc or als with m > −∞. Then the optimal set S of (PC ) is nonempty. Proof. Problem (PC ) is equivalent to m = inf{f (x) | x ∈ Rn }, where f = g + δC . Since f∞ (d) = +∞, if d ∈ C∞ and (g + δC )∞ (d) ≥ g∞ (d) + δC∞ (d) for d ∈ C∞ ,
(3.22)
it follows from (3.21) that (g)∞ (d) ≥ 0 for each d = 0, and H0 holds. Now x) = 0, it let {xk } be a sequence in Sf . Then {xk } ∈ C, and since f∞ (¯
100
3. Existence and Stability in Optimization Problems
follows from (3.21) and (3.22) that g∞ (¯ x) = 0 and x ¯ ∈ C∞ . Therefore, if g is adc, we have g(xk −ρ¯ x) = g(xk ) for each ρ ∈ R. Since C is asymptotically x ∈ C, so that f (xk −ρ¯ x) = linear, for k sufficiently large we also have xk −ρ¯ f (xk ) and H1 holds. If g is als and m > −∞, then from Proposition 3.3.3(c), f is also als, and thus from Corollary 3.4.2 the optimal set is nonempty. 2
Remark 3.4.3 If we suppose that g and C are convex, then the condition m > −∞ implies that (3.21) holds, since in this case (g+δC )∞ = g∞ +δC∞ .
3.5 Stability for Constrained Problems Let F : Rn ×Rm −→ R∪{+∞} be an lsc proper function. A very important problem in non linear analysis consists in studying the marginal function associated with F and defined by h(y) = inf{F (x, y) | x ∈ Rn },
(3.23)
and in particular to give simple conditions that imply that the infimum is attained and that h is lsc at some point y. The function F (x, y) stands in fact as a perturbation function, parameterized by the vector y ∈ Rm , and (3.23) can be viewed as a perturbed or parametric optimization problem. A key problem is thus to study the behavior of the optimal value h(y) and the corresponding optimal solution x(y), with respect to changes in the parameter y. This question is called stability, and it plays a central role in duality for optimization problems, a topic that will be developed in detail in Chapter 5. In this section we focus on the conditions that guarantee that there exists a minimum in (3.23) and that h is lsc. For this purpose we begin by studying a simpler problem, from which general results will easily follow. Let A be an m × n real matrix and f : Rn → R ∪ {+∞} an lsc, proper function. Consider the marginal function defined by h(y) := Af (y) := inf{f (x) | x ∈ C(y)},
(3.24)
where C(y) := {x ∈ Rn | Ax = y}. We wish to give conditions close to hypotheses H0 and H1 under which it may be proved that h is lsc at a fixed point y. First we remark that h(y) = inf{Fy (x) | x ∈ Rn },
(3.25)
with Fy (x) = f (x) + δC(y) (x). Then since C(y)∞ = {d | Ad = 0}, the hypothesis H0
f∞ (d) ≥ 0 ∀d ∈ ker A, d = 0,
(3.26)
3.5 Stability for Constrained Problems
101
implies H0 for Fy and coincides with H0 when f is convex. The second hypothesis, which corresponds to H1 , has to take into account perturbations around y. In similarity with the set Sf defined in (3.16), we thus let {yk } be a sequence converging to y and define by Sf [{yk }, y] the set of sequences {xk } satisfying Axk = yk such that ||xk || → ∞, {f (xk )} bounded above, xk ||xk ||−1 → x ¯ ∈ ker f∞ ∩ ker A. (3.27) Then H1 [y] holds if for any sequence {yk } converging to y: (a) either S[{yk }, y] is empty, or (b) for each {xk } ∈ S[{yk }, y] there exists zk , ρk ∈ (0, ||xk ||] such that for k sufficiently large we have f (xk − ρk zk ) ≤ f (xk ), Azk = 0, zk → z with ¯ x − z < 1.
(3.28)
Theorem 3.5.1 Suppose that f : Rn → R ∪ {+∞} is lsc and proper, and that H0 and H1 [y] hold. Then h is lsc at y, and when C(y) ∩ dom f = ∅, the optimal set S(y) is nonempty. Proof. Suppose that C(y) ∩ dom f is nonempty and set yk = y. To prove that S(y) is nonempty we have only to verify assumption H1 for Fy . But this is an immediate consequence of H1 [y] if we set yk = y for each k. It remains to prove that h is lsc at y; that is, h(y) ≤ lim inf k→∞ h(yk ). Without loss of generality we can suppose that lim inf h(yk ) = lim h(yk ) < ∞. k→∞
k→∞
(3.29)
Then it follows that for each k, C(yk ) ∩ dom f is nonempty. Let εk → 0+ , and set h(yk ) + εk if h(yk ) > −∞ (3.30) mk = −k otherwise, and Sk = {x | f (x) ≤ mk Ax = yk }.
(3.31)
Define xk as xk ∈ argmin{f (x) + εk x2 | x ∈ Sk }. Thanks to hypothesis H0 the objective function f (·) + k · 2 + δSk (·) is coercive, and from Corollary 3.1.1 such an xk exists. If the sequence {xk } is bounded, there exists a subsequence {xnk } converging to x∗ . Passing to the limit, since f is lsc, it follows from (3.31) that Ax∗ = y and f (x∗ ) ≤ lim inf k→∞ h(yk ). Since x∗ ∈ C(y), we then have h(y) ≤ f (x∗ ), which proves that h is lsc at y. Let us prove that the sequence {xk } is indeed bounded. In the contrary case we can suppose that ||xk || → ∞,
x k = xk ||xk ||−1 → x ¯ = 0.
(3.32)
102
3. Existence and Stability in Optimization Problems
Since f (xk ) ≤ mk , from (3.29), (3.30), there exists a constant c such that f (xk ) ≤ c for each k. Then using (2.18) it follows that f∞ (¯ x) ≤ 0. Furthermore, yk = 0. A¯ x = lim Axk ||xk ||−1 = lim k→∞ k→∞ xk As a consequence, using condition H0 it follows that A¯ x = 0 = f∞ (¯ x). Now we can use hypothesis H1 [y], and thus there exist zk , ρk ∈ (0 , ||xk ||] such that for k sufficiently large (3.28) holds. As a consequence, xk − ρk zk belongs to Sk , and by definition of xk the inequalities (3.18) hold, from which it follows that ||xk || ≤ xk − ρk zk . Then using the same arguments as in the proof of Theorem 3.4.1, we obtain the desired contradiction. 2 Corollary 3.5.1 Suppose that f is als and that f∞ (d) ≥ 0
∀d ∈ ker A,
d = 0.
Suppose also that cl h(y) > −∞. Then h is lsc at y, and when C(y) ∩ dom f = ∅, the optimal set S(y) is nonempty. Proof. We have only to verify that H1 [y] holds. Take a sequence {xk } ∈ ¯ ∀k . Then since Axk = yk , for each Sf [{yk }, y] with yk → y and set zk = x ε > 0 we have for k sufficiently large, f (xk ) > cl h(y) − ε, and it follows that the sequence {f (xk )} is bounded. As a consequence, since f is als, the 2 hypothesis H1 [y] is satisfied. Let us come back to our original motivation and study the marginal function h of an lsc proper function F : Rn × Rm : R ∪ {+∞}, i.e., h(y) := inf{F (x, y) | x ∈ Rn }. If we set A(x, y) = y, then h(y) = (AF )(y), and one can use Theorem 3.5.1. Condition H0 reduces to H0
F∞ (x, 0) ≥ 0 ∀x = 0,
and H1 [y] becomes H1 [y] as follows: Let {yk } be a sequence converging to y and define TF [{yk }, y] as the set of sequences {xk } satisfying ||xk || → ∞, {F (xk , yk )} bounded above, xk ||xk ||−1 → x ¯ ∈ ker F∞ (·, 0). (3.33) Then H1 [y] holds if for any sequence {yk } converging to y: (a) either TF [{yk }, y] is empty, or (b) for each sequence {xk } ∈ TF [{yk }, y] there exist zk , ρk ∈ (0, (xk , yk )] such that for k sufficiently large we have F (xk − ρk zk , yk ) ≤ F (xk , yk ), zk → z with ¯ x − z < 1.
(3.34)
3.5 Stability for Constrained Problems
103
As an immediate consequence of Theorem 3.5.1 we then obtain the following properties for the marginal function h defined in (3.23). Corollary 3.5.2 Suppose that F : Rn × Rm → R ∪ {+∞} is lsc, proper, and that H0 and H1 [y] hold. Then h is lsc at y and the optimal set S(y) = {x | F (x, y) = h(y)} is nonempty. Corollary 3.5.3 Suppose that F : Rn × Rm → R ∪ {+∞} is lsc, proper and that F∞ (x, 0) ≥ 0 ∀x = 0. Then h is lsc at each y, and the optimal set S(y) = {x | F (x, y) = h(y)} is nonempty under any one of the following conditions: (a) F∞ (x, 0) > 0 ∀x = 0. (b) F is convex and for each x such that F∞ (x, 0) = 0 we have F∞ (−x, 0) = 0. (c) cl h(y) > −∞ and F is asymptotically level stable. Proof. To prove the validity of the assertion under condition (a) or (b) we use Corollary 3.5.2, and we have only to verify that hypothesis H1 [y] holds. This is immediate for case (a), since for each sequence {yk } → y the set TF [{yk }, y] is obviously empty. In case (b), let {xk } ∈ TF [{yk }, y]. Then since F is convex and since for each x such that F∞ (x, 0) = 0 we have F∞ (−x, 0) = 0, it follows that F is constant on each direction (x, 0), x, yk ) = F (xk , yk ) ∀ ρ > 0 so that for k sufficiently large one has F (xk − ρ¯ and (3.34) holds. Finally, as an immediate consequence of Corollary 3.5.1 the assertion under (c) follows. 2 Corollary 3.5.4 Let f, g : Rn → R ∪ {+∞} be proper, convex and polyhedral. Then f 2g is polyhedral, and the infimum is attained in the infimal convolution whenever the latter is proper. Proof. Let F (x, y) := f (x) + g(y − x) and let h be the associated marginal function. Then epi h = epi f +epi g, so that epi h is a polyhedral set and thus h is polyhedral and hence also lsc. If we suppose that h is also proper, then h(y) > −∞, which implies that F∞ (x, 0) ≥ 0 for all x = 0. Furthermore, by Proposition 1.2.11, F is polyhedral, and then it follows that F is als and one can apply part (c) of Corollary 3.5.3. 2 Now we want to give a formula for the asymptotic function (Af )∞ in terms of the asymptotic function f∞ . In order to derive such a formula we need to suppose that H1 [y] holds for each y, and also that formula (3.27) does not depend on yk , y. More precisely, we introduce the following hypothesis H2 as follows: Let X be the set of sequences {xk } ∈ dom f satisfying ||xk || → ∞, xk ||xk ||−1 → x ¯ ∈ ker f∞ ∩ ker A.
(3.35)
104
3. Existence and Stability in Optimization Problems
Then H2 holds if: (a) either X is empty, or (b) for each sequence {xk } ∈ X there exists zk , ρk ∈ (0, ||xk ||] such that for k sufficiently large (3.28) holds; i.e., Azk = 0, f (xk − ρk zk ) ≤ f (xk ), zk → z, with ¯ x − z < 1.
(3.36)
Remark 3.5.1 If H2 is satisfied, then H1 [y] holds for every y ∈ Rn . Also, note that if f is asymptotically directionally constant, then H2 holds. Theorem 3.5.2 Let f : Rn → R ∪ {+∞} be an lsc, proper function and A a real m × n matrix. Suppose that there exists y ∈ Rn and x ∈ dom f such that Ax = y. Suppose also that H0 and H2 hold. Then the marginal function h = Af is lsc and proper. Furthermore, the optimal set S(y) is nonempty for each y such that C(y) ∩ dom f = ∅, and one has ∀y ∈ Rn :
(Af )∞ (y) = inf{f∞ (d) : Ad = y}.
Proof. Since H2 holds, it follows that H1 [y] holds for each y, and by Theorem 3.5.1 it follows that for any y, h is lsc and the optimal set S(y) in nonempty whenever C(y) ∩ dom f = ∅, so that h is proper, proving the ˜ ∞ (y) = inf{f∞ (d) | Ad = y}. We prove first first part of the theorem. Let h ˜ that h∞ ≤ h∞ = (Af )∞ . Let s ∈ R satisfy h∞ (y) ≤ s, which is equivalent to (y, s) ∈ (epi h)∞ . Then there exist sequences {(yk , sk )} and {tk } with tk → ∞ such that h(yk ) ≤ sk ,
(yk , sk ) → (y, s). tk
Let Sk = {x | f (x) ≤ sk , Ax = yk }. Since h(yk ) < ∞, it follows from Theorem 3.5.1 that the optimal set S(yk ) is nonempty, and we may define xk ∈ argmin f (x) + εk x2 | x ∈ Sk for a sequence {εk } → 0+ . Let us show that we cannot have limk→∞ t−1 k ||xk || = ∞. Suppose the contrary. Then limk→∞ ||xk || = +∞, and we may suppose ¯ with ¯ x = 1. Furtherwithout loss of generality that ξk := xk ||xk ||−1 → x more, one has (xk , sk ) sk tk = ξk , → (¯ x, 0). ||xk || tk ||xk || Since Aξk =
yk , ||xk ||
f (ξk ||xk ||) sk tk ≤ , ||xk || tk ||xk ||
then passing to the limit and using formula (2.18) we get A¯ x = 0, f∞ (¯ x) ≤ 0. Now, by hypothesis H2 , there exist zk and ρk ∈ (0, ||xk ||] such that f (xk − ρk zk ) ≤ f (xk ), A(xk − ρk zk ) = yk , zk → z, with ¯ x − z < 1.
3.5 Stability for Constrained Problems
105
Therefore, it follows that xk − ρk zk ∈ Sk and one has f (xk )+εk xk 2 ≤ f (xk −ρk zk )+εk xk −ρk zk 2 ≤ f (xk )+εk xk −ρk zk 2 , which implies that xk − ρk zk ≥ xk and then using the same arguments as given at the end of the proof of Theorem 3.4.1 we obtain the desired contradiction. Without loss of generality, we may now suppose −1 ¯. Furtherthat the sequence {t−1 k ||xk ||} is bounded and that tk xk → x −1 −1 −1 −1 −1 y . Passing to more, tk f (xk ) = tk f (tk xk tk ) ≤ tk sk , tk Axk = t−1 k k ˜ ∞ (y) ≤ s, x) ≤ s, A¯ x = y, and consequently h the limit, we get f∞ (¯ ˜ ∞ (y) ≤ h∞ (y) = (Af )∞ (y). To prove the from which it follows that h ˜ ∞ ≥ h∞ , it is only necessary to prove that for each reverse inequality, h ˜ ∞ (y) ≤ s we have h∞ (y) ≤ s. Let D = {(u, v) | Au = v} (y, s) with h and F (x, y) = f (x) + δD ((x, y)). Let P : Rn × Rm × R → Rm × R be the projection map P (x, y, r) = (y, r). Then one has P (epi F ) = epi h. Indeed, if (y, µ) ∈ epi h, then from Theorem 3.5.1 there exists an x such that F (x, y) = h(y) ≤ µ, and then (y, µ) = P (x, y, µ) ⊂ P (epi F ). Conversely, let (y, µ) = P (x, y, µ) with (x, y, µ) ∈ epi F. Then F (x, y) ≤ µ and h(y) ≤ µ. Thus, using epi h = P (epi F ) one has (y, s) ∈ epi h if and only if there exists an x such that Ax = y and f (x) ≤ s. As a consequence, one obtains that (y, s) ∈ (epi h)∞ if and only if there exists a sequence tk → +∞ and a sequence {xk } with Axk = yk , f (xk ) ≤ sk , t−1 k (yk , sk ) → (y, s). Define E = {(y, s) | ∃x with Ax = y, f∞ (x) ≤ s} and take (y, s) ∈ E. Then there exist x and sequences {tk }, {xk } satisfying tk → ∞, f (xk ) ≤ sk , t−1 k (xk , sk ) → (x, s), Ax = y. Set yk = Axk . Then t−1 k yk → Ax = y, and it follows that (y, s) ∈ (epi h)∞ . ˜ ∞ there exists a ˜ ∞ (y) ≤ s and εk → 0+ . By definition of h Now let h sequence {xk } such that Axk = y, f∞ (xk ) ≤ s + εk , and thus from the argument above it follows that (y, s + εk ) ∈ (epi h)∞ , which means that h∞ (y) ≤ s + εk . Passing to the limit, it thus follows that h∞ (y) ≤ s, and the proof is complete. 2 We now return to the general case of h defined in (3.23), to derive a formula for its asymptotic function in terms of F∞ . This will be obtained as an immediate consequence of Theorem 3.5.2. For that purpose, we replace the hypothesis H2 with H2 defined as follows: Let W be the set of sequences{xk , yk } ∈ dom F with (xk , yk ) → ∞,
(xk , yk ) → (¯ x, 0) ∈ ker F∞ . (xk , yk )
Then H2 holds if: (a) Either W is empty, or (b) for each sequence {xk , yk } ∈ W there exists zk , ρk ∈ (0, (xk , yk )] such that for k sufficiently large (3.34) holds.
106
3. Existence and Stability in Optimization Problems
Corollary 3.5.5 Let F : Rn ×Rm → R∪{+∞} be an lsc, proper function. Suppose that H0 and H2 hold. Then the marginal function y → h(y) = inf{F (x, y) | x ∈ Rn } is proper, lsc, and for each y ∈ Rm , h∞ (y) = inf{F∞ (x, y) :
x ∈ Rn }.
(3.37)
Proof. Set A(x, y) = y, so that h(y) = AF (y) and apply Theorem 3.5.2. 2 Using the same argument as in Corollary 3.5.3 we obtain from Corollary 3.5.5 the following results. Corollary 3.5.6 Let F : Rn × Rm → R ∪ {+∞} be an lsc, proper function and suppose that one of the following conditions holds: (a) F∞ (x, 0) > 0 ∀x = 0. (b) F is convex , F∞ (x, 0) ≥ 0 ∀x = 0, and for each x such that F∞ (x, 0) = 0 we have F∞ (−x, 0) = 0. Then the marginal function y → h(y) = inf{F (x, y) | x ∈ Rn } is lsc, proper, and for each y, the infimum is attained. Furthermore, one has for each y, h∞ (y) = inf{F∞ (x, y) | x ∈ Rn }. Proof. Since H0 holds in cases (a) and (b), we have only to verify the hypothesis H2 and use Corollary 3.5.5. In case (a), H2 holds since W is empty. In case (b), let (xk , yk ) ∈ W . Then since F is convex and since for each x such that F∞ (x, 0) = 0 we have F∞ (−x, 0) = 0, it follows that F is constant on (x, 0), and (3.34) holds. 2 Corollary 3.5.7 Let f : Rn → R ∪ {+∞} be lsc, proper, and let A be a linear map from Rn to Rm . Consider the function Af , defined by (3.24). Suppose that Af ≡ ∞ (i.e., dom f ∩ A−1 (Rm ) = ∅). Then, under either of the following conditions (a) f∞ (d) > 0, ∀0 = d ∈ ker A, (b) f is convex and Az = 0 for any z such that f∞ (z) ≤ 0 and f∞ (−z) > 0, one has Af proper, lsc, and when Af (y) = +∞, the infimum in the definition of (Af )(y) is attained for some x. Moreover, for each y, (Af )∞ (y) = Af∞ (y), and Af is convex whenever f is convex. Proof. Thanks to Theorem 3.5.2, we have only to prove that H0 and H2 hold. Under the case (a), the hypothesis H0 trivially holds, but also H2 . Indeed, the set X (cf. (3.35)) is empty in this case. Now consider the case (b). First, we show that H0 holds. In the contrary case, there would exist 0 = d with Ad = 0 and f∞ (d) < 0, and then by the assumptions in (b), we would have f∞ (−d) ≤ 0. But since f is convex, f∞ is also convex, and since it is positively homogeneous, we obtain 0 = f∞ (0) ≤ f∞ (d) + f∞ (−d) < 0, which is impossible. Now we prove that H2 holds as well. Consider a
3.6 Dual Operations and Subdifferential Calculus
107
sequence {xk } ⊂ X. Then f∞ (¯ x) = A¯ x = 0, x ¯ = 0, and by the assumption (b), it follows that f∞ (−¯ x) ≤ 0, which in turns implies by using (H0 ) that x) = 0. But since f is assumed convex, this means that x ¯ is a direction f∞ (−¯ ¯. 2 of constancy for f and (3.36) is satisfied by choosing for all k, zk = x An interesting application of these results is to the infimal convolution of a finite family of convex functions. Corollary 3.5.8 Let f1 , . . . , fp be lsc, proper, convex functions on Rn . Asp sume that i=1 zi = 0 for all vectors zi such that p
(fi )∞ (zi ) ≤ 0,
i=1
p
(fi )∞ (−zi ) > 0.
i
Then the infimal convolution h = f1 2 · · · 2fp is a proper, lsc, convex function, and the infimum is attained. Moreover, h∞ = (f1 )∞ 2 · · · 2(fp )∞ . p Proof. Take A as the linear map A : Rnp → R defined by i=1 xi , xi ∈ Rn , p 2 and f (x1 , . . . , xp ) = i=1 fi (xi ) in Corollary 3.5.7.
3.6 Dual Operations and Subdifferential Calculus Let fi : Rn → R ∪ {+∞}, i = 1, . . . , p, be a collection of proper convex functions. As recalled in Chapter 1, the following relations hold: (f1 2 · · · 2fp )∗ ∂f1 + · · · + ∂fp
= f1∗ + · · · + fp∗ , ⊂ ∂(f1 + · · · + fp ).
(3.38) (3.39)
In fact, both relations can be verified in a straightforward fashion, using the definition of the mathematical operations involved. However, to obtain, for example, the expected dual formula of (3.38), (f1 + · · · + fp )∗ = (f1∗ 2 · · · 2fp∗ ), and the reverse inclusion for the subdifferential inclusion (3.39), much more analysis is needed. In fact, one also needs to suppose a qualification condition on the domains of the function involved. For the two examples above, such a condition simply asks that the sets ri(dom fi ), i = 1, . . . , p, have a common point. Similar questions can be asked for other fundamental operations such as the composite function with a linear map. Despite its simplicity, the condition on the nonemptiness of the relative interiors of the domains has, in fact, a deep meaning, which turns out to be strongly
108
3. Existence and Stability in Optimization Problems
connected to the notion of asymptotic functions, and it is our purpose in this section to exhibit these connections. Dual Operations We first give a simple rule that will ensure in the convex case that the basic assumptions made in Corollary 3.5.7 and Corollary 3.5.8 are satisfied. Proposition 3.6.1 Let g : Rm → R ∪ {+∞} be a proper convex function and let A be a linear map from Rn to Rm . If there exists an x ∈ Rn such that Ax ∈ ri dom g, then A∗ y = 0 for every y satisfying (g ∗ )∞ (y) ≤ 0, and (g ∗ )∞ (y) > 0. Proof. Recall that cl g = g ∗∗ , an lsc proper convex function, and since ri dom g = ri dom cl g, the condition Ax ∈ ri dom g is equivalent to Ax ∈ ri dom g ∗∗ for some x ∈ Rn . Set f := g ∗ , h(y) := f (y) − y, Ax = f (y) − x, A∗ y. Then, using Proposition 2.5.8(b), one has Ax ∈ ri dom g =⇒ h∞ (y) > 0, ∀ y, except for those y ∈ Ch = {v : h∞ (y) = h∞ (−y) = 0}. Suppose the conclusion of the proposition does not hold. Then there exists y such that (g ∗ )∞ (y) ≤ 0, (g ∗ )∞ (−y) > 0, and A∗ y = 0. As a consequence, h = f = g ∗ and h∞ (y) ≤ 0 with y ∈ Ch , which is impossible. 2
Remark 3.6.1 In fact, it can be verified that the converse statement is also true in Proposition 3.6.1, although the proof of the converse part does not rely on the use of asymptotic calculus. Corollary 3.6.1 Let fi : Rn → R ∪ {+∞}, i = 1, . . . , p, be a collection of proper convex functions. If ∩pi=1 ri dom fi = ∅, then p i=1
yi = 0, ∀ yi such that
p p (fi∗ )∞ (yi ) ≤ 0, (fi∗ )∞ (−yi ) > 0. i=1
i=1
Proof. This is a direct application of Proposition 3.6.1. Indeed, ptake A : Rn → Rnp defined by Ax = (x, . . . , x) with x ∈ Rn , and g(x) = i=1 fi (xi ) p y , dom g = for all x = (x1 , . . . , xp ) with xi ∈ Rn . Then, A∗ y = i i=1 p dom f , with g proper convex. A direct calculation shows that g ∗ (y) = ∩ i i=1 p p ∗ 2 i=1 fi (yi ), and by Proposition 1.1.9, ri dom g = ∩i=1 ri dom fi . As already noted in Remark 3.6.1, it can also be shown that the converse statement of the last corollary holds true. Theorem 3.6.1 Let f be a convex function on Rn , let g be a convex function on Rm , and let A be a linear map from Rn to Rm . Then: (a) (Af )∗ = f ∗ A∗ .
3.6 Dual Operations and Subdifferential Calculus
109
(b) ((cl g)A)∗ = cl(A∗ g ∗ ). (c) If g is assumed proper, and there exists x ∈ Rn such that Ax ∈ ri dom g, the closure operation can be omitted in (b), so that (gA)∗ (u) = inf{g ∗ (v) | A∗ v = u}, and the infimum is attained for each u such that there exists v ∈ dom g with A∗ v = u. Proof. A direct calculation shows that (Af )∗ = f ∗ A∗ , proving (a), and applying this relation to A∗ and g ∗ we obtain (b). Under the assumption given in (c), Proposition 3.6.1 holds, and therefore we can apply Corollary 3.5.7(b) to the function g ∗ and the linear map A∗ , from which it follows that A∗ g ∗ = cl(A∗ g ∗ ) and for each u such that there exists v ∈ dom g with Av = u, the infimum in the formula A∗ g ∗ (u) is attained. Now, since Ax ∈ ri dom g, it follows from Proposition 2.6.3 that (cl g)A = cl(gA). Hence, ((cl g)A)∗ = (gA)∗ , and since we have just proved that ((cl g)A)∗ = cl(A∗ g ∗ ), we obtain finally A∗ g ∗ = (gA)∗ , which completes the proof. 2
Proposition 3.6.2 Let fi : Rn → R ∪ {+∞}, i = 1, . . . , p, be a collection of proper convex functions such that the sets ri(dom fi ), i = 1, . . . , p, have a common point. Then (f1 + · · · + fp )∗ (u) = inf{f1∗ (x1 ) + · · · + fp∗ (xp ) | x1 + · · · + xp = u}, where for each u the infimum is attained whenever it is finite. Proof. From Proposition 1.2.10 we have (cl f1 +· · ·+cl fp )∗ = cl(f1∗ 2 · · · 2fp∗ ). Since we assume that the sets ri(dom fi ), i = 1, . . . , p, have a common point, it follows from Proposition 1.2.7 that cl(f1 +· · ·+fp ) = cl f1 +· · · cl fp , so that (f1 + . . . + fp )∗ = cl(f1∗ 2 . . . 2fp∗ ). On the other hand, the same nonemptiness assumption on the relative interiors means that Corollary 3.6.1 is applicable, which in turn implies that we can invoke Corollary 3.5.8 on the functions fi∗ , ensuring that the infimal convolution of the fi∗ is lsc, and the infimum is attained whenever it is finite. 2 A useful particular case is the following Corollary 3.6.2 For two closed convex sets C1 , C2 of Rn satisfying 0 ∈ ri(C1 − C2 ) one has σC1 ∩C2 (d) = (σC1 2σC2 )(d), ∀d, with the infimum attained and dom σC1 ∩C2 = dom σC1 + dom σC2 . Proof. Apply Proposition 3.6.2 with fi = δCi , i = 1, 2, recalling that fi∗ = σCi . 2
110
3. Existence and Stability in Optimization Problems
It is useful to realize that we can relax the relative interior hypothesis in Proposition 3.6.2 when we consider in the infimal convolution a mixture of polyhedral functions with proper convex functions. In fact, we show below that as a consequence of Proposition 3.6.2, the relative interior on the domains of given polyhedral functions can simply be replaced by the domains of these functions. Corollary 3.6.3 Let fi : Rn → R ∪ {+∞}, i = 1, . . . , p be a collection of proper convex functions such that {fi : i = 1, . . . , r} are proper polyhedral and assume that ∩ri=1 dom fi ∩pi=r+1 ri(dom fi ) = ∅. Then (f1 + · · · + fp )∗ (u) = inf{f1∗ (x1 ) + · · · + fp∗ (xp ) | x1 + · · · + xp = u}, where for each u the infimum is attained whenever it is finite. Proof. The proof is based on Proposition 3.6.2 and a simple convex analysis argument. It is enough to prove the result for only two functions, since the general result follows by simple induction. Thus, let f1 be proper polyhedral and f2 be proper convex such that dom f1 ∩ ri dom f2 = ∅. Let E := aff(dom f2 ). Then the latter intersection property clearly implies that ri(E ∩ dom f1 ) ∩ ri dom f2 = ∅. Define g := f1 + δE . Then g is proper convex polyhedral (as the sum of proper polyhedral functions; cf. Chapter 1), with dom g = E ∩ dom f1 , so that one has ri(dom g) ∩ ri dom f2 = ∅.
(3.40)
We also note that by definition of g one has g + f2 = f1 + δE + f2 = f1 + f2 . The rest of the proof consists in applying twice Proposition 3.6.2. Indeed, under (3.40), we obtain (g +f2 )∗ = g ∗ 2f2∗ with the infimum attained in the infimal convolution. But since both δE and f1 are proper polyhedral convex, ∗ ). Since here one obviously has and thus lsc, on has (f1 + δE )∗ = cl(f1∗ 2δE ∗ is proper polyhedral with dom f1 ∩ E = ∅, then by Corollary 3.5.4, f1∗ 2δE the infimum attained in the infimal convolution, so that we have obtained ∗ . Thus one has that g ∗ = (f1 + δE )∗ = f1∗ 2δE ∗ (f1 + f2 )∗ = (g + f2 )∗ = f1∗ 2δE 2f2∗ = f1∗ 2f2∗ ,
with the infimum attained throughout in the infimal convolution opera∗ tions, and in the last equality above we use the fact δE 2f2∗ = (δE + f2 )∗ = ∗ f2 , which holds, once again by Proposition 3.6.2, since ri(aff dom f2 ) ∩ 2 ri dom f2 = aff(dom f2 ) ∩ ri dom f2 is clearly nonempty. Subdifferential Calculus Theorem 3.6.1 and Proposition 3.6.1 have important implications for computing the subdifferential of a composite function with a linear map and
3.6 Dual Operations and Subdifferential Calculus
111
the subdifferential of the sum of a finite collection of convex functions. In turn, these implications lead to important results for the calculus rules concerning normal cones. Theorem 3.6.2 Let g : Rm → R ∪ {+∞} be a proper convex function, let A be a linear map from Rn to Rm , and let h = g ◦ A. Then ∂h(x) ⊃ A∗ ∂g(Ax), ∀x ∈ Rn . Furthermore, if Az ∈ ri dom g for some z ∈ Rn , then the inclusion becomes an equality. Proof. Let v ∈ A∗ ∂g(Ax). Then there exists y ∈ ∂g(Ax) such that v = A∗ y, and for every u ∈ Rn we have h(u) = (g ◦ A)(u) = g(Au) ≥ g(Ax) + +y, Au − Ax = h(x) + v, u − x, which means that v ∈ ∂h(x). Conversely, let v ∈ ∂h(x). Then by Proposition 1.2.18, we have g(Ax)+(gA)∗ (v) = x, v, with (gA)∗ (v) ∈ R. Suppose now that there exists some z with Az ∈ ri dom g. Then we can apply Theorem 3.6.1, so that there exists some y satisfying A∗ y = v, (gA)∗ (v) = g ∗ (y). Thus, the formula g(Ax) + (gA)∗ (v) = x, v reduces to g(Ax) + g ∗ (y) = Ax, y, and hence by Proposition 1.2.18, one has y ∈ ∂g(Ax), and 2 therefore v ∈ A∗ ∂g(Ax). The next result is often referred to as the Theorem of Moreau-Rockafellar. Theorem 3.6.3 Let fi : Rn → R ∪ {+∞}, i = 1, . . . , p, be a collection of proper convex functions such that the sets ri(dom fi ), i = 1, . . . , p, have a common point. Then ∂(f1 + · · · + fp ) = ∂f1 + · · · + ∂fp . If some of the functions are polyhedral, say {fi | i = 1, . . . , r}, then the equality holds under the weaker condition ∩ri=1 dom fi ∩pi=r+1 ri(dom fi ) = ∅. Proof. By Proposition 1.2.19 we already have the inclusion ∂f1 +· · ·+∂fp ⊂ ∂(f1 +· · ·+fp ), and thus it remains to prove the reverse inclusion. To achieve this, we proceed exactly in the same manner as in the proof of Theorem 3.6.2, but here we invoke Proposition 3.6.2 instead of Theorem 3.6.1. Similarly, we leave the details to the reader to verify that the polyhedral case follows using similar arguments to those of Theorem 3.6.2, and with the help of Corollary 3.6.3. 2 We note that in the important particular case of two functions, i.e., when p = 2, the equation above holds and takes the simpler form 0 ∈ ri(dom f1 − dom f2 ) =⇒ ∂(f1 + f2 ) = ∂f1 + ∂f2 .
112
3. Existence and Stability in Optimization Problems
Let C ⊂ Rn be a nonempty closed set. Then since for each x ∈ Rn the normal cone NC (x) at x is the subdifferential of the indicator function δC (x) of C at x, the latter result leads to the following useful formula for a finite collection of normal cones. n Corollary 3.6.4 Let {Ci }m i=1 be a collection of closed convex sets in R m whose relative interiors have a point in common and let C = ∩i=1 Ci . Then
∀x ∈ C, NC (x) =
m
NCi (x).
i=1
If some of the sets, say {Ci | i = 1, . . . , r}, are polyhedral, then the equation holds under the weaker assumption ∩ri=1 (Ci ) ∩m i=r+1 ri Ci = ∅. Proof. Apply Theorem 3.6.3 with fi = δCi , recalling that with C := ∩m i=1 Ci m one has δC = i=1 δCi . 2
3.7 Additional Results in the Convex Case Suppose that an ordinary convex program is assumed only to have a finite infimum. Then two natural questions arise. For which class of functions is the optimum attained? For which class of functions is the marginal function of this convex program with (vertical) perturbations lsc? The second question is of key importance in the context of duality theory (see Chapter 5). We shall see that convex asymptotically level stable functions satisfy these two properties as a consequence of the main theorem proven below (Theorem 3.7.1). This theorem also implies two important corollaries. Consider a set described by a finite number of inequalities involving convex als functions. The first consequence says that the image of such a set is closed; the second consequence is a Helly-type theorem in that context and will be established in the next section.
Theorem 3.7.1 Let fi : Rn → R ∪ {+∞} be als and convex, i = 1, . . . , m with a common effective domain C := dom fi . Suppose there exist sequences {λki } converging to λi for i = 1, . . . , m such that m
lev(fi , λki ) = ∅.
i=1
Then
m i=1
lev(fi , λi ) is nonempty.
3.7 Additional Results in the Convex Case
113
Proof. The proof is by induction on m. Let m = 1. Let xk ∈ lev(f1 , λk1 ) be of minimal norm. If {xk } has a bounded subsequence, there exists at least a cluster point x, and since f1 is lsc, obviously f1 (x) ≤ λ1 . In the opposite case ¯. we can suppose without loss of generality that ||xk || → ∞, xk ||xk ||−1 → x Since xk ||xk || ||xk ||−1 f1 ≤ ||xk ||−1 λk1 , ||xk || passing to the limit, we obtain (f1 )∞ (¯ x) ≤ 0. If (f1 )∞ (¯ x) = 0, then since x ∈ lev(f1 , λk1 ) for k f1 is als, (3.15) is satisfied; i.e., for ρ > 0, xk − ρ¯ sufficiently large. But for k sufficiently large we also have x xk − ρ¯
= (1 − ρ||xk ||−1 )xk + ρ(xk ||xk ||−1 − x ¯) ≤ (1 − ρ||xk ||−1 )||xk || + ρxk ||xk ||−1 − x ¯ −1 ≤ ||xk || + ρ((xk ||xk || − x ¯ − 1).
(3.41)
Since xk is of minimal norm, it follows that ¯. 1 ≤ xk ||xk ||−1 − x
(3.42)
Passing to the limit we get 1 ≤ 0, which is impossible. As a consequence it x) < 0. Let x ∈ dom f1 . Since follows that (f1 )∞ (¯ x) ≤ f1 (x) + t(f1 )∞ (¯ x), f1 (x + t¯ x) < ε1 , and the result is it follows that for t sufficiently large f1 (x + t¯ proved. Now we assume by induction that the theorem holds for m ≤l. Consider the m case m = l + 1. Let xk be the solution of smallest norm in i=1 lev(fi , λki ). If the sequence {xk } has a bounded subsequence, there exists at least a cluster point x of this bounded subsequence, and since the functions fi are lsc, obviously fi (x) ≤ λi for each i. In the opposite case we can suppose without loss of generality that ||xk || → +∞,
xk ||xk ||−1 → x ¯.
By the same argument given in the first part of the proof we obtain x) ≤ 0 (fi )∞ (¯
∀i = 1, . . . , m.
(3.43)
We once again consider two cases. x) = 0 for each i. Since fi is als, it follows that for any ρ > 0, Case 1: (fi )∞ (¯ we have for k sufficiently large x∈ xk − ρ¯
m i=1
lev(fi , λki ).
114
3. Existence and Stability in Optimization Problems
Since xk is of minimal norm on this set, using (3.41) we obtain again (3.42), and passing to the limit, we get a contradiction. x) < 0. Without loss of generality, Case 2: There exists a j such that (fj )∞ (¯ l let j = l + 1. Since i=1 lev(fi , λki ) is nonempty, the induction hypothesis l implies that there exists some x ∈ i=1 lev(fi , λi ). Consider now the point l x + ρ¯ x. By (3.43) one has x ¯ ∈ ( i=1 lev(fi , λi ))∞ , and it follows that x) ≤ λi fi (x + ρ¯
∀i = 1, . . . , l
∀ρ > 0.
Furthermore, since dom fi is the same set for each i, it follows that x ∈ dom fl+1 , and we have x) ≤ fl+1 (x) + ρ(fl+1 )∞ (¯ x). fl+1 (x + ρ¯ Since (fl+1 )∞ (¯ x) < 0, it follows that for ρ sufficiently large we have fl+1 (x+ m x ∈ i=1 lev(fi , λi ). 2 ρ¯ x) < λl+1 , and for such ρ, x + ρ¯
n Corollary m 3.7.1 Let fi : R → R be als and convex for i =n 1, . . . , m . Set C = i=1 lev(fi , 0), and let A be a linear map defined on R . Then if C is nonempty, A(C) is closed.
Proof. Consider a convergent sequence {yk } ∈ A(C) with limk→∞ yk = y. Then there exists xk ∈ C such that yk = Axk . Set f0 (x) = y − Ax2 ,
k = f0 (xk ).
Then k → 0+ and f0 is als. Furthermore, for each k the system f0 (x) ≤ k ,
fi (x) ≤ 0
∀i = 1, . . . , m
admits a solution xk . Using Theorem 3.7.1 it follows that the system f0 (x) ≤ 0,
fi (x) ≤ 0
∀i = 1, . . . , m
admits a solution, which is equivalent to saying that y ∈ A(C).
2
We now turn to another application of Theorem 3.7.1 on the marginal function of a convex program. Let fi : Rn → R ∪ {+∞}, i = 0, 1 . . . , m, be lsc and proper functions. For y ∈ Rm consider the optimization problem (P (y))
h(y) = inf{f0 (x) | x ∈ F (y)},
where F (y) = {x | fi (x) ≤ yi optimal solutions of P (y).
∀i = 1, . . . , m}. Denote by S(y) the set of
3.7 Additional Results in the Convex Case
115
Corollary 3.7.2 Let fi : Rn → R ∪ {+∞} be als and convex for i = 0, . . . , m with common effective domain C = dom fi . Suppose there exists y such that h(y) is finite. Then the following properties hold: (a) The optimal set S(y) is nonempty. (b) h is proper and convex. (c) dom h is closed and h is lsc. Proof. (a) Let {νk } be a positive sequence converging to 0+ and consider the set Sk = F (y) ∩ lev(f0 , h(y) + νk ). By definition of h, Sk is nonempty, and then by Theorem 3.7.1 it follows that S(y) is nonempty. (b) Let us prove first that h is lsc at y. Let {εki }, i = 1, . . . , m, be sequences of reals converging to zero and εk = (εk1 , . . . , εkm )T . We have to prove that h(y) ≤ lim inf h(y + εk ). k→∞
(3.44)
Without loss of generality we can suppose that F (y + εk ) = ∅ and that lim inf h(y + εk ) = lim h(y + εk ) = α. k→∞
k→∞
Suppose that (3.44) is not satisfied, i.e., α < h(y). Then we consider two cases. Case 1: α is finite. Then h(y + εk ) is finite for k sufficiently large, and by part (a) the system of inequalities f0 (x) ≤ α + (h(y + εk ) − α), fi (x) ≤ yi + εki , i = 1, . . . , m, is nonempty. Again by Theorem 3.7.1 it follows that the system of inequalities f0 (x) ≤ α, fi (x) ≤ yi , i = 1, . . . , m, is nonempty. Since α < h(y), this is impossible. Case 2: α = −∞. Let τ < h(y). Then for k sufficiently large the system of inequalities f0 (x) ≤ τ, fi (x) ≤ yi + εki , i = 1, . . . , m, is nonempty. Using Theorem 3.7.1 again, it follows that there exists xτ satisfying f0 (xτ ) ≤ τ, fi (xτ ) ≤ yi , i = 1, . . . , m. Passing to the limit as τ → −∞, it follows that h(y) = −∞, which yields a contradiction. Since the functions fi are convex, it follows by Theorem 1.2.2 that h is convex. As a consequence h is proper. Indeed, suppose that there exists y˜ such that h(˜ y ) = −∞. Then for y(λ) := y + λ(˜ y − y), λ ∈ (0, 1), h(y(λ)) = −∞, and passing to the limit as λ → 0, this contradicts the lower semicontinuity of h at y.
116
3. Existence and Stability in Optimization Problems
(c) Now let us prove that dom h is closed. This will also obviously imply that h is lsc on the whole space. Suppose the contrary. Then there exists u ∈ cl(dom h) \ dom h. Take y˜ ∈ ri dom h and set y(λ) = u + λ(˜ y − u) for 0 < λ < 1. Then y(λ) ∈ ri dom h and h(y(λ)) is finite. As a consequence the system of inequalities fi (x) ≤ yi (λ), i = 1, . . . , m, is nonempty. Passing to the limit as λ → 0+ it follows from Theorem 3.7.1 that F (u) is nonempty. Then since dom f0 = dom fi for each i = 1, . . . , m, it follows that h(u) < +∞, which is impossible. 2
3.8 The Feasibility Problem. Given a finite or infinite system of convex inequalities, a fundamental question is to characterize whether such a system is empty. This leads to what is usually called alternative theorems or Helly-type theorems. We shall derive here such types of results as a consequence of Theorem 3.7.1 for systems of inequalities involving als convex functions. Theorem 3.8.1 Let fi : Rn −→ R ∪ {+∞} be als and convex for i = 1, . . . , m with common effective domain C = dom fi . Then one and only one of the following alternatives holds: (a) There exists a vector x such that fi (x) ≤ 0 ∀i = 1, 2 . . . , m. (b) There exist nonnegative real numbers λi , not all zero, such that for some ε > 0 one has m λi fi (x) ≥ ε, ∀x. (3.45) i=1
Proof. Obviously, (a) and (b) cannot hold simultaneously. Assume that (a) does not hold. We shall prove that (b) holds, and that will establish the theorem. Let G = {y | ∃x such that fi (x) ≤ yi ∀i = 1, . . . , m}. Then G is convex, 0 ∈ G, and by Theorem 3.7.1 G is closed. As a consequence we can separate strongly {0} and G by a hyperplane. Thus, for some nonzero λ = (λ1 , . . . , λm ) and two real numbers α ∈ R and > 0 we have 0 = 0, λ ≤ α − ≤ α + ≤ λ, y
∀y ∈ G.
(3.46)
Clearly, this implies that for each i, λi is nonnegative (if λj were negative the last inequality would be violated with yj → ∞). Furthermore, λ = 0. Since α + ≥ 2, setting yi = fi (x) in (3.46) for each i ends the proof. 2
3.8 The Feasibility Problem.
117
Theorem 3.8.2 Let {fi }i∈I be an arbitrary collection of proper, lsc, convex functions defined on Rn . Assume that the functions fi have no common asymptotic direction except 0. Then one and only one of the following alternatives holds: (a) There exists a vector x ∈ Rn such that fi (x) ≤ 0 ∀i ∈ I. (b) There exist nonnegative real numbers λi , only finitely many nonzero, such that for some > 0 one has
λi fi (x) ≥ , ∀x ∈ Rn .
i∈I
If alternative (b) holds, the multipliers λi can be chosen so that at most (n + 1) of them are nonzero. Proof. As in the proof of Theorem 3.8.1, obviously (a) and (b) cannot hold simultaneously. Assume that (a) does not hold, then we have only to prove that (b) holds. Let f (x) = supi∈I fi (x). Then the function f is convex, lsc, and as can be seen later, without loss of generality we can assume that f is proper. As a consequence f∞ = supi∈I (fi )∞ . Therefore, under the hypothesis of the theorem, it follows that f∞ (d) > 0, ∀d = 0, so that f is coercive and argmin f = ∅. Furthermore, since (a) does not hold, it follows that f ∗ (0) = − inf f < 0. Now from Proposition 1.2.12(b) we have f ∗ = cl(conv{fi∗ | i ∈ I}). Let us prove that f ∗ (0) = conv{fi∗ | i ∈ I}(0). From Proposition 1.2.5 it is sufficient to verify that 0 ∈ ri dom(conv{fi∗ | i ∈ I}). Suppose the contrary holds. Then we can separate the point 0 from dom(conv{fi∗ | i ∈ I}), which contains ∪i∈I dom fi∗ . As a consequence there exists 0 = d such that d, x ≤ 0, ∀x ∈ dom fi∗ , ∀i ∈ I. But the last inequality means precisely that (fi )∞ (d) ≤ 0, ∀i ∈ I, which is clearly impossible in view of the assumed hypothesis of the Theorem on the lack of asymptotic direction for fi . Thus we have proved that f ∗ (0) = conv{fi∗ | i ∈ I}(0). Set ε := −f ∗ (0) > 0. Then one has conv{fi∗ : i ∈ I}(0) = −ε and (0, −ε) ∈ conv{epi fi∗ | i ∈ I}. By definition of the convex hull of functions, and invoking Caratheodory’s Theorem 1 real numbers λi that 1.1.1, there exist yi and λi ≥ 0 with atmost n + ∗ are nonzero such that i∈I λi yi = 0, i∈I λi fi (yi ) = −ε. Without loss of generality, we can suppose that the indices corresponding to the nonzero n + 1), and if we set zi = λi yi , since λi are just the first m integers (m ≤ m m ∗ (λi fi )∗ (zi ) = λi fi∗ (yi ), we then have i=1 zi = 0, i=1 (λi fi ) (zi ) = −ε. By definition of the infimal convolution (cf. Definition 1.2.6), this im∗ ∗ Proposition 1.2.10, one has plies m that (λ∗1 f1 ) 2 · · · 2(λm f∗ m ) (0) ≤ −ε. By ( j=1 λj fj ) (0) = cl((λ1 f1 ) 2 · · · 2(λm fm )∗ )(0) ≤ −, and the latter inm equality is equivalent to inf x j=1 λj fj (x) ≥ ε, proving the desired result. 2
118
3. Existence and Stability in Optimization Problems
3.9 Notes and References Existence and stability of optimal solutions in extremum problems and conditions under which such solutions exist and change under perturbation of the data is a topic of fundamental importance in several branches of nonlinear and variational analysis. Existence of optimal solutions under level boundedness is the usual assumption in most optimization problems, and all the results in Section 3.1 are classical and can be found in [119] and [123]. The definition of asymptotically directional constant and weakly coercive functions was given by Auslender [18], and Proposition 3.2.1 takes its arguments in Rockafellar’s book [119, Corollary 27.3.1]. The Characterization of weakly coercive convex functions given in Proposition 3.2.2 and the equivalence between (a) and (b) in Theorem 3.2.1 were given by Auslender in [12] and [18]. The equivalence between (b) and (c) in this theorem is new, while the equivalence between (c) and (d) is classical. Proposition 3.2.3 is due to Goberna and Lopez [77], while the rest of the section is from the work of Auslender, Cominetti, and Crouzeix [10]. Asymptotically level stable functions, a terminology coined in this book, were introduced by Auslender in [18], and the results of Section 3.3 are taken from [18]. Existence results (without the level boundedness assumption) for the important special case of quadratic optimization originated with the work of Frank and Wolfe [73], and several extensions can be found in the works of Perold [109]. Other important contributions can be found in Rocakfellar [119], Belousov [25], and Baiocchi, Buttazo, Gastaldi and Tomarelli [22] and references therein. For recent extensions and variants in Banach spaces, we refer the reader to the recent work of Penot [108]. The necessary and sufficient condition given in Theorem 3.4.1 is due to [12], but the proof technique used for the sufficiency part is essentially taken from [22]. Likewise, Corollary 3.4.1 is from [22], while Corollaries 3.4.2–3.4.3 are from [18]. For classical results on stability in parametric optimization we refer to the monographs of Bank et al. [23], Belousov [25], and Kummer [84]. For a recent and comprehensive work on the analysis of perturbed optimiztion problems and related second-order optimality conditions, a topic not covered here, we refer the reader to the book of Bonnans and Shapiro [37] and references therein. Most of the stability results given in Section 3.5 are given in Auslender [12], while the results concerning the nonconvex coercive case was first established by Zalinescu [135], and the results concerning the weakly coercive convex case can be found in Rockafellar’s book [119]. The material on subdifferential calculus of Section 3.6 is classical and can be found in [119]. The additional results given for the convex case in Section 3.7 and Theorem 3.8.1 are due to Auslender [18], while Theorems 3.7.1–3.8.1 were first derived for quadratic convex functions in [95] and [96], and Theorem 3.8.2 is classical and can be found in [119].
4 Minimizing and Stationary Sequences
Solving an optimization problem usually consists in generating a sequence by some numerical algorithm. The key question is then to show that such a sequence converges to some solution as well as to evaluate the efficiency of the convergent procedure. In general, the coercivity hypothesis on the problem’s data is assumed to ensure the asymptotic convergence of the produced sequence. For convex minimization problems, if the produced sequence is stationary, i.e., the sequence of subgradients approaches zero, it is interesting to know for what class of functions we can reach convergence under a weaker assumption than coercivity. This chapter introduces the concept of well-behaved asymptotic functions, which in turn are linked to the problems of error bounds associated with a given subset of a Euclidean space. A general framework is developed around these two themes to characterize asymptotic optimality and error bounds for convex inequality systems.
4.1 Optimality Conditions in Convex Minimization A basic minimization principle is the well-known Fermat principle, which states that for a continuously differentiable function f , a necessary condition for x ¯ ∈ Rn to be a local minimum of the function f is that the gradient of f at x ¯ be equal to zero, i.e., ∇f (¯ x) = 0. This condition becomes sufficient whenever the function is also assumed convex. This result can be extended to handle more general problems, leading to the abstract Fermat principle.
120
4. Minimizing and Stationary Sequences
Theorem 4.1.1 Let f0 : Rn → R ∪ {+∞} be a proper, lsc, convex function, C ⊂ Rn a closed convex set, and consider the minimization problem ¯ minimizes f0 inf{f0 (x) | x ∈ C}. Suppose that ri dom f0 ∩ ri C = ∅. Then x on C if and only if 0 ∈ ∂f0 (¯ x) + NC (¯ x), or equivalently, if and only if x) such that x − x ¯, g ≥ 0, ∀x ∈ C. ∃g ∈ ∂f0 (¯ ¯ minimizes f on Rn , and from Proposition Proof. Let f := f0 + δC . Then x 1.2.17, this is equivalent to 0 ∈ ∂f (¯ x). Since we assume that ri dom f0 ∩ ri C x) + ∂δC (¯ x) = is nonempty, using Theorem 3.6.3 this means that 0 ∈ ∂f0 (¯ x)+NC (¯ x). The latter inclusion is equivalent to saying that there exists ∂f0 (¯ x) such that −g ∈ NC (¯ x), and by definition of the normal cone, g ∈ ∂f0 (¯ the second assertion follows. 2 We now specialize the abstract Fermat principle whenever the constraint set C is described by a finite number of convex and affine inequalities to obtain necessary and sufficient optimality conditions, often called in the literature the Karush–Kuhn–Tucker (KKT for short) theorem for convex programs. We thus consider the convex program (P)
inf{f0 (x) | x ∈ C},
with C = ∩m i=1 Ci , where the closed convex sets Ci are defined by Ci = {x | fi (x) ≤ 0, i = 1, . . . , r, ai , x ≤ αi , i = r + 1, . . . , m} where the functions fi : Rn → R∪{+∞}, i = 0, . . . , r, are supposed proper, convex, and lsc and where ai ∈ Rn , αi ∈ R are given. Unless otherwise specified, throughout this section we make the following standard regularity assumptions: (i) dom f0 ⊂ dom fi , ri dom f0 ⊂ ri dom fi , i = 1, . . . , m. x) < 0, ∀i = 1, . . . , r. (ii) ∃ˆ x ∈ ri dom f0 such that fi (ˆ Note that (i) is always satisfied whenever the functions fi are finite valued, while (ii) is a constraint qualification on the nonaffine constraints fi , usually called the Slater condition, which is needed for the application of subdifferential calculus rules. Theorem 4.1.2 Consider the convex optimization problem (P) described above and satisfying the hypotheses (i) and (ii). Then x ¯ ∈ C minimizes x), gi ∈ ∂fi (¯ x), λi ≥ 0 ∀i ∈ f0 on C if and only if there exist g0 ∈ ∂f0 (¯ x) = 0}, and µj ≥ 0 for j ∈ J(¯ x) := {j ∈ [r + 1, m] | I(¯ x) := {i ∈ [1, r] | fi (¯ aj , xj = αj } such that g0 +
i∈I(¯ x)
λi gi +
j∈J(¯ x)
aj µj = 0.
(4.1)
4.1 Optimality Conditions in Convex Minimization
121
Proof. We apply Theorem 4.1.1 with C = ∩m i=1 Ci , recalling that the subdifferential calculus rules given in Theorem 3.6.3 can be applied by hypothesis m x) = i=1 NCi (¯ x), and from Proposition 1.2.21, for (i). Thus one has NC (¯ i ≥ r + 1 we obtain R+ ai if ai , x ¯ = αi , x) = NCi (¯ otherwise. Rn For i = 1, . . . , r, from Slater’s condition using Proposition 1.2.22 it follows that R+ ∂fi (¯ x) if fi (¯ x) = 0, NCi (¯ x) = x) < 0. {0} if fi (¯ 2
Using these formulas assertion (4.1) follows.
Approximate Solutions Let f : Rn → R ∪ {+∞} be a proper lsc function with inf f > −∞. In general, as we have seen in Chapter 3, we need additional conditions to ensure that the infimum will be attained, but there are always approximatively ε solutions, i.e., points xε for ε > 0 satisfying inf f ≤ f (xε ) ≤ inf f + ε.
(4.2)
This leads to the notion of ε-subdifferential of f at x. Definition 4.1.1 Let f : Rn → R ∪ {+∞} be lsc. For any x ∈ dom f and any ε > 0 the ε-subdifferential of f at x, denoted by ∂ε f (x), is defined by ∂ε f (x) = {x∗ | f (y) ≥ f (x) + x∗ , y − x − ε
∀y}.
Obviously, whenever ε = 0, one has ∂0 f (x) = ∂f (x). In the convex case the ε-subdifferential enjoys many properties. In particular, as we shall see, it is never empty on dom f , and ε-subdifferential calculus rules can be developed. In the convex case we obtain immediately from the definition the following approximate optimality condition. Theorem 4.1.3 Let f : Rn → R ∪ {+∞} be lsc, proper, convex, and let ε > 0. Then for each x ∈ dom f, ∂ε f (x) is a nonempty closed convex set. Furthermore, if inf f > −∞, then a necessary and sufficient condition for xε to satisfy (4.2) is that 0 ∈ ∂ε f (xε ). Proof. From the definition of the ε-subdifferential we get ∂ε f (x) = {x∗ | f ∗ (x∗ ) + f (x) − x∗ , x ≤ ε },
122
4. Minimizing and Stationary Sequences
from which it follows that ∂ε f (x) is closed and convex. To see that ∂ε f (x) is nonempty, we argue by contradiction. Suppose ∂ε f (x) = ∅. Let g(x∗ ) = f ∗ (x∗ ) − x∗ , x. Then we have g(x∗ ) > ε − f (x) ∀x∗ , so that −g ∗ (0) ≥ ε − f (x). But g ∗ (y) = f (x + y), which leads to a contradiction. Finally, the second part of the theorem follows immediately from the definition of the ε-subdifferential. 2 The next result, called the Ekeland variational principle, is a fundamental tool with respect to several approximating approaches to optimization problems. Note that convexity is no longer needed, except for the last part of the result. Theorem 4.1.4 Let f : Rn → R ∪ {+∞} be an lsc, proper function with inf f > −∞. Let ε > 0 and suppose xε ∈ Rn satisfies f (xε ) ≤ inf f + ε. Then given λ > 0, there exists a point x ¯ ∈ dom f such that: (a) f (¯ x) ≤ f (xε ). (b) ¯ x − xε ≤ λ. ¯ ∀x = x ¯. (c) f (¯ x) < f (x) + ελ−1 x − x (d) With f assumed convex, there exists g ∈ ∂f (¯ x) satsisfying g ≤ ελ−1 . Proof. Let h(x) := f (x) + ελ−1 x − xε . Then h∞ (d) ≥ f∞ (d) + ελ−1 d, and since inf f > −∞, one has f∞ (d) ≥ 0 and hence h∞ (d) > 0 ∀d = 0, proving that h is a coercive function and argmin h is nonempty. Let x ¯∈ argmin h. Then x − xε ≤ f (x) + ελ−1 x − xε , ∀x ∈ Rn , f (¯ x) + ελ−1 ¯
(4.3)
and by setting x = xε in the last inequality one obtains x − xε ≤ f (xε ) − f (¯ x) ≤ f (xε ) − inf f ≤ ε, ελ−1 ¯ from which (a) and (b) follow. Part (c) is an immediate consequence of ¯ + ¯ x − xε , ∀x = x ¯. (4.3) and the triangle inequality x − xε < x − x To prove (d), since f is assumed convex, it follows from (c) that 0 ∈ ∂h(¯ x) ¯. Since dom · −¯ x = Rn , by Theorem 3.6.3 with h(x) = f (x) + ελ−1 x − x x)(¯ x) = ∂f (¯ x) + ελ−1 B, where we thus have ∂h(¯ x) = ∂f (¯ x) + ελ−1 ∂( · −¯ in the last equality we have used the fact ∂.(0) = B, with B the closed unit ball, and (d) is proved. 2 As an immediate consequence of Theorem 4.1.4 we obtain that x ¯ is a kind of “almost critical point” for the differentiable minimization problem inf{f (x) | x ∈ Rn }.
4.1 Optimality Conditions in Convex Minimization
123
Corollary 4.1.1 Let f be a proper lsc function on Rn such that inf f > −∞ and let λ > 0. Suppose f : Rn → R is differentiable. Then for each x) ≤ ελ−1 . ε > 0, there exists x ¯ ∈ Rn such that ∇f (¯ Proof. Invoking Theorem 4.1.4, there exists x ¯ ∈ Rn such that f (¯ x) ≤ f (x) + ελ−1 x − x ¯, ∀x ∈ Rn .
(4.4)
Let d ∈ Rn and t > 0 be arbitrary. Take x = x ¯ + td in (4.4), to obtain x +td)−f (¯ x)) ≥ −ελ−1 d, and passing to the limit as t → 0+ , since t−1 (f (¯ f is assumed differentiable, we get d, ∇f (¯ x) ≥ −ελ−1 d ∀d ∈ Rn . Since the last inequality is also true when d is replaced by −d, we thus obtain x) ≤ |d, ∇f (¯ x)| ≤ ελ−1 d, ∀d ∈ Rn , from which it follows that ∇f (¯ ελ−1 . 2 As an application we obtain the Bronsted–Rockafellar theorem. Theorem 4.1.5 Let f : Rn → R ∪ {+∞} be lsc, convex, and proper. Let ¯ ∈ dom f x ∈ dom f, ε > 0, λ > 0. Then for each g ∈ ∂ε f (x) there exist x and g¯ ∈ ∂f (¯ x) such that x − x ¯ ≤ λ, g − g¯ ≤ ελ−1 . Proof. Let h(u) = f (u) − u, g. Then since g ∈ ∂ε f (x), we have h(u) ≥ f (x) − x, g − ε = h(x) − ε
∀u,
from which it follows that h is lower bounded and that x is an ε-minimizer for h. Using Theorem 4.1.4(b)–(d) it follows that there exist x ¯ ∈ dom f and g¯ ∈ ∂f (¯ x) such that x − x ¯ ≤ λ, g − g¯ ≤ ελ−1 . 2 As announced in Section 3.1, using Theorem 4.1.4, we shall prove now that a function that is bounded below and satisfies the Palais–Smale condition is level bounded. Recall that for a C 1 function f , a sequence {xk } such that {f (xk )} is bounded and limk→∞ ∇f (xk ) = 0, is said to be a Palais–Smale (PS) sequence for f . Furthermore, f is said to satisfy the Palais–Smale condition if each (PS) sequence is bounded. Theorem 4.1.6 Suppose that f : Rn −→ R is a C 1 function bounded below on Rn that satisfies the Palais–Smale condition. Then f is level bounded. Proof. Define the following sets and functions, for a fixed k ∈ N: Xk αk
:= {x | x ≥ k }, := inf{fk (x) |x ∈ Rn },
fk (x) = f (x) + δXk (x), βk+1 = αk+1 − αk .
Then the sequence {αk } is a nondecreasing sequence of real numbers, and for each k, βk ≥ 0. Let yk ∈ Xk+1 be such that f (yk ) ≤ αk+1 + k −1 =
124
4. Minimizing and Stationary Sequences
αk + k −1 + βk . Then since fk is lsc, and proper with inf fk > −∞, we can use Ekeland’s Theorem 4.1.4, and for λ ∈ ]0, 1[ we can find xk ∈ Xk such that f (xk ) ≤ f (yk ), xk − yk ≤ λ. (4.5) But since xk ≥ yk −λ > (k+1)−1 = k, we have that fk is differentiable at xk with ∇f (xk ) = ∇fk (xk ), so that by Theorem 4.1.4(d), ∇f (xk ) ≤ λ−1 (βk + k −1 ).
(4.6)
Now suppose that f is not level bounded, i.e., there exists v > inf f such that f (x) ≤ v is unbounded. Then it follows that the sequence {αk } is bounded above by v, so that βk → 0+ , and from (4.5) and (4.6) it follows that {xk } is an unbounded (PS) sequence, which is impossible, since f satisfies the Palais–Smale condition. 2
4.2 Asymptotically Well-Behaved Functions In this section we consider the problem (P) inf f = inf{f (x) : x ∈ Rn }, where f : Rn → R∪{+∞} is lsc, convex, and proper. We are thus interested in finding a minimizing sequence {xk } for problem (P), that is, a sequence satisfying lim f (xk ) = inf f. k→∞
Usually, the sequence {xk } is produced by some numerical algorithm to solve (P). The standard approach to proving such a limiting result is based on the assumption that the sequence {xk } is bounded or that the objective function is coercive. For example, when f ∈ C 1 (Rn ), if the sequence generated by some algorithm is bounded, then whenever ∇f (xk ) → 0 one obtains f (xk ) → inf f as k → ∞. This is particularly the case when f is assumed coercive. However, these kinds of assumptions on the function f might often be violated, and therefore it is important to know what happens and how to handle more general situations. Example 4.2.1 Consider the function f defined by f (x1 , x2 ) =
x21 2x2
0 ∞
if x2 > 0, if(x1 , x2 ) = (0, 0), otherwise.
4.2 Asymptotically Well-Behaved Functions
125
It can be easily seen that f is the support functional of the closed convex set C, where x2 C = {x ∈ R2 | 1 + x2 ≤ 0}. 2 Then it follows that f is lsc, convex, and proper. Furthermore, inf f = 0, and the infimum of f is attained at (0,0). Let xk = (k, k2 ). Then 1 1 1 , − 2 → 0. f (xk ) = , ∇f (xk ) = 2 k 2k As a consequence, the sequence {xk } is a stationary sequence but not minimizing, and thus one can say that the function f is not well behaved “at large.” This example indicates that even for the minimization of a proper lsc convex function on a closed convex set, a quite standard model optimization, difficulties arise. This motivates the need for identifying other properties that should be shared by convex functions to guarantee the existence of a minimizing sequence. The following definition introduces the essential notion of asymptotically well-behaved (awb) functions. We first recall the formal definition of stationary and minimizing sequences. Definition 4.2.1 Let f : Rn → R ∪ {+∞} be a proper, lsc, convex function. A sequence {xk } is said to be stationary if there exists ck ∈ ∂f (xk ) with ck → 0. The sequence is said to be minimizing if limk→∞ f (xk ) = inf f . Definition 4.2.2 A proper, lsc, convex function f : Rn → R ∪ {+∞} is said to be asymptotically well behaved (awb) if each stationary sequence is minimizing. As we shall see later, the problem to know whether a stationary sequence is minimizing is related to the problem of global error bound. Let λ ≥ inf f . This problem consists in deriving necessary and sufficient conditions for the existence of a constant γ(λ) > 0 such that dist(x, lev(f, λ)) ≤ γ(λ)f (x)+ ∀x ∈ Rn ,
(4.7)
where dist(x, C) denotes the usual Euclidean distance of x to C, and a+ = sup(a, 0). An inequality of this kind is called a global error bound for the level set lev(f, λ), see Section 4.3 for more details. Going back to the nonasymptotically well-behaved√function given in Example 4.2.1, one can verify that dist(xk , lev(f, 0)) = k 4 + k 2 → ∞, and for λ = 0 there is no constant γ(λ) > 0 for which (4.7) holds. Characterization of Asymptotically Well-Behaved Functions We are now interested in characterizing awb functions in explicit terms of the data and information given on the function f . For that purpose, let us
126
4. Minimizing and Stationary Sequences
introduce the following constants, which will play a central role. For any λ ≥ inf f , define l(λ) = inf
f (x)>λ
f (x) − λ , r(λ) = inf dist(x, lev(f, λ)) f (x)=λ
and t(λ) = k(λ)
=
inf
f (x)=λ
inf
f (x)=λ
inf
x∗ ∈∂f (x)
x∗ ,
d f x; , d 0 =d∈Nlev(f,λ) (x) x∗ inf f x; . x∗ 0 =x∗ ∈∂f (x) inf
We begin with two useful technical results needed in our analysis. Lemma 4.2.1 Let C be a nonempty closed convex set in Rn and PC the usual Euclidean projection on C. Let x ∈ C, t > 0, d ∈ NC (x). Then PC (x + td) = x
∀t > 0 .
Proof. Set x(t) := PC (x+td). Since x(t) = arg min 12 x + td − y2 | y ∈ C , it follows from Theorem 4.1.1 that (x+td−x(t)) ∈ NC (x(t)), which is equivalent to x(t)−(x+td), u−x(t) ≥ 0 ∀u ∈ C. Set u = x in this inequality. Then since d ∈ NC (x), it follows that x − x(t)2 ≤ tx(t) − x, d ≤ 0. 2 Lemma 4.2.2 Let f : Rn → R ∪ {+∞} be a proper, lsc, and convex function. Let λ ≥ inf f and x ∈ lev(f, λ) (assumed nonempty when λ = inf f ) such that f (x) < ∞. Let y be the projection of x onto lev(f, λ). Then we have: (a) f (y) = λ, x − y ∈ Nlev(f,λ )(y), (b) when λ > inf f if y ∈ ri dom f there exists some α > 0 such that α(x − y) ∈ ∂f (y). Proof. Let us show first that f (y) = λ. From feasibility we have f (y) ≤ λ. On the other hand, for each t ∈ (0, 1) the point x + t(y − x) does not belong to lev(f, λ), and therefore λ < f (x + t(y − x)) ≤ (1 − t)f (x) + tf (y), which after letting t → 1 gives f (y) ≥ λ. From Theorem 4.1.1 one has y = Plev(f,λ) (x) if and only one has x − y ∈ Nlev(f,λ) (y), and (a) is proved. In the case where λ > inf f if y ∈ ri dom f , then by Proposition 1.2.22 it follows that Nlev(f,λ) (y) = R+ ∂f (y), and since x = y, there exists an α > 0 such that α(x − y) ∈ ∂f (y). 2
4.2 Asymptotically Well-Behaved Functions
127
Theorem 4.2.1 Let f : Rn → R ∪ {+∞} be an lsc, proper, and convex function. Then l(λ) = t(λ) ∀λ > inf f, (4.8) and (4.8) holds also for λ = inf f if argmin f = ∅. Furthermore, when λ > inf f if for each x such that f (x) = λ we have x ∈ ri dom f , then k(λ) = t(λ).
(4.9)
Proof. We must prove the two inequalities l ≥ t and t ≥ l. The first amounts to saying that for every x such that f (x) > λ we have t(λ) ≤
f (x) − λ , dist(x, lev(f, λ))
which follows from Lemma 4.2.2. In fact, it suffices to consider the case f (x) < ∞, and then taking the projection y of x onto the set lev(f, λ) we have x−y f (x) − f (y) t(λ) ≤ f y; , ≤ x − y dist(x, lev(f, λ)) from which the result follows, since f (y) = λ. For the converse inequality we must show that given x with f (x) = λ and given d ∈ Nlev(f,λ) (x), d = 0, we have l(λ) ≤ f (x; d). But from Lemma 4.2.1 we have Plev(f,λ) (x + td) = x ∀t > 0, so that dist(x + td, lev(f, λ)) = td > 0. We deduce f (x + td) − λ f (x + td) − f (x) = ≥ l(λ), td dist(x + td, lev(f, λ)) so that letting t → 0+ we get the desired conclusion. Now suppose that for each x such that f (x) = λ we have x ∈ ri dom f . Then from Proposition 1.2.22 it follows that R+ ∂f (x) = Nlev(f,λ) (x), from which (4.9) follows. 2
Proposition 4.2.1 Let f : Rn → R ∪ {+∞} be an lsc, proper, convex function such that f (x) > inf f ⇒ x ∈ ri dom f . Then: (a) k(λ ) ≥ r(λ ) ≥ l(λ) ≥ r(λ) ∀λ > λ > inf f . (b) For λ > inf f we have the following characterizations: x∗ ∗ inf f x; ∗ r(λ) = inf ∗ inf x , l(λ) = inf . x f (x)≥λ x ∈∂f (x) f (x)≥λ 0 =x∗ ∈∂f (x)
128
4. Minimizing and Stationary Sequences
Proof. (a) Since for x∗ ∈ ∂f (x) one has x∗ , d ≤ f (x; d) ∀d ∈ Rn , using formula (4.8) it follows easily that l ≥ r. Thus it suffices to show that r(λ ) ≥ l(λ). Let us take x with f (x ) = λ and let x be the projection of x on lev(f, λ). Then we can use Lemma 4.2.2 and write for some α > 0, α(x − x) (x − x) f (x ) − f (x) l(λ) = t(λ) ≤ f x; . x; = f ≤ α(x − x) x − x x − x Hence, for every x∗ ∈ ∂f (x ) we get l(λ) ≤
x∗ , x − x ≤ x∗ , x − x
and the desired inequality follows. (b) This is an immediate consequence of the monotonicity of l and r. 2
Theorem 4.2.2 Let f : Rn → R ∪ {+∞} be lsc, proper, convex, and such that f (x) > inf f =⇒ x ∈ ri dom f . Then the following statements are equivalent: (a) f is asymptotically well behaved. (b) All stationary sequences {xk } with f (xk ) bounded satisfy f (xk ) → inf f. (c) r(λ) > 0 for all λ > inf f . (d) t(λ) > 0 for all λ > inf f . (e) l(λ) > 0 for all λ > inf f . Proof. The implication (a) =⇒ (b) as well as the equivalence among (c), (d), and (e) are obvious from the definition of an awb function and the previous results, respectively. To prove (b) =⇒ (c) we observe that otherwise there exists λ > inf f with r(λ) = 0, so that we can find sequences xk and x∗k ∈ ∂f (xk ) with x∗k → 0 and f (xk ) = λ > inf f , contradicting (b). The implication (c) =⇒ (a) follows similarly. If (a) does not hold, we can find a stationary sequence that is not minimizing. Extracting a subsequence, we can find λ > inf f and sequences xk and x∗k ∈ ∂f (xk ) such that f (xk ) ≥ λ and x∗k → 0. The alternative characterization of r given in Proposition 4.2.1 yields r(λ) = 0, contradicting (c). 2 Next, we provide a simple sufficient condition that guarantees that a function is awb. Proposition 4.2.2 Let f : Rn → R ∪ {+∞} be an lsc, proper, convex function. Suppose that 0 ∈ cl dom f ∗ or equivalently that there exists a vector d such that f∞ (d) < 0. Then f is awb.
4.2 Asymptotically Well-Behaved Functions
129
Proof. Since 0 ∈ cl dom f ∗ , there exists r > 0 such that c ≤ r ⇒ c ∈ dom f ∗ . Since dom ∂f ∗ ⊂ dom f ∗ ,
cn ∈ ∂f (xn ) ⇐⇒ xn ∈ ∂f ∗ (cn ),
as a consequence it appears that there cannot exist a stationary sequence. 2 We shall see now that a weakly coercive convex function is awb, but in this case we can say even more. Proposition 4.2.3 Let h : Rn → R ∪ {+∞} be a convex function with 0 ∈ ri dom h and let E be the affine hull of its domain and hE = h ◦ PE , where PE is the projection operator on E. Then for each sequence uk → 0 and u∗k ∈ ∂h(uk ) we have dist(u∗k , ∂h(0)) = dist(PE (u∗k ), ∂hE (0)) → 0 and h∗ (u∗k ) → −h(0) = inf h∗ . Proof. Since ∂h(uk ) is nonempty, we must have uk ∈ E, and from Proposition 3.2.3 we get PE (u∗k ) ∈ ∂hE (uk ). From Proposition 1.4.3 the upper semicontinuity of ∂hE (·) at 0 ∈ int dom hE yields at once dist(u∗k , ∂h(0)) = dist(PE (u∗k ), ∂hE (0)) → 0 as well as the boundedness of the sequence PE (u∗n ). Therefore, since uk ∈ E, we have h∗ (u∗k ) + h(uk ) = PE (u∗k ), uk → 0, and since h is continuous relative to ri dom h and in particular at 0, we conclude that h∗ (u∗k ) → −h(0), as claimed. 2
Corollary 4.2.1 Let f : Rn → R ∪ {+∞} be an lsc, proper, convex function and weakly coercive. Then f is awb. Moreover, if {xk } is a sequence such that dist(0, ∂εk f (xk )) → 0 where εk → 0+ , then: (a) f (xk ) → inf f . (b) dist(xk , argmin f ) → 0. Proof. Since f is weakly coercive, 0 ∈ ri dom f ∗ , and when ε = 0 the conclusion of the corollary is an immediate consequence of Proposition 4.2.3. Let us consider the general case and select x∗k ∈ ∂εk f (xk ) with x∗k → √ 0. x such that y − x ≤ εk Using Theorem 4.1.5 we may find yk and k k √ k and yk∗ ∈ ∂f (yk ) with yk∗ − x∗k ≤ εk . It follows that yk∗ → 0 and so applying this result (in the case εk = 0) to the sequence yk , we get
130
4. Minimizing and Stationary Sequences
dist(yk , argmin f ) → 0 and f (yk ) → inf f . From this it is clear that (a) must hold, since inf f ≤ f (xk ) ≤ f (yk ) − x∗k , yk − xk + εk , and assertion (b) follows immediately. 2 Dual Characterization of Well-Behaved Functions We now consider dual characterizations for a given lsc proper convex function to be awb. In order to derive such characterizations we have to compute the conjugate of the positive part of a convex function and also the conjugate of the distance function to a closed convex set. The next results provide the necessary technical tools. Lemma 4.2.3 Let f : Rn → R ∪ {+∞} be a proper, lsc, convex function such that r > inf f > −∞ and let C := {x | f (x) ≤ r}. Then σC (d) = min t(f ∗ (t−1 d) + r) = inf t(f ∗ (t−1 d) + r), ∀d = 0, t>0
t≥0
where 0f ∗ ( d0 ) stands for (f ∗ )∞ (d) = limt→0 t(f ∗ (t−1 d)). Proof. Define g = f − r and let L : (x, α) → x be the projection map. Then one can write C = L(epi g ∩ Rn × R− ), and by Proposition 1.3.3(e) it follows that σC (d) = σepi g∩(Rn ×R− ) (d, 0). Since inf g < 0, one has epi g ∩ int(Rn × R− ) = ∅, and thus applying Corollary 3.6.2 we obtain σC (d)
=
(σepi g 2σRn ×R− )(d, 0)
= =
(σepi g 2δ{0}×R+ )(d, 0) min σepi g (d, −s)
=
min sg ∗ (s−1 d) = min t(f ∗ (t−1 d) + r),
s≥0 s≥0
t≥0
where the first part of the last equality uses Lemma 2.5.2. 2 Lemma 4.2.4 Let f : Rn → R ∪ {+∞} be a lsc, convex, proper function with inf f > −∞ and let λ ∈ R. Then for all y ∈ Rn one has y y + ∗ + λ = min sf ∗ +λ , (4.10) ((f − λ) ) (y) = inf sf ∗ s s s∈]0,1] s∈[0,1] where
f + = sup(f, 0) and 0f ∗
z 0
stands for (f ∗ )∞ (z).
4.2 Asymptotically Well-Behaved Functions
131
∗
Proof. Since (f − λ) = f ∗ + λ, it is sufficient to prove (4.10) for λ = 0. For all y ∈ Rn , using Lemma 2.5.2 one has ∗
(f + ) (y) = σepi f + (y, −1) = σepi f ∩Rn ×R+ (y, −1). Since int(Rn × R+ ) ∩ epi f = ∅, we apply Corollary 3.6.2, and noting that σRn ×R+ (·) = δ{0}×R− (·), we obtain ∗
(f + ) (y) = σepi f 2(δ{0}×R− )(y, −1)
= min σepi f (y, −1 − t) t≤0
=
y y min sf ∗ ( ) = inf sf ∗ ( ). s s s∈(0,1]
s∈[0,1]
2 Lemma 4.2.5 Let C be a nonempty closed convex subset of Rn . Then the ∗ conjugate dist(·, C) is given by the formula ∗
∀y ∈ B,
dist(·, C) (y) = σC (y)
(4.11)
where B = {y : y ≤ 1}. Proof. Set g(u) = u. Then we have dist(·, C) = (δC 2g)(·). Furthermore, it can be easily seen that g ∗ = δB . Then using Proposition 3.6.1 it follows 2 that dist(·, C)∗ = σC + g ∗ , which is equivalent to (4.11). Theorem 4.2.3 Let f : Rn → R∪{+∞} be an lsc, proper, convex function such that f (x) > inf f ⇒ x ∈ ri dom f , and inf f > −∞. Then f is awb if and only if for any λ > inf f there exists α > 0 such that ((f − λ)+ )∗ (y) = σlev(f,λ) (y)
∀y ∈ αB.
(4.12)
Proof. We know by Theorem 4.2.2 that f is awb if and only if ∀λ > inf f there exists α > 0 such that f (x) ≥ λ + α dist(x, lev(f, λ)),
x∈ / lev(f, λ),
which is equivalent to +
(f − λ) ≥ α dist(·, lev(f, λ)). Dually, and using Lemma 4.2.5, this is equivalent to ∗
((f − λ)+ )∗ ≤ (α dist(x, lev(f, λ))) = σlev(f,λ) + δαB . Using Lemma 4.2.3 and Lemma 4.2.4 this is equivalent to y y + λ ≤ min sf ∗ +λ ∀y ∈ αB, min sf ∗ s≥0 s s s∈[0,1]
132
4. Minimizing and Stationary Sequences
which obviously implies y y + λ = min sf ∗ +λ min sf ∗ s≥0 s s s∈[0,1]
∀y ∈ αB,
i.e., exactly formula (4.12). 2 As we have seen, the position of the origin relative to the domain of f ∗ is of main importance. When 0 ∈ / cl dom f ∗ or when 0 ∈ ri dom f ∗ , then f is awb. In fact, when 0 belongs to the relative boundary, this requires a particular analysis. The correct notion is the following. Definition 4.2.3 Let f : Rn → R ∪ {+∞} be an lsc, proper, convex function. The domain of its conjugate dom f ∗ is said to be locally conical at 0 if there exists α > 0 such that R+ dom f ∗ ∩ αB ⊂ dom f ∗ . Theorem 4.2.4 Let f : Rn → R∪{+∞} be an lsc, convex, proper function with inf f > −∞. Assume that dom f ∗ is locally conical at 0 and that f ∗ is bounded above on the intersection of a neighborhood of 0 and dom f ∗ . Then f is awb. Proof. Let α > 0 be such that R+ dom f ∗ ∩ αB ⊂ dom f ∗ , and c > f ∗ (0) such that f ∗ (y) ≤ c ∀y ∈ dom f ∗ ∩ αB. Given ε ∈ ]0, 1[ and y ∈ dom f ∗ ∩ εαB, one has ε−1 y ∈ R+ dom f ∗ ∩ αB ⊂ dom f ∗ ∩ αB, so that f ∗ (y) ≤ (1 − ε)f ∗ (0) + εf ∗ (ε−1 y) ≤ (1 − ε)f ∗ (0) + εc, yielding lim supy→0,y∈dom f ∗ f ∗ (y) ≤ f ∗ (0). Since f ∗ is lsc, we get limy→0,y∈dom f ∗ f ∗ (y) = f ∗ (0). As a consequence we have for δ > 1, lim
y→0,y∈dom f ∗
δf ∗ (y) − f ∗ (δy) = (δ − 1)f ∗ (0).
(4.13)
Now let {xk } be a stationary sequence and let us prove that {xk } is minimizing. Let yk ∈ ∂f (xk ) with yk → 0. Then xk ∈ ∂f ∗ (yk ), and we have xk , (δ − 1)yk + f ∗ (yk ) ≤ f ∗ (δyk ), −f ∗ (0) ≤ f (xk ) = xk , yk − f ∗ (yk ). Thus we get −(δ − 1)f ∗ (0) ≤ (δ − 1)f (xk ) = xk , (δ − 1)yk − (δ − 1)f ∗ (yk ) ≤ f ∗ (δyk ) − δf ∗ (yk ), so that 0 ≤ (δ − 1)(f (xk ) + f ∗ (0)) ≤ f ∗ (δyk ) − δf ∗ (yk ) + (δ − 1)f ∗ (0), and then using (4.13) it follows that limk→∞ f (xk ) = −f ∗ (0) = inf f.
2
4.3 Error Bounds for Convex Inequality Systems
133
4.3 Error Bounds for Convex Inequality Systems An initial motivation for studying error bounds in mathematical programming arose from a practical consideration in the implementation of numerical methods for solving optimization problems. Furthermore, on the theoretical side, error bounds are playing a key role in the convergence rate analysis of these numerical methods. Given a subset F of Rn , an error bound is an inequality that bounds the distance from an arbitrary vector to F in terms of some easily computable residual function ρ, that is, a function that satisfies ρ(x) ≥ 0, ∀x ∈ Rn , and ρ(x) = 0 if and only if x ∈ F. Thus formally an error bound consists in establishing that for some positive constant θ one has dist(x, F) ≤ θρ(x), ∀x ∈ B ⊂ Rn , for some (often bounded) subset B of Rn , in which case the error bound is called local, while when B = Rn , the error bound is called a global error bound for the set F. In fact, we are also interested in finding global error bounds that are sharp, i.e., to identify the best possible value of the constant θ, and this will be the main objective of this section. The origin of error bounds takes its root in the well-known Hoffman’s result for a polyhedral set F := {x ∈ Rn : Ax ≤ b}, with given data A ∈ Rm × Rn , b ∈ Rm , which asserts that there exists θ > 0 depending only on the data A and such that dist(x, F) ≤ θ(Ax − b)+ , ∀x ∈ Rn . Note that in this result the best possible value of θ is not known. As we shall see later, this result will be obtained as a byproduct of our analysis, which includes sharp error bounds for the more general problem described by polyhedral sets and convex systems. Thus, throughout this section we concentrate on error bounds for a set F described by a finite number of convex and affine inequalities. Let r be a nonnegative integer, and we suppose that for i = r + 1, . . . , m we have fi (x) = ai , x − αi . When r > 0 we assume that fi : Rn → R ∪ {+∞}, i = 1, . . . , r are lsc, proper, convex functions with dom fi open. We will also use the following notation. Let F (x) = (f1 (x), . . . , fm (x))t ,
+ F + (x) = (f1+ (x), . . . , fm (x))t ,
with fi+ (x) = max(fi (x), 0), i = 1, . . . , m, and define Ci = {x : fi (x) ≤ 0},
C = D ∩m i=r+1 Ci ,
(4.14)
with D = Rn if r = 0, and D = ∩ri=1 Ci otherwise. In the following, ||| · ||| denotes a monotonic norm on Rn , that is to say which satisfies |xi | ≤ |yi |
i = 1, . . . , m
=⇒
|||x||| ≤ |||y|||.
134
4. Minimizing and Stationary Sequences
In particular, the p-norms · p with 1 ≤ p ≤ +∞ are monotonic norms. Our aim is to derive error bounds for the set C, and if possible sharp, i.e., to find a constant θ > 0, the “best” possible, such that dist(x, C) ≤ θ|||F + (x)|||
∀x ∈ Rn .
To this end, we shall assume Slater’s condition for C; i.e., if r > 0, then there exists x0 ∈ C such that fi (x0 ) < 0 for i = 1, . . . , r. We know (cf. Corollary 3.6.4, Proposition 1.2.21, and Proposition 1.2.22) that under this condition we have for each x ∈ C, λi ci (4.15) NC (x) = d | ∃λi ≥ 0, ci ∈ ∂fi (x), i ∈ I(x) with d = i∈I(x)
with I(x) = {i | fi (x) = 0} and ci = ai for i ≥ r + 1. For y ∈ C we denote
by (fi+ ) (y; d) the directional derivative of fi+ at y in the direction d. We define, M (y) = {d ∈ NC (y) such that ||d = 1},
(Fa+ ) (y; d)
=
((fi+ ) (y; d) | i ∈ I(y)),
and the following real numbers: l = inf
x∈C /
|||F + (x)||| , dist(x, C)
t=
inf
inf
y∈bd(C) d∈M (y)
|||(Fa+ ) (y; d)|||.
Lemma 4.3.1 Let C be a nonempty closed convex set in Rn . (a) For x ∈ / C and y ∈ bd(C), one has y = PC (x) if and only if M (y). (b) Let y ∈ bd(C) and d ∈ M (y). Then for each θ > 0 we have x(θ) = y + θd ∈ / C,
x−y x−y
∈
dist(x(θ), C) = θ.
Proof. (a) Obviously, if y = PC (x), then y ∈ bd C. Furthermore, y = PC (x) minimizes u → x − u + δ(u|C), and then (a) is a direct consequence of the optimality conditions given in Theorem 4.1.1, while (b) is an immediate consequence of Lemma 4.2.1. 2 In order to obtain an error bound we need to introduce some kind of asymptotic constraint qualification for the system of convex inequalities. Definition 4.3.1 Let C be defined by the system of convex inequalities given in (4.14). (a) Let S be the set of sequences {xk } in C with ||xk || → ∞ and for which there exists a nonempty subset I of {1, . . . , m} such that for k sufficiently
4.3 Error Bounds for Convex Inequality Systems
135
large fi (xk ) = 0 ∀i ∈ I. Let {xk } ⊂ S and ci,k ∈ ∂fi (xk ) ∀k ∈ N ∀i ∈ I. Define the following index sets: Ib := {i ∈ I | {ci,k } is bounded},
Iub := I \ Ib .
Then the sequence {xk } is said to satisfy the property (P) if for each subsequence {nk } of {k} such that ci,nk {ci,nk } converges if i ∈ Ib , converges if i ∈ Iub , ci,nk with limit denoted by gi ∀i, one has necessarily 0 ∈ / conv{gi | i ∈ I}. (b) We say that C satisfies the asymptotic constraint qualification (A.C.Q.) if either S is empty or each sequence {xk } ⊂ S satisfies property (P). Note that an obvious situation where the A.C.Q. is automatically satisfied is when C is a compact set. Theorem 4.3.1 Let fi : Rn → R ∪ {+∞} be lsc and convex with dom fi open for i = 1, . . . , r, and let fi (x) = ai , x − αi for r + 1 ≤ i ≤ m. Suppose that Slater’s condition holds for the set C defined in (4.14). Then l := inf
x∈C /
|||F + (x)||| = inf inf |||(Fa+ ) (y; d)||| := t. dist(x, C) y∈bd(C) d∈M (y)
If in addition A.C.Q. holds, then t > 0. Proof. Let x ∈ / C and y = PC (x). Since fi+ is convex and since fi+ (y) = 0 ∀i ∈ I(y) we get fi+ (x) ≥ (fi+ ) (y; x − y). Since dist(x, C) = x − y, it follows that fi+ (x) x−y ≥ (fi+ ) y; ∀i ∈ I(y). dist(x, C) x − y Furthermore, we have
(fi+ ) (y; d) = (fi (y; d)) . +
Since ||| · ||| is monotonic, it follows that !! ! !!! |||F + (x)||| !!!!!! + x − y !!!!!! . ≥ !!!(Fa ) y; dist(x, C) ||x − y !!! Then by Lemma 4.3.1 we obtain |||F + (x)||| ≥ t, dist(x, C)
(4.16)
136
4. Minimizing and Stationary Sequences
which implies that l ≥ t. Now let {yk }, {dk } be sequences such that yk ∈ bd C, dk ∈ M (yk ) and
t = lim |||{(fi+ ) (yk ; d), i ∈ I(yk )}|||. k→∞
(4.17)
Let {θk } be a sequence of positive reals converging to 0 and set xk (θ) = / C and dist(xk (θ), C) = θ. yk + θdk for θ > 0. By Lemma 4.3.1, xk (θ) ∈ Since fi+ (yk ) = 0 for i ∈ I(yk ), by definition of the directional derivative we have fi+ (yk + θdk ) = (fi+ ) (yk ; dk ) + εk (θ) with lim εk (θ) = 0. θ→o θ Let j(k) be such that εk (θj(k) ) → 0 and set xk = xk (θj(k) ). Then fi (xk ) < 0 for i ∈ I(yk ) and fi+ (xk ) = (fi+ ) (yk ; dk ) + εk (θj(k) ), dist(xk , C)
i ∈ I(yk ),
and then from (4.17) it follows that |||F + (xk )||| = t, k→∞ dist(xk , C) lim
and we have proved l = t. Suppose now that A.C.Q. holds and let us prove that t > 0. Let I0 := {i ≥ r + 1 | ai , x = αi
∀x ∈ C},
I1 := {i ≥ r + 1 | i ∈ / I0 }.
Observing that C = {x | fi (x) ≤ 0 i = 1, . . . , r,
ai , x ≤ αi i ∈ I1 , ai , x = αi i ∈ I0 },
without loss of generality (deleting the other equations if necessary), we can suppose that the vectors {ai | i ∈ I0 } are linearly independent. We first prove that for each y ∈ C, and each subset I ⊂ I(y) we cannot have λi ci , λi > 0, ci ∈ ∂fi (y) ∀i ∈ I. (4.18) 0= i∈I
Three cases are possible: (i) I = I0 . In this case (4.18) is impossible, since the vectors {ai | i ∈ I0 } are linearly independent. (ii) I = I0 ∪ I1 with I1 nonempty. Then for each i ∈ I1 there exists xi ∈ C such that a i , xi < αi . Let |I1 | be the number of elements of I1 and let ¯ ∈ C, and we have x ¯ = |I1 |−1 i∈I1 xi . Then x ai , x ¯ < αi
∀i ∈ I1 .
4.3 Error Bounds for Convex Inequality Systems
137
As a consequence, we obtain from (4.18) 0= λi ci , x ¯ − y ≤ λi (ai , x ¯ − αi ) < 0, i∈I0 ∪I1
i∈I0 ∪I1
which is impossible. (iii) There exists i ≤ r ,with i ∈ I. Then using Slater’s condition with (4.18) we obtain 0= λi ci , x0 − y ≤ λi fi (x0 ) < 0, i∈I
i∈I
which again is impossible. Suppose now that t = 0. Then there exist sequences {yk } ∈ bd C, {dk } ∈ M (yk ) such that limk→∞ (fi+ ) (yk ; dk ) = 0 for i ∈ I(yk ). Without loss of generality we can suppose that I(yk ) = I for each k ∈ N. Then using (4.16) we get ∀i ∈ I. (4.19) lim sup fi (yk ; dk ) ≤ 0 k→∞
Since dk ∈ M (yk ), there exist λki ≥ 0, ci,k ∈ ∂fi (yk ), i ∈ I, such that dk = i∈I λki ci,k . Without loss of generality we can suppose that ci,k = gi for i ∈ Iub , lim ci,k = gi for i ∈ Ib . k→∞ ci,k k→∞
lim ci,k = ∞, lim
k→∞
Let tk = {λki , i ∈ Ib , λki ci,k , i ∈ Iub } and set λki λk ci,k for i ∈ Ib , λ¯ki = i for i ∈ Iub . tk 1 tk 1 ¯k ¯k ¯ i k ≥ 0, λ Since λ i∈I i = 1, it follows that {λi } is bounded, and again ¯ k } converges without loss of generality we can suppose that the sequence {λ i ¯ ¯ to λi ≥ 0 with i∈I λi = 1. Now two cases are possible, depending on the boundedness of the sequence {yk }: (i) Suppose that the sequence {yk } is bounded. Then since dom fi is open, the sequences {ci,k }, i ∈ I, are bounded, and there exist subsequences i ∈ I}, {dkl } converging respectively to y ∈ C, ci ∈ {ykl }, {ci,kl : ∂fi (y), i ∈ I ⊂ I(y), and d with d = 1. Furthermore, Iub is empty, and we cannot have liml→∞ tkl = ∞. Indeed, we have λ¯ki =
k dk l ¯ l ci,k → ¯ i gi , = λ λ l i tkl i∈Ib
so that 0=
i∈Ib
¯ i ci with ci ∈ ∂fi (y), λ
i∈Ib
i∈Ib
¯ i ≥ 0, ¯ i = 1, λ λ
138
4. Minimizing and Stationary Sequences
which is impossible from (4.18). Then we can suppose without loss of generality that the sequences {λki l }, i ∈ I, are bounded and converge to some λi ≥ 0 such that λi ci , d = 1. (4.20) d= i∈I
From (4.19) we have lim supci,k , dk ≤ 0
∀i ∈ I.
k→∞
Passing to the limit we obtain ci , d ≤ 0, ∀i ∈ I, and from (4.20) we obtain d = 0, which is impossible. (ii) Suppose now that the sequence {yk } is unbounded. Then we can write dk ¯ k di,k , with di,k = ci,k if i ∈ Ib and di,k = ci,k otherwise. = λ i tk ci,k i∈I
(4.21) Let us prove that the sequence {tk } is bounded. In the contrary case, without loss of generality we can suppose that lim tk = ∞, and passing ¯ k→∞ to the limit in (4.21) we obtain 0 = i∈I λ i gi , which is impossible from A.C.Q. Now from (4.19) we have " # ci,k , dk ≤ 0 ∀i ∈ Iub , lim sup lim supci,k , dk ≤ 0 ∀i ∈ Ib , ci,k k→∞ k→∞ from which it follows that gi , d ≤ 0 ∀i ∈ I,
(4.22)
where d is any cluster point of the sequence {dk } and satisfies d = 1. Since the sequence {tk 1 } is bounded, we can suppose without loss of generality that it converges to some α ≥ 0. It can be easily seen that α = 0; otherwise, by the definition of dk , passing to the limit we would obtain a vector d = 0, with d = 1, which Then from (4.21), passing to the limit is impossible. ¯ i gi . Using (4.22), it follows that d = 0, which is we obtain α−1 d = i∈I λ impossible. 2 As a corollary we obtain the following result, which concerns two important cases, and in particular allows for refining Hoffman’s error bound, which is now shown to be sharp. Corollary 4.3.1 Let fi : Rn → R ∪ {+∞} be lsc and convex with dom fi open for i = 1, . . . , p, and let C = {x | fi (x) ≤ 0, i = 1, . . . , p}. If either one of the following conditions is satisfied: (a) C is bounded and Slater’s condition holds, (b) C is a polyhedral set, i.e., fi (x) = ai , x − αi ∀i, then l = t and t > 0; i.e., we have a sharp error bound.
4.3 Error Bounds for Convex Inequality Systems
139
Proof. Part (a) is exactly given by Theorem 4.3.1. For part (b), when C is a polyhedral set, as remarked in the proof of Theorem 4.3.1, we can suppose that the vectors {ai | i ∈ I0 } are linearly independent. In this case A.C.Q. is equivalent to saying that for each sequence {xk } with ||xk || → ∞ one has 0 ∈ / conv{ai | i ∈ I}. But as seen in the proof of Theorem 4.3.1, this cannot occur. 2 As a final result for such kinds of bounds, we give now a simpler sufficient condition for the above set to satisfy a global error bound by invoking Theorem 4.2.2. However, the error bound will not be sharp in that case. Proposition 4.3.1 Let fi : Rn → R∪{+∞} be lsc and convex with dom fi open for i = 1, . . . , p, and let C = {x | fi (x) ≤ 0, i = 1, . . . , p}. Suppose that Slater’s condition holds and there exists a scalar δ > 0 such that ∀¯ x ∈ bd C,
∃¯ z ∈ int C such that
¯ x − z¯ ≤ δ. min1≤i≤p {−fi (¯ z )}
Then there exists θ > 0 such that dist(x, C) ≤ θ max fi+ (x) 1≤i≤p
∀x ∈ Rn .
Proof. Let f (x) = max1≤i≤p fi (x). Then one can write C = {x | f (x) ≤ 0},
int C = {x | f (x) < 0},
bd C = {x | f (x) = 0}.
We claim that our assumption implies that 0 ∈ / cl(∂f (f −1 (0)), and thus the desired error bound follows from Theorem 4.2.2. Assume that the claim is false. Then there exist sequences {xk } and {ck } such that ck ∈ ∂f (xk ),
f (xk ) = 0,
lim ck = 0.
k→∞
By the assumptions there exists zk ∈ int C such that xk − zk ≤ δ min {−fi (zk )}, 1≤i≤p
or equivalently, −f (zk ) ≥ δ −1 xk − zk . Since ck ∈ ∂f (xk ), we have f (zk ) = f (zk ) − f (xk ) ≥ ck , zk − xk ≥ −ck zk − xk , which implies that ck ≥ δ −1 ∀k, a contradiction.
2
Definition 4.3.2 Let C be defined by (4.14). Then C is said to be Hmetrically regular if there exist M > 0, γ ∈ ]0, 1] such that dist(x, C) ≤ M r(x)
∀x ∈ Rn
140
4. Minimizing and Stationary Sequences
with γ r(x) = max max (fi+ (x), fi+ (x) . 1≤i≤m
The H-metric regularity stipulates that error bounds of Holder type hold for C. When γ = 1, this means that a usual error bound with M > 0 exists for C. In this case we have Lipschitz type error bounds. We shall say that C is L-metrically regular. This concept is of particular importance, since as we shall see in the next section it ensures in the constrained case that asymptotically feasible stationary sequences or approximate Karush–Kuhn–Tucker sequences are minimizing sequences. Furthermore, this concept is particularly useful because not only does it enlarge the concept of L-metric regularity, but in addition it ensures in the quadratic case that C is H-metrically regular with γ = 12 .
4.4 Stationary Sequences in Constrained Minimization In this section we are back to the convex constrained optimization problem (P)
m = inf{f0 (x)|x ∈ C}
with C = {x ∈ Rn | fi (x) ≤ 0 i = 1, . . . , m}, where for i = 0, . . . , m, the functions fi : Rn → R ∪ {+∞} are lsc convex proper functions. We begin with some basic definitions. Definition 4.4.1 Let f = f0 + δC . A sequence {xk } is said to be: (a) asymptotically residual feasible(arf ) if lim sup fi (xk ) ≤ 0 for each i = 1, . . . , m; k→∞
(b) asymptotically feasible (af ) if limk→∞ dist(xk , C) = 0; (c) asymptotically residual feasible stationary (arfs) if {xk } is arf and if xk = PC (xk )) is stationary, i.e., if there exists the projected sequence {¯ xk } (¯ ak ∈ ∂f (x¯k ) with ak → 0. As a consequence of the concept of H-metric regularity, it follows that an arf sequence is af when C is H-metrically regular. We recall that a function F : Rn → (R ∪ {+∞})m is uniformly continuous near a given sequence {xk } if for every ε > 0 there exists δ > 0 such that for each k we have y − xk ≤ δ ⇒ F (y) − F (xk ) ≤ ε.
4.4 Stationary Sequences in Constrained Minimization
141
The next theorem gives conditions under which a sequence {xk } is asymptotically feasible and minimizing with respect to problem (P). Theorem 4.4.1 Let fi : Rn → R ∪ {+∞}, i = 0, . . . , m, be a collection of proper lsc convex functions with dom f0 open and dom f0 ∩ C = ∅. Suppose that C and L(λ) := {x ∈ C | f0 (x) ≤ λ} with λ > m are H-metrically regular. Let {xk } be an arfs sequence. Then if f0 is uniformly continuous near {xk }, the sequence {xk } is an af minimizing sequence; i.e., lim dist(xk , C) = 0 and
k→∞
lim f0 (xk ) = m.
k→∞
(4.23)
Proof. Since C is H-metrically regular and {xk } is an arf sequence, it follows that {xk } satisfies limk→∞ dist(xk , C) = 0 and we have only to ¯k = PC (xk ). Then since {xk } is an arfs prove limk→∞ f0 (xk ) = m. Let x xk ) sequence, it follows by Definition 4.3.1(c) that there exists ak ∈ ∂f (¯ xk } is feasible, and we want to prove first that with ak → 0. The sequence {¯ it is a minimizing sequence. Suppose the contrary. Then there exists λ > m xk ) > λ. Let yk = PL(λ) (¯ xk ). Such an yk exists, such that lim inf k→∞ f0 (¯ ¯k ∈ C, we have since L(λ) is nonempty. Since dom f0 is open and since x f0 (yk ) = λ. Since f0 is convex, then λ = f0 (yk ) ≥ f0 (¯ xk ) + ck , yk − x ¯k
∀ck ∈ ∂f0 (¯ xk ).
(4.24)
Now by Theorem 4.1.1 it follows that there exist ck ∈ ∂f0 (¯ xk ), bk ∈ NC (¯ xk ) xk ) and yk ∈ C, it follows that such that ak = ck + bk . But since bk ∈ NC (¯ ¯k , ck − ak ≥ 0, and then from (4.24) we get yk − x f0 (¯ xk ) − λ ≤ −ak , yk − x ¯k . Using the Cauchy–Schwarz inequality and recalling that L(λ) is assumed H-metrically regular, it follows that there exist constants M > 0, γ ∈ ]0, 1] such that f0 (x¯k ) − λ ≤ M ak max{f0 (¯ xk ) − λ, (f0 (¯ xk ) − λ)γ }. Dividing by f0 (¯ xk ) − λ, which is positive, we obtain 1 ≤ M ak max{1, (f0 (¯ xk ) − λ)γ−1 }. Since ak → 0 and lim inf k→∞ f0 (¯ xk ) > λ, we obtain a contradiction from the above expression by passing to the limit when k → ∞. Then we have x ¯k ∈ C,
lim f0 (¯ xk ) = m.
k→∞
Now since xk − x ¯k = dist(xk , C) → 0 and since f0 is uniformly continuous 2 near xk , it follows from the definition that lim inf k→∞ f0 (xk ) = m.
142
4. Minimizing and Stationary Sequences
Remark 4.4.1 If the functions fi are quadratic and convex, then it can be proved that C and L(λ) are H-metrically regular. If f0 is linear, then f0 is uniformly continuous. In many algorithms, in fact, one often computes an arf sequence {xk } and a sequence of multipliers µk = (µk1 , . . . , µkm ) that satisfies approximatively the Karush–Kuhn–Tucker conditions, i.e., µki ≥ 0, lim ∇f0 (xk ) +
k→∞
lim µki fi (xk ) k→∞ m µki ∇fi (xk ) i=1
=
0 ∀i = 1, . . . , m,
(4.25)
=
0.
(4.26)
In general, with Slater’s condition the algorithms generate a sequence {µk } that is bounded. Such typical situations arise in particular with augmented Lagrangian methods. The following theorem tells us when we can expect that such a sequence is minimizing. Theorem 4.4.2 Let fi : Rn → R ∪ {+∞} be lsc convex proper functions with dom fi open and fi differentiable on their domains, for each i = 0, . . . , m. We suppose that C and L(λ) := {x ∈ C : f0 (x) ≤ λ} with λ > m, are H-metrically regular. Let {xk } be an arf sequence and {µk } a bounded sequence of multipliers satisfying (4.25)–(4.26). Then if for each i = 0, . . . , m, fi , ∇fi are uniformly continuous near {xk }, then {xk } is an af minimizing sequence. Proof. The proof is very similar to the proof of Theorem 4.4.1. Let {xk } be an arf sequence. Since C is H-metrically regular, it follows that {xk } is ¯k = PC (xk ). arf and we have only to prove that limk→∞ f0 (xk ) = m. Let x xk } and since {µki } is Then since fi , ∇fi are uniformly continuous near {¯ bounded satisfying (4.25)–(4.26)we have εk :=
m
µki fi (¯ xk ) → 0,
i=1
ak := ∇f0 (¯ xk ) +
m
µki ∇fi (¯ xk ) → 0.
i=1
The sequence {¯ xk } is feasible, and we want to prove first that it is a minimizing sequence. Suppose the contrary, then there exists λ > m such that xk ) ≥ λ. Let yk = PL(λ) (¯ xk ). Such a yk exists, since L(λ) is lim inf k→∞ f0 (¯ ¯k ∈ C, we have f0 (yk ) = λ, and nonempty. Since dom f0 is open and since x by convexity it follows that λ = f0 (yk ) ≥ f0 (¯ xk ) + ∇f0 (x¯k ), yk − xk , and since yk ∈ C, we also have 0 ≥ fi (yk ) ≥ fi (x¯k ) + ∇fi (x¯k ), yk − xk .
(4.27)
4.5 Notes and References
143
Multiplying this last inequality by µki and summing up for i = 1, . . . , m, we obtain m −εk ≥ µki ∇fi (¯ xk ), yk − xk . i=1
xk )−λ ≤ εk +ak , yk −xk , Adding this last inequality to (4.27) we obtain f0 (¯ and the proof can be completed the same way as the proof of Theorem 4.4.1. 2 Note that all the assumptions of Theorem 4.4.2 are satisfied for linear programming. For more general situations the assumptions requiring fi for all i = 0, . . . , m, uniformly continuous near {xk }, are usually not easy to verify, and one has to consider stronger assumptions, as in the following theorem, which corresponds to the coercive case and will be proved in Chapter 6, in the more general framework of variational inequalities. Theorem 4.4.3 Let fi : Rn → R ∪ {+∞} be lsc convex proper functions with dom fi , i = 1, . . . , m, open. Suppose that the optimal set of (P) is nonempty and compact and that C ⊂ dom ∂f0 . Let {uk } be a bounded sequence in Rm + and consider a sequence {xk , gk , gi,k , i = 1, . . . , m} with gk ∈ ∂f0 (xk ), gi,k ∈ ∂fi (xk ), i = 1, . . . , m, such that m k lim gk + ui gi,k = 0, k→∞
i=1
lim sup fi (xk ) ≤ 0, ∀i = 1, . . . , m, k→∞
lim uki fi (xk )
k→∞
=
0, ∀i = 1, . . . , m.
Then the sequence {xk } is bounded, and each limit point of this sequence is an optimal solution of (P).
4.5 Notes and References Results given in Section 4.1 are standard except Theorem 4.1.6, whose proof is due to Corvallec [50]. Well-behaved functions were introduced by Auslender and Crouzeix [8], from which all the results presented here have been taken, except for Proposition 4.2.3 and Corollary 4.2.1, which were given in Auslender, Cominetti and Crouzeix [10]. Dual characterizations of well-behaved functions were first given by Angleraud [1]. An easier characterization was given by Cominetti [49], and more recently Aze and Michel [21] have completed the results of [49]. All the results and material given here are from [21]. For relationships between the notions of well-behaved functions and well-posedness, a notion not discussed here, we refer to the
144
4. Minimizing and Stationary Sequences
work of Penot [107]. In the last two decades, the study of error bounds has grown significantly. An important part of the literature is devoted to linear systems, starting with the classical result of Hoffman [79] and later on with the work of Walkups and Wets [130], Robinson [111], and Mangasarian [97]. Many results on linear systems providing new insights and further analysis can be found in Bergthaller and Singer [32], Burke and Tseng [45], Guler, Hoffman, and Rothblum [78], Klatte and Thiere [81], Li [88], Luo and Tseng [94], and Renegar [110]. Results on error bounds for convex inequality systems with compact convex sets under Slater’s condition are due to Robinson [111], [112]. Mangasarian [98] introduced an asymptotic constraint qualification (ACQ) and derived a global error bound for convex systems with differentiable functions under (ACQ) and assuming Slater’s condition. Auslender and Crouzeix then gave a sharp error bound for the case of nondifferentiable convex functions by varying ACQ; see Theorem 4.3.1. Luo and Luo [92] avoid having to assume any type of constraint qualifications by restricting their analysis to systems described by quadratic convex functions. To obtain computable error bounds, we refer, for example, to Mangasarian [99], from which Proposition 4.3.1 comes, and later on to Bertsekas [35] and Deng [62]. Several papers have also been devoted to global error bounds, not of Liptshiz type, but of Holder type; see, for example, the work of Luo and Tseng [94] which studies systems of analytic functions, and the work of Wang and Pang on convex quadratic functions but without Slater’s assumption. For further reading and references we refer the reader to Lewis and Pang [86] and to Pang [106]. The final section of this chapter summarizes some ideas and results developed by Chou, Ng, and Pang [48], and all the results described here come out from this work, except Theorem 4.4.3.
5 Duality in Optimization Problems
Duality theory plays a fundamental role in the analysis of optimization and variational problems. It not only provides a powerful theoretical tool in the analysis of these problems, but also paves the way to designng new algorithms for solving them. The basic idea behind a duality framework is greatly similar to the general mathematical line of thinking, namely to transform a hard problem into an easy or at least easier one to be analyzed and to be solved. A key player in any duality framework is the Legendre–Fenchel conjugate transform. Often, duality is associated with convex problems, yet it turns out that duality theory also has a fundamental impact even on the analysis of nonconvex problems. This chapter gives the elements of duality theory for optimization problems. Starting with a very general and abstract scheme for duality based on perturbation functionals, we derive the basic conditions leading to strong duality results, and characterization of optimal solutions. This covers well-known Fenchel and Lagrangian duality schemes, as well as minimax theorems for convex–concave functionals within a unified approach that emphasizes the importance of asymptotic functions.
5.1 Perturbational-Conjugate Duality This section introduces a general abstract duality framework that includes all the basic ingredients necessary to derive a viable duality theory for optimization problems. The general idea of duality can be described as follows.
146
5. Duality in Optimization Problems
Given an initial problem, called the primal, construct a new problem based on the same data, which is called the dual and is hopefully easier to analyze and to solve. The key question is then to find the relationships between the primal and dual problems, in particular, to know when these two problems are equivalent in the sense of having the same optimal values and in being able to characterize the existence of the corresponding optimal solutions as well as to recover optimal solutions of one problem from the optimal solutions of the other. We first outline results that do not assume convexity, and then establish stronger duality relations for the convex case. A General Abstract Dual Scheme Let f : Rn → R ∪ {+∞} be a proper function and consider the abstract minimization problem (P)
VP := inf{f (x) | x ∈ Rn }.
Problem (P) will be called the primal problem. Recall that since f is extended valued, the above formulation includes problems with a set of constraints C ⊂ Rn by simply redefining the function f with f + δC . Note that we also assume that f is proper in order to exclude the degenerate case f ≡ +∞. The solution set (possibly empty) of problem (P) is denoted by SP , and when inf f is finite one has
SP = x ¯ ∈ Rn | f (¯ x) = inf f (x) . x
We do not assume convexity for the moment, so that the above formalism encompasses a wide range of optimization problems, and the results described below hold in particular for nonconvex problems. A powerful concept for developing and studying a duality theory is through the use of perturbations. More precisely, we em