Nonlinear Programming (Classics in Applied Mathematics)

10.3

Nonlinear Programming

It is obvious also that x = — 1 solves the primal problem and that B(x) = — 2. Hence x = x, and 6(x) = t(x,u). We end this chapter with a corollary that follows directly from the proof of Theorem 8.1.6 or from the proof of Theorem 1 above. Corollary Let all the assumptions of Theorem 1 above hold, except that 6 need not be pseudoconvex at x nor g be quasiconvex at x. If (x,u) is a local solution of DP 8.1.2, then (x,u) is a Kuhn-Tucker point, that is, (x,u) solves KTP7.14.

160

Chapter Eleven Optimality and Duality in the Presence of Nonlinear Equality Constraints

Up until now we have dealt almost exclusively with minimization problems in which the feasible region X is determined by inequality constraints only, that is,

where X° is a subset of Rn, and g is an w-dimensional vector function defined on X°. (For exceptions see 6.8.8, 5.4-8, 7.2.2, and 7.3.18, where equality constraints are also considered.) In this chapter we shall examine in some detail minimization problems in which the feasible region X is determined by inequality and nonlinear equality constraints, that is,

where h is a fc-dimensional vector function defined on X°. In particular we shall obtain sufficient optimality criteria, necessary optimality criteria, and some duality results for the following minimization problem

We remark here that all the necessary optimality conditions of Chap. 7 that involved differentiable functions required that the set X° be open. We derive in this chapter an important necessary optimality condition of the maximum-principle type which does not require that X° be open.

11.1

1,

Nonlinear Programming

Sufficient optixnality criteria

The sufficient optimality criteria given here follow directly from Theorem 10.1.1 and Theorem 10.1.2 by observing that the equality constraint h(x) = 0 can be written as h(x) ^ 0 and — h(x) ^ 0, and that the negative of a quasiconcave function is quasiconvex. Sufficient optimality theorem Let X° be an open set in Rn, and let d, g, and h be respectively a numerical function, an m-dimensional vector function, and a k-dimensional vector function, all defined on X°. Let x G X°, let let 0 be pseudoconvex at x, let gr be differentiate and quasiconvex at x, and let h be differentiate, quasiconvex, and quasiconcave at x. If there exist u £j Rm and v £ Rk such that (x,u,v) satisfies the following conditions

then x is a solution of the following minimization problem

Kuhn-Tucker sufficient optimality theorem Theorem 1 above holds with the condition replacing the condition

2.

"Minimum-principle" necessary optimality criteria: X° not open

We consider in this section the minimization problem

162

Optimality and Duality with Nonlinear Equalities

11.2

We do not assume here that X° is an open subset of Rn, that g is convex on X°, or that h is linear on\ff n .f Without the assumption that X° is open, we cannot derive necessary optimality criteria of either the Fritz John type (7.3.2} or of the Kuhn-Tucker type (7.3.7). We can, however, derive necessary optimality criteria of the minimum-principle type, which, when X° is open, become necessary optimality criteria of the Fritz John or Kuhn-Tucker type. (The minimum-principle necessary optimality criterion we are about to derive has been referred to as a maximum principle in [Halkin 66, Halkin-Neustadt 66, Canon et al. 66, MangasarianFromovitz 67] because of its close relation to Pontryagin's maximum principle [Pontryagin et al. 62], which is the fundamental optimality condition of optimal control. We have a minimum principle here instead of a maximum principle because the adjoint variables of Pontryagin's maximum principle, which play the role of Lagrange multipliers, are the negative of our Lagrange multipliers f 0 , f, s, to be introduced below. Halkin [Halkin 66] has a maximum principle because he has a maximization problem instead of a minimization problem.) We point out now the main difference between a minimum-principle necessary optimality criterion and the main condition of the Fritz John necessary optimality criterion. The minimum principle requires that where x is a solution of the minimization problem, f 0 , f, and s are Lagrange multipliers and X° is the closure of X°. The Fritz John criterion requires that Obviously, the latter condition follows from the first one if X° is open. If X° is not open, the latter condition does not hold in general. We begin by establishing a generalization of a linearization lemma of [Mangasarian-Fromovitz 67a]. Linearization lemma Let X° be a convex set in Rn with a nonempty interior, int (X°), and let A be an open set in Rn. Let f be an ^-dimensional vector function, and let h be a k-dimensional vector function, both defined on some open set containing X°. Let t For the case when X° is convex, g is convex on X°, and h is linear on Rn, we have the necessary optimality conditions 5.4-8. 163

ll.X

Nonlinear Programming

Let f be differentiable at x, let h have continuous first partial derivatives at x, and let Vhj(x), j = I , . . . , k, be linearly independent, that is,

Then

PROOF k = n: The linear independence of Vhj(x), j = 1, . . , , k, is equivalent to the nonsingularity of Vh(x). Hence Vf(x)(x — x) < 0 cannot hold, because Vh(x)(x — x) = 0 implies that x — x = 0. k > n: This case is excluded because of the assumption that Vhj(x),j j = 1, . . . , k, are linearly independent, which implies that k ^ n. 0 < k < n: We shall establish the lemma by proving the following backward implication

which is the logical equivalent of the implication of the lemma. Since h(x) = 0, and since Vhj(x), j = 1, . . . , k, are linearly independent, we have, by the implicit function theorem D.3.1, that there exist: (i) a partition (XJ,XK) of x such that xi G Rn~k, XK £ Rk, (ii) an open set Si in Rn~k containing xr, and (iii) a fc-dimensional differentiable vector function e on ft such that

164

Optlmality and Duality with Nonlinear Equalities

11.1

and

VXKh(x) is nonsingular Since h and e are differentiable, and since h[xi,e(xi)] = 0 for all XT E fy we have, by the chain rule D.L6, that Postmultiplication by (XT — XT) gives But by assumption Vh(x)(£ — x) = 0, or equivalently

Hence the last two distinct equalities and the nonsingularity of VXKh(x) imply that

Now, since ft is open, since XT (E ft, and since e is differentiable at x, there exists a 50 > 0 such that for S < do and

where c is a /c-dimensional vector function that tends to zero as 5 tends to zero. By combining the last two equalities and the fact that XK = e(xi), we have that for 5 < 50 But this relation and the fact that / is differentiable at x imply that there exists a 61 such that 0 < 81 < 80 and such that for 5 < di

188

11.1

Nonlinear Programming

where ||a;/,x/c|| denotes (xixi + XKXR)^, and where 6 is an /-dimensional vector function which tends to zero as 6 tends to zero. Since c tends to zero as 5 tends to zero, and since (by assumption) Vf(x)(x — x) < 0, that is, it follows that there exists a 52 such that 0 < 52 < 5i, and such that for 0 < 5 < 82, the quantity in the curly brackets { } in the expression next to the last is strictly negative. Hence (since /(£) = 0) we have that and, since h[xi,e(xi}} — 0 for XT (E 12, we also have that Now, we have already shown that for 5 < 80 Because x G int (-X"0), and because c tends to zero as 8 tends to zero, there exists a 83 such that 0 < 83 < 52 and 53 < 1, and such that Since x (— X°, it follows from the last two relations and the convexity of X ° that Since x G A, and since A is open, it follows from the expression established above for e[xi -+- 8(xi — Xi}} that there exists a 5 4 such that 0 < 54 < 53 and such that Hence, combining the above results, we have that any x given by satisfies f(x] < 0, h(£) = 0 and is in X° C\ A. We have thus established the lemma for the case when n > k > 0. k = 0: This is the case where there is no h(x) — 0. We prove here that We have now that for 0 < 8 < 80

166

Optimality and Duality with Nonlinear Equalities

11.9

Since b tends to zero as 5 tends to zero, and since Vf(x) (x — x) < 0, there exists a 6 > 0 such that

Since x G X° C\ A, and since X° is convex and A is open, there exists a £ < 0 such that £ < 8 and $ < 1, and such that

Hence any x = x + 8(x — x ) . 0 < 5 < §. satisfies f(£] < 0 and is in

x°nA.

We derive now another lemma from the above one by using the fundamental theorem for convex functions, Theorem ^$.1 (which is a consequence of the separation theorem for convex sets, Theorem 3.2.8). This will be the key lemma in deriving the minimum-principle necessary optimality criterion. Lemma Let X° be a convex set in Rn with a nonempty interior, int (X°~), and let A be an open set in Rn. Let f be an ^-dimensional vector function, and let h be a k-dimensional vector function, both defined on some open set containing X°. Let

Let f be differentiable at x. Then

at x, and let h have continuous first partial derivatives

PROOF If Vhj(x), j = 1, . . . , k, are linearly dependent, then there exists a q 9^ 0 such that qVh(x) = 0, and the lemma is trivially satisfied by p = 0, q ^ 0, and qVh(x)(x - x) = 0 for all x £ X°. If Vhj(x),j = 1, ... , k, are linearly independent, then by Lemma 1 above we have that

has solution has aa solution 167

11.3

Nonlinear Programming

Since the interior, int (X°), of a convex set is convex (see 3.1.7), it follows by Theorem 4-2.1 that there exist p G R*, q G Rk, P ^ 0, (p,q) ^ 0 such that and since the above expression is continuous in x, and in fact linear, the above inequality holds also on the closure X° of X°. We are now ready to derive the fundamental necessary optimality criterion of this chapter. Minimum-principle necessary optimality theorem Let X° be a convex set in Rn with a nonempty interior: int (X°). Let 6 be a numerical function, let g be an m-dimensional vector function, and let h be a k-dimensional vector function, all defined on some open set containing X°. Let x be a solution of

Let 6 and g be differentidble at x, and let h have continuous first partial derivatives at x. Then there exist f 0 G R> f G Rm, s G Rk such that the following conditions are satisfied

REMARK The first of the above four relations is called a minimum principle because it is a necessary optimality condition for f06(x) -frg(x) -f- sh(x) to have a minimum at x (see Theorem 9.3.3). If in addition we have that f08 + fg + sh is pseudoconvex or convex at x (this, in general, is not the case under the assumptions of the above theorem), then the minimum-principle condition implies that fod(x) + fg(x) -f- sh(x) = min f08(x) + fg(x) -f- sh(x) x£.X<>

The above condition becomes then equivalent to the saddlepoint condition of Problem 6.4.2. PROOF

168

Let

Optimality and Duality with Nonlinear Equalities

11.2

Let m/ and mj denote the number of elements in the sets / and J respectively, so mi -\- mj = m. Since g is defined on some open set containing X°, and since g is differentiate at x, there exists a 8 > 0 such that for i 6E J and \\x — x \ < 8

Hence the set is an open set in Rn. Now, we have that the system

for if it did have a solution, then x would not be a solution of the minimization problem. But we also have that Hence by Lemma 2 above, there exist f 0

such that

By defining f / = 0 G: Emj and observing that and

the theorem is established. It should be remarked that the restriction that the convex set X° have a nonempty interior is a mild restriction, since in general a convex set without an interior is equivalent to the intersection of a convex set with a nonempty interior and a linear manifold \x \ x £ Rn, h(x] = 0} (h linear on Rn). Since we allow equalities h(x) = 0 in Theorem 3 above, 169

11,8

Nonlinear Programming

convex sets with empty interiors can, in effect, be handled by the above results. The convexity requirement on X° is of course a restriction which cannot be dispensed with easily. (See however [Halkin 66, Canon et al. 66].) If we replace the convexity requirement on X° by the requirement that XQ be open, a stronger necessary optimality condition than the above one can be obtained. In effect this will be an extension of the Fritz John stationary-point necessary optimality theorem 7.3.2 to the case of nonlinear equalities. We shall give this result in the next section of this chapter.

3. Fritz John and Kuhn-Tucker stationary-point necessary optimality criteria: X° open We derive in this section necessary optimality criteria of the Fritz John and Kuhn-Tucker types from the minimum principle of the previous section. Fritz John stationary-point necessary optimality theorem [Mangasarian-Fromovitz 67a] Let X° be an open set in Rn. Let 6 be a numerical function on X°, let g be an m-dimensional vector function on X°, and let h be a k-dimensional vector function on X°. Let x be a (global) solution of the minimization problem

or a local solution thereof, that is,

where B&(x) is an open ball around x with radius 8. Let 6 and g be differentiable at x, and let h have continuous first partial derivatives at x. Then there exist fo (E R, r G Rm, s G Rk such that the following conditions are satisfied

PROOF ITU

Let x be a global or local solution of the minimization problem.

Optlmality and Duality with Nonlinear Equalities

11.8

In either case (since X° is open) there exists an open ball Bp(x) around x with radius p such that Bp(x) C BI(X) C. X°, and

Since Bp(x) is a convex set with a nonempty interior, we have by the minimum principle 11.2.3 that there exist f 0 E R, ? £ Rm, s £ Rk such that

Since for some small positive f we have from the first inequality above that To derive Kuhn-Tucker conditions from the above, we need to impose constraint qualifications on the problem. The Kuhn-Tucker constraint qualification (see 7.3.3) Let X° be an open set in Rn, let g and h be w-dimensional and /c-dimensional vector functions on X°, and let

g and h are said to satisfy the Kuhn-Tucker constraint qualification at x G X if g and h are differentiate at x and

there exists an n-dimsional vector funtion on the interval suuch that

is differentiable at where

171

11.3

Nonlinear Programming

The weak Arrow-Hurwicz-Uzawa constraint qualification (see 10.8.3) Let X° be an open set in Rn, let g and h be m-dimensional and /c-dimensional vector functions on X°, and let

g and h are said to satisfy the weak Arrow-Hurwicz-Uzawa constraint qualification at x (E X if g and h are differentiate at x, h is both pseudoconvex and pseudoconcave at x, and

has a solution

where

ispseudoconcave pseudoconcave

The weak reverse convex constraint qualification (see 10.2.4) Let X° be an open set in Rn, let g and & be m-dimensional and /c-dimensional vector functions on X°, and let

g and h are said to satisfy the weak reverse constraint qualification at x G X if a and /i are differentiate at x, and if for each

either Qi is pseudoconcave at x or linear on Rn, and /i is both pseudoconvex and pseudoconcave at x. The modified Arrow-Hurwicz-Uzawa constraint qualification [Mangasarian-Fromovitz 67a] Let X° be an open set in Rn, let g and h be w-dimensional and fc-dimensional vector functions on X°, and let

g and /i are said to satisfy the modified Arrow-Hurwicz-Uzawa constraint qualification at x G X if g is differentiable at x, A is continuously differen172

Optimality and Duality with Nonlinear Equalities

11.3

liable at x, Vhi(x), i — 1, . . . , k, are linearly independent, and

has a solution where

Kuhn-Tucker stationary-point necessary optimality theorem Let X° be an open set in Rn, and let 9, g, and h be respectively a numerical function, an m-dimensional vector function, and a k-dimensional vector function, all defined on X°. Let x be a (global) solution of the minimization problem

or a local solution thereof, that is,

where B$(x) is some open ball around x with radius 5. Let d, g, and h be differentiate at x, and let g and h satisfy (i) (ii) (m) (iv)

the Kuhn-Tucker constraint qualification 2 at x, or the weak Arrow-Hurwicz-Uzawa constraint qualification 3 at x, or the weak reverse convex constraint qualification 4 at x, or the modified Arrow-Hurwicz-Uzawa constraint qualification 5 at x.

Then there exist u G Rm and a v G Rk such that

PROOF (i)-(ii)-(iii) These parts of the theorem follow from Theorem 10.2.7, parts (i), (ii), and (iii), by replacing h(x) = 0 in the above theorem by h(x) ^ 0 and -h(x) ^ 0. 173

U.*

Nonlinear Programming

(iv) All we have to show here is that f 0 > 0 in the theorem 1 above, for then we have that x,w = f/f 0 and v = the Kuhn-Tucker conditions above. We assume now that exhibit a contradiction. If 7 = [i | Qi(x) = 0} is empty, that is, there are no straints, then f — 0 (since fg(x) = 0, f ^ 0 and g(x) ^ 0). Theorem 1

Fritz John s/f0 satisfy f 0 = 0 and active conHence by

which contradicts the assumption of 5 that Vhi(x), i = 1, . . . , k, are linearly independent. (If there are no equality constraints h(x) — 0, then (fo,f) = 0 contradicts the condition (f 0 ,f) ^ 0 of Theorem 1, since there is no s.) If / is not empty, then by Theorem 1 we have that

If fj = 0, then s 5^ 0, and we have a contradiction to the assumption of 5 that Vhi(x), i = 1, . . . , k, are linearly independent (if there is no s, then fi = 0 implies (f 0 ,f) = 0, which contradicts the condition (f 0 , f) 5^ 0 of Theorem /). If rz 9* 0, then fj > 0. But by 5 there exists a 2 such that Hence which contradicts the equality above that

4. Duality with nonlinear equality constraints The following strict converse duality results follow respectively from Theorem 10.3.1 and Corollary 10.3.2 by replacing the equality constraints h(x) = 0 by h(x) £ 0 and -h(x) ^ 0. Strict converse duality theorem Let X° be an open set in Rn, and let 6, g, and h be respectively a numerical function, an m-dimensional vector function, and a k-dimensional vector function, all defined and differentiable on X°. Let (x,u,v) be a (global) 174

Optimallty and Duality with Nonlinear Equalities

11.4

solution of the dual problem

or a local solution of the dual problem, that is,

where BS(X,U,&) is an open ball in Rn X Rm X Rk with radius 8 > 0 around (£,w,$). Let 6 be pseudoconvex at x, let g be quasiconvex at x, and let h be both quasiconvex and quasiconcave at x. If either (i) (ii)

\l/(x,u,6) is twice continuously differentiate at x and the n X n Hessian matrix Vxz\l/(x,u,D) is nonsingular, or there exists an open set A in Rm X Rk containing (u,v} and an n-dimensional differentiate vector function e on A such that

and

then A solves the minimization problem

and

If x solves the minimization pro blem 3, it does not follow that x and some u and v solve the dual problem 2 (unless X° is convex, 6 and g are convex on X°, and h is linear on Rn), nor does it follow that &(x) ^ \l/(£,u,v) for any dual feasible point (£,#,£), that is, (£,u,v) £ Y. 175

11.4

Nonlinear Programming

Corollary Let all the assumptions of Theorem 1 above hold, except that 6 need not be pseudoconvex at x, nor g quasiconvex at x, nor h both quasiconvex and quasiconcave at x. If (x,u,v) is a local solution of the dual problem 2, then (£,ufi} is a Kuhn-Tucker point, that is,

176

Appendix A Vectors and Matrices

We collect in this appendix some of the bare essentials of real linear algebra which are needed in this book. For more detail the reader is referred to [Gale 60, Halmos 58]. We assume here that the reader is familiar with the notation and definitions of Chap. 1.

1. Vectors Fundamental theorem of vector spaces Let each of the vectors y°, yl, . . . , ym in Rn be a linear combination of the vectors xl, xz, . . . , xm in Rn; then y°, yl, . . . , ym are linearly dependent.

PROOF The proof will be by induction on m. If m = 1, then

then y° = yl = 0, and y° and yl are linearly dependent. If not, then po T* 0, say. Then and T/° and y1 are linearly dependent because p0 ^ 0. Now assume that the theorem holds for TO = k — 1, and we will show that it also holds for m = k. By hypothesis we have that

If all pi' are zero then all y3' are zero and hence linearly dependent. Assume now that at least one p*' is not zero, say pi°. Define

A.I

Nonlinear Programming

Then each of the k vectors zj is a linear combination of the k — 1 vectors x*, . . . , xk, and hence by the induction hypothesis, the zj are linearly dependent, that is, there exist numbers qi, . . . , qk, not all zero, such that

which shows that the y' are linearly dependent. Corollary Any n + 1 vectors in Rn are linearly dependent. PROOF Let e ' b e the vector in Rn with zero for each element except the ith, which is 1. Then any vector x in Rn is a linear combination of the ei for x = V xte{. The corollary then follows from Theorem 1 above. Corollary Any m vectors in Rn are linearly dependent if m > n. PROOF By Corollary 2 any n + 1 vectors of the m vectors are linearly dependent, hence the m vectors are linearly dependent. Corollary Each system of n homogeneous linear equations in m unknowns, m > n, has a nonzero solution. PROOF Consider the matrix equation Ax = 0 where A is any n X m matrix. The m columns Aj of A are vectors in Rn and hence by Corollary 3 above are linearly dependent. Hence there exists an x ^ 0 such that Ax = 0. Basis of a subset of Rn Let S be a subset of Rn, and let r be the maximum number of linearly independent vectors which can be chosen from S. Any set of r linearly independent vectors of S is called a basis for S. 178

A.2

Appendix A

Basis theorem The linearly independent vectors xl, . . . , xr are a basis for a subset n S of R if and only if every vector y in S is a linear combination of the x\ PROOF [Gale 60] Suppose every y in S is a linear combination of the x\ Then S contains no larger set of linearly independent vectors, for any set of more than r vectors must be dependent since they are combinations of the x\ Therefore r is the maximum number of linearly independent vectors which can be chosen from S, and hence the xi are a basis for S. Conversely, suppose that the xi are a basis for S. Then by definition, r is the number of vectors in the largest set of linearly independent vectors that can be found in S. So if y is in S, then x1, . . . , xr, y are linearly dependent, that is,

and po T^ 0, for otherwise the x{ would be linearly dependent.

Hence

which is the desired linear combination.

2.

Matrices

Row and column rank of matrix The maximum number of linearly independent rows of a matrix is called the row rank of the matrix, and the maximum number of linearly independent columns of a matrix is called the column rank of the matrix. Rank theorem For any matrix A, the row rank and column rank are equal. PROOF [Gale 60] Let r be the row rank of A, s its column rank, and suppose that r < s. Choose a row basis for A, which we may assume consists of the rows AI, . . . , Ar (renumbering rows and columns of A clearly does not affect its row or column rank), and a column basis, which we assume consists of the columns A.i, . . . , A.,. Let

179

A.a

Nonlinear Programming:

and note that the equations are r equations in s (> r) unknowns and hence have a nonzero solution y by Corollary A.I4- Also since AI, . . . , Ar are a row basis, it follows 1

from the basis theorem A.1.6 that for all k, Ak — Y pikAi for some numbers pik. Hence

and for all k

which is equivalent to

This shows that the columns A.I, . . . , A.t of A are linearly dependent, which contradicts the assumption that they were a basis. This contradiction shows that r ^ s. The same argument applied to the transpose of A shows that s ^ r. Hence r = s. Rank of a matrix The rank of a matrix is the column or row rank of the matrix (which are equal by the above theorem). Corollary Let A be an r X n matrix with rank r (r ^ ri). Then for any b in Rr, the system Ax = b has a solution x in R". PROOF By Theorem 2 above, the column rank of A is r, and thus if A.i, . . . , A.r, say, are a column basis, we have r linearly independent vectors in Rr, and the vector b £ Rr can be expressed as a linear combination of them, that is,

By defining Xj• = 0 for.; = r + 1, . . . , n, we have that 6 = Ax. 180

Appendix A

A.2

Nonsingular matrix An n X n matrix is said to be nonsingular if it has rank n. Semidefinite matrix An n X n matrix A is said to be positive semidefinite if xAx ^ 0 for all x in Rn and negative semidefinite if xAx ^ 0 for all x in Rn. Definite matrix An n X n matrix A is said to be positive definite if and negative definite if Obviously the negative of a positive semidefinite (definite) matrix is a negative semidefinite (definite) matrix and conversely. Also, each positive (negative) definite matrix is also positive (negative) semidefinite. Proposition Each positive or negative definite matrix is nonsingular. PROOF Let A be an n X n positive or negative definite matrix. If A is singular, then its rank is less than n, and there exists an x G R", x -^ 0, such that Ax = 0. Hence xAx = 0 for some x ?** 0, which contradicts the assumption that A is positive or negative definite. Hence A is nonsingular.

181

Appendix B Resume' of Some Topological Properties ofR n

We collect here some well-known topological properties of R" which can be found in greater detail in any modern book on real analysis or general topology [Buck 65, Fleming 65, Rudin 64, Berge 63, Simmons 63J.

1.

Open and closed sets Open ball

Given a point x° £ Rn and a real number X > 0, the set

is called an open ball around x° with radius X. (Sometimes the subscript X from B^(x°) will be dropped for convenience.) Interior point A point x is said to be an interior point of the set T C Rn if there exists a number e > 0 such that Bt(x) C T. Point of closure A point x is said to be a point of closure of the set F C Rn if for each number e > 0, Note that a point of closure of T need not be in T. For example 0 E R is a point of closure of the infinite set T = {1,H>M> • • • } > but 0 £ T. On the other hand, every point in the set F is also a point of closure of T. In other words, a point of closure x° of a set T is a point such that there exist points in T that are arbitrarily close to it (closeness being measured by the distance 5(x,x°) = \\x — x°\\, see 1.8.9).

Appendix B

B.I

Open set A set T C Rn such that every point of F is an interior point is said to be open. Closed set A set T C Rn such that every point of closure of F is in F is said to be closed. Closure of a set The closure T of a set T C Rn is the set of points of closure of F. Obviously F C T, and for a closed set F = T. Interior of a set The interior int (F) of a set F C Rn is the set of interior points of F. Obviously int (F) C F, and for an open set F = int (F). Relatively open (closed) sets Let F and A be two sets such that F C A C Rn- F is said to be open (closed) relative to A if F = A C\ Q, where 0 is some open (closed) set in Rn. Obviously an open (closed) set F in Rn is open (closed) relative to n R . If F C A, and if F is open (closed), then F is open (closed) relative to A for F = A r> F. Problem Show that: (i) Every open ball B\(a) in Rn is an open set. (ii) The closure B*(a) of an open ball B^(a) in Rn is and is closed. (The closure of an open ball is called a closed ball and is denoted by -Bx(a).) (iii) The interior of a closed ball #x(a) is the open ball B\(a). Theorem The family of open sets in Rn has the following properties: (i) Every union of open sets is open. (ii) Every finite intersection of open sets is open. (iii) The empty set 0 and Rn are open. us

B.I

Nonlinear Programming

PROOF (i) Let (I\-),-e/ be a family, finite or infinite, of open sets in Rn. If x £ r = U r,, then x £ I\ for some i £ /, and there exists an « > 0 such that

B.(x) C r, C r Hence £ is an interior point of T, and T is open. (ii) Let r,6/ be a finite family of open sets in Rn. If x £ F = O I\, then a: £ I\ for each i £ /. Because each F, is open, there exist c* > 0 such that Bt<(x) C I\

for V* £ 7

Take e = minimum e' > 0 (this is where the finiteness of / is used), then

B.(X) c r and F is open. (iii) Since the empty set 0 contains no points, we need not find an open ball surrounding any point, and hence 0 is open. The set Rn is open because for each x £ Rn, Bt(x) C Rn for all e > 0. The family of open sets in Rn, as defined in 4> is called a topology n in R . (In fact any family of sets—called open—which has properties (i), (ii), (iii) above is also called a topology in Rn. For example the sets 0, Rn also form a topology in Rn. We shall, however, be concerned here only with open sets as defined by 40 Theorem Let T C Rn-

PROOF

Then

Let x £ Rn.

Then

Hence

Corollary The complement (relative to Rn) of an open set in Rn is closed, and vice versa. 184

B.«

Appendix B

PROOF

T is closed *=> T = T

By using Corollary 12, the following theorem is a direct consequence of Theorem 10. Theorem The family of closed sets in Rn has the following properties: (i) (ii) (iii)

Every intersection of closed sets is closed. Every finite union of closed sets is closed. The empty set 0 and R" are closed. Problem Show that the interior of each set F C Rn is an open set. Problem Show that the closure of each set F C Rn is a closed set. Problem Let A be a fixed m X n matrix and b a fixed m-vector.

(i) (ii) (iii)

Show that the set {x \ x G Rn, Ax ^ b}, and hence also the set {a: | x G Rn, Ax = 6} are closed sets in Rn. Show that the set {x \ x G Rn, Ax < b} is an open set in Rn. Show that the set {x \ x £j Rn, Ax < b} is neither a closed nor an open set in Rn.

2. Sequences and bounds Sequence A sequence in a set X is a function / from the set / of all positive integers into the set X. If /(n) = xn G X for n G I, it is customary to denote the sequence / by the symbol [ x n \ or by x1, z2, . . . . For any sequence of positive integers n1, n2, . . . , such that n1 < n 2 < • • • , the sequence z"1, z"2, . . . is called a subsequence of x1, x2, . . . . If X = Rn, then x1, z2, . . . , is a sequence of points in Rn, and if X = R, then z1, z2, . . . , is the familiar sequence of real numbers. 18*

B.a

Nonlinear Programming

Limit point Let x1, x2, ..., be a sequence of points in Rn. A point x£ Rn is said to be a limit point of the sequence if

(In the literature, a limit point is sometimes called an accumulation point or a cluster point.} Limit Let x1, x2, . . . , be a sequence of points in Rn. A point x° G Rn is said to be a limit of the sequence, or we say that the sequence converges or tends to a limit x° if

Remark

Obviously a. limit of a sequence is also a limit point of the sequence, but the converse is not necessarily true. (For example the sequence 1, — 1, 1, — 1, . . . , has the limit points 1 and —1, but no limit. The sequence 1, 3^j, %, • • • > has a limit, and hence a limit point, zero.) Also note that if x° is a limit of xl, x2, . . . , then it is also a limit of each subsequence of x1, a;2, . . . . Theorem (i) // x is a limit point of the sequence x1, x2, . . . , then there exists a subsequence xn\ xn\ . . . , which has x as a limit, and conversely. (ii) // a sequence x1, x2, . . . , tends to a limit x°, then there can be no limit point (and hence no limit) other than x°. PROOF

(i) Let x be a limit point of x1, re2, . . . .

Then

and so on. The subsequence xn\ xn\ . . . , converges to x. 186

Appendix B

B.3

Conversely, if the subsequence xn\ xn\ . . . , converges to x, then for given e > 0 and n there exists an n* ^ n (in fact all n{ ^ n) such that ||xn< — x|| < t, and hence x is a limit point of xl, a;2, . . . . (ii) Let the sequence x1, x2, . . . , tend to the limit x°, and x ?* x°. Then for 0 < e < ||x — x°||, there exists an integer n° such that (by triangle inequality, see 1.3.10) and hence x is not a limit point of the sequence. Remark Note that a limit point (and hence a limit) of a sequence x1, x2, . . . , in Rn is a point of closure of any set F in Rn containing {x^x2, . . . }. Conversely, if x is a point of closure of the set F C Rn, then there exists a sequence x1, x2, . . . C F (and hence also a subsequence x"1, x"2, . . . ) such that x is a limit point of x1, x2, . . . (and hence a limit of l n2 , . . . )"» . T n , r4 X

Lower and upper bounds Let F be a nonempty set of real numbers.

Then

a is a lower bound of F <=> (x (E F => x ^ a) /3 is an upper bound of F <=» (x G F => x ^ /3) Greatest lower and least upper bounds Let F be a nonempty set of real numbers. A lower bound a is a greatest lower bound of F (infimum of F, or inf F) if no bigger number is a lower bound of F. An upper bound /3 is a least upper bound of F (supremum of F, or sup F) if no smaller number is an upper bound of F. Equivalently we have

We shall take the following axiom as one of the axioms of the real number system [Birkhoff-Maclane 53]. 1ST

B.S

Nonlinear Programming

Axiom Any nonempty set T of real numbers which has a lower (upper) bound has a greatest (least) lower (upper) bound. If the set F has no infimum (or equivalently by the above axiom if it has no lower bound), we say that F is unbounded from below and we write inf F = — «. Similarly if F has no supremum (or equivalently by the above axiom if it has no upper bound), we say T is unbounded from above and we write sup F = + ». Hence by augmenting the Euclidean line R by the two points + °° and — °° any nonempty set F will have an infimum, which may be — °°, and a supremum, which may be + °o. We shall follow the convention of writing inf 0 = -f » and sup 0 = — °°. We observe that neither inf F nor sup F need be in F. For example inf {iy2,lA, . . .} is 0, but 0 is not in the set |1,H,H» • • • } • Theorem Every bounded nondecreasing (nonincreasing) sequence of real numbers has a limit. PROOF Let xl, x2, . . . , be a bounded nondecreasing sequence of real numbers. By the above Axiom 9, the sequence has a least upper bound 0. Hence

(by 8) (because sequence is nondecreasing) (because 0 ^ xn for all n) The proof for a bounded nonincreasing sequence is similar. Cauchy convergence criterion A sequence xl, x2, . . . , in Rn converges to a limit XQ if and only if it is a Cauchy sequence, that is, for each e > 0 there exists an n* such that \\xm — xn\\ < efor each m,n ^ n*. For a proof see [Buck 65, Fleming 65, Rudiri 64].

3.

Compact sets in Rn

Bounded set A set F C Rn is bounded if there exists a real number a such that for each x £ F, ||x|| ^ a. 188

Appendix B

B.S

A set F C Rn which is both closed and bounded has some very interesting properties. In fact such sets constitute one of the most important classes of sets in analysis, the class of compact sets. We give in the theorem below a number of equivalent characterizations of a compact set in Rn. (These characterizations are not necessarily equivalent in spaces more general than Rn.) Compact sets (theorem-definition) A set T C Rn is said to be compact if it satisfies any of the following equivalent conditions: (i) (ii) (iii)

T is closed and bounded. (Bolzano-Weierstrass) Every sequence of points in T has a limit point in T. (Finite intersection property) For any family (r,)ier of sets, closed relative to T, it follows that

or equivalently

(iv)

(Heine-Borel) From every family (I\)t-e/ of open sets whose union VJ I\ contains T we can extract a finite subfamily T tl , . . . , Tim

te/

whose union F,, \J F,2 • • • W I\m contains T (or equivalently stated: every open covering of T has a finite subcovering).

We shall not give here a proof of the equivalence of the above four conditions; such a proof can be found in any of the references given at the beginning of this Appendix. An especially lucid proof is also given in chap. 2 of [Berge-Ghouila Houri 65]. Corollary Let T and A be sets in Rn which are respectively compact and closed. Then the sum ft is closed. PROOF Let x° belong to the closure of Q. Then there exists a sequence xl, x2, . . . , in 8 which converges to x° (see B.2.6). Then we can find 189

Nonlinear Programming

B.3

a sequence y1, yz, . . . , in T and a sequence 21, 22, . . . , in A such that z n = yn + 2"

forn = 1, 2, . . .

Since T is compact, there exists a subsequence i/"1, ynt, . . . , which converges to y° £ T. Then

Hence and ft is closed.

1*0

Appendix C Continuous and Semicontinuous Functions, Minima and Infima

We collect in this appendix some basic definitions and properties of continuous and semicontinuous functions defined on a subset of Rn. We also state some facts about the minima and maxima of such functions. More detailed discussions can be found in [Berge 63, BergeGhouila Houri 65.]

1. Continuous and semicontinuous functions Continuous function A numerical function 8 defined on a set r C Rn is said to be continuous at x° G r (with respect to T) if either of the two following equivalent conditions are satisfied: (i)

For each e > 0 there exists a 5 > 0 such that

(ii)

For each sequence xl, x*, . . . , in T converging to x°, lim 6(xn) = 0(lim xn) = d(x°)

n—»«o

n—»«

6 is said to be continuous on T (with respect to F) if it is continuous (with respect to F) at each point x° £ F, or equivalently if any of the following equivalent conditions hold: (iii) The sets

and

(iv)

are closed relative to F for each real a. The sets

C.I

Nonlinear Programming

and

(v)

are open relative to F for each real a. The epigraph of 6 and the hypograph of 6 are closed relative to F X R.

In each of the above conditions a pair of symmetric requirements can be found. By dropping one requirement from each pair, we arrive at the concept of semicontinuous functions. Lower semicontinuous function A numerical function 0 defined on a set F C Rn is said to be lower semicontinuous at x° G F (with respect to F) if either of the two following equivalent conditions is satisfied : (i)

For each e > 0 there exists a 5 > 0 such that

(ii)

For each sequence x1, x2, . . . , in F converging to x°,

where lim inf 8(x") denotes the infimum of the limit points of the n—+»

sequence of real numbers 8(x1}i 0(z2), . . . . 6 is said to be lower semicontinuous on T (with respect to F) if it is lower semicontinuous (with respect to F) at each point x° G F, or equivalently if any of the following equivalent conditions holds:

(iii)

The set

(iv)

is closed relative to F for each real a. The set

198

C.I

Appendix C

is open relative to F for each real a. (v)

The epigraph of 6 is closed relative to I' X R.

Upper semicontinuous function A numerical function 0 defined on a set F C Rn is said to be upper semicontinuous at x° G F (with respect to F) if either of the two following equivalent conditions is satisfied: (i)

For each e > 0 there exists a 8 > 0 such that

(ii)

For each sequence x1, x2, . . . , in F converging to x°,

where lim sup 0(xn) denotes the supremum of the limit points of the n—»«o

sequence of real numbers 0(x1), 0(z2), . . . . 0 is said to be upper semicontinuous on T (with respect to F) if it is upper semicontinuous (with respect to F) at each point x° G F, or equivalently if any of the following equivalent conditions hold:

(iii)

The set

(iv)

is closed relative to F for each real a. The set

(v)

is open relative to F for each real a. The hypograph of 6 is closed relative to F X R. Remark 6 is lower semicontinuous at x° 6: F (with respect to F) if and only 193

C.I

Nonlinear Programming

if —6 is upper semicontinuous at x° £ T (with respect to r). 8 is continuous at x° (E T (with respect to T) if and only if it is both lower semicontinuous and upper semicontinuous at x° £ F (with respect to r). Examples is lower semicontinuous on R, see Fig.

C.I.I is upper semicontinuous on R, see Fig.

rU1.-£1 .«9

Fig. C.1.1 A lower semicontinuous function on R.

Fig. C.1.2 A upper semicontinuous function on R.

Theorem Let (6i)i£i be a (finite or infinite) family of lower semicontinuous functions on T C Rn> Its least upper bound e(x) = sup 8t(x) i&

is lower semicontinuous on T. If I is finite, then its greatest lower bound 4>(x) = inf ef(x) i&

is also lower semicontinuous on T. PROOF The first part follows directly from #(iii) above and Theorem B,L13(i) if we observe that for any real X, the set

194

Appendix C

C.S

is closed (relative to F). The second part follows from #(iii) and Theorem B.IJ3(ii) if we observe that for any real X the set

is closed (relative to F). Corollary Let (Qi)i£i be a (finite or infinite) family of upper semicontinuous functions on F C Rn. Us greatest lower bound

is upper semicontinuous on F. If I is finite, then its least upper bound

is also upper semicontinuous on T.

2.

Intimum (supremum) and minimum (maximum) of a set of real numbers

We recall that in Appendix B (B.2.8] we defined the infimum and supremum of a set of real numbers F as follows

and

We also noted that neither a nor 0 need be in F. However when a and |5 are in F, they are called respectively min F and max F. Minimum (maximum) Let F be a set of real numbers. If (inf F) £ F, then inf F is called the minimum of F and is denoted by min F. If (sup F) (E F, then sup F is called the maximum of F and is denoted by max F. Equivalently

195

C.S

Nonlinear Programming

3. Infimum (supremum) and minimum (maximum) of a numerical function Bounded functions A numerical function 6 defined on the set F is said to be bounded from below on T if there exists a number a such that The number a is a lower bound of 8 on F. 6 is said to be bounded from above on T if there exists a number /3 such that The number ft is an upper bound of 0 on F. Infimum of a numerical function Let 0 be a numerical function denned on the set F. If there is a number a such that

and

then 5 is called the infimum of 9 on F, and we write

Supremum of a numerical function Let 6 be a numerical function defined on the set F. number 0 such that

If there is a

and

then /3 is called the supremum of 6 on F, and we write

If we admit the points + », then every numerical function 6 has a supremum and infimum on the set F on which it is defined. 196

Appendix C

C.S

Examples

Minimum of a numerical function Let 6 be a numerical function denned on the set F. If there exists an x G F such that

then 0(z) is called the minimum of 6 on F, and we write

We make the following remarks: Not every numerical function has a minimum; for example, e~x and x have no minima on R. (ii) The minimum of a numerical function, if it exists, must be finite, (iii) The minimum of a numerical function, if it exists, is an attained infimum, that is, (i)

Maximum of a numerical function Let 6 be a numerical function denned on the set F. If there exists an x G F such that

then 6(x) is called the maximum of 0 on F and we write

Remarks similar to (i), (ii), and (iii) above, which applied to the minimum of a function, also apply here to the maximum of a function. 197

Nonlinear Programming

C.4

4. Existence of a minimum and a maximum of a numerical function We give now sufficient conditions for a numerical function defined on a subset of Rn to have a minimum and a maximum on that subset. Theorem A lower (upper) semicontinuous function 6 defined on a compact— that is, closed and bounded—set T in Rn is bounded from below (above) and attains in T the value

In other words there exists an x G r such that

PROOF the set

We prove the lower semicontinuous case.

Let 7 > a.

is not empty and is closed relative to T by C.1.2(i\\). finite intersection theorem B.3.2(iii)

Then

Hence by the

Choose x in this intersection; then d(x) = a, and hence a is finite. It should be remarked here that the above theorem cannot be strengthened by dropping the semicontinuity or the compactness assumptions. In other words, in order to ensure that

we cannot weaken the conditions that (i) (ii) (iii)

8 is lower semicontinuous on F, T is closed, and T is bounded.

We give examples below where the infimum is not attained whenever any one of the above conditions is violated. 198

Appendix C

C.4

(i)

inf

8(x) = 0, but no minimum exists on the compact set

(ii) inf 8(x) = 0, but no minimum exists of this continuous function

0<x<1

on the open set

(iii) inf 6(x) = 0, but no minimum exists of this continuous function on

*££

the closed unbounded set R.

199

Appendix D Differentiable Functions, Mean-value and Implicit Function Theorems

We summarize here the pertinent definitions and properties of differentiable functions and give the mean-value and implicit function theorems. For details the reader is referred to [Fleming 65, Rudin64, Hestenes 66, Bartle 64].

1. Differentiable and twicedifferentiable functions Differentiable numerical function Let 6 be a numerical function defined on an open set F in Rn, and let x be in F. 6 is said to be differentiable at x if for all x G Rn such that x + x G F we have that

where t(£) is an n-dimensional bounded vector, and a is a numerical function of x such that 6 is said to be differentiable on F if it is differentiable at each x in F. [Obviously if 6 is differentiable on the open set F, it is also differentiable on any subset A (open or not) of F. Hence when we say that 6 is differentiable on some set A (open or not), we shall mean that 6 is differentiable on some open set containing A.] Partial derivatives and gradient of a numerical function Let 6 be a numerical function defined on an open set F in Rn, and let x be in F. 0 is said to have a partial derivative at x with respect to Xi, i = 1, . . . , n, if

Appendix D

D.I

tends to a finite limit when 5 tends to zero. This limit is called the partial derivative of 6 with respect to x, at x and is denoted by 66(x)/dXi. The n-dimensional vector of the partial derivatives of 6 with respect to Xi, . . . , xn at x is called the gradient of 6 at x and is denoted by V0(z), that is,

Theorem Let 6 be a numerical function defined on an open set Y in Rn, and let x be in T. (i) // 6 is differentiate at x, then 6 is continuous at x, and V0(z) exists (but not conversely), and

(ii) // 6 has continuous partial derivatives at x with respect to xi, . . . , xn, that is, V6(x) exists and V0 is continuous at x, then 6 is differentiate at x. We can summarize most of the above result as follows

Differentiable vector function Let / be an w-dimensioiial vector function defined on an open set F in Rn, and let x be in F. /is said to be differentiate at x (respectively on T) if each of its components f i , . . . ,m f is differentiate at x (respectively on F). Partial derivatives and Jacobian of a vector function Let / be an m-dimensional vector function defined on an open set F in Rn, and let x be in T. f is said to have partial derivatives at x with 201

Nonlinear Programming

D.I

respect to Xi, . . . , xn if each of its components f i , . . . , f derivatives at x with respect to Xi, . . . , xn. We write

m

has partial

The m X n matrix V/(x) is called the Jacobian (matrix) of f at x. Chain rule theorem Let f be an m-dimensional vector function defined on an open set F in Rn, and let $ be a numerical function which is defined on- Rm. The numerical function 6 defined on F by 6(x) = *[/(*)] is differentiate at x G F if f is differentiate aty = f(x), and

at x and if is

differentiate

V6(x) = V0(£)V/(z) Twice-differentiable numerical function and its Hessian Let 6 be a numerical function denned on an open set F in Rn, and let x be in F. $ is said to be twice differentiate at x if for all x G Rn such that x -f- x G r we have that

where V26(x) is an n X n matrix of bounded elements, and /3 is a numerical function of x such that lim 0(x,x) = 0 z-*0

The n X n matrix V20(z) is called the Hessian (matrix) of 6 at x and its ijth element is written as

Obviously if 8 is twice differentiate at x, it must also be differentiate at x. Theorem Let 6 be a numerical function defined on an open set F in Rn, and kt x be in F. Then aoa

Appendix D

(i) (ii)

D.I

V0 differentiable at x => 6 twice differentidble at x V0 has continuous partial derivatives at x => 6 twice differentiable at x

Remark The numbers dd(x)/dxi, i' = 1, . . . , n, are also called the first partial derivatives of 6 at x, and d*Q(x)/dXi dx, i,j = 1, . . . , n, are also called the second partial derivatives of 6 at x. In an analogous way we can define /cth partial derivatives of 6 at x. Remark Let 6 be a numerical function defined on an open set F C Rn X Rk which is differentiable at (x,y) (E T. We define then

Let / be an m-dimensional function defined on an open set F C Rn X Rk which is differentiable at (x,y) G r. We define then

and

108

Nonlinear Programming

D.3

2. Mean-value theorem and Taylor's theorem Mean-value theorem Let 6 be a differentiate numerical function on the open convex set T C Rn, and let xl,x2 E T. Then for some real number y, 0 < 7 < 1. Taylor's theorem (second order) Let 6 be a twice-differentiable numerical function on the open convex set T C Rn, and let x\x2 E T. Then

for some real number

3. Implicit function theorem Implicit function theorem Let f be an m-dimensional vector function defined on an open set A C Rn X Rm, let f have continuous first partial derivatives at (x,y) G: A, let f(x,y) = 0, and let Vvf(x,y) be nonsingular. Then there exist a ball Bx(x,y) with radius X > 0 in Rn+m, an open set F C -Kn containing x, and an m-dimensional vector function e with continuous first partial derivatives on F such that Vvf(x>y)

and

204

*s nonsingular for all (x,y} £ B\(x,y)

References Abadie, J.: On the Kuhn-Tucker Theorem, in J. Abadie (ed.), "Nonlinear Programming," pp. 21-36, North Holland Publishing Company, Amsterdam, 1967. Anderson, K. W., and D. W. Hall: "Sets, Sequences and Mappings, the Basic Concepts of Analysis," John Wiley & Sons, Inc., New York, 1963. Arrow, K. J., and A. C. Enthoven: Quasiconcave Programming, Econometrica 29:779-800 (1961). Arrow, K. J., L. Hunvicz, and H. Uzawa (eds.): "Studies in Linear and Nonlinear Programming," Stanford University Press, Stanford, Calif., 1958. Arrow, K. J., L. Hurwicz, and H. Uzawa: Constraint Qualifications in Maximization Problems, Naval Research Logistics Quarterly 8: 175-191 (1961). Bartle, R. G.: "The Elements of Real Analysis," John Wiley & Sons, Inc., New York, 1964. Berge, C.: "Topological Spaces," The MacMillan Company, New York, 1963. Berge, C., and A. Ghouila Houri: "Programming, Games and Transportation Networks," John Wiley & Sons, Inc., New York, 1965. Birkhoff, G. and S. Maclane: "A Survey of Modern Algebra," The MacMillan Company, New York, 1953. Bohnenblust, H. F., S. Karlin, and L. S. Shapley: "Solutions of Discrete, Twoperson Games," Contributions to the Theory of Games, vol. I, Annals of Mathematics Studies Number 24, Princeton, 1950, 51-72. Bracken, J., and G. P. McCormick: "Selected Applications of Nonlinear Programming," John W T iley & Sons, Inc., New York, 1968. Brondsted, A.: Conjugate Convex Functions in Topological Vector Spaces, Matematisk-fysiske Meddelel ser udgivet af Dei Kongelige Danske Videnskabernes Selskab, 34(2): 1-27 (1964). Browder, F. E.: On the Unification of the Calculus of Variations and the Theory of Monotone Nonlinear Operators in Banach Spaces, Proc. Nat. Acad. Sci. U.S., 66: 419-425 (1966). Buck, R. C.: "Advanced Calculus," McGraw-Hill Book Company, New York, 1965. Canon, M., C. Cullum, and E. Polak: Constrained Minimization Problems in Finite-dimensional Spaces, Society for Industrial and Applied Mathematics Journal on Control, 4: 528-547 (1966). Carathe'odory, C.: tlber den variabilitatsberich der Koeffizienten von Potenzreihen, die gegebene Werte nicht annehmen, Mathematische Annalen 64: 95-115 (1907).

205

Nonlinear Programming

Charnes, A., and W. W. Cooper: "Management Models and Industrial Applications of Linear Programming," vols. I, II, John Wiley & Sons, Inc., New York, 1961. Cottle, R. W.: Symmetric Dual Quadratic Programs, Quarterly of Applied Mathematics 21: 237-243 (1963). Courant, R.: "Differential and Integral Calculus," vol. II, 2d ed., rev., Interscience Publishers, New York, 1947. Courant, R., and D. Hilbert: "Methods of Mathematical Physics," pp. 231-242, Interscience Publishers, New York, 1953. Dantzig, G. B.: "Linear Programming and Extensions," Princeton University Press, Princeton, N.J., 1963. Dantzig, G. B., E. Eisenberg, and R. W. Cottle: Symmetric Dual Nonlinear Programs, Pacific Journal of Mathematics, 15: 809-812 (1965). Dennis, J. B.: "Mathematical Programming and Electrical Networks," John Wiley & Sons, Inc., New York, 1959. Dieter, U.: Dualitat bei konvexen Optimierungs—(Programmierungs—)Aufgaben, Unternehmensforschung 9: 91-111 (1965a). Dieter, U.: Dual External Problems in Locally Convex Linear Spaces, Proceedings of the Colloquium on Convexity, Copenhagen, 52-57 (1965b). Dieter, U.: Optimierungsaufgaben in topologischen Vektorraumen I: Dualitatstheorie, Zeitschrift fur \Vahrscheinlichkeitstheorie und Verwandte Gebiete, 5: 89-117 (1966). Dorn, W. S.: Duality in Quadratic Programming, Quarterly of Applied Mathematics, 18: 155-162 (1960). Dorn, W. S.: Self-dual Quadratic Programs, Society for Industrial and Applied Mathematics Journal on Applied Mathematics, 9: 51-54 (1961). Duffin, R. J.: Infinite Programs, in [Kuhn-Tucker 56], pp. 157-170. Fan, K., I. Glicksburg, and A. J. Hoffman: Systems of Inequalities Involving Convex Functions, American Mathematical Society Proceedings, 8: 617-622 (1957). Farkas, J.: Uber die Theorie der einfachen Ungleichungen, Journal fur die Heine und Angewandte Mathematik, 124: 1-24 (1902). Fenchel, W.: "Convex Cones, Sets and Functions," Lecture notes, Princeton University, 1953, Armed Services Technical Information Agency, AD Number 22695. Fiacco, A. V.: Second Order Sufficient Conditions for Weak and Strict Constrained Minima, Society for Industrial and Applied Mathematics Journal on Applied Mathematics, 16: 105-108 (1968). 106

References

Fiacco, A. V., and G. P. McCormick: "Nonlinear Programming: Sequential Unconstrained Minimization Techniques," John Wiley &Sons, Inc., New York, 1968. Fleming, W. H.: "Functions of Several Variables," McGraw-Hill Book Company, New York, 1965. Friedrichs, K. 0.: Ein Verfahren der Variationsrechnung des Minimum eines Integral als das Maximum eines Anderen Ausdruckes Darzustellen, Nachrichten von der Gesellschajl der Wissenschaften zu Gottingen Mathematische—Physikalische Klasse, 1320(1929). Gale, D.: "The Theory of Linear Economic Models," McGraw-Hill Book Company, New York, 1960. Gale, D., H. W. Kuhn, and A. W. Tucker: Linear Programming and the Theory of Games, in [Koopmans 51], pp. 317-329. Gass, S.: "Linear Programming," 2d ed., McGraw-Hill Book Company, New York, 1964. Gol'stein, E. G.: Dual Problems of Convex and Fractionally-convex Programming in Functional Spaces, Soviet Mathematics-Doklady (English translation), 8: 212-216 (1967). Gordan, P.: Uber die Auflosungen linearer Gleichungen mit reelen Coefficienten, Mathematische Annalen, 6: 23-28 (1873). Graves, R. L., and P. Wolfe (ed.): "Recent Advances in Mathematical Programming," McGraw-Hill Book Company, New York, 1963. Hadley, G. F.: "Linear Programming," Addison-Wesley Publishing Company, Inc., Mass., 1962. Hadley, G. F.: "Nonlinear and Dynamic Programming," Addison-Wesley Publishing Company, Inc., Reading, Mass., 1964. Halkin, H.: A Maximum Principle of the Pontryagin Type for Systems Described by Nonlinear Difference Equations, Society for Industrial and Applied Mathematics Journal on Control, 4: 90-111 (1966). Halkin, H., and L. W. Neustadt: General Necessary Conditions for Optimization Problems, Proc. Nat. Acad. Sci. U.S., 56: 1066-1071 (1966). Halmos, P. R.: "Finite-dimensional Vector Spaces," D. Van Nostrand Company, Inc., Princeton, N.J., 1958. Hamilton, N. T., and J. Landin: "Set Theory: The Structure of Arithmetic," Allyn and Bacon, Inc., Boston, 1961. Hanson, M. A.: A Duality Theorem in Nonlinear Programming with Nonlinear Constraints, Australian Journal of Statistics 3: 64-71 (1961). 207

Nonlinear Programming:

Hanson, M. A.: Bounds for Functionally Convex Optimal Control Problems, Journal of Mathematical Analysis and Applications, 8: 84-89 (1964). Hanson, M. A.: Duality for a Class of Infinite Programming Problems, Society for Industrial and Applied Mathematics Journal on Applied Mathematics, 16: 318-323 (1968). Hanson, M. A., and B. Mond: Quadratic Programming in Complex Space, Journal of Mathematical Analysis and Applications, 23: 507-514 (1967). Hestenes, M. R.: "Calculus of Variations and Optimal Control Theory," John Wiley & Sons, Inc., New York, 1966. Hu, T. C.: "Integer Programming and Network Flows," Addison-Wesley Publishing Company, Inc., Reading, Mass., 1969. Huard, P.: Dual Programs, 75.17 J. Res. Develop., 6: 137-139 (1962). Hurwicz, L.: Programming in Linear Spaces, in [Arrow et al. 58], pp. 38-102. Jacob, J.-P., and P. Rossi: "General Duality in Mathematical Programming," IBM Research Report, 1969. Jensen, J. L. W. V.: Sur les fonctions convexes et les ine'galite's entre les valeurs moyennes, Acta Mathematica, 30: 175-193 (1906). John, F.: Extremum Problems with Inequalities as Subsidiary Conditions, in K. 0. Friedrichs, O. E. Neugebauer, and J. J. Stoker, (eds.), "Studies and Essays: Courant Anniversary Volume," pp. 187-204, Interscience Publishers, New York, 1948. Karamardian, S.: "Duality in Mathematical Programming," Operations Research Center, University of California, Berkeley, Report Number 66-2, 1966; Journal of Mathematical Analysis and Applications, 20: 344-358 (1967). Karlin, S.: "Mathematical Methods and Theory in Games, Programming, and Economics," vols. I, II, Addison-Wesley Publishing Company, Inc., Reading, Mass., 1959. Koopman, T. J. (ed.): "Activity Analysis of Production and Allocation," John Wiley & Sons, Inc., New York, 1951. Kowalik, J., and M. R. Osborne: "Methods for Unconstrained Optimization Problems," American Elsevier Publishing Company, Inc., New York, 1968. Kretschmar, K. S.: Programmes in Paired Spaces, Canadian Journal of Mathematics, 13: 221-238 (1961). Kuhn, H. W., and A. W. Tucker: Nonlinear Programming, in J. Neyman (ed.), "Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability," pp. 481-492, University of California Press, Berkeley, Calif., 1951. 908

Beferencea

Kuhn, H. W., and A. W. Tucker: "Linear Inequalities and Related Systems," Annals of Mathematics Studies Number 38, Princeton University Press, Princeton, N.J., 1956. Ktinzi, H. P., W. Krelle, and W. Oettli: "Nonlinear Programming," Blaisdell Publishing Company, Waltham, Mass., 1966. Larsen, A., and E. Polak: Some Sufficient Conditions for Continuous-linearprogramming Problems," International Journal of Engineering Science, 4: 583604 (1966). Lax, P. D.: Reciprocal Extremal Problems in Function Theory, Communications on Pure and Applied Mathematics, 8: 437-453 (1955). Levinson, N.: Linear Programming in Complex Space, Journal of Mathematical Analysis and Applications, 14: 44-62 (1966). Levitin, E. S., and B. T. Polyak: Constrained Minimization Methods, USSR Computational Mathematics and Mathematical Physics (English translation), 6(5):1-50 (1966). McCormick, G. P.: Second Order Conditions for Constrained Minima, Society for Industrial and Applied Mathematics Journal on Applied Mathematics, 16: 641-652 (1967). Mangasarian, 0. L.: Duality in Nonlinear Programming, Quarterly of Applied Mathematics, 20: 300-302 (1962). Mangasarian, 0. L.: Pseudo-convex Functions, Society for Industrial and Applied Mathematics Journal on Control, 3: 281-290 (1965). Mangasarian, O. L., and S. Fromovitz: A Maximum Principle in Mathematical Programming, in A. V. Balakrishnan and L. W. Neustadt (eds.), "Mathematical Theory of Control," pp. 85-95, Academic Press Inc., New York, 1967. Mangasarian, 0. L., and S. Fromovitz: The Fritz John Necessary Optimality Conditions in the Presence of Equality and Inequality Constraints, J. Math. Analysis and Applications, 17:37-47 (1967a). Mangasarian, 0. L., and J. Ponstein: Minmax and Duality in Nonlinear Programming, Journal of Mathematical Analysis and Applications, 11: 504-518, (1965). Martos, B.: The Direct Power of Adjacent Vertex Programming Methods, Management Science, 12: 241-252 (1965). Minty, G. J.: On the Monotonicity of the Gradient of a Convex Function, Pacific Journal of Mathematics, 14: 243-247 (1964). Mond, B.: A Symmetric Dual Theorem for Nonlinear Programs, Quarterly of Applied Mathematics, 23: 265-269 (1965). S09

Nonlinear Programming

Mond, B., and R. W. Cottle: Self-duality in Mathematical Programming, Society for Industrial and Applied Mathematics Journal on Applied Mathematics, 14: 420-423 (1966). Mond, B., and M. A. Hanson: Duality for Variational Problems, Journal of Mathematical Analysis and Applications, 18: 355-364 (1967). Moreau, J.-J.: Proximite* et dualite* dans un espace Hilbertien, Bulletin de la Societe Mathematique de France, 93: 273-299 (1965). Motzkin, T. S.: "Beitrage zur Theorie der Linearen Ungleichungen," Inaugural Dissertation, Basel, Jerusalem, 1936. Nikaido, H.: On von Neumann's Minimax Theorem, Pacific Journal of Mathematics, 4: 65-72 (1954). Opial, Z.: "Nonexpansive and Monotone Mappings in Banach Spaces," Lecture Notes 67-1, Division of Applied Mathematics, Brown University, Providence, Rhode Island, 1967. Ponstein, J.: Seven Types of Convexity, Society for Industrial and Applied Mathematics Review, 9: 115-119 (1967). Pontryagin, L. S., V. G. Boltyanskii, R. V. Gamkrelidze, and E. F. Mishchenko: "The Mathematical Theory of Optimal Processes," John Wiley & Sons, Inc., New York, 1962. Rissanen, J.: On Duality Without Convexity, Journal of Mathematical Analysis and Applications, 18: 269-275 (1967). Ritter, K.: Duality for Nonlinear Programming in a Banach Space, Society for Industrial and Applied Mathematics Journal on Applied Mathematics, 15: 294-302 (1967). Rockafellar, R. T.: "Convex Functions and Dual Extremum Problems," Doctoral dissertation, Harvard University, 1963. Rockafellar, R. T.: Duality Theorems for Convex Functions, Bulletin of the American Mathematical Society, 70: 189-192 (1964). Rockafellar, R. T.: Conjugates and Legendre Transforms of Convex Functions, Canadian Journal of Mathematics, 19: 200-205 (1967a). Rockafellar, R. T.: Convex Programming and Systems of Elementary Monotonic Relations, Journal of Mathematical Analysis and Applications, 19: 543-564 (1967b). Rockafellar, R. T.: Duality and Stability in Extremum Problems Involving Convex Functions, Pacific Journal of Mathematics, 21: 167-187 (1967c). Rockafellar, R. T.: "Convex Analysis," Princeton University Press, Princeton, N.J., 1969.

210

References

Rosen, J. B.: The Gradient Projection Method for Nonlinear Programming, Society for Industrial and Applied Mathematics Journal on Applied Mathematics, 8: 181-217 (1960) and 9: 514-553 (1961). Rubinstein, G. S.: Dual Extremum Problems, Soviet Mathematics-Doklady (English translation), 4: 1309-1312 (1963). Rudin, W.: "Principles of Mathematical Analysis," 2d ed., McGraw-Hill Book Company, New York, 1964. Saaty, T. L., and J. Bram: "Nonlinear Mathematics," McGraw-Hill Book Company, New York, 1964. Simmonard, M.: "Linear Programming," Prentice-Hall, Inc., Englewood Cliffs, N.J., 1966. Simmons, G. F.: "Introduction to Topology and Modern Analysis," McGrawHill Book Company, New York, 1963. Slater, M.: "Lagrange Multipliers Revisited: A Contribution to Nonlinear Programming," Cowles Commission Discussion Paper, Mathematics 403, November, 1950. Slater, M. L.: A Note on Motzkin's Transposition Theorem, Econometrica, 19: 185-186 (1951). Stiemke, E.: t)ber positive Losungen homogener linearer Gleichungen, Mathematische Annalen, 76: 340-342 (1915). Stoer, J.: Duality in Nonlinear Programming and the Minimax Theorem, Numerische Mathematik, 5: 371-379 (1963). Stoer, J.: tlber einen Dualitatssatz der nichtlinearen Programmierung, Numerische Mathematik, 6: 55-58 (1964). Trefftz, E.: "Ein Gegenstiick zum Ritzschen Verfahren," Verhandlungen des Zweiten Internationalen Kongresses fur Technische Mechanik, p. 131, Zurich, 1927. Trefftz, E.: Konvergenz und Fehlerschatzung beim Ritzschen Verfahren, Mathematische Annalen, 100: 503-521 (1928). Tucker, A. W.: Dual Systems of Homogeneous Linear Relations, in [KuhnTucker 56], pp. 3-18. Tuy, Hoang: Sur les ine'galite's line*aires, Colloquium Mathematicum, 13: 107-123 (1964). Tyndall, W. F.: A Duality Theorem for a Class of Continuous Linear Programming Problems, Society for Industrial and Applied Mathematics Journal on Applied Mathematics, 13: 644-666 (1965). 111

Nonlinear Programming

Tyndall, W. F.: An Extended Duality Theorem for Continuous Linear Programming Problems, Notices of the American Mathematical Society, 14: 152-153 (1967). Uzawa, H.: The Kuhn-Tucker Theorem in Concave Programming, in [Arrow et al. 58], pp. 32-37. Vajda, S.: "Mathematical Programming," Addison-Wesley Publishing Company, Inc., Reading, Mass., 1961. Valentine, F. A.: "Convex Sets," McGraw-Hill Book Company, New York, 1964. Van Slyke, R. M., and R. J. B. Wets: "A Duality Theory for Abstract Mathematical Programs with Applications to Optimal Control Theory," Mathematical Note Number 538, Mathematics Research Laboratory, Boeing Scientific Research Laboratories, October, 1967. Varaiya, P. P.: Nonlinear Programming in Banach Space, Society for Industrial and Applied Mathematics Journal on Applied Mathematics, 16: 284-293 (1967). von Neumann, J.: Zur Theorie der Gesellschaftsspiele, Mathematische Annalen, 100: 295-320(1928). Whinston, A.: Conjugate Functions and Dual Programs, Naval Research Logistics Quarterly, 12: 315-322 (1965). Whinston, A.: Some Applications of the Conjugate Function Theory to Duality, in [Abadie 67], pp. 75-96. Wolfe, P.: A Duality Theorem for Nonlinear Programming, Quarterly of Applied Mathematics, 19: 239-244 (1961). Zangwill, W. I.: "Nonlinear Programming: A Unified Approach," Prentice-Hall, Inc., Englewood Cliffs, N.J., 1969. Zarantonello, E. H.: "Solving Functional Equations by Contractive Averaging," Mathematics Research Center, University of Wisconsin, Technical Summary Report Number 160, 1960. Zoutendijk, G.: "Methods of Feasible Directions," Elsevier Publishing Company, Amsterdam, 1960. Zukhovitskiy, S. I., and L. I. Avdeyeva: "Linear and Convex Programming," W. B. Saunders Company, Philadelphia, 1966.

ata

Indexes

This page intentionally left blank

Name Index Abadie, J., 97, 99, 100, 205 Almgren, F. J., 62 Anderson, K. W., 3, 205 Arrow, K. J., 102, 103, 151, 205 Avdeyeva, L. I., 212

Fiacco, A. V., 206, 207 Fleming, W. H., 2, 62, 71, 182, 188, 200, 207 Friedrichs, K. 0., 113, 207 Fromovitz, S., 72, 163, 170, 172, 209

Bartle, R. G., 200, 205 Berge, C., 3, 46, 55, 68, 122, 132, 182 189, 191, 205 Birkhoff, G., 187, 205 Bohnenblust, H. F., 67, 205 Boltyanskii, V. G., 72, 163, 210 Bracken, J., 205 Bram, J., 211 Brondsted, A., 122, 205 Browder, F. E., 87, 205 Buck, R. C., 182, 188, 205

Gale, D., 16, 33, 35, 113, 127, 177, 179, 207 Gamkrelidze, R. V., 72, 163, 210 Gass, S., 15, 207 Ghouila Houri, A., 55, 68, 122, 132, 189, 191, 205 Glicksburg, I., 63, 65, 206 Gol'stein, S., 122, 207 Gordan, P., 31, 207 Graves, R. L., 207

Canon, M., 72, 163, 170, 205 Carathe"odory, C., 43, 205 Charnes, A., 206 Cooper, W. W., 206 Cottle, R. W., 122, 206, 210 Courant, R., 2, 113, 206 Cullum, C., 72, 163, 170, 205 Daniel, J. W., 130 Dantzig, G. B., 15, 18, 113, 122, 206 Dennis, J. B., 113, 206 Dieter, U., 122, 206 Dorn, W. S., 117, 122, 124, 125, 206 Duffin. R. J.. 32, 206 Eisenberg, E., 122, 206 Enthoven, A. C., 151, 205

Fan, K., 63, 65, 206 Farkas, J., 16, 31, 206 Fenchel, W., 55, 122, 206

Hadley, G. F., 15, 207 Halkin, H., 72, 163, 170, 207 Hall, D. W., 3, 205 Halmos, P. R., 177, 207 Hamilton, N. T., 3, 207 Hanson, M. A., 113, 117, 118, 122, 137, 207, 208, 210 Hestenes, M. R., 200, 208 Hilbert, D., 113, 206 Hoffman, A. J., 63, 65, 206 Hu, T. C., 208 Huard, P., 115, 117, 118, 122, 208 Hurwicz, L., 102, 103, 122, 205, 208 Jacob, J.-P., 122, 208 Jensen, J. L. W. V., 61, 208 John, F., 77, 99, 208 Karamardian, S., 87, 117, 122, 137, 139, 208 Karlin, S., 67, 77-79, 205, 208 Koopmans, T. J., 208 Kowalik, J., 208 Krelle, W., 209

Nonlinear Programming

Kretschmar, K. S., 122, 208 Kuhn, H. W., 79, 94, 102, 105, 113, 127, 207-209 Kiinzi, H. P., 209

Rosen, J. B., 211 Rossi, P., 122, 208 Rubinstein, G. S., 122, 211 Rudin, W., 182, 188, 200, 211

Landin, J., 3, 207 Larsen, A., 122, 209 Lax, P. D., 113, 209 Levinson, N., 122, 209 Levitin, E. S., 209

Saaty, T. L., 211 Shapley, L. S., 67, 205 Simmonard, M., 15, 113, 211 Simmons, G. F., 182, 211 Slater, M. L., 27, 78, 211 Stiemke, E., 32, 211 Stoer, J., 122, 211

McCormick, G. P., 205, 207, 209 MacLane, S., 187, 205 Mangasarian, 0. L., 72, 117, 118, 122, 131, 140, 151, 157, 163, 170, 172, 209 Martos, B., 132, 137, 151, 209 Minty, G. J., 87, 209 Mishchenko, E. F., 72, 163, 210 Mond, B., 113, 122, 208-210 Moreau, J.-J., 122, 210 Motzkin, T. S., 28, 210 Neustadt, L. W., 163, 207 Nikaid6, H., 131, 132, 210 Oettli, W., 209 Opial, Z., 87, 210 Osborne, M. R., 208 Polak, E., 72, 163, 170, 205, 209 Polyak, B. T., 209 Ponstein, J., 117, 118, 122, 209, 210 Pontryagin, L. S., 72, 163, 210 Rissanen, J., 122, 210 Ritter, K., 122, 210 Rockafellar, R. T., 87, 122, 210

216

Trefftz, E., 113, 211 Tucker, A. W., 16, 20, 21, 24, 25, 28, 29, 79, 94, 102, 105, 113, 127, 207209, 211 Tuy, Hoang, 131, 140, 151, 211 Tyndall, W. F., 122,211, 212 Uzawa, H., 77, 79, 80, 102, 103, 205, 212 Vajda, S., 212 Valentine, F. A., 46, 55, 212 Van Slyke, R. M., 122, 212 Varaiya, P. P., 122, 212 von Neuman, J., 113, 212 Wets, R. J. B., 122, 212 Whinston, A., 122, 212 Wolfe, P., 114, 115, 121, 122, 207, 212 Zangwill, W. L, 212 Zarantonello, E. H., 87, 212 Zoutendijk, G., Ill, 212 Zukhovitskiy, S. L, 212

Subject Index Alternative: table of theorems of, 34 theorems of, 27-37 Angle between two vectors, 8 Axiom of real number system, 188 Ball: closed, 183 open, 182 Basis, 178 Bi-nonlinear function, 149 Bolzano-Weierstrass theorem, 189 Bounded function, 196 Bounded set, 188 Bounds: greatest lower and least upper, 187 lower and upper, 187 Caratheodory's theorem, 43 Cauchy convergence, 188 Cauchy-Schwarz inequality, 7 Closed set, 183 Closure of a set, 183 Compact set, 189 Concave function, 56 differentiable, 83-91 minimum of, 73 strictly, 57 strictly concave: and differentiable, 87-88 and twice differentiable, 90-91 twice differentiable, 88-90 Concave functions, infimum of, 61 Constraint qualification: Arrow-Hurwicz-Uzawa, 102 modified, 172 weak, 154 with convexity, 78-81 with differentiability, 102-105, 154-156, 171-172 Karlin's, 78 Kuhn-Tucker, 102, 171

Constraint qualification: reverse convex, 103 weak, 155, 172 Slater's, 78 weak, 155 strict, 79 Constraint qualifications, relationships between, 79, 103, 155 Continuous function, 191-192 Convex: and concave functions, 54-68 generalized, 131-150 and pseudoconvex functions, relation between, 144, 146 Convex combinations. 41 Convex function, 55 continuity of, 62 differentiable, 83-91 at a point, 83 on a set, 84 strictly, 56 strictly convex: and differentiable, Q7 fifi oi~ oo

and twice differentiable, 90-91 twice differentiable, 88-90 Convex functions: fundamental theorems for, 63-68 supremum of, 61 Convex hull, 44 Convex set, 39 Convex sets, 38-53 intersection of, 41 Differentiable function: numerical, 200 vector, 201 Distance between two points, 7 Dual linear problem (LDP), 127, 130 Dual (maximization) problem (DP), 114, 123 Dual problem, unbounded, 119, 121, 126, 127 Dual variables, 71

Nonlinear Programming

Duality, 113-130, 157-160, 174-176 in linear programming, 126-130 with nonlinear equality constraints, 174-176 in nonlinear programming, 114-123 in quadratic programming, 123-126 Duality theorem: Dorn's, 124 Dorn's strict converse, 125 Hanson-Huard strict converse, 118 linear programming, 127 strict converse, 117, 118, 124, 157, 160, 174, 176 weak, 114, 124 Wolfe's, 115 Equality constraints: linear, 80, 95 nonlinear, 75, 112, 161-176 Euclidean space Rn , 6 Farkas' theorem, 16, 31, 37, 53 nonhomogeneous, 32 Feasible directions, method of, 111 Finite-intersection property, 189 Fractional function: linear, 149 nonlinear, 148 Fritz John saddlepoint necessary optimality theorem, 77 Fritz John saddlepoint problem (FJSP), 71 Fritz John stationary-point necessary optimality theorem, 99, 170 Fritz John stationary-point problem (FJP), 93 Function, 11 linear vector, 12 numerical, 12 vector, 12 Gale's theorems, 33 Gordan's theorem, 31

ais

Gordan's theorem: generalized to convex functions, 65 Gradient of a function, 201 Halfspace, 40 Heine-Borel theorem, 189 Hessian matrix, 202 Implicit function theorem, 204 Inequalities, linear, 16-37 Infimum, 187, 195 of a numerical function, 196 Interior of a set, 183 Interior point, 182 Jacobian matrix, 202 Kuhn-Tucker saddlepoint necessary optimality theorem, 79 in the presence of equality constraints, 80 Kuhn-Tucker saddlepoint problem (KTSP), 71 Kuhn-Tucker stationary-point necessary optimality criteria, 105, 111, 112, 156, 173 Kuhn-Tucker stationary-point problem (KTP), 94 Kuhn-Tucker sufficient optimality theorem, 94, 153, 162 Lagrange multipliers, 71 Lagrangian function, 71 Limit, 186 Limit point, 186 Line, 38 segments, 39 Linear combination, 6 nonnegative, 7 Linear dependence, 6

Subject Index

Linear inequalities, 16-37 Linear minimization problem (LMP), 127, 130 Linear programming: duality, 126-130 optimality, 18-21 Linear systems, existence theorems, 21-26 Linearization lemma, 97, 153, 163 Local minimization problem (LMP), 70, 93 Mapping, 11 Matrices, 8-11 Matrix: nonvacuous, 11 submatrices of, 10 Maximum, 195 of a numerical function, 197 existence of, 198 Maximum principle (see Minimum principle) Mean-value theorem, 204 Metric space, 7 Minimization problem (MP), 70, 93 local, 70, 93 solution set, 72 uniqueness, 73 Minimum, 195 of a numerical function, 197 existence of, 198 Minimum principle, 72, 162-170, 141-142 Motzkin's theorem, 28 Negative definite matrix, 91, 181 Negative semidefinite matrix, 89, 90, 181 Nonlinear programming problem 1-3, 14-15 Nonsingular matrix, 181 Norm of a vector, 7 Notation, 13-15 Numerical function, 12

Open set, 183 Optimality criteria: with differentiability, 92-112, 151-176 necessary, 97-112, 153-157, 162-174 sufficient, 94-97, 151-153, 162 necessary saddlepoint, 76-82 saddlepoint, 69-82 sufficient saddlepoint, 74-76 Partial derivatives: of a numerical function, 200-201 of a vector function, 201-202 Plane, 40 Point of closure, 182 Polyhedron, 41 Poly tope, 41 Positive definite matrix, 91, 181 Positive semidefinite matrix, 89, 90, 181 Primal minimization problem (MP), 114, 122 Primal problem, no minimum, 121,126, 127 Product of a convex set with a real number, 45 Pseudoconcave function, 141 Pseudoconvex and strictly quasiconvex functions, relation between, 143, 146 Pseudoconvex function, 140-141 Quadratic dual problem (QDP), 124 Quadratic minimization problem (QMP), 123 Quadratic programming, 123-126 Quasiconcave function, 132 differentiable, 134 strictly, 137 Quasiconvex and strictly quasiconvex functions, relation between, 139, 146 119

Nonlinear Programming

Quasiconvex function, 131 differentiable, 134 strictly, 137 Rank of a matrix: column, 179 row, 179 Relatively closed set, 183 Relatively open set, 183 Rn, 6 Saddlepoint problem: Fritz John (FJSP), 71 Kuhn-Tucker (KTSP), 71 Scalar product, 7 Semicontinuous function: lower, 192-193 upper, 193 Separating plane, 46 Separation theorem, 49 strict, 50 Separation theorems, 46-53 Sequence, 185 Set, 3 complement of, 4 element of, 3 empty, 4 product with a real number, 45 subset of, 3 Set theoretic symbols, 5 Sets: difference of, 4 disjoint, 4

120

Sets: intersection of, 4 product of, 4-5 sum of, 45 union of, 4 Simplex, 42 Slater's theorem, 27 Stiemke's theorem, 32 Subspace, 40 Sum of two convex sets, 45 Supremum, 187, 195 of a numerical function, 196 Symbols, 5 Taylor's theorem, 204 Triangle inequality, 8, 41 Tucker's theorem, 29 Twice differentiable numerical function, 202 Uniqueness of minimum solution, 73 Vector, 6 addition, 6 multiplication by a scalar, 6 norm of, 7 Vector function, 12 Vector space, fundamental theorem of, 177 Vectors: angle between two, 8 scalar product of, 7 Vertex, 41