Abstract
An implementable algorithm for solving nonsmooth nonconvex constrained optimization is proposed by combining bundle ideas, proximity control, and the exact penalty function. We construct two kinds of approximations to nonconvex objective function; these two approximations correspond to the convex and concave behaviors of the objective function at the current point, which captures precisely the characteristic of the objective function. The penalty coefficients are increased only a finite number of times under the conditions of Slater constraint qualification and the boundedness of the constrained set, which limit the unnecessary penalty growth. The given algorithm converges to an approximate stationary point of the exact penalty function for constrained nonconvex optimization with weakly semismooth objective function. We also provide the results of some preliminary numerical testing to show the validity and efficiency of the proposed method.
1. Introduction and Motivation
Nonsmooth optimization problems (NSO) arise from many fields of applications in engineering [1], economics [2], mechanics [3], and optimal control [4]. For example, multiobjective nonsmooth optimization has also been applied in many fields of engineering where optimal decisions need to be taken in the presence of trade-offs between two or more conflicting objectives [5]. There exist several approaches to solving NSO, see [6–10]. Bundle methods are currently among the most efficient optimization methods; they can be used to study the engineering problem of the safe evaluation technology for concrete dams by applying nonsmooth bundle ideas to hydrostructure antiseismic fields [11–14]. These methods are based on the cutting plane method [15, 16], where the convexity of the objective function is the fundamental assumption. If the objective function is convex, the model functions are lower approximations to the objective function. This feature is crucial to prove the convergence of most bundle methods. There exist lots of bundle methods [7, 17–19] for solving convex constrained optimization problems. In [7], the author presents a version of proximal bundle method for convex constrained optimization, where and exact penalty functions are employed and a new penalty update is used to limit unnecessary penalty growth; the global convergence of the method is established. In [17], an infeasible bundle method for convex constrained optimization is proposed, which does not use either a penalty function or a filter, and in fact the method can be viewed as an unconstrained proximal bundle method applied directly to the improvement function, and it should be noted that the serious steps need neither be monotone nor feasible. The algorithm presented in [18] inherits attractive features from the proximal bundle methods and the filter strategy, which makes the criterion for accepting a candidate point as a serious step easier to satisfy.
However, for nonconvex cases, the corresponding model function does not stay below the objective function and may even cut off a region containing a minimizer. There are few systematic studies for extending convex bundle methods to nonconvex cases. Most authors have considered forcing linearization errors to be positive by replacing negative values with a quadratic term or with the absolute value of the linearization errors; the piecewise affine models embedding possible downward shifting of the affine pieces are also considered [20, 21]. In [22], the author presents a substitute for the cutting plane without convexity assumption and proves that every accumulation point of the sequence of serious steps is critical. Based on cutting plane models, a local convexification model of the objective function is constructed in [23]; it opens a new way to create nonconvex algorithms; Fuduli et al. [24] partitioned the bundle information into two subsets to capture convex and concave behaviors of the objective function around the current point. Other literatures about bundle methods for nonsmooth nonconvex optimization can be found in [25–28].
In this paper, we propose a new algorithm for constrained nonconvex optimization. The algorithm is based on the construction of two kinds of approximations to the objective function, and these two kinds of approximations correspond to the convex and concave behaviors of the objective function at the current point. If the linearization error is positive, we build a local lower approximation to the objective function, otherwise a local upper approximation is constructed. Besides that, the method employs exact penalty functions with a new penalty update rule that limits unnecessary penalty growth. Our method extends the exact penalty function algorithms for constrained convex minimization to nonconvex optimization. The following is the main difference of our paper in many respects from the existing ones [6, 7, 19, 29]. The proposed algorithm in this paper is quite different from the ones in [7, 17–19] since the objective function in our paper is nonconvex although the constraint function is convex, while both the objective function and the constraint function are convex in [7, 17–19]. In [7, 19], the and exact penalty functions are employed to consider convex constraint optimization problems by combining with proximal bundle method. In this paper, we also use the similar exact penalty functions for solving nonconvex optimization problems, but we have to adjust suitably the construction of quadratic programming subproblems since the presence of nonconvexity can make the linearization errors negative, which enhances the difficulty to solve the problem. To solve this problem, we divide the bundle index set into two sets according to the signs of linearization errors and use the partitioned bundles during the process of the construction of the objective function model. Therefore, the direction finding subproblem is quite different from the existing ones, which leads to the overall changes for the design of the algorithm. The algorithms in [17, 18] use neither penalty functions nor relatively complex filters; they build on the theory of the well-developed unconstrained bundle methods by introducing the improvement function, which is essential for the convergence of the proposed algorithm. It is another approach proposed in recent years for solving nonsmooth convex constrained optimization problems. It should be noted that the descent condition in our method used to decide when the candidate point can be accepted as the next serious step is different from the ones in [17, 18], where the improvement function involving the objective function is employed to form the descent criterion, but in our method, the penalty function is used to serve the same role.
This paper is organized as follows: in Section 2, the model for constrained nonconvex optimization is established by using the exact penalty function. The nonconvex bundle algorithm is presented in Section 3. In Section 4, we prove that the sequence of serious steps generated by the proposed algorithm converges to an approximate stationary point of exact penalty function with weakly semismooth objective functions. Preliminary numerical experiments are provided in Section 5. Finally, some conclusions are given in Section 6.
2. Derivation of the Model
Consider the following constrained nonconvex optimization problem:where is a real-valued locally Lipschitz function and is a real-valued convex function. It is well known that for a locally Lipschitz function , the generalized subdifferential (Clarke’s subdifferential) at each point is defined bywhere “conv” denotes the convex hull of a set and is the set where is not differentiable. The set is locally bounded [30]. An extension of the generalized subdifferential is the Goldstein -subdifferential defined as
For convex function , the subdifferential of at is defined bywhich is locally bounded. Assume that we are able to compute at each point both the function values and subgradients . We denote the current iteration point (stability center) and the trial point by and , respectively. The bundles of available information are the sets and of elementswhere , , , , , and .
If and are real-valued convex functions on , under Slater constraint qualification, problem (1) can be solved by minimizing the following exact penalty functionwhere is the penalty parameter which is greater than the Lagrange multiplier of problem (1), see [7]. For real-valued locally Lipschitz function and real-valued convex function , we try to combine the ideas of bundle methods, proximity control, and exact penalty functions to solve problem (1). We define the cutting plane approximations of , , and by their linearizations:where , is the corresponding penalty parameter. The next trial point is obtained by solving the following problem:where is the proximal parameter. Note that if is feasible, . Therefore, we introduce additionally a trial point , the subgradient , and the linearization error with respect to , and then we define . Hence, problem (8) can be written in equivalent formwhere . The introduction of index set into can make sure in (9) for .
It should be noted that may be negative since is nonconvex; we divide into two sets and defined bywhere is nonempty since . We define the following two piecewise affine functions:
Let , the affine function , be considered as the approximation of since and . Because , if , therefore . Summing up, around (around the stability center ), it appears that the set is more important and reliable.
Let be the optimal solution to the following problem:
Let . Since is the optimal solution of problem (12), , similarly, . We define the predicted descent , it is not difficult to find that . We notice that (therefore, ) since is a feasible point of problem (12). Set
Let ; we obtainwhere and are matrices whose columns are the vectors and . and are the vectors with components and , respectively. Let ; we obtaini.e., , where . Let ; we obtain
Substituting (14)–(16) into , we havewhere and are vectors whose components are and , respectively. Then, the duality problem of is the following minimization problem:and the primal optimal solution is related to the dual optimal solution by the following formulae:
3. Algorithm
In this section, we present a nonconvex bundle algorithm for constrained nonconvex optimization problem (1). Our method is based on repeatedly solving problem (12).
A few comments on Algorithm 1 are in order.
|
The stopping criterion in Step 1 is used to assess the stationarity of current stability center. If it is satisfied, the approximate stationarity of exact penalty function is achieved, Algorithm 1 stops, and the approximate solution is obtained.
The solution of at Step 2 may be obtained by using the dual quadratic programming method of [29] or [31], which can efficiently solve sequences of subproblems with varying and .
The result is never a consequence of the choice of too big . In fact, we note that if , it holds that
The right-hand side of the above inequality is more than , so too big may not lead to , therefore we intend to decrease the value of in Step 2.
Large may force the iterate points generated by Algorithm 1 to approach closely to the boundary of the feasible set and may damage the fast convergence of Algorithm 1. Our rule increases (from ) only to ensure a significant predicted decrease in constraint violation of the form , where [0,1) is a contraction factor, see Step 3.
Finally, note that the insertion of a bundle index into or at Step 6 is not simply based on the sign of , see [15].
4. Convergence Results
The presented work in this section follows a line of investigation initiated in [6, 7], where nonconvex bundle algorithm is used to solve unconstrained minimization problem and the idea of exact penalty function is employed in proximal bundle method for constrained convex minimization problem. Here, we expand and generalize the central idea [6] to constrained nonconvex minimization problems; some techniques have to be adjusted to the new situations for the presence of constraints and nonconvexity.
Throughout the section, we make the following assumptions: (A1) and are weakly semismooth ( is said to be weakly semismooth if the directional derivative exists for all and , and where ). (A2) The set is compact, where is the initial point provided by the user in Algorithm 1. (A3) The feasible set is bounded. (A4) The Slater constraint qualification holds, i.e., there exists such that .
The assumption that the feasible set of problem (1) is bounded is usual and reasonable; it was also assumed in [7, 32–34]. In [27], the boundedness of the feasible set was assumed in order to guarantee the existence of the supremum of the range of a set-valued mapping on the feasible set. In [32], the authors assumed the feasible sets were bounded closed convex for finding the saddle point of the objective function.
Lemma 1. Let be the sequence generated within an inner iteration such that andwith Algorithm 1 looping between Step 2 and Step 8. Then, the following conclusions hold:(i)There is an index such that for each , every new bundle index with respect to is inserted into and remains unchanged.(ii)Step 6(c) is appropriate, feasible, and not difficult to realize.(iii)Whenever a new bundle index is inserted into , the condition holds, where are the subgradients of and at , respectively.
Proof. (i)Since increases at Step 6(a) of Algorithm 1, the situation that infinite bundle indices are inserted into can not happen. Hence, once exceeds , no bundle index with respect to can be inserted into .(ii)According to Assumption (A1), is weakly semismooth, the directional derivative exists for any . It follows from the mean value theorem that for some , where Since the sufficient decrease condition Algorithm 1 is not satisfied, we have there exists a scalar such that By weak semismoothness of , it is not difficult to find a scalar such that satisfies the condition , where , .(iii)By construction of Algorithm 1, we have . If (the next Lemma 4 shows that can not be increased for infinitely many times, therefore, is possible), we also have the condition also holds.The next lemma shows the finite termination of the inner iteration.
Lemma 2. The inner iteration terminates after a finite number of steps.
Proof. It is enough to demonstrate that, in a finite number of steps, either the condition of the stop at Step 1 or the exit at Step 4 is satisfied. Firstly, we prove Algorithm 1 cannot pass through Step 4 infinitely many times. Assume that such a case occurs, since at each iteration, the algorithm enters Step 4, then we have and . Observe that and will exceed the threshold in a finite number of steps. It follows that , therefore we obtain , which means that the indices of the new bundle elements are inserted into and are never removed.
According to Step 6, we insert an index into only if , which implies that whenever entering Step 4, all the elements in are removed. Taking into account (15), (16), (19), and (20), there is an index such that for all ,where . But since and , we havewhich leads to a contradiction.
Next, we show that it is impossible to have for infinitely many times and the descent condition Algorithm 1 is not satisfied with the algorithm looping between Step 4 and Step 8. Indexing by , the jth passage through such a loop, we observe that, by Lemma 1(i), there exists an index such that for every , the index of each new bundle element is put into with remaining unchanged. Therefore, for , the sequence is nondecreasing and bounded and hence convergent. Since is bounded, suppose is its convergent subsequence. The sequence also converges to a nonpositive limit . Now assume that . Let and be two successive indices in and let with and , ; we have and . If ; we haveif , it holds thatCombing (28) and (29), we obtain , hence by taking the limits , it contradicts , hence . It follows from that , which contradicts the fact that .
The next lemma shows that the penalty coefficients are increased finitely many times under the conditions of Slater constraint qualification and the boundedness of .
Lemma 3. There exists such thatif.
Proof. Denote the Lagrangian of byThe Lagrange multipliers satisfy the usual saddle-point condition:for all . For the above inequality, we take , where is the one in Assumption (A4), and then by using (15) and (16), we obtainwhere . Remove the same terms from both sides of (33) and note that , , and the subgradient inequality of convex function ; we haveSince , , and Assumption (A4) tells us that , the following inequality holds:Recalling that is locally bounded, let and . The existence of can be guaranteed by the aggregate technique of bundle methods. Then,where is the upper bound of . Now, if we take , the conclusion can be obtained.
Lemma 4. There exist an indexandsuch thatfor all.
Proof. According to the result of Lemma 3, there exists one such that does not hold for . Therefore, Step 3 in Algorithm 1 will not be executed once . The penalty coefficient remains constant .
Note that Lemmas 3 and 4 ensure the number of iterations between Step 2 and Step 3 is finite, and the penalty coefficients stay unchanged after finitely many iterations.
Now, we are ready to prove the overall finiteness of Algorithm 1.
Theorem 5. Suppose Assumptions (A1)–(A4) hold, then for any and , Algorithm 1 stops at a point satisfying the approximate stationarity conditionwith .
Proof. For contradiction, assume that the approximate stationarity condition (37) cannot be satisfied for an infinite number of iterations. In other words, the termination condition in Step 4 is not satisfied for each iteration. Therefore, Algorithm 1 is executed for infinitely many times. It follows from Lemma 2 that the descent condition Algorithm 1 is satisfied for each iteration. Let be the stability center at the jth iteration through inner iteration, then and , henceSince and , it follows that, if is big enough,where is the constant appeared in Lemma 3 and and are the locally Lipschitz constants of and on . Therefore, is bounded away from zero. It follows from that . Hence, is bounded away from zero as well. Therefore, by passing to the limit, we obtainNote that is bounded from below as a consequence of the semismoothness of and the compactness of the level set of . By combining the fact and (40), we obtain , and is unbounded, which leads to a contradiction. Hence, Algorithm 1 cannot be executed for infinitely many times; it stops at a point satisfying condition (37).
5. Numerical Experiments
To assess practical performance of the presented method, we coded Algorithm 1 in MATLAB and ran it on a PC with 1.80 GHz CPU.
5.1. Examples for Nonconvex Optimization Problems
In this subsection, we first introduce the nonconvex test problems. We prefer a series of polynomial functions developed in [35], also see [23, 36]. For each , the function is defined by
There are four classes of test functions defined by in [36] as objective functions. It has been proved in [23, 36] that they are nonconvex, globally lower , level coercive, and bounded on compact . We use one of these test functions to verify the validity and efficiency of the proposed method.
Example 1. Consider problem (1):For objective function, we define the nonconvex functionFor constraint function, we consider the pointwise maximum of a finite collection of quadratic functions:
For problem (1) with (43) and (44), we can obtain that and . The results of numerical experiment are as follows: The initial point: ; Optimal solution: ; The final objective function value: ; The final constraint function value: ; The CPU time: 0.25 seconds.
Example 2. Consider problem (1):For objective function, we define the nonconvex functionFor constraint function, we consider the pointwise maximum of a finite collection of quadratic functions:
For problem (1) with (46) and (47), we can obtain that and . The results of the numerical experiment are as follows: The initial point: ; Optimal solution: ; The final objective function value: ; The final constraint function value: ; The CPU time: 15.47 seconds.
The above two examples show that the proposed Algorithm 1 does perform not badly since the optimal solutions and computed, respectively, by Algorithm 1 are not far away from the true optimal solution to problem (1).
6. Conclusion
For constrained nonconvex optimization problem, we propose an implementable algorithm by combining bundle ideas, proximal control, and exact penalty functions. The results extend the ideas of cutting plane and proximity control to the constrained nonconvex case. We present some techniques for choosing penalty coefficients which ensures the limitation of penalty growth. The penalty parameters are increased only a finite number of times which prevents the algorithm from following closely the curvature boundary of the constrained set. For weakly semismooth functions, the convergence of the presented algorithm to an approximate stationary point of the exact penalty function is proved without any additional assumptions except for the conditions of Slater constraint qualification and the boundedness of the constrained set.
Data Availability
No data were used to support this study.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this article.
Acknowledgments
This work was supported by the National Nature Science Foundation of China (grant numbers 61877032 and 11601061) and Basic Scientific Research Project of Educational Department of Liaoning Province (grant number JYTMS20231042).