Zeroth-order Gradient and Quasi-Newton Strategies for Nonsmooth Nonconvex StochasticOptimization
Authors: Luke Marrinan, Uday V. Shanbhag, Farzad Yousefian
Summary: We contemplate the minimization of a Lipschitz steady and expectation-valued perform outlined as f(x)≜E[f~(x,ξ)], over a closed and convex set. Our focus lies on acquiring each asymptotics in addition to price and complexity ensures for computing an approximate stationary level (in a Clarke sense) through zeroth-order schemes. We undertake a smoothing-based strategy reliant on minimizing fη the place fη(x)=Eu[f(x+ηu)], u is a random variable outlined on a unit sphere, and η>0. It has been noticed {that a} stationary level of the η-smoothed downside is a 2η-stationary level for the unique downside within the Clarke sense. In such a setting, we develop two units of schemes with promising empirical conduct. (I) We develop a smoothing-enabled variance-reduced zeroth-order gradient framework (VRG-ZO) and make two units of contributions for the sequence generated by the proposed zeroth-order gradient scheme. (a) The residual perform of the smoothed downside tends to zero virtually certainly alongside the generated sequence, permitting for making ensures for η-Clarke stationary options of the unique downside; (b) To compute an x that ensures that the anticipated norm of the residual of the η-smoothed downside is inside ε requires no higher than O(η−1ε−2) projection steps and O(η−2ε−4) perform evaluations. (II) Our second scheme is a zeroth-order stochastic quasi-Newton scheme (VRSQN-ZO) reliant on a mixture of randomized and Moreau smoothing; the corresponding iteration and pattern complexities for this scheme are O(η−5ε−2) and O(η−7ε−4), respectively △