Deep Neural Networks (DNNs) are considered one of the crucial efficient instruments for locating patterns in massive datasets by coaching. On the core of the coaching issues, we have now advanced loss landscapes and the coaching of a DNN boils all the way down to optimizing the loss because the variety of iterations will increase. A couple of of essentially the most generally used optimizers are Stochastic Gradient Descent, RMSProp (Root Imply Sq. Propagation), Adam (Adaptive Second Estimation) and so forth.
Lately (September 2024), researchers from Apple (and EPFL) proposed a brand new optimizer, AdEMAMix¹, which they present to work higher and sooner than AdamW optimizer for language modeling and picture classification duties.
On this publish, I’ll go into element concerning the mathematical ideas behind this optimizer and focus on some very fascinating outcomes offered on this paper. Subjects that can be coated on this publish are:
- Overview of Adam Optimizer
- Exponential Shifting Common (EMA) in Adam.
- The Important Concept Behind AdEMAMix: Combination of two EMAs.
- The Exponential Decay Charge Scheduler in AdEMAMix.