介绍了一种基于低阶矩自适应估计的随机目标函数一阶梯度优化算法Adam。该方法易于实现，计算效率高，对内存的要求少，对梯度的对角重新缩放是不变的，并且非常适合于数据和/或参数很大的问题。该方法也适用于非平稳目标和具有非常嘈杂和/或稀疏梯度的问题。超参数有直观的解释，通常需要很少的调整。本文讨论了一些与相关算法的联系，Adam 正是在这些算法上受到启发。我们还分析了算法的理论收敛性，并给出了与在线凸优化框架下的最优结果相当的收敛遗憾界。实验结果表明，该方法在实际应用中效果良好，与其它随机优化方法相比具有一定的优越性。最后，我们讨论了AdaMax，一个基于无穷范数的Adam变体。

Stochastic gradient-based optimization is of core practical importance in many fields of science and engineering. Many problems in these fields can be cast as the optimization of some scalar parameterized objective function requiring maximization or minimization with respect to its parameters. If the function is differentiable w.r.t. its parameters, gradient descent is a relatively efficient optimization method, since the computation of first-order partial derivatives w.r.t. all the parameters is of the same computational complexity as just evaluating the function. Often, objective functions are stochastic. For example, many objective functions are composed of a sum of subfunctions evaluated at different subsamples of data; in this case optimization can be made more efficient by taking gradient steps w.r.t. individual subfunctions, i.e. stochastic gradient descent (SGD) or ascent. SGD proved itself as an efficient and effective optimization method that was central in many machine learning success stories, such as recent advances in deep learning (Deng et al., 2013; Krizhevsky et al., 2012; Hinton & Salakhutdinov, 2006; Hinton et al., 2012a; Graves et al., 2013). Objectives may also have other sources of noise than data subsampling, such as dropout (Hinton et al., 2012b) regularization. For all such noisy objectives, efficient stochastic optimization techniques are required. The focus of this paper is on the optimization of stochastic objectives with high-dimensional parameters spaces. In these cases, higher-order optimization methods are ill-suited, and discussion in this paper will be restricted to first-order methods.

基于随机梯度的优化方法在许多科学和工程领域具有核心的实际意义。这些领域中的许多问题都可以转化为对某个标量参数化目标函数的优化，该函数的参数需要最大化或最小化。如果函数的参数是可微的，梯度下降法是一种比较有效的优化方法，因为一阶偏导数的计算与函数的求值具有相同的计算复杂度。通常，目标函数是随机的。例如，许多目标函数由在不同数据子样本上求值的子函数和组成;在这种情况下，优化可以通过采取梯度步骤w.r.t.单独的子函数，即随机梯度下降(SGD)或上升来提高效率。SGD证明了自己是一种高效和有效的优化方法，这在许多机器学习成功的故事中都是核心，比如最近在深度学习方面的进展(Deng et al.， 2013;Krizhevsky等，2012;Hinton & Salakhutdinov, 2006;Hinton等，2012a;Graves 等人，2013)。目标也可能有数据子采样之外的其他噪音源，如dropout (Hinton et al.， 2012b)正则化。对于所有这些有噪声的目标，都需要有效的随机优化技术。本文主要研究具有高维参数空间的随机目标的优化问题。在这种情况下，高阶优化方法是不合适的，本文的讨论将局限于一阶方法。