Skip to content

class Num::NN::AdamOptimizer(T)
inherits Num::NN::Optimizer #

Adam (short for Adaptive Moment Estimation) is an update to the RMSProp optimizer. In this optimization algorithm, running averages of both the gradients and the second moments of the gradients are used.

Constructors#

.new(learning_rate : Float64 = 0.001, beta1 : Float64 = 0.9, beta2 : Float64 = 0.999, epsilon : Float64 = 1e-8) #

Initializes an Adam optimizer, disconnected from a network. In order to link this optimizer to a Num::NN::Network, calling build_params will register each variable in the computational graph with this optimizer.

Arguments#
  • learning_rate : Float - Learning rate of the optimizer
  • beta1 : Float - The exponential decay rate for the 1st moment estimates
  • beta2 : Float - The exponential decay rate for the 2nd moment estimates
  • epsilon : Float - A small constant for numerical stability
View source

Methods#

#build_params(l : Array(Layer(T))) #

Adds variables from a Num::NN::Network to the optimizer, to be tracked and updated after each forward pass through a network.

Arguments#
View source

#update #

Updates all Num::Grad::Variables registered to the optimizer based on weights present in the network and the parameters of the optimizer. Resets all gradients to 0.

View source