How the weights are updated. optimizer_adam() unless there is a reason not to: it adapts the step per parameter, so it usually trains without anything tuned except learning_rate. optimizer_sgd() with momentum is the recipe most of the classification literature’s best numbers were produced with, in exchange for depending on a tuned rate and usually a schedule - see callbacks for those.
optimizer_adam(
learning_rate = 0.001,
beta_1 = 0.9,
beta_2 = 0.999,
eps = 1e-08,
weight_decay = 0,
amsgrad = FALSE
)
optimizer_adamw(
learning_rate = 0.001,
beta_1 = 0.9,
beta_2 = 0.999,
eps = 1e-08,
weight_decay = 0.01
)
optimizer_sgd(
learning_rate = 0.001,
momentum = 0,
nesterov = FALSE,
weight_decay = 0
)
optimizer_rmsprop(
learning_rate = 0.001,
alpha = 0.99,
momentum = 0,
eps = 1e-08,
weight_decay = 0
)
optimizer_adagrad(
learning_rate = 0.001,
lr_decay = 0,
eps = 1e-10,
weight_decay = 0
)
optimizer_adadelta(
learning_rate = 0.001,
rho = 0.9,
eps = 1e-06,
weight_decay = 0
)
optimizer_adamax(
learning_rate = 0.001,
beta_1 = 0.9,
beta_2 = 0.999,
eps = 1e-08,
weight_decay = 0
)
optimizer_nadam(
learning_rate = 0.001,
beta_1 = 0.9,
beta_2 = 0.999,
eps = 1e-08,
weight_decay = 0
)
learning_rate
|
How big a step to take. The one setting worth trying first when a run will not learn: too high and the loss moves without improving, too low and it improves too slowly to tell from not improving at all. Each parameter group decays from its own rate, so an encoder_learning_rate is not flattened by a schedule.
|
beta_1
|
Decay for the running average of the gradient. Lower reacts to the last few batches, higher smooths over more of them; 0.9 is the usual value and rarely worth moving. |
beta_2
|
Decay for the running average of the squared gradient, which is what scales each parameter’s step. Below about 0.99 training gets noticeably noisier. In optimizer_adamax() it decays the largest gradient seen instead.
|
eps
|
Added to the denominator so a parameter whose gradient has been zero for a while does not take an enormous step. A guard, not a knob. |
weight_decay
|
Pull the weights towards zero at every step, which is L2 regularisation. Off by default because it interacts with the loss and is not free: on a small dataset it can cost more than the overfitting it prevents. In optimizer_adamw() it is applied to the weights directly rather than through the gradient, which is the whole difference between the two.
|
amsgrad
|
Keep the largest squared-gradient average seen so far rather than the current one, so the step size never grows back. It fixes a convergence case in Adam’s proof and usually changes nothing in practice; worth trying if a run plateaus and then gets worse. |
momentum
|
Carry a fraction of the previous step into this one. 0 is plain gradient descent; 0.9 is the usual choice and is what makes SGD competitive with the adaptive methods. |
nesterov
|
Look ahead to where momentum is about to carry the weights before measuring the gradient. Needs momentum above 0, and giving it without momentum is refused rather than ignored.
|
alpha
|
For optimizer_rmsprop(), the decay for its running average of the squared gradient - the same work beta_2 does in the Adam family, under that family’s name.
|
lr_decay
|
For optimizer_adagrad(), shrink the rate further as training goes on, on top of the shrinking Adagrad already does by accumulating every squared gradient it has seen.
|
rho
|
For optimizer_adadelta(), how long a window of squared gradients to remember. Adadelta exists to avoid Adagrad’s ever-shrinking step and this is the parameter that does it.
|
No run in this project has compared them on segmentation, so this page says what each one does differently rather than which is better. available_optimizers() lists them with the arguments each accepts.
The rest are here because a configuration should be able to name what torch offers. optimizer_adamw() is Adam with weight decay applied to the weights rather than through the gradient, which is why its decay defaults to 0.01 rather than 0 - choosing it and leaving the decay at zero leaves nothing to choose. optimizer_adagrad() accumulates every squared gradient it has seen, so its step only ever shrinks and a long run can stop learning; optimizer_adadelta() is the same idea with a window instead of a sum. optimizer_adamax() tracks the largest gradient rather than the average of their squares, which is steadier where a few batches carry very large ones. optimizer_rmsprop() is Adam’s denominator without its momentum, and optimizer_nadam() is Adam with the look-ahead that SGD calls nesterov.
Every argument left unset is decided by the engine, which is where the defaults live.
An optimiser specification.
available_optimizers() for the same list with its arguments, and callbacks() for moving the rate during a run.
library("rplatypus")
optimizer_adam(learning_rate = 1e-4)$name
[1] "adam"
$learning_rate
[1] 1e-04
$beta_1
[1] 0.9
$beta_2
[1] 0.999
$eps
[1] 1e-08
$weight_decay
[1] 0
$amsgrad
[1] FALSEoptimizer_sgd(learning_rate = 0.01, momentum = 0.9, nesterov = TRUE)$name
[1] "sgd"
$learning_rate
[1] 0.01
$momentum
[1] 0.9
$nesterov
[1] TRUE
$weight_decay
[1] 0