AdamW
also: AdamAdam with decoupled weight decay: per-parameter step sizes from running averages of gradients (β₁) and squared gradients (β₂), both fixed constants.
Reference ↗Related
2 articles
The fractal optimizer that was plain gradient descent
A blueprint reported a fractal-orbit optimizer reaching loss 0.0176 on two-spirals. We ran its code. The fractal term never switched on: in 36,000 steps the measured dimension never crossed the threshold that activates it.
The Mandelbrot question we have not answered
If the loss landscape is self-similar across scales, a fixed-window optimiser is the wrong instrument by construction. A question, written down before the work, with what it would take to become a claim.