A second-order-like optimizer with adaptive gradient scaling for deep learning - Argumentation, Décision, Raisonnement, Incertitude et Apprentissage
Pré-Publication, Document De Travail Année : 2024

A second-order-like optimizer with adaptive gradient scaling for deep learning

Résumé

In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order information and rescaling while keeping the memory requirements of standard DL methods as AdamW or SGD with momentum. After having recalled our geometrical motivations, we provide quite extensive experiments. On image classification (CIFAR-10, ImageNet) and language modeling (GPT-2), INNAprop consistently matches or outperforms AdamW both in training speed and accuracy, with minimal hyperparameter tuning in large-scale settings. Our code is publicly available at \url{https://github.com/innaprop/innaprop}.
Fichier principal
Vignette du fichier
innaprop/innaprop_arxiv.pdf (971.87 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04724894 , version 1 (07-10-2024)

Identifiants

Citer

Jérôme Bolte, Ryan Boustany, Edouard Pauwels, Andrei Purica. A second-order-like optimizer with adaptive gradient scaling for deep learning. 2024. ⟨hal-04724894⟩
0 Consultations
0 Téléchargements

Altmetric

Partager

More