How to Scale Your EMA
Paper
•
2307.13813
•
Published
•
9
•
4
Dan Busbridge
dbusbridge
AI & ML interests
Deep learning, optimization, self-supervised learning, representation learning, large language modeling, equivariance, geometric deep learning, attention mechanisms, transformers
Recent Activity
authored
a paper
9 days ago
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for
Mixture-of-Experts Language Models
authored
a paper
5 months ago
Theory, Analysis, and Best Practices for Sigmoid Self-Attention
commented on
a paper
over 1 year ago
How to Scale Your EMA
Organizations
dbusbridge's activity
commented
2
papers
over 1 year ago