orybkin
Abstract
Scaling data and compute is critical to the success of machine learning. However, scaling demands predictability: we want methods to not only perform well with more compute or data, but also have their performance be predictable from small-scale runs, without running the large-scale experiment. In this paper, we show that value-based off-policy RL methods are predictable despite community lore regarding their pathological behavior. First, we show that data and compute requirements to attain a given performance level lie on a Pareto frontier, controlled by the updates-to-data (UTD) ratio. By estimating this frontier, we can predict this data requirement when given more compute, and this compute requirement when given more data. Second, we determine the optimal allocation of a total resource budget across data and compute for a given performance and use it to determine hyperparameters that maximize performance for a given budget. Third, this scaling behavior is enabled by first estimating predictable relationships between hyperparameters, which is used to manage effects of overfitting and plasticity loss unique to RL. We validate our approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI gym, and IsaacGym, when extrapolating to higher levels of data, compute, budget, or performance.
Community
We establish that value-based online RL can be scaled predictably to larger data, larger compute, or generally larger budget
thanks, very interesting.
at my opinion RL will be more progressing to more predefined world models, like more physical laws for real world. And for RL the bigger the model is, the more complex it's to small-scale
Combining this with pretrained models would definitely be very interesting! One big question there is how much pretraining vs finetuning helps you, so how to allocate compute across both.
Combining this with pretrained models would definitely be very interesting! One big question there is how much pretraining vs finetuning helps you, so how to allocate compute across both.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Towards General-Purpose Model-Free Reinforcement Learning (2025)
- Sample complexity of data-driven tuning of model hyperparameters in neural networks with structured parameter-dependent dual function (2025)
- Flow Q-Learning (2025)
- Multilinear Tensor Low-Rank Approximation for Policy-Gradient Methods in Reinforcement Learning (2025)
- Are Expressive Models Truly Necessary for Offline RL? (2024)
- Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data (2025)
- Adaptive Data Exploitation in Deep Reinforcement Learning (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper