Knowledge of training best practices can reduce the cost of training a desired model significantly. Here, we link to readings and resources on effectively using a given resource budget for model training, including canonical papers on fitting scaling laws.
Provide scaling laws to determine the optimal allocation of a fixed compute budget.
A calculator to apply compute-optimal scaling laws for a given budget, including factoring expected total inference usage.
Demonstrates an optimal allocation of compute when dataset size is bounded
Proposes an optimal allocation of computational budget between model and dataset size, and shows experimental design for fitting scaling laws for compute allocation in a new setting.
