👉 Capacity math, also known as capacity scaling, is a technique used in deep learning to train very large neural networks by gradually increasing the model's capacity (the ability to fit complex functions) during training. Initially, the model has a small number of parameters (capacity) and is trained with a high learning rate to ensure it can learn quickly. As training progresses, the learning rate is reduced, and the model's capacity is increased through techniques like adding more layers or neurons. This allows the network to learn more complex representations without becoming unstable due to the large number of parameters. The key idea is to start with a model that can fit simple functions and then incrementally increase its capacity, effectively scaling up the learning rate to match the model's growing complexity. This method helps in training deep networks that would otherwise be too large to train efficiently with standard techniques.
