How to Make Knowledge Distillation Affordable

Knowledge distillation - it's a technique in machine learning that's all about training a smaller student model to mimic the performance of a larger teacher model. And with the recent surge in open-source large language models, it's become a hot topic again. The thing is, deploying these massive models can be ridiculously expensive, so compressing them into smaller ones through knowledge distillation has become the norm. But here's the catch - the distillation step, which pretty much decides the final quality, is also usually the most expensive part of the whole process.

The standard setup for knowledge distillation, which involves online distillation using the Kullback-Leibler divergence loss, is a real resource hog - it needs a ton of memory and compute power. You see, it requires loading both the teacher and student models at the same time, and generating a probability distribution over the entire vocabulary for every single token. This can be crazy expensive, requiring hundreds of GPUs and some serious tensor-parallelism strategies. To get around this, researchers have suggested a couple of system changes: caching the teacher's top-K logits once, and using a new, memory-efficient KL-divergence loss that avoids having to materialize that huge vocabulary-size × sequence-length matrix.

By making these changes, developers can seriously cut down on the cost of knowledge distillation. The cached top-K logits approach means you don't have to recompute the teacher at every step, and the fused chunked KL loss reduces memory requirements by processing the data in chunks. This makes it possible to train models on just a single GPU, which slashes the cost and makes large-scale experimentation actually feasible. As a result, developers can now explore new applications and models that were previously out of reach due to the high cost of knowledge distillation.

Source: Hugging Face