FIELD NOTES TOPIC
memory efficiency
2 notes, newest first.
Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
The researchers present a study to make knowledge distillation training efficient for small language models deployed under tight constraints. They propose two methods: offline knowledge distillation (caching teacher's…
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
The authors propose a new programming paradigm called fuzzy-function programming, where functions can be compiled from natural language specifications into compact, locally executable neural networks. They introduce…
