基本信息
- Title: Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
- Source type: paper
- Related topic notes: Knowledge Distillation, Sequence-level Distillation, Offline KD
TODO
- 阅读论文原文，整理 rationale / reasoning trace 如何辅助 student 学习。
- 回填 distillation 与 chain-of-thought supervision 的关系。
- 补充小模型、少数据和 rationale 质量之间的实验结论。