论文阅读分析，记录论文的研究问题、核心方法、实验依据、局限和可以回填到主题笔记的稳定知识。
Notes
- Deep Residual Learning for Image Recognition
- Sequence-Level Knowledge Distillation
- Neural Machine Translation of Rare Words with Subword Units
- Training Deep Nets with Sublinear Memory Cost
- Layer Normalization
- Deep Reinforcement Learning from Human Preferences
- Proximal Policy Optimization Algorithms
- Attention Is All You Need
- Outrageously Large Neural Networks
- GPipe
- SentencePiece
- Megatron-LM
- Fast Transformer Decoding
- Root Mean Square Layer Normalization
- T5
- ZeRO
- Big Bird
- GShard
- GLU Variants Improve Transformer
- Language Models are Few-Shot Learners
- Longformer
- Learning to summarize from human feedback
- Scaling Laws for Neural Language Models
- ALiBi
- Deduplicating Training Data Makes Language Models Better
- Documenting Large Webtext Corpora
- The Pile
- LoRA
- RoFormer
- Switch Transformer
- ZeRO-Infinity
- Finetuned Language Models Are Zero-Shot Learners
- Multitask Prompted Training Enables Zero-Shot Task Generalization
- Constitutional AI
- Training language models to follow instructions with human feedback
- Self-Instruct
- Training Compute-Optimal Large Language Models
- A Pretrainer’s Guide to Training Data
- Distilling Step-by-Step
- Direct Preference Optimization
- Grouped-Query Attention
- RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
- LongLoRA
- Position Interpolation
- QLoRA
- RefinedWeb
- ROOTS
- YaRN
- DataComp-LM
- DeepSeekMath
- DeepSeek-V2
- DeepSeek-V3
- DeepSeekMoE
- Dolma
- FineWeb
- LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
- LongRoPE
- Repoformer: Selective Retrieval for Repository-Level Code Completion
- CodeRAG: Finding Relevant and Necessary Knowledge for Retrieval-Augmented Repository-Level Code Completion
- LongCodeZip: Compress Long Context for Code Language Models
- SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents
- Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents
- CodePromptZip: Code-specific Prompt Compression for Retrieval-Augmented Generation in Coding Tasks with LMs
- RLP: Reinforcement as a Pretraining Objective