跳转至

LLM

文献

文献¶

主题页保留与具体主张相邻的来源，并在页末以 Reference 汇集本页书目；本页只提供贯穿全站的原始论文、官方实现与规范入口。技术谱系按问题演进组织阅读路径，并为关键工作提供论文、实现、公式与代码相互对照的深读页。书目格式见阅读方法。技术事实最后核验于 2026-07-28，持续演进的软件应同时记录版本或 commit。

基础、数据与缩放¶

对应基础、数据与缩放实验设计。

Elman, Finding Structure in Time
Hochreiter & Schmidhuber, Long Short-Term Memory
Bengio et al., A Neural Probabilistic Language Model
Sutskever et al., Sequence to Sequence Learning with Neural Networks
Bahdanau et al., Neural Machine Translation by Jointly Learning to Align and Translate
Sennrich et al., Neural Machine Translation of Rare Words with Subword Units
Vaswani et al., Attention Is All You Need
Radford et al., Improving Language Understanding by Generative Pre-Training
Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Kudo & Richardson, SentencePiece
Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Brown et al., Language Models are Few-Shot Learners
Kaplan et al., Scaling Laws for Neural Language Models
Hoffmann et al., Training Compute-Optimal Large Language Models
Muennighoff et al., Scaling Data-Constrained Language Models
Xie et al., DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
Liu et al., RegMix: Data Mixture as Regression for Language Model Pre-training
Lee et al., Deduplicating Training Data Makes Language Models Better
Penedo et al., The RefinedWeb Dataset for Falcon LLM
Soldaini et al., Dolma: An Open Corpus of Three Trillion Tokens
Li et al., DataComp-LM
Penedo et al., The FineWeb Datasets
Groeneveld et al., OLMo: Accelerating the Science of Language Models
Longpre et al., The Data Provenance Initiative

结构、位置与长序列¶

对应模型结构、位置编码、长上下文与非 Transformer 路线。

Zhang & Sennrich, Root Mean Square Layer Normalization
Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding
Press et al., Train Short, Test Long: Attention with Linear Biases
Peng et al., YaRN: Efficient Context Window Extension of Large Language Models
Shazeer, Fast Transformer Decoding: One Write-Head is All You Need
Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models
Dao et al., FlashAttention
Dao, FlashAttention-2
Shah et al., FlashAttention-3
Fedus et al., Switch Transformers
Dai et al., DeepSeekMoE
Wang et al., Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
Gu & Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Dao & Gu, Transformers are SSMs
Peng et al., RWKV: Reinventing RNNs for the Transformer Era
Sun et al., Retentive Network

DeepSeek 家族与 V4¶

DeepSeek 演化区分架构、训练、推理、权重与产品节点；DeepSeek-V4 总深读连接 mHC、CSA / HCA、Muon、MegaMoE、长上下文后训练和评测协议。公式与实现分别进入压缩注意力、mHC、On-Policy Distillation 和系统闭环；103 项引用图谱保留完整的一跳文献角色。

DeepSeek-AI et al., DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
DeepSeek-AI, DeepSeek-V4 official model collection
DeepSeek-AI, DeepSeek-V4-Pro model card and inference implementation
DeepSeek-AI, DeepSeek-V4-Flash model card and inference implementation
DeepSeek-AI et al., mHC: Manifold-Constrained Hyper-Connections
Thinking Machines Lab, On-Policy Distillation
DeepSeek-AI, DeepGEMM and MegaMoE

Kimi 家族与 K3¶

Kimi 技术谱系区分各节点的报告、权重、代码、API、许可证与发布日期；K3 工作深读连接结构、训练、系统和评测；150 项引用图谱进一步标明每项文献是直接来源、前身、并行工作、benchmark 还是比较对象。KDA、depth mixing、latent routing 与 expert parallel 分别见 Kimi Linear 与 FlashKDA、Attention Residuals、Stable LatentMoE 与 Quantile Balancing 和 MoonEP；视觉与音频分支另见 Kimi 家族的多模态路线。

Kimi Team, Kimi k1.5: Scaling Reinforcement Learning with LLMs
Kimi Team, Kimi-VL Technical Report
Kimi Team, Kimi K2: Open Agentic Intelligence
Kimi Team, Kimi Linear: An Expressive, Efficient Attention Architecture
Kimi Team, Kimi K2.5: Visual Agentic Intelligence
Kimi Team, Attention Residuals
Kimi Team, Kimi K3 Technical Report
Moonshot AI, Kimi K3 model card and weights
Moonshot AI, FlashKDA
Moonshot AI, MoonEP

GLM 家族与 GLM-5¶

GLM 演化区分论文、模型、权重、代码、API 与后续版本；GLM-5 总深读覆盖报告的架构、数据、训练、Agentic RL、环境、硬件和评测；63 项引用图谱保留正文实际使用文献的角色。架构、IndexCache、slime 与 Agentic Engineering 分别展开可复用机制。

Du et al., GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Zeng et al., GLM-130B: An Open Bilingual Pre-trained Model
GLM Team, ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
GLM Team, GLM-4.5: Agentic, Reasoning, and Coding Foundation Models
GLM Team, GLM-5: from Vibe Coding to Agentic Engineering
Z.ai, GLM-5 官方仓库
Z.ai, GLM-5.2 官方模型卡
THUDM, slime

多模态与生成¶

对应多模态与生成、共同表示、视觉理解、图像生成、音频、视频和 Any-to-Any 系统。

Dosovitskiy et al., An Image is Worth 16x16 Words
Radford et al., Learning Transferable Visual Models From Natural Language Supervision
Alayrac et al., Flamingo
Li et al., BLIP-2
Liu et al., Visual Instruction Tuning / LLaVA
Girdhar et al., ImageBind
Team Chameleon, Chameleon: Mixed-Modal Early-Fusion Foundation Models
Yue et al., MMMU
Ho et al., Denoising Diffusion Probabilistic Models
Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models
Lipman et al., Flow Matching for Generative Modeling
Peebles & Xie, Scalable Diffusion Models with Transformers
Kim et al., OCR-free Document Understanding Transformer / Donut
Lee et al., Pix2Struct
Cheng et al., SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision / Whisper
Baevski et al., wav2vec 2.0
Zeghidour et al., SoundStream
Défossez et al., High Fidelity Neural Audio Compression / EnCodec
Borsos et al., AudioLM
Wang et al., Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers / VALL-E
Tong et al., VideoMAE
Bardes et al., V-JEPA
Kondratyuk et al., VideoPoet

世界模型与具身智能¶

对应世界模型、潜在动力学与规划、表示预测与生成式世界、具身智能与 VLA 数据谱系。

Sutton, Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming
Ha & Schmidhuber, World Models
Hafner et al., Learning Latent Dynamics for Planning from Pixels / PlaNet
Hafner et al., Dream to Control: Learning Behaviors by Latent Imagination
Schrittwieser et al., Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model / MuZero
Assran et al., V-JEPA 2
Bruce et al., Genie: Generative Interactive Environments
Driess et al., PaLM-E
Brohan et al., RT-1
Zitkovich et al., RT-2
Open X-Embodiment Collaboration, Open X-Embodiment
Octo Model Team, Octo: An Open-Source Generalist Robot Policy
Kim et al., OpenVLA
Black et al., π0: A Vision-Language-Action Flow Model for General Robot Control

预训练、适配与压缩¶

对应预训练、参数高效适配、蒸馏与量化。

Loshchilov & Hutter, Decoupled Weight Decay Regularization
Yang et al., Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Wei et al., Finetuned Language Models Are Zero-Shot Learners / FLAN
Wang et al., Self-Instruct
Hu et al., LoRA
Dettmers et al., QLoRA
Liu et al., DoRA: Weight-Decomposed Low-Rank Adaptation
Hinton et al., Distilling the Knowledge in a Neural Network
Agarwal et al., Generalized Knowledge Distillation for Auto-Regressive Sequence Models
Liu et al., Muon is Scalable for LLM Training
Frantar et al., GPTQ
Lin et al., AWQ
Xiao et al., SmoothQuant

强化学习基础与策略优化¶

对应强化学习总览、历史与脉络、序贯决策、价值与 Bellman、策略优化和手撕强化学习。

Bellman, Dynamic Programming
Sutton and Barto, Reinforcement Learning: An Introduction, Second Edition
Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Watkins and Dayan, Q-learning
Sutton et al., Policy Gradient Methods for Reinforcement Learning with Function Approximation
Mnih et al., Human-level control through deep reinforcement learning
Schulman et al., Trust Region Policy Optimization
Schulman et al., High-Dimensional Continuous Control Using Generalized Advantage Estimation
Schulman et al., Proximal Policy Optimization Algorithms
Haarnoja et al., Soft Actor-Critic
Espeholt et al., IMPALA
Kumar et al., Conservative Q-Learning for Offline Reinforcement Learning
Ho and Ermon, Generative Adversarial Imitation Learning
Sutton, Precup, and Singh, Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction

语言模型反馈与策略学习¶

对应反馈制度、推理 RL 配方地图、GAE、PPO、GRPO、Ratio 与 Gate、在线 RL 与语言模型信用分配。

Ouyang et al., Training Language Models to Follow Instructions with Human Feedback
Bai et al., Constitutional AI
Schulman et al., Proximal Policy Optimization Algorithms
Rafailov et al., Direct Preference Optimization
Azar et al., A General Theoretical Paradigm to Understand Learning from Human Preferences / IPO
Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization
Meng et al., SimPO: Simple Preference Optimization with a Reference-Free Reward
Ahmadian et al., Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback
Espeholt et al., IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
Lightman et al., Let's Verify Step by Step
Shao et al., DeepSeekMath
DeepSeek-AI et al., DeepSeek-R1

研究前沿¶

下列工作适合研究复现与条件化比较，不应直接视为跨模型、跨环境的默认配方。

Yu et al., DAPO
Liu et al., Understanding R1-Zero-Like Training / Dr. GRPO
Cui et al., Process Reinforcement through Implicit Rewards / PRIME
Fu et al., AReaL: A Large-Scale Asynchronous Reinforcement Learning System
Yue et al., VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
MiniMax Team, MiniMax-M1 / CISPO
Zheng et al., Group Sequence Policy Optimization
Yao et al., On the Rollout-Training Mismatch in Modern RL Systems
Ma et al., Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers / R3
Ling Team, Every Step Evolves / IcePop
Gao et al., Soft Adaptive Policy Optimization
Qwen Team, Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
Hou et al., Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
Li et al., CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

推理、搜索与智能体¶

对应推理、搜索与验证、工具使用、Agent 运行时与安全边界。

Cobbe et al., Training Verifiers to Solve Math Word Problems
Wang et al., Self-Consistency Improves Chain of Thought Reasoning
Zelikman et al., STaR: Bootstrapping Reasoning With Reasoning
Yao et al., Tree of Thoughts
Lewis et al., Retrieval-Augmented Generation
Karpukhin et al., Dense Passage Retrieval
Khattab & Zaharia, ColBERT
Nakano et al., WebGPT
Yao et al., ReAct
Schick et al., Toolformer
Wang et al., Voyager

训练与推理系统¶

对应训练系统、并行策略、注意力 kernel 与推理服务。

Shoeybi et al., Megatron-LM
Rajbhandari et al., ZeRO
Huang et al., GPipe
Qi et al., Zero Bubble Pipeline Parallelism
Liu et al., Ring Attention
Jacobs et al., DeepSpeed Ulysses
Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention
Agrawal et al., Sarathi-Serve
Zhong et al., DistServe
Leviathan et al., Fast Inference from Transformers via Speculative Decoding
Cheng et al., DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
Chen et al., DFlash: Block Diffusion for Efficient Speculative Decoding
Li et al., EAGLE-3
Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs

官方实现与规范¶

评测、可靠性与安全¶

对应评测、统计推断、校准、生成式评审、污染与安全评测。

Liang et al., Holistic Evaluation of Language Models / HELM
Biderman et al., Lessons from the Trenches on Reproducible Evaluation of Language Models
Magnusson et al., PALOMA: A Benchmark for Evaluating Language Model Fit
Chen et al., Evaluating Large Language Models Trained on Code / HumanEval
Zhou et al., Instruction-Following Evaluation for Large Language Models / IFEval
Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Wang et al., Large Language Models are not Fair Evaluators
Min et al., FActScore
Wei et al., Long-Form Factuality in Large Language Models / SAFE
Kuhn et al., Semantic Uncertainty
Lin et al., TruthfulQA
Manakul et al., SelfCheckGPT
Mazeika et al., HarmBench
Souly et al., A StrongREJECT for Empty Jailbreaks
Chao et al., JailbreakBench
Röttger et al., XSTest
Wallace et al., The Instruction Hierarchy

Agent 与多模态评测¶

对应 Agent 与工具评测和多模态评测。

Jimenez et al., SWE-bench
Zhou et al., WebArena
Xie et al., OSWorld
Yao et al., tau-bench
Debenedetti et al., AgentDojo
Liu et al., AgentBench
Qin et al., ToolLLM / ToolBench
Yue et al., MMMU
Li et al., MMBench

引用论文结果时，应回到具体版本、表格、数据切分、预算和限制；本页是阅读索引，不替代原文。