Wangding Zeng | Chlience Paper Archive

Jun 24, 2026

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

DeepSeekMoE 把 MoE (Mixture of Experts，混合专家) 的效率问题改写成专家专门化问题：fine grained expert segmentation 通过切小 FFN (Feed Forward Network，前馈网络) 专家并增加激活专家数，提高每个 token 的专家组合分辨率；shared expert isolation 通过固定激活共享专家承载通用知识，让 routed experts ...

2401.06066-deepseekmoe-expert-specialization MoE Architecture

Jun 24, 2026

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Engram 把大模型稀疏性从 MoE (Mixture of Experts，混合专家，用条件计算扩大 total 参数并控制 active compute) 扩展到 conditional memory：用 hashed N gram lookup 存静态局部模式，用 context aware gate 决定是否把查到的记忆注入 hidden state；在 iso parameter / iso FLOPs 的 27B MoE...

2601.07372-conditional-memory-engram-scalable-lookup Memory Architecture MoE Architecture

Jun 23, 2026

DeepSeek V2: A Strong, Economical, and Efficient Mixture of Experts Language Model

DeepSeek V2 最独立的贡献是 MLA (Multi head Latent Attention，多头潜变量注意力)：它把 generation cache 从完整 per head K/V 改成由 hidden state 下投影得到的 512 维联合 KV latent 与 64 维 RoPE key，再用 projection absorption 让 latent 直接参与 content score 和 value ...

2405.04434-deepseek-v2-mla-moe-efficient-llm KV Cache MoE Architecture MoE Systems

Related Notes

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

DeepSeek V2: A Strong, Economical, and Efficient Mixture of Experts Language Model