tracked author

Samyam Rajbhandari

Recurring DeepSpeed author in the local archive: ZeRO coauthor and DeepSpeed Ulysses coauthor. Public sources identify him as DeepSpeed co-founder / system architect at Microsoft and later AI Systems Lead at Snowflake; X bio aligns with Snowflake and MSFT DeepSpeed.

2 archived notes X: high X

Related Notes

按论文归档时间排序，展示该作者在本站已经出现的材料。

Jun 21, 2026

DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

DeepSpeed Ulysses 的核心贡献是把长序列训练中的 activation / context 维瓶颈变成一次可逆的数据布局转换：Transformer 其他算子沿 sequence 维分片，进入 attention 前用 all to all 把 sequence partitioned, all heads 的 QKV 变成 full sequence, head partitioned 的 QKV，让每张 GPU 只...

2309.14509-deepspeed-ulysses-long-sequence-training Distributed Training Long Context Attention Kernel

Jun 18, 2026

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

ZeRO 的核心贡献是把数据并行中每张 GPU 都完整复制的 optimizer states、gradients、parameters 拆成可按 data parallel rank 分片保存和按需通信的动态状态系统，使 data parallelism 获得接近 model parallelism 的内存效率，同时保留 data parallelism 的大粒度计算和接近原始 DP 的通信量；论文实际评估了 $P {os+g}$ ...

1910.02054-zero-memory-optimizations-trillion-parameter-models Training Memory Distributed Training