tracked author

Qian Liu (刘乾)

TRM and OTB coauthor. Homepage identifies him as a researcher at TikTok AI Innovation Center and links GitHub/Twitter; GitHub profile currently shows `coding @xai-org`, so the archive keeps both TikTok and xAI as current-public-profile signals.

2 archived notes X: high Homepage GitHub X

Related Notes

按论文归档时间排序，展示该作者在本站已经出现的材料。

Jun 22, 2026

The Optimal Token Baseline: Variance Reduction for Long Horizon LLM RL

OTB 解决的是 long horizon LLM RL 中 outcome reward 经过整条 response 回传时的高方差问题：它从 causal policy gradient 的方差最小化目标推导出 token level optimal baseline $B t^ =\mathbb E[G tW t]/\mathbb E[W t]$，其中 $W t$ 是到第 $t$ 个 token 为止累计的 gradient r...

2602.07078-optimal-token-baseline-long-horizon-llm-rl Credit Assignment RL Algorithm RL Theory

Jun 21, 2026

Trust Region Masking for Long Horizon LLM Reinforcement Learning

这篇论文把 long horizon LLM RL 中 pi roll 与 pi theta 的实现级分布差异形式化为 surrogate objective error：经典 trust region bound 随长度呈 $O(T^2)$ 并迅速失效，作者给出更紧的 KL/TV 组合界，指出 max token divergence 是单靠 sequence average KL 无法替代的控制量，并提出 Trust Region...

2512.23075-trust-region-masking-long-horizon-llm-rl Training Stability RL Algorithm Rollout Optimization