Skip to content
BranchesTags

Repository files navigation

Awesome Credit Assignment in LLM RL Awesome

arXiv License: MIT PRs Welcome

Long-horizon LLM reasoning and agents often learn from sparse outcome rewards. Which token, step, turn, action, or agent deserves the credit?

47-method survey taxonomy | 28 newer entries | 16 benchmarks | Regularly updated

Start Here: 3-Minute Guide | Method Decision Guide | Machine-Readable Catalog | 2026.08 Update

A curated research map of credit assignment in reinforcement learning for large language models, organized by granularity and methodology across reasoning, agentic, and multi-agent settings.

This repository accompanies our survey paper:

From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models

Chenchen Zhang

[Paper]

The original survey snapshot covers 47 credit-assignment methods (41 core, 6 adjacent enablers) published between 2024 and early 2026. The living catalog adds 28 newer entries and related resources beyond that snapshot.

Contents

Overview

Taxonomy Grid

Taxonomy Tree

Credit assignment (CA) in LLM RL addresses a fundamental question: when the only feedback is a sparse terminal reward, which actions were responsible for the outcome? This problem manifests in two regimes:

  • Reasoning RL: Credit must be distributed across tokens and steps within a single chain-of-thought generation (500--30K+ tokens)
  • Agentic RL: Multi-turn environment interaction introduces stochastic transitions, partial observability, and horizons of 100+ turns (100K--1M tokens)

Our taxonomy organizes methods along two axes:

  • Granularity: Token / Segment / Step-Turn / Multi-Agent
  • Methodology: Monte Carlo / Temporal Difference / LLM-as-Critic / Game-theoretic / Information-theoretic / uncertainty-control / verifiable-feedback shaping

Recent agentic-RL papers increasingly move beyond generic trajectory-level rewards toward turn/action-aware optimization: execution feedback for coding agents, response-level entropy or uncertainty control, information-gain normalization across turns, and structured action credit for CLI-style environments.

Updates

  • [2026.08] Latest: Backfilled $\Delta$Belief-RL as a core turn-level method for intrinsic belief-change credit assignment.
  • [2026.07] Added the grouped paper refresh, method decision guide, machine-readable catalog, release notes, and contribution workflow.
  • [2026.05] Added recent agentic, coding-agent, uncertainty-control, multi-agent orchestration, and adjacent credit-assignment papers.
  • [2026.04] First version of the survey released on arXiv. Repository created.

Recent Additions

Recent additions are newly added to this repository, not necessarily newly published papers. We mark papers as core when the method directly changes how sparse outcome signal is assigned to tokens, turns, actions, or agents. Some core papers are backfilled; non-core entries are listed as near-core, related signals, or evaluation resources.

2026.08 Additions

Backfilled Core

Paper Year Type Granularity Why it is relevant
Intrinsic Credit Assignment for Long Horizon Interaction ($\Delta$Belief-RL) [Paper] 2026 Core Turn Uses changes in the policy's log-probability of the known target as dense per-turn rewards, then computes turn-wise group-relative advantages without a separate critic or PRM.

2026.07 Refresh and Gap-Fill

Backfilled Core

Paper Year Type Granularity Why it is relevant
GRPO-$\lambda$: Credit Assignment improves LLM Reasoning [Paper] 2025 Core Token/Sequence Extends GRPO with critic-free lambda-return and eligibility-trace style weighting for finer token-sequence credit.

Core Methods

Paper Year Type Granularity Why it is relevant
DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards [Paper] 2026 Core Token Reweights RLVR updates with discriminative token coefficients to amplify token-gradient directions associated with reward differences.
From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning (SCRL) [Paper] 2026 Core Subproblem/Span Converts reasoning chains into verifiable subproblems and assigns normalized advantages to corresponding answer spans.
TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning [Paper] 2026 Core Segment/Turn Adds semantic role labels to outcome credit and maps them to bounded segment-level process rewards for agentic RL.
Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning (OAR) [Paper] 2026 Core Token/Step Redistributes GRPO advantages using outcome-sensitivity signals from counterfactual perturbations or gradient proxies.
CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning [Paper] 2026 Core Token/Step Uses free sibling rollouts to estimate signed counterfactual token credit for self-distilled agentic RL.
Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards (SC-GRPO) [Paper] 2026 Core Token Uses verified self-conditioned trajectories to weight GRPO gradients with per-token KL-based credit.
GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewards [Paper] 2026 Core Token Applies gradient-activation saliency to reweight sequence-level advantages into intrinsic token-wise advantages.
Verifiable Process Rewards for Agentic Reasoning (VPR) [Paper] 2026 Core Turn (verifiable feedback) Converts intermediate symbolic or algorithmic oracles into dense turn-level supervision for long-horizon agentic credit assignment.
Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry (PivoARL) [Paper] 2026 Core Turn/Action Identifies the pivotal erroneous turn via structured self-reflection and retries locally from that state, assigning cross-episode credit that rewards correct prefixes while isolating erroneous suffixes.

Near-Core / CA-Adjacent Optimization

Paper Year Type Granularity Why it is relevant
APPO: Agentic Procedural Policy Optimization [Paper] 2026 Near-Core Decision/Procedure Moves agent optimization toward fine-grained procedure decisions and procedure-level advantage scaling.
OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning [Paper] 2026 Near-Core Token/Skill Extracts episode- and step-level hindsight skills from on-policy trajectories to form token-level self-distillation advantages.
Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control (PAPO) [Paper] 2026 Near-Core Token/Entropy Uses signed token-level entropy polarity to adaptively reweight advantages across entropy-expanding and entropy-contracting updates.

Related Signals

Paper Year Type Granularity Why it is relevant
Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents [Paper] 2026 Related Signal Step/Turn Derives an implicit progress signal from the log-probability ratio between an RL-trained policy and its reference policy for step-level scoring and failure attribution.

Evaluation / Benchmark

Paper Year Type Granularity Why it is relevant
QVal: Cheaply Evaluating Dense Supervision Signals for Long-Horizon LLM Agents [Paper] 2026 Evaluation State/Action Provides a training-free benchmark for testing whether dense supervision scores align with reference-policy Q-values in long-horizon agent tasks.

Additional related threads kept out of the main method lists: STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability [Paper] for entropy-aware token advantage reweighting; Not only where, But when: Temporal Scheduling for RLVR [Paper] for scheduling credit-allocation criteria over training; and Self-Distilled Agentic Reinforcement Learning (SDAR) [Paper] as a self-distillation background line for agentic dense supervision.

Earlier 2026.05 Additions

Paper Year Type Granularity Why it is relevant
AT$^2$PO: Agentic Turn-based Policy Optimization via Tree Search [Paper] 2026 Core Turn Uses turn-level tree structure, entropy-guided expansion, and turn-wise credit assignment for multi-turn agentic RL.
GVPO: Group Verification-based Policy Optimization for Interactive Coding Agents [OpenReview] 2026 Core Step/Action Shapes advantages using both outcome-verifiable rewards and process-verifiable execution feedback for interactive coding agents.
AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning [Paper] 2026 Core Response/Turn Uses response-level entropy dynamics to modulate advantages without external process supervision.
T$^2$PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning [Paper] 2026 Core Token/Turn Controls exploration through token-level uncertainty interventions and turn-level resampling.
Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces [Paper] 2026 Related Trace/Agent/Team Frames reward and credit-bearing units from token to team in LLM-based multi-agent orchestration.
A$^2$TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping [Paper] 2026 Core Turn Refines information-gain credit with turn-group normalization, variance-rescaled accumulation, and adaptive turn-level clipping.
Learning CLI Agents with Structured Action Credit under Selective Observation [Paper] 2026 Core Turn/Action Introduces Action Advantage Assignment (A$^3$) for CLI agents using structured action sub-chain residuals and trajectory margins.
Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning [Paper] 2026 Adjacent Criterion/Step Provides structured partial-credit judge rewards; relevant as reward shaping and process-supervision infrastructure.
In-Context Credit Assignment via the Core [Paper] 2026 Related Contributor/Coalition Applies cooperative game theory to in-context credit assignment for AI-generated content; related to game-theoretic CA but not LLM-policy RL.

Foundational & Background

Surveys & Overviews

  1. A Survey of Temporal Credit Assignment in Deep Reinforcement Learning (2023)

    Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin

    [Paper] -- Comprehensive review of temporal CA in classical deep RL (56 pages), predates the LLM era.

  2. The Landscape of Agentic Reinforcement Learning for LLMs: A Survey (2025)

    Guibin Zhang, Luyang Zheng, Zhiwei Zhang, Guang Yu, Zongxin Wen, Kun Li

    [Paper] -- 100-page overview of agentic RL for LLMs (500+ papers), treats CA as one sub-topic.

  3. A Survey of Reinforcement Learning for Large Reasoning Models (2025)

    Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, et al.

    [Paper] -- Covers RL algorithms broadly for reasoning LLMs.

Classical RL Foundations

  1. Proximal Policy Optimization Algorithms (2017)

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov

    [Paper] -- PPO: the workhorse of RLHF with learned value functions for token-level baselines.

  2. High-Dimensional Continuous Control Using Generalized Advantage Estimation (ICLR 2016)

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel

    [Paper] -- GAE: interpolation between TD and MC advantage estimation.

  3. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023)

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

    [Paper] -- DPO: bypasses explicit reward modeling via preference pairs.

  4. Hindsight Credit Assignment (NeurIPS 2019)

    Anna Harutyunyan, Will Dabney, Thomas Mesnard, et al.

    [Paper] -- Reweights past actions based on observed outcomes.

  5. RUDDER: Return Decomposition for Delayed Rewards (NeurIPS 2019)

    Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, et al.

    [Paper] -- Decomposes episodic return into per-step contributions.

RL Algorithms for LLMs

  1. Training language models to follow instructions with human feedback (NeurIPS 2022)

    Long Ouyang, Jeffrey Wu, Xu Jiang, et al.

    [Paper] -- InstructGPT / RLHF: established the paradigm of PPO-based LLM alignment.

  2. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024)

    Zhihong Shao, Peiyi Wang, Qihao Zhu, et al.

    [Paper] -- Introduces GRPO: group relative policy optimization, episode-level credit.

  3. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025)

    DeepSeek-AI

    [Paper] -- Demonstrates GRPO with binary rewards can elicit sophisticated chain-of-thought reasoning.

  4. Toolformer: Language Models Can Teach Themselves to Use Tools (NeurIPS 2024)

    Timo Schick, Jane Dwivedi-Yu, et al.

    [Paper] -- Foundational work on tool-using LLMs.

  5. WebArena: A Realistic Web Environment for Building Autonomous Agents (ICLR 2024)

    Shuyan Zhou, Frank F. Xu, Hao Zhu, et al.

    [Paper] -- Realistic web navigation benchmark for LLM agents.

PRM Foundations

  1. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (ACL 2024)

    Peiyi Wang, Lei Li, Zhihong Shao, et al.

    [Paper] -- Pioneered automatic step-level labeling via MC sampling.

  2. Improve Mathematical Reasoning in Language Models by Automated Process Supervision (OmegaPRM, 2024)

    Liangchen Luo, Yinxiao Xu, Anirudh Sahoo, et al.

    [Paper] -- Scaled PRM via divide-and-conquer exploration.

Credit Assignment in Reasoning RL

Token-Level Methods

  1. VinePPO: Refining Credit Assignment in RL Training of LLMs (ICML 2025) MC Token

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, Nicolas Le Roux

    [Paper] -- Replaces learned value network with unbiased MC value estimates at each token position via "vine" rollouts.

  2. RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution (2024) Redistribution Token

    Jiahui Li, Lin Li, Tai-wei Chang, Kun Kuang, et al.

    [Paper] -- Probes reward model internals via linear regression for token-level credit signals.

  3. T-REG: Preference Optimization with Token-Level Reward Regularization (2024) Self-generated Token

    Wenxuan Zhou, Shujian Zhang, Lingxiao Zhao, Tao Meng

    [Paper] -- Contrastive self-prompting to identify discriminative tokens without external models.

  4. From r to Q*: Your Language Model is Secretly a Q-Function (2024) Implicit Token

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

    [Paper] -- Shows DPO implicitly learns token-level Q-values; theoretical foundation for implicit CA.

Segment-Level Methods

  1. SPO: Segment Policy Optimization (2025) MC Segment

    Yiran Guo, Lijie Xu, Jie Liu, Dan Ye, Shuang Qiu

    [Paper] -- Partitions reasoning chains at semantic cutpoints; MC advantages per segment.

  2. SCAR: Shapley Credit Assignment Rewards (2025) Game-theoretic Segment

    Meng Cao, Shuyuan Zhang, Xiao-Wen Chang, Doina Precup

    [Paper] -- Shapley value decomposition across reasoning segments; theoretically principled.

  3. TEMPO: Exploiting Tree Structure for Credit Assignment in RL Training of LLMs (2025) Tree-TD Token/Segment

    Hieu Tran, Zonghai Yao, Hong Yu

    [Paper] -- Branch-gated TD corrections on a tree of reasoning paths; critic-free.

Step-Level Methods

  1. PURE: Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning (ICML 2025) Min-form PRM Step

    Jie Cheng, Gang Xiong, Ruixi Qiao, Lijun Li, et al.

    [Paper] -- Min-form credit prevents reward hacking in step-level PRMs.

  2. SPRO: Self-Guided Process Reward Optimization (2025) Masked Advantage Step

    Wu Fei, Hao Kong, Shuxian Liang, Yang Lin, et al.

    [Paper] -- Leave-one-out masked step advantage; 3.4x efficiency over GRPO.

  3. CAPO: Credit Assignment Policy Optimization (2025) LLM-as-Critic Step

    Guofu Xie, Yunsheng Shi, Hongtao Tian, Ting Yao, Xiao Zhang

    [Paper] -- LLM as Generative PRM: self-critiques each reasoning step.

  4. ACPO: Pinpointing Crucial Steps: Attribution-based Credit Assignment for Verifiable Reinforcement Learning (2025) Attribution Step

    Junxi Yin, Haisen Luo, Zhenyu Li, Yihua Liu, Dan Liu, Zequn Li, Xiaohang Xu

    [Paper] -- Gradient-based attribution + difficulty-aware curriculum.

  5. HICRA: Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning (2025) Hierarchy Step

    Haozhe Wang, Qixin Xu, Che Liu, Junhong Wu, Fangzhen Lin, Wenhu Chen

    [Paper] -- Focuses credit on planning tokens over procedural tokens.

  6. PRL: Process Reward Learning (2026) Entropy-RL Step

    Jiarui Yao, Ruida Wang, Tong Zhang

    [Paper] -- Derives optimal step-level process rewards from entropy-regularized RL.

  7. InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning (2026) Intervention Step

    Matthew Y. R. Yang, Hao Bai, Ian Wu, Gene Yang, Amrith Setlur, Aviral Kumar

    [Paper] -- Model proposes counterfactual interventions to assess step importance.

  8. FinePO: Fine-Grained Process Reward via SketchVL (2026) Fine PRM Sub-step

    Muye Huang, Lingling Zhang, Yifei Li, Yaqiang Wu, Jun Liu

    [Paper] -- Sub-step granularity PRM for domain-specific (visual) reasoning.

Credit Assignment in Agentic RL

Turn-Level Process Reward Models

  1. AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress (2025) TD+GAE Step/Turn

    Zhiheng Xi, Chenyang Liao, Guanyu Li, Yajie Yang, et al.

    [Paper] -- TD+GAE for turn-level value estimation; 8x sample efficiency vs MC-based PRM.

  2. SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks (2025) Privileged Critic Turn

    Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, Xian Li

    [Paper] -- Asymmetric critic with privileged training-time information (Meta/FAIR).

  3. Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design (NeurIPS 2025) Hybrid Turn

    Quan Wei, Siliang Zeng, Chenliang Li, William Brown, et al.

    [Paper] -- Hybrid reward: automated verification for verifiable turns + LLM-judge for subjective turns.

  4. Turn-PPO: Turn-Level Optimized Policy Optimization for Multi-Turn LLM Agents (EACL 2026) Turn-level MDP Turn

    Junbo Li, Peng Zhou, Rui Meng, Meet P. Vadera, Lihong Li, Yang Li

    [Paper] -- Reformulates multi-turn RL as turn-level MDP; turn-level importance ratios.

  5. SORL: Stabilizing Off-Policy RL for Long-Horizon Agent Training (2025) Bias-corrected Turn

    Chenliang Li, Adel Elmahdy, Alex Boyd, et al.

    [Paper] -- Turn-level importance sampling with clipping-triggered normalization (SO-PPO/SO-GRPO).

  6. TARL: Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents (2025) LLM-Judge Turn

    Weiting Tan, Xinghua Qu, Ming Tu, et al.

    [Paper] -- LLM-as-judge for turn-level evaluation + mixed-task curriculum.

  7. ITPO: Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction (2026) Implicit Turn

    Haoyu Wang, Yuxin Chen, Liang Luo, et al.

    [Paper] -- Derives implicit turn-level rewards from model's own log-probability changes.

Hindsight & Counterfactual Methods

  1. HCAPO: Hindsight Credit Assignment for Long-Horizon LLM Agents (2026) Hindsight Turn

    Hui-Ze Tan, Xiao-Wen Yang, Hao Chen, Jie-Jing Shao, Yi Wen, Yuteng Shen, Weihong Luo, Xiku Du, Lan-Zhe Guo, Yu-Feng Li

    [Paper] -- Retrospective LLM critic with counterfactual continuation analysis.

  2. C3: Contextual Counterfactual Credit Assignment for Multi-Agent Reinforcement Learning in LLM Collaboration (2026) Counterfactual Turn

    Yanjun Chen, Yirong Sun, Hanlin Wang, Xinming Zhang, Xiaoyu Shen, Wenjie Li, Wei Zhang

    [Paper] -- Leave-one-out counterfactual credit; extends to multi-agent settings.

  3. CCPO: Counterfactual Credit Policy Optimization for Multi-Agent Collaboration (2026) Counterfactual Turn

    Zhongyi Li, Wan Tian, Yikun Ban, Jinju Chen, Huiming Zhang, Yang Liu, Fuzhen Zhuang

    [Paper] -- Structural causal model for trajectories; average treatment effect as credit.

  4. CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic (2025) Retrospective Critic Turn

    Yaocheng Zhang, Haohuan Huang, Zijun Song, Yuanheng Zhu, Qichao Zhang, Zijie Zhao, Dongbin Zhao

    [Paper] -- Retrospective asymmetric critic specialized for search agent turn-level credit.

Critic-Free Step-Level Methods

  1. GiGPO: Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025) MC (group) Step

    Lang Feng, Zhenghai Xue, Tingcong Liu, Bo An

    [Paper] -- Two-level advantage: outer trajectory group + inner anchor state group; critic-free.

  2. POAD: Reinforcing Language Agents via Policy Optimization with Action Decomposition (2024) Action Decomposition Token/Turn

    Muning Wen, Ziyu Wan, Weinan Zhang, Jun Wang, Ying Wen

    [Paper] -- Intra-action (token-level) + inter-action (turn-level) credit decomposition.

Hierarchical Methods

  1. ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL (ICML 2024) TD (hierarchical) Turn

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, Aviral Kumar

    [Paper] -- Pioneering hierarchical CA: off-policy turn-level critic + on-policy token-level actor.

  2. PilotRL: Global Planning-Guided Progressive Reinforcement Learning (2025) Progressive Step

    Keer Lu, Chong Chen, Xili Wang, Bin Cui, Yunhuai Liu, Wentao Zhang

    [Paper] -- Three-stage progressive CA: plan-level -> step-level -> token-level.

  3. CARL: Focusing Agentic Reinforcement Learning on Critical Actions (NeurIPS 2025) Entropy-based Step

    Leyang Shen, Yang Zhang, Chun Kai Ling, Xiaoyan Zhao, Tat-Seng Chua

    [Paper] -- Identifies critical bifurcation points via action entropy; 72% fewer gradient updates.

Information-Theoretic Methods

  1. IGPO: Information Gain-based Policy Optimization (2025) Info-theoretic Turn

    Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, et al.

    [Paper] -- Turn-level credit = information gain about task success probability.

Implicit & DPO-Based Methods

  1. iStar: Agentic Reinforcement Learning with Implicit Step Rewards (2025) Implicit DPO Step

    Xiaoqian Liu, Ke Wang, Yuchuan Wu, Fei Huang, et al.

    [Paper] -- Extracts implicit step-level rewards from trajectory-level DPO; multi-level advantage fusion.

  2. StepAgent: From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning (2024) Implicit+IRL Step

    Zhirui Deng, Zhicheng Dou, Yutao Zhu, Ji-Rong Wen, Ruibin Xiong, Mang Wang, Weipeng Chen

    [Paper] -- Inverse RL from expert demonstrations + novice-to-expert curriculum.

Infrastructure & Practical Methods

  1. Agent Lightning: Train ANY AI Agents with Reinforcement Learning (2025) Decoupled Architecture Step

    Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K. Qiu, Yuqing Yang

    [Paper] -- Microsoft Research. Decoupled training architecture for RL-based LLM agent training.

  2. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning (2025) Uncertainty Step

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, et al.

    [Paper] -- StarPO framework; identifies "echo trap" failure mode; open-source benchmark.

  3. SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution (2025) MLP estimator Step

    Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, Wenjie Li

    [Paper] -- Lightweight MLP progress estimator; minimal computational overhead.

  4. LaRe: Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning (AAAI 2025) LLM-Critic Step

    Yun Qu, Yuhang Jiang, Boyuan Wang, Yixiu Mao, Cheems Wang, Chang Liu, Xiangyang Ji

    [Paper] -- Natural language credit explanations converted to scalar rewards.

  5. SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models (2026) Skill-prototype Step

    Yuxuan Jiang, Francis Ferraro

    [Paper] -- Skill prototype library for semantic-level credit assignment.

  6. PRS: Enhancing Agentic RL with Progressive Reward Shaping and VSPO (2025) Progressive Step

    Jianghao Su, Xia Zeng, Luhui Liu, Chao Luo, Ye Chen, Zhuoran Zhuang

    [Paper] -- Progressive reward densification from format to accuracy.

  7. Adaptive Segment-Level Reward: Bridging the Gap Between Action and Reward Space in Alignment (2024) Segmentation Segment

    Yanshi Li, Shaopan Xiong, Gengru Chen, Xiaoyang Li, et al.

    [Paper] -- Semantic segmentation for consistent reward granularity across trajectory lengths.

Multi-Agent Credit Assignment

  1. M-GRPO: Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO (2025) Hierarchical Multi-Agent

    Haoyang Hong, Jiajun Yin, Yuan Wang, Jingnan Liu, et al.

    [Paper] -- Two-level credit: inter-agent (team compositions) + intra-agent (GRPO); decoupled training.

  2. LLM-MCA: Leveraging Large Language Models for Effective and Explainable Multi-Agent Credit Assignment (2025) LLM-Critic Multi-Agent

    Kartik Nagpal, Dayi Dong, Jean-Baptiste Bouvier, Negar Mehr

    [Paper] -- LLM centralized critic reads full interaction history; semantic credit reasoning.

  3. QLLM: Do We Really Need a Mixing Network for Credit Assignment in Multi-Agent Reinforcement Learning? (2025) LLM-generated Multi-Agent

    Yuanjun Li, Zhouyang Jiang, Bin Zhang, Mingchao Zhang, Junhao Zhao, Zhiwei Xu

    [Paper] -- LLM generates a Python credit assignment function; training-free.

  4. SHARP: Shapley Credit-based Optimization for Multi-Agent System (2026) Shapley Multi-Agent

    Yanming Li, Xuelin Zhang, WenJie Lu, Ziye Tang, Maodong Wu, Haotian Luo, Tongtong Wu, Zijie Peng, Hongze Mi, Yibo Feng, Naiqiang Tan, Chao Huang, Hong Chen, Li Shen

    [Paper] -- Three-component Shapley decomposition; +23.7% over single-agent baselines.

  5. MAPPA: Scaling Multiagent Systems with Process Rewards (2026) Per-action PRM Multi-Agent

    Ed Li, Junyu Ren, Cat Yan

    [Paper] -- Per-action process rewards from AI feedback; +5.0--17.5pp on AIME.

  6. Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (2026) Agent-wise Advantage Multi-Agent

    Lang Feng, Longtao Zheng, Shuo He, Fuxiang Zhang, Bo An

    [Paper] -- Agent-wise advantage normalization to stabilize multi-agent GRPO; +5.6% avg@16.

Benchmarks

Reasoning RL Benchmarks

Benchmark Domain Scale Verifiable
GSM8K Grade-school math 8.5K test Yes
MATH Competition math 5K, 5 levels Yes
MATH-500 MATH subset 500 Yes
AIME'24/'25 Competition math 30 problems Yes
AMC Competition math ~25 problems Yes
CodeContests Competitive programming Varies Yes

Agentic RL Benchmarks

Benchmark Domain Typical Turns Typical Tokens
ALFWorld Embodied/simulated 5--20 5K--30K
WebShop Tool-use / shopping 5--15 5K--20K
WebArena Web navigation 10--30 30K--100K
SWE-bench Software engineering 20--100+ 100K--500K+
AppWorld Interactive coding / tool use 5--50+ 20K--200K+
ShellOps CLI repository tasks 5--50+ 20K--200K+
TextCraft Tool-use crafting 5--20 5K--30K
HotpotQA Multi-hop QA 5--15 5K--20K
Search QA Search-based tool use 5--30 10K--100K
ColBench Collaborative coding 10--30 30K--100K

Citation

If you find this repository useful, please cite our survey:

@article{zhang2026reasoning,
  title={From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models},
  author={Zhang, Chenchen},
  journal={arXiv preprint arXiv:2604.09459},
  year={2026}
}

Contributing

We welcome carefully reviewed paper suggestions, metadata corrections, and taxonomy improvements. Please read the contribution guidelines before opening an issue or pull request.

For any questions or suggestions, feel free to contact: zcc1959339538@gmail.com

About

Curated papers, taxonomy, benchmarks, and decision guides for credit assignment in reasoning and agentic LLM reinforcement learning.

arxiv.org/abs/2604.09459

Topics

agentic-rlawesomeawesome-listcredit-assignmentlarge-language-modelsllmmulti-agent-systemsprocess-reward-modelreinforcement-learningrlvr

Resources

Readme
MIT license

Contributing

Contributing
Activity

Stars

153 stars

Watchers

1 watching

Forks

3 forks
Report repository

Releases

Packages

Contributors

Languages