A Survey of Reflective Reinforcement Learning

Mehrdad Zakershahrak, Samira Ghodratnama, and Amin Beheshti

Macquarie University

Read the paper | Explore the evidence records | Use the specification | Contribute

Reinforcement learning can optimize actions, but when does it genuinely learn through an explicit reasoning process? This survey introduces reflective reinforcement learning (R2L) as a shared framework for systems in which an identifiable reasoning object - such as a symbolic plan, causal hypothesis, logical specification, natural-language derivation, or tool-use trajectory - is functionally coupled to policy learning or action selection.

The paper's operational boundary and two complementary R2L lineages.

The core claim is deliberately narrow: readable output alone is not reflective RL. The reasoning object must affect the RL update or the actions selected by the learned system.

Three paper highlights

1. Reasoning must be inside the learning loop

R2L is not defined by whether a trace looks interpretable. An explicit reasoning object must shape the policy update or the action selected by the learned system. The unified loop makes that information flow testable.

2. R2L systems have three independent design choices

The survey classifies each system by representation x learning signal x integration point. This separates what carries the reasoning from what provides credit and where the object changes the learned system. Lifecycle and training-versus-inference phase remain additional reporting tags rather than being collapsed into the taxonomy.

3. Final accuracy is not enough

An overall score can hide failures in evidence quality, grounding, intermediate reasoning, verification, or execution. The paper therefore proposes a six-stage evaluation stack and recommends reporting local diagnostics alongside the final outcome.

What the full survey provides

• an operational definition that separates R2L from ordinary model-based, hierarchical, or post-hoc explanation methods;
• a unified formalism for environment actions, internal reasoning actions, reasoning objects, learning signals, verifiers, and stopping;
• a taxonomy organized by representation, learning signal, and integration point;
• a critical synthesis across control, mathematical reasoning, programmatic tasks, tool use, and multiagent systems;
• an evaluation stack for grounding, credit assignment, verifier quality, policy improvement, transfer, cost, and safety; and
• reporting requirements and a research agenda for auditable and robust R2L.

Start with the paper PDF. Use this repository when you want to inspect how quantitative claims are represented, validate a record, or contribute a correction.

Companion evidence record

This repository contains version 1.0.0 of the R2L Evidence Record Specification, the machine-readable companion to the survey. It captures the provenance, configuration, resources, controls, measurements, uncertainty, limitations, and permitted claim scope needed to interpret one R2L evidence unit.

The specification is an auditability tool, not an automatic scientific reviewer. Passing validation means that required fields and internal references are consistent. It does not establish scientific validity, comparability, or synthesis eligibility.

Release v1.0.0 provides:

• a versioned JSON Schema for one R2L claim or measurement;
• a dependency-free structural and semantic validator;
• regression tests and continuous integration;
• two visibly fictional examples spanning the survey's two R2L lineages;
• draft source-reported records linked to claims discussed in the manuscript; and
• contributor guidance for corrections and future records.

The versioned v1.0.0 release contains no empirical dataset. Records currently under records/ are draft source-reported extractions. They remain synthesis-pending until two named authors independently verify each extraction. The fictional examples are always excluded from synthesis.

Quick start

Python 3.10 or newer is required. No third-party packages are needed.

git clone https://github.com/mehrdadxzaker/r2l-evidence-record.git
cd r2l-evidence-record

python3 validate_record.py \
  examples/schematic_language_record.json \
  examples/schematic_control_record.json

python3 -m unittest discover -s tests -v

Validate a directory of candidate records:

python3 validate_record.py path/to/records/

Use --strict to treat completeness warnings as failures:

python3 validate_record.py --strict path/to/records/

The validator exits with status 0 when no errors are found and status 1 when a structural or semantic error is present. Messages use JSON Pointer paths so the failing field can be located directly.

The two schematic examples pass ordinary validation but intentionally produce completeness warnings because no experiment was run and no resource quantity or comparator parity is claimed. Their strict validation is therefore expected to fail. Strict mode is intended for candidate evidence records.

Repository map

Path	Purpose
paper/	Current manuscript PDF
assets/figures/	README-ready figures extracted from the manuscript
schema/r2l_evidence_record.schema.json	JSON Schema Draft 2020-12 record definition
validate_record.py	Dependency-free structural and semantic validator
records/	Draft source-reported records and admission policy
examples/	Fictional examples for the two R2L lineages
docs/recording-guide.md	Provenance and review workflow
docs/versioning.md	Schema compatibility and release policy
tests/	Validator regression tests

Record unit

One source-reported record represents exactly one source, one evaluated system configuration, and one task or benchmark protocol under a fixed resource and comparator configuration. Create another record when any of the following changes:

• model checkpoint or policy version;
• task split, environment version, or evaluation protocol;
• inference, sampling, interaction, or compute budget;
• comparator configuration; or
• tool access, prior knowledge, demonstrations, or supplied task structure.

An author-computed record identifies its input record IDs, formula, normalization, and uncertainty procedure. A theoretical record identifies its assumptions and theorem or derivation location. A qualitative synthesis record identifies its supporting record IDs and synthesis rationale. Fields that do not apply must be marked explicitly as not applicable rather than filled with invented values.

Evidence and review status

The schema preserves five evidence classes:

• reported_by_source requires a citation key and exact source locators;
• computed_by_authors requires input record IDs, a formula, normalization, and an uncertainty procedure;
• derived_or_theoretical requires assumptions and a theorem or derivation locator;
• qualitative_synthesis requires supporting record IDs and an explicit synthesis rationale; and
• schematic_example must be marked illustrative and excluded from synthesis.

Three statuses remain independent:

• record_status describes the editorial state;
• evidence_class describes how the claim was obtained; and
• synthesis_eligibility.status records whether the claim may enter a declared synthesis after scientific and compatibility review.

A verified record requires a documented verification check. Synthesis eligibility requires a verified record and checks by at least two distinct reviewers. Validation enforces those lifecycle relationships but never promotes a record between states.

Contributing

Corrections, schema improvements, and carefully sourced records are welcome through GitHub issues and pull requests. Read CONTRIBUTING.md and the recording guide first. Never infer missing experimental values; use explicit reporting states and explain the omission.

Citation

If the survey is useful in your work, cite the manuscript:

@misc{zakershahrak2026reflective,
  title   = {A Survey of Reflective Reinforcement Learning},
  author  = {Zakershahrak, Mehrdad and Ghodratnama, Samira and Beheshti, Amin},
  year    = {2026},
  note    = {Manuscript}
}

If you use the schema or validator, cite release v1.0.0 using CITATION.cff.

The release is archived at https://github.com/mehrdadxzaker/r2l-evidence-record/releases/tag/v1.0.0.

License

The evidence-record specification and validator are released under the MIT License.