Add reward breakdowns, level curves, W&B metric organization (AMC-76)#23
Merged
Merged
Conversation
…-76) Enhances MonitoringCallback with three capabilities that were missing from the observability stack: 1. Per-episode reward component breakdowns: accumulates reward_components on every env step and logs reward/exploration_mean, reward/badge_mean, etc. as W&B scalars so researchers can see which reward signals the agent exploits over training. 2. Game-state tracking: records player_level, pokemon_count, and money at episode boundaries and logs game/player_level_mean, _max, _latest plus pokemon and money curves to W&B. 3. W&B define_metric() calls: organizes panels under episode/*, game/*, and reward/* sections with global_step as the x-axis so the dashboard auto-creates useful panel groups. Also extends the episode table and dashboard JSON snapshot with the new fields, and adds pokemon_count and money to MONITORED_INFO_KEYS. 16 new tests (30 total), all 696 project tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
amcheste
added a commit
that referenced
this pull request
Apr 30, 2026
…#23) ## Summary - Enhances `MonitoringCallback` with three capabilities the observability stack was missing: - **Reward component breakdowns**: Accumulates `reward_components` on every env step (not just episode end) and logs `reward/exploration_mean`, `reward/badge_mean`, `reward/time_mean`, etc. as W&B scalars. Researchers can now see which reward signals the agent is exploiting over training. - **Game-state level/money/party curves**: Records `player_level`, `pokemon_count`, and `money` at episode boundaries. Logs `game/player_level_mean`, `game/player_level_max`, `game/player_level_latest`, plus pokemon count and money curves. - **W&B `define_metric()` organization**: Groups panels under `episode/*`, `game/*`, and `reward/*` with `global_step` as the x-axis so the W&B dashboard auto-creates structured panel groups. - Extends the episode table with 3 new columns: `player_level`, `pokemon_count`, `money` - Extends `dashboard_state.json` with `level_history`, `money_history`, `pokemon_count_history`, and `reward_component_summary` - Adds `pokemon_count` and `money` to `MONITORED_INFO_KEYS` - Multi-env safe: reward accumulators are keyed by env index and reset per-episode ## W&B Dashboard After This PR When you run training, the W&B dashboard will auto-organize into: - **episode/** — reward_mean, reward_max, length_mean, best_reward - **game/** — maps_visited, badges, event_flags, player_level, pokemon_count, money, screen captures - **reward/** — per-component breakdown (exploration, badge, time, death, event_flags, etc.) ## Test plan - [x] 16 new tests added (30 total in `test_monitoring_callback.py`) - [x] All 696 project tests pass - [x] Verified reward accumulation across steps, accumulator reset between episodes, multi-env isolation - [x] Verified graceful handling of missing fields and old wandb versions 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MonitoringCallbackwith three capabilities the observability stack was missing:reward_componentson every env step (not just episode end) and logsreward/exploration_mean,reward/badge_mean,reward/time_mean, etc. as W&B scalars. Researchers can now see which reward signals the agent is exploiting over training.player_level,pokemon_count, andmoneyat episode boundaries. Logsgame/player_level_mean,game/player_level_max,game/player_level_latest, plus pokemon count and money curves.define_metric()organization: Groups panels underepisode/*,game/*, andreward/*withglobal_stepas the x-axis so the W&B dashboard auto-creates structured panel groups.player_level,pokemon_count,moneydashboard_state.jsonwithlevel_history,money_history,pokemon_count_history, andreward_component_summarypokemon_countandmoneytoMONITORED_INFO_KEYSW&B Dashboard After This PR
When you run training, the W&B dashboard will auto-organize into:
Test plan
test_monitoring_callback.py)🤖 Generated with Claude Code