Fix TensorboardCallback logging for off-policy algorithms (DDPG/TD3/SAC)

[jenish-25]

Summary

Fixes #1395.

TensorboardCallback._on_rollout_end assumed the model always exposes a rollout_buffer, which only on-policy algorithms (A2C, PPO) have. Off-policy algorithms (DDPG, TD3, SAC) expose a replay_buffer instead, so the callback raised a KeyError on every rollout, fell into the except branch, logged None for train/reward_min|mean|max, and printed Logging Error: ... noise. Reward curves were effectively broken for off-policy training.

Fix

Read whichever buffer the model actually exposes and compute the reward statistics over the valid (filled) region:

buffer = getattr(self.model, "rollout_buffer", None) or getattr(self.model, "replay_buffer", None)
if buffer is None:
    return True
rewards = np.asarray(buffer.rewards)
if not getattr(buffer, "full", False):          # replay buffer isn't full early on
    rewards = rewards[: getattr(buffer, "pos", 0)]  # ignore the zero-initialized tail
rewards = rewards.flatten()
if rewards.size == 0:
    return True
self.logger.record(key="train/reward_min", value=float(rewards.min()))
self.logger.record(key="train/reward_mean", value=float(rewards.mean()))
self.logger.record(key="train/reward_max", value=float(rewards.max()))

This reads the buffer from self.model (stable across SB3 versions) rather than self.locals, so reward logging now works for both algorithm families. The import statistics it replaced is removed.

Tests

Added unit_tests/test_tensorboard_callback.py covering:

• on-policy rollout_buffer → logs min/mean/max
• off-policy replay_buffer → logs min/mean/max, no None leaks (the DDPG / off-policy algorithms fail due to rollout_buffer logging in FinRL 0.3.8 #1395 regression)
• partially-filled replay buffer → stats ignore the unfilled tail
• model with neither buffer → no-op, no exception

All four pass; black is clean on the changed files.

Note

This is a clean re-submission of the previously-closed #1404 (a maintainer asked for a fresh PR in that thread). The implementation reads the buffer off the model and adds the missing tests.

cc @bruceyang @spencerromo @xiaoyang