Skip to content

feature(sandbox): Fix heartbeat timeout recovery and session-bound sandbox lifecycle#413

Merged
AgentScopeTeam merged 16 commits intoagentscope-ai:mainfrom
rayrayraykk:dev/0122/rec_sandbox
Jan 23, 2026
Merged

feature(sandbox): Fix heartbeat timeout recovery and session-bound sandbox lifecycle#413
AgentScopeTeam merged 16 commits intoagentscope-ai:mainfrom
rayrayraykk:dev/0122/rec_sandbox

Conversation

@rayrayraykk
Copy link
Copy Markdown
Member

@rayrayraykk rayrayraykk commented Jan 22, 2026

Description

[Describe what this PR does and why]

This PR improves sandbox heartbeat handling and recovery after inactivity.

  • Add session_ctx_id to sandbox creation (sync/async) for better session tracking.
  • Normalize environment variables to always use string key/value pairs.
  • Extend ContainerModel with lifecycle fields (state, last_active_at, recycled_at, etc.) and keep backward compatibility with meta["session_ctx_id"].
  • Update HeartbeatMixin to read/write heartbeat and recycle status on container models, and only call restore_session when implemented.
  • Add an integration test to verify: heartbeat timeout → reap → restore → run_shell_command works again.
  • Update CI workflow to include the new heartbeat restore test.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Refactoring

Component(s) Affected

  • Engine
  • Sandbox
  • Tools
  • Common
  • Documentation
  • Tests
  • CI/CD

Checklist

  • Pre-commit hooks pass
  • Tests pass locally
  • Documentation updated (if needed)
  • Ready for review

@gemini-code-assist
Copy link
Copy Markdown

Summary of Changes

Hello @rayrayraykk, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the robustness and reliability of sandbox lifecycle management by introducing a more detailed container state model and refining the heartbeat mechanism. The changes enable better tracking of session-bound sandboxes, automatic recovery from inactivity, and consistent handling of environment variables, ultimately leading to a more stable and resilient sandbox environment.

Highlights

  • Enhanced Sandbox Lifecycle Management: The ContainerModel has been extended with new fields such as state, last_active_at, recycled_at, released_at, updated_at, and recycle_reason to provide a more granular and robust tracking of sandbox container lifecycles. This centralizes lifecycle state within the container model itself.
  • Improved Heartbeat Handling and Recovery: The HeartbeatMixin has been refactored to store heartbeat and recycle status directly on the ContainerModel instances, moving away from separate heartbeat_mapping and recycled_mapping. This change enables a more consistent and integrated approach to managing sandbox activity and inactivity. A new restore_session method has been implemented in SandboxManager to handle the automatic restoration of recycled sandboxes upon user re-engagement.
  • Session-Bound Sandbox Creation: Sandboxes now incorporate a session_ctx_id during creation (both sync and async), allowing for better session tracking and binding of containers to specific user sessions. This session_ctx_id is now a top-level field in ContainerModel for easier access and logic, with backward compatibility for older payloads.
  • Environment Variable Normalization: Environment variables passed during sandbox creation are now consistently normalized to string key/value pairs, ensuring predictable behavior across different environments.
  • Comprehensive Integration Test: A new integration test has been added to validate the complete flow of heartbeat timeout, session reaping, automatic restoration, and successful command execution in the restored sandbox, ensuring the reliability of the new lifecycle management features.
Ignored Files
  • Ignored by pattern: .github/workflows/** (1)
    • .github/workflows/unit_test_sandbox.yml
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request significantly enhances the sandbox's heartbeat handling and recovery mechanisms. Key improvements include integrating session tracking directly into the ContainerModel, normalizing environment variables for robustness, and introducing a comprehensive restore_session logic. The changes are well-structured, reflecting a clear intent to improve the lifecycle management of sandbox containers. The accompanying test updates, including a new integration test, provide good coverage for the new functionality.

Comment thread src/agentscope_runtime/sandbox/manager/heartbeat_mixin.py
Comment thread src/agentscope_runtime/sandbox/manager/heartbeat_mixin.py
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enhances sandbox heartbeat handling and recovery from inactivity timeouts by introducing session-bound container lifecycle management.

Changes:

  • Introduces ContainerState enum and extends ContainerModel with lifecycle fields (state, last_active_at, recycled_at, etc.) for tracking container status
  • Refactors heartbeat and recycle tracking from separate mappings to container-level fields within ContainerModel
  • Implements restore_session() method to recreate recycled containers when users return after timeout
  • Adds session_ctx_id binding to sandbox creation in sync/async contexts
  • Normalizes environment variables in boxlite_client to string key/value pairs
  • Adds integration test verifying the full heartbeat timeout → reap → restore → usage cycle

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 18 comments.

Show a summary per file
File Description
tests/sandbox/test_heartbeat_timeout_restore.py New integration test verifying heartbeat timeout recovery flow
tests/sandbox/test_heartbeat.py Updated tests using stub clients for heartbeat functionality
src/agentscope_runtime/sandbox/model/container.py Extended ContainerModel with state tracking and lifecycle timestamps
src/agentscope_runtime/sandbox/model/init.py Exported ContainerState enum
src/agentscope_runtime/sandbox/manager/sandbox_manager.py Implemented restore_session, updated cleanup/reap logic, modified create_from_pool binding
src/agentscope_runtime/sandbox/manager/heartbeat_mixin.py Refactored heartbeat/recycle tracking to use container-level fields
src/agentscope_runtime/sandbox/box/sandbox.py Added session_ctx_id generation and binding in sandbox creation
src/agentscope_runtime/common/container_clients/boxlite_client.py Normalized environment variables to string pairs
.github/workflows/unit_test_sandbox.yml Added new test to CI workflow
Comments suppressed due to low confidence (1)

src/agentscope_runtime/sandbox/manager/sandbox_manager.py:1482

  • The docstring for scan_heartbeat_once does not follow the required template. According to custom guideline 1000000, all methods must have complete docstrings with Args and Returns sections in the specified format.
    def scan_heartbeat_once(self) -> dict:
        """
        Scan all session_ctx_id in session_mapping and reap those idle
        beyond timeout. Uses redis distributed lock to avoid multi-instance
        double reap.

Comment thread src/agentscope_runtime/sandbox/manager/sandbox_manager.py
Comment thread src/agentscope_runtime/sandbox/manager/sandbox_manager.py
Comment thread src/agentscope_runtime/sandbox/manager/sandbox_manager.py
Comment thread tests/sandbox/test_heartbeat.py
Comment thread src/agentscope_runtime/sandbox/model/container.py Outdated
Comment thread src/agentscope_runtime/sandbox/manager/heartbeat_mixin.py
Comment thread src/agentscope_runtime/common/container_clients/boxlite_client.py
Comment thread src/agentscope_runtime/sandbox/model/container.py Outdated
Comment thread src/agentscope_runtime/sandbox/manager/sandbox_manager.py Outdated
Comment thread src/agentscope_runtime/sandbox/manager/sandbox_manager.py Outdated
@rayrayraykk
Copy link
Copy Markdown
Member Author

Link to #289

Copilot AI review requested due to automatic review settings January 22, 2026 12:37
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Comment thread src/agentscope_runtime/sandbox/manager/heartbeat_mixin.py Outdated
Comment thread src/agentscope_runtime/sandbox/manager/heartbeat_mixin.py
Comment thread src/agentscope_runtime/sandbox/manager/heartbeat_mixin.py
Comment thread src/agentscope_runtime/sandbox/manager/heartbeat_mixin.py
Copilot AI review requested due to automatic review settings January 23, 2026 04:27
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

src/agentscope_runtime/sandbox/manager/sandbox_manager.py:1499

  • Docstring does not follow the required format. According to the coding guidelines [MUST], all methods must have complete docstrings with Args and Returns sections following the specified template. The method should document:

Args:
(No arguments)

Returns:
dict:
Dictionary containing scan metrics with keys: "scanned_sessions", "reaped_sessions", "skipped_no_heartbeat", "skipped_no_running_containers", "skipped_lock_busy", "skipped_not_idle_after_double_check", "errors"

    def scan_heartbeat_once(self) -> dict:
        """
        Scan all session_ctx_id in session_mapping and reap those idle
        beyond timeout. Uses redis distributed lock to avoid multi-instance
        double reap.
        """

Comment thread src/agentscope_runtime/sandbox/manager/sandbox_manager.py Outdated
Comment thread src/agentscope_runtime/sandbox/model/container.py Outdated
Comment thread src/agentscope_runtime/sandbox/manager/sandbox_manager.py
Comment thread src/agentscope_runtime/engine/services/sandbox/sandbox_service.py
Comment thread src/agentscope_runtime/sandbox/manager/sandbox_manager.py
Comment thread tests/sandbox/test_heartbeat_timeout_restore.py
@AgentScopeTeam AgentScopeTeam merged commit c9ec719 into agentscope-ai:main Jan 23, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants