[Prompt registry] Phase 2 — Port static-tier sections behind the snap…#3659
Conversation
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Coverage Report •
|
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
⚠️ QA Report: PASS WITH ISSUES
The new static prompt registry works as an additive SDK API and matches legacy section content after documented gap normalization, but literal byte-for-byte parity with Agent.static_system_message is not true.
Does this PR achieve its stated goal?
Partially. I exercised the SDK as a user would by importing build_default_registry(), constructing real Agent/LLM objects, building prompt contexts, and comparing the registry output to the existing static prompt path across default, analyzer, sandbox-tier, browser-enabled, and Windows-simulated scenarios. Those scenarios matched after the PR's documented inter-section gap normalization and produced no dynamic block, so the static-tier port is functionally usable behind the snapshot. However, exact equality was False for the real default prompt (15187 vs 15191 bytes), so the PR does not literally satisfy the repeated byte-for-byte parity claim.
| Phase | Result |
|---|---|
| Environment Setup | ✅ make build completed successfully. |
| CI Status | 🟡 21 checks passing, 8 pending, 2 skipped at review time; no checks were rerun. |
| Functional Verification |
Functional Verification
Test 1: Establish base behavior without the PR registry
Step 1 — Reproduce / establish baseline (without the fix):
Ran git worktree add --detach /tmp/qa-pr3659-base origin/main and then executed the SDK from that base worktree:
base_has_default_registry= False
base_static_len= 15191
base_starts= You are OpenHands agent, a helpful AI assistant that can interact with a computer to solve tasks.
base_has_role= True
base_has_browser= False
base_has_security_policy= True
This shows the base branch has the legacy Agent.static_system_message path but no openhands.sdk.context.prompts.default_registry entry point.
Step 2 — Apply the PR's changes:
Returned to PR head 3fd47bf95719a408dfc5132369a20d811c655da1 on vasco/static-part after bootstrapping with make build.
Step 3 — Re-run with the PR in place:
Executed real SDK agents and the new registry API:
CASE=default_gpt5
registry_static_len= 15187
legacy_static_len= 15191
exact_equal= False
canonical_equal= True
dynamic_is_none= True
has_security= True
has_security_risk= True
has_important= True
CASE=anthropic_with_analyzer
registry_static_len= 14489
legacy_static_len= 14493
exact_equal= False
canonical_equal= True
dynamic_is_none= True
has_security= True
has_security_risk= True
has_important= True
CASE=sandbox_risk_tiers
registry_static_len= 15139
legacy_static_len= 15143
exact_equal= False
canonical_equal= True
dynamic_is_none= True
has_security= True
has_security_risk= True
has_important= True
This shows the new API is usable and the static bodies match after gap normalization, but exact byte-for-byte parity with the live prompt path is not achieved.
Test 2: Browser, Windows, and edge-path behavior
Step 1 — Reproduce / establish baseline (without the fix):
On base, the new registry import was absent, so these sectionized paths could not be exercised there.
Step 2 — Apply the PR's changes:
Used the PR branch's new registry and section classes directly with SDK PromptContext/Agent objects.
Step 3 — Re-run with the PR in place:
Ran browser-enabled, Windows-simulated, and disabled-security-policy cases:
CASE=browser_enabled_agent
ctx_enable_browser= True
registry_has_browser= True
legacy_has_browser= True
canonical_equal= True
CASE=browser_guard_direct_context
disabled= False
enabled= True
CASE=windows_agent
ctx_platform= windows
registry_contains_powershell= True
registry_contains_execute_powershell= False
canonical_equal= True
CASE=security_policy_disabled
registry_static_len= 13674
legacy_static_len= 13703
exact_equal= False
canonical_equal= False
dynamic_is_none= True
has_security= False
has_security_risk= True
has_important= True
This confirms browser and Windows paths behave as expected after normalization. The disabled security policy divergence matches the PR notes and is not runtime-wired yet, but it reinforces that the new registry is not a literal byte-for-byte replacement for all legacy static prompt outputs.
Exact mismatch evidence
Ran a compact diff summary on the default PR prompt:
exact_equal= False
registry_len= 15187 legacy_len= 15191 delta= 4
first_diff_index= 9476
registry_excerpt= 'ser has explicitly requested and would expect\n\n</SECURITY>\n\n<SECURITY_RISK_ASSESSMENT>\n# Security Risk Policy\nWhen using tools that support '
legacy_excerpt= 'ser has explicitly requested and would expect\n\n</SECURITY>\n\n\n<SECURITY_RISK_ASSESSMENT>\n# Security Risk Policy\nWhen using tools that support'
This pinpoints the observed difference as inter-section blank-line spacing.
Issues Found
- 🟡 Minor: The new registry is functionally usable and section bodies match after normalization, but
registry.build(ctx).staticis not literally byte-for-byte equal toAgent.static_system_messagedespite the docstring/PR wording implying that level of parity.
This review was created by an AI agent (OpenHands) on behalf of the user.
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
⚠️ QA Report: PASS WITH ISSUES
Verified the new prompt registry by importing it as SDK library code and building real Agent/LLM instances; nominal static-tier scenarios match legacy output, with one documented out-of-matrix divergence for disabled security policy.
Does this PR achieve its stated goal?
Yes, for the stated additive/static-tier goal. On the PR commit, build_default_registry().build(ctx).static matched canonicalized Agent.static_system_message for real GPT-5/browser/security, Anthropic sandbox/security, and simulated win32 shell-refinement scenarios, and the legacy Agent.static_system_message hashes stayed identical to main for the same Linux scenarios. The registry also remained all-static (dynamic_is_none=True). I found one edge divergence for security_policy_filename=""; the PR description already calls this out as outside the matrix/follow-up, so it is not blocking for the behind-snapshot phase but should be tracked before runtime cutover.
| Phase | Result |
|---|---|
| Environment Setup | ✅ make build completed and installed the uv-managed workspace packages. |
| CI Status | 🟡 At review time: 22 checks successful, 8 in progress, and one unresolved-review-threads failure present. |
| Functional Verification |
Functional Verification
Test 1: Baseline on main — registry is absent, legacy prompt still renders
Step 1 — Establish baseline without the PR:
Ran git checkout --detach origin/main && OPENHANDS_SUPPRESS_BANNER=1 uv run python /tmp/qa_prompt_registry_baseline.py:
DEFAULT_REGISTRY_IMPORT=unavailable (ModuleNotFoundError: No module named 'openhands.sdk.context.prompts.default_registry')
SCENARIO gpt5-cli-browser-security
static_chars=15908 static_sha256_16=eb14bbdf59f0f3c0
has_browser=True
has_security=True
has_security_risk=True
has_important=True
SCENARIO anthropic-sandbox-security
static_chars=14460 static_sha256_16=06f4ecdf0f669296
has_browser=False
has_security=True
has_security_risk=True
has_important=True
SCENARIO gemini-no-security-custom-soul
static_chars=12797 static_sha256_16=10128628f125ef51
has_browser=False
has_security=True
has_security_risk=True
has_important=True
RESULT legacy Agent.static_system_message renders for all baseline scenarios
This shows the new user-facing library entry point did not exist on main, while the legacy Agent.static_system_message path rendered successfully for the exercised scenarios.
Step 2 — Apply the PR changes:
Checked out PR commit 505e243ea318470fb9f8f0b16d988a5cc8283bcb.
Step 3 — Re-run with the PR in place:
Ran OPENHANDS_SUPPRESS_BANNER=1 uv run python /tmp/qa_prompt_registry_pr.py:
SCENARIO gpt5-cli-browser-security
legacy_chars=15908 legacy_sha256_16=eb14bbdf59f0f3c0 legacy_same_as_main=True
canonical_chars=15904 registry_chars=15904 static_matches_legacy=True
dynamic_is_none=True
has_browser=True
has_security=True
has_security_risk=True
has_important=True
SCENARIO anthropic-sandbox-security
legacy_chars=14460 legacy_sha256_16=06f4ecdf0f669296 legacy_same_as_main=True
canonical_chars=14456 registry_chars=14456 static_matches_legacy=True
dynamic_is_none=True
has_browser=False
has_security=True
has_security_risk=True
has_important=True
SCENARIO gemini-no-security-custom-soul
legacy_chars=12797 legacy_sha256_16=10128628f125ef51 legacy_same_as_main=True
canonical_chars=12793 registry_chars=12768 static_matches_legacy=False
dynamic_is_none=True
has_browser=False
has_security=False
has_security_risk=True
has_important=True
first_diff_at=7922 registry='_' legacy='>'
SCENARIO win32-gpt5-cli-security
legacy_chars=15218 legacy_sha256_16=36fda475513147a3 legacy_same_as_main=False
canonical_chars=15214 registry_chars=15214 static_matches_legacy=True
dynamic_is_none=True
has_browser=False
has_security=True
has_security_risk=True
has_important=True
win32_terms powershell=True bash=False
RESULT nominal registry scenarios match canonical legacy static output; documented empty-security-policy edge diverges
This shows the PR adds the registry entry point and that nominal registry output matches the legacy static prompt after the PR's documented inter-section gap canonicalization. The unchanged legacy_sha256_16 values for the Linux scenarios confirm the live legacy prompt path remains additive/unchanged. The win32 simulation confirms the shell-term refinement still matches legacy behavior for the exercised Windows path.
Issues Found
- 🟡 Minor:
security_policy_filename=""diverges from legacy static output (static_matches_legacy=False; registry omits<SECURITY>while legacy includes an empty wrapper). This is documented in the PR as outside the Phase 0 matrix/follow-up, so I do not consider it blocking for this behind-snapshot PR, but it should be resolved or explicitly carried into the Phase 3 cutover plan.
This review was created by an AI agent (OpenHands) on behalf of the user.
| </SECURITY>""" | ||
|
|
||
| def guard(self, ctx: PromptContext) -> bool: | ||
| return bool(ctx.template_kwargs.get("security_policy_filename")) |
There was a problem hiding this comment.
🟡 Suggestion: QA functional verification found an out-of-matrix parity divergence for Agent(security_policy_filename=""): legacy static_system_message still contains an empty <SECURITY> wrapper, while the registry output omits it (static_matches_legacy=False, has_security=False). The PR description notes this as intentional/follow-up, so it is not blocking for the current behind-snapshot phase, but please track it before the registry is wired into runtime to avoid a cutover behavior change for users who disable the default policy.
This comment was created by an AI agent (OpenHands) on behalf of the user.
There was a problem hiding this comment.
^^ I wonder why would it miss <SECURITY>?
Resolve conflicts after #3659 (static-part) landed on main: the static-tier files now exist on both sides. Keep the dynamic-tier additions (sections, registry registration, equivalence tests) on top of the merged static tier. Co-authored-by: openhands <openhands@all-hands.dev>
3 participants
HUMAN:
This is pahse 2 of #3606. In this PR, we just port the static part of the system prompt inside the new structure.
AGENT:
Why
Part of the prompt-registry roadmap (#3606; proposal #2827), which replaces the monolithic
system_prompt.j2+ post-renderrefine()with a typed section registry whose static/dynamic cache split is declared per-section andunit-testable.
This PR is the static-tier half of Phase 2 (#3610): it ports the ~17 static blocks of
system_prompt.j2into typedPromptSectionclasses assembled by a default registry, soregistry.build(ctx).staticreproduces today'sprompt byte-for-byte. It is purely additive and lands behind the Phase 0 snapshot — nothing is wired into the runtime prompt yet (that is the Phase 3 cutover).
Summary
PromptSectionclasses (no Jinja) incontext/prompts/sections/static.py, ported verbatim fromsystem_prompt.j2, plusbuild_default_registry()(context/prompts/default_registry.py) registering them intemplate order.
registry.build(ctx).staticreproducesAgentBase.static_system_messagebyte-for-byte across the full Phase 0 matrix (model family × browser × security-analyzer × cli_mode) and the win32 cell; every section also has astandalone unit test (no Jinja environment).
system_prompt.j2, the snapshot oracle, or any runtime path.refine()(win32 shell-term swap) is applied only in the two blocks that actually contain shell terms.Issue Number
#3610 (roadmap #3606, proposal #2827)
How to Test
Library-only change with no runtime wiring yet (behind the snapshot), so the end-to-end evidence is the byte-for-byte equivalence test — it builds real
Agentinstances (not mocks), renders the realstatic_system_message,and asserts the registry reproduces it:
Regression (legacy render path untouched) + Phase 1:
Full lint/type gate:
Video/Screenshots
N/A — no UI or runtime surface (the registry is not wired into the prompt yet). Evidence is the test output above; the equivalence test exercises real
Agent/LLMobjects, not mocks.Type
Notes
static_system_messageis not yet routed through the registry — that is the Phase 3 cutover. This PR only proves equivalence.{% if %}blocks. The equivalence test normalizes only those</TAG>…3+ blanks…<TAG>boundaries (onetag-anchored regex), so every section body is asserted byte-for-byte. The literal whitespace shift lands at the Phase 3 cutover.
static.pycarries a file-level# ruff: noqa: E501for its verbatim long prompt lines;pyproject.tomlis untouched.<SECURITY>is guarded onsecurity_policy_filename(omitted when empty, vs. the template's empty tags); a customsecurity_policy_filenamewould resolve its contentinto the context (follow-up).
DateTime/RepoContext/AvailableSkills/CustomSuffix/CustomSecrets) fromsystem_message_suffix.j2.Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:505e243-pythonRun
All tags pushed for this build
About Multi-Architecture Support
505e243-python) is a multi-arch manifest supporting both amd64 and arm64505e243-python-amd64) are also available if needed