Skip to content

feat: orchestrate on-demand runtime residency #494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
schickling-assistant wants to merge 1 commit into from
Draft

feat: orchestrate on-demand runtime residency #494

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/vrs/spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -716,6 +716,20 @@ state, and Resource observation are not demand. Host policy owns idle thresholds
and warm capacity. Telemetry measures resume-to-ready and demand-to-delivery
latency before the project assigns a latency objective.

The folder-catalog supervisor enables this policy only when `st2 up` receives
both `--residency-idle-after <duration>` and
`--residency-warm-capacity <count>`. If a catalog contains a running on-demand
agent and the host omits this policy, reconciliation holds that agent
`adopt-only` and reports an error instead of launching it. `st2 wake` writes an
explicit durable wake request. `st2 pty attach <agent-address>` writes the same
request before attaching. Each launch attempt has a host-minted incarnation
fence. A retry after a missing provider rotates this fence. A live provider
retains the fence while reconciliation completes its sidecars. Provider
readiness must match the independent fence, so a binding left by an earlier
failed attempt cannot make a later attempt active. A wake request that races
with checkpoint or stop remains pending until the complete task group is absent
and exact native resume can start.

## Host-local scheduling and supervision

```text
Expand Down
103 changes: 83 additions & 20 deletions src/claude_session.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ pub fn run(
runtime_id: String,
claude_argv: Vec<String>,
) -> Result<()> {
run_with_required_resume(catalog_root, identity, runtime_id, claude_argv, None)
run_with_required_resume(catalog_root, identity, runtime_id, claude_argv, None, None)
}

pub fn run_residency(
Expand All @@ -44,12 +44,36 @@ pub fn run_residency(
claude_argv: Vec<String>,
resume_generation: crate::residency::Generation,
) -> Result<()> {
run_residency_attempt(
catalog_root,
identity,
runtime_id,
claude_argv,
resume_generation,
harness_state::session_token(),
)
}

/// Run one host-owned cold-residency attempt under its exact incarnation.
pub fn run_residency_attempt(
catalog_root: &Path,
identity: String,
runtime_id: String,
claude_argv: Vec<String>,
resume_generation: crate::residency::Generation,
required_incarnation: String,
) -> Result<()> {
anyhow::ensure!(
!required_incarnation.is_empty(),
"Claude required runtime incarnation is empty"
);
run_with_required_resume(
catalog_root,
identity,
runtime_id,
claude_argv,
Some(resume_generation),
Some(required_incarnation),
)
}

Expand All @@ -59,11 +83,16 @@ fn run_with_required_resume(
runtime_id: String,
claude_argv: Vec<String>,
required_resume_generation: Option<crate::residency::Generation>,
required_incarnation: Option<String>,
) -> Result<()> {
anyhow::ensure!(
!claude_argv.is_empty(),
"Claude driver '{runtime_id}' has no provider argv"
);
anyhow::ensure!(
required_resume_generation.is_some() == required_incarnation.is_some(),
"Claude residency launch has an incomplete attempt fence"
);
let workspace = std::env::current_dir().context("reading the Claude driver workspace")?;
let required_native_session = match required_resume_generation {
Some(generation) => Some(required_residency_resume(
Expand All @@ -85,7 +114,12 @@ fn run_with_required_resume(
crate::pretrust::pretrust_claude(std::slice::from_ref(&workspace))
.with_context(|| format!("admitting Claude driver workspace {}", workspace.display()))?;
install_signal_handler();
let observer = SessionObserver::new(&agent_dir, &identity, "claude", &runtime_id)?;
let observer = match required_incarnation {
Some(session) => {
SessionObserver::with_session(&agent_dir, &identity, "claude", &runtime_id, session)?
}
None => SessionObserver::new(&agent_dir, &identity, "claude", &runtime_id)?,
};
let mut env = vec![
(RUNTIME_ID_ENV.to_string(), runtime_id.clone()),
(SESSION_ENV.to_string(), observer.session().to_string()),
Expand Down Expand Up @@ -689,6 +723,13 @@ fn load_checkpoint(
&& checkpoint.binding.runtime_id == runtime_id,
"Claude residency checkpoint belongs to a different agent runtime"
);
anyhow::ensure!(
!checkpoint.binding.runtime_incarnation.is_empty()
&& valid_uuid(&checkpoint.binding.native_session_id)
&& checkpoint.binding.canonical_workspace.is_absolute()
&& checkpoint.binding.transcript_path.is_absolute(),
"Claude residency checkpoint has an incomplete native session binding"
);
anyhow::ensure!(
checkpoint.transcript_sha256.len() == 64
&& checkpoint
Expand All @@ -700,6 +741,25 @@ fn load_checkpoint(
Ok(checkpoint)
}

fn validate_residency_candidate(
checkpoint: &ClaudeResidencyCheckpoint,
candidate: &ClaudeSessionBinding,
resume_generation: crate::residency::Generation,
) -> Result<()> {
anyhow::ensure!(
candidate.resume_generation == Some(resume_generation)
&& candidate.runtime_incarnation != checkpoint.binding.runtime_incarnation,
"Claude SessionStart does not prove the required residency generation and a new wrapper incarnation"
);
anyhow::ensure!(
candidate.native_session_id == checkpoint.binding.native_session_id
&& candidate.canonical_workspace == checkpoint.binding.canonical_workspace
&& candidate.transcript_path == checkpoint.binding.transcript_path,
"Claude resumed a different native session than the residency checkpoint"
);
Ok(())
}

pub fn required_residency_resume(
state_dir: &Path,
agent: &str,
Expand Down Expand Up @@ -739,15 +799,11 @@ pub fn residency_ready(
let current = load_binding(state_dir, agent, runtime_id)?;
if let Some(current) = &current
&& current.resume_generation == Some(resume_generation)
&& current.runtime_incarnation == expected_runtime_incarnation
{
anyhow::ensure!(
current.native_session_id == checkpoint.binding.native_session_id
&& current.canonical_workspace == checkpoint.binding.canonical_workspace
&& current.transcript_path == checkpoint.binding.transcript_path,
"Claude ready binding does not match the residency checkpoint"
);
return Ok(true);
validate_residency_candidate(&checkpoint, current, resume_generation)?;
if current.runtime_incarnation == expected_runtime_incarnation {
return Ok(true);
}
}
let Some(candidate) = load_pending_binding(state_dir, agent, runtime_id)? else {
return Ok(false);
Expand All @@ -759,17 +815,10 @@ pub fn residency_ready(
"Claude native session changed after this residency generation became ready"
);
}
validate_residency_candidate(&checkpoint, &candidate, resume_generation)?;
anyhow::ensure!(
candidate.resume_generation == Some(resume_generation)
&& candidate.runtime_incarnation != checkpoint.binding.runtime_incarnation
&& candidate.runtime_incarnation == expected_runtime_incarnation,
"Claude SessionStart does not prove the required residency generation and wrapper incarnation"
);
anyhow::ensure!(
candidate.native_session_id == checkpoint.binding.native_session_id
&& candidate.canonical_workspace == checkpoint.binding.canonical_workspace
&& candidate.transcript_path == checkpoint.binding.transcript_path,
"Claude resumed a different native session than the residency checkpoint"
candidate.runtime_incarnation == expected_runtime_incarnation,
"Claude SessionStart does not prove the expected wrapper incarnation"
);
validate_transcript(
&candidate.transcript_path,
Expand Down Expand Up @@ -2481,6 +2530,20 @@ mod tests {
}

let temp = tempfile::tempdir().unwrap();
let empty_incarnation = run_residency_attempt(
temp.path(),
"no-spawn.worker".into(),
"no-spawn.worker".into(),
vec!["claude".into()],
crate::residency::Generation(2),
String::new(),
)
.unwrap_err();
assert!(
empty_incarnation
.to_string()
.contains("required runtime incarnation is empty")
);
let marker = temp.path().join("provider-started");
let provider = temp.path().join("claude");
fs::write(
Expand Down
Loading