feat: implement Moho recursive proof generation

[prajwolrg]

Description

• Implement end-to-end Moho recursive proof generation in the proof orchestrator, building on existing ASM step proofs. The orchestrator now checks prerequisites (ASM proof + previous Moho proof), constructs the recursive input with Merkle inclusion proofs for the predicate key, and submits to a dedicated Moho remote prover.
• Extract ProverEnv from BasicEnv in functional tests, making the orchestrator config optional so non-prover tests don't spin up proving infrastructure. Add Moho proof assertions to the existing proof generation test.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature/Enhancement (non-breaking change which adds functionality or enhances an existing one)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactor
• New or updated tests
• Dependency update
• Security fix

Notes to Reviewers

Checklist

• I have performed a self-review of my code.
• I have commented my code where necessary.
• I have updated the documentation if needed.
• My changes do not introduce new warnings.
• I have added tests that prove my changes are effective or that my feature works.
• New and existing tests pass with my changes.

Related Issues

[codecov]

Codecov Report

❌ Patch coverage is 95.62500% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
bin/asm-runner/src/bootstrap.rs	68.75%	5 Missing ⚠️
bin/asm-runner/src/block_watcher.rs	71.42%	2 Missing ⚠️

Files with missing lines	Coverage Δ
bin/asm-runner/src/prover/input.rs	99.30% <100.00%> (+1.47%)	⬆️
bin/asm-runner/src/prover/orchestrator.rs	86.95% <100.00%> (+1.98%)	⬆️
crates/proof/types/src/lib.rs	75.43% <100.00%> (+2.36%)	⬆️
bin/asm-runner/src/block_watcher.rs	83.15% <71.42%> (+0.74%)	⬆️
bin/asm-runner/src/bootstrap.rs	91.20% <68.75%> (-0.26%)	⬇️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Commit: 9279c4c
SP1 Execution Results

program	cycles	success
ASM STF	118,483,823	yes
Moho Recursive	11,586,718	yes

[storopoli]

ProofId::Moho is submitted through self.moho, but reconciliation still always goes back through self.asm, so a real dedicated Moho prover will never be polled or fetched correctly. Submission dispatches to the Moho host here:

• asm/bin/asm-runner/src/prover/orchestrator.rs

  Lines 256 to 271 in e422361

  	ProofId::Moho(block) => {
  	let prerequisite = match self.input_builder.check_moho_prerequisite(*block).await {
  	Ok(prereq) => prereq,
  	Err(e) => {
  	warn!(?e, "moho proof generation cannot be done yet, re-enqueuing");
  	self.queue.enqueue(proof_id);
  	return Ok(());
  	}
  	};
  	let input = self
  	.input_builder
  	.build_moho_runtime_input(prerequisite, *block)
  	.await?;
  	MohoRecursiveProgram::start_proving(&input, &self.moho)
  	.await
  	.map_err(|e| anyhow::anyhow!("failed to submit proof to remote prover: {e}"))?

But status polling and proof retrieval are still hard-wired to the ASM host here:

• asm/bin/asm-runner/src/prover/orchestrator.rs

  Lines 123 to 165 in e422361

  	async fn reconcile_one(
  	&self,
  	remote_id: &RemoteProofId,
  	old_status: &RemoteProofStatus,
  	) -> Result<()> {
  	let typed_id = to_typed_proof_id::<R>(remote_id)?;
  	let new_status = self
  	.asm
  	.get_status(&typed_id)
  	.await
  	.map_err(|e| anyhow::anyhow!("failed to query remote proof status: {e}"))?;
  	if &new_status == old_status {
  	return Ok(());
  	}
  	debug!(
  	%remote_id,
  	?old_status,
  	?new_status,
  	"remote proof status changed"
  	);
  	match &new_status {
  	RemoteProofStatus::Completed => {
  	self.handle_completed(remote_id, &typed_id).await?;
  	}
  	RemoteProofStatus::Failed(reason) => {
  	error!(?remote_id, %reason, "remote proof generation failed");
  	self.db
  	.remove(remote_id)
  	.await
  	.context("failed to remove failed proof status")?;
  	}
  	_ => {
  	self.db
  	.update_status(remote_id, new_status)
  	.await
  	.context("failed to update proof status")?;
  	}
  	}
  	Ok(())

• asm/bin/asm-runner/src/prover/orchestrator.rs

  Lines 168 to 194 in e422361

  	/// Retrieves a completed proof and stores it in the proof DB.
  	async fn handle_completed(
  	&self,
  	remote_id: &RemoteProofId,
  	typed_id: &R::ProofId,
  	) -> Result<()> {
  	let receipt = self
  	.asm
  	.get_proof(typed_id)
  	.await
  	.map_err(|e| anyhow::anyhow!("failed to retrieve completed proof: {e}"))?;
  	let proof_id = self
  	.db
  	.get_proof_id(remote_id)
  	.await
  	.context("failed to look up proof ID from remote ID")?
  	.context("no mapping found for completed remote proof")?;
  	proof_store::store_completed_proof(&self.db, proof_id, receipt).await?;
  	self.db
  	.remove(remote_id)
  	.await
  	.context("failed to remove completed proof status")?;
  	Ok(())

This passes with NativeHost because the proof ID is effectively the receipt, but it breaks the PR's stated design once ASM and Moho use different prover backends. The reconciler needs to dispatch by proof kind, or persist backend identity alongside the remote ID, before this is safe.

[MdTeach]

Overall looks good!!
Few questions and nits

[prajwolrg]

ProofId::Moho is submitted through self.moho, but reconciliation still always goes back through self.asm, so a real dedicated Moho prover will never be polled or fetched correctly. Submission dispatches to the Moho host here:

We use self.asm here but this could be any ZkVmRemoteHost instance. get_status only requires a network client and proof ID — not the ELF or proving key. Since the orchestrator is generic over a single R: ZkVmRemoteHost, both asm and moho share the same concrete type, so either works.

But status polling and proof retrieval are still hard-wired to the ASM host here:

We use self.asm here but this could be any ZkVmRemoteHost instance. get_status only requires a network client and proof ID — not the ELF or proving key. Since the orchestrator is generic over a single R: ZkVmRemoteHost, both asm and moho share the same concrete type, so either works.

I've added a comment to clarify this in c3595e6. Note that since the plan is to later move the orchestration logic to use paas once alpenlabs/alpen#1573 merges. This will be much cleaner then.

This passes with NativeHost because the proof ID is effectively the receipt, but it breaks the PR's stated design once ASM and Moho use different prover backends. The reconciler needs to dispatch by proof kind, or persist backend identity alongside the remote ID, before this is safe.

This works with remote host as well, because the R::ProofId will be a proper identifier for the remote proofs. I did proof generation as part of https://github.com/alpenlabs/asm/pull/54/changes and it works.

[evgenyzdanovich]

This look good, thanks.

[evgenyzdanovich]

re-ACKing

[storopoli]

I'm not sure about the AlwaysAccept hard-coded, maybe that was intentional from you. But the SP1 4 byte receipts is a real blocker.

[storopoli]

ACK 0f62fd1

Great! Rebase on top of alpenlabs/strata-common#88 (might need a new tag)