[P1] foundry-hosting: auto-grant Foundry User to per-version MI in deploy_hosted_agent.py

[Cataldir]

Problem statement

When a Foundry V3 hosted-agent version is created via the SDK path (lib/src/holiday_peak_lib/foundry_hosting/deploy.py ÔåÆ client.agents.create_version), the platform mints a new per-version managed identity (instance_identity.principal_id) without granting it the Foundry User role on the project. As a result, the container starts, accepts requests, runs the agent handler successfully, and then fails to persist the response: every POST /storage/responses from the container returns HTTP 401 and the public /responses call surfaces as HTTP 500 to the caller ÔÇö even though the agent code is healthy.

The azd up / VS Code Foundry-extension deploy path does perform this RBAC grant automatically. The SDK path does not, so any agent deployed by scripts/ops/deploy_hosted_agent.py requires a manual operator step before it can serve traffic. That manual step blocked PR #1103 for ~40 minutes and is the only remaining wart in the otherwise fully working pilot.

Once granted, the role is permanent for that specific per-version MI, but a fresh MI is minted on every create_version, so the grant must be applied on each new version. The blueprint MI (blueprint.principal_id) is stable across versions of the same agent name and only needs to be granted once.

Current behavior evidence

• lib/src/holiday_peak_lib/foundry_hosting/deploy.py ÔÇö deploy_hosted_agent_version (line ~430) calls client.agents.create_version, then _resolve_latest_version, then optionally polls until terminal. No role-assignment step exists.
• scripts/ops/deploy_hosted_agent.py ÔÇö the CLI wrapper around the library function. Same behaviour.
• App Insights trace from inventory-health-check v15ÔÇôv19 (before manual grant ÔÇö PR feat(#990): Foundry V3 hosted-agents pilot end-to-end (with portal-visibility fixes) #1103 pilot logs):

  Foundry storage POST .../storage/responses?api-version=v1 -> 401
  Inbound POST /responses completed with status 500
  

• App Insights trace from v20 (after manual grant of Foundry User to the per-version MI):

  DefaultAzureCredential acquired a token from ManagedIdentityCredential
  Foundry storage POST .../storage/responses?api-version=v1 -> 201
  Response caresp_ completed: status=completed output_count=1
  Inbound POST /responses completed with status 200
  

• Manual fix that unblocked PR feat(#990): Foundry V3 hosted-agents pilot end-to-end (with portal-visibility fixes) #1103:

  $projectScope = "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<account>/projects/<project>"
  az role assignment create \
    --assignee-object-id <instance_identity.principal_id> \
    --assignee-principal-type ServicePrincipal \
    --role "Foundry User" \
    --scope $projectScope

• Runbook reference: memories/session/foundry-v3-pilot-status.md ┬º"Three root causes (in order of discovery)" ÔÇö case 3.

Required change

deploy_hosted_agent_version must, after _resolve_latest_version returns and before entering the poll loop:

1. Extract instance_identity.principal_id and blueprint.principal_id from version_obj (both are nested objects on HostedAgentVersion).
2. Resolve the project ARM resource ID from the project endpoint (https://<account>.services.ai.azure.com/api/projects/<project> ÔåÆ /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<account>/projects/<project>). The account-name / project-name segments come from the endpoint URL; the subscription + RG can come from AZURE_SUBSCRIPTION_ID + AZURE_RESOURCE_GROUP env vars or from an az resource show lookup on the account.
3. Use azure.mgmt.authorization.AuthorizationManagementClient.role_assignments.create to assign Foundry User (role definition ID is stable ÔÇö record it as a module constant). Use a deterministic name=guid(scope, principal_id, role_def_id) so the call is idempotent (HTTP 201 on first call, HTTP 409 on subsequent ÔÇö treat 409 as success).
4. Apply the same grant to the blueprint MI (idempotent ÔÇö usually a no-op after the first version of a given agent name).
5. Emit a structured log foundry_hosted_agent_rbac_granted capturing principal_id, role, scope, and the assignment ID.
6. Fail-soft behaviour: if the calling identity does not have Microsoft.Authorization/roleAssignments/write on the project scope (HTTP 403 from the API), log a clear warning explaining the manual grant needed and surface a RoleAssignmentSkipped field on HostedAgentDeploymentResult ÔÇö do not fail the overall deploy, because in CI scenarios the caller may legitimately delegate RBAC to a separate workflow.

Suggested API surface

Add a grant_rbac: bool = True flag and an optional rbac_credential: TokenCredential | None = None (defaults to the same credential used for the project client). New module constant:

_FOUNDRY_USER_ROLE_DEFINITION_ID = (
    "/providers/Microsoft.Authorization/roleDefinitions/"
    "<role-id-for-Foundry User>"
)

(Confirm the exact role definition GUID with az role definition list --name "Foundry User" during implementation ÔÇö do not hard-code an unverified GUID.)

CLI flag wiring

Expose --no-grant-rbac on scripts/ops/deploy_hosted_agent.py for CI flows that already delegate RBAC, and --rbac-scope <arm-id> for projects whose scope cannot be inferred from the endpoint.

Acceptance criteria

• deploy_hosted_agent_version performs the two role grants (instance MI + blueprint MI) by default after _resolve_latest_version.
• Grant is idempotent (deterministic assignment name; HTTP 409 treated as success).
• On HTTP 403 from the role-assignment API, a warning is logged and HostedAgentDeploymentResult.extras["rbac_skipped"] = True; the deploy still returns the version result.
• grant_rbac=False / --no-grant-rbac flag fully skips the section and emits a structured log noting the skip.
• Unit tests cover: happy path (201), idempotent path (409), permission-denied path (403 ÔåÆ warn + extras flag), and skip path (grant_rbac=False).
• Integration test (live, opt-in via env var) deploys an agent, asserts the version is active, and asserts an invocation against the public endpoint returns HTTP 200 with Foundry storage POST  -> 201 in the App Insights trace.
• memories/session/foundry-v3-pilot-status.md is updated to drop case 3 from the manual runbook and reference this PR/issue.
• scripts/ops/deploy_hosted_agent.py CLI help documents the new flags.

Risks and dependencies

• Risk: Hard-coding the Foundry User role definition GUID. Mitigation: look it up dynamically once on first call and cache per-process; record the resolved GUID in the structured log.
• Risk: azure-mgmt-authorization adds a transitive dependency. Mitigation: it is already in the dependency closure of azure-identity / azure-mgmt-core; verify and add an explicit pin in lib/pyproject.toml.
• Risk: Calling-identity RBAC mismatch in CI. Mitigation: the fail-soft behaviour above + the --no-grant-rbac flag let workflows opt out cleanly.
• Dependency: PR feat(#990): Foundry V3 hosted-agents pilot end-to-end (with portal-visibility fixes) #1103 must merge first ÔÇö it lands the _pick_latest_version + _normalize_status patches this issue's implementation will build on.
• Dependency: Sibling issue [P2] foundry-hosting: codify ACR prerequisites for Foundry V3 hosted-agents in IaC #1108 (ACR prerequisites in IaC) ÔÇö together they close the operational gap that PR feat(#990): Foundry V3 hosted-agents pilot end-to-end (with portal-visibility fixes) #1103 ran into.

ADR impact

No ADR is invalidated. This implements the operator-facing contract for ADR-046 (Foundry V3 hosted-agents) ÔÇö recommended to add a short addendum to ADR-046 capturing "SDK deploy path auto-grants Foundry User on per-version MI; manual azd flow continues to rely on azd auth permissions".

Labels

• priority: high
• type:backend
• type:devops
• foundry

BPMN process

%%{init: {'theme':'base', 'themeVariables': {
  'primaryColor':'#FFB3BA',
  'primaryTextColor':'#000',
  'primaryBorderColor':'#FF8B94',
  'lineColor':'#BAE1FF',
  'secondaryColor':'#BAE1FF',
  'tertiaryColor':'#FFFFFF'
}}}%%
flowchart LR
  A[Confirm Foundry User Role GUID] --> B[Design RBAC Step]
  B --> C[Implement on Issue Branch]
  C --> D[Unit + Integration Tests]
  D --> E[Open PR]
  E --> F[Validation and Fixes]
  F --> G[Merge to Main]
  G --> H[Update Runbook + Close Issue]

Loading

---

Filed as a follow-up to PR #1103. The manual grant described here was applied during the pilot and proved the design works; this issue tracks moving the grant into the SDK path so future hosted-agent deployments don't require operator RBAC steps.

[Cataldir]

Sequencing note: this issue's implementation depends on PR #1103 (which lands _pick_latest_version / _normalize_status patches that the new RBAC step in deploy_hosted_agent_version will build on). Branch issue/1107-foundry-user-auto-grant from main after #1103 merges, not before, to avoid a 3-way merge inside deploy.py.

[Cataldir]

Status: implementation landed on feature/foundry-hosted-agents-pilot as commit 582f443e. Closes when #1103 merges.

The auto-grant is now built into deploy_hosted_agent_version and the
scripts/ops/deploy_hosted_agent.py CLI. No separate PR is needed; the
work is bundled into the same PR that introduced the SDK deploy path
(#1103).

Summary

• On reaching active, the helper:
  1. Reads instance_identity.principal_id from the version object
     (with fallbacks for managed_identity / identity field aliases
     across preview SDK builds, plus Mapping-style models).
  2. Derives the project ARM scope from the project_endpoint URL
     (https://{account}.services.ai.azure.com/api/projects/{project})
     via az resource list --resource-type Microsoft.CognitiveServices/accounts --name {account}. Operators
     can override with --project-scope <arm-id>.
  3. Calls az role assignment create --assignee-object-id <principal_id> --assignee-principal-type ServicePrincipal --role 'Foundry User' --scope <project-scope> — matching the
     manual runbook one-for-one.
  4. Treats RoleAssignmentExists as success (idempotent).
• New CLI flags: --no-auto-grant-foundry-user (opt out),
  --foundry-role-name <name> (override role), --project-scope <id>
  (skip lookup). JSON output surfaces role_grant.status.
• A failed grant does NOT mask a successful version activation. The
  failure is captured in result.extras["role_grant"] with
  status=failed, error=<stderr> so operators can re-run.

Verification matrix

Path	Test
End-to-end deploy auto-grants on active	test_deploy_auto_grants_foundry_user_after_active
--no-auto-grant-foundry-user skips the call	test_deploy_skips_grant_when_auto_grant_disabled
Idempotent on existing assignment	test_deploy_records_already_exists_when_granter_returns_none
Failure recorded, deploy still succeeds	test_deploy_surfaces_grant_failure_without_breaking_active
Missing principal-id is graceful	test_deploy_records_skipped_when_principal_id_missing
--project-scope override bypasses az lookup	test_deploy_uses_explicit_project_scope_override
Principal-id extraction: 3 SDK field shapes	test_extract_principal_id_from_* (3 tests)
Scope derivation: resolver / malformed / no-account	test_derive_project_scope_* (3 tests)
Default granter: success / already-exists / failure	test_grant_role_via_az_* (3 tests)

All tests pass; full lib suite (1376) green.

Operator impact

The verify steps in #1103's body have been updated. Old step 4 (manual
az role assignment create) is gone. The new flow is:

python scripts/ops/deploy_hosted_agent.py `
  --agent-yaml apps/inventory-health-check/agent.hosted.yaml `
  --image-uri "holidaypeakhub405devacr.azurecr.io/inventory-health-check@sha256:..." `
  --project-endpoint $env:PROJECT_ENDPOINT --json
# (Foundry User auto-granted; no extra step required)

Closing this issue is gated on #1103 merging.