Skip to content

Feature: api-proxy should export models.json artifact with available models #2338

@lpcox

Description

@lpcox

Summary

The api-proxy should write a models.json file at the end of a workflow run (or on-demand) that describes which models were available from each configured provider. This artifact would be uploaded alongside existing workflow artifacts (agent, firewall-audit-logs) for observability and debugging.

Motivation

  • Debugging model availability issues: When a workflow fails because a requested model is unavailable (e.g., sweagentd#11264), having a record of what models were actually available at runtime makes diagnosis trivial.
  • Model selection strategy: Upcoming model-selection policy support ([aw] Security Guard failed #2334) needs a ground-truth list of available models to implement fallback logic.
  • Audit trail: Teams can track which models were offered over time, detect regressions in model availability, and validate that expected models are present.

Proposed Behavior

  1. At startup (after fetchStartupModels() completes), write /var/log/api-proxy/models.json with the discovered models from each provider.
  2. Update on refresh if models are re-fetched during the run.
  3. The existing api-proxy-logs volume mount already maps /var/log/api-proxy/ to the host, so the file is automatically available for artifact upload.

Proposed Schema

{
  "timestamp": "2026-05-01T03:00:00Z",
  "providers": {
    "openai": {
      "configured": true,
      "models": ["gpt-4.1", "gpt-4.1-mini", "o3", "o4-mini", ...],
      "target": "api.openai.com"
    },
    "anthropic": {
      "configured": false,
      "models": null,
      "target": null
    },
    "copilot": {
      "configured": true,
      "models": ["gpt-4.1", "claude-sonnet-4", ...],
      "target": "api.githubcopilot.com"
    },
    "gemini": {
      "configured": false,
      "models": null,
      "target": null
    },
    "opencode": {
      "configured": false,
      "models": null,
      "target": null
    }
  },
  "model_aliases": {
    "fast": ["gpt-4.1-mini"],
    "smart": ["o3"]
  }
}

Implementation Notes

  • The data is already available in the cachedModels object and reflectEndpoints() function in server.js
  • Write the file after fetchStartupModels() resolves (line ~1553 in server.js)
  • Use the existing /var/log/api-proxy/ directory which is volume-mounted to ${workDir}/api-proxy-logs/
  • Include model_aliases from AWF_MODEL_ALIASES if configured
  • Workflows that upload firewall-audit-logs could include this file, or a separate models artifact could be defined

Files to Change

  • containers/api-proxy/server.js — Write models.json after model fetch completes
  • Optionally: workflow templates to upload the artifact

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions