Skip to content

[codex] Add full auth ops canary coverage#2376

Closed
serrrfirat wants to merge 1 commit intocodex/auth-oauth-canary-unificationfrom
codex/pr2367-auth-workflow-port
Closed

[codex] Add full auth ops canary coverage#2376
serrrfirat wants to merge 1 commit intocodex/auth-oauth-canary-unificationfrom
codex/pr2367-auth-workflow-port

Conversation

@serrrfirat
Copy link
Copy Markdown
Collaborator

Stacks on #2367.\n\nAdds the missing full-workflow auth canary coverage on top of the unified live canary framework:\n\n- adds a Linear seeded auth case\n- adds an opt-in ops_workflow seeded case that installs and probes Gmail, Calendar, Drive, Docs, Sheets, Slides, GitHub, Web Search, LLM Context, Slack, Telegram, Composio, Notion, and Linear in one deterministic /v1/responses turn\n- seeds the extra token/manual-auth material needed by Brave, Slack, Composio, Telegram, and Linear\n- documents the fixture objects and CI env/secrets for the combined workflow\n\nValidated locally:\n\n- python3 -m py_compile scripts/live_canary/auth_registry.py scripts/live_canary/auth_runtime.py scripts/auth_live_canary/run_live_canary.py tests/e2e/mock_llm.py\n- bash -n scripts/live-canary/run.sh && bash -n scripts/auth_live_canary/config.example.env\n- git diff --check origin/codex/auth-oauth-canary-unification..HEAD\n- seeded registry selection with fake fixture envs

@github-actions github-actions bot added size: L 200-499 changed lines scope: ci CI/CD workflows scope: docs Documentation risk: medium Business logic, config, or moderate-risk modules contributor: core 20+ merged PRs and removed size: L 200-499 changed lines labels Apr 12, 2026
@serrrfirat serrrfirat closed this Apr 12, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new ops_workflow canary case and a linear provider case to the live canary testing suite. It expands the Google integration to include Drive, Docs, Sheets, and Slides, and adds support for Brave Search, Slack, Telegram, and Composio. Key architectural changes include updating the registry to support multiple extension installations per probe, adding a write_memory utility for seeding workspace files, and updating the mock LLM to handle the new tool calls. Feedback focuses on improving the robustness of the mock LLM by using environment variables in regex patterns for tool matching and safely parsing integer environment variables to prevent runtime crashes.

Comment thread tests/e2e/mock_llm.py
"Notion search completed successfully.",
),
(
re.compile(r"Tool `linear_.*` returned:", re.IGNORECASE | re.DOTALL),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The regex for matching Linear tool output in CANNED_RESPONSES is hardcoded to expect a tool name starting with linear_. If a custom tool name is provided via the AUTH_LIVE_LINEAR_TOOL_NAME environment variable that does not follow this pattern, the mock LLM will fail to produce the expected completion text, causing the canary test to fail. It's better to use the environment variable in the regex with a fallback.

Suggested change
re.compile(r"Tool `linear_.*` returned:", re.IGNORECASE | re.DOTALL),
re.compile(rf"Tool {os.getenv('AUTH_LIVE_LINEAR_TOOL_NAME', 'linear_.*')} returned:", re.IGNORECASE | re.DOTALL),

Comment thread tests/e2e/mock_llm.py
def _auth_ops_workflow_calls() -> list[dict]:
github_owner = os.environ["AUTH_LIVE_GITHUB_OWNER"]
github_repo = os.environ["AUTH_LIVE_GITHUB_REPO"]
github_issue_number = int(os.environ["AUTH_LIVE_GITHUB_ISSUE_NUMBER"])
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Directly calling int() on os.environ["AUTH_LIVE_GITHUB_ISSUE_NUMBER"] can raise a ValueError if the environment variable is not a valid integer, which would crash the mock LLM process. While the registry logic checks for the presence of the variable, it doesn't validate its format. A safer approach would be to handle the potential conversion error.

Suggested change
github_issue_number = int(os.environ["AUTH_LIVE_GITHUB_ISSUE_NUMBER"])
try:
github_issue_number = int(os.environ.get("AUTH_LIVE_GITHUB_ISSUE_NUMBER", "0"))
except ValueError:
github_issue_number = 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor: core 20+ merged PRs risk: medium Business logic, config, or moderate-risk modules scope: ci CI/CD workflows scope: docs Documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant