[codex] Add full auth ops canary coverage#2376
[codex] Add full auth ops canary coverage#2376serrrfirat wants to merge 1 commit intocodex/auth-oauth-canary-unificationfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new ops_workflow canary case and a linear provider case to the live canary testing suite. It expands the Google integration to include Drive, Docs, Sheets, and Slides, and adds support for Brave Search, Slack, Telegram, and Composio. Key architectural changes include updating the registry to support multiple extension installations per probe, adding a write_memory utility for seeding workspace files, and updating the mock LLM to handle the new tool calls. Feedback focuses on improving the robustness of the mock LLM by using environment variables in regex patterns for tool matching and safely parsing integer environment variables to prevent runtime crashes.
| "Notion search completed successfully.", | ||
| ), | ||
| ( | ||
| re.compile(r"Tool `linear_.*` returned:", re.IGNORECASE | re.DOTALL), |
There was a problem hiding this comment.
The regex for matching Linear tool output in CANNED_RESPONSES is hardcoded to expect a tool name starting with linear_. If a custom tool name is provided via the AUTH_LIVE_LINEAR_TOOL_NAME environment variable that does not follow this pattern, the mock LLM will fail to produce the expected completion text, causing the canary test to fail. It's better to use the environment variable in the regex with a fallback.
| re.compile(r"Tool `linear_.*` returned:", re.IGNORECASE | re.DOTALL), | |
| re.compile(rf"Tool {os.getenv('AUTH_LIVE_LINEAR_TOOL_NAME', 'linear_.*')} returned:", re.IGNORECASE | re.DOTALL), |
| def _auth_ops_workflow_calls() -> list[dict]: | ||
| github_owner = os.environ["AUTH_LIVE_GITHUB_OWNER"] | ||
| github_repo = os.environ["AUTH_LIVE_GITHUB_REPO"] | ||
| github_issue_number = int(os.environ["AUTH_LIVE_GITHUB_ISSUE_NUMBER"]) |
There was a problem hiding this comment.
Directly calling int() on os.environ["AUTH_LIVE_GITHUB_ISSUE_NUMBER"] can raise a ValueError if the environment variable is not a valid integer, which would crash the mock LLM process. While the registry logic checks for the presence of the variable, it doesn't validate its format. A safer approach would be to handle the potential conversion error.
| github_issue_number = int(os.environ["AUTH_LIVE_GITHUB_ISSUE_NUMBER"]) | |
| try: | |
| github_issue_number = int(os.environ.get("AUTH_LIVE_GITHUB_ISSUE_NUMBER", "0")) | |
| except ValueError: | |
| github_issue_number = 0 |
Stacks on #2367.\n\nAdds the missing full-workflow auth canary coverage on top of the unified live canary framework:\n\n- adds a Linear seeded auth case\n- adds an opt-in ops_workflow seeded case that installs and probes Gmail, Calendar, Drive, Docs, Sheets, Slides, GitHub, Web Search, LLM Context, Slack, Telegram, Composio, Notion, and Linear in one deterministic /v1/responses turn\n- seeds the extra token/manual-auth material needed by Brave, Slack, Composio, Telegram, and Linear\n- documents the fixture objects and CI env/secrets for the combined workflow\n\nValidated locally:\n\n- python3 -m py_compile scripts/live_canary/auth_registry.py scripts/live_canary/auth_runtime.py scripts/auth_live_canary/run_live_canary.py tests/e2e/mock_llm.py\n- bash -n scripts/live-canary/run.sh && bash -n scripts/auth_live_canary/config.example.env\n- git diff --check origin/codex/auth-oauth-canary-unification..HEAD\n- seeded registry selection with fake fixture envs