feat(proxy): attribute usage to every team a user belongs to (opt-in) #35356
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Agent Shin — reconsider | |
| # Comment-trigger workflow: when the PR/issue author (or an internal | |
| # collaborator) comments `@agent-shin reconsider` on a CLOSED PR/issue, | |
| # Agent Shin re-runs LLM-judge triage on the current title+body and: | |
| # | |
| # - on PASS: posts a "re-evaluated and reopened" comment + reopens. | |
| # - on FAIL: posts a "still missing X" comment and leaves it closed, | |
| # so the contributor can iterate again. | |
| # | |
| # This exists because GitHub does NOT let an external (non-write-access) | |
| # OSS contributor reopen a PR/issue closed by a bot or maintainer. Without | |
| # this comment trigger, a contributor whose PR Agent Shin auto-closed | |
| # would have no path back into the review queue except opening a fresh PR | |
| # (which loses the original PR's history). The bot, on the other hand, | |
| # has write access via GH_TOKEN and can reopen on their behalf. | |
| # | |
| # DRY-RUN BY DEFAULT — gated on `vars.AGENT_SHIN_ENABLED == 'true'` just | |
| # like the other Agent Shin workflows. The workflow also gates on the | |
| # commenter being either the PR/issue author or an internal collaborator | |
| # (OWNER/MEMBER/COLLABORATOR) so random commenters cannot DOS the LLM | |
| # judge or force a reopen. | |
| on: | |
| issue_comment: | |
| types: [created] | |
| permissions: | |
| contents: read | |
| issues: write | |
| pull-requests: write | |
| jobs: | |
| reconsider: | |
| if: | | |
| github.repository == 'BerriAI/litellm' | |
| && contains(github.event.comment.body, '@agent-shin reconsider') | |
| runs-on: ubuntu-latest | |
| steps: | |
| - name: Authorize commenter | |
| # Only the PR/issue author OR an internal collaborator may trigger | |
| # a reconsider. Outside random commenters could otherwise spam the | |
| # phrase to burn LLM budget or, if a fail-open bug were ever | |
| # introduced, force a reopen on someone else's behalf. | |
| # | |
| # We expose the authorization decision as a step output and gate | |
| # every subsequent (potentially destructive) step on it. A `run:` | |
| # step with `exit 0` would NOT stop the job — only `if:` gating | |
| # on a known-true output is safe here. | |
| id: auth | |
| env: | |
| COMMENTER: ${{ github.event.comment.user.login }} | |
| AUTHOR: ${{ github.event.issue.user.login }} | |
| ASSOCIATION: ${{ github.event.comment.author_association }} | |
| run: | | |
| set -euo pipefail | |
| if [ "${COMMENTER}" = "${AUTHOR}" ]; then | |
| echo "::notice::Authorized: commenter is the PR/issue author." | |
| echo "authorized=true" >> "$GITHUB_OUTPUT" | |
| exit 0 | |
| fi | |
| case "${ASSOCIATION}" in | |
| OWNER|MEMBER|COLLABORATOR) | |
| echo "::notice::Authorized: commenter is an internal collaborator (${ASSOCIATION})." | |
| echo "authorized=true" >> "$GITHUB_OUTPUT" | |
| ;; | |
| *) | |
| echo "::notice::Commenter '${COMMENTER}' (${ASSOCIATION}) is not authorized to trigger reconsider; skipping subsequent steps." | |
| echo "authorized=false" >> "$GITHUB_OUTPUT" | |
| ;; | |
| esac | |
| - name: React 👀 to acknowledge the reconsider | |
| # Add an eyes reaction to the triggering comment the moment we accept | |
| # it, so the contributor gets instant feedback that the bot saw their | |
| # `@agent-shin reconsider` before the slower triage steps run. Gated on | |
| # AGENT_SHIN_ENABLED so dry-run leaves no visible trace. Best-effort: | |
| # a reactions API hiccup must never fail the actual reconsider. | |
| if: steps.auth.outputs.authorized == 'true' && vars.AGENT_SHIN_ENABLED == 'true' | |
| env: | |
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| COMMENT_ID: ${{ github.event.comment.id }} | |
| run: | | |
| set -euo pipefail | |
| gh api --method POST \ | |
| -H "Accept: application/vnd.github+json" \ | |
| "repos/${{ github.repository }}/issues/comments/${COMMENT_ID}/reactions" \ | |
| -f content=eyes \ | |
| || echo "::warning::failed to add 👀 reaction (non-fatal)" | |
| - name: Checkout triage script | |
| if: steps.auth.outputs.authorized == 'true' | |
| uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0 | |
| with: | |
| sparse-checkout: .github/scripts | |
| persist-credentials: false | |
| - name: Set up Python | |
| if: steps.auth.outputs.authorized == 'true' | |
| uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 | |
| with: | |
| python-version: "3.12" | |
| - name: Install LLM client | |
| if: steps.auth.outputs.authorized == 'true' | |
| run: pip install --no-cache-dir --require-hashes -r .github/scripts/triage-requirements.txt | |
| - name: Run Agent Shin reconsider | |
| if: steps.auth.outputs.authorized == 'true' | |
| env: | |
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| # Only expose the LLM key when the bot is enabled, so a PR/issue | |
| # author can't force paid LLM calls by spamming `@agent-shin | |
| # reconsider` while the bot is still in dry-run. The Python script | |
| # calls the LLM whenever this var is set (regardless of `--close`); | |
| # stripping `--close` doesn't suppress the API call, only the | |
| # destructive side effects. Mirror the gating used by every other | |
| # Agent Shin workflow (triage_pr_with_llm.yml, review_gate.yml, ...). | |
| OPENAI_API_KEY: ${{ vars.AGENT_SHIN_ENABLED == 'true' && secrets.OPENAI_API_KEY || '' }} | |
| OPENAI_BASE_URL: ${{ vars.OPENAI_BASE_URL }} | |
| TRIAGE_MODEL: ${{ vars.TRIAGE_MODEL }} | |
| AGENT_SHIN_ENABLED: ${{ vars.AGENT_SHIN_ENABLED }} | |
| # `issue_comment` events fire for both issues and PR comments. | |
| # `issue.pull_request` is set iff this is a PR comment, so we use | |
| # its presence to decide whether to invoke `--pr N` or `--issue N`. | |
| IS_PR: ${{ github.event.issue.pull_request != null }} | |
| NUMBER: ${{ github.event.issue.number }} | |
| run: | | |
| set -euo pipefail | |
| if [ "${IS_PR}" = "true" ]; then | |
| ARGS=(--repo "${{ github.repository }}" --pr "${NUMBER}" --reconsider) | |
| else | |
| ARGS=(--repo "${{ github.repository }}" --issue "${NUMBER}" --reconsider) | |
| fi | |
| # Reconsider's destructive actions (post comment + reopen) are | |
| # gated on `--close`, mirroring the regular triage workflows. | |
| # When AGENT_SHIN_ENABLED is not the EXACT string "true", we | |
| # still run the script so its verdict + would-X action lands in | |
| # the step summary for QA — but without `--close`, the script | |
| # returns `would-reopen` / `would-reconsider-still-failing` | |
| # instead of touching GitHub state. | |
| # | |
| # Use the positive `= "true"` gate (not `!= "true" -> exit`) so | |
| # the workflow guardrails in | |
| # tests/test_litellm/test_github_triage_workflows.py see the | |
| # canonical fail-safe enable pattern. Unknown values like | |
| # "True", "yes", "1", or typos fall through to the dry-run | |
| # branch, which is the safe default. | |
| if [ "${AGENT_SHIN_ENABLED:-false}" = "true" ]; then | |
| ARGS+=(--close) | |
| echo "::notice::Agent Shin reconsider ENABLED — running real triage (close=true)." | |
| else | |
| echo "::notice::AGENT_SHIN_ENABLED is not 'true' -> reconsider stays in dry-run (no comment, no reopen)." | |
| fi | |
| python3 .github/scripts/triage_with_llm.py "${ARGS[@]}" | |
| - name: React 👍 when the reconsider finishes | |
| # Once the reconsider run has completed successfully, add a thumbs-up so | |
| # the contributor sees the bot is done (the 👀 stays, signalling | |
| # seen -> handled). `success()` keeps this from firing if the run | |
| # errored, and the AGENT_SHIN_ENABLED gate keeps dry-run inert. | |
| if: success() && steps.auth.outputs.authorized == 'true' && vars.AGENT_SHIN_ENABLED == 'true' | |
| env: | |
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| COMMENT_ID: ${{ github.event.comment.id }} | |
| run: | | |
| set -euo pipefail | |
| gh api --method POST \ | |
| -H "Accept: application/vnd.github+json" \ | |
| "repos/${{ github.repository }}/issues/comments/${COMMENT_ID}/reactions" \ | |
| -f content=+1 \ | |
| || echo "::warning::failed to add 👍 reaction (non-fatal)" |