refactor: migrate Deequ Bot to the shared Shadow engine (#756)

* Migrate Deequ Bot to the shared Shadow engine

Replace the vendored copy of the bot (src/scripts/issue_bot/) with consuming
the shared engine sudsali/shadow via its reusable workflow, SHA-pinned to
3a057597420d7123df162bfe4686aaafb5f7eccd (v1.2). One engine, maintained once
upstream, instead of a per-repo fork.

Zero intended behavior change:
- Deequ's language-tuned prompts stay in Secrets Manager (deequ-bot/* namespace)
  and are selected via prompt_sm_prefix: deequ-bot — the engine fetches all
  eight (pr-investigator/critic/reporter + 2 commit nudges + issue-classify/
  issue-respond/followup) in place of its bundled language-agnostic defaults.
- .shadow.yml sets bot.name: deequ-bot so the clean-review marker stays
  <!-- deequ-bot:clean --> (auto-approve.yml keys on it) and the Scala codebase
  config (src/main/scala, .scala) matches the vendored setup.
- All four surfaces (PR review, issue triage, issue-respond, follow-up), all
  four triggers, and the job-level trigger guard are preserved. Workflow name
  stays "Deequ Bot" (auto-approve.yml keys on it too).

BOT_AGENT_PIPELINE=1 is already the live path, so the dropped legacy file-review
prompts are dead code. generate_kb.py stays (update-kb.yml still runs it); its
stale paths-ignore entries for the removed dirs are cleaned up.

Requires a one-time OIDC trust update on the bot's IAM role (job_workflow_ref →
shadow's reusable workflow), applied out-of-band with an @* version wildcard so
future engine bumps need no IAM change.

* Clarify OIDC trust comment: hardening is optional and post-merge only

The caller works on the current sub-only IAM trust with no change (its OIDC sub
is still repo:awslabs/deequ:*). The job_workflow_ref pin is optional
defense-in-depth to be applied only after this PR merges — applying it while the
base branch still runs the vendored bot denies AssumeRole. Comment-only.

* Add top-level permissions to the Shadow caller workflow

A reusable-workflow caller must grant, at the top level, the union of
permissions its nested jobs request — GitHub caps nested jobs at the caller's
token and otherwise fails at startup ('is requesting ... but is only allowed
none'). The reusable workflow's analyze (id-token:write, pull-requests:read)
and act (issues:write, pull-requests:write) jobs need this union; the vendored
workflow declared these per-job, which doesn't carry across the workflow_call
boundary. Fixes the workflow_dispatch startup_failure.

* Trim workflow/config comments to WHY-only

Drop WHAT-narration and multi-paragraph explanations; keep only comments that
prevent a concrete mistake (name coupling, dual-SHA pin, pull_request_target
no-ref, permissions ceiling, marker, if-guard).

* Rename bot identity deequ-bot -> shadow

Surface the Shadow product to the OSS community. Renames the user-facing
identity in lockstep across the three coupled points: .shadow.yml bot.name
(the clean-review marker), the workflow name, and auto-approve.yml (both the
workflows[] trigger and the CLEAN_MARKER it greps). prompt_sm_prefix stays
deequ-bot (private Secrets Manager namespace).

* Pin engine to v1.3 and set Shadow attribution footer

Bumps the shared-engine SHA to v1.3 (adds bot.attribution) and sets the footer
attribution so posted comments surface the Shadow product to the OSS community.

Author: sudsali

.github/workflows/auto-approve.yml

@@ -2,7 +2,7 @@ name: Auto-Approve Clean PRs
 
 on:
   workflow_run:
-    workflows: ["Java CI with Maven", "Deequ Bot"]
+    workflows: ["Java CI with Maven", "Shadow"]
     types: [completed]
 
 permissions:
@@ -72,7 +72,7 @@ jobs:
               owner, repo, pull_number: prNumber
             });
 
-            const CLEAN_MARKER = '<!-- deequ-bot:clean -->';
+            const CLEAN_MARKER = '<!-- shadow:clean -->';
 
             const latestBot = reviews
               .filter(r => r.user.login === 'github-actions[bot]')

.github/workflows/issue-bot.yml

@@ -1,9 +1,11 @@
-name: Deequ Bot
+name: Shadow   # load-bearing: auto-approve.yml keys on this exact name
+# To upgrade the engine, bump the SHA in BOTH `uses:` and `shadow_ref` below
+# (GitHub forbids expressions in `uses:`, so they can't share a variable).
 
 on:
   issues:
     types: [opened, reopened]
-  pull_request_target: # Runs base branch code with secrets; safe because bot fetches diff via API, never executes PR code. NEVER add ref: to checkout.
+  pull_request_target:  # base-branch checkout only; never add ref: (see SECURITY A1)
     types: [opened, reopened, synchronize]
   issue_comment:
     types: [created]
@@ -17,128 +19,42 @@ on:
         type: boolean
         default: true
 
-# Serialize per issue/PR to prevent duplicate comments
+# Union of the reusable workflow's nested-job permissions; a caller must grant
+# these or the call fails at startup (each nested job still narrows its own set).
+permissions:
+  contents: read
+  id-token: write
+  pull-requests: write
+  issues: write
+
+# inputs.issue_number fallback: workflow_dispatch has no event PR/issue number.
 concurrency:
   group: bot-${{ github.event.issue.number || github.event.pull_request.number || inputs.issue_number }}
   cancel-in-progress: false
 
 jobs:
-  analyze:
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
+  shadow:
+    # Avoid self-triggered loops (bot's own comments) and double-reviewing a PR
+    # from both issue_comment and pull_request_target; dispatch is always manual.
     if: >-
       (github.event_name == 'workflow_dispatch') ||
       (github.actor != 'github-actions[bot]' &&
        (github.event.issue.pull_request == null || github.event_name == 'pull_request_target'))
-    permissions:
-      contents: read
-      id-token: write
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
-        with:
-          persist-credentials: false
-
-      - name: Configure AWS credentials
-        uses: aws-actions/configure-aws-credentials@e3dd6a429d7300a6a4c196c26e071d42e0343502 # v4.0.2
-        with:
-          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
-          aws-region: us-east-1
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Install dependencies
-        run: pip install requests==2.33.1 boto3==1.42.94
-
-      - name: Run analysis
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          GITHUB_REPOSITORY: ${{ github.repository }}
-          ISSUE_NUMBER: ${{ github.event.issue.number || github.event.pull_request.number || inputs.issue_number }}
-          EVENT_TYPE: ${{ github.event_name }}
-          EVENT_ACTION: ${{ github.event.action }}
-          EVENT_BEFORE: ${{ github.event.before }}
-          EVENT_AFTER: ${{ github.event.pull_request.head.sha || github.event.after }}
-          GITHUB_ACTOR: ${{ github.actor }}
-          KB_S3_BUCKET: ${{ secrets.KB_S3_BUCKET }}
-          KB_S3_KEY: ${{ secrets.KB_S3_KEY }}
-          BEDROCK_MODEL_ID: ${{ secrets.BEDROCK_MODEL_ID }}
-          BEDROCK_REPORTER_MODEL_ID: ${{ vars.BEDROCK_REPORTER_MODEL_ID }}
-          BEDROCK_CRITIC_MODEL_ID: ${{ vars.BEDROCK_CRITIC_MODEL_ID }}
-          GUARDRAIL_ID: ${{ secrets.GUARDRAIL_ID }}
-          GUARDRAIL_VERSION: ${{ secrets.GUARDRAIL_VERSION }}
-          SM_ISSUE_CLASSIFY_PROMPT: deequ-bot/issue-classify-prompt
-          SM_ISSUE_RESPOND_PROMPT: deequ-bot/issue-respond-prompt
-          SM_PR_FILE_REVIEW_PROMPT: deequ-bot/pr-file-review-prompt
-          SM_PR_FILE_REVIEW_REPORT_PROMPT: deequ-bot/pr-file-review-report-prompt
-          SM_PR_INVESTIGATOR_PROMPT: deequ-bot/pr-investigator-prompt
-          SM_PR_CRITIC_PROMPT: deequ-bot/pr-critic-prompt
-          SM_PR_REPORTER_PROMPT: deequ-bot/pr-reporter-prompt
-          SM_PR_INVESTIGATOR_COMMIT_PROMPT: deequ-bot/pr-investigator-commit-prompt
-          SM_PR_CRITIC_COMMIT_PROMPT: deequ-bot/pr-critic-commit-prompt
-          SM_FOLLOWUP_PROMPT: deequ-bot/followup-prompt
-          # Flip BOT_AGENT_PIPELINE to "1" to enable the 3-agent (Investigator+Critic+Reporter) pipeline.
-          # When unset/empty, the legacy two-phase flow runs unchanged.
-          # NOTE: Set as a REPOSITORY VARIABLE (Settings → Secrets and variables → Actions → Variables tab),
-          # NOT a secret. Misplacing it under "Secrets" leaves vars.BOT_AGENT_PIPELINE empty → legacy flow
-          # runs silently. To verify: `gh api repos/awslabs/deequ/actions/variables` should list it.
-          BOT_AGENT_PIPELINE: ${{ vars.BOT_AGENT_PIPELINE || '' }}
-          CODEBASE_SRC_DIR: src/main/scala
-          CODEBASE_FILE_EXT: .scala
-          DRY_RUN: ${{ inputs.dry_run || 'false' }}
-          ARTIFACT_PATH: ${{ runner.temp }}/bot_result.json
-        run: python -m issue_bot.main analyze
-        working-directory: src/scripts
-
-      - name: Upload artifact
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
-        with:
-          name: bot-result
-          path: ${{ runner.temp }}/bot_result.json
-          retention-days: 30
-
-  act:
-    runs-on: ubuntu-latest
-    timeout-minutes: 1
-    needs: analyze
-    permissions:
-      contents: read
-      issues: write
-      pull-requests: write
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
-        with:
-          persist-credentials: false
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
-        with:
-          python-version: "3.12"
-
-      - name: Install dependencies
-        run: pip install requests==2.33.1 boto3==1.42.94
-
-      - name: Download artifact
-        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
-        with:
-          name: bot-result
-          path: ${{ runner.temp }}
-
-      - name: Execute actions
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          GITHUB_REPOSITORY: ${{ github.repository }}
-          ISSUE_NUMBER: ${{ github.event.issue.number || github.event.pull_request.number || inputs.issue_number }}
-          EVENT_TYPE: ${{ github.event_name }}
-          EVENT_ACTION: ${{ github.event.action }}
-          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
-          DRY_RUN: ${{ inputs.dry_run || 'false' }}
-          ARTIFACT_PATH: ${{ runner.temp }}/bot_result.json
-        run: python -m issue_bot.main act
-        working-directory: src/scripts
+    uses: sudsali/shadow/.github/workflows/shadow-review.yml@0ff3d759e3860fe5439682f51b6cd6352bf271c3
+    with:
+      pr_number: ${{ inputs.issue_number }}
+      dry_run: ${{ inputs.dry_run && 'true' || 'false' }}
+      shadow_ref: 0ff3d759e3860fe5439682f51b6cd6352bf271c3
+      aws_region: us-east-1
+      prompt_sm_prefix: deequ-bot
+    secrets:
+      AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }}
+      GUARDRAIL_ID: ${{ secrets.GUARDRAIL_ID }}
+      GUARDRAIL_VERSION: ${{ secrets.GUARDRAIL_VERSION }}
+      KB_S3_BUCKET: ${{ secrets.KB_S3_BUCKET }}
+      KB_S3_KEY: ${{ secrets.KB_S3_KEY }}
+      BEDROCK_MODEL_ID: ${{ secrets.BEDROCK_MODEL_ID }}
+      # Repo variables (not secrets); empty falls back to engine defaults.
+      BEDROCK_REPORTER_MODEL_ID: ${{ vars.BEDROCK_REPORTER_MODEL_ID }}
+      BEDROCK_CRITIC_MODEL_ID: ${{ vars.BEDROCK_CRITIC_MODEL_ID }}
+      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

.github/workflows/update-kb.yml

@@ -5,8 +5,6 @@ on:
     branches: [master]
     paths-ignore:
       - '.github/workflows/**'
-      - 'src/scripts/issue_bot/**'
-      - 'src/scripts/tests/**'
   workflow_dispatch:
 
 jobs:

.shadow.yml

@@ -0,0 +1,19 @@
+# Shadow engine config for awslabs/deequ.
+
+codebase:
+  src_dir: src/main/scala
+  file_ext: .scala
+  test_dir: src/test/scala
+  language: scala
+
+bot:
+  name: shadow   # renders the <!-- shadow:clean --> marker auto-approve.yml greps for
+  attribution: "Reviewed by Shadow · github.com/sudsali/shadow"
+  escalate_label: needs-human
+  max_replies: 2
+  max_runs_per_hour: 20
+
+models:
+  investigator: us.anthropic.claude-opus-4-7
+  critic: us.anthropic.claude-opus-4-7
+  reporter: us.anthropic.claude-haiku-4-5-20251001-v1:0