Merge remote-tracking branch 'base/main' into sdcg-poc

# Conflicts:
#	be/src/exec/file_scanner/json_scanner.cpp

Author: meegoo

.github/workflows/ai-sr-skills.yml

@@ -230,15 +230,35 @@ jobs:
   # the GitHub REST API (GITHUB_TOKEN injected). --comment applies a strict confidence gate and emits
   # either a sticky-marked briefing or the sentinel "MODULE-RISK: none" (then we post nothing).
   #
-  # SECURITY (pull_request_target): this job injects secrets (GITHUB_TOKEN, CLAUDE_CODE_OAUTH_TOKEN)
-  # into `claude --dangerously-skip-permissions`, which reads attacker-influenceable PR body/diff and
-  # whose output is posted back to the PR. An untrusted author could prompt-inject the model to run
-  # bash / emit secrets (e.g. after the marker) and have them published. So it is GATED to TRUSTED
-  # authors only (author_association OWNER/MEMBER/COLLABORATOR = write access); external / first-time
-  # contributors never reach the model. (assign-reviewer tolerates all authors because its host step
-  # extracts only login-shaped tokens — a strict whitelist; module-risk's output is free-form, so it
-  # needs the author gate. The marker-bounded extraction below is defense-in-depth, not the control.)
-  # For an external PR a maintainer can still run /github-pr-module-risk:review manually.
+  # TRIGGER: every human-authored PR event (opened/reopened/synchronize/ready_for_review), minus
+  # mergify/ and *-sync-pr-* automation branches and Bot-typed authors. It intentionally does NOT
+  # restrict to org members / collaborators — external contributors are briefed too (deliberate; see
+  # the SECURITY note below for the accepted risk).
+  #
+  # Why no trusted-author gate: the prior `author_association in [OWNER,MEMBER,COLLABORATOR]` check
+  # silently skipped EVERY PR. In the pull_request_target payload `author_association` reflects only
+  # PUBLICLY-visible org membership, and StarRocks committers keep membership private, so they arrive
+  # as CONTRIBUTOR and the gate matched nobody (module-risk never ran once since it was added). The
+  # replacement `github.event.pull_request.user.type != 'Bot'` keeps automation (mergify[bot],
+  # dependabot, ...) out while briefing all human PRs.
+  #
+  # SECURITY (pull_request_target): the briefing runs attacker-influenceable PR body/diff through
+  # `claude --dangerously-skip-permissions` with secrets in env, so a malicious PR can prompt-inject
+  # the model to run bash and try to exfiltrate whatever the container holds. To bound the blast
+  # radius the work is SPLIT into two jobs:
+  #   * module-risk (runs the model): its GITHUB_TOKEN is scoped READ-only (job `permissions:` below),
+  #     so an injected run cannot write to the repo or steal a write-capable token. It never posts —
+  #     it only hands the briefing text to the next job via a job output.
+  #   * module-risk-comment (posts the comment): holds the pull-requests:write token, never runs the
+  #     model, and receives the untrusted body through env (never spliced into a command via ${{ }}),
+  #     so an injected body cannot inject shell. The body crosses the job boundary base64-encoded, so
+  #     it also cannot forge the job-output framing.
+  # Residual risk: CLAUDE_CODE_OAUTH_TOKEN still lives in the model container and a determined
+  # injection could exfiltrate it — that is a Claude billing/abuse exposure (rotatable), NOT repo
+  # write or supply-chain. Closing it fully needs container network-egress allowlisting
+  # (anthropic / github / context-base only); tracked separately. (assign-reviewer stays safe for any
+  # author because its host step extracts only login-shaped tokens — a strict whitelist; module-risk's
+  # output is free-form.)
   module-risk:
     runs-on: [self-hosted, normal]
     # Dedup rapid pushes for THIS job only (a new event cancels the in-flight briefing run).
@@ -248,57 +268,85 @@ jobs:
       group: ai-sr-skills-module-risk-${{ github.event.number }}
       cancel-in-progress: true
     continue-on-error: true
+    # READ-only token: this job runs the model on untrusted PR content, so it must NOT hold a
+    # write-capable token (see SECURITY above). The sibling module-risk-comment job does the posting.
+    permissions:
+      contents: read
+      pull-requests: read
     if: >
       (github.event.action == 'opened' || github.event.action == 'reopened' ||
        github.event.action == 'synchronize' || github.event.action == 'ready_for_review') &&
       !startsWith(github.head_ref, 'mergify/') &&
       !contains(github.head_ref, '-sync-pr-') &&
-      contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.pull_request.author_association)
+      github.event.pull_request.user.type != 'Bot'
+    outputs:
+      # base64 of the marker-to-EOF briefing (empty when there is nothing to post). base64 keeps the
+      # untrusted body to a single [A-Za-z0-9+/=] line so it can neither forge the job-output framing
+      # nor inject anything when the comment job consumes it.
+      body: ${{ steps.brief.outputs.body }}
     env:
       GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
       PR_URL: ${{ github.event.pull_request.html_url }}
-      PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
     steps:
       - name: Module-risk briefing (AI, read-only)
+        id: brief
         run: |
           # Namespaced plugin command (a bare /review would NOT resolve in headless -p — it would be
-          # treated as prompt text and silently no-op). tee captures stdout for the upsert step.
+          # treated as prompt text and silently no-op). tee captures stdout for extraction below.
           docker exec -i -u claudeuser \
             -e GITHUB_TOKEN="${GITHUB_TOKEN}" \
             -e CLAUDE_CODE_OAUTH_TOKEN="${CLAUDE_CODE_OAUTH_TOKEN}" \
             claude-cli claude --dangerously-skip-permissions \
             -p "/github-pr-module-risk:review ${PR_URL} --comment" 2>&1 | tee "${RUNNER_TEMP}/module_risk.txt" || true
+          # The skill emits the briefing starting at the marker line, or "MODULE-RISK: none" (no
+          # marker) when nothing clears the confidence gate. Take from the marker to EOF as the body
+          # (empty for no marker / sentinel / error => the comment job posts nothing); sed from the
+          # marker also strips any claude preamble before it. Hand it to the comment job base64-encoded
+          # on a single line (see the `outputs:` note above).
+          out="${RUNNER_TEMP}/module_risk.txt"
+          marker="<!-- module-risk-briefing -->"
+          b64=""
+          [ -f "$out" ] && b64="$(sed -n "/$marker/,\$p" "$out" 2>/dev/null | base64 -w0 || true)"
+          echo "body=${b64}" >> "$GITHUB_OUTPUT"
 
+  module-risk-comment:
+    needs: module-risk
+    runs-on: [self-hosted, normal]
+    continue-on-error: true
+    # Holds the write token and does the posting; it never runs the model, so untrusted PR content
+    # never meets a write-capable token. Skipped when the brief produced no briefing (empty output),
+    # and (via `needs`) when the brief job itself was gated out.
+    permissions:
+      pull-requests: write
+    if: ${{ needs.module-risk.outputs.body != '' }}
+    env:
+      GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      PR: ${{ github.event.number }}
+      REPO: ${{ github.repository }}
+      PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+      # Untrusted (model-produced) body, base64. Passed via env, never spliced into a command via
+      # ${{ }}, so a prompt-injected body cannot inject shell.
+      BODY_B64: ${{ needs.module-risk.outputs.body }}
+    steps:
       - name: Upsert sticky comment
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          PR: ${{ github.event.number }}
-          REPO: ${{ github.repository }}
         run: |
-          out="${RUNNER_TEMP}/module_risk.txt"
           marker="<!-- module-risk-briefing -->"
-          [ -f "$out" ] || { echo "no skill output, skip"; exit 0; }
-          # The skill emits the briefing starting at the marker line, or "MODULE-RISK: none" (no
-          # marker) when nothing clears the confidence gate. Take from the marker to EOF as the body;
-          # empty (no marker / sentinel / error) => post nothing. sed from the marker also strips any
-          # claude preamble before it.
-          body="$(sed -n "/$marker/,\$p" "$out" 2>/dev/null || true)"
+          body="$(printf '%s' "$BODY_B64" | base64 -d 2>/dev/null || true)"
           if [ -z "$body" ]; then
             echo "no high-confidence, diff-aligned module risk — not commenting"
             exit 0
           fi
-          # Stamp the PR head SHA as line 2 (right after the marker, which sed guaranteed is line 1)
-          # so reviewers can detect a briefing that predates the latest push. awk avoids sed escaping
-          # pitfalls; the SHA is hex so it is injection-safe. Applies to both the real briefing and the
-          # incomplete-coverage caveat (both carry the marker as line 1).
+          # Stamp the PR head SHA as line 2 (right after the marker, which the brief job's sed
+          # guaranteed is line 1) so reviewers can detect a briefing that predates the latest push.
+          # awk avoids sed escaping pitfalls; the SHA is hex so it is injection-safe.
           sha_line="<!-- module-risk-head-sha: ${PR_HEAD_SHA} -->"
           printf '%s\n' "$body" \
             | awk -v s="$sha_line" 'NR==1{print; print s; next} {print}' \
             > "${RUNNER_TEMP}/mr_body.md"
           # Sticky: edit our existing marked comment if present, else create one. Idempotent across
-          # reopen / workflow rerun (the marker identifies our comment; gh is host-side, not in the
-          # container). PATCH body is JSON-built with jq so markdown is escaped correctly.
+          # reopen / workflow rerun (the marker identifies our comment). PATCH body is JSON-built with
+          # jq so markdown is escaped correctly.
           existing="$(gh api --paginate "repos/$REPO/issues/$PR/comments" \
                         --jq ".[] | select(.body | contains(\"$marker\")) | .id" 2>/dev/null | head -1 || true)"
           if [ -n "$existing" ]; then

.github/workflows/inspection-pipeline.yml

@@ -1,11 +1,13 @@
 name: INSPECTION PIPELINE
 
 on:
-  schedule:
-    - cron: "0 0 * * 1-5"
-    - cron: "30 4 * * 1-5"
-    - cron: "0 11 * * 1,3,5"
-    - cron: "0 11 * * 2,4"
+  # Scheduled inspection runs disabled 2026-07-08: the runs are not currently
+  # meaningful. Kept here (commented) for easy re-enable. Manual dispatch stays.
+  # schedule:
+  #   - cron: "0 0 * * 1-5"
+  #   - cron: "30 4 * * 1-5"
+  #   - cron: "0 11 * * 1,3,5"
+  #   - cron: "0 11 * * 2,4"
   workflow_dispatch:
     inputs:
       BRANCH:

be/AGENTS.md

@@ -199,15 +199,15 @@ Format module for core format primitives and concrete CSV, JSON, Avro, ORC, and
 - Remediation: Keep format code inside the Format module; move connector orchestration and higher execution policy upward.
 
 ### StoragePrimitive (`storageprimitive`)
-Shared storage-access primitives, predicate contracts, predicate trees, iterator contracts, and value/schema types used by Storage, Exec, Connector, Format, and ComputeEnv without concrete Storage engine, tablet, rowset, lake, service, or full Exec coupling.
+Shared storage-access primitives, predicate contracts, predicate trees, iterator/merge/aggregation helpers, and value/schema types used by Storage, Exec, Connector, Format, and ComputeEnv without concrete Storage engine, tablet, rowset, lake, service, or full Exec coupling.
 - Targets: `StoragePrimitive`
 - Allowed internal include prefixes: `storage_primitive/`, `exprs/`, `runtime/`, `column/`, `types/`, `common/`, `base/`, `gutil/`, `gen_cpp/`
-- Allowed target deps: `Expr`, `Runtime`, `ChunkCore`, `ColumnCore`, `Types`, `Common`, `Base`, `Gutil`, `StarRocksGen`
+- Allowed target deps: `Expr`, `Runtime`, `ColumnSortCore`, `ChunkCore`, `ColumnCore`, `Types`, `Common`, `Base`, `Gutil`, `StarRocksGen`
 - Core tests: `storage_primitive_test`
-- Remediation: Keep StoragePrimitive limited to reusable storage-access contracts, predicate contracts, predicate trees, iterator contracts, and value/schema types; move concrete storage engine, tablet, rowset, lake, service, concrete connector, and full Exec integration upward.
+- Remediation: Keep StoragePrimitive limited to reusable storage-access contracts, predicate contracts, predicate trees, iterator/merge/aggregation helpers, and value/schema types; move concrete storage engine, tablet, rowset, lake, service, concrete connector, and full Exec integration upward.
 
 ### StorageBase (`storagebase`)
-Base storage algorithms and mask-buffer helpers above StoragePrimitive and ComputeEnv without concrete Storage engine, tablet, rowset, lake, service, or full Exec coupling.
+Base storage algorithms above StoragePrimitive and ComputeEnv without concrete Storage engine, tablet, rowset, lake, service, or full Exec coupling.
 - Targets: `StorageBase`
 - Allowed internal include prefixes: `storage/base/`, `storage_primitive/`, `column/`, `types/`, `common/`, `base/`, `gutil/`, `gen_cpp/`
 - Allowed target deps: `ComputeEnv`, `StoragePrimitive`, `Runtime`, `ColumnSortCore`, `ChunkCore`, `ColumnCore`, `Types`, `Common`, `Base`, `Gutil`, `StarRocksGen`
@@ -269,12 +269,12 @@ Operator-tree execution framework for query and fragment contexts, driver lifecy
 - Remediation: Keep ExecRuntime limited to the operator-tree execution framework and runtime behavior that can be expressed through ComputeEnv and ExecPrimitive contracts; move concrete operators, storage, service, connector, cache, HTTP, and broad Exec integration upward.
 
 ### ConnectorPrimitive (`connectorprimitive`)
-Read-side connector contracts, DataSource, and DataSourceProvider default mechanics without concrete connectors, sinks, registry composition, storage, service, or full Exec coupling.
+Connector contracts, DataSource, DataSourceProvider default mechanics, and primitive connector sink commit/profile types without concrete connectors, sinks, registry composition, storage, service, or full Exec coupling.
 - Targets: `ConnectorPrimitive`
-- Allowed internal include prefixes: `connector_primitive/`, `exec_primitive/`, `exprs/`, `runtime/`, `column/`, `types/`, `common/`, `base/`, `gutil/`, `gen_cpp/`
-- Allowed target deps: `ExecPrimitive`, `Expr`, `Runtime`, `ChunkCore`, `ColumnCore`, `Types`, `Common`, `Base`, `Gutil`, `StarRocksGen`
+- Allowed internal include prefixes: `connector_primitive/`, `exec_primitive/`, `exprs/`, `runtime/`, `formats/`, `column/`, `types/`, `common/`, `base/`, `gutil/`, `gen_cpp/`
+- Allowed target deps: `FormatCore`, `ExecPrimitive`, `Expr`, `Runtime`, `ChunkCore`, `ColumnCore`, `Types`, `Common`, `Base`, `Gutil`, `StarRocksGen`
 - Core tests: `connector_primitive_test`
-- Remediation: Keep ConnectorPrimitive limited to read-side connector contracts and default scan-range-to-morsel mechanics; move concrete connectors, sinks, registry wiring, storage, service, and full Exec integration upward.
+- Remediation: Keep ConnectorPrimitive limited to connector contracts, default scan-range-to-morsel mechanics, and primitive sink commit/profile types; move concrete connectors, concrete sinks, registry wiring, storage, service, and full Exec integration upward.
 
 ### ConnectorBuiltinRegistry (`connectorbuiltinregistry`)
 Top-level built-in connector registration composition above connector contracts and concrete connector libraries.

be/CMakeLists.txt

@@ -649,6 +649,7 @@ set(STARROCKS_DEPENDENCIES
     zstd
     streamvbyte
     arrow
+    arrow_testing
     arrow_flight
     arrow_flight_sql
     gRPC::grpc