Skip to content
Daily Sanitized Differential Fuzz

Daily Sanitized Differential Fuzz #41

Sign in to view logs

Daily Sanitized Differential Fuzz

Daily Sanitized Differential Fuzz #41

Workflow file for this run

.github/workflows/daily-sanitized-fuzz.yml at 9b52773
name: Daily Sanitized Differential Fuzz
# Runs the differential fuzz suite with pine-cpp built under sanitizers so
# data races / UAFs / heap-overflows surface at the moment they occur, not
# after a Release-mode divergence triggers a reactive 40x TSan replay (see
# nightly-diff-fuzz.yml lines 152-209).
#
# Motivation: issues #109 / #124 showed two mirrored go-vs-cpp divergences
# that could not be reproduced locally, only surfaced under CI runner
# pressure. A reactive replay sees them only after the rare window already
# opened; running the main fuzz with -fsanitize=thread / address makes the
# first hit produce a full stack instead of `cpp_rc=1` with no diagnostic.
#
# Cost: ASan ~3x slower, TSan ~5-10x slower than Release. Initial
# smoke-run measurements (#128, 100/50 rounds) projected ~1.4 s/round and
# 5000+3500 fitting in ~205m. The 2026-06-20 / 06-21 production runs
# (#132 / #133) tripped exit=124 on both passes — throughput was cut to
# 3000 ASan + 2000 TSan (570b4312). Production runs 06-24..07-04
# (#142 / #144 / #146-152) then showed worst-case throughput is really
# ~2.0 s/round (ASan) / ~1.9 s/round (TSan) on slow runner days — 55 %
# above the fast-day ~1.3 s/round — so ASan tripped its 90m timeout on
# 8 of 10 days with 5-11 % of rounds left, losing the whole pass each
# time. Two-layer fix:
# 1. --time-budget-seconds makes the fuzz script stop launching new
# rounds when the budget is exhausted and emit its normal Results:
# summary — a slow runner now degrades to fewer completed rounds
# instead of losing the pass to a timeout kill.
# 2. Budgets re-derived from worst measured throughput: ASan 3000
# rounds × 2.03 s = ~102m → budget 105m; TSan 2000 × 1.92 s = ~64m
# → budget 75m. The outer `timeout` stays as a last-resort kill
# ~5m above each budget (hang protection, not pacing).
# Nightly diff-fuzz keeps driving 10k Release rounds for raw
# throughput / different seed coverage; this workflow is the
# deep-diagnostic complement, NOT a replacement.
on:
schedule:
- cron: "0 4 * * *" # 12:00 UTC+8 daily
workflow_dispatch:
inputs:
asan-rounds:
description: "ASan fuzz rounds (cpp ASan main run)"
default: "3000"
tsan-rounds:
description: "TSan fuzz rounds (smaller — TSan is 5-10x slower)"
default: "2000"
stability-runs:
description: "Stability check reruns per case"
default: "3"
concurrency:
group: daily-sanitized-fuzz-${{ github.ref }}
cancel-in-progress: false
# Same rationale as nightly-diff-fuzz.yml: issues:write so the "open issue
# on divergence" steps can succeed; default token is read-only.
permissions:
contents: read
issues: write
jobs:
sanitized-fuzz:
runs-on: ubuntu-latest
# Hard budget: setup/builds ~7m + ASan step ≤115m + TSan step ≤85m +
# evaluate/packaging ~5m ≈ 212m worst. 215m covers that path; anything
# beyond it means multiple layers of protection already failed.
timeout-minutes: 215
steps:
- uses: actions/checkout@v6
- uses: actions/setup-go@v6
with:
go-version-file: pine-go/go.mod
cache-dependency-path: pine-go/go.sum
- uses: actions/setup-python@v6
with:
python-version: "3.13"
- uses: actions/setup-java@v5
with:
distribution: temurin
java-version: "25"
cache: maven
- name: Install C++ build deps
# ci-apt-install.sh retries with a per-attempt timeout so one slow
# mirror rotation doesn't burn the whole budget (#125, #164). cmake /
# g++-13 / make are preinstalled on the runner image — assert them
# instead of apt-installing shadowed duplicates.
run: |
bash scripts/ci-apt-install.sh libluajit-5.1-dev libcurl4-openssl-dev
cmake --version | head -1
g++-13 --version | head -1
- name: Build Go binary
run: go build -o pineapple-run ./cmd/pineapple-run/
working-directory: pine-go
- name: Build Java engine
run: mvn package -B -q -DskipTests
working-directory: pine-java
- name: Build C++ ASan binary
# Address+UB sanitizer build for the main fuzz pass. Catches
# heap-buffer-overflow / use-after-free / use-after-return / UB
# at the moment they happen, not after a Release divergence
# reactivates the rare path. jemalloc must be off (ASan replaces
# the allocator). -O1 keeps debug info readable in stacks without
# killing inliner-dependent hot paths the fuzz exercises.
run: |
cmake -S pine-cpp -B pine-cpp/build-asan \
-DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_CXX_FLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -O1 -g" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address,undefined" \
-DPINE_USE_JEMALLOC=OFF
cmake --build pine-cpp/build-asan --target pineapple-run -j2
- name: Build C++ TSan binary
# Thread sanitizer build for the smaller second pass. ASan and TSan
# are mutually exclusive (both rewrite allocator + instrumentation),
# so we run them as separate fuzz passes.
run: |
cmake -S pine-cpp -B pine-cpp/build-tsan \
-DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_CXX_FLAGS="-fsanitize=thread -fno-omit-frame-pointer -O1 -g" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=thread" \
-DPINE_USE_JEMALLOC=OFF
cmake --build pine-cpp/build-tsan --target pineapple-run -j2
- name: Codegen schema parity gate
# nightly-diff-fuzz also runs this as a low-cost guard against
# silent three-way codegen drift between PR runs. _prebuild.sh
# builds pine-cpp/build/ Release binaries as a side effect; we
# don't use those (we want sanitized binaries above), but the
# schema diff itself is still worth running.
run: bash scripts/cross-validate.sh 1
- name: Run ASan differential fuzz
# Main fuzz pass: cpp uses ASan binary so heap UAF / overflow /
# UB surfaces directly. Worst measured throughput (06-24..07-04,
# #142 et al.) is ~2.03 s/round → 3000 rounds ≈ 102m. The script's
# own --time-budget-seconds (105m) is the pacing mechanism: on a
# slow runner it stops launching rounds and still emits Results:,
# so partial coverage stays a valid signal. The outer `timeout`
# (110m) and step timeout (115m) are hang protection only — if
# they fire, the run is genuinely wedged, not just slow.
timeout-minutes: 115
env:
ASAN_OPTIONS: "halt_on_error=0 detect_leaks=0 abort_on_error=0 print_stacktrace=1"
UBSAN_OPTIONS: "halt_on_error=0 print_stacktrace=1"
run: |
set -o pipefail
ROUNDS="${{ inputs.asan-rounds || '3000' }}"
rc=0
timeout 110m python3 scripts/differential-fuzz.py \
--rounds "$ROUNDS" \
--stability-runs ${{ inputs.stability-runs || '3' }} \
--time-budget-seconds 6300 \
--go-bin pine-go/pineapple-run \
--cpp-bin pine-cpp/build-asan/pineapple-run \
--engines go,java,cpp \
--save-dir /tmp/diff-fuzz-asan \
| tee asan-fuzz-output.log || rc=$?
echo "$rc" > /tmp/asan-fuzz-exit-code
# halt_on_error=0 lets ASan keep dumping; do NOT exit non-zero
# on the first divergence so the TSan pass below also runs.
exit 0
- name: Run TSan differential fuzz
# Race-only pass. Worst measured throughput (07-03) is ~1.92
# s/round → 2000 rounds ≈ 64m, which had shaved the old 65m inner
# timeout to a <1m margin. Same two-layer scheme as ASan:
# --time-budget-seconds (75m) paces, outer `timeout` (80m) and
# step timeout (85m) only catch genuine hangs. Distinct save
# dir so the two passes don't share case numbering (and so
# packaging can pick them apart).
timeout-minutes: 85
env:
# halt_on_error=0: keep going so one race doesn't kill the sweep.
# second_deadlock_stack=1: surface both sides of a lock-order race.
TSAN_OPTIONS: "halt_on_error=0 second_deadlock_stack=1"
run: |
set -o pipefail
ROUNDS="${{ inputs.tsan-rounds || '2000' }}"
rc=0
timeout 80m python3 scripts/differential-fuzz.py \
--rounds "$ROUNDS" \
--stability-runs ${{ inputs.stability-runs || '3' }} \
--time-budget-seconds 4500 \
--go-bin pine-go/pineapple-run \
--cpp-bin pine-cpp/build-tsan/pineapple-run \
--engines go,java,cpp \
--save-dir /tmp/diff-fuzz-tsan \
| tee tsan-fuzz-output.log || rc=$?
echo "$rc" > /tmp/tsan-fuzz-exit-code
exit 0
- name: Evaluate results
if: always()
id: evaluate
run: |
parse_count() {
local pattern=$1
local file=$2
local val
val=$(grep -oP "$pattern" "$file" 2>/dev/null | tail -1 || echo 0)
echo "${val:-0}"
}
ASAN_FAIL=$(parse_count 'FAIL: +\K[0-9]+' asan-fuzz-output.log)
ASAN_UNSTABLE=$(parse_count 'UNSTABLE: +\K[0-9]+' asan-fuzz-output.log)
TSAN_FAIL=$(parse_count 'FAIL: +\K[0-9]+' tsan-fuzz-output.log)
TSAN_UNSTABLE=$(parse_count 'UNSTABLE: +\K[0-9]+' tsan-fuzz-output.log)
ASAN_HAS_RESULTS=$(grep -c '^Results:' asan-fuzz-output.log 2>/dev/null || true)
TSAN_HAS_RESULTS=$(grep -c '^Results:' tsan-fuzz-output.log 2>/dev/null || true)
ASAN_HAS_RESULTS=${ASAN_HAS_RESULTS:-0}
TSAN_HAS_RESULTS=${TSAN_HAS_RESULTS:-0}
ASAN_EXIT=$(cat /tmp/asan-fuzz-exit-code 2>/dev/null || echo 1)
TSAN_EXIT=$(cat /tmp/tsan-fuzz-exit-code 2>/dev/null || echo 1)
# Actual rounds from the Results: line. A budget-stopped pass
# reports "Results: N/M rounds (time budget), seed=..."; a full
# pass reports "Results: M rounds, seed=...". Show what really
# ran so partial coverage is visible in the summary table.
parse_rounds() {
local file=$1 planned=$2 val
val=$(grep -oP '^Results: \K[0-9/]+(?= rounds)' "$file" 2>/dev/null | tail -1 || true)
echo "${val:-$planned}"
}
ASAN_ROUNDS=$(parse_rounds asan-fuzz-output.log "${{ inputs.asan-rounds || '3000' }}")
TSAN_ROUNDS=$(parse_rounds tsan-fuzz-output.log "${{ inputs.tsan-rounds || '2000' }}")
TOTAL_FAIL=$((ASAN_FAIL + TSAN_FAIL))
TOTAL_UNSTABLE=$((ASAN_UNSTABLE + TSAN_UNSTABLE))
# Per-pass status. "incomplete" = no Results: summary line. With
# the in-script time budget, a slow runner stops early and still
# prints Results: (rounds shows N/M) — so "incomplete" now only
# means the outer `timeout` killed a genuinely wedged process or
# the script crashed mid-run. Without this distinction a TSan
# kill would silently report as TSAN_FAIL=0 / TSAN_UNSTABLE=0
# — exactly the diagnostic signal #109 needs us to preserve.
asan_status=$([[ "$ASAN_HAS_RESULTS" -gt 0 ]] && echo "complete" || echo "incomplete")
tsan_status=$([[ "$TSAN_HAS_RESULTS" -gt 0 ]] && echo "complete" || echo "incomplete")
{
echo "### Daily Sanitized Fuzz Results"
echo ""
echo "| Pass | rounds | status | FAIL | UNSTABLE | exit |"
echo "|---|---|---|---|---|---|"
echo "| ASan | $ASAN_ROUNDS | $asan_status | $ASAN_FAIL | $ASAN_UNSTABLE | $ASAN_EXIT |"
echo "| TSan | $TSAN_ROUNDS | $tsan_status | $TSAN_FAIL | $TSAN_UNSTABLE | $TSAN_EXIT |"
echo ""
if [[ "$asan_status" == "incomplete" ]]; then
echo "⚠️ **ASan pass did not produce a Results: summary** (exit=$ASAN_EXIT) — the outer timeout killed a wedged process, or the script crashed. ASan findings, if any, are lost."
echo ""
fi
if [[ "$tsan_status" == "incomplete" ]]; then
echo "⚠️ **TSan pass did not produce a Results: summary** (exit=$TSAN_EXIT) — the outer timeout killed a wedged process, or the script crashed. TSan findings, if any, are lost; #109 race diagnosis is degraded for this run."
echo ""
fi
echo "#### ASan tail"
echo '```text'
tail -30 asan-fuzz-output.log 2>/dev/null || echo "(no asan fuzz output captured)"
echo '```'
echo ""
echo "#### TSan tail"
echo '```text'
tail -30 tsan-fuzz-output.log 2>/dev/null || echo "(no tsan fuzz output captured)"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
# Three orthogonal output signals so downstream steps can pick:
# has_divergences — at least one pass surfaced FAIL/UNSTABLE; gates issue-on-divergence
# has_incomplete — at least one pass failed to produce a Results: summary; gates issue-on-incomplete
# Both can be true (e.g. ASan found a divergence and TSan timed out).
has_divergences=false
if [[ "$TOTAL_FAIL" -gt 0 || "$TOTAL_UNSTABLE" -gt 0 ]]; then
has_divergences=true
fi
has_incomplete=false
if [[ "$asan_status" == "incomplete" || "$tsan_status" == "incomplete" ]]; then
has_incomplete=true
fi
echo "has_divergences=$has_divergences" >> "$GITHUB_OUTPUT"
echo "has_incomplete=$has_incomplete" >> "$GITHUB_OUTPUT"
echo "asan_status=$asan_status" >> "$GITHUB_OUTPUT"
echo "tsan_status=$tsan_status" >> "$GITHUB_OUTPUT"
echo "asan_fail=$ASAN_FAIL" >> "$GITHUB_OUTPUT"
echo "asan_unstable=$ASAN_UNSTABLE" >> "$GITHUB_OUTPUT"
echo "tsan_fail=$TSAN_FAIL" >> "$GITHUB_OUTPUT"
echo "tsan_unstable=$TSAN_UNSTABLE" >> "$GITHUB_OUTPUT"
echo "asan_exit=$ASAN_EXIT" >> "$GITHUB_OUTPUT"
echo "tsan_exit=$TSAN_EXIT" >> "$GITHUB_OUTPUT"
- name: Package divergences
if: ${{ !cancelled() && steps.evaluate.outputs.has_divergences == 'true' }}
run: |
cd /tmp
tar czf /tmp/daily-sanitized-divergences.tar.gz \
diff-fuzz-asan/divergence_* diff-fuzz-asan/unstable_* \
diff-fuzz-tsan/divergence_* diff-fuzz-tsan/unstable_* \
2>/dev/null || true
- name: Upload divergences
if: ${{ !cancelled() && steps.evaluate.outputs.has_divergences == 'true' }}
uses: actions/upload-artifact@v7
with:
name: daily-sanitized-divergences-${{ github.run_number }}
path: /tmp/daily-sanitized-divergences.tar.gz
retention-days: 28
- name: Open issue on divergence
if: ${{ !cancelled() && steps.evaluate.outputs.has_divergences == 'true' }}
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
ASAN_FAIL="${{ steps.evaluate.outputs.asan_fail || '0' }}"
ASAN_UNSTABLE="${{ steps.evaluate.outputs.asan_unstable || '0' }}"
TSAN_FAIL="${{ steps.evaluate.outputs.tsan_fail || '0' }}"
TSAN_UNSTABLE="${{ steps.evaluate.outputs.tsan_unstable || '0' }}"
RUN_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
ARTIFACT_URL="${RUN_URL}#artifacts"
gh issue create \
--title "Daily sanitized-fuzz: ASan=${ASAN_FAIL}f/${ASAN_UNSTABLE}u TSan=${TSAN_FAIL}f/${TSAN_UNSTABLE}u ($(date -u +%Y-%m-%d))" \
--label "bug,fuzz,sanitized" \
--body "$(cat <<EOF
## Daily Sanitized Differential Fuzz Divergence
Sanitizer-instrumented fuzz pass surfaced divergence(s). Unlike
the nightly Release pass which only sees \`cpp_rc!=0\` on a
divergence, the sanitizer build attached to each case should
carry full UAF / heap-overflow / race stacks at the point of
first occurrence.
| | |
|---|---|
| **Run** | ${RUN_URL} |
| **Date** | $(date -u +%Y-%m-%d) |
| **ASan FAIL / UNSTABLE** | ${ASAN_FAIL} / ${ASAN_UNSTABLE} |
| **TSan FAIL / UNSTABLE** | ${TSAN_FAIL} / ${TSAN_UNSTABLE} |
| **Artifact** | [daily-sanitized-divergences-${{ github.run_number }}](${ARTIFACT_URL}) |
### Reproduce locally
1. Download and extract the artifact (note: contains two subtrees,
\`diff-fuzz-asan/\` and \`diff-fuzz-tsan/\`):
\`\`\`bash
tar xzf daily-sanitized-divergences.tar.gz
cd diff-fuzz-{asan,tsan}/divergence_NNNNNN/ # pick a case
\`\`\`
2. Run each engine. The artifact case directory's \`cpp_output.json\`
was produced by the *sanitized* binary; for full re-run you can
use the same sanitizer build locally:
\`\`\`bash
# Build sanitized C++ binary (mirrors the workflow's recipe)
cmake -S pine-cpp -B pine-cpp/build-asan \\
-DCMAKE_BUILD_TYPE=Debug \\
-DCMAKE_CXX_FLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -O1 -g" \\
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address,undefined" \\
-DPINE_USE_JEMALLOC=OFF
cmake --build pine-cpp/build-asan --target pineapple-run -j
# Run cpp with ASan
ASAN_OPTIONS="halt_on_error=0 detect_leaks=0 print_stacktrace=1" \\
pine-cpp/build-asan/pineapple-run \\
-config config.json -request request.json
\`\`\`
3. Per-case files mirror nightly:
- \`config.json\` / \`request.json\` — inputs
- \`info.txt\` — which engines diverged
- \`{go,java,cpp}_output.json\` — actual outputs
(sanitizer stack will appear in the cpp run's stderr, not in
the saved \`cpp_output.json\` — check the workflow log or
re-run locally per step 2.)
### ASan tail
\`\`\`text
$(tail -30 asan-fuzz-output.log 2>/dev/null || echo "(no asan output)")
\`\`\`
### TSan tail
\`\`\`text
$(tail -30 tsan-fuzz-output.log 2>/dev/null || echo "(no tsan output)")
\`\`\`
### Relationship to #109
This workflow was introduced to surface the race / non-deterministic
path implicated by #109 / #124. If sanitizer stacks appear in this
report, link them on #109 — they are the first concrete diagnostic
beyond \`cpp_rc=1\` for the mirrored CI-only divergences.
EOF
)"
- name: Open issue on incomplete pass
# Fires when at least one pass did not produce a Results: summary,
# regardless of whether the other pass found divergences. With the
# in-script time budget, slow runners degrade to partial coverage
# WITH a Results: line — so this now only fires on a genuinely
# wedged process (outer timeout kill) or a script crash. Without
# this, such a kill would be silently swallowed because
# TSAN_FAIL=TSAN_UNSTABLE=0 from an empty log indistinguishably
# means "clean run" or "didn't run". That matters most for the
# TSan pass — its whole purpose is the #109 race signal.
if: ${{ !cancelled() && steps.evaluate.outputs.has_incomplete == 'true' }}
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
RUN_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
ASAN_STATUS="${{ steps.evaluate.outputs.asan_status }}"
TSAN_STATUS="${{ steps.evaluate.outputs.tsan_status }}"
ASAN_EXIT="${{ steps.evaluate.outputs.asan_exit }}"
TSAN_EXIT="${{ steps.evaluate.outputs.tsan_exit }}"
gh issue create \
--title "Daily sanitized-fuzz: pass incomplete (ASan=$ASAN_STATUS TSan=$TSAN_STATUS) ($(date -u +%Y-%m-%d))" \
--label "bug,infra,fuzz" \
--body "$(cat <<EOF
## Daily Sanitized Fuzz Pass Incomplete
At least one fuzz pass did not produce a \`Results:\` summary
line. The fuzz script paces itself with --time-budget-seconds
and emits a Results: summary even when it stops early on a slow
runner — so a missing summary means the outer \`timeout\` killed
a genuinely wedged process, or the script crashed mid-run. Any
sanitizer findings from the incomplete pass were lost — the run
cannot claim coverage of that sanitizer's surface area.
This matters for #109: the TSan pass is the primary race
diagnostic; if it was killed, the run produced no race signal
regardless of what ASan reported.
| | |
|---|---|
| **Run** | ${RUN_URL} |
| **Date** | $(date -u +%Y-%m-%d) |
| **ASan status / exit** | $ASAN_STATUS / $ASAN_EXIT |
| **TSan status / exit** | $TSAN_STATUS / $TSAN_EXIT |
### Suggested triage
Since the --time-budget-seconds mechanism (ASan 105m / TSan
75m), a slow runner no longer produces an incomplete pass — the
script stops early and still emits Results:. An incomplete pass
now means one of:
- exit=124: the outer \`timeout\` (ASan 110m, TSan 80m) killed a
genuinely wedged process — a single round hung (engine
deadlock / subprocess leak), or the script itself stalled
after the budget check. Look at the last progress line in the
step log to find the wedged round, then reproduce with its
seed.
- exit≠124: script crash; check the workflow logs for the
actual failure step / traceback.
Re-run via workflow_dispatch with smaller \`asan-rounds\` /
\`tsan-rounds\` to bisect.
EOF
)"
- name: Open issue on workflow failure
if: ${{ (failure() || cancelled()) && steps.evaluate.outputs.has_divergences != 'true' && steps.evaluate.outputs.has_incomplete != 'true' }}
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
RUN_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
gh issue create \
--title "Daily sanitized-fuzz: workflow failed ($(date -u +%Y-%m-%d))" \
--label "bug,infra" \
--body "$(cat <<EOF
## Daily Sanitized Fuzz Workflow Failure
The daily sanitized-fuzz workflow failed due to an infrastructure
error (build failure, dependency issue, or script crash). This is
NOT a sanitizer divergence — see the workflow logs for the actual
failure step.
| | |
|---|---|
| **Run** | ${RUN_URL} |
| **Date** | $(date -u +%Y-%m-%d) |
Check the [workflow logs](${RUN_URL}) for details.
EOF
)"