fix(cpu): detect PCLMULQDQ support to prevent SIGILL on Hygon/QEMU#934
fix(cpu): detect PCLMULQDQ support to prevent SIGILL on Hygon/QEMU#934liuq19 wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #934 +/- ##
==========================================
- Coverage 51.86% 51.27% -0.60%
==========================================
Files 127 172 +45
Lines 10893 14149 +3256
==========================================
+ Hits 5650 7255 +1605
- Misses 4920 6503 +1583
- Partials 323 391 +68
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds PCLMULQDQ (CLMUL) CPU feature detection and uses it to fail fast on amd64 CPUs/VMs that would otherwise hit illegal SIMD instructions during native dispatch, preventing SIGILL crashes in Sonic’s SIMD JSON parsing path.
Changes:
- Add
HasPCLMULQDQfeature flag viacpuid.CLMULininternal/cpu. - Guard
internal/nativeamd64 dispatch with a PCLMULQDQ check and panic with an actionable message when missing.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| internal/cpu/features.go | Introduces HasPCLMULQDQ based on CPUID CLMUL support. |
| internal/native/dispatch_amd64.go | Adds an init-time check to prevent dispatching into SIMD code paths that can SIGILL without PCLMULQDQ. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Both AVX2 and SSE native code paths previously used PCLMULQDQ (carry-less multiplication) for computing the in-quote bitmask in the SIMD JSON string scanner. This caused SIGILL crashes on CPUs or VMs that lack PCLMULQDQ, such as Hygon Dhyana under QEMU/KVM. Changes: - native/scanning.h: gate `_mm_clmulepi64_si128` behind `#ifdef USE_PCLMUL`; provide scalar prefix-XOR fallback for the SSE path - scripts/build-x86.sh: SSE compiles with `-mno-pclmul`; AVX2 compiles with `-mpclmul -DUSE_PCLMUL=1` - Regenerate all SSE native code (no PCLMULQDQ instructions) - internal/cpu/features.go: add `HasPCLMULQDQ` detection via `cpuid.CLMUL` - internal/native/dispatch_amd64.go: AVX2 path now requires both `HasAVX2 && HasPCLMULQDQ`; SSE path works without PCLMULQDQ Dispatch logic: HasAVX2 + HasPCLMULQDQ → AVX2 (with VPCLMULQDQ) HasSSE (no PCLMULQDQ) → SSE (scalar prefix-XOR fallback) Fixes bytedance#867 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
86bdeb2 to
54a512c
Compare
- SONIC_MODE=noavx/noavx2 now also sets HasPCLMULQDQ=false - Add TestNoPCLMULQDQ_SubProcess: verifies CPU flags are correctly disabled under SONIC_MODE=noavx - Add TestNoPCLMULQDQ_JSON: end-to-end test running JSON marshal, unmarshal, GetByPath, and validation through the SSE path (scalar prefix-XOR) in a subprocess with SONIC_MODE=noavx Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Summary
HasPCLMULQDQruntime detection; AVX2 path requires it, SSE path does notBackground
Both AVX2 and SSE native code paths used
PCLMULQDQ/VPCLMULQDQinstructions for carry-less multiplication in the SIMD JSON string scanner (native/scanning.h:_mm_clmulepi64_si128). The CPU feature detection only checked for AVX2 and SSE — not PCLMULQDQ.This caused
SIGILLcrashes on:The previous workaround
SONIC_MODE=noavx2(suggested in #867) was insufficient because the SSE fallback path also usedPCLMULQDQ(non-VEX encoded66 0F 3A 44).Changes
native/scanning.h_mm_clmulepi64_si128behind#ifdef USE_PCLMUL; scalar prefix-XOR fallbackscripts/build-x86.sh-mno-pclmul; AVX2:-mpclmul -DUSE_PCLMUL=1internal/native/sse/*internal/cpu/features.goHasPCLMULQDQ = cpuid.CPU.Has(cpuid.CLMUL)internal/native/dispatch_amd64.goHasAVX2 && HasPCLMULQDQ; SSE works withoutDispatch logic
The scalar prefix-XOR is mathematically equivalent to
clmul(x, 0xFFFFFFFFFFFFFFFF):Fixes #867
Test plan
go build ./...passesgo test ./... -count=1SONIC_MODE=noavx go test ./... -count=1pclmulqdqinstructionsvpclmulqdq(unchanged)🤖 Generated with Claude Code