Add option to enable PCRE2 JIT for regex subsignature matching - #1749
Add option to enable PCRE2 JIT for regex subsignature matching#1749j20120307 wants to merge 1 commit into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a710fc1348
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| * is left interpreter-only and matching proceeds unchanged. Disabled by | ||
| * default; controlled by the PCREJit / --pcre-jit option. */ | ||
| if (try_jit) { | ||
| int jitrc = pcre2_jit_compile(pd->re, PCRE2_JIT_COMPLETE); |
There was a problem hiding this comment.
When PCREJit/--pcre-jit is enabled, this compiles every pattern for JIT even though the match context just set PCRERecMatchLimit via pcre2_set_recursion_limit(). PCRE2 documents that the depth/recursion limit is ignored for JIT matches (https://man7.org/linux/man-pages/man3/pcre2api.3.html), so signatures whose recursive backtracking would previously stop with the configured recmatch limit can now continue and match/run until another limit is hit; this changes the advertised detection/resource-bound behavior for deployments that tune PCRERecMatchLimit. Please avoid JIT for matches where the recmatch limit must be enforced, or otherwise force the interpreter for those cases.
Useful? React with 👍 / 👎.
ClamAV compiles regex subsignatures with pcre2_compile() and matches them with the interpreted pcre2_match(), and never enables the PCRE2 JIT compiler even when the linked PCRE2 library supports it. For scans where the Aho-Corasick prefilter promotes many PCRE subsignatures over a large buffer, the interpreted matcher dominates scan time. Add a 'PCREJit' clamd.conf option and a '--pcre-jit' clamscan option, exposed through a new CL_ENGINE_PCRE_JIT engine setting, mirroring how the existing PCRE limit options are plumbed (engine field, set/get, settings copy/apply). The option is disabled by default to preserve existing behavior on upgrade. When enabled, cli_pcre_compile() performs a best-effort pcre2_jit_compile() after setting the match limits. pcre2_match() automatically dispatches to the JIT code when a pattern has been JIT-compiled, so no change to the match call is required for the common path. If JIT compilation fails (PCRE2 built without JIT, or an unsupported construct), the failure is logged at debug level and matching falls back to the interpreter. To guarantee identical results to the interpreter when JIT is enabled, cli_pcre_match() now retries a match with PCRE2_NO_JIT if the JIT matcher returns PCRE2_ERROR_JIT_STACKLIMIT (the JIT uses a fixed stack and does not grow on demand like the interpreter). On a workload that promotes several hundred PCRE subsignatures over a ~37 MiB buffer, enabling the option reduced the PCRE phase roughly 3-4x and overall scan time by about 2x, with no change in detection.
a710fc1 to
01dc512
Compare
|
Thanks — good catch. Confirmed against the PCRE2 docs: Worth noting the overall Since this is an opt-in feature (disabled by default), I've documented the tradeoff rather than changing the match path: the caveat is now in the Happy to instead force the interpreter when |
|
Independent verification of this patch, FWIW — evaluating it for a production Setup: applied cleanly against Benchmark: real 74.6MB PPTX (not synthetic/random bytes — file structure
~2.75x, consistent across 3 separate runs (different content each time, to Detection verified identical with the option on/off: EICAR detected correctly Happy to share more detail if useful for review. |
2 participants
Add option to enable PCRE2 JIT for regex subsignature matching
Summary
cli_pcre_compile()compiles regex subsignatures withpcre2_compile()andcli_pcre_match()runs them with the interpretedpcre2_match(). The PCRE2JIT compiler is never enabled, even when the linked PCRE2 library is built
with JIT support.
This PR adds an opt-in option to JIT-compile PCRE subsignatures:
PCREJitinclamd.conf--pcre-jitforclamscanCL_ENGINE_PCRE_JITengine settingIt is disabled by default, so existing behavior is unchanged on upgrade.
The option is plumbed exactly like the existing
PCREMatchLimit/PCRERecMatchLimit/PCREMaxFileSizesettings (engine field, set/get,settings copy/apply, option table).
Implementation
cli_pcre_compile()performs a best-effortpcre2_jit_compile(pd->re, PCRE2_JIT_COMPLETE)after the match limits areset.
pcre2_match()automatically dispatches to the JIT code when a patternhas been JIT-compiled, so no change to the normal match call is required.
does not support), the failure is logged via
cli_dbgmsg()and the patternis matched by the interpreter as before.
cli_pcre_match()retries a match withPCRE2_NO_JITif the JIT matcherreturns
PCRE2_ERROR_JIT_STACKLIMIT. The JIT uses a fixed stack and, unlikethe interpreter, does not grow on demand; this retry keeps behavior
equivalent rather than surfacing a JIT-only error.
Motivation
When the Aho-Corasick prefilter promotes a large number of PCRE subsignatures
for full evaluation against a large buffer, the interpreted matcher dominates
scan time. The literal anchors of many filename-style phishing subsignatures
are short and commonly appear in text-heavy input, so a single buffer can
legitimately trigger several hundred PCRE evaluations, each scanning the full
buffer. JIT executes the identical automaton as native code, so this is a pure
latency improvement with no effect on detection.
Benchmark
Measured with two builds of the same revision — option off vs option on —
using an identical signature database and scan configuration (
ScanArchivedisabled, cache disabled). The test input was a ~37 MiB archive composed of
many small text files, which the prefilter promotes to roughly several hundred
PCRE subsignatures.
PCREJit/--pcre-jitonclamscan(standalone, includes DB load)clamdscan(resident daemon, scan-only)Phase decomposition (daemon, scan-only) attributes the gain to the PCRE phase:
Detection results were verified identical with the option on and off (EICAR
detected in both; the benign sample reported clean in both).
Testing
libclamavunit test suite (ENABLE_TESTS=ON) on both this branchand the unmodified base revision: both report
Checks: 1235with anidentical set of results, so this change introduces no new test failures. The
pcre,regex, andmatcherssuites — which exercise the changed compileand match paths — pass.
check_matchers.cwas updated to build with the JITpath enabled so the suite validates that matches are unchanged.
the prior release),
--pcre-jit=yesengages JIT, and--pcre-jit=noisidentical to the default.
containing a PCRE subsignature: matching input is detected and non-matching
input is reported clean, identically with the option on and off. A
known-malicious test file is detected and benign input reported clean in both
modes.
clamscanandclamdscan/clamdhonor the option.A note on benchmarking
The repository has no in-tree performance/benchmark test harness — the test
suite is correctness-only — so the timing numbers above were measured manually
by timing identical scans with the option off and on. A wall-clock assertion
was intentionally not added, since it would be flaky across CI hardware. The
per-signature
--statistics=pcrediagnostic can be used to reproduce thePCRE-phase timing breakdown.
Compatibility
CL_ENGINE_PCRE_JITis appended to the end of thecl_engine_fieldenum, so existing enumerator values are unchanged.