Fw backports/v7.2f - #15854
Draft
victorjulien wants to merge 64 commits into
Draft
Conversation
Add minimal SMTP progress states to support envelope validation before moving to data. Update SMTP, file and email keywords to hook into the appropriate states. Purposefully kept minimal for now as to not break the current idea of an SMTP transaction, which is probably not ideal for firewall mode. Ticket: OISF#8393 (cherry picked from commit c2728ee)
Add directionality to completion states, and replace tx->done by checking for both directions being complete. This means that the transaction is now not complete until the server responds to the clients of data marker, previously the tx was completed when the client send end of data without waiting for the server response. This keeps smtp:response_complete from being exposed before the server response is parsed. Ticket: OISF#8393 (cherry picked from commit 7b31f41)
(cherry picked from commit d81be73)
(cherry picked from commit 0e3c946)
(cherry picked from commit 0123cfd)
Don't allow trailing : (cherry picked from commit 7fe8f63)
It used the alstate name where it meant tx. (cherry picked from commit 280a1e5)
Since pstate can't be NULL, remove the conditional logic. (cherry picked from commit f4f5212)
(cherry picked from commit c1392d9)
(cherry picked from commit 695c9d8)
(cherry picked from commit 3b351e2)
Extend the app-layer-protocol keyword to accept a pipe-separated list of
protocol values, so a single rule can match any of several protocols:
app-layer-protocol:[!]<proto1>|<proto2>[|...][,<qualifier>]...;
A non-negated list matches when the flow's protocol equals any listed value
(OR); a negated list matches when it equals none of them (NOR). The
single-value form and the trailing mode qualifier are unchanged.
Matching keeps the historical AppProtoEquals() equivalences by default
(dns/doh2, http/http1/http2, dcerpc/smb, ...). An `exact` qualifier selects
strict identity matching with no equivalences and no http umbrella; it
combines with a direction mode in any order. Because a flow is never the
generic ALPROTO_HTTP, `http,exact` is rejected at load.
Values are expanded once at rule load into an effective match-set bitmask, so
the per-packet match is a single bitmask test. Single-value rules remain
prefilterable; multi-value rules are excluded from prefiltering and an
explicit prefilter on them is rejected. Conflicting keyword combinations
(duplicate or overlapping negations, mixed positive/negated) are rejected at
load. Engine-analysis reports the effective match set.
Ticket: 7705
(cherry picked from commit 43bc2db)
Ticket: 8518 Keywords that work for HTTP2 headers match now as soon as possible A push promise is now considered like a headers frame with regards to the progress (no dedicated "reserved" progress/state) http.protocol and http.stat_msg keywords are now registered at earliest progress, since these are synthetic like "HTTP/2" and not really seen on the wire. http.request_line and http.response_line match only on data, and not on headers, since we must wait the end of headers to be sure to have the full line http2.size_update now matches at headers progress as it should http2.frametype, http2.errorcode, http2.priority now match like http2.window, when the tx is complete from both sides, as a half-closed client may still send priority, rst_stream or window_update frames (cherry picked from commit daf68dc)
Ticket: 8518 Meaning they will now match only once per tx instead of twice: once for each direction (cherry picked from commit 8eed90c)
Ticket: 8518 (cherry picked from commit 26bb18c)
Allow registration of per substate progress name mappings. Implement logic for getting sub-state names, id's. Add transaction type and end of progress values to AppLayerTxData. Introduce helpers to keep logic clean. (cherry picked from commit b6bc025)
Support for per transaction sub states: different state machines per transaction type. Skip engines not belonging to our substate. Store tx_type in DetectTransaction. Each protocol supporting sub states will register states from 1 and up. To support: Ticket: OISF#8386. (cherry picked from commit 95365ce)
(cherry picked from commit 59e2c5a)
(cherry picked from commit 67e2cb5)
(cherry picked from commit 2f41e0e)
Split into 2 sub-states: - stream, which has the "HTTP" requests and responses, including DOH2 - global, which has the settings and other global or control handling Introduce a simpler progress tracking for the global sub state: - HTTP2ProgGlobalStart and HTTP2ProgGlobalComplete. The stream sub state uses the same state machine as before. Ticket: OISF#8386. (cherry picked from commit 7ebc699)
Hard coded for now. (cherry picked from commit c943de4)
(cherry picked from commit 416b9c1)
Each keyword supporting DOH2 must register explicitly (cherry picked from commit 1bb843d)
Now that DNS keywords are no longer registered for DOH2, the keywords need to manually registered for DOH2. (cherry picked from commit b36ec14)
Now that HTTP keywords are no longer automatically registered for HTTP/2, register them manually. (cherry picked from commit 17953d2)
Now that there is a policy hash, make the alert signature object a member of that to avoid another hash table lookup. (cherry picked from commit 3b6b381)
(cherry picked from commit af1c72c)
Check that sub state handling is correct. (cherry picked from commit 2e1003a)
(cherry picked from commit 17a3e0d)
In preparation of being able to log it. (cherry picked from commit 7e60c32)
(cherry picked from commit e1e24d6)
(cherry picked from commit 2511070)
(cherry picked from commit e4176d2)
http2 states were not yet supported even if the built-in states could already work. Clearly error out on a hook w/o substate. (cherry picked from commit 7517ba9)
(cherry picked from commit cfc98ae)
(cherry picked from commit 110e746)
(cherry picked from commit f02a427)
(cherry picked from commit 56d9670)
In firewall mode the alproto logic when building prefilter and rule engines is strict, but was out of sync between prefilter and rule app inspect engines. For SIGNATURE_HOOK_TYPE_APP rules the logic is strict, with an exception for HTTP/2 and DOH2. DOH2 is not a full protocol implementation, but rather HTTP/2 with a different alproto. Fixes: d64954a ("detect: don't register unrelated inspect engines") (cherry picked from commit f8f28c6)
This is to make the hook use names consistent with the default hooks and other protocols. (cherry picked from commit cf216b6)
DOH2 depends on HTTP/2, so it makes no sense to enable it separately. It would also put the sub state handling in a weird state, as the DOH2 side reuses the registered HTTP/2 callbacks. (cherry picked from commit f545f06)
(cherry picked from commit 9a294bf)
(cherry picked from commit c5d4a25)
When building the per signature app_inspect list, only add engines that match the sub state specified in the hook. (cherry picked from commit a824d97)
Register HTTP/2 and DOH2 this way. (cherry picked from commit 89baff0)
Make sure that every buffer in a signature is actually used in the setup of the engines. Reject sigs that are locked to a certain substate and use buffers that require another. (cherry picked from commit 8a5e8c6)
Add substates to the --list-app-layer-hooks option. (cherry picked from commit 8455efd)
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main-8.0.x #15854 +/- ##
==============================================
- Coverage 81.67% 76.64% -5.03%
==============================================
Files 1016 1016
Lines 276979 278189 +1210
==============================================
- Hits 226213 213231 -12982
- Misses 50766 64958 +14192
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
WARNING:
Pipeline = 32560 |
Contributor
Contributor
|
@lukashino, can we also merge |
Contributor
|
We also found a backward-compatibility regression with muti-value support in |
Contributor
|
Hi, yes, let me look at that tomorrow. I'll rebase your work on top of #15951
Thank you. Let us know how it goes. |
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Not for merge: backports for the tx substate work. Also includes #15837 and a backport of the app-layer-protocol work by @yashda.
SV_BRANCH=OISF/suricata-verify#3229