Sieve gets :regex, wearing the same circuit breaker as the rules engine

The :regex match type (draft-ietf-sieve-regex) on header, address,
envelope, body and hasflag, advertised as "regex" over ManageSieve.
The draft never became an RFC, but Dovecot implements it and clients
offer it, which is what a capability name is for.

Semantics: a SEARCH - the pattern may hit anywhere in the value, and a
script anchors with ^ and $ when it means the whole - where the rules
engine's regex criterion is a whole-value match. Case folds under the
default comparator (i;ascii-casemap) and does not under i;octet.
i;ascii-numeric with :regex is refused, as for :contains and :matches.

Two protections, because a script author is exactly as able to write a
catastrophic pattern as a rule author, and this is the delivery
thread:

- Every key is compiled at upload. A pattern that cannot compile is
  refused with its line number, instead of being stored and silently
  never matching - the failure mode the legacy rules engine had for
  years (HM6042).

- Evaluation runs under RuleGuard's budget-and-suspend breaker: 250 ms
  per evaluation, five-minute suspension keyed on the pattern text,
  reported once per window. SieveRegexMatches is a deliberately
  parallel sibling of RegexCriteriaMatches rather than a shared core -
  same suspension table, same budget, same error codes, each keeping
  its exact log wording, and the rules path character-for-character
  unchanged. The header says: if you change one, look at the other.

The breaker test delivers a message whose crafted subject detonates
(a+)+<unique>$ and asserts the thing that matters: the message ARRIVES,
in INBOX, with the suspension reported - a hostile sender must never be
able to turn a filter into mail loss. The pattern carries a per-run
unique tail because suspension outlives a test run and reports only on
insertion; a reused pattern would be answered from the table, report
nothing, and fail the assertion for the wrong reason.

Suite: 1513/1513.

Author: chrisholloway5

Roadmap.md

@@ -57,7 +57,7 @@ strong and where it is thin far more honestly than any prose summary.
 | [Transport security and deliverability](#transport-security-and-deliverability) | 43 | – | 4 | 1 |
 | [IMAP](#imap) | 58 | – | 17 | 3 |
 | [POP3](#pop3) | 21 | – | 6 | – |
-| [Sieve, ManageSieve and rules](#sieve-managesieve-and-rules) | 50 | 0 | 14 | – |
+| [Sieve, ManageSieve and rules](#sieve-managesieve-and-rules) | 51 | 0 | 13 | – |
 | [Authentication and cryptography](#authentication-and-cryptography) | 57 | 0 | 18 | – |
 | [Anti-spam, anti-virus and content control](#anti-spam-anti-virus-and-content-control) | 54 | – | 13 | – |
 | [Storage, accounts and data model](#storage-accounts-and-data-model) | 86 | – | 10 | – |
@@ -72,7 +72,7 @@ strong and where it is thin far more honestly than any prose summary.
 | [Future-proofing: standards and protocols](#future-proofing-standards-and-protocols) | 2 | – | 4 | 2 |
 | [Future-proofing: platform and supply chain](#future-proofing-platform-and-supply-chain) | 5 | 1 | 2 | 2 |
 | [Future-proofing: deployment and operations](#future-proofing-deployment-and-operations) | 4 | 1 | 4 | – |
-| **Total** | **625** | **12** | **143** | **15** |
+| **Total** | **626** | **12** | **142** | **15** |
 
 Three things stand out and are worth naming rather than leaving to be inferred.
 **Storage and the administration surface are the best-covered areas**, and the
@@ -448,7 +448,7 @@ the source, not from documentation.
 
 ### Sieve, ManageSieve and rules
 
-50 shipped · 0 underway · 14 not started · 0 deferred
+51 shipped · 0 underway · 13 not started · 0 deferred
 
 | | Capability | Detail |
 |:-:|---|---|
@@ -506,7 +506,7 @@ the source, not from documentation.
 | ⬜ | include (RFC 6609) | Not implemented, and refused rather than ignored: `include`, `return` and `global` are not in the known-command allowlist, so a script using them fails at upload with "unknown command". There is no personal/global script namespace in SieveStorage to include from. |
 | ✅ | mailbox / mboxmetadata (RFC 5490) | **The mailbox extension shipped 16 August 2026**: the `mailboxexists` test and `fileinto :create`. mailboxexists is the first test in this engine answered from outside the message - the delivery path hands the evaluator a callback over the recipient's real folder list, using the same lookup fileinto itself uses (`MessageUtilities::FolderExistsForDelivery`, kept beside `MoveToIMAPFolder` so the two cannot drift), and for a public folder the same insert-permission gate, because RFC 5490 3.1 defines "exists" as exists-and-deliverable. A caller with no store to ask (the COM test evaluator) gets false for every name - the safe answer. `:create` was already this server's behaviour - MoveToIMAPFolder has always created missing account folders - so the tag is accepted under its require and the semantics were proven rather than added: an end-to-end test SELECTs the folder the delivery created, over IMAP, from a real client session. Advertised over ManageSieve under the standing rule. Still absent from RFC 5490: the mboxmetadata half (METADATA is not implemented in the IMAP server at all). |
 | ⬜ | Out-of-office scheduling and scope | Only an end date exists — there is no start date, so a future absence cannot be scheduled and must be switched on manually. There is also no domain-level or server-level auto-reply, no separate internal/external message… |
-| ⬜ | regex (draft-ietf-sieve-regex) | Not implemented as a Sieve match type, even though the server already carries a regex engine used by the legacy rules engine (RuleCriteria::MatchesRegEx). Wiring it into MatchValue_ would be small |
+| ✅ | regex (draft-ietf-sieve-regex) | **Shipped 16 August 2026** as the `:regex` match type on header/address/envelope/body/hasflag, advertised as "regex" over ManageSieve. Never became an RFC, but Dovecot implements it and clients offer it. A search (anchor with `^`/`$` to mean the whole value), case-folded under the default comparator and case-sensitive under `i;octet`; `i;ascii-numeric` with `:regex` is refused. Two protections, both tested end to end: every key is COMPILED AT UPLOAD, so a pattern that cannot compile is refused with its line number instead of silently never matching (the failure mode the legacy rules engine had for years); and evaluation runs under the same RuleGuard budget-and-suspend breaker as the legacy regex criterion - a deliberately parallel sibling sharing the suspension table and error codes - so a catastrophic pattern costs one bounded evaluation per five-minute window, never the message. The breaker test delivers a crafted subject against `(a+)+…$` and asserts the message ARRIVES in INBOX with the suspension reported. |
 | ⬜ | reject / ereject (RFC 5429) | Not implemented. **The hazard this row used to describe is gone and the correction matters more than the gap:** neither is in the known-command allowlist and `require "reject"` is refused, so a script that tries to reject mail is rejected AT UPLOAD with a message naming the unsupported extension. The author is told; mail is not silently kept while they believe it is being refused. `SieveSyntax.cs` asserts exactly that script is refused. Implementing them means an SMTP-time refusal for `ereject` and an RFC 3834 bounce for `reject`. |
 | ✅ | relational (RFC 5231) | Not implemented. No :count or :value match types, and no i;ascii-numeric comparator to make them meaningful — SplitArguments recognises only is/contains/matches **Shipped in dc9301a.** |
 | ⬜ | RENAMESCRIPT and UNAUTHENTICATE | Neither implemented; both fall through to NO "Unknown command." A client renaming a script must GETSCRIPT/PUTSCRIPT/DELETESCRIPT by hand |

hmailserver/source/Server/Common/Rules/RuleGuard.cpp

@@ -330,6 +330,75 @@ namespace HM
       return false;
    }
 
+   bool
+   RuleGuard::SieveRegexMatches(const String &pattern, const String &subject, bool caseSensitive)
+   {
+      if (PatternIsSuspended_(pattern))
+         return false;
+
+      bool compiled = false;
+
+      try
+      {
+         // Perl syntax, as for a rule criterion; icase is what the script's default
+         // comparator (i;ascii-casemap) means, and i;octet turns it off.
+         boost::wregex expression(pattern,
+            caseSensitive ? boost::regex::normal : (boost::regex::normal | boost::regex::icase));
+         compiled = true;
+
+         const ULONGLONG startTick = ::GetTickCount64();
+
+         // regex_search, not regex_match: draft-ietf-sieve-regex matches anywhere in
+         // the value, and the script anchors with ^ and $ when it means the whole.
+         const bool matched = boost::regex_search(subject, expression);
+
+         const ULONGLONG elapsed = ::GetTickCount64() - startTick;
+
+         if (elapsed >= static_cast<ULONGLONG>(EvaluationBudgetMilliseconds) &&
+             SuspendPatternAndClaimReport_(pattern))
+         {
+            String errorMessage = Formatter::Format("A Sieve ':regex' match took {0} ms against a {1} character value, over the {2} ms budget, and will not be evaluated again for {3} seconds. Any sender can make this cost recur once per message, so it is being paid once per pattern instead. Pattern begins: {4}",
+               elapsed, subject.GetLength(), EvaluationBudgetMilliseconds, SuspensionSeconds, DescribePattern_(pattern));
+
+            ErrorManager::Instance()->ReportError(ErrorManager::Medium, 6043, "RuleGuard::SieveRegexMatches", errorMessage);
+         }
+
+         return matched;
+      }
+      catch (const std::runtime_error &)
+      {
+         if (!compiled)
+         {
+            // Upload validation compiles every ':regex' key, so reaching this means a
+            // script stored before that check existed, or edited on disk. Saying so
+            // matters for the same reason it does for a rule: a test that can never
+            // match looks exactly like a test that happens not to.
+            if (SuspendPatternAndClaimReport_(pattern))
+            {
+               String errorMessage = Formatter::Format("A Sieve ':regex' key is not a valid regular expression and can never match. The script containing it is active and that test is doing nothing. Pattern: {0}",
+                  DescribePattern_(pattern));
+
+               ErrorManager::Instance()->ReportError(ErrorManager::Medium, 6042, "RuleGuard::SieveRegexMatches", errorMessage);
+            }
+
+            return false;
+         }
+
+         if (SuspendPatternAndClaimReport_(pattern))
+         {
+            String errorMessage = Formatter::Format("A Sieve ':regex' match was abandoned as too complex against a {0} character value, and will not be evaluated again for {1} seconds. Treated as no match. Pattern begins: {2}",
+               subject.GetLength(), SuspensionSeconds, DescribePattern_(pattern));
+
+            ErrorManager::Instance()->ReportError(ErrorManager::Medium, 6043, "RuleGuard::SieveRegexMatches", errorMessage);
+         }
+
+         return false;
+      }
+
+      // Unreachable, for the same /WX reason as in RegexCriteriaMatches.
+      return false;
+   }
+
    void
    RuleGuard::ReportActionFailed(const String &actionName, const String &detail)
    {

hmailserver/source/Server/Common/Rules/RuleGuard.h

@@ -139,6 +139,22 @@ namespace HM
       // this into a log flood of its own.
       static bool RegexCriteriaMatches(const String &pattern, const String &subject);
 
+      // The same breaker for the Sieve ":regex" match type (draft-ietf-sieve-regex),
+      // which differs from a rule criterion in exactly two ways: it is a SEARCH
+      // (the pattern may hit anywhere in the value; a rule criterion must match the
+      // whole value), and its case behaviour follows the script's comparator
+      // (i;ascii-casemap folds case, i;octet does not) rather than being fixed.
+      //
+      // Deliberately a parallel sibling of RegexCriteriaMatches rather than a shared
+      // core: the two share the suspension table, the budget and the error codes
+      // (HM6042/HM6043), but each keeps its own exact log wording, and the rules
+      // path stays character-for-character what it was. If you change one, look at
+      // the other. Suspension is keyed on the pattern text alone, so a pattern
+      // suspended by a rule is also suspended for scripts and vice versa - the
+      // breaker's question is "is this pattern affordable", which does not depend
+      // on who asked.
+      static bool SieveRegexMatches(const String &pattern, const String &subject, bool caseSensitive);
+
       // A rule action failed and may have left the message file in a state nobody
       // chose.
       //

hmailserver/source/Server/Common/Sieve/ManageSieveServer.cpp

@@ -234,7 +234,17 @@ namespace HM
       // delivered message was filed, with the negative control being a
       // mailboxexists on a folder that is absent: the message must stay in INBOX,
       // which an implementation answering "true" for everything cannot pass.
-      const char *AdvertisedSieveExtensions = "fileinto copy relational subaddress vacation vacation-seconds imap4flags body mailbox";
+      // regex moved in on 16 August 2026, under the same rule. Its delivery-side
+      // step is MatchValue_'s dispatch into RuleGuard::SieveRegexMatches - the same
+      // budget-and-suspend breaker the legacy rules engine's regex criterion runs
+      // under, so a catastrophic pattern costs one 250 ms evaluation per five-minute
+      // window rather than one per message, and delivery always continues. Keys are
+      // compiled at upload, so a pattern that cannot compile is refused with a line
+      // number instead of silently never matching. End-to-end tests in
+      // SieveRegexDelivery.cs include the one that matters for the breaker: a
+      // message whose crafted subject makes the pattern catastrophic is still
+      // DELIVERED, to INBOX, with the pattern suspended rather than the thread lost.
+      const char *AdvertisedSieveExtensions = "fileinto copy relational subaddress vacation vacation-seconds imap4flags body mailbox regex";
 
       AnsiString EscapeQuoted(const String &value)
       {

hmailserver/source/Server/Common/Sieve/SieveEvaluator.cpp

@@ -6,6 +6,7 @@
 #include "SieveEvaluator.h"
 
 #include "../Util/Parsing/StringParser.h"
+#include "../Rules/RuleGuard.h"
 
 #ifdef _DEBUG
 #define DEBUG_NEW new(_NORMAL_BLOCK, __FILE__, __LINE__)
@@ -1132,6 +1133,15 @@ namespace HM
    bool
    SieveEvaluator::MatchValue_(const String &matchType, bool caseSensitive, const String &value, const String &key)
    {
+      if (matchType == _T("regex"))
+      {
+         // draft-ietf-sieve-regex: a search, case-folded under the default
+         // comparator. Runs under RuleGuard's budget-and-suspend breaker, because a
+         // script author is exactly as able to write a catastrophic pattern as a
+         // rule author, and this is the delivery thread.
+         return RuleGuard::SieveRegexMatches(key, value, caseSensitive);
+      }
+
       if (matchType == _T("contains"))
       {
          if (caseSensitive)

hmailserver/source/Server/Common/Sieve/SieveParser.cpp

@@ -5,6 +5,10 @@
 
 #include "SieveParser.h"
 
+// For the upload-time compile check on ':regex' keys. The evaluator matches with
+// Boost too (through RuleGuard), so what compiles here is what will run there.
+#include <boost/regex.hpp>
+
 #ifdef _DEBUG
 #define DEBUG_NEW new(_NORMAL_BLOCK, __FILE__, __LINE__)
 #define new DEBUG_NEW
@@ -48,6 +52,22 @@ namespace HM
                 comparator.Compare(_T("i;octet")) == 0 ||
                 comparator.Compare(_T("i;ascii-numeric")) == 0;
       }
+
+      // Whether a ':regex' key compiles, using the same construction the evaluator
+      // will use (Boost, Perl syntax). Called at upload validation so a bad pattern
+      // is refused with a line number instead of silently never matching.
+      bool RegexKeyCompiles_(const String &key)
+      {
+         try
+         {
+            boost::wregex expression(key);
+            return true;
+         }
+         catch (const std::runtime_error &)
+         {
+            return false;
+         }
+      }
    }
 
    SieveParser::SieveParser() :
@@ -102,6 +122,11 @@ namespace HM
          L"copy",
          L"relational",
          L"subaddress",
+         // draft-ietf-sieve-regex. Never became an RFC, but Dovecot's Pigeonhole
+         // implements it and clients offer it, which is what a capability name is
+         // for. The evaluation runs under the same budget-and-suspend breaker the
+         // legacy rules engine's regex criterion uses (RuleGuard).
+         L"regex",
          L"comparator-i;ascii-casemap",
          L"comparator-i;octet",
          L"comparator-i;ascii-numeric"
@@ -229,7 +254,8 @@ namespace HM
 
          if (!TagTakesValue(tag))
          {
-            if (tag == _T("is") || tag == _T("contains") || tag == _T("matches"))
+            if (tag == _T("is") || tag == _T("contains") || tag == _T("matches") ||
+                tag == _T("regex"))
             {
                result.matchType = tag;
                result.matchTypeGiven = true;
@@ -857,14 +883,38 @@ namespace HM
          }
       }
       else if (set.comparatorGiven && set.comparator.Compare(_T("i;ascii-numeric")) == 0 &&
-               (set.matchType == _T("contains") || set.matchType == _T("matches")))
+               (set.matchType == _T("contains") || set.matchType == _T("matches") ||
+                set.matchType == _T("regex")))
       {
          // i;ascii-numeric only defines equality (RFC 4790 9.1.1).
          errorMessage.Format(_T("Line %d: the \"i;ascii-numeric\" comparator cannot be used with ':%s'."),
             line, set.matchType.c_str());
          return false;
       }
 
+      if (set.matchType == _T("regex"))
+      {
+         if (!NeedExtension_(_T("regex"), _T("':regex'"), line, errorMessage))
+            return false;
+
+         // Compile every key NOW, so the author hears about a bad pattern at upload
+         // rather than the test silently never matching at delivery. The keys are
+         // the last positional list; the earlier one, when there are two, is the
+         // header/part names, which are not patterns.
+         if (!set.stringLists.empty())
+         {
+            for (const String &key : set.stringLists.back())
+            {
+               if (!RegexKeyCompiles_(key))
+               {
+                  errorMessage.Format(_T("Line %d: ':regex' key '%s' is not a valid regular expression."),
+                     line, key.c_str());
+                  return false;
+               }
+            }
+         }
+      }
+
       if (set.addressPart == _T("user") || set.addressPart == _T("detail"))
       {
          if (!NeedExtension_(_T("subaddress"), _T("':") + set.addressPart + _T("'"), line, errorMessage))
@@ -1241,7 +1291,7 @@ namespace HM
 
       if (name == _T("header"))
       {
-         if (!CheckTags_(set, _T("comparator is contains matches value count"), _T("'header'"), errorMessage))
+         if (!CheckTags_(set, _T("comparator is contains matches value count regex"), _T("'header'"), errorMessage))
             return false;
 
          return ValidateMatchArguments_(set, _T("'header'"), test->line, errorMessage);
@@ -1253,7 +1303,7 @@ namespace HM
              !NeedExtension_(_T("envelope"), _T("the 'envelope' test"), test->line, errorMessage))
             return false;
 
-         if (!CheckTags_(set, _T("comparator is contains matches value count all localpart domain user detail"),
+         if (!CheckTags_(set, _T("comparator is contains matches value count regex all localpart domain user detail"),
                          _T("'") + name + _T("'"), errorMessage))
             return false;
 
@@ -1300,7 +1350,7 @@ namespace HM
          if (!NeedExtension_(_T("body"), _T("the 'body' test"), test->line, errorMessage))
             return false;
 
-         if (!CheckTags_(set, _T("comparator is contains matches value count raw text content"),
+         if (!CheckTags_(set, _T("comparator is contains matches value count regex raw text content"),
                          _T("'body'"), errorMessage))
             return false;
 
@@ -1331,7 +1381,7 @@ namespace HM
          if (!NeedExtension_(_T("imap4flags"), _T("the 'hasflag' test"), test->line, errorMessage))
             return false;
 
-         if (!CheckTags_(set, _T("comparator is contains matches value count"), _T("'hasflag'"), errorMessage))
+         if (!CheckTags_(set, _T("comparator is contains matches value count regex"), _T("'hasflag'"), errorMessage))
             return false;
 
          if (set.comparatorGiven && !IsKnownComparator(set.comparator))

hmailserver/test/RegressionTests/RegressionTests.csproj

@@ -350,6 +350,7 @@
     <Compile Include="Sieve\SieveBodyDelivery.cs" />
     <Compile Include="Sieve\SieveFlagDelivery.cs" />
     <Compile Include="Sieve\SieveMailboxDelivery.cs" />
+    <Compile Include="Sieve\SieveRegexDelivery.cs" />
     <Compile Include="Sieve\SieveVacationDelivery.cs" />
     <Compile Include="Sieve\ManageSieveSecurity.cs" />
     <Compile Include="Sieve\SieveSyntax.cs" />

hmailserver/test/RegressionTests/Sieve/ManageSieve.cs

@@ -247,6 +247,11 @@ public void TestManageSieveRoundTrip()
                // delivery created.
                StringAssert.Contains("mailbox", greeting);
 
+               // regex followed the same day, with SieveRegexDelivery.cs behind it -
+               // including the breaker test: a catastrophic pattern costs a bounded
+               // evaluation and a suspension, never the message.
+               StringAssert.Contains("regex", greeting);
+
                // Still absent on purpose: envelope is implemented and evaluated, and is
                // held back until a test proves the envelope TEST command itself works.
                // If somebody adds it here, that test should exist first.