Conversation threading, and the reply chain that could have taken the server down

THREAD and UID THREAD (RFC 5256), both algorithms, advertised as
THREAD=ORDEREDSUBJECT THREAD=REFERENCES and riding the same bounded search
machinery as SORT.

REFERENCES is the actual algorithm rather than subject grouping wearing its
name. Containers are linked by Message-ID / References / In-Reply-To, with
In-Reply-To consulted only when References is absent; a message's own References
line authoritatively breaks a parent that somebody else's line asserted; cycles
are refused rather than linked; dummies are pruned except at the root, where a
dummy with two or more children is what holds two replies together when the
message they both answer never arrived. Threads are then grouped by RFC 2.1 base
subject - the whole step machine, so "[fwd: Re: [list] x]" reduces the way the
RFC says rather than the way one regular expression can - and sorted by sent
date. ORDEREDSUBJECT is the flat approximation the RFC requires of anyone
advertising THREAD at all.

Not gated behind a setting, unlike SORT, and the asymmetry is deliberate: the
work is bounded by the same per-command ceilings as SEARCH, and a gate would
have cost a seeded settings row on every backend plus a schema bump - a missing
row reports HM5015 on every read rather than defaulting quietly - to guard
something already bounded.

An adversarial pass over 11 passing tests found three defects, and the one that
mattered was measured rather than argued. Prune_ and SortChildrenRecursively_
recurse once per tree level on a 1MB default worker stack; a probe compiled
against those exact frames survived 6000 levels and terminated with
STATUS_STACK_OVERFLOW at 8000, while the old cycle guard permitted chains near
10000. A stack overflow is not reliably catchable, so that was a crash of the
whole service, reachable by any authenticated user willing to mail themselves a
long enough reply chain. Links are now refused past depth 512, which bounds all
three recursive walks at their source.

The header pass was uninterruptible: the ceiling was checked after BuildThreads
returned, which can only disown a result the connection thread has already spent
minutes producing. The predicate is handed in now and stops before the next file
is opened.

And a Date header that was present but unparseable fell through to the 1899 OLE
epoch, which the comparator reads as a raw value without consulting the validity
flag - so the most malformed message in a conversation sorted to the front of
it. RFC 5256 2.2's INTERNALDATE fallback now covers unparseable as well as
absent.

Two pre-existing weaknesses of exactly these shapes are recorded in the roadmap
rather than silently inherited: SORT checks its ceiling only after its own
uninterruptible header pass, and IMAPSort::CacheHeaderFields_ tests the Date
header for emptiness rather than validity.

13 tests assert exact trees - chains, forks, dummy roots, subject grouping, UID
mode against UIDs read back rather than assumed, BAD for an unadvertised
algorithm, a same-connection stale-state control, a 600-message chain that must
not crash, and the INTERNALDATE fallback. The fixture earned itself twice: it
caught a stray space between root threads, which a threading client would parse
as an error, and its own first date assertion was wrong rather than the code -
crafted headers dated today at 10:xx against a suite running at 09:xx.

Full suite: 1490/1490.

Author: chrisholloway5

README.md

@@ -29,7 +29,7 @@ Mail protocols
 --------------
 
 * **SMTP** with PIPELINING, ENHANCEDSTATUSCODES, 8BITMIME, SIZE, CHUNKING/BDAT (RFC 3030), DSN delivery status notifications (RFC 3461/3464) and SMTPUTF8/EAI for internationalised addresses.
-* **IMAP4rev1**, plus **IMAP4rev2** (RFC 9051) advertised with its behavioural deltas implemented — a few of the extensions rev2 folds in (LIST-STATUS, non-synchronising literals, BINARY) are still missing — with IDLE, MOVE (RFC 6851), UIDPLUS (RFC 4315), CONDSTORE/QRESYNC (RFC 7162), SEARCHRES (RFC 5182), ESEARCH (RFC 4731), SORT, ACL, NAMESPACE, ID (RFC 2971), SPECIAL-USE (RFC 6154, including explicit designation via `CREATE ... (USE (\Sent))`) and QUOTA.
+* **IMAP4rev1**, plus **IMAP4rev2** (RFC 9051) advertised with its behavioural deltas implemented — a few of the extensions rev2 folds in (LIST-STATUS, non-synchronising literals, BINARY) are still missing — with IDLE, MOVE (RFC 6851), UIDPLUS (RFC 4315), CONDSTORE/QRESYNC (RFC 7162), SEARCHRES (RFC 5182), ESEARCH (RFC 4731), SORT and THREAD (RFC 5256, both ORDEREDSUBJECT and REFERENCES), ACL, NAMESPACE, ID (RFC 2971), SPECIAL-USE (RFC 6154, including explicit designation via `CREATE ... (USE (\Sent))`) and QUOTA.
 * **POP3**, including retrieval from external POP3 accounts on a schedule.
 * **Public folders**, shared across accounts with per-user ACLs.
 

Roadmap.md

@@ -55,7 +55,7 @@ strong and where it is thin far more honestly than any prose summary.
 | **The capability matrix** | | | | |
 | [SMTP and ESMTP](#smtp-and-esmtp) | 24 | – | 4 | – |
 | [Transport security and deliverability](#transport-security-and-deliverability) | 43 | – | 4 | 1 |
-| [IMAP](#imap) | 57 | – | 18 | 3 |
+| [IMAP](#imap) | 58 | – | 17 | 3 |
 | [POP3](#pop3) | 21 | – | 6 | – |
 | [Sieve, ManageSieve and rules](#sieve-managesieve-and-rules) | 48 | 0 | 16 | – |
 | [Authentication and cryptography](#authentication-and-cryptography) | 57 | 0 | 18 | – |
@@ -72,7 +72,7 @@ strong and where it is thin far more honestly than any prose summary.
 | [Future-proofing: standards and protocols](#future-proofing-standards-and-protocols) | 2 | – | 4 | 2 |
 | [Future-proofing: platform and supply chain](#future-proofing-platform-and-supply-chain) | 5 | 1 | 2 | 2 |
 | [Future-proofing: deployment and operations](#future-proofing-deployment-and-operations) | 4 | 1 | 4 | – |
-| **Total** | **621** | **12** | **147** | **14** |
+| **Total** | **622** | **12** | **146** | **14** |
 
 Three things stand out and are worth naming rather than leaving to be inferred.
 **Storage and the administration surface are the best-covered areas**, and the
@@ -329,7 +329,7 @@ the source, not from documentation.
 
 ### IMAP
 
-57 shipped · 0 underway · 18 not started · 3 deferred
+58 shipped · 0 underway · 17 not started · 3 deferred
 
 | | Capability | Detail |
 |:-:|---|---|
@@ -407,7 +407,7 @@ the source, not from documentation.
 | ⬜ | REPLACE (RFC 8508) | Not implemented and not advertised; there is no REPLACE or UID REPLACE command, so clients must emulate draft updates with APPEND + STORE \Deleted + EXPUNGE. |
 | ⬜ | RFC 9208 QUOTA (QUOTA=RES-STORAGE / RES-MESSAGE, SETQUOTA) | Not implemented. The bare "QUOTA" capability atom is advertised rather than the RFC 9208 QUOTA=RES-* form, there is no SETQUOTA, no per-mailbox quota roots and no OVERQUOTA response code. Listed in the roadmap's IMAP-extension backlog. |
 | ⬜ | SAVEDATE (RFC 8514) | Not implemented. No SAVEDATE FETCH item and no SAVEDATE/SAVEDBEFORE/SAVEDSINCE/SAVEDATESUPPORTED search keys. Listed in the roadmap's IMAP-extension backlog. |
-| ⬜ | THREAD (RFC 5256) | Not implemented and not advertised. No THREAD command in the dispatch table and no reference anywhere in the source. SORT ships from the same RFC but THREAD does not… |
+| ✅ | THREAD (RFC 5256) | **Shipped 15 August 2026, both algorithms.** `THREAD` and `UID THREAD` advertise `THREAD=ORDEREDSUBJECT THREAD=REFERENCES` and ride the same bounded search machinery as SORT. REFERENCES is the full RFC algorithm rather than a subject approximation wearing its name: containers are linked by `Message-ID`/`References`/`In-Reply-To` (falling back to In-Reply-To only when References is absent), a message's own References line authoritatively breaks a parent another message asserted, cycles are refused, dummies are pruned except at the root where they hold orphaned siblings together, threads are then grouped by RFC 2.1 base subject - the full step machine, so `[fwd: Re: [list] x]` reduces the way the RFC says - and sorted by sent date with the INTERNALDATE fallback. **An adversarial pass found three defects the 11 passing tests did not.** A deep reply chain overflowed the 1MB worker stack: `Prune_` and `SortChildrenRecursively_` recurse per level, and a probe compiled against those exact frames survived 6000 levels and died with `STATUS_STACK_OVERFLOW` at 8000 - a whole-service crash any authenticated user could reach by mailing themselves a long enough chain, now bounded by a 512 depth cap at link time. The header pass was uninterruptible, so `IMAPSearchTimeout` could only disown a result the connection thread had already spent minutes producing; the ceiling is now handed in and stops the pass before the next file opens. And a Date header that was present but unparseable fell through to the 1899 OLE epoch and sorted to the FRONT of its thread, because the comparator reads the raw value and not the validity flag - RFC 5256 2.2's INTERNALDATE fallback now covers unparseable as well as absent. 13 tests assert exact trees. **Two pre-existing weaknesses of the same shape are recorded, not silently inherited:** SORT checks its ceiling only after its own uninterruptible header pass, and `IMAPSort::CacheHeaderFields_` tests the Date header for emptiness rather than validity. |
 | ⬜ | UNAUTHENTICATE (RFC 8437) | Not implemented and not advertised; there is no way to return an authenticated session to the not-authenticated state for connection reuse. |
 | ⬜ | URLAUTH (RFC 4467) / BURL | Not implemented and not advertised; no GENURLAUTH, URLFETCH or RESETKEY commands and no IMAP URL parser. |
 | ✅ | WITHIN (RFC 5032) OLDER / YOUNGER search keys | Implemented and advertised. Both compare the message's internal date against (now - n seconds) using the whole timestamp rather than the calendar date, which is the point of the extension and what makes "YOUNGER 3600" answerable at all; relative-age searches no longer have to be rewritten as absolute BEFORE/SINCE. The interval is validated as an nz-number and clamped at a century, so a zero, negative or non-numeric interval is BAD instead of silently becoming "0 seconds ago". |

hmailserver/source/Server/IMAP/IMAPCommandCapability.cpp

@@ -32,6 +32,13 @@ namespace HM
       if (pConfig->GetUseIMAPSort())
          sResponse += " SORT";
 
+      // RFC 5256. Unconditional where SORT is gated, and deliberately so: the work
+      // is bounded by the same per-command ceilings as SEARCH and SORT, and a gate
+      // would cost a seeded settings row on every backend plus a schema bump - see
+      // the reasoning on IMAPThread. ORDEREDSUBJECT is required of any server that
+      // advertises THREAD at all; REFERENCES is the one clients actually want.
+      sResponse += " THREAD=ORDEREDSUBJECT THREAD=REFERENCES";
+
       if (pConfig->GetUseIMAPACL())
          sResponse += " ACL";
 

hmailserver/source/Server/IMAP/IMAPCommandSearch.cpp

@@ -7,6 +7,7 @@
 #include "IMAPCommandSEARCH.h"
 #include "IMAPConnection.h"
 #include "IMAPSort.h"
+#include "IMAPThread.h"
 #include "IMAPConfiguration.h"
 #include "IMAPListLookup.h"
 
@@ -27,8 +28,9 @@
 
 namespace HM
 {
-   IMAPCommandSEARCH::IMAPCommandSEARCH(bool bIsSort) :
-      is_sort_(bIsSort),
+   IMAPCommandSEARCH::IMAPCommandSEARCH(IMAPSearchCommandMode mode) :
+      is_sort_(mode == IMAPSearchModeSort),
+      is_thread_(mode == IMAPSearchModeThread),
       is_uid_(false),
       is_esearch_(false),
       esearch_min_(false),
@@ -93,8 +95,9 @@ namespace HM
 
       // RFC 4731 (ESEARCH): an optional "RETURN (...)" result-options clause may
       // follow the SEARCH keyword. Detect and consume it before criteria parsing.
-      // (RETURN with SORT is ESORT/RFC 5267 and is intentionally not handled here.)
-      if (!is_sort_)
+      // (RETURN with SORT is ESORT/RFC 5267 and is intentionally not handled here,
+      // and RFC 5267's THREAD variant likewise.)
+      if (!is_sort_ && !is_thread_)
       {
          String sTrimmed = pArgument->Command();
          sTrimmed.TrimLeft();
@@ -132,7 +135,7 @@ namespace HM
       // RFC 9051 (IMAP4rev2): once the client has enabled IMAP4rev2, a SEARCH (or UID
       // SEARCH) without an explicit RETURN clause still returns its results in an
       // ESEARCH response (with the ALL result option), not the legacy "* SEARCH" line.
-      if (!is_sort_ && !is_esearch_ && pConnection->GetImap4Rev2Enabled())
+      if (!is_sort_ && !is_thread_ && !is_esearch_ && pConnection->GetImap4Rev2Enabled())
       {
          is_esearch_ = true;
          esearch_all_ = true;
@@ -143,13 +146,23 @@ namespace HM
       // RFC 5182 (SEARCHRES): the parser resolves a "$" search key against this.
       pParser->SetSavedSearchResult(pConnection->GetSavedSearchResult());
 
-      IMAPResult result = pParser->ParseCommand(pArgument, is_sort_);
+      IMAPResult result = pParser->ParseCommand(pArgument,
+         is_thread_ ? IMAPSearchModeThread : (is_sort_ ? IMAPSearchModeSort : IMAPSearchModeSearch));
       if (result.GetResult() != IMAPResult::ResultOK)
          return result;
 
       if (is_sort_ && !pParser->GetSortParser())
          return IMAPResult(IMAPResult::ResultBad, "Incorrect search commands.");
 
+      // RFC 5256: naming an algorithm the server did not advertise is a protocol
+      // error. BAD rather than a silent fallback, because a client that asked for
+      // REFERENCES and silently got ORDEREDSUBJECT would render wrong conversation
+      // trees with no way to know it.
+      IMAPThread::Algorithm threadAlgorithm = IMAPThread::AlgorithmReferences;
+
+      if (is_thread_ && !IMAPThread::ParseAlgorithm(pParser->GetThreadAlgorithm(), threadAlgorithm))
+         return IMAPResult(IMAPResult::ResultBad, "Unsupported threading algorithm.");
+
       // Mails in current box
       std::shared_ptr<IMAPFolder> pCurFolder =  pConnection->GetCurrentFolder();
 
@@ -215,6 +228,26 @@ namespace HM
                return AbortSearch_(pConnection, message_count_);
          }
 
+         if (is_thread_)
+         {
+            // Threading reads a header per matching message, and the ceiling is
+            // handed IN rather than checked afterwards. SORT above does the latter,
+            // and the difference matters: a post-hoc check can only disown a result
+            // the connection thread has already spent minutes producing, whereas
+            // this stops before the next file is opened. On a folder large enough
+            // for the timeout to bite, that is the difference between a bounded
+            // command and a bounded-looking one.
+            IMAPThread oThreader;
+
+            const bool completed = oThreader.BuildThreads(pConnection, vecMatchingMessages, is_uid_,
+                                                          threadAlgorithm,
+                                                          [this]() { return BoundExceeded_(); },
+                                                          thread_response_body_);
+
+            if (!completed || BoundExceeded_())
+               return AbortSearch_(pConnection, message_count_);
+         }
+
          typedef std::pair<int, std::shared_ptr<Message> > MessagePair;
          for(MessagePair messagePair : vecMatchingMessages)
          {
@@ -284,6 +317,12 @@ namespace HM
       }
       else if (is_sort_)
          sResponse = "* SORT" + sMatching + "\r\n";
+      else if (is_thread_)
+      {
+         // The body carries its own leading space when there is one - an empty
+         // result is a bare "* THREAD", not "* THREAD ".
+         sResponse = "* THREAD" + thread_response_body_ + "\r\n";
+      }
       else
       {
          sResponse = "* SEARCH" + sMatching;
@@ -678,6 +717,10 @@ namespace HM
       bytes_examined_ = 0;
       bound_reason_.Empty();
 
+      // The handler is reused per connection like everything else here, so a THREAD
+      // that matched nothing must not answer with the PREVIOUS thread's tree.
+      thread_response_body_.Empty();
+
       // Read once, here. If they were read per message a configuration reload
       // during a long search could raise the ceiling out from under it.
       search_timeout_seconds_ = IniFileSettings::Instance()->GetIMAPSearchTimeout();

hmailserver/source/Server/IMAP/IMAPCommandSearch.h

@@ -18,7 +18,7 @@ namespace HM
    class IMAPCommandSEARCH : public IMAPCommand
    {
    public:
-	   IMAPCommandSEARCH(bool bIsSort);
+	   IMAPCommandSEARCH(IMAPSearchCommandMode mode);
 	   virtual ~IMAPCommandSEARCH();
 
       void SetIsUID() { is_uid_ = true; }
@@ -57,6 +57,7 @@ namespace HM
       std::shared_ptr<MimeHeader> mime_header_;
 
       bool is_sort_;
+      bool is_thread_;
       bool is_uid_;
 
       // RFC 4731 (ESEARCH) result-option state, parsed from a "RETURN (...)" clause.
@@ -91,5 +92,10 @@ namespace HM
 
       // Which ceiling was passed, and by how much, for the application log entry.
       String bound_reason_;
+
+      // The rendered RFC 5256 tree for a THREAD command, built where SORT sorts and
+      // consumed where the response is assembled. Cleared by ResetSearchState_,
+      // because the handler is reused across commands on a connection.
+      String thread_response_body_;
    };
 }

hmailserver/source/Server/IMAP/IMAPCommandUID.cpp

@@ -192,7 +192,7 @@ namespace HM
       }
       else if (sTypeOfUID.CompareNoCase(_T("SEARCH")) == 0)
       {
-         std::shared_ptr<IMAPCommandSEARCH> pCommand = std::shared_ptr<IMAPCommandSEARCH> (new IMAPCommandSEARCH(false));
+         std::shared_ptr<IMAPCommandSEARCH> pCommand = std::shared_ptr<IMAPCommandSEARCH> (new IMAPCommandSEARCH(IMAPSearchModeSearch));
          pCommand->SetIsUID();
          IMAPResult result = pCommand->ExecuteCommand(pConnection, pArgument);
 
@@ -203,10 +203,23 @@ namespace HM
       }
       else if (sTypeOfUID.CompareNoCase(_T("SORT")) == 0)
       {
-         std::shared_ptr<IMAPCommandSEARCH> pCommand = std::shared_ptr<IMAPCommandSEARCH> (new IMAPCommandSEARCH(true));
+         std::shared_ptr<IMAPCommandSEARCH> pCommand = std::shared_ptr<IMAPCommandSEARCH> (new IMAPCommandSEARCH(IMAPSearchModeSort));
          pCommand->SetIsUID();
          IMAPResult result = pCommand->ExecuteCommand(pConnection, pArgument);
-         
+
+         if (result.GetResult() == IMAPResult::ResultOK)
+            pConnection->SendAsciiData(sTag + " OK UID completed\r\n");
+
+         return result;
+      }
+      else if (sTypeOfUID.CompareNoCase(_T("THREAD")) == 0)
+      {
+         // RFC 5256: UID THREAD is THREAD with UIDs in the tree instead of
+         // sequence numbers. Same shape as UID SORT above.
+         std::shared_ptr<IMAPCommandSEARCH> pCommand = std::shared_ptr<IMAPCommandSEARCH> (new IMAPCommandSEARCH(IMAPSearchModeThread));
+         pCommand->SetIsUID();
+         IMAPResult result = pCommand->ExecuteCommand(pConnection, pArgument);
+
          if (result.GetResult() == IMAPResult::ResultOK)
             pConnection->SendAsciiData(sTag + " OK UID completed\r\n");
 

hmailserver/source/Server/IMAP/IMAPConnection.cpp

@@ -168,8 +168,9 @@ namespace HM
    {
       mapCommandHandlers[IMAP_UID] = std::shared_ptr<IMAPCommandUID>(new IMAPCommandUID());
       mapCommandHandlers[IMAP_APPEND] = std::shared_ptr<IMAPCommandAppend>(new IMAPCommandAppend());
-      mapCommandHandlers[IMAP_SEARCH] = std::shared_ptr<IMAPCommandSEARCH>(new IMAPCommandSEARCH(false));
-      mapCommandHandlers[IMAP_SORT] = std::shared_ptr<IMAPCommandSEARCH>(new IMAPCommandSEARCH(true));
+      mapCommandHandlers[IMAP_SEARCH] = std::shared_ptr<IMAPCommandSEARCH>(new IMAPCommandSEARCH(IMAPSearchModeSearch));
+      mapCommandHandlers[IMAP_SORT] = std::shared_ptr<IMAPCommandSEARCH>(new IMAPCommandSEARCH(IMAPSearchModeSort));
+      mapCommandHandlers[IMAP_THREAD] = std::shared_ptr<IMAPCommandSEARCH>(new IMAPCommandSEARCH(IMAPSearchModeThread));
       mapCommandHandlers[IMAP_IDLE] = std::shared_ptr<IMAPCommandIdle>(new IMAPCommandIdle(std::dynamic_pointer_cast<IMAPConnection>(shared_from_this())));
 
       mapStaticHandlers = StaticIMAPCommandHandlers::Instance()->GetStaticHandlers();
@@ -685,6 +686,8 @@ namespace HM
          return IMAP_IDLE;
       else if (sCommand == _T("SORT"))
          return IMAP_SORT;
+      else if (sCommand == _T("THREAD"))
+         return IMAP_THREAD;
       else if (sCommand == _T("NAMESPACE"))
          return IMAP_NAMESPACE;
       else if (sCommand == _T("MYRIGHTS"))
@@ -868,10 +871,15 @@ namespace HM
       command.
       */
 
+      // RFC 5256 carries SORT's rule over verbatim: untagged EXPUNGE responses are
+      // not permitted while responding to THREAD, but are permitted during a UID
+      // THREAD - and active_command is IMAP_UID in the UID case, so the exclusion
+      // below covers exactly the non-UID form, which is the required behaviour.
       bool send_expunge = active_command != IMAP_FETCH &&
                           active_command != IMAP_STORE &&
                           active_command != IMAP_SEARCH &&
-                          active_command != IMAP_SORT;
+                          active_command != IMAP_SORT &&
+                          active_command != IMAP_THREAD;
 
       notification_client_->SendCachedNotifications(send_expunge);
    }

hmailserver/source/Server/IMAP/IMAPConnection.h

@@ -84,7 +84,8 @@ namespace HM
          IMAP_MOVE = 136,
          IMAP_ID = 137,
          IMAP_UNSELECT = 138,
-         IMAP_ENABLE = 139
+         IMAP_ENABLE = 139,
+         IMAP_THREAD = 140
       };
 
       void ParseData(const AnsiString &Request);