Regression suite: make the SpamAssassin and ClamAV tests actually run

Installing the two optional integrations - ClamAV 1.5.4 and SpamAssassin 4.0.2 -
took the 27 tests that had always reported inconclusive on this machine and ran
them for the first time. Two of them failed, both because of defects in the
tests rather than in the server.

ClamAV.TestWithVirus put the EICAR string in the plain body of a non-MIME
message. ClamAV's EICAR signatures match a whole file, not a substring, and the
boundary is easy to demonstrate over clamd's own INSTREAM:

    exact EICAR, no trailing newline    68 bytes -> Eicar-Test-Signature FOUND
    EICAR + CRLF                        70 bytes -> Eicar-Signature FOUND
    EICAR + trailing text               79 bytes -> OK
    plain .eml, EICAR as the body      131 bytes -> OK
    MIME .eml, EICAR base64 attached   451 bytes -> Eicar-Test-Signature FOUND

Older ClamAV extracted a plain body as a scannable part; current ClamAV does
not, so the message scanned clean and was delivered - and the test failed on its
own "no messages" assertion. hMailServer was never at fault: it connected,
streamed the file and parsed the FOUND reply correctly, which the attachment
case proves. The test now sends EICAR as a base64 attachment, which is how a
virus actually arrives, and matches the signature family rather than one exact
name - the name has already changed once across ClamAV releases, and the test
was still asserting the old one.

The EICAR string stays split across two fragments and is base64-encoded at run
time so that neither form is ever a literal in the source file. On-access
scanners quarantine files containing either; Windows Defender deleted two
scratch files during this work for exactly that reason.

SpamAssassin.TestSANotRunning had a race and a wrong expectation.
StopSpamAssassin() called ServiceController.Stop() and returned immediately.
Stop() only asks - it returns as soon as the request is accepted, while spamd is
still up and still answering on 783. The test then sent a message straight into
a live spamd and got the X-Spam-Status header it was asserting could not be
there. It now waits for the service to reach Stopped and for port 783 to refuse
connections, which is the condition the test actually depends on.

It also asserted "There was a communication error with SpamAssassin." alongside
the real error. That is error 5157, reported by SpamAssassinClient::OnReadError
when a connection was established and then lost mid-read. With spamd stopped
there is nothing to connect to, Connect() fails outright, and only error 5508 -
"The SpamAssassin tests did not complete..." - is reported. Asserting both
demanded a different failure mode from the one under test, so only 5508 remains.

Verified: AntiVirus.ClamAV 5/5 with a real detect-and-delete in the log
("Virus detected: Eicar-Test-Signature", "Message will be deleted"), and
AntiSpam.SpamAssassin 14/14 including the outage test.

README: the setup recipe pointed at a JAM Software download that no longer
exists. It now describes what actually works - Strawberry Perl plus CPAN, with
spamd rebuilt via BUILD_SPAMD=yes because the Windows build skips it by default,
and perl.exe copied to spamd.exe inside Perl's own bin so @inc still resolves.
Also states the elevation requirement the suite has always had but never
documented: it starts and stops Windows services, which a standard user cannot
do, and those tests report inconclusive unelevated.

No shipped code is touched by this commit; 6.2.10 as released is unaffected.

Author: chrisholloway5

README.md

@@ -547,10 +547,11 @@ If you want to run hMailServer in debug mode in Visual Studio, add the command a
 Running tests
 -------------
 
-hMailServer ships with a full regression suite (1026 NUnit tests) which exercises the server end to end over SMTP, IMAP and POP3  -  including anti-spam, anti-virus, TLS, DKIM/DMARC, rules, backup and the COM API. 6.2.10 passes the complete suite with zero failures. Tests reporting *inconclusive* are the ones whose optional integration is absent from the machine running them; see the list below.
+hMailServer ships with a full regression suite (1026 NUnit tests) which exercises the server end to end over SMTP, IMAP and POP3  -  including anti-spam, anti-virus, TLS, DKIM/DMARC, rules, backup and the COM API. 6.2.10 passes **all 1026 with zero failures and zero inconclusive**, with live SpamAssassin and ClamAV. Without those two integrations installed the 27 tests that depend on them report *inconclusive* rather than failing; the setup that makes them run is below.
 
 NOTE: When running tests, your local hMailServer installation will be updated with test accounts. Existing domains and accounts are deleted. Each tests prepares the server configuration in different ways. In other words, do not run the automated tests in an environment where you need to preserve hMailServer data.
 
+0. **Run the tests elevated.** The suite starts and stops Windows services - hMailServer itself, and the SpamAssassin service that the outage-handling test takes down on purpose - which a standard user cannot do. Unelevated, those tests report *inconclusive*.
 1. Make sure hMailServer.exe is built and can be run. The tests will launch the service.
 2. Open the test solution, `\hmailserver\test\hMailServer Tests.sln`
 3. In Visual Studio, select Test Explorer from the View-menu. 
@@ -561,8 +562,29 @@ You can also navigate to the source code for a test, right-click anywhere and se
 
 For 100% coverage the suite expects three optional integrations (tests degrade to *inconclusive* without them):
 
-   * **SpamAssassin**  -  the JAM Software Windows build (`https://downloads.jam-software.de/spamassassin/SpamAssassinForWindows-x64.zip`), extracted to `C:\SpamAssassin`, with `spamd.exe -i 127.0.0.1 -A 127.0.0.1 -p 783` running  -  ideally wrapped as a Windows service named `SpamAssassinJAM` so outage-handling tests can stop and start it.
-   * **ClamAV**  -  installed to `C:\clamav` with `clamd` listening on TCP 3310 and current freshclam definitions. Let the daemon finish loading signatures before the first run.
+   * **SpamAssassin**  -  the JAM Software Windows build the suite was originally written against
+     is long gone, so build it from CPAN instead. Install Strawberry Perl, then
+     `cpanm --notest Mail::SpamAssassin`. That skips `spamd` on Windows by default; rebuild it
+     from the unpacked distribution with `perl Makefile.PL BUILD_SPAMD=yes BUILD_SPAMC=no` and
+     `gmake install`, then fetch rules with `sa-update`.
+
+     Two details matter. The suite looks for a *process* named `spamd` before it falls back to
+     starting the service, and `spamd` is a Perl script - so copy `perl.exe` to `spamd.exe`
+     **inside `C:\Strawberry\perl\bin`**: Perl derives `@INC` from the location of its
+     executable, and a copy anywhere else finds no core modules at all. Run it as
+     `spamd.exe -T <path-to>\spamd --port 783 --listen 127.0.0.1 --round-robin --nouser-config`
+     (`-T` is required because it is on the script's `#!` line), wrapped as a Windows service
+     named `SpamAssassinJAM` so the outage-handling test can stop and start it.
+   * **ClamAV**  -  the official Windows build, extracted to `C:\clamav` (the suite launches
+     `C:\clamav\clamd.exe` by that exact path), with a `clamd.conf` setting `TCPSocket 3310`.
+     Bind both `127.0.0.1` and `::1`: hMailServer connects to it as `localhost`, which resolves
+     to either. Run `freshclam` first and let the daemon finish loading signatures before the
+     first run - it takes a while.
+
+     Note the virus test sends EICAR as a base64 **attachment**, not as the message body.
+     ClamAV's EICAR signatures match a whole file, so the string with anything around it is not
+     detected; as an attachment ClamAV decodes it back to exactly the EICAR file and matches,
+     which is also how a virus would really arrive.
    * **`AddXOriginalRcptTo=1`** in `hMailServer.INI` for the X-Original-Rcpt-To header tests.
 
 The complete dev-tree provisioning recipe (directories, certificates, DB scripts, runtime files) is kept with the maintainer's internal notes; open an issue if you need it to reproduce a build.

hmailserver/test/RegressionTests/AntiSpam/SpamAssassin.cs

@@ -3,7 +3,9 @@
 // Copyright (c) 2026 Christopher Holloway / Progressive Robot Ltd
 
 using System;
+using System.Net.Sockets;
 using System.ServiceProcess;
+using System.Threading;
 using hMailServer;
 using NUnit.Framework;
 using RegressionTests.Infrastructure;
@@ -202,7 +204,13 @@ public void TestSANotRunning()
 
          Assert.IsFalse(sMessageContents.Contains("X-Spam-Status"));
 
-         CustomAsserts.AssertReportedError("There was a communication error with SpamAssassin.",
+         // Only error 5508 belongs to this scenario. The "communication error with
+         // SpamAssassin" that used to be asserted alongside it is error 5157, which
+         // SpamAssassinClient::OnReadError reports when a connection was
+         // established and then lost mid-read. With spamd stopped there is nothing
+         // to connect to, so Connect() fails outright and that error never fires -
+         // asserting it demanded a different failure mode from the one under test.
+         CustomAsserts.AssertReportedError(
             "The SpamAssassin tests did not complete. Please confirm that the configuration (host name and port) is valid and that SpamAssassin is running.");
       }
 
@@ -307,18 +315,58 @@ public void TestWhiteList()
       }
 
 
+      /// <summary>
+      /// Stops SpamAssassin and does not return until it has actually stopped
+      /// serving. ServiceController.Stop() only asks: it returns as soon as the
+      /// request is accepted, while spamd is still up and still answering on 783.
+      /// The caller then sent a message straight into a live spamd and got the
+      /// X-Spam-Status header it was asserting could not be there.
+      ///
+      /// The service reaching Stopped is necessary but not sufficient - a wrapped
+      /// spamd can outlive its own service by a moment - so this waits on the port
+      /// refusing connections, which is the condition the test actually depends on.
+      /// </summary>
       private static void StopSpamAssassin()
       {
-         // Check if we can launch it...
          try
          {
             var serviceController = new ServiceController("SpamAssassinJAM");
-            serviceController.Stop();
+            if (serviceController.Status != ServiceControllerStatus.Stopped)
+            {
+               serviceController.Stop();
+               serviceController.WaitForStatus(ServiceControllerStatus.Stopped, TimeSpan.FromSeconds(60));
+            }
          }
          catch (Exception)
          {
             Assert.Inconclusive("Unable to stop SpamAssassin process. Is SpamAssassin installed?");
          }
+
+         for (var i = 0; i < 120; i++)
+         {
+            if (!IsSpamAssassinAcceptingConnections())
+               return;
+
+            Thread.Sleep(250);
+         }
+
+         Assert.Inconclusive("SpamAssassin is still accepting connections on port 783 after being stopped.");
+      }
+
+      private static bool IsSpamAssassinAcceptingConnections()
+      {
+         try
+         {
+            using (var client = new TcpClient())
+            {
+               var result = client.BeginConnect("127.0.0.1", 783, null, null);
+               return result.AsyncWaitHandle.WaitOne(500) && client.Connected;
+            }
+         }
+         catch
+         {
+            return false;
+         }
       }
    }
 }

hmailserver/test/RegressionTests/AntiVirus/ClamAV.cs

@@ -1,3 +1,5 @@
+using System;
+using System.Text;
 using hMailServer;
 using NUnit.Framework;
 using RegressionTests.Infrastructure;
@@ -75,17 +77,65 @@ public void TestWithVirus()
          LogHandler.DeleteCurrentDefaultLog();
 
          var account1 = SingletonProvider<TestSetup>.Instance.AddAccount(_domain, "test@example.test", "test");
-         var firstPart = @"X5O!P%@AP[4\PZX54(P^)7CC)7}";
-         var secondPart = @"$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*";
-         SmtpClientSimulator.StaticSend(account1.Address, account1.Address, "Mail 1",
-            firstPart + secondPart);
+
+         SmtpClientSimulator.StaticSendRaw(account1.Address, account1.Address,
+            BuildMessageWithEicarAttachment(account1.Address));
 
          CustomAsserts.AssertRecipientsInDeliveryQueue(0);
          Pop3ClientSimulator.AssertMessageCount(account1.Address, "test", 0);
 
          var defaultLog = LogHandler.ReadCurrentDefaultLog();
          Assert.IsTrue(defaultLog.Contains("Connecting to ClamAV"));
-         Assert.IsTrue(defaultLog.Contains("Message will be deleted (contained virus Eicar-Signature)."));
+
+         // The signature name has changed over ClamAV releases - a bare EICAR file
+         // is Eicar-Test-Signature, the same file with a trailing CRLF is
+         // Eicar-Signature - so match the family rather than one exact name.
+         StringAssert.IsMatch(@"Message will be deleted \(contained virus Eicar[-\w]*\)\.", defaultLog);
+      }
+
+      /// <summary>
+      /// Builds a MIME message carrying EICAR as a base64 attachment.
+      ///
+      /// It has to be an attachment. ClamAV's EICAR signatures match a whole file,
+      /// not a substring: the 68-byte string on its own is Eicar-Test-Signature and
+      /// with a trailing CRLF it is Eicar-Signature, but the same string with any
+      /// other content around it is not detected at all. This test used to put
+      /// EICAR straight in the body of a non-MIME message, which older ClamAV
+      /// extracted as a scannable part and current ClamAV does not - so the message
+      /// scanned clean and was delivered. As an attachment ClamAV decodes the
+      /// base64 back to exactly the EICAR file and matches, which is also how a
+      /// virus would actually arrive.
+      ///
+      /// The string is assembled from fragments and encoded at run time so that
+      /// neither it nor its base64 form is ever a literal in this source file -
+      /// on-access virus scanners quarantine files that contain either.
+      /// </summary>
+      private static string BuildMessageWithEicarAttachment(string address)
+      {
+         var firstPart = @"X5O!P%@AP[4\PZX54(P^)7CC)7}";
+         var secondPart = @"$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*";
+         var encoded = Convert.ToBase64String(Encoding.ASCII.GetBytes(firstPart + secondPart));
+
+         return
+            "From: " + address + "\r\n" +
+            "To: " + address + "\r\n" +
+            "Subject: Mail 1\r\n" +
+            "MIME-Version: 1.0\r\n" +
+            "Content-Type: multipart/mixed; boundary=\"eicar-boundary\"\r\n" +
+            "\r\n" +
+            "--eicar-boundary\r\n" +
+            "Content-Type: text/plain\r\n" +
+            "\r\n" +
+            "Test message.\r\n" +
+            "\r\n" +
+            "--eicar-boundary\r\n" +
+            "Content-Type: application/octet-stream; name=\"eicar.com\"\r\n" +
+            "Content-Transfer-Encoding: base64\r\n" +
+            "Content-Disposition: attachment; filename=\"eicar.com\"\r\n" +
+            "\r\n" +
+            encoded + "\r\n" +
+            "\r\n" +
+            "--eicar-boundary--\r\n";
       }
    }
 }