diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9782b55a..4631ee03 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -127,8 +127,13 @@ jobs: file: coverage.cobertura.xml language: C# label: code-coverage-agent - # The upload needs "Code quality" enabled in repository settings - # (currently 403 without it). The coverage report is also kept as - # a build artifact, so a failed upload is a warning, not a failed - # pipeline. + # The upload needs the "Code quality" repository setting, and + # returns 403 without it. Tested on 22 Aug 2026: with the setting + # configured and NO languages the upload succeeds - but merely having + # the setting on re-exposes the 1,482 stale findings of the managed + # quality scan that was switched off on purpose (it ignored this + # repository's analysis configuration). That page being clean is + # worth more than this upload, so the setting stays off and the + # cobertura report kept as a build artifact is the coverage record. + # A failed upload is therefore a warning, not a failed pipeline. fail-on-error: false diff --git a/RELEASE.md b/RELEASE.md index d09d5abc..9fe4e91b 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -55,12 +55,16 @@ already cost a release cycle or nearly shipped a defect. `hMailServer.exe` and `publish\hMailCP.dll`. If `hMailServer.idl` changed in this range, regenerate the checked-in COM - wrapper after the server build and before the tools build - the command is in - `source\Tools\Interop\README.md`. A stale wrapper still compiles, which is why - this is easy to skip: the tools use a small stable subset of the API, so nothing - fails, and the members added this release are simply invisible to them. - Regenerate AFTER any interface-ordering fix, never before, or the old vtable - layout is baked into the shipped wrapper permanently. + wrapper after the server build and before the tools build: + `build egenerate-interop.ps1`. It runs TlbImp AND rewrites the wrapper's + SHA-256 and size in `hmailserver\docs hird-party-binaries.json` - the + binary-provenance workflow fails when those disagree, and regenerating by + hand without the manifest did exactly that twice in one day. A stale wrapper + still compiles, which is why this is easy to skip: the tools use a small + stable subset of the API, so nothing fails, and the members added this + release are simply invisible to them. Regenerate AFTER any interface-ordering + fix, never before, or the old vtable layout is baked into the shipped wrapper + permanently. 9. **Full regression suite on the stamped binary** — every test, nothing skipped. If *anything* changes after this run, the run is void: rebuild @@ -73,8 +77,23 @@ already cost a release cycle or nearly shipped a defect. 11. **Installer**: ISCC on `hMailServer64.iss`. Never run the installer on the dev machine — validation is the CI smoke-test workflow (`installer-smoke.yml`), which installs it on a throwaway runner. -12. **Commit** (as chrisholloway5, no co-author trailers), **tag** `vX.Y.Z`, - push branch then tag. Then publish **as a draft first** - this repository has +12. **Commit** (as chrisholloway5, no co-author trailers - history has been + rewritten once to remove them, and will be again). `master` is protected: + changes arrive by pull request, force-pushes and deletions are refused, and + the maintainer's bypass exists for emergencies, not for the release flow. + So push the working branch, open the PR and merge it with a rebase so the + history stays linear and the commits keep their own messages: + + ``` + git push origin server-fixes-wave + gh pr create --base master --head server-fixes-wave --fill + gh pr merge --rebase # self-merge is allowed; no reviewer is required + git pull --ff-only origin master + ``` + + Then **tag** `vX.Y.Z` on master and push the tag (tags matching `v*` are + protected too: no deletion, no rewrite - a tag is spent once). Then publish + **as a draft first** - this repository has immutable releases enabled, so a published release refuses every further asset upload and would be stuck with whatever it was created with: diff --git a/Roadmap.md b/Roadmap.md index abe6a43d..28c917ea 100644 --- a/Roadmap.md +++ b/Roadmap.md @@ -48,7 +48,7 @@ strong and where it is thin far more honestly than any prose summary. |---|--:|--:|--:|--:| | **Urgent — the OpenSSF gold badge** | | | | | | [What stands between here and gold](#what-stands-between-here-and-gold) | – | – | 9 | 3 | -| [Housekeeping that fell out of the badge work](#housekeeping-that-fell-out-of-the-badge-work) | – | – | 4 | – | +| [Housekeeping that fell out of the badge work](#housekeeping-that-fell-out-of-the-badge-work) | 1 | – | 2 | 1 | | [Dated items — the forcing functions](#dated-items--the-forcing-functions) | 2 | – | 4 | 1 | | [Defects found by the audit](#defects-found-by-the-audit) | 30 | – | – | – | | **The next generation** | | | | | @@ -78,7 +78,7 @@ strong and where it is thin far more honestly than any prose summary. | [Future-proofing: standards and protocols](#future-proofing-standards-and-protocols) | 8 | – | – | 2 | | [Future-proofing: platform and supply chain](#future-proofing-platform-and-supply-chain) | 5 | 1 | 2 | 2 | | [Future-proofing: deployment and operations](#future-proofing-deployment-and-operations) | 6 | 2 | 1 | – | -| **Total** | **729** | **13** | **70** | **18** | +| **Total** | **730** | **13** | **68** | **19** | Three things stand out and are worth naming rather than leaving to be inferred. **Storage, the administration surface and the core protocol layer are the @@ -166,8 +166,8 @@ small change rather than a project. | | Item | Detail | |:-:|---|---| -| ⬜ | **Interop manifest drift** | Regenerating `Interop.hMailServer.dll` after a COM API change without updating its SHA-256 in `hmailserver/docs/third-party-binaries.json` failed the binary-provenance check **twice on 21 Aug alone**. The regeneration step in `hmailserver/source/Tools/Interop/README.md` should rewrite its own manifest entry so the hash and the binary can never disagree. | -| ⬜ | **Repository setting: Code quality** | The CI coverage upload returns 403 until *Code quality* is enabled in repository settings; `ci.yml` treats the failure as a warning meanwhile. **Do not simply enable it**: the same setting drives GitHub's managed buildless quality scan, which was switched off deliberately on 21 August after it produced 1,482 findings that ignored the repository's analysis configuration (see the commit *The Code quality page was a second scanner disagreeing with the first*). First establish whether the feature can be enabled for coverage upload alone — `PATCH /repos/{o}/{r}/code-quality/setup` with no languages — and only then remove `fail-on-error: false`. If it cannot, the build artifact is the coverage record and the warning stays. | +| ✅ | **Interop manifest drift** | **Closed 22 August 2026.** Regenerating `Interop.hMailServer.dll` after a COM API change without updating its SHA-256 in `hmailserver/docs/third-party-binaries.json` failed the binary-provenance check twice on 21 August alone, after two different people had each done the first half correctly. Two steps that must always happen together are now one step: `build/regenerate-interop.ps1` runs TlbImp and rewrites the manifest entry's hash and size in place, refuses to run from a type library older than the IDL's last commit or its uncommitted edits (an mtime alone is not evidence - git bumps it on every checkout, which is how the first version of that guard refused a current library), and prefers the intermediate `.tlb` MIDL just wrote over a stale staged copy. `RELEASE.md` step 8 and `Interop/README.md` point at it. Verified by running it against a fresh Release build: manifest and disk agreed afterwards. | +| ⏸️ | **Repository setting: Code quality** | **Tested 22 August 2026 and settled.** The CI coverage upload needs the *Code quality* repository setting, and with the setting configured and *no languages* the upload does succeed (`PATCH /repos/{o}/{r}/code-quality/setup`, `{"state":"configured","languages":[]}`; CI run 32552896421). But the setting being on at all re-exposes the 1,482 stale findings of the managed buildless scan that was switched off on 21 August for ignoring the repository's analysis configuration — and with no languages configured nothing ever re-scans them closed. There is no API to dismiss findings. So the setting stays off, the cobertura build artifact is the coverage record, and `fail-on-error: false` stays with the reasoning written beside it. Revisit only if GitHub adds per-rule filtering or a dismissal API to the feature. | | ⬜ | **Repository setting: automatic dependency submission** | GitHub's own NuGet submission job is red on every push because it restores Windows-targeting projects on a Linux runner. It duplicates what `sbom.yml` already submits. Turn it off under *Dependency graph*, or accept a permanent red X. | | ⬜ | **Organisation setting: require 2FA** | The maintainer account uses a passkey; the org-wide requirement is still off (the API refused it — it would eject any member without 2FA, and the org has a second, read-only member). Enable it in *Authentication security* once that member is confirmed enrolled. | diff --git a/build/regenerate-interop.ps1 b/build/regenerate-interop.ps1 new file mode 100644 index 00000000..92b0e291 --- /dev/null +++ b/build/regenerate-interop.ps1 @@ -0,0 +1,135 @@ +# Copyright (c) 2026 Christopher Holloway / Progressive Robot Ltd and the hMailServer contributors +# SPDX-License-Identifier: AGPL-3.0-or-later + +<# + Regenerates the checked-in COM wrapper, Interop.hMailServer.dll, from the + Release type library - AND updates its entry in the binary-provenance + manifest, which is the half that kept getting forgotten. + + WHY ONE SCRIPT. The wrapper is TlbImp output, committed so the .NET tools + build with plain `dotnet build` on any machine. Every tracked binary is + listed in hmailserver/docs/third-party-binaries.json with its SHA-256, and + .github/workflows/verify-binary-provenance.yml fails the build when the + file on disk and the manifest disagree. Regenerating the wrapper by hand + and not touching the manifest failed that check twice on 21 August 2026 + alone - after two different people had each done the first half + correctly. Two steps that must always happen together are one step. + + WHEN. After any change to hMailServer.idl, and after building the server + in Release - the wrapper is read from the Release type library, and a + Debug-only build leaves it describing whatever Release last contained. + Regenerate AFTER any interface-ordering fix, never before: the wrapper + freezes the vtable layout it was generated from. + + A stale wrapper still compiles, which is what makes this easy to skip. + The tools use a small, stable subset of the API, so nothing fails; the + members added since the last regeneration are simply invisible to them. +#> + +[CmdletBinding()] +param( + [string] $TlbImp = 'C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8.1 Tools\x64\TlbImp.exe' +) + +$ErrorActionPreference = 'Stop' + +$root = (Get-Item (Split-Path -Parent $MyInvocation.MyCommand.Path)).Parent.FullName +$typeLibrary = Join-Path $root 'hmailserver\source\Server\hMailServer\x64\Release\hMailServer.tlb' +$intermediate = Join-Path $root 'hmailserver\source\Server\hMailServer\hMailServer\x64\Release\hMailServer.tlb' +$wrapper = Join-Path $root 'hmailserver\source\Tools\Interop\Interop.hMailServer.dll' +$manifestPath = Join-Path $root 'hmailserver\docs\third-party-binaries.json' +$manifestKey = 'hmailserver/source/Tools/Interop/Interop.hMailServer.dll' + +if (-not (Test-Path $TlbImp)) +{ + throw "TlbImp.exe not found at $TlbImp. It ships with the .NET Framework SDK (Windows SDK component)." +} + +# MIDL writes the type library into the intermediate directory; post-build.bat +# stages it into the output directory, but only when the post-build event +# runs, and build.ps1 suppresses build events. Take the newer of the two so a +# build that skipped staging cannot regenerate the wrapper from yesterday. +$source = $typeLibrary +if (Test-Path $intermediate) +{ + if (-not (Test-Path $typeLibrary) -or (Get-Item $intermediate).LastWriteTime -gt (Get-Item $typeLibrary).LastWriteTime) + { + $source = $intermediate + } +} + +if (-not (Test-Path $source)) +{ + throw "No Release type library found. Run build\build.ps1 -Configuration Release first - and if the IDL changed, delete the stale .tlb before building, because MSBuild has been seen to skip MIDL for one configuration while running it for the other." +} + +# Staleness guard. A file's mtime is not evidence on its own - git bumps it on +# every checkout and merge, which is how the first version of this guard +# refused to run against a type library that was in fact current. So the +# question is asked two ways, and either answer of "stale" is final: +# * the IDL has UNCOMMITTED edits newer than the type library, or +# * the IDL's last COMMIT is newer than the type library. +$idl = Join-Path $root 'hmailserver\source\Server\hMailServer\hMailServer.idl' +$idlRelative = 'hmailserver/source/Server/hMailServer/hMailServer.idl' +$libraryTime = (Get-Item $source).LastWriteTime + +& git -C $root diff --quiet HEAD -- $idlRelative +$idlDirty = ($LASTEXITCODE -ne 0) + +if ($idlDirty -and (Get-Item $idl).LastWriteTime -gt $libraryTime) +{ + throw "hMailServer.idl has uncommitted edits newer than the type library ($source). Rebuild the server in Release first." +} + +$lastCommitEpoch = [long](& git -C $root log -1 --format=%ct -- $idlRelative) +$lastCommitTime = [DateTimeOffset]::FromUnixTimeSeconds($lastCommitEpoch).LocalDateTime + +if ($lastCommitTime -gt $libraryTime) +{ + throw "hMailServer.idl was last committed at $lastCommitTime, after the type library was built ($libraryTime). Rebuild the server in Release first." +} + +Write-Host ("Generating {0}" -f $wrapper) +Write-Host (" from {0} (built {1})" -f $source, (Get-Item $source).LastWriteTime) + +& $TlbImp $source /out:$wrapper /namespace:hMailServer /machine:X64 /silent +if ($LASTEXITCODE -ne 0) +{ + throw "TlbImp returned $LASTEXITCODE." +} + +$hash = (Get-FileHash -LiteralPath $wrapper -Algorithm SHA256).Hash.ToLower() +$size = (Get-Item $wrapper).Length + +# Update the manifest in place with a text edit rather than ConvertTo-Json, so +# every other entry, the key order and the formatting survive byte-for-byte. +$manifest = Get-Content -LiteralPath $manifestPath -Raw +$pattern = '(?s)("path":\s*"' + [regex]::Escape($manifestKey) + '".*?"sha256":\s*")([0-9a-f]{64})(".*?"size":\s*)(\d+)' +$match = [regex]::Match($manifest, $pattern) + +if (-not $match.Success) +{ + throw "No manifest entry for $manifestKey in $manifestPath - add one before regenerating, so the provenance check can see it." +} + +$oldHash = $match.Groups[2].Value +$oldSize = $match.Groups[4].Value + +if ($oldHash -eq $hash) +{ + Write-Host (" unchanged: the manifest already records {0} ({1} bytes)" -f $hash.Substring(0, 12), $size) -ForegroundColor Green + exit 0 +} + +$updated = $manifest.Substring(0, $match.Groups[2].Index) + $hash + + $manifest.Substring($match.Groups[2].Index + $oldHash.Length, $match.Groups[4].Index - ($match.Groups[2].Index + $oldHash.Length)) + + $size + + $manifest.Substring($match.Groups[4].Index + $oldSize.Length) + +# Preserve the file's own line endings and the absence of a BOM. +[System.IO.File]::WriteAllText($manifestPath, $updated, (New-Object System.Text.UTF8Encoding($false))) + +Write-Host (" manifest: sha256 {0}... -> {1}..., size {2} -> {3}" -f $oldHash.Substring(0, 12), $hash.Substring(0, 12), $oldSize, $size) +Write-Host '' +Write-Host 'Regenerated and recorded. Commit the wrapper and the manifest together, then rebuild the tools:' -ForegroundColor Green +Write-Host ' build\build-tools.ps1' diff --git a/hmailserver/source/Tools/Interop/README.md b/hmailserver/source/Tools/Interop/README.md index d3e86762..3497ac5e 100644 --- a/hmailserver/source/Tools/Interop/README.md +++ b/hmailserver/source/Tools/Interop/README.md @@ -1,23 +1,33 @@ -# Interop.hMailServer.dll - -Checked-in tlbimp wrapper for the hMailServer COM API. The .NET tools -(DBSetup, DBSetupQuick, DBUpdater, DataDirectorySynchronizer, Shared, -ImportTool) reference this assembly directly so they build with plain -`dotnet build` on any machine — no registered typelib required. The -installer ships it to `{app}\Bin` for use by external .NET scripts. - -## Regenerating - -Regenerate whenever the COM API surface changes (`hMailServer.idl`), after -building the server: - -```powershell -& "C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8.1 Tools\x64\TlbImp.exe" ` - hmailserver\source\Server\hMailServer\x64\Release\hMailServer.tlb ` - /out:hmailserver\source\Tools\Interop\Interop.hMailServer.dll ` - /namespace:hMailServer /machine:X64 -``` - -A stale wrapper still builds and runs — the tools use a small, stable -subset of the API — but new COM members are invisible to them until the -wrapper is regenerated. +# Interop.hMailServer.dll + +Checked-in tlbimp wrapper for the hMailServer COM API. The .NET tools +(DBSetup, DBSetupQuick, DBUpdater, DataDirectorySynchronizer, Shared, +ImportTool) reference this assembly directly so they build with plain +`dotnet build` on any machine — no registered typelib required. The +installer ships it to `{app}\Bin` for use by external .NET scripts. + +## Regenerating + +Regenerate whenever the COM API surface changes (`hMailServer.idl`), after +building the server in Release: + +```powershell +build egenerate-interop.ps1 +``` + +That runs the TlbImp command below **and** updates this file's SHA-256 and +size in `hmailserver/docs/third-party-binaries.json`, which the +binary-provenance workflow checks on every push. Doing only the first half +by hand is how that check was failed twice on one day. The underlying +command, for reference: + +```powershell +& "C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8.1 Tools\x64\TlbImp.exe" ` + hmailserver\source\Server\hMailServer\x64\Release\hMailServer.tlb ` + /out:hmailserver\source\Tools\Interop\Interop.hMailServer.dll ` + /namespace:hMailServer /machine:X64 +``` + +A stale wrapper still builds and runs — the tools use a small, stable +subset of the API — but new COM members are invisible to them until the +wrapper is regenerated.