LibreSSL ML-KEM compatibility fixes and OpenBSD hardening cleanup

[daviduhden]

Summary

This PR brings LibreSSL ML-KEM compatibility to parity with the OpenSSL branch intent, while keeping the OpenBSD hardening cleanup and staying rebased on top of openssl.

What changed

1) ML-KEM / LibreSSL compatibility

• switched transport/config/ratchet PQ gating from OPENSSL_PQ to OPENSSL_MLKEM
• enabled OPENSSL_MLKEM for LibreSSL >= 4.3.0
• kept ML-DSA-specific paths under OPENSSL_MLDSA
• fixed ML-KEM mapping/indexing for LibreSSL variants (768/1024)
• added safe fallback paths for unsupported ML-KEM variants on LibreSSL
• added targeted NTCP2/SSU2 diagnostics for PQ selection and handshake framing

2) NTCP2 upstream behavior alignment

• restored intentional ClientLogin() random SessionRequest size variation (upstream behavior), to avoid divergence in handshake shape.

3) OpenBSD hardening cleanup

• preserved the minimal pledge/unveil cleanup included in this branch.

4) Added deterministic PQ test coverage

• added tests/test-postquantum.cpp
• wired into tests/Makefile as test-postquantum
• validates ML-KEM encapsulation/decapsulation roundtrip

Validation performed in this environment

Toolchain/runtime:

• clang++ (Homebrew LLVM)
• Homebrew LibreSSL 4.3.2
• Homebrew OpenSSL 3.6.2

Checks completed:

• build and run with LibreSSL
• build and run with OpenSSL
• test-postquantum passed on both LibreSSL and OpenSSL builds
• long runtime windows with ntcp2.version=4 and ssu2.version=4

Current production-readiness status

• deterministic ML-KEM encaps/decaps coverage is now in-tree and passing
• SSU2 runtime remains healthy
• NTCP2 runtime still shows intermittent SessionCreated read error in public-peer conditions, in patterns comparable to current upstream runtime behavior

Given the above, this PR is prepared for preview/review as the compatibility-correct baseline and keeps parity with upstream behavior while improving LibreSSL correctness and testability.

[orignal]

AI slope will not be accepted anyway

[daviduhden]

Why upstream openssl implementation is failing in this LibreSSL PQ4 scenario

Sharing a concrete diagnosis from runtime and code-path analysis.

@wipedlifepotato (original implementation context)

1) Capability gating mismatch

Upstream PQ transport paths are primarily guarded by OPENSSL_PQ (OpenSSL 3.5 capability bucket), while LibreSSL exposes ML-KEM separately.

Result on LibreSSL: PQ transport behavior is either partially disabled, inconsistently gated, or enters fallback paths that are not equivalent to full PQ4 support.

2) Feature coupling problem (OPENSSL_PQ vs per-feature gating)

OPENSSL_PQ couples KEM + signature assumptions. On LibreSSL we need ML-KEM transport enablement without implicitly requiring ML-DSA behavior.

This is why ML-KEM transport paths must be gated by OPENSSL_MLKEM, and signature-specific paths by OPENSSL_MLDSA.

3) Runtime evidence in PQ4 mode

In real peer runtime with ntcp2.version=4 / ssu2.version=4:

• selection reaches PQ peers (server=4, remote=4, crypto type for ML-KEM-768),
• but NTCP2 repeatedly fails after SessionRequest with SessionCreated read error.

That means it is not enough to “advertise/select PQ”; interoperability is still not stable end-to-end.

4) Why this blocks production readiness

Production-complete PQ4 requires at least:

• stable NTCP2 PQ session establishment,
• stable SSU2 PQ session establishment,
• successful encapsulation/decapsulation paths observed in live handshake outcomes.

Current upstream behavior does not satisfy that on LibreSSL in this test matrix.

---

I updated this branch to move transport gating to OPENSSL_MLKEM and separate ML-DSA gating, which fixes local correctness and improves diagnostics, but final production readiness still requires stable cross-peer PQ handshake completion.

[daviduhden]

@orignal thanks for the clarification about the intentional random SessionRequest sizing in ClientLogin().

I aligned this branch with that behavior again (restored the random NTCP2 PQ downgrade branch for size variance) and re-ran long runtime checks in this environment (Homebrew + clang) for both TLS backends.

Build/runtime environment used

• clang++ (Homebrew LLVM)
• Homebrew LibreSSL 4.3.2
• Homebrew OpenSSL 3.6.2
• ntcp2.version=4, ssu2.version=4, loglevel=debug
• 320s runtime windows on public peers

Results after restoring random SessionRequest size variation

Variant	NTCP2 connected	NTCP2 crypto6	NTCP2 crypto4	NTCP2 SessionCreated read errors	SSU2 established	SSU2 PQ4 effective	SSU2 PQ2 effective
LibreSSL	79	28	51	18	19	0	11
OpenSSL	92	36	56	26	17	1	14

Current status

• Conflicts with openssl base are resolved and PR is merge-clean.
• SSU2 transport behavior is healthy.
• NTCP2 still shows repeated SessionCreated read error in real-peer runtime, so I cannot call this production-complete for PQ4 yet.

I will keep this branch aligned with upstream behavior while continuing NTCP2 interoperability diagnosis.

[daviduhden]

@orignal update with the latest production-readiness work, while keeping this branch aligned with upstream/openssl.

1) Upstream alignment / conflicts

• Rebased on top of upstream/openssl and resolved conflicts.
• PR merge state is now clean.
• Restored intentional NTCP2 SessionRequest random size behavior in ClientLogin() (to stay aligned with upstream intent).

2) Added deterministic ML-KEM encaps/decaps coverage

I added a dedicated roundtrip test:

• tests/test-postquantum.cpp
• wired in tests/Makefile (test-postquantum)

Roundtrip test result in this environment:

• LibreSSL 4.3.2: pass (LIBRESSL_PQ_TEST_OK)
• OpenSSL 3.6.2: pass (OPENSSL_PQ_TEST_OK)

This gives deterministic proof of at least one successful encapsulation/decapsulation path.

3) Runtime evidence (320s windows, ntcp2.version=4, ssu2.version=4, clang + Homebrew)

Variant	NTCP2 connected	NTCP2 crypto6	NTCP2 crypto4	SessionCreated read errors	SSU2 established	SSU2 PQ4 effective	SSU2 PQ2 effective
LibreSSL (this branch)	79	28	51	18	19	0	11
OpenSSL (this branch)	92	36	56	26	17	1	14
OpenSSL (pure upstream binary)	101	n/a	n/a	30	27	n/a	n/a

Additional NTCP2 stability indicators on LibreSSL run:

• SessionConfirmed sent: 61
• Received message decrypted: 347
• I2NP blocks processed: 333
• ML-KEM native decapsulation successes observed repeatedly.

4) Status for preview

Given the deterministic ML-KEM roundtrip coverage plus sustained runtime traffic on both transports, this PR is now in preview-ready shape from my side.