Add full-stack performance and resilience lab

Author: ammarheidari

README.md

@@ -19,8 +19,8 @@ Native Linux SCTP, external SCTP/M3UA, and cross-implementation full-stack MAP
 SMS traffic evidence are retained. Stateful M3UA, M2PA, SCCP, TCAP, and MAP SMS
 runtimes are available. Full stable-product readiness remains gated on
 operator/vendor profile interoperability, independent M2PA evidence,
-operator-sized performance evidence, production operations, trusted signing,
-and stable release execution.
+20K peak and operator-sized multi-host performance evidence, production
+operations, trusted signing, and stable release execution.
 
 For release history and governance, see:
 
@@ -55,6 +55,8 @@ The current RC engineering track focuses on:
   comparison evidence.
 - Cross-implementation MAP SMS traffic through TCAP, SCCP, M3UA, and native SCTP
   with retained PCAP, traces, field comparison, and digest evidence.
+- Full-stack performance and resilience tooling with a retained 62,000-operation
+  baseline, latency/resource metrics, SCTP failover, and zero-loss recovery.
 - Stateful SCCP, TCAP, and MAP SMS service layers.
 - M2PA as a parallel MTP2 path.
 - Byte-level tests and protocol validation.
@@ -74,6 +76,7 @@ The current RC engineering track focuses on:
 | SCCP | Stateful `ISccpService` supports UDT/XUDT/LUDT, GT translation, routing, bounded segmentation/reassembly, UDTS return policy, metrics, cancellation, and MTP3 receive ownership. Independent C-peer UDT evidence is retained; operator/vendor profile evidence remains. |
 | TCAP | Concurrent `TcapDialogueManager` provides transaction correlation, Begin/Continue/End/Abort, tracked invokes, Result/Error/Reject outcomes, shared timeout scanning, bounded queues, cleanup, snapshots, and metrics. Independent C-peer transaction evidence is retained; operator/vendor profile evidence remains. |
 | MAP | Stateful client/server SMS workflows cover SRI-SM, MO/MT ForwardSM, ReportSM-DeliveryStatus, and AlertServiceCentre with operation profiles, result/error correlation, typed dispatch, cancellation, and metrics. All five operations have cross-implementation repository-profile evidence; operator/vendor profile evidence remains. |
+| Performance | The retained WSL baseline completed 62,000 full-stack operations without protocol loss at about 13.5K sustained/peak TPS. The 20K peak and multi-host operator qualification gates remain open. |
 | Tooling | Byte-level tests, protocol diagnostics, trace comparison, and interoperability evidence are core project principles. |
 
 ---
@@ -164,6 +167,7 @@ Start here:
 - [MAP SMS service](docs/MAP.md)
 - [Phase 51 MAP SMS service](docs/PHASE51_MAP_SMS_SERVICE.md)
 - [Phase 52 end-to-end SS7 traffic lab](docs/PHASE52_END_TO_END_SS7_TRAFFIC_LAB.md)
+- [Phase 53 performance and resilience](docs/PHASE53_OPERATOR_PERFORMANCE_RESILIENCE.md)
 - [Interoperability and tooling](docs/INTEROPERABILITY.md)
 - [Compatibility policy](docs/COMPATIBILITY.md)
 - [Quality and contribution rules](docs/QUALITY.md)
@@ -220,9 +224,10 @@ Stable production support requires:
 - Stable API lifecycle validation.
 - Security, release, compliance, and operations review.
 
-The Linux SCTP, external M3UA, and repository-profile full-stack traffic gates
-are complete. Until the remaining gates are complete, the package should be
-treated as release-candidate infrastructure for controlled integrations.
+The Linux SCTP, external M3UA, repository-profile full-stack traffic, and
+controlled performance execution gates are complete. Until the remaining gates
+are complete, the package should be treated as release-candidate infrastructure
+for controlled integrations.
 
 ---
 

docs/COMMERCIAL_READINESS_REPORT.md

@@ -38,10 +38,22 @@ retained manifests instead of stale hard-coded flags.
   invokes and built five results. The retained PCAP contains 23 SCTP packets
   and ten M3UA DATA messages. The complete bundle is under
   `docs/evidence/PHASE52_END_TO_END_20260724T085858Z/`.
+- Full-stack performance run `performance-20260724T093659Z` completed 62,000
+  MAP SMS transactions through TCAP, SCCP, M3UA, and native SCTP without
+  protocol failure. Sustained, peak, and soak throughput were approximately
+  13.5K TPS; sustained/peak P95 remained below 15 ms; peak working set was
+  70 MB; and hot-path allocation was about 9.5 KB per transaction. Injected
+  association failure recovered in about 1.04 seconds and recovery traffic
+  completed without loss. Compressed PCAP, TShark fields, metrics, logs,
+  configuration, trace, comparison, report, and digests are retained under
+  `docs/evidence/PHASE53_PERFORMANCE_20260724T093659Z/`.
 
 ## Remaining Production Blockers
 
-- The current benchmark is real Linux SCTP peer traffic evidence, but it is single-host loopback. Do not use it for broad operator capacity claims until an operator-sized deployment benchmark is retained.
+- The current full-stack benchmark is real Linux SCTP peer traffic evidence,
+  but it is single-host WSL loopback. It reached about 13.5K TPS and did not
+  meet the 20K peak target. Do not use it for broad operator capacity claims
+  until a representative multi-host deployment benchmark passes.
 - Package publication evidence is closed for the public RC prerelease. Stable commercial release gates are foundation-complete, but live stable publication still requires retained stable release evidence, a completed protected stable publication run, and verified stable NuGet publication evidence.
 - Public/stable signing must use the organization's approved trusted certificate in the protected release environment; the current signing evidence is internal self-signed RC evidence.
 - Hosted GitHub provenance/SBOM attestations were skipped for dry-run and prerelease because private repository or organization attestation persistence can require a supported plan or public repository. These runs retained local provenance markers; stable runs keep hosted attestation reserved for the protected stable gate.
@@ -51,8 +63,8 @@ retained manifests instead of stale hard-coded flags.
   cross-implementation traffic evidence for the repository profile. A separate
   operator or vendor profile run is still required before broad SS7 network
   interoperability can be claimed.
-- The retained benchmark is not an operator-sized multi-host capacity or soak
-  result.
+- The retained soak contains 20,000 operations, not the long-duration
+  multi-host soak required for an operator capacity claim.
 
 ## Production Decision
 

docs/PHASE14_PERFORMANCE.md

@@ -74,3 +74,15 @@ Long-running benchmarks remain opt-in so normal pull-request validation stays fa
 ## Phase Status
 
 `SigtranPerformanceStatus.Describe()` summarizes the completed Phase 14 units and separates performance foundation readiness from production performance readiness.
+
+## Measured Full-Stack Baseline
+
+Phase 53 adds an executable native SCTP benchmark and retained run
+`performance-20260724T093659Z`. The run completed 62,000 MAP SMS transactions
+through TCAP, SCCP, and M3UA with no failures, measured approximately 13.5K TPS
+for sustained, peak, and soak stages, and restored traffic after association
+failure without loss.
+
+The run is a single-host WSL baseline and did not reach the 20K peak target. See
+[Phase 53](PHASE53_OPERATOR_PERFORMANCE_RESILIENCE.md) for metrics, evidence,
+and the remaining multi-host gate.

docs/PHASE33_PERFORMANCE_RESILIENCE_EVIDENCE.md

@@ -114,7 +114,11 @@ This keeps retained benchmark evidence separate from production claims. A report
 
 The final unit completes the status count and validates the phase with naming, package-neutrality, build, test, and pack checks. The final status capability is `final-sweep-validation`.
 
-Phase 33 is foundation-complete for performance and resilience evidence contracts. Production performance claims still require a retained real peer-traffic benchmark report and wider commercial readiness.
+Phase 33 is foundation-complete for performance and resilience evidence
+contracts. Phase 53 now supplies a retained 62,000-operation native SCTP
+full-stack baseline and an executable runner. Production performance claims
+still require a passing 20K peak result and representative multi-host,
+long-duration operator evidence.
 
 ## Validation
 

docs/PHASE53_OPERATOR_PERFORMANCE_RESILIENCE.md

@@ -0,0 +1,91 @@
+# Phase 53 - Operator-Sized Performance And Resilience
+
+Phase 53 replaces smoke timing with an executable full-stack benchmark over
+native Linux SCTP and an independent C peer.
+
+## Completed Units
+
+| Unit | Capability | Status |
+| --- | --- | --- |
+| 1 | Full-stack performance lab executable | Complete |
+| 2 | Configurable warmup and sustained stages | Complete |
+| 3 | Configurable peak and soak stages | Complete |
+| 4 | P50, P95, P99, and maximum latency | Complete |
+| 5 | CPU, working set, allocation, and Gen2 metrics | Complete |
+| 6 | Association failover and reconnect orchestration | Complete |
+| 7 | Recovery traffic and loss validation | Complete |
+| 8 | PCAP, trace, peer logs, metrics, and comparison artifacts | Complete |
+| 9 | Hot-path allocation and TCAP queue corrections | Complete |
+| 10 | Retained baseline, docs, build, test, and pack | Complete |
+
+## Run The Benchmark
+
+The runner requires Linux kernel SCTP, lksctp headers, GCC, .NET 10,
+`tcpdump`, `tshark`, and `gzip`.
+
+```bash
+SIGTRAN_ARTIFACT_ROOT="$HOME/sigtran-lab/artifacts" \
+SIGTRAN_WARMUP_OPERATIONS=1000 \
+SIGTRAN_SUSTAINED_OPERATIONS=20000 \
+SIGTRAN_PEAK_OPERATIONS=20000 \
+SIGTRAN_RECOVERY_OPERATIONS=1000 \
+SIGTRAN_SOAK_OPERATIONS=20000 \
+bash scripts/run-full-stack-performance-lab.sh
+```
+
+Counts, concurrency, addresses, point codes, and peer identity are environment
+configurable. Publication targets remain fixed in the lab report unless the
+executable is explicitly given alternative qualification arguments.
+
+## Runtime Corrections
+
+The first measured run exposed two production bottlenecks:
+
+- Native SCTP receive allocated a buffer as large as the maximum M3UA PDU for
+  every message, and send copied an already array-backed payload.
+- The optional TCAP dialogue observation queue could block correlated result
+  processing when no observation consumer was present.
+
+The native path now uses array-backed memory directly and falls back to
+`ArrayPool<byte>` only when required. `SCTP_NODELAY` is enabled by default
+through `NativeSctpTransportOptions.EnableNoDelay`.
+
+TCAP correlated processing no longer waits for the optional dialogue event
+consumer. A full observation queue drops the event and increments
+`TcapDialogueManagerMetrics.DroppedDialogueEvents`; inbound application Invoke
+components retain bounded backpressure.
+
+## Retained Baseline
+
+Run `performance-20260724T093659Z` completed on WSL2 kernel
+`6.18.33.2-microsoft-standard-WSL2` with .NET `10.0.9`, eight visible
+processors, native SCTP, packet capture, and the independent C peer.
+
+| Stage | Operations | Failed | TPS | P95 | P99 | Allocation |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| Warmup | 1,000 | 0 | 4,913.2 | 2.352 ms | 92.041 ms | 10,179 B/op |
+| Sustained | 20,000 | 0 | 13,515.3 | 7.230 ms | 8.419 ms | 9,627 B/op |
+| Peak | 20,000 | 0 | 13,491.5 | 14.471 ms | 16.380 ms | 9,585 B/op |
+| Recovery | 1,000 | 0 | 4,333.8 | 13.255 ms | 89.833 ms | 9,626 B/op |
+| Soak | 20,000 | 0 | 13,582.4 | 6.911 ms | 8.058 ms | 9,501 B/op |
+
+All 62,000 requests and responses were counted at M3UA, SCCP, and TCAP.
+Association recovery completed in `1,039.1 ms`, recovery traffic completed in
+`1,270.4 ms`, and no recovery operations were lost.
+
+The retained bundle under
+`docs/evidence/PHASE53_PERFORMANCE_20260724T093659Z/` includes compressed PCAP
+and TShark output, peer logs, host profile, SDK trace, structured metrics,
+configuration, comparison, report, and SHA-256 manifest.
+
+## Qualification Decision
+
+The controlled execution passed, including sustained load, failover, recovery,
+and soak. It did not satisfy the `20,000 TPS` peak target: observed peak
+throughput was `13,491.5 TPS`.
+
+This result is also single-host WSL loopback. It cannot support an
+operator-sized multi-host capacity claim. That claim requires a dedicated
+Linux host or VM pair, representative CPU limits, production network latency,
+long-duration soak, and a separately operated peer. The runner is ready for
+that environment without code changes.

docs/PHASE53_SUMMARY.md

@@ -0,0 +1,26 @@
+# Phase 53 Summary - Performance And Resilience
+
+Phase 53 implementation and controlled baseline execution are complete.
+
+## Delivered
+
+- Added `Sigtran.NET.PerformanceLab` and a repeatable Linux runner.
+- Added warmup, sustained, peak, recovery, and soak workloads.
+- Added latency percentiles, throughput, CPU, memory, allocation, GC, layer
+  counters, failover timing, reconnect count, and message-loss reporting.
+- Parameterized the independent C peer for long-running and quiet workloads.
+- Enabled `SCTP_NODELAY` through a documented native transport option.
+- Removed per-message 65KB native receive allocation and redundant send copies.
+- Prevented an unused TCAP observation stream from blocking correlated results
+  and exposed dropped observation events in metrics.
+- Retained a 62,000-operation baseline with no protocol failures and successful
+  association recovery.
+- Updated README, performance guidance, SCTP/TCAP docs, roadmap, readiness
+  report, and phase index.
+
+## Readiness Position
+
+The execution gate is complete and the SDK demonstrated approximately
+13.5K full-stack MAP SMS transactions per second in this WSL loopback profile.
+The 20K peak target and operator-sized multi-host evidence remain open. No
+stable operator-capacity claim should be made from the retained baseline.

docs/PHASE_INDEX.md

@@ -57,7 +57,7 @@ This index is the complete phase map for Sigtran.NET. It links each phase to the
 | 50 | TCAP dialogue manager | [Phase 50 TCAP Dialogue Manager](PHASE50_TCAP_DIALOGUE_MANAGER.md), [Phase 50 Summary](PHASE50_SUMMARY.md), [TCAP](TCAP.md) | Implementation complete; external peer evidence pending |
 | 51 | MAP SMS service | [Phase 51 MAP SMS Service](PHASE51_MAP_SMS_SERVICE.md), [Phase 51 Summary](PHASE51_SUMMARY.md), [MAP SMS](MAP.md) | Implementation complete; independent peer evidence pending |
 | 52 | End-to-end SS7 traffic lab | [Phase 52 End-To-End SS7 Traffic Lab](PHASE52_END_TO_END_SS7_TRAFFIC_LAB.md), [Phase 52 Summary](PHASE52_SUMMARY.md), [Interoperability](INTEROPERABILITY.md) | Complete for repository-profile cross-implementation traffic; operator/vendor profile evidence remains |
-| 53 | Operator-sized performance and resilience | Planned | Pending |
+| 53 | Operator-sized performance and resilience | [Phase 53 Performance And Resilience](PHASE53_OPERATOR_PERFORMANCE_RESILIENCE.md), [Phase 53 Summary](PHASE53_SUMMARY.md), [Performance Foundation](PHASE14_PERFORMANCE.md) | Implementation and controlled 62K-operation baseline complete; 20K peak and multi-host operator qualification pending |
 | 54 | Production operations package | Planned | Pending |
 | 55 | Stable release execution | Planned | Pending |
 
@@ -68,5 +68,5 @@ peer traffic, M3UA interoperability, RC SBOM/provenance/API baseline, and
 prerelease publication evidence. Phase 52 also retains cross-implementation
 M3UA/SCCP/TCAP/MAP SMS traffic evidence. Full product readiness remains blocked
 on operator/vendor profile validation, independent M2PA peer traffic,
-operator-sized performance, production operations, a trusted release identity,
-and protected stable publication.
+20K peak and multi-host operator performance, production operations, a trusted
+release identity, and protected stable publication.

docs/SCTP_TRANSPORT.md

@@ -221,3 +221,15 @@ Run `phase45-native-sctp-20260701T103951Z` passed this script on a Linux VM and
 `NativeSctpLab.CreateFromEnvironment()` keeps native SCTP integration verification opt-in through `SIGTRAN_NATIVE_SCTP_LAB=1`.
 
 `NativeSctpReadiness.GetReport()` marks the Phase 8 native SCTP foundation complete while keeping production readiness blocked until Linux verification passes.
+
+## Low-Allocation Native Traffic
+
+`NativeSctpTransportOptions.EnableNoDelay` enables `SCTP_NODELAY` by default for
+latency-sensitive signaling. Set it to `false` only when deliberate SCTP
+bundling is preferred and measured.
+
+The lksctp interop path uses array-backed `ReadOnlyMemory<byte>` and
+`Memory<byte>` directly. Non-array-backed buffers fall back to
+`ArrayPool<byte>` instead of allocating a maximum-PDU array for every receive.
+Phase 53 reduced measured full-stack allocation from approximately 77 KB to
+about 9.5 KB per MAP SMS transaction with these changes.

docs/SDK_ROADMAP.md

@@ -500,7 +500,15 @@ profile validation and independent M2PA traffic remain separate gates.
   separate hosts or VMs.
 - Report TPS, P95/P99 latency, CPU, memory, allocations, queue pressure, and loss.
 
-Status: Pending.
+Status: Implementation and controlled baseline execution are complete.
+`Sigtran.NET.PerformanceLab` and
+`scripts/run-full-stack-performance-lab.sh` execute warmup, sustained, peak,
+failover, recovery, and soak against the independent C/lksctp peer. Run
+`performance-20260724T093659Z` completed 62,000 full-stack transactions without
+loss, measured about 13.5K TPS across sustained/peak/soak, kept sustained and
+peak P95 below 15 ms, and restored traffic in about 1.27 seconds. The run did
+not satisfy the 20K peak target and is single-host WSL loopback, so
+operator-sized multi-host qualification remains open.
 
 ## Phase 54 - Production Operations Package
 

docs/TCAP.md

@@ -161,6 +161,12 @@ dedicated delay task for every invoke while preserving per-invoke timeout
 overrides. `SnapshotDialogues` and `GetMetrics` expose current state without
 returning mutable manager internals.
 
+The dialogue event stream is observational and cannot block correlated
+ReturnResult, ReturnError, or Reject processing. When its bounded queue is full,
+the manager drops the observation event and increments
+`TcapDialogueManagerMetrics.DroppedDialogueEvents`. Inbound Invoke components
+still use their bounded component queue and retain application backpressure.
+
 ## Allocation
 
 `TcapTransactionIdAllocator` and `TcapInvokeRegistry` provide deterministic allocation and duplicate detection for transaction and invoke identifiers.