Merge pull request #802 from dahlia/feat/bench/remaining-scenarios

Run remaining benchmark scenarios in `fedify bench`

Author: dahlia

CHANGES.md

@@ -304,8 +304,22 @@ To be released.
     discovery-aware `--dry-run` planning, and ships with a local benchmark
     fixture used by the scenario tests.  [[#744], [#783], [#784]]
 
+ -  Added `actor`, `object`, `fanout`, `failure`, and `mixed` scenario runners
+    to `fedify bench`.  Read scenarios can now benchmark actor and object
+    document fetches, including authenticated GET requests; fanout scenarios
+    drive the benchmark trigger endpoint and wait for queue task drain; failure
+    scenarios report expected fault outcomes as successes; and mixed scenarios
+    run weighted child scenario blends.  The `collection` scenario type remains
+    reserved but not executable.  Fanout and remote failure scenarios can set
+    `sinkBase` to generate deterministic benchmark sink inbox URLs for targets
+    that keep `triggerSinks` allowlisting enabled.  This change is published
+    as benchmark scenario schema version 2.  [[#744], [#785], [#801], [#802]]
+
 [#783]: https://github.com/fedify-dev/fedify/issues/783
 [#784]: https://github.com/fedify-dev/fedify/issues/784
+[#785]: https://github.com/fedify-dev/fedify/issues/785
+[#801]: https://github.com/fedify-dev/fedify/pull/801
+[#802]: https://github.com/fedify-dev/fedify/pull/802
 
 ### @fedify/fixture
 

docs/manual/benchmarking.md

@@ -94,9 +94,9 @@ delivery with the same `@fedify/fedify` signer a real peer uses, so the measured
 crypto cost is real.
 
 > [!NOTE]
-> This version runs the `inbox` and `webfinger` scenario types.  The scenario
-> format can express the others (`actor`, `object`, `fanout`, `collection`,
-> `failure`, and `mixed`), but they are not executed yet.  Within the runnable
+> This version runs the `inbox`, `webfinger`, `actor`, `object`, `fanout`,
+> `failure`, and `mixed` scenario types.  The `collection` scenario type is
+> reserved by the suite format but is not executed yet.  Within the runnable
 > types, a few options the format accepts are also not implemented yet and are
 > rejected up front with a clear message:
 >
@@ -115,7 +115,7 @@ is a superset).  The suite declares the `target`, shared `defaults`, the
 block of pass/fail thresholds:
 
 ~~~~ yaml
-# yaml-language-server: $schema=https://json-schema.fedify.dev/bench/scenario-v1.json
+# yaml-language-server: $schema=https://json-schema.fedify.dev/bench/scenario-v2.json
 version: 1
 target: http://localhost:3000
 defaults:
@@ -161,7 +161,56 @@ list, deliveries are rotated across the recipients (and across the synthetic
 `actors` signing them), modeling a server that receives from many peers into
 many local inboxes.
 
-[published schema]: https://json-schema.fedify.dev/bench/scenario-v1.json
+[published schema]: https://json-schema.fedify.dev/bench/scenario-v2.json
+
+### Scenario types
+
+The runnable scenario types cover the main benchmark surfaces:
+
+ -  `inbox`: discovers recipient inboxes and sends signed `Create(Note)`
+    deliveries through the target's inbound ActivityPub path.
+ -  `webfinger`: drives direct `/.well-known/webfinger` lookups on the target.
+ -  `actor`: resolves actor URLs from the scenario recipients and fetches actor
+    documents.  Set `authenticated: true` to sign those GET requests.
+ -  `object`: fetches object URLs from `source`.  Set `authenticated: true` to
+    sign those GET requests.
+ -  `fanout`: posts to `/.well-known/fedify/bench/trigger` so the target calls
+    `sendActivity()` and drains its fanout/outbox queue to benchmark-owned sink
+    inboxes.  The command starts those sink inboxes locally.  A non-loopback
+    target therefore needs `--advertise-host` unless the scenario sets
+    `sinkBase` to a reachable `http://host:port/` URL.  The target must either
+    allow the generated sink inboxes through `triggerSinks` or run with
+    `allowUnsafeTriggerRecipients` in a controlled benchmark environment.  Use
+    `sinkBase` when you want those inboxes to be deterministic, for example
+    `http://127.0.0.1:9090/inbox/0` through
+    `http://127.0.0.1:9090/inbox/4` for `followers: 5`.
+    `fedify bench` does not switch the target's queue backend; run the same
+    suite against targets configured with the queue implementations you want to
+    compare.  Fanout triggers are serialized while the runner observes queue
+    drain, so client latency includes time spent waiting for earlier fanout
+    drains under high-rate or concurrent load.  Use `deliveryThroughput` and
+    `queueDrain` expectations for delivery performance, and keep request
+    latency expectations conservative for this scenario type.
+ -  `failure`: records expected fault outcomes as successes.  For this
+    scenario type, `successRate` means “the expected failure was observed,”
+    not “the HTTP request succeeded.”  The `invalid-signature` and
+    `missing-actor` faults send malformed signed deliveries to a recipient
+    inbox.  The `remote-404`, `remote-410`, `slow-inbox`, and `network-error`
+    faults post to the benchmark trigger endpoint with `sender`, so the target
+    uses its normal outbound delivery path against controlled benchmark-owned
+    sink inboxes.  Like `fanout`, these remote failure faults need
+    `--advertise-host` for a non-loopback target unless `sinkBase` gives a
+    reachable, fixed sink base URL that the target's `triggerSinks` can
+    preconfigure.  Remote failure deliveries are also serialized while the
+    runner waits for the target's queue to
+    observe the expected failure or retry signal, so request latency can include
+    earlier wait time when the configured load is concurrent or high-rate.
+ -  `mixed`: runs referenced child scenarios concurrently, splitting the
+    `mixed` scenario's load by each entry's `weight`.  The referenced
+    scenarios are named scenarios in the same suite and are still run as normal
+    suite entries when listed.  The mixed result merges client-side request,
+    throughput, delivery throughput, latency, and error measurements;
+    server-side metric snapshots are not merged across child runners.
 
 ### Actors
 
@@ -239,7 +288,7 @@ CI check.  Keep CI gates on robust signals such as success rate, error counts,
 and gross throughput or latency floors; precise latency-percentile regression
 belongs in a controlled environment, not a shared CI runner.
 
-[report schema]: https://json-schema.fedify.dev/bench/report-v1.json
+[report schema]: https://json-schema.fedify.dev/bench/report-v2.json
 
 ### Safety
 
@@ -347,6 +396,11 @@ allowlist.  To bypass this guard for a controlled run, set
 `~FederationBenchmarkOptions.allowUnsafeTriggerRecipients` to `true` in the
 application configuration.
 
+For `fanout` and remote `failure` scenarios, set a `sinkBase` value such as
+`http://host:port/` in the scenario when the target keeps the safe default and
+you need stable sink URLs for `triggerSinks`.  With `followers: 5`, the runner
+generates `/inbox/0` through `/inbox/4` under that base.
+
 A successful trigger returns `202 Accepted`:
 
 ~~~~ json

packages/cli/src/bench/__fixtures__/invalid/failure-missing-fault.yaml

@@ -0,0 +1,10 @@
+# Server-side metrics are not merged for mixed scenario results.
+version: 1
+target: http://localhost:3000
+scenarios:
+  - name: realistic-blend
+    type: mixed
+    mix:
+      - { scenario: fanout-1k, weight: 1 }
+    expect:
+      queueDrain.p95: "< 2s"

packages/cli/src/bench/__fixtures__/invalid/mixed-server-metric.yaml

@@ -0,0 +1,9 @@
+# signatureVerification.* is only valid for authenticated object scenarios.
+version: 1
+target: http://localhost:3000
+scenarios:
+  - name: object-fetch
+    type: object
+    source: http://localhost:3000/objects/1
+    expect:
+      signatureVerification.p95: "< 10ms"

packages/cli/src/bench/__fixtures__/invalid/object-signature-expect-unauthenticated.yaml

@@ -1,6 +1,6 @@
 {
-  "$schema": "https://json-schema.fedify.dev/bench/report-v1.json",
-  "schemaVersion": 1,
+  "$schema": "https://json-schema.fedify.dev/bench/report-v2.json",
+  "schemaVersion": 2,
   "tool": { "name": "@fedify/cli", "version": "2.3.0" },
   "environment": {
     "runtime": "deno",

packages/cli/src/bench/__fixtures__/reports/inbox-report.json

@@ -1,6 +1,6 @@
 # yaml-language-server: $schema=https://json-schema.fedify.dev/bench/scenario-v1.json
-# Exercises every scenario type the format can express, even though only
-# `inbox` and `webfinger` have runners in this version.
+# Exercises every scenario type the format can express; `collection` is still
+# reserved but not executable in this version.
 version: 1
 target: http://localhost:3000
 defaults:

packages/cli/src/bench/__fixtures__/scenarios/all-types.yaml

@@ -0,0 +1,10 @@
+# yaml-language-server: $schema=https://json-schema.fedify.dev/bench/scenario-v2.json
+version: 1
+target: http://localhost:3000
+scenarios:
+  - name: signed-object-fetch
+    type: object
+    authenticated: true
+    source: http://localhost:3000/objects/1
+    expect:
+      signatureVerification.p95: "< 10ms"