[sophiecarreras]

Test-only change. Lets the credential-based integration tests take an optional realm so they can also run against a B2 simulator. When B2_REALM is unset nothing changes.

• liveRealmOptions in tests/helpers/live-b2.ts: {} unset, else { realm, allowedHostSuffixes: [] }, spread into each B2Client.
• Evidence: simulator only (not real B2): 32 pass, 5 fail, 12 skip.
• Question: is bypassing the SSRF guard under B2_REALM acceptable, or prefer a loopback exception?

Tracking: backblaze-labs/demand-side-ai#1556

🤖 Generated with Claude Code

[quality-keeper]

Quality Keeper — Advisory Review

Source: backblaze-labs/b2-sdk-typescript

Advisory only — this is not a required status check. Quality Keeper reviewed this repository snapshot and is reporting what it found. It changes no status and merges nothing; suggested scaffolds are starting points, not proof.

Quality Keeper assessed the repository, executed checks, and collected results, covering 19 quality expectations.

Five repository-quality areas

Hard expectations keep their exact Evidence State. Advisory applicability questions are separate. Implementation or suite presence is not proof of a trusted run.

Area	Hard expectations	Evidence States	Implementation status	Advisory questions
Observability	0	—	not_evaluated 1	0
Product Analytics	0	—	not_evaluated 1	0
User Feedback	0	—	not_evaluated 1	0
Reliability	3	pending_run 3	not_evaluated 3	0
Security	4	pending_run 2, verified 2	not_evaluated 4	0
Other registered quality	12	pending_run 10, verified 2	not_evaluated 12	0

Observability — 0 hard, 0 advisory

Can operators understand and diagnose repository-owned runtime behavior?

Hard expectations

None selected.

Applicability Review Gaps (advisory)

None.

Product Analytics — 0 hard, 0 advisory

Does repository-owned product analytics instrumentation behave safely and as declared?

Hard expectations

None selected.

Applicability Review Gaps (advisory)

None.

User Feedback — 0 hard, 0 advisory

Do repository-owned feedback and support-intake flows behave safely and accessibly?

Hard expectations

None selected.

Applicability Review Gaps (advisory)

None.

Reliability — 3 hard, 0 advisory

Does repository-owned software fail, recover, time out, and report health predictably?

Hard expectations

• Cancellation or idempotency controls need behavior testing — Testing Type reliability_resilience_testing — Evidence State pending_run
  • Why it applies: Production source declares a cancellation primitive or idempotency control, so its externally observable behavior needs repository-owned verification.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: No seed proposal — an operator must design acceptable evidence for this expectation.
• Outbound calling assets need timeout testing — Testing Type reliability_resilience_testing — Evidence State pending_run
  • Why it applies: Production source contains a recognized outbound network-client call site, so explicit timeout behavior needs repository-owned verification.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: No seed proposal — an operator must design acceptable evidence for this expectation.
• Published sdk packages need reliability testing — Testing Type reliability_resilience_testing — Evidence State pending_run
  • Why it applies: The interpreted asset is a published package or library with a detected external service or network boundary — consumers depend on predictable error classification and failure-path behavior, so those paths need verification against the declared public contract. Timeout and retry expectations are selected separately when their owned call sites or policies are detected.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: No seed proposal — an operator must design acceptable evidence for this expectation.

Applicability Review Gaps (advisory)

None.

Security — 4 hard, 0 advisory

Do static controls and repository-owned security behaviors protect the intended boundaries?

Hard expectations

• Authorization boundaries need permission behavior testing — Testing Type dynamic_security_testing — Evidence State pending_run
  • Why it applies: The interpreted asset contains a production authorization, role, permission, tenant, resource, or ownership boundary, so repository-owned tests must prove forbidden cross-boundary access is denied.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: No seed proposal — an operator must design acceptable evidence for this expectation.
• Credential handling assets need non exposure testing — Testing Type dynamic_security_testing — Evidence State pending_run
  • Why it applies: The interpreted asset contains production credential, token, password, API-key, redaction, or sanitization handling, so repository-owned tests must prove sensitive values are not logged, returned, or durably exposed.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: No seed proposal — an operator must design acceptable evidence for this expectation.
• Code and configuration assets need static security analysis — Testing Type static_security_analysis — Evidence State verified
  • Why it applies: The interpreted asset contains code, configuration, dependencies, secrets coordinates, infrastructure-as-code, or service boundaries, so static security analysis is required before runtime evidence is trusted.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains verified; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / current
  • Partial/unbound proof: implementation detection not evaluated
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: None — already verified before this run.
• Published packages need provenance proof — Testing Type static_security_analysis — Evidence State verified
  • Why it applies: The interpreted asset is a published package with source code, so published release artifacts should carry provenance metadata (npm provenance, SLSA attestation, or sigstore signature) to protect the supply chain.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains verified; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / current
  • Partial/unbound proof: implementation detection not evaluated
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: None — already verified before this run.

Applicability Review Gaps (advisory)

None.

Other registered quality — 12 hard, 0 advisory

Registered Testing Types not assigned to one of the five focused areas.

Hard expectations

• Packaged or runnable assets need build verification — Testing Type build_verification — Evidence State pending_run
  • Why it applies: The interpreted asset has package, script, workflow, UI, service, CLI, edge-runtime, or game-runtime signals, so it needs a reproducible build or equivalent construction check.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: None — exercised this run.
• Published packages need sdk build artifact review — Testing Type build_verification — Evidence State pending_run
  • Why it applies: The interpreted asset is a published package with source code, so the distribution manifest completeness, pack tarball integrity, and declared file list need verification as part of the release artifact.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: None — exercised this run.
• Consumer or platform boundaries need compatibility testing — Testing Type compatibility_testing — Evidence State pending_run
  • Why it applies: The interpreted asset is published or consumed across environments (published package, exported API, command-line tool, or edge target), so compatibility across platforms, versions, and consumers needs verification.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: None — exercised this run.
• Published typescript packages need type declaration check — Testing Type compatibility_testing — Evidence State pending_run
  • Why it applies: The interpreted asset is a published TypeScript package (tsconfig.json or has_types in package.json detected), so generated .d.ts declaration files and type resolution must be verified from a consumer project's perspective.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: None — exercised this run.
• Every asset needs configuration review — Testing Type configuration_artifact_review — Evidence State verified
  • Why it applies: Every interpreted asset needs a static review of its declared files, manifests, configuration, and artifacts so Quality Keeper can explain its shape before evidence is harvested.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains verified; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / current
  • Partial/unbound proof: implementation detection not evaluated
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: None — already verified before this run.
• Public or network boundaries need contract testing — Testing Type contract_testing — Evidence State pending_run
  • Why it applies: The interpreted asset publishes or exposes a boundary that other software depends on (network API, CLI, package exports, or an importable Go external client), so the public interface contract needs verification.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: It was selected but produced no result — opt in (e.g. --include-expensive) or fix the command; do not just re-select it.
• Published sdk packages need integration testing — Testing Type integration_testing — Evidence State pending_run
  • Why it applies: The interpreted asset is a published package or library that creates an integration boundary for consumers, so the package's integration paths, API contracts, and external service behaviors need integration-level verification.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: It was selected but produced no result — opt in (e.g. --include-expensive) or fix the command; do not just re-select it.
• Every asset needs discoverability review — Testing Type metadata_discoverability_review — Evidence State verified
  • Why it applies: Every interpreted asset needs enough metadata and documentation for humans and systems to identify what it is and how to work with it.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains verified; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / current
  • Partial/unbound proof: implementation detection not evaluated
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: None — already verified before this run.
• Documented assets need specification conformity review — Testing Type specification_conformity_review — Evidence State pending_run
  • Why it applies: The asset carries documentation, so documented commands, setup instructions, examples, and claims need to stay aligned with the implementation.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: None — exercised this run.
• Published packages with docs need example validation — Testing Type specification_conformity_review — Evidence State pending_run
  • Why it applies: The interpreted asset is a published package with detected documentation, so code examples in README, docs, or examples directories should be compiled or type-checked to confirm they match the actual public API.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: None — exercised this run.
• Cli sdk iac assets need system testing — Testing Type system_testing — Evidence State pending_run
  • Why it applies: The interpreted asset is a CLI, published package/SDK, or IaC provider that makes outbound calls to a real external dependency, so its own entry point needs an end-to-end test of that real flow — not just unit or integration coverage of individual pieces.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: It was selected but produced no result — opt in (e.g. --include-expensive) or fix the command; do not just re-select it.
• Code assets need unit testing — Testing Type unit_testing — Evidence State pending_run
  • Why it applies: The interpreted asset contains runnable or reusable source code, so the smallest testable units need isolated verification.
  • Implementation presence: not_evaluated
  • Implementation explanation: Implementation detection was not evaluated because no applicable standards-owned detection contract was available. Evidence State remains pending_run; if verified, that verdict still comes only from Evaluate's accepted evidence.
  • Applicability signals: none recorded
  • Behavior suite / source: not required by the selected criteria
  • Native GitHub result: not observed
  • Structured summary: not_observed
  • Trusted run: not required by the selected criteria
  • Revision / freshness: 8a27b890b1064fcbb36440d4403221aad14cde9f / unknown
  • Partial/unbound proof: implementation detection not evaluated; Evidence State remains pending_run
  • Owner / target: backblaze-labs/b2-sdk-typescript / src:fa9511dc2356b03d
  • Next: None — exercised this run.

Applicability Review Gaps (advisory)

None.

Result

• 11 expectations are covered by acceptable evidence.
• 8 still need attention — Quality Keeper cannot count them as covered yet.

Covered

Quality Keeper found acceptable evidence for one or more expectations in these Testing Types.

• Build Verification — 2 expectations covered (Ran and passed).
  Confirms the software actually builds/packages into a runnable or publishable artifact.
  • What Quality Keeper ran:
    • pnpm run build → passed @. (asset scope) — declared in package.json
    • pnpm run typecheck → passed @. (asset scope) — declared in package.json
    • Full command output/logs are in the run artifacts (see Proof below).
• Compatibility Testing — 2 expectations covered (Ran and passed).
  Checks the software works across the consumers, platforms, or versions it claims to support.
  • What Quality Keeper ran:
    • pnpm run verify:release → passed @. (asset scope) — declared in package.json
    • Full command output/logs are in the run artifacts (see Proof below).
• Configuration & Artifact Review — 1 expectation covered (Verified before execution).
  Checks that config files, manifests, and build artifacts are present, well-formed, and say what they should.
• Metadata & Discoverability Review — 1 expectation covered (Verified before execution).
  Checks that a repository or package is findable and understandable — description, topics, README, keywords.
• Specification Conformity Review — 2 expectations covered (Ran and passed).
  Checks that documented assets actually match the specification or documented behavior they claim.
  • What Quality Keeper ran:
    • pnpm run lint:docs → passed @. (asset scope) — declared in package.json
    • Full command output/logs are in the run artifacts (see Proof below).
• Static Security Analysis — 2 expectations covered (Verified before execution).
  Scans code and configuration for security problems without running it — secrets, unsafe patterns, misconfigurations.
  • What Quality Keeper ran:
    • pnpm run audit:deps → passed @. (asset scope) — declared in package.json
    • pnpm run test:audit:deps → passed — 7 passed @. (asset scope) — declared in package.json
    • Full command output/logs are in the run artifacts (see Proof below).
• Unit Testing — 1 expectation covered (Ran and passed).
  Exercises individual functions or modules in isolation.
  • What Quality Keeper ran:
    • pnpm run test → passed — 2628 passed, 2 skipped @. (asset scope) — declared in package.json
    • pnpm run test:all → passed — 2628 passed, 2 skipped @. (asset scope) — declared in package.json
    • Full command output/logs are in the run artifacts (see Proof below).

Needs attention

Quality Keeper does not yet have acceptable evidence for these Testing Types.

Some have partial checks that need to run, be fixed, or be connected. Others need new proof or seeded tests. In both cases, Quality Keeper cannot count them as covered yet.

• Contract Testing
  Checks that a public or network boundary (API, SDK surface, schema) matches its agreed shape.
  • Blocked before execution: expensive/invasive check gated for safety; opt in explicitly with --include-expensive to run it
• Dynamic Security Testing
  Exercises security behavior before release or against a staged runtime, including access controls, secret safety, secure defaults, and sanitized audit records.
• Integration Testing
  Checks that two or more components work together — a module plus a real dependency, service, or datastore.
  • Blocked before execution: expensive/invasive check gated for safety; opt in explicitly with --include-expensive to run it
• Reliability & Resilience Testing
  Checks that the software behaves predictably when dependencies fail, time out, degrade, or recover — without mistaking retry libraries, health routes, or alert files for proof.
• System / End-to-End Testing
  Drives a real user or system flow through the whole application, end to end.
  • Blocked before execution: expensive/invasive check gated for safety; opt in explicitly with --include-expensive to run it

Quality Keeper changes no Evidence State. Pending, missing, executed-but-not-exercised, and review items are not passing, and seeded scaffolds are next actions, not evidence, until applied and re-evaluated.

Full breakdown (19 expectations)

Area	Status	Expectation	Why it applies
Build Verification	Ran and passed	packaged_or_runnable_assets_need_build_verification	The interpreted asset has package, script, workflow, UI, service, CLI, edge-runtime, or game-runtime signals, so it needs a reproducible build or equivalent construction check.
Build Verification	Ran and passed	published_packages_need_sdk_build_artifact_review	The interpreted asset is a published package with source code, so the distribution manifest completeness, pack tarball integrity, and declared file list need verification as part of the release artifact.
Compatibility Testing	Ran and passed	consumer_or_platform_boundaries_need_compatibility_testing	The interpreted asset is published or consumed across environments (published package, exported API, command-line tool, or edge target), so compatibility across platforms, versions, and consumers needs verification.
Compatibility Testing	Ran and passed	published_typescript_packages_need_type_declaration_check	The interpreted asset is a published TypeScript package (tsconfig.json or has_types in package.json detected), so generated .d.ts declaration files and type resolution must be verified from a consumer project's perspective.
Configuration & Artifact Review	Verified before execution	every_asset_needs_configuration_review	Every interpreted asset needs a static review of its declared files, manifests, configuration, and artifacts so Quality Keeper can explain its shape before evidence is harvested.
Contract Testing	Selected but not executed	public_or_network_boundaries_need_contract_testing	The interpreted asset publishes or exposes a boundary that other software depends on (network API, CLI, package exports, or an importable Go external client), so the public interface contract needs verification.
Dynamic Security Testing	Missing — no proof yet	authorization_boundaries_need_permission_behavior_testing	The interpreted asset contains a production authorization, role, permission, tenant, resource, or ownership boundary, so repository-owned tests must prove forbidden cross-boundary access is denied.
Dynamic Security Testing	Missing — no proof yet	credential_handling_assets_need_non_exposure_testing	The interpreted asset contains production credential, token, password, API-key, redaction, or sanitization handling, so repository-owned tests must prove sensitive values are not logged, returned, or durably exposed.
Integration Testing	Selected but not executed	published_sdk_packages_need_integration_testing	The interpreted asset is a published package or library that creates an integration boundary for consumers, so the package's integration paths, API contracts, and external service behaviors need integration-level verification.
Metadata & Discoverability Review	Verified before execution	every_asset_needs_discoverability_review	Every interpreted asset needs enough metadata and documentation for humans and systems to identify what it is and how to work with it.
Reliability & Resilience Testing	Missing — no proof yet	outbound_calling_assets_need_timeout_testing	Production source contains a recognized outbound network-client call site, so explicit timeout behavior needs repository-owned verification.
Reliability & Resilience Testing	Missing — no proof yet	cancellation_or_idempotency_controls_need_behavior_testing	Production source declares a cancellation primitive or idempotency control, so its externally observable behavior needs repository-owned verification.
Reliability & Resilience Testing	Missing — no proof yet	published_sdk_packages_need_reliability_testing	The interpreted asset is a published package or library with a detected external service or network boundary — consumers depend on predictable error classification and failure-path behavior, so those paths need verification against the declared public contract. Timeout and retry expectations are selected separately when their owned call sites or policies are detected.
Specification Conformity Review	Ran and passed	documented_assets_need_specification_conformity_review	The asset carries documentation, so documented commands, setup instructions, examples, and claims need to stay aligned with the implementation.
Specification Conformity Review	Ran and passed	published_packages_with_docs_need_example_validation	The interpreted asset is a published package with detected documentation, so code examples in README, docs, or examples directories should be compiled or type-checked to confirm they match the actual public API.
Static Security Analysis	Verified before execution	code_and_configuration_assets_need_static_security_analysis	The interpreted asset contains code, configuration, dependencies, secrets coordinates, infrastructure-as-code, or service boundaries, so static security analysis is required before runtime evidence is trusted.
Static Security Analysis	Verified before execution	published_packages_need_provenance_proof	The interpreted asset is a published package with source code, so published release artifacts should carry provenance metadata (npm provenance, SLSA attestation, or sigstore signature) to protect the supply chain.
System / End-to-End Testing	Selected but not executed	cli_sdk_iac_assets_need_system_testing	The interpreted asset is a CLI, published package/SDK, or IaC provider that makes outbound calls to a real external dependency, so its own entry point needs an end-to-end test of that real flow — not just unit or integration coverage of individual pieces.
Unit Testing	Ran and passed	code_assets_need_unit_testing	The interpreted asset contains runnable or reusable source code, so the smallest testable units need isolated verification.

---

New to these terms? Start with the Quality Keeper Standards Wiki to learn what each testing type means, why it applies, and what counts as evidence.

[sophiecarreras]

Test-only: B2_REALM is optional and changes nothing when unset. Evidence is simulator only (32 pass, 5 fail, 12 skip), not real B2. One question for you: is bypassing the SSRF guard under B2_REALM acceptable, or would you prefer a loopback exception? Local typecheck/lint/test are green on current main.

Update: the commit message is now a single-line subject without a body or trailer, as CONTRIBUTING.md asks. The tree is identical to before; I force-pushed (with lease) to this PR branch only, new head a96ea2c. CI on the new head: 37 success, 4 skipped, 0 failing.