CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

Unreleased

Breaking

Changes

• CI
  • The Agave toolchain install retries, and a failed one now fails the job. Eight workflow steps across five workflows ran sh -c "$(curl -sSfL .../install)" once with no retry, so a transient reset from release.anza.xyz killed a job before it ran anything. That form also swallowed a failed fetch: the command substitution comes back empty, sh -c "" exits 0, and the step passed having installed nothing. The eight are now one composite action at .github/actions/solana-toolchain that fetches and runs as separate steps, checks solana --version actually runs, clears partial state between attempts, bounds every wait so a stalled handshake or hung transfer reaches the backoff instead of sitting until the job times out, and backs off. Each caller keeps the version it used before; solana.yml and offchain.local-validator.yml are on v3.0.12 and the other three on v3.0.4, which is drift worth settling separately.
• Serviceability
  • write_stake_mirror in the instruction crate and WriteStakeMirrorCommand in the Rust SDK, so the relayer has a caller-side surface for the instruction A6 added. The command departs from its neighbours in one way: every other one uses append_payer_permission_account, which attaches the caller's Permission account only when it already exists, because a legacy GlobalState key might authorize instead. No legacy key satisfies STAKE_ORACLE, so a missing Permission account is always fatal here, and the command says so locally rather than sending a transaction that can only come back NotAllowed. It checks what authorize checks, not just ownership: an account that does not decode, a suspended Permission, and one lacking the flag are each refused with the reason named, which matters because suspending a Permission is what revoking the relayer's key looks like.
  • New WriteStakeMirror instruction (variant 119) and a STAKE_ORACLE permission, which is what a relayer will call to copy a builder's Solana stake onto the DZ ledger. Nothing could write a StakeMirror before this. The instruction refuses a write whose source_slot is not newer than the stored one: polling makes a repeated write harmless but says nothing about ordering, and a retry carrying an older read could otherwise walk the mirror backwards, so the program enforces it rather than trusting the caller. It also refuses to reassign a mirror to a different builder, since the builder is not a PDA seed, and it carries feed_key forward rather than taking it from the caller, because CreateFeed writes that to spend the stake and zeroing it would let one bond back two feeds. No legacy GlobalState key maps to STAKE_ORACLE, so a holder needs a real Permission account even while require-permission-accounts is clear. Gated on allow-staked-feeds with the rest of RFC-28. STAKE_ORACLE is grantable through doublezero permission set --add stake-oracle, named by permission audit and bitmask_to_names, listed in AUTHORIZE_GATED_FLAGS and in the Go SDK's flag constants. It is the first flag no legacy GlobalState key can satisfy, which the audit's own test now asserts through a PERMISSION_ONLY_FLAGS list rather than treating as a gap in the enumeration.
  • Feed carries the RFC-28 stake terms: builder, stake_ref, spec_id, sla_hash, committed_rate_bits_per_sec and a lifecycle status. Setting them needs the new allow-staked-feeds feature flag, which no cluster has, so CreateFeed refuses a builder until the stake mirror and the attestor exist; without the flag the instruction behaves as before. The fields are appended rather than versioned, so a feed written before this decodes with them defaulted and try_acc_write resizes the account on the next update. status is the exception to defaulting: a short account reads Active, because reading it as Pending would pull every live catalog feed out of service. The rate is bits per second, not basis points, which is what bps means elsewhere in DoubleZero.
  • New StakeMirror account (AccountType 19), one per stake, holding the stake a relayer saw on Solana: tier, the rate the builder declared, the source slot, and the key that vouched for it. The PDA is seeded by stake_ref, the builder-stake account it mirrors, not by the builder: RFC-28 collateralizes each feed on its own bond, so a builder-keyed mirror would let two feeds pass the coverage check against one stake. The DZ ledger cannot read a Solana account, so this is an assertion rather than a proof, and relayer records whose assertion it is. StakeTier::None is the default and covers no rate, so an unwritten mirror and an absent one are the same answer to a reader. Nothing writes one yet; who is allowed to is the open trust-anchor question.
  • CreateFeed refuses a staked feed whose committed rate the stake tier does not cover, reading the tier from the stake's StakeMirror, and refuses a stake that already backs a feed. Creating a feed claims the stake by writing the feed's key onto the mirror, which is what makes RFC-28's one feed per stake true; a relayer updating a mirror has to carry feed_key forward rather than zero it. The mirror account is found by matching its PDA rather than by position, so a caller that sends one is not obliged to also send a Permission account and a caller that sends neither is unaffected. Four errors rather than one generic argument failure: StakeMirrorMissing (119) when the account is left out, StakeDoesNotCoverRate (120) when the stake is absent or too small, StakeAlreadyBacksFeed (121) when it is already spent, and StakedFeedCannotBeDeleted (122) because DeleteFeed now refuses a staked feed: closing it would leave the claim pointing at an account that no longer exists, so the bond would back nothing and still refuse to back anything else. Retirement is the path out and releases the stake with it, both in the lifecycle work.
  • A feed admits a subscriber only while its status is Active. SubscribeFeed and CreateSubscribeUser both check it where the seat is spent, so a feed that is pending conformance, halted, or retired stops taking subscribers the moment its status changes, and retiring one needs no sweep over the access passes already holding a seat for it. The check is deliberately not in the shared coverage path, which UnsubscribeFeed also runs: gating there would leave a user holding a seat on a retired feed with no way to release it. New error FeedNotActive (123). Live subscribers are unaffected until they disconnect; evicting them is separate work.
  • Test coverage for the RFC-28 publish-rights path, which needs no new instruction. A feed's multicast groups are created with owner set to the builder, and AddMulticastGroupPubAllowlist authorizes on mgroup.owner == payer, so the builder grants its own publish rights without the catalog admin that created the feed. The test walks it with two distinct signers.
• SDK
  • The Rust SDK sends program logs and errors through the log facade instead of straight to stderr. A failed transaction's program logs now arrive as one error! record carrying the signature, so a service embedding the SDK can capture, filter and ship them. Four eprintln! sites bypassed logging entirely, while the same function logged the successful path at debug level: an operator saw every transaction that worked and got raw stderr for one that failed. RFC-20 already asks modules to use the standard log macros. One behavior change for the CLI: --log-level off silences program logs, which previously printed whatever the level (#4299).
  • sdk/shreds/go carries the feed subscription program's FeedDistribution account: how much USDC one feed collected for one calendar month. The program is a second program alongside shred subscription and had nothing in this SDK, so each consumer decoded the account at fixed byte offsets itself, lake included. The account is a bytemuck Pod read here field by field, which agrees with the Pod bytes only because the field order leaves no interior padding; TestStructSizes pins the 120-byte total and a new test pins every field against a real mainnet account. Client is built around one program ID and so gains no fetch method, and DeserializeFeedDistribution is exported for a caller that makes its own getProgramAccounts call. make sdk-test never ran ./sdk/shreds/go/..., so this package's layout pins have never run in CI; it runs them now. (#4216)
  • The TypeScript and Python Feed deserializers read the RFC-28 tail and synthesize Active for an account that carries no status byte, matching the Rust program. New feed_legacy fixture covers that path alongside the updated feed fixture.
• Solana programs (solana/)
  • builder-stake carries its own instruction builders in instruction::builders, moved out of the test harness now that a caller outside the crate needs them. Each one fixes the account order the processor expects, so a change there has one place to update rather than one per caller.
  • builder-stake exposes its processor module and try_process_instruction under the existing entrypoint feature, so a test in another crate can load the program natively through processor! rather than building it to BPF first. doublezero-serviceability already exposes its own for the same reason. The crate's own tests still run against the .so, and nothing is exposed to a consumer that does not ask for the feature.
  • builder-stake holds a bond for six months and returns the excess after. The first bond starts the hold and a later one does not restart it, so a repricing that forces a top-up cannot push a builder's withdrawal date out. Withdraw returns anything above the stake's requirement to a token account the builder names, and refuses both before the hold elapses and below the requirement, which is what stops a builder walking its bond out from under a live feed. The hold is 180 days rather than calendar months, because a month has no fixed length and the alternative is calendar arithmetic onchain to move a one-off boundary by at most three days. A stake that has never been bonded has no hold, and a zero expiry means the hold has not started rather than that it ended in 1970. Every instruction taking a stake checks the account is at the address its own fields derive, which the zero-copy reader does not do. A SetHoldExpiry instruction lets a devnet demo show a withdrawal without waiting: it exists in every build and refuses outside a development one, rather than sitting behind a #[cfg] that would give the two binaries different instruction encodings for the same bytes.
  • New builder-stake program at dzbschFChpPoWihZFdnYjyzHJicZwPHb6QTntHjhLki, holding the 2Z bond a builder posts before deploying a feed under RFC-28. A BuilderStake PDA, a 2Z token account owned by it, and InitializeProgram, SetAdmin, ConfigureProgram, InitializeBuilderStake and PostBond. A bond rather than a deposit: it is returnable after the hold and forfeitable by slashing, and deposit carries neither. The address is keyed on (builder, stake_index) rather than the builder alone, because RFC-28 collateralizes each feed on its own bond and a builder-only address would cap a builder at one stake for life. The program starts paused, so a deployment with no admin and no tier table holds nothing. No slash instruction yet: the burn authority is what makes this its own deployable, and writing it before the verdict signer is settled means writing it twice. Bond sizing and the six-month hold are not here either.
  • builder-stake sizes a bond against the rate its feed commits to. An admin sets three 2Z amounts, one per RFC-28 rate tier; RFC-28 quotes the tiers in dollars but fixes the bond in 2Z at the price prevailing when the tier is set, so this is a table rather than a price feed. The rate ceilings are compiled in, because they have to agree with StakeTier in the serviceability program. A stake's requirement follows the tier table while the stake is short and stops moving once it is funded, so repricing cannot under-fund a builder who already paid in full, and cannot be dodged by pre-creating stakes for the cost of rent and funding them after a rise. A table with a hole, or one where more rate costs less, is refused.

v0.39.0 - 2026-09-04

Breaking

• CLI
  • Remove revenue-distribution convert-2z and harvest-2z (malbeclabs/infra#2527).
  • Validator deposit no longer accepts --convert-2z-limit-price.
• SDK
  • Remove the Go, Python, and TypeScript revenue-distribution clients that fetched /swap-rate.

Changes

• Monitor
  • Stop polling the SOL/2Z swap oracle (malbeclabs/infra#2527).
  • -twoz-oracle-interval still parses so existing ansible extra args do not fail the process.
• CLI
  • revenue-distribution fetch sol-conversion no longer requests a swap quote.
• SDK
  • The TypeScript and Python GlobalState deserializers expose ip_verifier_authority_pk, the RFC-27 trust root the Go SDK and the Rust state already carried, so those consumers can read which key signs IP ownership proofs. The field is appended, so an account written before the upgrade decodes it as the default pubkey rather than failing. (#4231)
• CI
  • The e2e matrix runs 5 round-robin shards instead of 4. Shards are filled by test count, not by duration, so the heaviest one was carrying ~3735s of tests against a 15-minute job timeout and was cancelled mid-run; the extra shard brings the worst case back to ~3022s. Adding e2e (shard 6) to the required status checks in the main ruleset is a separate, manual step.
  • shreds-e2e pins one heavy test to its own shard instead of three. TestE2E_MultiUserInstantAllocationAndWithdrawal and TestE2E_DeviceScale no longer exist in doublezero-shreds, so the pin validation failed every run and the matrix was never built. Only TestE2E_FeedSubscriptionOracleExpiryTeardown stays pinned, leaving 1 pinned + 2 round-robin shards. Dropping shard-e2e (shard 4) and shard-e2e (shard 5) from the required status checks in the main ruleset is a separate, manual step — until it happens those contexts are required but never reported.
  • .cursor/BUGBOT.md and .github/copilot-instructions.md now tell Bugbot and Copilot to read the nearest sibling, flag a path that skips a zero or a duplicate, and assert a specific error and the exact log line at the expected index. Onchain checks apply only when the repository has onchain code. The eight path-scoped files under .github/instructions/ are removed so Copilot reads only the repo-wide file. (#4247)
  • The e2e shard step passes GITHUB_TOKEN to go test. TestE2E_BackwardCompatibility asks the GitHub API which client releases exist, and unauthenticated that is 60 requests an hour shared across every runner on the same address, so shard 1 failed on a 403 rate limit before reaching a devnet. The test already read the variable; nothing was setting it.
  • The 11 workflows imported with doublezero-offchain and doublezero-solana are folded into this repo's set, and the imported .github directories are deleted. GitHub only reads workflows at the repository root, so those files did nothing where they sat.
  • Five release workflows come across, one per component: contributor-rewards, doublezero-solana-cli, the offchain sentinel, solana-validator-debt and the offchain scheduler. They run goreleaser from the repository root rather than from offchain/, because the merged workspace writes to the root target/ directory, which is where goreleaser's rust builder looks for the binary it packages. Every path inside the five goreleaser configs is rewritten to match, and release.github.name moves from doublezero-offchain to doublezero, so the releases land on this repository. The three secrets they need are already configured here.
  • New solana workflow. The solana/ tree is excluded from the root workspace, so rust.yml never reaches it and it would otherwise have no CI here at all. It runs that tree's own lint, library tests, docs and SBF tests, plus the checksum gate that rebuilds both networks' artifacts and verifies them against solana/programs/sha256sums_*.txt, path-scoped to solana/**.
  • New elixir workflow for the offchain scheduler, path-scoped to offchain/scheduler/**: format check, compile with warnings as errors, credo and tests. Those checks only ran on a release tag before, so a pull request that broke the scheduler passed.
  • New offchain-local-validator workflow carrying the two live fork tests. It runs from the repository root, since the shell scripts resolve their binaries as target/debug/<name>. Every cargo call names its package, because the root workspace sets default-members = [] and a bare cargo build --bin selects nothing. It is path-scoped to the trees that can affect it rather than running on every pull request as it did before. The two jobs that were disabled or commented out upstream are not carried over.
  • rust-cli-static now asserts static musl linkage for all five released CLIs instead of the client alone, which is what offchain's rust-musl-static job did for its four. Each package is built on its own so feature unification matches the release.
  • changelog-reminder keeps the per-crate changelog check that offchain enforced, with paths moved under offchain/. A change to one of those 13 subprojects needs both the root CHANGELOG.md and that subproject's own.
  • Offchain's ci.yml is dropped: make rust-build, make rust-lint and make rust-test cover those crates now that they are workspace members. Its just ci coverage floor (cargo llvm-cov --fail-under-lines 25) is not carried over, since the workspace it measured no longer exists. offchain/Justfile keeps its Elixir recipes and loses the Rust ones, which would otherwise act on the whole workspace with a different fmt and clippy line than CI uses.
  • offchain/scripts/release-rc.sh resolves the repository root two levels up rather than one, so local release candidates build against the merged workspace.
  • The three contributor-rewards tests that set and remove REWARDER_KEYPAIR_PATH are marked #[serial]. That variable is process-wide, and cargo test runs tests as threads in one process, so they raced: one test removing it while another was between setting it and reading it made the read fail. It never fired in doublezero-offchain, whose CI ran cargo nextest, which gives each test its own process.
• E2E/QA
  • New e2e coverage for RFC-27 proof enforcement with require-ip-ownership-proof set: the working path still reaches BGP, a client with no verifier to reach is rejected with IpOwnershipProofRequired, a wildcard (0.0.0.0) access pass binds client_ip only when a proof is attached, the sentinel authority stays exempt so the oracle path keeps working, and connect refuses a proof whose address disagrees with the one it provisions. (#4243)
  • Remove TestQA_MulticastSettlement. It funded a seat through doublezero-solana shreds pay, which is going away. The agent seat-pay RPC now returns Unimplemented if something still calls it. Unused settlement helpers go with the test. (#4248)
• Offchain
  • The scheduler refuses to boot when SOLANA_RPC is unset or empty, rather than handing nil or an empty string to the Rust NIF. All three workers read the one config key, so the check sits in config/runtime.exs where the single cause was. The test environment is exempt, since the tests set the key themselves. DZ_LEDGER_RPC no longer sets a ledger_rpc key that nothing in the application read. (#4240 follow-up)
  • The scheduler README closes its shell code block, so the Installation heading and everything after it stop rendering inside it.
• Solana programs
  • Rename test_lifetime_swept_2z_amount to match the lifetime_swapped_2z_amount field and method it covers, so a search for the swapped-amount tests finds it.
• Repo
  • The offchain crates join the root Cargo workspace. offchain/Cargo.toml, offchain/Cargo.lock and offchain/rust-toolchain.toml are gone, so all 15 crates build on this repo's 1.97.1 toolchain against one lockfile, and the ten path dependencies resolve inside a single workspace. The solana/ tree stays excluded with its own lockfile and its 1.91 toolchain: 62 of the 96 crates in the programs' build closure resolve differently here, which would change the compiled bytes and break solana/programs/sha256sums_*.txt. (#4240)
  • Each offchain crate now states edition = "2024" and its own version. Both were inherited from the offchain workspace, so folding them in would have moved all 15 crates to edition 2021, where they do not build, and reversioned four of them from 0.0.1 to 0.38.0.
  • bincode and reqwest are declared per crate in the offchain tree rather than taken from the workspace. This repo is on bincode 2, whose API differs, and on reqwest with its default TLS backend, while the offchain crates use bincode 1 and rustls with the OS root store. Unifying either is a code change, not a manifest change.
  • The e2e sentinel binary is renamed from doublezero-sentinel to dz-e2e-sentinel. Offchain ships a deb of the former name, and two members of one workspace cannot write the same file into target/release. The e2e base image and the sentinel image follow the new name.
  • The e2e base image builds its four binaries by name instead of --workspace, so it no longer compiles the offchain crates and their arrow, parquet and AWS dependencies on the way to a client and a sentinel.
  • The revdist fixture generator reaches doublezero-program-tools and doublezero-revenue-distribution by path into solana/ instead of by an unpinned git dependency, matching the other three generators. Nothing there floats on the next cargo update any more.
  • Offchain's release profile (lto = true, codegen-units = 1) is not carried over. Cargo profiles are workspace-wide, so keeping it would apply full link-time optimization to every release build in this repo, including the ones e2e waits on. The offchain release binaries lose that optimization, which is accepted rather than deferred.
  • smartcontract's test-sbf target runs from each program's directory, the way build-programs already does, instead of once from smartcontract/. An unscoped cargo test-sbf resolves the whole workspace against the platform-tools rustc, which reports itself as 1.89.0-dev, and the workspace now carries crates declaring a higher minimum: aws-sdk-s3 at 1.94.1 and rustler at 1.91, both reached through the offchain crates and neither in any program's dependency graph. Serviceability runs before telemetry, whose test helper loads the serviceability .so from target/deploy.
  • smartcontract's test-sbf target names its four program packages instead of running unscoped. An unscoped cargo test-sbf resolves the whole workspace against the platform-tools rustc (1.89.0-dev), and the workspace now carries crates declaring a higher minimum, aws-sdk-s3 at 1.94.1 and rustler at 1.91, both reached through the offchain crates. Neither is in any program's dependency graph. The sibling test-programs and lint-programs targets were already package-scoped, which is why only this one failed.
  • The imported offchain/ tree catches up with the three pull requests that landed in doublezero-offchain after the import point: doublezero-solana shreds pay is removed (withdraw, list, payments and price stay), the fund-seat instruction is removed from the offchain Solana SDK while shreds payments keeps reading leftover fund-seat transactions, and the solana-cli crate moves to 0.5.12. Author, date and message come across unchanged, with the pull request references named the way the import named its own. solana/ needed no sync: its upstream tip is the imported tip.
  • offchain/CONTRIBUTING.md is removed and its code of conduct moves to the root README. It told contributors to fork doublezero-offchain and open pull requests and issues there, which stops being true when that repository is archived, and the rest of it duplicated the root's contributing guidance.
  • DEVELOPMENT gains a section on the two imported trees, naming the parts the root make targets do not cover: the Solana L1 programs, which keep their own workspace and toolchain, and the Elixir scheduler.

v0.38.0 - 2026-08-28

Breaking

• SDK
  • UpdateMulticastGroupRolesCommand.group_pk: Pubkey becomes group_pks: Vec<Pubkey> and CreateSubscribeUserCommand.mgroup_pk: Pubkey becomes mgroup_pks: Vec<Pubkey> (non-empty; the first entry is the instruction's primary group). The RFC-26 builders update_multicast_group_roles and create_subscribe_user gain an extra_groups: &[Pubkey] parameter and derive the new extra_group_count arg from it. Single-group callers pass a one-element vec / empty slice. CreateSubscribeUserCommand measures the built transaction's wire size and rejects a group set that cannot fit under the 1232-byte limit, naming how many groups do fit: the create also carries the device's dz_prefix accounts and an optional feed, so the 16-group role-update chunk does not bound it. (malbeclabs/infra#2114)

Changes

• CI
  • The new release-bump-dry-run job dry-runs the release version bump on every PR, so a cargo update --workspace dependency rebind fails on the PR that causes it instead of a week later at release time, as it did for v0.37.0 (#4213, fixed in #4219). Advisory until its context is added to the main ruleset. (#4220)
  • Add .cursor/BUGBOT.md to enable Cursor review. This is an experiment. (malbeclabs/infra#2387)
• Repo
  • The ten git dependencies that offchain/ carried on malbeclabs/doublezero and malbeclabs/doublezero-solana become path dependencies into this repo. All ten version pins stop existing, so the two trees can no longer disagree about which revision of a shared crate they build, and a change spanning them lands in one commit instead of a pin bump per consumer. network-shapley-rs stays a git dependency, since it lives in another organization. (#4245)
• Repo
  • doublezero-offchain and doublezero-solana are imported into this repo, with their history and all 126 of their release tags, under new top-level offchain/ and solana/ directories. Nothing that was already here moves. Both trees stay excluded from the root Cargo workspace and keep their own Cargo.toml, Cargo.lock and rust-toolchain.toml, so they build exactly as they did in their own repos, and their workflows do not run yet. Imported commit messages are rewritten so that a reference that meant a pull request in a source repo now names that repo explicitly. .github/dependabot.yml enumerates its cargo directories instead of globbing them, so no update lands in a tree this repo does not build. The source repos are still the ones that release. (#4240)
• CLI
  • doublezero connect with no mode provisions everything the server's AccessPass authorizes in one run — the IBRL tunnel plus a multicast tunnel joined to the groups or purchased feeds the pass grants — instead of requiring the operator to know their entitlements and issue connect ibrl and connect multicast separately. Each mode is reported on its own line; one the pass does not cover is skipped with the reason rather than failing the run. The two are attempted independently, so a failure in one keeps the other's work and names the command that finishes the rest, and the run exits non-zero if an attempted mode failed or the pass authorized nothing. Epoch expiry still gates unicast only, so an expired pass connects multicast and skips IBRL. The bare form also enables the reconciler up front, so a run that provisions nothing still leaves the daemon managing tunnels, and takes --tenant/--allocate-addr for its IBRL half (connect ibrl keeps its own positional tenant and -a). connect ibrl and connect multicast are unchanged.
  • doublezero balance takes an optional address, so doublezero balance <pubkey> reports that account's balance while the bare form keeps reporting the configured keypair's. Querying another account needs no local keypair. An address that was never funded prints 0 Credits instead of failing the account lookup.
  • New doublezero transfer <RECIPIENT> <AMOUNT> sends credits from the configured keypair to another account on the DoubleZero Ledger, mirroring solana transfer: AMOUNT is a credit amount, or ALL to send the whole balance minus the transaction fee. A recipient that does not exist yet is created by the transfer, with the amount raised to the rent-exempt minimum when it falls below it, so no opt-in flag is needed for an unfunded recipient. A transfer that would leave the sender holding a nonzero balance below that same minimum is refused up front, with the largest payable amount named, since the runtime would reject it.
  • doublezero connect Multicast with N groups is one transaction in the common case (all groups sharing one publisher/subscriber flag pair fold into the create, skipping the activation wait); doublezero multicast subscribe|unsubscribe|publish|unpublish, doublezero user subscribe, and the role-strip cleanup in user delete/request-ban batch their role changes by flag pair, chunked to 16 groups per transaction. Failure reporting in the multicast verbs is per batch: a failed batch lists every group it carried, since none was applied. (malbeclabs/infra#2114)
  • doublezero feed update|delete --force-unsubscribe strips each user's orphaned groups with one batched role update per user (chunked to 16 groups per transaction) instead of one transaction per group. (malbeclabs/infra#2114)
  • doublezero connect obtains an RFC-27 IP ownership proof from the verification service and attaches it to user creation; the address the service observes is authoritative, and where it disagrees with the daemon's own discovery connect stops and names both. A verifier that is unreachable, unconfigured, or that declines is reported and the connect continues without a proof, which the program accepts until require-ip-ownership-proof is set; --ip-verifier-url or DZ_IP_VERIFIER_URL points at one, and no environment has a built-in default yet. (#4201)
• Client
  • From-source builds per client/INSTALL.md now succeed. client/Makefile defaulted CARGO_FLAGS to empty, so make build produced target/debug/doublezero while make install copied from target/release/doublezero, which never existed; CARGO_FLAGS now defaults to --release so the two agree. make install also called Debian-only addgroup/adduser, which are absent on RHEL/Rocky/Amazon Linux (in the documented support matrix), failing with addgroup: command not found; it now uses the portable groupadd/useradd with equivalent flags. (#4175)
• Serviceability
  • UpdateMulticastGroupRoles (58) and CreateSubscribeUser (59) accept additional writable MulticastGroup accounts (counted by a new borsh-incremental extra_group_count: u8 arg), so subscribing a user to N groups is one atomic transaction instead of N: one signature/fee, and a failure rolls back every group. Each batch member is authorized exactly like a single-group call (per-group allowlist checks; in CreateSubscribeUser, EdgeSeat extras are coverage-checked against the single passed feed and the seat still ticks once per user per feed, so a seat tick can no longer outlive a partial subscription). Duplicate group accounts in a batch are rejected. Old encodings without the count byte decode as 0, so existing clients are unaffected. Deploy ordering (RFC-1): the program must deploy to all clusters before any client that emits batches — an old program would misread the extra group accounts as the trailing optional accounts. (malbeclabs/infra#2114)
• SDK
  • Append the payer Permission account on doublezero feed create, feed delete, feed update, user delete, user update, and multicast role updates (UpdateMulticastGroupRoles, including the role-strip that user delete / request-ban and feed update|delete --force-unsubscribe run first) when that account exists and the serviceability program owns it. (malbeclabs/infra#2343)
  • The revdist fixture generator pulls doublezero-program-tools and doublezero-revenue-distribution from malbeclabs/doublezero-solana, which moves there from the doublezerofoundation org. The pinned commit does not change, so the generator resolves the same code.
  • CreateUserCommand and CreateSubscribeUserCommand take an optional RFC-27 ip_proof, which attaches the native Ed25519SigVerify instruction the program looks for and sends both as one transaction, with the verifier key read from GlobalState.ip_verifier_authority_pk. A proof naming a different owner, address, or user type is refused before the transaction is paid for; omitting it produces the pre-RFC-27 transaction unchanged. (#4200)
  • DoubleZeroClient gains send_instructions, for a transaction that needs more than one instruction. send_transaction is unchanged. (#4200)
• Utility crates
  • doublezero-serviceability-instruction gains ip_proof::ed25519_verification_instruction and ip_proof::with_ed25519_verification, which lay out the Ed25519 precompile instruction for a proof and pair it with a user-creation instruction. With a proof attached a CreateUser transaction still fits 10 dz_prefix_block accounts and CreateSubscribeUser 8. (#4200)

v0.37.0 - 2026-08-21

Breaking

Changes

• Monitor
  • The serviceability watcher's epoch estimates now use a per-chain slot time instead of 400ms everywhere: 350ms for Solana mainnet, 200ms for Solana testnet (which DoubleZero devnet also dials), and 400ms for the DoubleZero ledger. A 432,000-slot Solana epoch was over-estimated by ~6 hours on mainnet and ~24 hours on testnet in the previous_epoch_start / next_epoch_start log fields. The mainnet value must move to 200ms once Solana finishes its 200ms slot rollout. (malbeclabs/infra#2319)
• RFCs
  • RFC-27: IP Ownership Verification Service for user connection
• Serviceability
  • GlobalState carries ip_verifier_authority_pk, the RFC-27 trust root for IP ownership proof validation, which SetAuthority and doublezero global-config authority set --ip-verifier-authority <pubkey|me> rotate without a program upgrade. (#4196)
  • CreateUser and CreateSubscribeUser validate an optional RFC-27 IpOwnershipProof, verified through the native Ed25519 precompile and signed by globalstate.ip_verifier_authority_pk, so a caller can no longer bind a client_ip it cannot originate traffic from. Enforcement is gated on the new require-ip-ownership-proof feature flag: while it is clear a missing proof is accepted, and a supplied proof is validated in full either way. The sentinel authority may omit the proof, because the shred-oracle provisions users owned by validators and has no proof it could obtain; a proof it does supply is still validated (#4215). (#4197)
• IP verifier
  • New doublezero-ip-verifier service signs the source address it observes as an RFC-27 IpOwnershipProof, over POST /v1/proof. Forwarded headers count only for connections from a --trusted-proxy CIDR, and only the --forwarded-header the proxy actually writes is read; the chain is walked from the right so a client-prepended hop is ignored. With no trusted proxies configured the connection peer address is the only address it will sign. Non-routable and IPv6 sources are refused, as is a request the cached ledger epoch is too old to answer. The verifier key is checked against GlobalState.ip_verifier_authority_pk at startup and periodically after, so a rotation this service was not redeployed for takes it out of rotation instead of silently failing every user creation onchain. Built on axum, the first HTTP server framework in the Rust workspace. (#4198)
• Utility crates
  • New doublezero-ip-proof crate defines the RFC-27 IpOwnershipProof and the exact bytes the verifier signs, in one place the serviceability program, the CLI, and the verification service all share. (#4195, #4206)

v0.36.0 - 2026-08-14

Breaking

Changes

• CI
  • The devnet daily deploy waits for CloudSmith to list the nightly it just built before it runs ansible. On 2026-08-12 the client package finished uploading at 14:37:00 UTC, and four devnet hosts ran apt-get update between 14:37:21 and 14:37:24. Those four hosts read the old index and stayed on the 2026-08-11 nightly, while the four hosts that ansible reached 7 seconds later upgraded. The new step polls the jammy and noble package indexes for up to 10 minutes, then fails the deploy. CloudFront serves those indexes with max-age=30, so the step waits another 35 seconds after the indexes list the version, because a host reaches a different edge than the runner. A component that builds no deb skips the step. (#4181)
• E2E/QA
  • TestQA_DeviceProvisioning reads the CLI's stdout on its own, rather than merging stderr into it. runCLI used CombinedOutput, so the upgrade banner the CLI writes to stderr reached the JSON parser, and the devnet run on 2026-08-12 failed with invalid character 'A' looking for beginning of value. The CLI's tracing diagnostics and ssh's host-key warning write to stderr as well, and each one broke the same parse. The test now logs that banner once per run as a warning. (#4181)

v0.35.0 - 2026-08-11

Breaking

• CLI
  • doublezero feed get is removed, and doublezero feed list gains --code and --exchange filters in its place. doublezero feed list --code shreds --exchange xlax returns the feed that doublezero feed get --pubkey <pubkey> --exchange xlax returned, plus the group_codes column the list view already carried. A code that matches no feed now prints an empty table instead of failing. The exchange column now carries the metro code, such as xlax, in place of the exchange pubkey, in both the table and the JSON. (#4171, #4172)
  • doublezero feed update and doublezero feed delete now name the feed as --pubkey <PUBKEY>, or as --code <CODE> with --exchange <EXCHANGE>. --pubkey shreds-lax used to accept a code and resolve it by reading every feed, which failed as soon as a second metro carried that code. Write doublezero feed update --code shreds-lax --exchange xlax --name "Shreds LAX v2" instead. (#4172)

Changes

• CLI
  • doublezero access-pass list now renders the groups a feed grants in the multicast column, with an F: prefix next to the existing P: and S: entries. An EdgeSeat pass carries its multicast entitlement on its feeds, and the column read only the two allowlists, so such a pass showed an empty column and looked like it granted nothing. access-pass get gains a feeds row and a feed_groups row, and user get gains a feeds row naming the feeds whose seats the user holds. Both name each feed as code:metro — its code qualified by its exchange's code, e.g. lashay1-feed:xams — because a feed is keyed by (code, exchange), so a pass holding one code in three metros holds three different feeds and printing their codes named all three alike. A feed whose exchange account cannot be read falls back to the exchange key, which is uglier but keeps same-coded feeds apart; a feed that cannot be read at all still renders as its own key. The feed seats in access-pass get --json now carry feed_code and the new exchange_code alongside the existing feed_key, both unqualified, so nothing has to be split back out of a joined string. --multicast-group-subscriber and --not-multicast-group-subscriber now match a group a feed grants, so they agree with the column beside them. The publisher filters are unchanged, because a feed grants subscribe rights only. (malbeclabs/infra#2178, #4167, #4177)
  • doublezero connect multicast with no arguments now joins the feeds an EdgeSeat pass has purchased, instead of subscribing the host to nothing while reporting success. The no-arguments path resolves groups from the pass's mgroup_pub_allowlist / mgroup_sub_allowlist, which the feed purchase path never writes — only the administrative subscriber-allowlist instruction does — so a feed pass had both empty, printed The AccessPass has no authorized multicast groups; nothing to connect to. and exited 0. doublezero-edge-connect's installers run exactly that form, so their || warn guard could not fire and an install announced success while the host held no subscription. Connect now takes a seat on every purchased feed that has one free in the connected device's metro, and names each feed it skipped with the reason. Joining nothing is an error when the host holds no feed and could take none, whether the seats are full or serve another metro, because exiting 0 with nothing subscribed is indistinguishable from success to an unattended installer; a host that already holds a feed still succeeds and activates, so repeated installs stay safe. Headroom is measured against FeedSeat.max_users, the field try_add_feed_user enforces — max_future_users is written and documented to flip the cap at window_end but is read nowhere, so using it would propose feeds the program rejects. Candidate devices are restricted to the metros the purchased feeds serve, as resolve_feed_join already does, so a host that bought an Amsterdam feed while sitting nearest Frankfurt connects through Amsterdam rather than failing; a nearer device in a metro the pass holds nothing in is passed over, and an informational line names it with the latency difference. Only metros holding a feed with a free seat are candidates, so latency cannot land the host in a metro where everything it bought is already full. A user is keyed on (client_ip, user_type) with no device, so one host can still only ever hold one metro's feeds, and feeds in the others are named as skipped. When no eligible device serves any candidate metro the failure names those feeds, and any feed skipped as full, rather than reporting a metro problem for a cluster that simply has no activated devices. A --device naming a device other than the existing Multicast user's now fails rather than being silently ignored, matching the guard resolve_feed_join already applies. (#4173)
  • doublezero feed create and doublezero feed update now read back every --exchange and --group argument, so a feed cannot name a metro or a multicast group that the ledger does not carry. A base58 argument used to pass straight through with no read, so --group 4Nd1mBQtrMJVYVfKf2PJy9NZUZdTAsp7D4xWLs4gDB4T created a feed whose group nobody can join. A code was always read back, so only the pubkey form changes. (#4172)
• E2E/QA
  • TestQA_MulticastSettlement's validate_instant_allocation_price_matches_chain no longer names a specific doublezero_solana_version in its skip path. Both the comment and the skip message said the pin was 0.5.10-1; testnet has since moved to 0.5.11-1, so a reader was told the pin was merely behind when in fact instant_allocation_price is in no release yet. They now name what actually gates the field — a doublezero-offchain release carrying doublezero-offchain#405 — and where the pin lives, neither of which goes stale as versions move. Comment and message only, no behaviour change.
  • TestQA_AllDevices_UnicastConnectivity no longer counts a device that cannot accept users against its failure thresholds. Five sites already checked activated && max_users > 0 and logged Ignoring <x> failure for device not ready for users, but each incremented FailedTests before the check, so the carve-out suppressed only the log line while the device still counted as failed — and gating that counter alone would not have been enough, since Success() also requires a non-zero packet count a device that never connected cannot produce. Such devices are now excluded from ComputeFailureStats entirely, per-host denominator included. This is what failed mainnet-beta QA three times over 2026-08-08/09: laconic-dfw-sw01, laconic-mia-sw01 and laconic-was-sw01 have been activated at max_users=0 since 08-06, the client CLI refuses those connects outright, and cmh-mn-qa01 draws from a 13-device pool, so three unusable devices read as a 21-29% per-host rate against the 20% gate. The excluded codes are now reported in test output, so a skip is distinguishable from a pass. The exclusion is deliberately a subset of the program's is_device_eligible_for_provisioning: a device at users_count + reserved_seats >= max_users hits the same CLI rejection and still counts as a failure, since narrowing that too would restore the capacity pre-filtering #3697 removed. A run that could not attempt more than half of the devices assigned to it — fleet-wide or on any single host — now fails (-skipped-threshold, default 0.5) rather than reporting green over the remnant, and testing nothing at all fails regardless of that threshold, where previously the rate was 0/0 and NaN > threshold passed silently. The gate is per host as well because a drained metro is a few percent of the fleet but all of one host's coverage. The count also publishes as devices_skipped next to devices_tested in InfluxDB and ClickHouse, so a collapse that stays under the threshold shows up on the dashboard and not only in the test log. Separately, a ping that never gets a reply reports its packet counts rather than failed to ping after 3 retries: %!w(<nil>); the wrapped error was always nil, because the retry loop returns early on any real failure. (#4168)

v0.34.0 - 2026-08-07

Breaking

Changes

• CLI
  • doublezero feed list gains a group_codes column naming the multicast groups the feed holds, alongside the existing groups count. A group the ledger no longer carries renders as its raw pubkey. The JSON output gains the field as well. (malbeclabs/infra#2172, #4150)
• Collector
  • The RIPE Atlas collector no longer re-exports the boundary result on every poll. RIPE's ?start= filter is inclusive and the collector passed it the newest timestamp it had already consumed, so each cycle re-fetched that result and appended a duplicate sample to the circuit's onchain account. Samples therefore accumulated at roughly twice the declared 600s sampling interval, and because a sample's time is derived as start_timestamp + index × sampling_interval, the derived times outran wall clock — 403 of 435 mainnet pairs drifted more than an hour within epoch 196, worst case +43.7h, which made event_ts-keyed views unreliable and left consecutive epochs overlapping by days. wheresitup was unaffected. Existing accounts keep their inflated sample counts; the drift stops accumulating from this change forward. The timestamp not updated (old results?) warning only fired because duplicates were being exported, so it would have gone silent with this fix; the stall signal is now keyed off the export cursor not advancing, and fires once a measurement has produced no new sample for over 30 minutes — whether it returned nothing at all or only timeouts. Expect the latency_samples_per_collection_interval_missing counter to start reporting on deploy: the duplicate was always a successful result, so every circuit exported at least one record every cycle and that counter could not fire at all. It is now accurate rather than newly broken, but with one probe per circuit and the sampling and export intervals both at 10 minutes, poll jitter will trip it intermittently in steady state — treat a sustained rise, not an isolated one, as signal. (#4154)
  • The hourly RIPE Atlas measurement cycle now reports source probes that have produced no successful sample since their measurement was created. A source's last_response_at is zeroed whenever its measurement is created and is only ever advanced by a sample with a latency above zero, so a zero past a 2h grace period means that probe has contributed nothing to it — and because a circuit is enlisted exactly once (a measurement for target T enlists only sources whose code sorts after T), each such source silently darks its circuit with no other trace. Between 2026-08-05 and 08-07 this hit muc (11 circuits), then hkg (6), then dub (4), each stopping within the same second across every one of that probe's enlistments; they were found only by reading the state file by hand. New gauge doublezero_internet_latency_collector_ripeatlas_sources_without_samples labelled by source_location, plus a warning naming up to 20 affected sources per cycle; both are silent when there are none. The grace period is deliberately longer than the 1h probe timeout that drives probe rotation, and separate from it so the two can be tuned independently: RIPE dispatch is not immediate, an accepted enlistment has been observed producing its first result 80–100 minutes after creation, so a shorter window reports probes that are merely warming up. This is observation only — it does not mark probes or recreate measurements, because recreation invalidates every measurement whose target sorts before that metro and every drop observed so far recovered unaided. It also names no cause: a source RIPE never dispatched and a source whose path is at total packet loss are indistinguishable in this state, since a result carrying no rtt parses to zero latency. (#4155)
  • A RIPE Atlas result carrying no successful ping no longer looks like a dead source probe. parseLatencyFromResult returns zero for a result whose ping array holds no rtt, and the caller skipped everything behind an if latency > 0 gate, so a path at 100% packet loss never advanced that source's last_response_at. It aged past the 1h threshold and the measurement cycle marked a probe that was uploading results on schedule as unresponsive — observed on mainnet-beta on 2026-08-06, where probe 7447 was marked 16 seconds after its most recent upload and the rotation recreated 28 measurements. A result the probe uploaded now counts as a response regardless of whether anything came back. What gets exported is unchanged: a total-loss result still writes no sample, so the export cursor still stalls when a whole measurement goes quiet and the target-level staleness check still fires. The trade is that a single source stuck at sustained total loss to one target, while its siblings keep returning RTTs, is no longer marked or rotated: its last_response_at stays fresh and its siblings keep the export cursor advancing, so doublezero_internet_latency_collector_latency_samples_per_collection_interval_missing for that circuit is the remaining signal. That is deliberate — a wrongly-marked probe recreates every measurement whose target sorts before its metro — but it is a detector that used to exist. (#4153, #4160)
  • A failed internet-latency submission now retries from the first unwritten sample rather than restarting at the beginning of the flushed partition, and only the unwritten remainder is requeued — the same fix the device telemetry submitter got in #4145. A partition larger than one transaction is written in batches, so any failure part-way through left the earlier batches onchain while the retry and the next tick re-sent them, appending those samples a second time and skewing the latency they feed. Reachable today from an RPC timeout mid-partition; surfacing program rejections adds another way in. (malbeclabs/infra#1703, #4152)
• SDK
  • The Go telemetry SDK no longer reports a transaction the program rejected as a success. Finalization only means the cluster agreed on the transaction: a rejected instruction finalizes too, carrying the rejection in err, which the executor never read. It now returns a *telemetry.ProgramError holding the ledger's error and the program's log output, and leads the message with the program's own explanation so a caller that just prints the error still gets the reason. This is the check the serviceability executor already made. (malbeclabs/infra#1703, #4152)
  • A samples-account-full or missing-account rejection that reaches execution now returns the same ErrSamplesAccountFull / ErrAccountNotFound the equivalent preflight rejection does, via the new ProgramError.CustomErrorCode(). Preflight catches nearly all of these, but a write that simulated cleanly and then failed against the bank it landed on reported its code only through the finalized transaction, so a caller's account-full handling worked on one side of preflight and not the other. (malbeclabs/infra#1703, #4152)
• Device Telemetry
  • A submission the telemetry program rejects onchain no longer burns the tick's remaining attempts: the agent logs the rejection with the program's explanation at Error and moves on, counting submitter_program_error on the errors counter. Before this, the init half of the init→write path could not be seen to fail — it skips preflight, so the rejection only showed up on the finalized transaction, which the SDK read as success — and the agent looped init→write→account not found every few seconds with nothing in the log naming the cause. Observed on chi-dn-dzd4, where the device's metrics_publisher had been set to a key the agent did not hold. Samples are requeued as with any other failure, so the next tick retries once the cause is fixed. An init the program rejects because the account already exists is excepted: that leaves the write with what it needed, so the write now runs either way and only a write that still finds nothing there reports the init failure as the reason. (malbeclabs/infra#1703, #4152)
  • A ledger RPC outage no longer stops TWAMP probing on the device telemetry agent: the pinger caches the last known epoch and refreshes it off the probe path, instead of fetching it inline and skipping the tick on failure. Probing stops when no epoch has ever been fetched, when the cached one exceeds the new -max-epoch-staleness (default 10h, clamped to what the sample buffer holds at -probe-interval), or when the cached epoch's projected end has passed. Samples taken against a cached epoch are written to that epoch's account, so a query scoped to a later epoch will not return them — the projected-end bound is what keeps that from spanning a rollover. The refresh cadence follows -probe-interval and can be set with the new -epoch-refresh-interval. (#4143)
  • A peer discovery refresh that fails after reading the ledger no longer wipes the agent's peer list. It cleared the cache before calling LocalNet.Interfaces(), so a transient failure there left the pinger iterating zero peers and probing nothing until a later refresh succeeded. The cache is now replaced only once the new list is built, which also shortens the critical section to the assignment. (#4146)
  • The telemetry agent now logs and counts samples it discards when a submission fails and the partition buffer is already at capacity; that path previously recycled the batch with no signal at all. New counter doublezero_device_telemetry_agent_samples_dropped_total with a reason label (buffer_full), plus submitter_buffer_full on the existing errors counter. Requeue behavior below capacity is unchanged, and neither signal fires in steady state. (#4144)
  • Samples dropped because a partition's onchain account is full are now counted too, under reason="account_full" plus submitter_account_full on the errors counter. That path reports success to its caller, so a warning was the only trace, and its count was wrong: it reported the whole flushed partition rather than the samples actually lost. (#4145)
  • A failed submission now retries from the first unwritten sample rather than restarting at the beginning of the flushed partition, and only the unwritten remainder is requeued. Previously a mid-partition error made every subsequent attempt re-send batches that were already onchain, appending those samples a second time and pushing the account toward the sample cap it is measured against. (#4145)
  • Agent logs now identify the ledger RPC endpoint in use, and the peer count and the stale-program-data warning report state transitions instead of firing on every refresh. New lines name the resolved remote address of each connection (a bad load balancer address behind a hostname was invisible before), and a refresh that finds no peers is now a warning rather than a Debug line. The stale-cache warning also moves off the package-global slog onto the agent's own logger, so it is formatted and leveled with everything else. New: doublezero_device_telemetry_agent_peers gauge, and pinger_epoch_fetch on the errors counter for every exhausted epoch fetch. (#4147)
• QA
  • Rework existing TestQA_MulticastSettlement and adapt it to the new FLAG_RETRANSMIT_ONLY_ONBOARDING_ENFORCED_BIT flag. Test now checks for this flag in the ProgramConfig solana account and depending on if it's on or off tries to assert that no new user can subscribe to a non retransmit-only metro unless that metro has the retransmit-only flag enabled in the MetroHistory account. (#4156)

v0.33.0 - 2026-07-31

Breaking

• CLI
  • doublezero feed update (with --group) and doublezero feed delete now fail closed if the change would leave an EdgeSeat user holding a multicast group role that no consumed feed seat on their access pass covers and the pass's own allowlists don't authorize, printing the affected users, groups, and roles and submitting nothing. Pass --force-unsubscribe to remove those roles first and then apply the change; a group where one role must go while the allowlist still authorizes the other fails closed even with the flag and needs manual cleanup. A rename or an update without --group is unaffected; an additive --group set scans but finds nothing to do. The removals need USER_ADMIN (or foundation membership) on the payer in addition to FEED_AUTHORITY. (malbeclabs/infra#2113, #4119)

Changes

• CLI
  • doublezero connect multicast gains --subscribe-feed and --unsubscribe-feed (feed codes or pubkeys) for EdgeSeat access passes: one command creates the bare multicast user if needed and joins whole feeds, or leaves them. When both flags are given the leave runs first, and if one half fails the output names it and says which flag to rerun. (#4111)
• SDK
  • New SubscribeFeedCommand and UnsubscribeFeedCommand in the Rust SDK derive the exact group lists the onchain instructions demand and check the metro, pass, and feed-cap constraints before sending; the daemon-cli LedgerClient gains list_feed, subscribe_feed, and unsubscribe_feed. (#4111)
• Controller
  • A multicast user with no publisher role now gets the subscriber ingress ACL and a deny-all announce prefix-list; a user with no roles at all previously fell through to the publisher versions of both. (#4110)
• Serviceability
  • New SubscribeFeed and UnsubscribeFeed instructions join or leave whole feeds on an EdgeSeat access pass in a single atomic transaction, charging one seat per feed rather than per group. A feed is all-or-nothing: the caller names feeds, the processor derives which groups change and rejects a group list that does not match, so two feeds carrying the same group stay unambiguous. UpdateMulticastGroupRoles now enforces the multicast-group allowlists for every access-pass type, EdgeSeat included, so purchased groups go through the feed instructions and individually comped groups through the allowlist. CreateSubscribeUser skips the allowlist only for the case its feed gate actually covers, closing two paths that could join a group with no check. MAX_FEED_GROUPS drops from 64 to 20, bounded by what one transaction can carry: because joining passes every group a feed holds, a larger feed could never be joined. No feed onchain is affected. For the same reason a user may hold at most 5 feeds (a leave names every held feed plus the departing groups behind the client's compute-budget prelude), and a held feed the pass no longer carries is pruned on leave instead of blocking it. New errors: EdgeSeatRequired (101), UserDeviceMismatch (102), UserFeedLimitExceeded (103), EdgeSeatIsMulticastOnly (104). (#4109)
  • CreateUser can now create a bare multicast user (no group, no feed) under an EdgeSeat pass: the feed gate moved to CreateSubscribeUser, its only caller, and feed seats are otherwise charged by SubscribeFeed. The pass-level max_multicast_users cap is enforced again (it had been vestigial under the feed-seat model) so bare users stay bounded; the oracle already provisions it as the total purchased seats. CreateUser is also idempotent: re-running it for an existing user matching the requested owner, device, user type, and tenant succeeds without changing state, so a client can retry safely; a mismatch still fails with AccountAlreadyInitialized, and a banned user with InvalidStatus. This applies to the current client_ip-keyed user PDA; the legacy index-keyed PDA cannot be re-run, since its index has moved on. Deploy note: max_multicast_users becomes load-bearing on existing EdgeSeat passes; before deploying to a cluster, confirm each pass's value covers its feed seats and backfill via SetAccessPass where short. Checked 2026-07-30: mainnet-beta and devnet hold no EdgeSeat pass; testnet's single one (5x9DTsWC…) has max_multicast_users 1, zero users, zero connections. (#4110)
• CI
  • The Rust toolchain moves from 1.90.0 to 1.97.1, seven releases forward. 1.90.0 sat below the supported floor of the rust-analyzer builds shipping with current editors (1.94.0), so the language server warned on every workspace load; that floor tracks stable at roughly minus-three releases, so the gap widens with each Rust release rather than holding steady. rust-toolchain.toml, the twelve dtolnay/rust-toolchain CI pins across seven workflows, the devcontainer Rust feature, and the rustup pin baked into release/Dockerfile.release move together — contributors must rebuild their devcontainer after this merges. The release image needs no manual step: scripts/build-snapshot.sh keys its rebuild on the Dockerfile's hash, so changing the pin is what triggers it. Left stale, that image would have kept 1.90.0 while goreleaser ran against a 1.97.1 rust-toolchain.toml, re-downloading a full toolchain on every snapshot build (RUSTUP_HOME is not one of the named volumes and the container runs --rm). The accompanying source changes are only the clippy lints the newer toolchain adds (useless_borrows_in_formatting, unnecessary_sort_by, iter_kv_map, to_string_in_format_args, collapsible_match, result_large_err). The serviceability ones are redundant & removals inside Display/Debug impls, where format_args! auto-references, so formatted output is byte-identical; they were required rather than optional because lint-programs runs from smartcontract/ and so picks up the root toolchain, not smartcontract/programs/rust-toolchain.toml. Onchain codegen is unaffected either way: cargo build-sbf --tools-version v1.54 supplies its own rustc. (#4118)
  • dispatch-and-wait now reports to Slack which jobs of a dispatched cross-repo run did not pass, each with its conclusion, so the release thread says what broke instead of only that something did. It names every concluded job that is not success or skipped — a leg that exceeds its timeout-minutes concludes timed_out, not failure — and it fires when the calling job is cancelled or hits its own timeout as well as when the run fails, which is where a stalled stage would otherwise go unannounced. The report is best-effort: it warns and exits 0 on a token or API failure, so it can never mask or replace the failure the calling job already reports. The job list is capped at 10 names with an overflow count. Wired up on the orchestrator's qa job (the infra qa.testnet.yml dispatch), which now also lists preflight directly in needs to read its thread_ts. (#4121)

v0.32.0 - 2026-07-29

Breaking

• SDK

  • The Rust SDK no longer attaches the payer's Permission PDA to serviceability transactions. The RFC-26 builders derive account layout offline and their Permission append is still deferred, so every migrated instruction reaches authorize() with no Permission account and takes the legacy GlobalState allowlist/authority path. A payer whose only grant is an activated Permission account — not on foundation_allowlist/qa_allowlist, not a matching authority key — gets NotAllowed on every gated command. Re-enabling the append is per-instruction future work and must be sequenced with the permission-model rollout. Verified against live chain state at time of writing, no cluster is exposed: RequirePermissionAccounts is off everywhere (so the legacy fallback is live), mainnet-beta's 8 Permission accounts all grant FOUNDATION to keys already on foundation_allowlist, and the one grant on testnet/devnet that no legacy authority covers (ACCESS_PASS_ADMIN to EWFXDTBWhNxsmj4mbahZL7HH1XpmC6Ao3LZUhJGCyyPh) belongs to the shred oracle, which assembles its serviceability instructions itself and appends its own Permission account — it never uses this send path, so it is unaffected. Note doublezero permission audit does not answer this question: it reports the inverse (strict-mode) direction only, so checking it means diffing permission list against the GlobalState allowlists. (#4060)

• Serviceability

  • MIN_COMPATIBLE_VERSION moves from 0.21.0 to 0.30.0, excluding every client that predates the EdgeSeat(Vec<FeedSeat>) AccessPass decoder (#3954, #4030). Such clients misparse every field after the variant tag and can abort on the bogus allowlist length that follows; testnet already holds an EdgeSeat pass, so this is a live crash rather than a hypothetical one. The oldest release this admits is v0.31.0, not v0.30.0: the client/v0.30.0 tag was cut before its version-bump commit merged, so its binaries self-report 0.29.0 — indistinguishable from the genuine v0.29.0 release, which has no EdgeSeat decoder — and a floor low enough to admit them would admit v0.29.0 too. The v0.30.0 release is therefore excluded by its version string rather than by any real incompatibility (its code has the decoder). Tag/version drift is fixed for v0.32.0 onward by #4068. The floor is stamped onto ProgramConfig by doublezero init, so it takes effect per cluster at that cluster's next program deploy — not at merge — at which point the window is [0.30.0, <deployed version>], admitting v0.31.0 and the release being deployed. A client below the floor fails at startup with "A new version of the client is available".
  • User account: the single-feed slot feed_pk is replaced in place by feed_pks: Vec<Pubkey> across the Rust struct, serde JSON, and all SDKs (Go FeedPk → FeedPks, TypeScript feedPk → feedPks, Python feed_pk → feed_pks). Consumers reading the old key must update. The vec occupies the former 32-byte scalar slot; this is safe because no account ever recorded a feed there (no EdgeSeat access pass has existed on any cluster — verified against the mainnet-beta data lake), so existing accounts carry 32 zero bytes that parse as an empty vec with the leftover zero bytes ignored as trailing data. Deploy note: confirm no nonzero feed_pk account exists on each cluster at upgrade time; an EdgeSeat multicast connect before the upgrade would record a feed the new layout cannot see. (#4080)

Changes

• CLI
  • doublezero user create-subscribe gains a --feed flag (a Feed pubkey or an unambiguous feed code) passed as the trailing Feed account the EdgeSeat feed metro gate requires. Without it, connecting a Multicast user to a feed pass failed with FeedAccountRequired. (#4087)
• Serviceability
  • User accounts can hold EdgeSeat feed seats on multiple feeds: feed_pks: Vec<Pubkey> records every feed whose per-feed seat the user consumed at connect, and delete releases all of them. Added to the Go/TypeScript/Python SDK User decoders as well. (#4080)
  • Bound the preallocation in deserialize_vec_with_capacity against the remaining input. A garbage or attacker-controlled u32 length prefix in an account (e.g. a pre-FeedSeat SDK misparsing an EdgeSeat AccessPass) could request tens of GiB via Vec::with_capacity, aborting the process through the uncatchable alloc-error handler; the capacity is now capped at the remaining byte count. Decoding of valid accounts is unchanged. (#4072)
  • Reject duplicate entries in a Link's link_topologies, and decrement each topology's reference_count once per unique entry when a link drops it — on LinkDelete and on the LinkUpdate removal path. LinkUpdate diffs old vs new as a set, so a duplicate (--link-topology "TOPO-A,TOPO-A", one comma typo by a NETWORK_ADMIN or foundation key) stored two entries against a single increment while both drop paths decremented per entry — which could zero a reference_count another link still contributes to and let TopologyDelete close the topology PDA that guard protects, leaving a dangling pubkey in the surviving link and a permanently retired admin-group bit. The duplicate check is on the instruction argument, deliberately not in Link::validate(): validate() runs inside try_acc_write, so a link already holding duplicates would become unwritable and therefore unrepairable. No cluster carries existing skew — mainnet-beta has no link, current or in any historical snapshot, with more than one topology entry. (#4090)
  • AccessPass gains a dzf_locked flag (bit 0 of flags) marking a pass as foundation-managed so automated reconcilers such as the Feed Oracle leave it alone. A new SetAccessPassFlags instruction (gated on ACCESS_PASS_ADMIN) sets/clears access-pass flags without disturbing the others, and SetAccessPass now preserves the flag across unrelated updates. Set it with the doublezero access-pass dzf-lock / dzf-unlock verbs or access-pass set --dzf-locked. (#4083)
• CI
  • The daily release build populates the Go module cache in a retried step before GoReleaser runs. proxy.golang.org occasionally drops a module zip mid-transfer (HTTP/2 INTERNAL_ERROR) and the go command does not retry, so the failure previously surfaced as a GoReleaser build error partway through the release. Retrying up front is cheap and side-effect free. The step is best-effort and bounded (continue-on-error, 10-minute timeout): it fetches a superset of what the released binaries import, so a durable proxy failure on a test-only dependency, or a stalled transfer, degrades to the previous behavior — GoReleaser fetches whatever is still missing — instead of becoming a new way for the release to fail or hang. (#4105)
  • Add Copilot code-review instructions: a repo-wide .github/copilot-instructions.md (review posture plus cross-cutting rules) and eight path-scoped .github/instructions/*.instructions.md files covering the onchain programs, the Rust SDK/builders/CLI, Go, E2E/QA, release workflows, the language SDKs and their golden fixtures, tests, and the changelog. Rules were extracted from the team's own inline PR reviews over the previous month and cross-checked against the code, plus the standards in RFC-1 (semver classification, deploy-ordering prerequisites), RFC-10 (compatibility-window schema evolution), and RFC-20. No effect on builds or runtime. (#4093)
• Client daemon
  • A device being added to or removed from the fleet no longer tears down and recreates every DoubleZero user's tunnel. DoubleZeroPrefixes in a ProvisionRequest is the union of every device's dz_prefixes fleet-wide, so one device changing shifts it for everyone, yet Equal/InfraEqual compared it for all user types and pushed the reconciler onto the full-reprovision path. Deleting the Linux interface drops every multicast group membership bound to it — consumers that join only at startup silently stop receiving traffic while BGP reconverges and all DZ-side health checks pass. The comparison is now scoped to UserTypeEdgeFiltering, the field's only consumer (EdgeFilteringService.createIPRules); IBRL and multicast never read it. Diff still reports the prefix-count change unconditionally so the log line stays useful for diagnosis. (#4102, malbeclabs/infra#2117)
  • Route liveness recovers from a transient loss in roughly one backed-off probe interval plus two round trips, instead of up to three such intervals. A state-changing RX now advertises the new state immediately and re-arms the normal transmit cadence, rather than waiting out the timer armed while the session was Down (exponentially backed off up to -route-liveness-backoff-max). Two defects contributed: the RX path never enqueued a TX on a state change, and the scheduler's sleep had no wakeup path, so an event pushed earlier than the deadline it was sleeping on could not shorten that sleep — which also delayed the first probe of a newly registered route by up to another peer's backed-off interval. The first probe after the path recovers is still bounded by the backed-off interval, so the backoffMax default (60s) still sets the floor on recovery time; that default is unchanged here. The immediate transmit takes over the pending-TX slot rather than adding to it, so recovery re-arms the normal cadence instead of leaving the session silent until the stale backed-off deadline — which would have made the peer time it back out and flap the route every detect interval. It is also paced by the transmit interval measured from the last send, so a peer whose inbound traffic is blocked and which therefore repeats Down (flipping the session Down↔Init on every packet) cannot drive the transmit rate 1:1 with its send rate past minTxFloor/maxTxCeil; and a displaced transmit event is now dropped as stale when it pops, as detect events already were, so a takeover emits no duplicate packet and leaves no backlog. Also logs the effective minTxFloor/maxTxCeil/backoffMax at startup, since a wrong value was otherwise only visible as slow reconvergence. (#3935)
• Device controller
  • Escalate onchain account fetch failures to ERROR only when sustained; a transient blip that recovers on the next poll now logs at WARN, so a single flaky fetch no longer pages via the generic ERROR-level alert. A weighted score (+1 per failure, -0.5 per success, floored at 0, capped at 6) crosses the threshold on a persistently failing endpoint, so real outages still surface. Each fetch is bounded by a 30s timeout so a hung endpoint fails the tick promptly rather than blocking for minutes. (#4081)
• Tools
  • Treat truncated or partial JSON-RPC response bodies (unexpected end of JSON input, unexpected EOF) as retryable, so a cut-off 200 response is retried in-call; genuinely malformed but complete responses remain non-retryable. (#4081)
  • Retry HTTP 429/500/502/503/504 from a Solana JSON-RPC endpoint. The classifier asserted on StatusCode() int / Code() int interfaces that no solana-go error type implements — both expose their code as a struct field — so the entire status-code branch was dead and every Go ledger reader gave up after one attempt. On 2026-07-28 that meant 4+ hours of 100% fetch failure across ~20 components against a healthy chain, absorbed only because each happens to keep a last-good cache. Classification now reads the concrete *jsonrpc.HTTPError and *jsonrpc.RPCError values, covering both the undecodable-body case and the decodable JSON-RPC error envelope the incident actually arrived in, and matches transient wording from providers that set no machine-readable code. Backoff is jittered so the ~60 doublezerod hosts and dozen services reading one endpoint do not retry in lockstep against an endpoint already shedding load. sendTransaction and requestAirdrop are never retried regardless of the options passed, so no caller can resubmit a transaction the endpoint already accepted. New doublezero_solana_rpc_retries_total and doublezero_solana_rpc_retries_exhausted_total counters (labelled by method) make the next episode visible without reading logs. Only codes that a later identical request can plausibly succeed on are retried — -32005 (node unhealthy), -32004 (block not available) and the provider-minted -32429 (rate limit); -32003 (transaction signature verification failure) is deterministic and is not retried, so a bad signature fails on the first attempt instead of spending the whole budget against the endpoint. (#4098)
  • Consolidate Go Solana RPC construction on tools/solana/pkg/rpc.New, replacing the duplicated transport-level retry loops in sdk/shreds, sdk/revdist and the internet-latency collector (each ignored ctx and slept a fixed unjittered backoff) and adding retries to the ledger readers that had none: doublezerod, monitor, funder, device-health-oracle, telemetry (data-api, geoprobe-agent, telemetry including its netns/on-device path, data CLI), cdiff and the Go SDK. Note for consumers of sdk/shreds and sdk/revdist: the retry budget contracts from 6 attempts over ~30s of unjittered time.Sleep to 4 attempts over ~3s that honor ctx, so a blip longer than a few seconds now surfaces as an error rather than blocking for half a minute — pass Retry via tools/solana/pkg/rpc.New for a longer budget. ~30s is the established budget convention for QA-test callers in this codebase, so those callers should pass Retry with a larger MaxAttempts explicitly rather than silently inheriting the shorter default. The shared transport's default per-request timeout drops from 5 minutes to 10s, which also bounds a single retry attempt: retry multiplies a hung endpoint by the attempt count, so the previous default let one logical call consume up to 20 minutes on the readers that inherit it (doublezerod, monitor, funder, device-health-oracle, telemetry, data-api, cdiff, state-ingest, flow-enricher, global-monitor, the Go SDK and sdk/revdist). 10s leaves more than an order of magnitude of headroom over the heaviest call we make — an unfiltered getProgramAccounts over the serviceability program, ~2.9MB raw / ~1.0MB gzipped and ~0.4s against mainnet — and keeps an exhausted budget inside the tightest unbounded caller cadence (state-ingest's 60s refresh); sdk/shreds and the internet-latency collector keep their explicit 15s. The dial timeout drops from 5 minutes to 10s, matching its TLS handshake bound — a dial that can hang for minutes makes any retry budget meaningless. (#4098)
• E2E/QA
  • TestQA_MulticastSettlement skips (with an expected epoch-tail closed window: ... message) instead of failing when wait_for_open_phase times out during the by-design closed window at the tail of every Solana epoch. The classification is verified against live chain state — the closed_for_requests_grace_period_slots read from the shred-subscription ProgramConfig, the execution controller phase and last-close slot, and the epoch schedule from the target cluster's RPC — and requires the whole timed-out wait (not just its end) to fall inside the window, so nothing is hardcoded and a timeout outside the window still fails as loudly as before. (#4069)
  • TestQA_MulticastSettlement recovers from the failure modes that kept mainnet-beta QA red: ensure_multicast_disconnected self-heals seats left stuck-active onchain by a previous run's failed withdraw (scanning client seats for the client's public IP via the shreds SDK FetchAllClientSeats and withdrawing any with TenureEpochs > 0), and every withdraw — the withdraw_seat step, the self-heal, and the cleanup — retries over a bounded window instead of failing on a single spurious "request in flight" preflight bail. The retry rotates to a different Solana RPC endpoint on the in-flight bail (the stale getMultipleAccounts read behind it is per-endpoint) and confirms completion against fresh onchain state rather than the CLI's error text. The wait_for_seat_allocation_acked step is removed: polling the seat's pending flag cannot distinguish a fast ack from a re-fund of an active seat that never creates a request; the retrying withdraw instead confirms completion against the seat's onchain tenure and pending-request state. (#4066, supersedes #4065)
  • Add TestQA_RetransmitOnlySettlement, an E2E QA test demonstrating retransmit-only feed subscription and settlement via shred pay. (#4077)
  • TestE2E_MultiClientIBRL_RouteLiveness diagnoses its own route-convergence timeouts: requireEventuallyRoute renders the pass number (its pass %d messages previously printed literally) and dumps the failing client's daemon /routes view, liveness counters and iptables INPUT counters instead of only Condition never satisfied. Clients 1-3 run with route-liveness debug logging. (#3935)
• SDK
  • Add the doublezero-serviceability-instruction crate (RFC-26): pure, RPC-free build_xxx(...) -> Instruction builders — one per buildable serviceability instruction — that assemble account layout and borsh-pack args offline (SPL-style), with the trailing-account convention centralized in common::build. Backed by golden ix_* fixtures (CI drift-guarded) and a solana-program-test suite that runs the highest-cardinality builders (create_device, create_link, create_subscribe_user, atomic delete_device, and clear_topology) against the real program to catch account-order drift.
  • Migrate every serviceability commands/* execute() to delegate to those builders and a single new DoubleZeroClient::send_transaction (compute-budget prelude + sign + send), replacing the four execute_*(instruction, accounts) methods and the client-side account assembly. Command execute() signatures/returns are unchanged, so CLI/sentinel/daemon consumers are unaffected. The _quiet send variant and its SimulationError/SimulationTransactionError types are dropped: their only caller was the activator/ crate, deleted in #3647. (#4060)
  • Fix doublezero topology clear reverting with "Topology Account is not writable" on a live topology. The SDK passed the topology PDA read-only, but the processor asserts it writable whenever it actually clears a link reference (it decrements reference_count), so the command only worked on the already-closed-topology path — blocking topology delete, which requires reference_count == 0. The RFC-26 clear_topology builder passes the account writable, which is a strict superset privilege, so no other path changes. (#4078)

v0.31.0 - 2026-07-17

Breaking

Changes

• CI
  • Fix the event-driven doublezero-edge-connect rebuild: the base-image publishers now mint a short-lived, least-privilege token from the release-bot GitHub App (scoped to doublezero-edge-connect) to fire the cross-repo repository_dispatch, replacing the never-created EDGE_CONNECT_DISPATCH_TOKEN secret whose absence silently no-op'd every notify. Also add the missing devnet notify step, so all three variants (testnet, mainnet-beta, devnet) trigger a rebuild on base publish. The mint/notify steps stay non-fatal to the publish and edge-connect's daily poll remains the fallback.
  • Collapse the testnet release tag phase to a single approval: gate 1 now confirms the version-PR merges and authorizes the tag push in one click. The reusable tag workflow no longer carries an environment (it is pure mechanism); the approval prompt lives with its callers — the orchestrator's gate 1, and a new approve job (on testnet) in the manual components dispatcher, so manual tag pushes prompt exactly as before — and a github.workflow_ref caller allowlist makes any other caller fail closed. No new environments. Gate job display names and the Slack posts now spell out what each approval means and that gate 1 must only be approved after both PRs merge.
  • Add a pipeline-complete guard job (if: !cancelled(), needs every stage through qa) to the testnet release orchestrator that fails unless every stage concluded success, so a run that silently skipped stages can never end green; hollow successes also trigger the Slack failure alert. Deliberately cancelled runs stay quiet, and announce is excluded so an announce-only failure can't fire a "release failed" alert for a release that succeeded.
  • Guard the version-bump script against resolver side effects: cargo update --workspace may only add the members' new version lines to Cargo.lock — any other added line (e.g. a dependency-edge rebind like the observed solana-system-interface 3.2.0→2.0.0 flip) fails the bump before a poisoned PR is opened; pure removals (stale-entry pruning) are logged for the reviewer.
• CLI
  • Documentation only: update docs/cli-standard.md to reference the doublezero-daemon-cli module crate (final PR of the RFC-20 daemon-cli extraction stack). The transition shims (client/doublezero/src/servicecontroller.rs, client/doublezero/src/command/) were already removed by the earlier stack PRs; the binary is unchanged. (#4047)
  • The feed verbs accept an exchange code for --exchange and multicast group codes for --group, in addition to pubkeys; pubkey inputs behave exactly as before. (#4027)
• Serviceability
  • Add the doublezero-serviceability-instruction crate (RFC-26 R0): pure, RPC-free instruction builders for the serviceability program (the SPL instruction::* pattern). R0 ships the scaffold, the shared trailing-account keystone, and four exemplar builders (create_device, create_link, delete_device, create_subscribe_user). Nothing consumes it yet; remaining domains and golden-fixture / solana-program-test coverage land in follow-up PRs. (#4049)
  • access-pass get --json includes a feed_seats array exposing each EdgeSeat pass's per-feed seat state (user counts and billing windows); the table view is unchanged. (#4063)

v0.30.0 - 2026-07-10

Breaking

Changes

• Serviceability
  • Feed account: a catalog entry for one metro's multicast group set, keyed by (code, exchange) (one feed_key is one feed in one metro), managed by a catalog admin (FEED_AUTHORITY Permission or FOUNDATION) via CreateFeed/UpdateFeed/DeleteFeed. (#3953)
  • SetAccessPassFeeds provisions feed_keys (SKU seats) onto an EdgeSeat pass, each FeedSeat carrying the feed's full per-feed billing state (current cap, future cap, the window boundary between them, the termination date, and the renewal anniversary day); the oracle calls it via its ACCESS_PASS_ADMIN Permission. (#3954, #4030)
  • The AccessPass EdgeSeat variant now carries a Vec<FeedSeat> payload (feed_key + per-feed cap) instead of being a bare marker. This changes the AccessPass borsh layout for EdgeSeat passes. (#3954)
  • EdgeSeat multicast connect is metro-gated: a device whose exchange is not covered by any of the pass's feeds is rejected with MetroMismatch, and the matching feed's per-feed cap is enforced. (#3955)
• Controller
  • Track the latest config agent version per device in a new controller_agent_versions ClickHouse table, updated on GetConfig polls. (#3578)
• SDK
  • Give the shreds SDK RPC client a bounded per-request timeout (15s) and a sized connection pool instead of the unbounded http.DefaultClient, so a slow or degraded RPC endpoint fails fast rather than blocking until a transaction's blockhash expires and its send fails preflight with BlockhashNotFound.
  • Go, TypeScript, and Python deserialization for the Feed account and the EdgeSeat FeedSeat payload. (#3956, #4030)
• Serviceability
  • Gate Device and device-interface instructions on NETWORK_ADMIN (and HEALTH_ORACLE for sethealth) or the contributor owner via authorize(); internal foundation-only sub-gates now also accept NETWORK_ADMIN holders. (#3980)
  • Gate UpdateUser on USER_ADMIN, CheckAccessPass on ACTIVATOR, and accesspass CheckStatus on ACTIVATOR|USER_ADMIN via authorize(); user create and set_bgp_status remain owner-authorized (not part of the admin Permission system). (#3984)
  • Gate Tenant instructions (create/update/delete/add_administrator/remove_administrator/update_payment_status) on TENANT_ADMIN or foundation/sentinel via authorize(). (#3983)
  • Gate MulticastGroup CRUD on MULTICAST_ADMIN and publisher/subscriber allowlist add/remove on mgroup.owner OR MULTICAST_ADMIN/ACCESS_PASS_ADMIN via authorize(); add handlers use split_trailing_permission. (#3982)
  • Require a Permission account (or the legacy foundation authority) for GlobalState, GlobalConfig, and foundation/QA allowlist admin instructions, gated on GLOBALSTATE_ADMIN via authorize(). (#3977)
  • Gate Contributor instructions (create/update/suspend/resume/delete) on CONTRIBUTOR_ADMIN or foundation; the contributor owner retains the ops-manager-only update path. (#3978)
  • Gate Location and Exchange instructions (create/update/suspend/resume/delete, exchange setdevice) on INFRA_ADMIN or foundation via authorize(). (#3979)
  • Gate Link instructions (create/update/delete/suspend/resume/accept/sethealth) on NETWORK_ADMIN (and HEALTH_ORACLE for sethealth) or the contributor owner via authorize(); variable-length delete/update use split_trailing_permission. (#3981)
• Collector
  • Harden ledger writes against a slow/degraded RPC endpoint: bound each RPC request (default 15s, --ledger-rpc-timeout), size the connection pool above the submitter concurrency (default 128, --ledger-rpc-max-conns), and deadline each submission attempt so it fails fast and retries with a fresh blockhash instead of sending an expired one and failing preflight with BlockhashNotFound. (#3973)
• Onchain programs
  • Restrict granting the FOUNDATION permission flag: a plain PERMISSION_ADMIN holder can no longer grant FOUNDATION (a privilege escalation). Only a foundation_allowlist member or an existing FOUNDATION holder may grant it, enforced independently of RequirePermissionAccounts so foundation members are never locked out.
• CLI
  • Move the multicast transport verbs (multicast subscribe/unsubscribe/publish/unpublish) into the doublezero-daemon-cli crate per RFC-20. Each now takes &CliContext + generic &D: DaemonClient + &L: LedgerClient + &mut W writer; informational/result lines route through the shared writer (stdout, previously the stderr spinner). They stay nested under doublezero multicast (not hoisted as top-level daemon verbs); onchain multicast group CRUD is unchanged. The binary's command/helpers.rs (resolve_client_ip) and the servicecontroller.rs remnant are deleted — the crate copies survive. Flags, output content, and exit codes are unchanged. (#4037)
  • Move the connect verb into the doublezero-daemon-cli crate per RFC-20. It now takes &CliContext + generic &D: DaemonClient + &L: LedgerClient + &mut W writer; informational/result lines route through the shared writer (stdout, previously the stderr spinner), spinners stay on stderr, pre-flight diagnostics route through tracing, and device selection uses the crate's latency utilities (from #3995). The binary's dzd_latency.rs and the orphaned check_doublezero pre-flight are deleted. Flags, output content, --verbose, exit codes, and version-check semantics are unchanged. (#4010)
  • Move the disconnect verb into the doublezero-daemon-cli crate per RFC-20. It now takes &CliContext + generic &D: DaemonClient + &L: LedgerClient + &mut W writer; informational/result lines route through the shared writer, spinners stay on stderr, and diagnostics route through tracing. Behavior (flags, output, --verbose, --no-wait, version-check) is unchanged. (#4008)
  • Move the latency and routes verbs, plus the device-selection/latency-polling utilities and resolve_client_ip, into the doublezero-daemon-cli crate per RFC-20. Both verbs now take &CliContext + generic &D: DaemonClient + &L: LedgerClient + &mut W writer; diagnostics route through tracing and output through the shared writer helper. Behavior (flags, output, --json, version-check) is unchanged. (#3995)
  • Add doublezero permission audit to check legacy→Permission parity before enabling require-permission-accounts: it reports which legacy keys would lose access to migrated instructions (coverage gaps, non-zero exit), super-admin holders, and the non-migrated subsystems that still depend on the GlobalState allowlists.
  • Block doublezero device delete when the device is still enabled in the shred-subscription program (checked via its DeviceHistory account on Solana L1), preventing an orphaned device that deadlocks shred oracle epoch settlement. (#3989)
• E2E
  • Fix the multicast settlement QA test's seat-allocation ack wait. It read the reused client seat at finalized commitment and could accept the previous run's already-acked state, then withdraw while the current request was still pending. It now waits to observe the request pending before treating a cleared flag as the ack. (#3972)
• Sentinel
  • Enable the reqwest json feature explicitly for the sentinel crate, which relies on Response::json() in the validator metadata reader and multicast publisher after the workspace dependency stopped enabling it by default. (#3986)
  • Make the all-devices unicast QA test tolerate a host reporting multiple tunnel statuses. It now selects the IBRL status via GetUserStatuses/FindIBRLStatus instead of erroring on a lingering Multicast tunnel and dropping the host, and logs a warning when a host reports more than one status. (#3976)
  • Add a doublezero feed CRUD lifecycle test (create/get/list/update/delete) against a live devnet. (#3994)
• CI
  • Auto-publish the mainnet-beta client base image daily when the stable Cloudsmith channel advances (idempotency-gated so a run with no new version is a no-op), and notify doublezero-edge-connect to rebuild its testnet and mainnet-beta variants when a new base image is published. Serialize publishes with per-job concurrency groups, keep the notify steps non-fatal to the publish, and skip Debian pre-release versions when resolving the mainnet-beta tag. (#3990)
  • Add a testnet release orchestrator workflow (release.testnet.yml) that drives the release end to end: preflight checks, version-bump PRs for doublezero and infra, a human-approved gate before pushing the 9 component tags, CloudSmith package verification, Solana program build and staging with a manual deploy gate, onchain version verification, infra core/client deploys, QA, and Slack notifications. Supports dry_run for plumbing validation and safe re-runs (existing PRs are reused; already-pushed tags are skipped via a new skip_existing input on the tag workflow). Runbook at docs/testnet-release.md.
  • Thread all testnet release orchestrator Slack posts under a single per-run parent message in #bots, posted via the Slack Web API (chat.postMessage with a bot token) instead of incoming webhooks, which cannot start threads. Adds a threaded PR-links post after open-prs covering the merge-both-PRs / approve-gate-1 human steps, and a tag-approval nudge after gate-tags for the tag jobs' second testnet environment prompt. Slack failures degrade to workflow warnings and flat posts, never failing the release. (#4036)
  • Testnet release: the generated DEPLOY.md now embeds the exact program-deploy commands (keypairs under ~/testnet-ops/, artifacts from the staged release directory, doublezero init to refresh the onchain version) and links the infra deploy runbook, which replaced the Notion doc. Runbook recovery guidance updated to prefer "Re-run all jobs" after mid-pipeline failures.
  • Fix dry runs dead-ending after stage-programs: the job-level skip of push-tags transitively skipped every downstream default-condition job (a skipped ancestor poisons success() for the whole graph, past verify-cloudsmith's own override), so gate-programs through announce never ran in dry-run mode. The tag workflow gains a dry_run input and the tag jobs now run as validated no-ops instead of skipping — which also means dry runs exercise the testnet tag-approval prompt — and verify-cloudsmith's special-case condition is deleted.
• Dependencies
  • Pin the workspace solana-system-interface requirement to "3" and solana-loader-v3-interface to "6" (were multi-major ranges >=1,<=3 and >=5,<=6). The wide ranges let cargo update --workspace silently rebind our crates' edges onto the older major the agave tree also pulls in, making lock resolution non-deterministic (poisoned the v0.30.0 release bump). No resolved versions change. (#4043)
• RFCs
  • Add RFC-26 proposing a pure, RPC-free Rust instruction-builder library (doublezero_serviceability_instruction) with one SPL-style builder per serviceability instruction, decoupling instruction construction from signing/sending and centralizing account-order conventions.

v0.29.0 - 2026-07-02

Breaking

• SDK
  • revdist Python SDK migrated to the async solana-py RPC API (solana-py 0.40.0 removed the sync Client). The Client read methods (fetch_config, fetch_distribution, etc.) are now coroutines and must be awaited; new_rpc_client returns an AsyncClient. (#3945)

Changes

• Dependencies
  • Migrate the entire Rust workspace from solana-sdk 2.3.x to the solana 3.0 line plus the granular split crates (solana-pubkey, solana-instruction, solana-cpi, solana-sdk-ids, solana-system-interface, solana-commitment-config, solana-compute-budget-interface), aligning with the doublezero-solana programs. Onchain account layouts are unchanged (regenerated fixtures are byte-identical), so the Go, TypeScript, and Python SDKs are unaffected. (#3830)
• Onchain programs
  • Adapt to the solana 3.0 APIs: AccountInfo::realloc becomes resize, system-program and BPF-upgradeable-loader IDs move to solana-sdk-ids, ProgramError::BorshIoError is now a unit variant, and AccountInfo::new drops its rent_epoch argument. Bump the programs build toolchain to Rust 1.91.
• Client
  • Add a -route-liveness-backoff-max daemon flag to cap the Down-state liveness probe interval. Defaults to 60s (production behavior unchanged); the e2e harness pins a small value to avoid a probe gap that flaked the multi-client IBRL tests. (#3949)
  • Add a structured subscriptions array to doublezero status (after multicast_groups) with per-group detail — group pubkey, code, multicast IP, max bandwidth, and publisher/subscriber booleans — so consumers no longer have to parse the flattened P:/S: string. (#3964)
  • Originate a PIM Register beacon for multicast publishers: doublezerod periodically sends a PIM Register (encapsulating the publisher heartbeat) unicast to the RP over the tunnel, so the device originates the MSDP SA for the published source even on a dual-role publisher/subscriber tunnel, where pim ipv4 border-router source injection is suppressed by the subscriber-side PIM neighbor. (RFC-22)
• CI
  • Install agave v3.0.4 and build/test the SBF programs with platform-tools v1.54 (SBF_TOOLS_VERSION), required because the solana 3.0 dependency tree pulls edition2024 crates that need Cargo >= 1.85 (agave's default platform-tools v1.51 ships Cargo 1.84.1).
• Controller
  • Permit the unicast PIM Register to the RP (permit pim any host 10.0.0.0) on publisher multicast tunnels so the client-originated Register reaches the device; pim ipv4 border-router is retained as a backstop. (RFC-22)
• E2E tests
  • Bump the e2e base image to agave v3.0.4 and build the onchain programs with platform-tools v1.54 to match the solana 3.0 migration.
  • Pin the e2e ledger solana-test-validator to the deploy floor (agave 2.2.16, testnet) so a green e2e proves a change actually deploys and runs on the production cluster runtime. Previously the runtime validator rode the SBF build toolchain version (2.3.13); it is now decoupled and pinned independently. The build toolchain is unchanged. (#3957)
  • Fix a TestE2E_Multicast flake where the post-connect doublezero status check could observe only the first multicast group. After incrementally adding the second group, the test relied on WaitForTunnelUp, which returns immediately because the first tunnel is already up, so the single-shot status assertion could race the onchain propagation and the daemon's cached program data. Add an Eventually poll on doublezero user list for both groups before the post-connect checks.
• Smartcontract (Serviceability)
  • Honor a Permission account bearing ACCESS_PASS_ADMIN / USER_ADMIN on the UpdateMulticastGroupRoles grant path and the CreateSubscribeUser owner-override, so the feed oracle can subscribe validator-owned users and provision cross-owner users on a Permission account instead of foundation_allowlist membership. Granting multicast roles requires ACCESS_PASS_ADMIN; removal-only cleanup stays USER_ADMIN. Both handlers disambiguate the optional trailing Permission account from the EdgeSeat feed/device accounts by PDA match. The change is additive: every existing caller keeps its current authority. (#3966)

v0.28.0 - 2026-06-26

Breaking

• CLI
  • Remove the doublezero-admin binary. Its commands now live in the doublezero CLI as hidden subcommands (e.g. doublezero sentinel ..., doublezero migrate flex-algo).

Changes

• Client
  • Add a --no-wait flag to doublezero disconnect that skips waiting for the daemon to tear down the tunnel(s), exiting once the onchain user deletion is confirmed. (#3911)
• CLI
  • doublezero user subscribe can now remove multicast roles: --publisher/--subscriber accept an explicit value (--publisher false / --subscriber false) to drop a role, an omitted flag preserves the user's current role for the group, and the command errors when neither flag is given. Bare --publisher/--subscriber still mean true. (#3914)
  • Add hidden migrate flex-algo (RFC-18 link-topology and Vpnv4 loopback FlexAlgoNodeSegment backfill); the prior migrate command is now migrate user-pda. Moved from doublezero-admin.
  • Add hidden device migrate-multicast-counts and device migrate-unicast-counts to reconcile stale per-device subscriber, publisher, and unicast-user counts. Moved from doublezero-admin.
  • Add hidden sentinel find-validator-multicast-publishers and sentinel create-validator-multicast-publishers commands. Moved from doublezero-admin.
  • Feature-gate doublezero-sentinel's server-mode deps (Prometheus exporter) behind a default-on server feature and depend on it with default-features = false, so the doublezero binary no longer links rustls/aws-lc-sys. Restores the glibc floor (binaries built on Ubuntu 24.04 load on 22.04 again) and shrinks the binary.
  • Add a --narrow flag to device list, link list, and access-pass list that renders a width-reduced table for wide output — dropping low-value columns, abbreviating pubkeys to a copyable leading-prefix, and shortening headers — while leaving --json and the default table unchanged. (#3938)
• Onchain programs
  • Validate the device mgmt_vrf field against the account-code charset ([A-Za-z0-9:_-]) and a 32-byte length cap, matching the device code field. Empty (the default VRF) is still accepted.
  • Transfer connect/disconnect credits to the user's account when adding a user to a multicast group's publisher or subscriber allowlist, so the user can connect immediately. The airdrop is atomic with the allowlist update and mirrors set_access_pass (scaled for allow_multiple_ip passes). (#3851)
• SDK
  • Pass the user_payer account on the multicast allowlist add instructions so the onchain credit transfer can fund it.
• Controller
  • Skip rendering device config when a string field would not survive as a single config token (contains control or whitespace characters).
  • Prune a device's per-pubkey Prometheus series when it is removed from the on-chain ledger, so the Network: Device Stopped Calling Controller alert auto-resolves instead of firing forever on a frozen counter. Check-ins from ledger-absent pubkeys are rejected, counted on the new aggregate controller_grpc_getconfig_unknown_pubkey_total, and logged at WARN (rate-limited). Register controller_link_metrics/controller_link_metrics_invalid_total (previously populated but never exposed); on each cache update delete only the controller_link_metrics gauge series that have gone stale (active last cycle, absent now — covering inactive links, removed/renamed interfaces, code changes, and devices that gain a pathology) while leaving still-active series in place so scrapes never see a gap; and prune controller_link_metrics_invalid_total by device code when a device is removed from the ledger. (#3931)
• Device agents
  • Reduce agent CPU usage by continuing to fetch the full config every 5 seconds but only applying when it has changed or after 60s timeout
• E2E tests
  • Route all devnet networks (CYOA, default, and miscellaneous) through a shared collision-safe subnet allocator to prevent overlapping subnet assignments across concurrent test runs. (#3919)
  • Harden the mainnet-beta QA client against flaky/stale Solana RPC: multi-endpoint failover (with a public-RPC default fallback when SOLANA_RPC_FALLBACK_URLS is unset), active slot-lag detection, poll-until-consistent post-write reads, and configurable timeout/retry budgets. Eliminates manual SOLANA_RPC_URL repointing during RPC outages. (#3930)
  • Make device selection deterministic in the maxusers rollover test by waiting until the nearby device is measured faster than the faraway device by more than the client's 5ms latency tolerance, so the client cannot connect to the wrong device on a tie. (#3936)
  • Match CLI validation errors case-insensitively in the interface validation test, decoupling assertions from the program's error-message casing. (#3936)

v0.27.1 - 2026-06-10

Breaking

Changes

• Client
  • Revert auto-enabling allocated-IP mode on doublezero connect ibrl behind NAT (#3861): the RFC1918 heuristic misfires on 1:1 NAT hosts where plain IBRL works, silently changing the user type.

v0.27.0 - 2026-06-10

Breaking

Changes

• Client
  • Auto-enable allocated-IP mode for doublezero connect ibrl when the daemon detects a private RFC1918 default-route source (behind NAT), unless -a or --client-ip is set.
  • doublezero connect multicast with no groups now auto-joins every group authorized in the caller's AccessPass — publishing to mgroup_pub_allowlist and subscribing to mgroup_sub_allowlist. An AccessPass with no authorized groups is a no-op.
• Onchain programs
  • Add per-category seat caps to EdgeSeat access passes (errors 89/90 on overflow), scale the SetAccessPass airdrop by the cap sum when allow_multiple_ip is set, and drop the dynamic-pass IP-lock and IS_DYNAMIC flag. (#3859)
• CLI
  • access-pass set gains --max-unicast-users / --max-multicast-users; get/list show the per-category counts and caps.
• SDK
  • Decode the four new AccessPass cap fields (and the previously-missing tenant_allowlist) in the Go, Python, and TypeScript layouts.
  • GetAccessPassCommand and the multicast allowlist resolver both resolve a shared dynamic-seat AccessPass (the UNSPECIFIED PDA) before the exact-IP pass, matching the onchain create_user lookup. (#3853)

v0.26.0 - 2026-06-05

Breaking

Changes

• Onchain programs
  • Deprecate the AccessPassStatus::Expired access-pass status (renamed ExpiredDeprecated; discriminant 3 retained for wire compatibility). Access-pass epoch expiry no longer demotes users to OutOfCredits: update_status stops producing the status, and both try_activate (user creation) and CheckUserAccessPass (periodic re-check) keep users Activated. Epoch validity is still enforced at user creation for unicast users only; multicast publishers and subscribers are governed by mgroup_*_allowlist, not by epoch.
• SDK
  • Mirror the access-pass status rename in the Go, Python, and TypeScript deserializers: AccessPassStatusExpiredDeprecated (Go), EXPIRED_DEPRECATED (Python), and the "expired (deprecated)" string across all three.
• SDK (Rust)
  • Remove the client-side User not active precheck from the multicast subscribe/publish command (UpdateMulticastGroupRoles) so non-Activated users are no longer blocked before submission; authorization is enforced onchain.
• CLI
  • Extract doublezero-daemon-cli crate housing the DaemonClient trait and the enable, disable, and status daemon verbs. The new crate owns all daemon HTTP interaction (Unix-socket client, response types, shared output helpers) and is consumed by the doublezero binary. check_daemon binds get_environment() once per invocation instead of calling it per-check.
  • Fold version, account, accounts, log, and subscribe diagnostic verbs from the binary's top-level Command enum into ServiceabilityCommand per RFC-20. Each verb now takes &CliContext + generic &C: CliCommand + &mut W writer and is async. Add --json to account, accounts, and log (RFC-20 §Output). The binary-level subscribe override uses the real blocking DZClient::subscribe for live event streaming; the module crate's implementation falls back to a get_all() snapshot for testability.
  • Change geolocation user update-payment to update-payment-status for clarity.
  • geolocation user get: Show probe code, rather than probe pubkey in target list.
  • geolocation probe get: Show exchange code, rather than exchange pubkeys.
  • Remove env/argv reads and eprintln! from the serviceability CLI module per RFC-20 §67. The keypair-source pre-flight check moves from a standalone has_keypair_source() (which read std::env::args and stdin) to a CliCommand::has_keypair_source method computed once by the binary at startup; diagnostic output in check_id, check_balance, check_allowlist, and print_error now routes through tracing::error! instead of writing to stderr directly.
  • Add --json output to globalconfig feature-flags get and accesspass user-balances per RFC-20 §Output. feature-flags get now renders a two-column (flags, raw) table by default instead of the prior Enabled feature flags: <names> (raw: <N>) / No feature flags enabled (raw: <N>) sentence.
• CI
  • e2e: report trusted fork e2e/shreds shard results onto the PR head SHA so branch protection's required e2e (shard N) / shard-e2e (shard N) checks are satisfied by a /run-e2e dispatch, removing the need for a maintainer to bypass the ruleset to merge fork PRs; also make the dispatcher's confirmation comment non-fatal so a capped GITHUB_TOKEN no longer fails the job after the runs have already launched (follow-up to #3777)
• Tools
  • tools/stress/device-reporter: surface device CPU + memory + agent RSS in the post-run summary. The summary subcommand now reads the observer's per-tick show processes top once JSON captures and observer.agent_metrics.json to render a ## Resource usage section: device CPU peak / p95 / sustained ≥ 80 % windows (matching the observer's cpu_sustained abort threshold), memory peak free / used / floor-violation count, and doublezero-agent resident-memory peak / end / per-minute slope. New -free-mem-floor-mb flag (default 1024, set 0 to disable) (#3845)
  • Complete the device-stress orchestrator (part 3): replace the stubbed agent runner with an SSH-backed runner that execs doublezero-agent -verbose on the DUT and tees its output to orchestrator.agent.log, and a log parser that turns the agent's commit-diff lines into pre_commit_log / applied runlog events. Adds --dut-ssh-user and --no-agent flags.
  • Add tools/stress/device-observer/, a per-device sampling tool that analyzes the output of the device-stress orchestrator and observer
• Telemetry
  • Drop the redundant ip-msdp-sa-cache kind from the state-ingest server's default state-collect command list. show ip msdp sa-cache rejected already returns the full SA cache (accepted SAs in the acceptedSaMsg array plus any rejected SAs in rejectedSaMsg), so the bare show ip msdp sa-cache collection is redundant — devices were running both commands per tick and uploading the same accepted-SA data twice. The ip-msdp-sa-cache-rejected kind is retained.
• Telemetry (geoprobe)
  • Retry transient bind: invalid argument failures when allocating per-probe UDP sockets in Publisher.AddProbe, matching the existing retry-on-bind pattern in Pinger. The shared retry helper is lifted into retry.go so the publisher and pinger paths use the same exponential-backoff logic. Fixes intermittent TestPublisher_RemoveProbe/TestPublisher_AddProbe CI flakes caused by concurrent ephemeral-port allocation (#3765)
• Makefile
  • Add unreadable_literal to make cargo clippy alert on large numbers written without _.

v0.25.1 - 2026-06-01

Breaking

Changes

• SDK (Go)
  • Add CreateUser / DeleteUser to the serviceability executor with cross-language wire-format fixtures and four new PDA helpers (GetUserPDA, GetAccessPassPDA, GetTunnelIdsPDA, GetDzPrefixBlockPDA)
• SDK (Rust)
  • Add DZClient::from_context and GeoClient::from_context, which build clients directly from a resolved RFC-20 CliContext instead of re-reading ~/.config/doublezero/cli/config.yml and re-applying moniker conversion. The context already carries the fully resolved ledger RPC/WS URLs and program IDs, so these constructors consume them verbatim, making the context the single source of truth and removing the double-resolution the binary previously incurred. Keypair precedence is preserved exactly (CLI flag > DOUBLEZERO_KEYPAIR > stdin > context keypair path > default): the raw --keypair flag is passed as the highest-precedence source and the context keypair path is used only as the low-precedence fallback, so the env var still wins. The new constructors and their doublezero-cli-core dependency are gated behind a cli-context cargo feature so non-CLI SDK consumers (controlplane, telemetry, e2e) keep a dependency-light default build. DZClient::new / GeoClient::new are unchanged for callers that do not build a CliContext (e.g. controlplane/doublezero-admin).
  • Drop the pre-submit simulate_transaction call in DZClient::execute_transaction_inner and submit with skip_preflight: true, eliminating the redundant double-simulation (the explicit simulate plus send_and_confirm_transaction's default preflight) on the happy path. Program logs are now recovered from get_transaction on the failure path so SimulationError / SimulationTransactionError and DoubleZeroError mapping in CLI output are unchanged. Trade-off: failing transactions now land onchain and burn fees instead of failing for free at simulation (#3750)
• CLI
  • Honor the build-configured default environment (Testnet by default, MainnetBeta under the default-mainnet-beta feature) when neither --env nor a persisted config.yml selects one. The RFC-20 context-build previously fell back to Environment::default(), which is always Devnet regardless of the build, so a testnet build with no config silently targeted Devnet's ledger URLs and program IDs. The binary now resolves the fallback through the new doublezero_sdk::default_environment(), matching the legacy DZClient::new defaults (default_program_id, ClientConfig::default) which already key off the compiled-in environment (#3810)
  • Construct the serviceability and geolocation SDK clients in the doublezero binary via DZClient::from_context / GeoClient::from_context, replacing the legacy DZClient::new(Option<String>, ...) bridge. The binary no longer round-trips the already-resolved CliContext values back through the SDK's config-file re-resolution. No user-facing command, flag, or output change.
  • Restore environment-moniker support for the --program-id and --geo-program-id global flags. The context-build resolved both flags with a raw parse::<Pubkey>(), so a moniker (e.g. --geo-program-id testnet) failed to parse, was silently dropped, and the binary fell back to the environment default. Both flags now accept monikers in their full (mainnet-beta, testnet, devnet, local) and short (m, t, d, l) forms, resolving to the matching program ID; a literal pubkey still passes through. A value that is neither a known moniker nor a valid pubkey is now a hard error instead of being silently ignored. convert_program_moniker is broadened to cover all four environments (previously only devnet/testnet), matching convert_geo_program_moniker.
  • Treat --env as a base that the per-field flags override, rather than being mutually exclusive with them. --env <name> resolves the whole network (ledger URL, WS URL, Solana L1 URL, serviceability and geolocation program IDs); --url, --ws, --solana-url, --program-id, and --geo-program-id each override only their own value on top (precedence: explicit flag > --env, per RFC-20 §override hierarchy). Applies to both the global doublezero flags and doublezero config set. Previously the global flag rejected the combination at the clap layer (ArgumentConflict) and config set printed Invalid flag combination and exited without writing, so --env local --program-id <X> was not possible.
  • Add --solana-url <SOLANA_RPC_URL> global flag to doublezero per RFC-20 §Global flags. Distinct from --url, which continues to override the DZ ledger transport; --solana-url targets the Solana L1 transport. The flag is parsed and exposed on the binary's App struct; per-verb consumption lands when verbs migrate to construct typed Solana L1 clients from CliContext.
  • Add --log-level <LEVEL> global flag and initialize the tracing subscriber at startup. LEVEL is one of off, error, warn (default), info, debug, trace. Diagnostic logs go to stderr so --json output on stdout remains parseable. Honors the RUST_LOG environment variable when set, overriding the CLI-flag level for per-module filtering. Replaces the previous println!("using keypair: ...") stdout line with a tracing::info! event; the keypair confirmation now appears only at --log-level info or higher and no longer pollutes parseable stdout. (Named --log-level rather than the RFC-20 §Global-flags suggested --verbose / -v because the existing doublezero connect / disconnect subcommands already own a --verbose flag with bool type; the global flag deviation will be revisited when the daemon-control module crate is carved out.)
  • Build a CliContext once at binary startup from --env, the per-field global overrides (--url, --ws, --solana-url, --program-id, --geo-program-id, --keypair, --sock-file), and the persisted ~/.config/doublezero/cli/config.yml (overridable via DOUBLEZERO_CONFIG_FILE), per RFC-20 (§CliContext). Precedence (highest wins): CLI flag > persisted config > env-derived default. When --env is not set and the persisted config has a serviceability program ID, the environment is derived from that program ID via Environment::from_program_id; otherwise the binary falls back to Environment::default(). The legacy DZClient is now constructed from the fully resolved CliContext URL, WebSocket, and program-ID values directly, so verbs that migrate to read CliContext see the same backend as the legacy bridge. Keypair resolution is intentionally left to DZClient::new's internal load_keypair precedence (CLI --keypair flag > DOUBLEZERO_KEYPAIR env var > stdin > persisted config) so the DOUBLEZERO_KEYPAIR env var continues to override the persisted keypair path, as relied on by the e2e contributor-auth negative-authz suite. File reads happen only in the binary; module crates remain forbidden from touching the filesystem (RFC-20 §67).
  • Centralize top-level error rendering through doublezero_cli_core::error::render_eyre. Replaces three ad-hoc eprintln!("Error: {e}") sites in client/doublezero/src/main.rs (env-parse failure, env-config resolution failure, top-level command failure) with a single helper that prints Error: <head> followed by the full chain of causes on stderr.
  • Rename the smartcontract/cli/ crate from doublezero_cli to doublezero-serviceability-cli to satisfy RFC-20's module-crate naming contract (doublezero-<module>-cli in kebab-case). The crate stays at smartcontract/cli/; only the [package].name and [lib].name change (lib name is doublezero_serviceability_cli because Rust requires underscores in import paths). All in-tree consumers are updated: client/doublezero, client/doublezero-geolocation-cli, controlplane/doublezero-admin, and the workspace Cargo.toml. External operators who depend on the workspace crate by its old name (doublezero_cli) must update their Cargo.toml and use statements. No user-facing command, flag, or output change.
  • Migrate location get to the RFC-20 conforming verb pattern as the project's reference. GetLocationCliCommand::execute is now async fn, takes &CliContext as its first non-self argument, and emits a tracing::debug! event so -v surfaces what the verb is doing. The verb's user-facing args, flags, table layout, and JSON schema are unchanged. The unit test consumes the shared doublezero_cli_core::testing::cli_context_default_for_tests() helper and continues to use the existing MockCliCommand (auto-generated by #[automock]) as the backend. Binary dispatch arms in client/doublezero and controlplane/doublezero-admin are updated to .await the new method; other location verbs (Create, Update, List, Delete) keep their current sync signatures and migrate opportunistically.
  • Add docs/cli-standard.md, the contributor-facing summary of RFC-20 with the location get worked example and pointers to the shared validators, formatters, logging facade, and test helpers in doublezero-cli-core.
  • Update CLAUDE.md with a CLI-standard section pointing at RFC-20, the contributor doc, and the reference verb so future contributors land on the standard quickly.
  • Move the per-resource serviceability subcommand wrapper files (accesspass, config, contributor, device, exchange, globalconfig, link, location, multicastgroup, permission, resource, tenant, user) from client/doublezero/src/cli/ into the module crate at smartcontract/cli/src/cli/ per RFC-20 §Module contract item 2. Internal imports in the moved files switch from doublezero_serviceability_cli::<resource>::* to crate::<resource>::*. Binary import paths in client/doublezero/src/{cli/command.rs,main.rs} switch to doublezero_serviceability_cli::cli::<resource>::*. cli/multicast.rs stays in the binary because its Subscribe/Unsubscribe/Publish/Unpublish variants are async and depend on binary-local daemon-control infrastructure (ServiceControllerImpl, crate::command::helpers::resolve_client_ip); the binary now imports MulticastGroupCliCommand from the library.
  • Introduce doublezero_serviceability_cli::cli::ServiceabilityCommand, the module crate's top-level subcommand enum + async fn execute(ctx, client, out) dispatcher per RFC-20 §Module contract item 2. Aggregates 17 serviceability variants (Init, Migrate, Address, Balance, Config, GlobalConfig, Location, Exchange, Contributor, Permission, Tenant, Device, Link, AccessPass, User, Export, Keygen, Resource) and owns the full dispatch tree currently inlined in client/doublezero/src/main.rs. Defined but not yet wired into the unified binary; the next PR adds #[command(flatten)] Serviceability(ServiceabilityCommand) to the binary's Command enum and collapses main.rs to a single dispatch arm.
  • Hoist ServiceabilityCommand into the unified doublezero binary via #[command(flatten)] on the binary's Command enum. Drops 17 explicit variants and collapses the main.rs dispatch match block from roughly 270 lines to one arm (Command::Serviceability(cmd) => cmd.execute(&ctx, &client, &mut handle).await). The binary retains daemon-control verbs (Connect, Enable, Disable, Status, Disconnect, Latency, Routes), raw-DZClient diagnostics (Account, Accounts, Log), the binary-local geolocation tree, InitGeolocationConfig, the multicast dispatch (whose Subscribe/Unsubscribe/Publish/Unpublish async arms depend on daemon-control infrastructure), and the Completion generator. User-facing doublezero --help is byte-identical to the pre-refactor output (29 visible top-level commands); no flag, name, or output change. Binary Command enum and main.rs dispatch are unchanged; this is pure file relocation.
  • Move MulticastGroupCommands dispatch out of client/doublezero/src/main.rs and into a pub fn execute(&client, &mut out) method on the enum itself, defined next to the enum in smartcontract/cli/src/cli/multicastgroup.rs. Mirrors the per-resource dispatch pattern in ServiceabilityCommand::execute and finishes the flatten/collapse work for the one module-crate subtree reached through the binary's MulticastCliCommand wrapper (which has to stay binary-local because its Subscribe/Unsubscribe/Publish/Unpublish arms depend on ServiceControllerImpl). The binary's Multicast arm shrinks from a 5-level nested match (Allowlist → Publisher/Subscriber → Add/Remove/List, plus Create/Update/List/Get/Delete at the Group level) to MulticastCommands::Group(args) => args.command.execute(&client, &mut handle). No flag, name, or user-facing output change.
  • Add cross-verb helpers in doublezero-cli-core to drop the per-verb boilerplate that every list/get/create/update/delete verb repeats today: the require! macro (one-line readiness check that expands to client.check_requirements(flags.bits())? so the legacy u8 trait signature stays unchanged and MockCliCommand keeps working), render_collection<T: Tabled + Serialize> and render_record<T: Tabled + Serialize> (the --json / --json-compact / table three-branch switch), print_signature and print_signature_and_then (the Signature: <sig> write-verb tail and its --wait companion), and OutputFormat::from_flags(json, json_compact) (resolves the per-verb boolean flags to the enum). tabled becomes a doublezero-cli-core dependency so the rendering helpers can construct tables; module-crate verbs continue to import tabled::Tabled directly for their per-verb display types. Helpers ship with unit-tested byte-identical output to the pre-refactor code paths.
  • Add resolve_location_pk(client, pubkey_or_code) to smartcontract/cli/src/helpers.rs, the per-resource pubkey-or-code resolver used by location update and location delete. Centralizes the existing client.get_location(GetLocationCommand { pubkey_or_code: ... })? pattern so the verb body no longer carries the lookup boilerplate; additional resolvers (resolve_device_pk, resolve_link_pk, ...) land per-resource as their sweep PRs migrate the other verbs.
  • Migrate all five location verbs (create, update, list, get, delete) to the RFC-20 conforming shape ahead of the per-resource sweeps. Every verb is now pub async fn execute(self, ctx: &CliContext, client: &C, out: &mut W) -> eyre::Result<()>, consumes the new helpers (require!, render_collection, render_record, print_signature, resolve_location_pk), and forwards through the dispatcher with .await. Behavior is byte-identical: table layout, JSON schema, Signature: <sig> line, and the --json / --json-compact semantics all match the pre-refactor output exactly (existing tests pass without assertion changes). controlplane/doublezero-admin's LocationCommands arm is updated to forward &ctx and await every verb. Other resources (exchange, contributor, tenant, device, link, user, multicastgroup, accesspass, globalconfig, permission, resource) continue to compile against their existing sync pub fn execute(self, client, out) signatures and migrate opportunistically in subsequent PRs.
  • Lift the block_on async-test helper into doublezero_cli_core::testing instead of redeclaring it verbatim in every async verb's test module (it was already copied five times across the location verbs). The helper is gated behind a new testing cargo feature so tokio stays an optional dependency and the default doublezero-cli-core build remains dependency-light; module crates enable doublezero-cli-core = { workspace = true, features = ["testing"] } in their dev-dependencies. The five location verb tests now import block_on from the shared module.
  • Ship shell-completion scripts in the client installer and recommend bash-completion so apt/dnf pull it in when available. build/ is added to .gitignore.
  • Migrate all six exchange verbs (create, update, list, get, delete, set-device) to the RFC-20 conforming shape on top of the shared CLI helpers. Every verb is now pub async fn execute(self, ctx: &CliContext, client: &C, out: &mut W) -> eyre::Result<()>, consumes the helpers (require!, render_collection, render_record, print_signature), and the update/delete/set-device paths route their pubkey-or-code argument through a new resolve_exchange_pk helper in smartcontract/cli/src/helpers.rs. The pre-existing BGP community range check in exchange update is preserved. exchange set-device retains its legacy Option<String>::and_then semantics for --device1 / --device2 (an unknown device silently resolves to None, which clears the slot) under an explanatory comment. controlplane/doublezero-admin, the unified doublezero binary, and the serviceability dispatcher all forward &ctx and await every exchange arm. Behavior is byte-identical: table layout, JSON schema, Signature: <sig> line, and --json / --json-compact semantics match pre-refactor output exactly; all 7 exchange unit tests pass without assertion changes.
  • Migrate all five contributor verbs (create, update, list, get, delete) to the RFC-20 conforming shape on top of the shared CLI helpers. Every verb is now pub async fn execute(self, ctx: &CliContext, client: &C, out: &mut W) -> eyre::Result<()>, consumes the helpers (require!, render_collection, render_record, print_signature), and update / delete route their pubkey-or-code argument through a new resolve_contributor_pk helper in smartcontract/cli/src/helpers.rs. The duplicate-code precondition in create and update is preserved, as is the owner = "me" short-circuit in create that resolves to the payer. update's pubkey resolution now goes through the shared helper rather than an in-line Pubkey::from_str (the old code-by-pubkey path was a code-or-pubkey path despite the variable name; the resolver accepts both). controlplane/doublezero-admin, the unified doublezero binary, and the serviceability dispatcher all forward &ctx and await every contributor arm. Behavior is byte-identical: table layout, JSON schema, Signature: <sig> line, and --json / --json-compact semantics match pre-refactor output exactly; all 5 contributor unit tests pass without assertion changes.
  • Migrate the 5 multicastgroup CRUD verbs (create, update, list, get, delete), the 6 multicastgroup allowlist verbs (publisher + subscriber add/list/remove), the 6 standalone foundation/QA allowlist verbs (foundation add/list/remove, qa add/list/remove), the 8 user verbs (create, create-subscribe, subscribe, request-ban, update, list, get, delete), and the 9 globalconfig verbs and sub-tree verbs (get, set, set-version, airdrop get/set, authority get/set, feature-flags get/set) to the RFC-20 pub async fn execute(self, ctx: &CliContext, client, out) signature. Signature-only sweep: verb bodies (including the post-write --wait polling in user create-subscribe/subscribe and the bespoke multicastgroup update re-fetch flow) are unchanged. MulticastGroupCommands::execute itself flips from sync to async and propagates ctx through its nested allowlist arms; the binary's Multicast arm becomes args.command.execute(&ctx, &client, &mut handle).await (single line). Test files gain the per-file block_on shim and cli_context_default_for_tests() import. controlplane/doublezero-admin, the unified doublezero binary, and the serviceability dispatcher all forward &ctx and await every multicastgroup, allowlist, user, and globalconfig arm. All 345 unit tests pass byte-identically. Helper adoption (require!, print_signature, render_collection, render_record) lands opportunistically in follow-up PRs.
  • Migrate the 11 device and device interface verbs (device create, update, list, get, delete, set-health plus interface create, update, list, get, delete) and the 14 link and topology verbs (link accept, delete, wan create, dzx create, get, latency, list, set-health, update plus topology assign-node-segments, clear, create, delete, list) to the RFC-20 pub async fn execute(self, ctx: &CliContext, client, out) signature. Signature-only sweep: verb bodies (including --wait polling via poll_for_*_activated, the per-verb requirement checks, and the Signature: writes) are unchanged. Test files gain a per-file block_on shim and cli_context_default_for_tests() import so the existing sync #[test] bodies can drive the now-async execute. controlplane/doublezero-admin, the unified doublezero binary, and the serviceability dispatcher all forward &ctx and await every device, interface, link, and topology arm. All 345 unit tests pass byte-identically (92 in the migrated modules: device 46, link 29, topology 17). Helper adoption (require!, print_signature, render_collection, render_record) lands opportunistically in follow-up PRs; the --wait polling flow on device create/update, device interface create/update, link wan-create/dzx-create/accept/update needs special handling there since the post-signature poll has to be preserved.
  • Migrate all six accesspass verbs (set, close, list, get, user-balances, fund) and all six resource verbs (allocate, create, deallocate, get, close, verify), plus the eight leaf single-file verbs (address, balance, init, migrate, keygen, export, config get, config set), to the RFC-20 pub async fn execute(self, ctx: &CliContext, client, out) -> eyre::Result<()> signature. The five small leaf verbs (address, balance, init, migrate, keygen) also adopt the require! macro and (where applicable) the print_signature helper since their bodies were one-line readiness checks paired with a single Signature: write. The larger and more idiosyncratic verbs (config get/set which manipulate the persisted YAML, export which serializes the whole graph, the accesspass and resource verbs which contain bespoke output and progress-spinner logic) keep their existing bodies for now and only get the signature flip; helper adoption for those lands opportunistically in follow-up PRs. controlplane/doublezero-admin, the unified doublezero binary, and the serviceability dispatcher all forward &ctx and await every accesspass, resource, and leaf-verb arm. config get/set tests gain a per-file block_on shim and a cli_context_default_for_tests() import so the existing sync #[test] bodies can still drive the now-async execute. The bespoke accesspass fund signature (R: BufRead for stdin) is preserved — only the _ctx parameter is inserted after self. Behavior is byte-identical: table layouts, JSON schemas, Signature: lines, the fund interactive flow, and the config text output all match the pre-refactor strings exactly; all 345 unit tests pass without assertion changes.
  • Migrate all eight tenant verbs (create, update, list, get, delete, administrator add, administrator remove, update-payment-status) and all six permission verbs (set, suspend, resume, delete, get, list) to the RFC-20 conforming shape on top of the shared CLI helpers. Every verb is now pub async fn execute(self, ctx: &CliContext, client: &C, out: &mut W) -> eyre::Result<()>, consumes the helpers (require!, render_collection, render_record, print_signature), and tenant verbs that accept a pubkey-or-code identifier (update, delete, add-administrator, remove-administrator, update-payment-status) route through a new resolve_tenant_pk helper in smartcontract/cli/src/helpers.rs. The duplicate-code precondition in tenant create is preserved, as is the administrator = "me" short-circuit. tenant delete's bespoke two-line output ("✓ Tenant 'X' deleted successfully\n Signature: ..."), its cascade-delete progress spinners, and its reference-count polling loop are preserved with manual writeln! calls and an explanatory comment. permission set's bespoke two-line aligned output ("Signature: ..." + "Permissions: ...") is preserved the same way. Permission verbs derive the on-chain PDA from user_payer rather than going through a pubkey-or-code resolver. controlplane/doublezero-admin, the unified doublezero binary, and the serviceability dispatcher all forward &ctx and await every tenant and permission arm. Behavior is byte-identical: table layout, JSON schema, Signature: line shape, and --json / --json-compact semantics match pre-refactor output exactly; all 18 tenant and 18 permission unit tests pass without assertion changes.
  • Migrate doublezero geolocation subcommands into the new doublezero-geolocation-cli module crate per RFC-20. The probe and user subtrees and the hidden init verb are now owned by the crate; the binary mounts them via GeolocationArgs from doublezero-geolocation-cli. The hidden top-level doublezero init-geolocation-config alias is removed; use doublezero geolocation init instead.
  • Validate CYOA/DIA interfaces have non-zero --bandwidth in doublezero device interface create and doublezero device interface update, and validate interface[a|z].bandwidth >= link.bandwidth in doublezero link wan-create and doublezero link dzx-create (DZX checks side A only). Mirrors the new onchain enforcement so misconfigured commands fail before submitting a transaction.
  • Validate interface[a|z].bandwidth >= link.bandwidth in doublezero link accept (DZX accept path) before submitting the transaction, with the same human-readable message style as link wan-create / link dzx-create. Mirrors the new onchain check.
  • Restore the default value for doublezero device interface create --bandwidth so it is optional again. #3077 dropped default_value from the clap attribute on bandwidth; because the field is u64 (not Option<u64>), clap then treated omission as a missing required argument, even though the PR description stated --bandwidth was now optional (#3775)
  • Route the doublezero user get "no Access Pass found" warning through tracing::warn! instead of eprintln!, so diagnostics go to the logging facade per RFC-20 §Diagnostic logging and no longer write directly to stderr. No user-facing output, flag, or schema change.
  • Reconcile the RFC-20 docs with the implemented global logging flag: rfcs/rfc20-cli-standardization.md and docs/cli-standard.md now describe --log-level <off|error|warn|info|debug|trace> (default warn) instead of the never-implemented repeatable --log-verbose. Documentation only; the binary is unchanged.
• Tools
  • Add tools/stress/device-orchestrator/, the device-stress orchestrator skeleton for the GRE Tunnel Capacity Study (part 2 of #3746). Runs a batched provision-then-reverse-deprovision sweep against a live serviceability program, dumping orchestrator-config.json and emitting a JSONL runlog of submit | confirm | activate | deprovision_* events. Cooperates with an abort sentinel file: finish the in-flight user, tear down everything created, exit non-zero. The SSH-backed agent runner (pre_commit_log / applied events) is stubbed behind pkg/agent.Runner and lands in part 3 (#3771).
  • Add tools/stress/device-observer/ initial scaffolding plus eAPI device sampler that writes per-tick snapshots of five show commands and an observer-config.json with the observer PID; collectors for Prometheus scrape, log tailers, and abort decider are stubbed and will be replaced in follow-up PRs.
  • Implement the tools/stress/device-observer/ Prometheus scraper for the doublezero-agent metrics endpoint. Each tick fetches --agent-metrics-url, parses the exposition response, and appends one NDJSON row per metric sample to observer.agent_metrics.json. Counter family totals are also exposed via a thread-safe Scraper.Snapshot() for downstream consumers. Per-tick HTTP, parse, or write failures log at WARN and the loop continues.
• E2E/QA
  • Configure the manager and client CLI with doublezero config set --env local, keeping per-field overrides for the container-specific ledger URLs and the deployed serviceability program ID (and geolocation program ID on the manager). The override is required because the sentinel multicast-publisher test deploys serviceability at a generated keypair and the compatibility tests clone testnet/mainnet program IDs; for standard stacks the env defaults already match the fixed localnet pubkeys. Exercises the new --env-as-base behavior end-to-end.
  • add trusted fork PR e2e dispatch (#3777)
  • e2e/qa: remove client-side capacity pre-filtering from ValidDevices, because the QA user pubkey bypasses capacity limits using the serviceability global-config qa-allowlist. Individual device failures no longer fail the test; instead, overall and per-host failure rates are evaluated after all batches and the test only fails if either exceeds --failure-threshold (default 10%) or --per-host-failure-threshold (default 20%).
• Controller
  • Enable eos-native gnmi provider (#3781)
• Smartcontract (Serviceability)
  • Enforce non-zero bandwidth on CYOA/DIA interfaces in process_create_device_interface and process_update_device_interface. On update the rule fires only when the transaction is changing CYOA, DIA, or bandwidth, so legacy zero-bandwidth CYOA/DIA interfaces already onchain can still be updated for unrelated fields without first being repaired. Enforce side_a_iface.bandwidth >= link.bandwidth (and side_z_iface.bandwidth >= link.bandwidth for WAN; DZX side Z is external) in process_create_link. Enforce the same rule for both side A and side Z in process_accept_link (the DZX accept path), so DZX side Z's bandwidth is validated when it is first bound and side A is re-validated in case it was lowered via process_update_device_interface between create and accept. All rejections surface as DoubleZeroError::InvalidBandwidth (Custom(31)).
• Telemetry
  • Replace the geoprobe MinCache best/backup eviction with a guarded-backup pattern: a backup is only collected while best is within its final maxAge/2 ("guard") window, so on best's expiry the promoted value is always a recent-window minimum rather than a stale fallback. A new record low resets best and clears backup, and expiry now promotes in a loop so a backup that is itself already expired cannot be promoted. Best / BestRttNs become pure read-through accessors returning the lower of the two non-expired slots without mutating or promoting.
  • Add ip-mroute, ip-mroute-count, and the four MSDP show-commands (ip-msdp-summary, ip-msdp-pim-sa-cache, ip-msdp-sa-cache, ip-msdp-sa-cache-rejected) to the state-ingest server's default state-collect command list. Devices with --state-collect-enable will run each command via Arista eAPI and upload signed JSON snapshots to S3; downstream parsing into ClickHouse lands in separate lake/indexer/pkg/dzingest PRs (one per kind family).

v0.24.0 - 2026-05-22

Breaking

Changes

• Smartcontract
  • Deprecate the 13 contributor-side program instructions whose only client was the now-deleted activator: ActivateDevice (21), RejectDevice (22), CloseAccountDevice (27), ActivateLink (29), RejectLink (30), CloseAccountLink (35), ActivateMulticastGroup (47), RejectMulticastGroup (48), DeactivateMulticastGroup (53), ActivateDeviceInterface (72), RemoveDeviceInterface (75), UnlinkDeviceInterface (77), and RejectDeviceInterface (78). Dispatch arms now short-circuit to DoubleZeroError::Deprecated (custom code 67); processor files and argument structs are removed. Borsh variant tags are preserved (unit variants) so the wire format is unchanged — old clients receive a deterministic deprecation error rather than an unknown-instruction decode failure. Bumps MIN_COMPATIBLE_VERSION to 0.15.0 (the client/v0.14.1 git tag was a patch release built from a commit whose workspace Cargo version was still 0.14.0, so the v0.14.1 binary self-reports as 0.14.0 in its startup version check; v0.15.0 is the first release whose embedded version actually satisfies the intended ≥ 0.14.1 gate). Gated on onchain ProgramConfig.min_compatible_version ≥ 0.15.0 (#3623)
  • Deprecate the ActivateUser, RejectUser, CloseAccountUser, and BanUser user-lifecycle program instructions: dispatch arms now return DoubleZeroError::Deprecated (custom code 67), and the processor files / argument structs are removed. Borsh variant tags 37/38/43/45 are preserved so the wire format is unchanged. The activator was the only client of all four — CreateUser has been atomic-to-Activated since RFC-11, closeaccount was activator-driven only, and RequestBanUser is now atomic. Gated on onchain min_compatible_version ≥ 0.12.0 (#3622)
  • Extend SetUserBGPStatus with bgp_rtt_ns: u64 (smoothed BGP TCP RTT in nanoseconds, same unit as Link.delay_ns / Link.jitter_ns); append bgp_rtt_ns to the User account. Old payloads (status-only) decode with bgp_rtt_ns = 0 via BorshDeserializeIncremental; old serialized accounts decode with bgp_rtt_ns = 0 via the existing append-only field pattern. Deploy order is unconstrained.
• SDK (Rust)
  • Delete the now-dead {Activate,Reject,CloseAccount}DeviceCommand, {Activate,Reject,CloseAccount}LinkCommand, {Activate,Reject,Deactivate}MulticastGroupCommand, and {Activate,Reject,Remove,Unlink}DeviceInterfaceCommand wrappers and the corresponding orphaned trait methods on DoubleZeroProgram — none are reachable from any live caller (#3623)
  • Delete the now-dead ActivateUserCommand, RejectUserCommand, CloseAccountUserCommand, and BanUserCommand wrappers — none are reachable from any live caller. RequestBanUserCommand (operator-driven, atomic) is unaffected (#3622)
  • Unconditionally request the protocol maximum compute (1,400,000 CU) and heap frame (256 KiB) on every serviceability transaction sent through DZClient. Removes DoubleZeroClient::execute_transaction_with_compute_unit_limit and the per-instruction SET_GLOBAL_CONFIG_COMPUTE_UNIT_LIMIT / ASSIGN_TOPOLOGY_NODE_SEGMENTS_COMPUTE_UNIT_LIMIT constants — serviceability runs on a dedicated private Solana cluster, so raising every transaction to the protocol max is free (#3742)
• SDK (Go/TS/Python)
  • Add the bgp_rtt_ns field to the User deserializer in all three SDKs; the Go executor builder accepts and serializes the new field via a 10-byte instruction payload.
• Telemetry
  • bgpstatus now discovers VRFs by enumerating /var/run/netns/ instead of deriving namespace names from onchain tenant data and a hardcoded base prefix. Fixes multicast user BGP status never being collected on Arista EOS: the previous code assumed the default VRF was the agent's current Linux namespace, but on the device the default VRF is exposed as /var/run/netns/default while the agent runs in ns-management. Drops the --bgp-namespace flag wiring from the BGP status submitter (the flag is still used by the state collector); adds NetnsDir config (default /var/run/netns); removes the empty-string short-circuit in netns.RunInNamespace.
  • bgpstatus reports per-user BGP RTT onchain on every status-change or periodic-refresh write. RTT comes from the same INET_DIAG snapshot already used to detect ESTABLISHED sessions (state.BGPSocketState.RTTms × 1_000_000 to ns), so no new collection cost. RTT does not by itself trigger a submission; it piggybacks on writes that would already happen, capped by PeriodicRefreshInterval staleness. A Down submission always carries bgp_rtt_ns = 0 to avoid stale RTT outliving a missing session.
  • gnmi-writer now collects ISIS global state (isis_global_state: instance, NET, level capability per network instance) and the ISIS LSP overload bit (isis_overload_bit) into ClickHouse, with companion *_latest views; moves another slice of ISIS telemetry off the S3 pipeline and onto gnmi-writer so we can eventually retire the S3 ISIS path
  • gnmi-writer now stores per-interface unicast/multicast/broadcast packet counters (in/out) and FCS errors in the interface_state table, bringing it closer to parity with the interface counters already collected in InfluxDB. These already stream in the gNMI interface-state subtree, so there is no added collection cost; existing interface_state rows read back 0 for the new columns
  • Remove the InfluxDB writer from global-monitor; ClickHouse is now the sole telemetry backend, dropping the INFLUX_* env var configuration
  • Signed-TWAMP reflector in geoprobe-agent issues a per-pair 8-byte challenge nonce in Reply0.SinceLastRxNs and flags Reply1.NumOffsets bit 7 (new Challenged field on ReplyPacket) when Probe1.Sec || Frac echoes the nonce — proves the sender received Reply 0 before sending Probe 1, closing the pre-emit-Probe-1 attack on SinceLastRxNs. Backwards-compatible: legacy senders never echo the nonce so the flag bit stays 0, and they always ignored Reply 0's previously-zero SinceLastRxNs. Documented in RFC16's new "Challenge-Response Inbound Probing" subsection (#3737)
  • geoprobe-target-sender gains opt-in --challenged flag (default off). When set, the sender extracts the nonce from Reply0.SinceLastRxNs, writes it into Probe1.Sec || Frac, signs Probe 1 only after Reply 0 is parsed, and surfaces Reply1.Challenged on every per-pair log line (JSON "challenged", text Challenged Inbound:). Default off preserves the existing pre-sign-both / fire-Probe-1-immediately fast path byte-for-byte. Trade-off: challenged mode inflates Reply1.SinceLastRxNs by the sender's Probe 1 signing latency (#3738)
  • state-ingest no longer logs a spurious server exited with error: use of closed network connection at shutdown; the listener-closed race during graceful shutdown (net.ErrClosed) is now treated as a clean stop alongside http.ErrServerClosed
• CLI
  • Introduce doublezero-cli-core (crates/doublezero-cli-core/), the shared library crate that every doublezero-<module>-cli will reuse per RFC-20. Ships CliContext + CliContextBuilder (resolved configuration value carried into every verb), RequirementCheck bitflags aligned with the legacy CHECK_ID_JSON | CHECK_BALANCE | CHECK_FOUNDATION_ALLOWLIST bit values, the shared validator set (validate_pubkey, validate_pubkey_or_code, validate_code, validate_parse_bandwidth, validate_parse_delay_ms, validate_parse_jitter_ms, validate_parse_delay_override_ms), display formatters (DisplayVec, stringify_vec), a tracing + tracing-subscriber init_logging(verbosity) helper that writes to stderr, and a testing module with a CliContext builder for verb unit tests. Existing call sites in smartcontract/cli continue to compile unchanged: smartcontract/cli/src/validators.rs and formatters.rs are now thin pub use shims over the core crate.
  • Add solana_l1_rpc_url to doublezero-config::NetworkConfig. Per RFC-20 §Environments: mainnet-beta resolves to https://api.mainnet-beta.solana.com, testnet to https://api.testnet.solana.com, devnet to https://api.testnet.solana.com (intentional asymmetry, see RFC), and local to http://localhost:8899. A new DZ_SOLANA_RPC_URL environment variable overrides the resolved value, mirroring the existing DZ_LEDGER_RPC_URL / DZ_LEDGER_WS_RPC_URL overrides.
  • Drop the activator-only pollers from doublezero (user and multicastgroup activation waits). The --wait flag on user create, user create-subscribe, user subscribe, multicastgroup create, and multicastgroup update now fetches the post-create state once instead of polling; creates are atomic to Activated post-RFC-11, so the wait loop was watching a transition that no longer happens (#3614)
  • doublezero geolocation probe ... and user ... mirrors doublezero-geolocation versions; new --geo-program-id global flag, config get/set include Geolocation Program ID; new -init-geolocation-config for init of geolocation program
  • cli: doublezero geolocation probe ... and user ... mirrors doublezero-geolocation versions; new --geo-program-id global flag, config get/set include Geolocation Program ID.
  • geolocation probe list now includes signing pubkeys
  • Drop the activator-only pollers from doublezero (user and multicastgroup activation waits). The --wait flag on user create, user create-subscribe, user subscribe, multicastgroup create, and multicastgroup update now fetches the post-create state once instead of polling — creates are atomic to Activated post-RFC-11, so the wait loop was watching a transition that no longer happens (#3614)
  • Trim the Rejected status arm from the device and link activation pollers; Rejected was itself an activator-driven transition (#3614)
  • doublezero user get and doublezero user list surface BGP RTT as an rtt column (e.g. 5.50 ms, or - when no sample has been observed). JSON output includes raw bgp_rtt_ns alongside the pretty bgp_rtt string.
  • Remove standalone doublezero-geolocation binary; use doublezero geolocation ... instead.
• Client
  • Simplify doublezero connect's post-create user fetch to a fixed retry-on-RPC-lag get instead of waiting for UserStatus::Activated; the activator-driven transition is gone, so the fetch only needs to ride out replica lag (#3614)
• E2E
  • Switch geolocation invocations to doublezero geolocation ... and doublezero init-geolocation-config
• Agent: log after Arista eapi commit
• Agent: log received config size in bytes and expose doublezero_agent_config_size_in_lines and doublezero_agent_config_size_in_bytes Prometheus gauges (#3741)
• Controller
  • Add --max-user-tunnel-slots flag to override the per-device user-tunnel slot count of 128 at runtime.

v0.23.0 - 2026-05-15

Breaking

Changes

• Smartcontract
  • Tenant administrators can now create or update access passes scoped to their tenant without being added to the foundation allowlist; non-privileged callers cannot remove a tenant they do not administer from an existing access pass.
  • Skip last_access_epoch enforcement for UserType::Multicast in CreateSubscribeUser and CheckUserAccessPass. Multicast access is gated by mgroup_pub_allowlist / mgroup_sub_allowlist on the access pass, not by epoch, so multicast users can be created and remain Activated regardless of the access-pass expiry. IBRL/unicast epoch enforcement is unchanged.
• Client
  • add --tenant flag to doublezero user update for foundation-driven tenant reassignment
  • break latency ties with avg latency (#362)
  • doublezero connect multicast no longer fails the client-side check_accesspass epoch check; only the AccessPass existence is verified for multicast. IBRL paths still enforce last_access_epoch >= current_epoch.
  • Break latency ties using average latency when ranking candidate devices (#3692)
  • Delete InterfaceV3 and the InterfaceDeprecated::V3 variant from the serviceability program. V3 was added by an earlier change, never written to production accounts, and reverted in #3653 / no longer produced after #3667; this removes the dead type. Discriminant 3 is now an unused reserved slot in InterfaceDeprecated's encoding space — unknown discriminants fall through to InterfaceV2::default(). Removes the V3 struct, its helper impls (From<InterfaceV2>, TryFrom<&InterfaceV1>, Default, TryFrom<&InterfaceV3> for InterfaceV2), V3 match arms in InterfaceDeprecated::to_v2/size/Device::TryFrom, and the V3 cross-language byte-layout debug test. On-disk write format is unchanged (#3664)
  • break latency ties with avg latency (#362)
• SDK
  • Drop V3 handling from the Go, Python, and TypeScript serviceability readers: remove DeserializeInterfaceV3 (Go) and the version === 3 / version == 3 legacy-slot branches (Python/TS); remove the TestDeserializeInterfaceV3CrossLanguage Go test. The forward-compat trailing interfaces vec continues to carry flex_algo_node_segments via the size-prefixed body — that path is unchanged (#3664)
  • Let side-Z contributors update a link's status / desired_status / delay_override_ns via UpdateLinkCommand; the Rust SDK now auto-detects the signer's side and builds the 4-account side-Z preamble the on-chain processor expects, instead of always sending the side-A layout (#3702)
• Controller:
  • Enforce interface MTU during config render from interface role (CYOA/DIA → 1500, fabric → 9000) instead of trusting onchain Interface.Mtu / Link.Mtu; render each parent interface exactly once with max of its subinterface MTUs; change the tunnel.tmpl fallback from 2048 to 9000. Guards against stale V1 onchain interfaces (Mtu = 0) and duplicate parent blocks that previously caused silent IS-IS adjacency failures (#3696)
• E2E tests
  • Make multicast QA failures self-explanatory: require a Multicast-typed status entry after multicast connect (instead of accepting a stale IBRL one), retry MulticastJoin briefly on "interface not found" to absorb the daemon/kernel race, snapshot host state once after 30s of zero packets, and heartbeat the publisher's tunnel status during send windows to catch silent regressions

v0.22.0 - 2026-05-08

Breaking

Changes

• Smartcontract
  • Rename the BackfillTopology instruction to AssignTopologyNodeSegments across the program, CLI, and Rust SDK; the instruction discriminant (110) and on-disk semantics are unchanged (#3648)
  • Extend CreateDeviceInterface with optional trailing topology PDA accounts (topology_count: u8); for Vpnv4 loopbacks under onchain allocation the processor allocates a FlexAlgoNodeSegment per topology atomically with interface creation, so newly-provisioned devices no longer need a separate AssignTopologyNodeSegments step. The CLI/SDK auto-discover existing topologies and pass them. Topology accounts are validated by program-owner and by first-byte AccountType::Topology (#3648)
  • Extend UpdateDeviceInterface with update_topologies: bool + topology_count: u8 to reconcile flex-algo node segments against a desired topology set on a Vpnv4 loopback (deallocate removed, allocate added, preserve unchanged); contributor owner or foundation allowlist can call it (#3648)
  • DeleteDeviceInterface now deallocates flex-algo node segments alongside the loopback's base SR ID under onchain deallocation (#3648)
  • Canonicalize the forward-compatible Device interface names: rename the new struct from NewInterface to Interface, the legacy enum from Interface to InterfaceDeprecated, the Device::new_interfaces field to Device::interfaces, and the legacy Device::interfaces field to Device::deprecated_interfaces. On-disk format and behavior are unchanged — Borsh is positional, so the regenerated device fixture is byte-identical. Mechanical rename across the serviceability program, processors, CLI, sentinel, client, controlplane admin, and the Rust SDK. The activator/ crate was already deleted in an earlier change so the rename does not touch it (#3663)
  • Stop reading Device::deprecated_interfaces outside of backward-compat tests. The delete_device processor and Device::validate now check the canonical Device::interfaces vec; the legacy slot is treated as wire-format-only state populated by the deserializer and projected by the serializer. CLI, processor, and Rust SDK construction sites use ..Default::default() to avoid initializing the legacy field. The unused UpdateDeviceCommand::deprecated_interfaces SDK command field is removed. Device::deprecated_interfaces itself is retained for backward-compat fixture tests that verify legacy-slot decoding (#3663)
  • Remove the CurrentInterfaceVersion type alias and the unused Device::find_interface_legacy helper. Tests that used to construct CurrentInterfaceVersion {...} (an InterfaceV2 literal) and convert into either the canonical Interface or the legacy InterfaceDeprecated enum now build Interface {...} directly. The legacy enum's into_current_version() method is replaced by to_v2() for the few backward-compat sites that still need an InterfaceV2 projection (#3663)
  • Stop writing InterfaceV3 from CreateDeviceInterface and UpdateDeviceInterface; CurrentInterfaceVersion is now InterfaceV2. MigrateDeviceInterfaces and BackfillTopology continue to write InterfaceV3 since they are admin-controlled and need the flex_algo_node_segments field
  • Add forward-compatible NewInterface struct in state/interface.rs with a size: u16 + version: u8 on-disk prefix, V3-shaped body, and flex_algo_node_segments. Older readers can use the size prefix to skip past unknown future versions in constant time. Additive only — no callers, processors, or SDKs change in this PR (#3666)
  • Append new_interfaces: Vec<NewInterface> to Device after max_multicast_publishers, behind a custom BorshSerialize that projects the on-disk legacy interfaces slot from new_interfaces (always Interface::V2 per #3653) and writes new_interfaces at the end of the layout. Legacy accounts with no trailing bytes deserialize cleanly: Device::try_from rebuilds new_interfaces from the legacy enum vec via per-variant TryFrom. Older readers continue to parse the legacy slot at its existing offset; newer readers gain forward-compat via the trailing vec. Mutations now go through Device::replace_interface / push_interface / remove_interface so both vecs stay in sync; find_interface returns &NewInterface and find_interface_legacy is a temporary helper for unrelated callers (#3665)
  • Migrate serviceability processors (device/interface/{create,update,activate,delete,reject,remove,unlink}, link/{accept,activate,closeaccount,create,delete,update}, topology/backfill) to read and mutate Device::new_interfaces directly. device.interfaces is no longer touched in processors/, and Device::push_interface now takes a NewInterface. BackfillTopology no longer mirrors flex_algo_node_segments into the legacy in-memory interfaces vec — segments live only in new_interfaces and are intentionally dropped on the V2-projected on-disk legacy slot (#3658)
  • Delete the MigrateDeviceInterfaces processor and integration tests from the serviceability program. Device::TryFrom<&[u8]> (#3665) now auto-promotes legacy interfaces into new_interfaces when the trailing vec is missing, and the next account write persists the promoted vec — no standalone migration step is needed. Variant 111 is retained as a Deprecated111() tombstone (no-op dispatch, slot reserved so it isn't reused) for compatibility with older clients still emitting the old discriminator (#3662)
• SDK
  • Go, Python, and TypeScript serviceability readers parse the trailing new_interfaces vec on Device with size-prefixed (u16 size + u8 version + body) forward-compat framing. Empty trailing falls back to rebuilding new_interfaces from the legacy enum vec, matching the Rust device reader. Length mismatch between the legacy and trailing vecs is surfaced as an error (Python/TS raise; Go sets Device.DeserializeError). Bumps CURRENT_INTERFACE_VERSION / CurrentInterfaceVersion to 4 across SDKs to match Rust's CURRENT_INTERFACE_SCHEMA_VERSION (#3660)
  • Regenerate device.{bin,json} through Device's custom serializer with a populated new_interfaces vec (one Vpnv4 loopback carrying a FlexAlgoNodeSegment, one physical user-tunnel-endpoint), and add device_legacy.{bin,json} (legacy interfaces vec only, no trailing bytes — exercises the SDK legacy-fallback path) and device_future_version.{bin,json} (last trailing-vec element doctored to version=5 with 8 trailing junk bytes — exercises the SDK skip-to-end path). Adds fixture-driven Go SDK tests; extends the existing Python/TS fixture tests to cover all three Device fixtures (#3661)
  • Expose SubscriptionStartSlot and LastUSDCPriceDollars on the shreds Go SDK ClientSeat struct, mapped to the existing prorated-billing fields in the onchain layout (#3684)
• SDK
  • Apply the same rename in the Go, Python, and TypeScript serviceability readers: Go gets Device.DeprecatedInterfaces ↔ Device.Interfaces (with the new Interfaces taking the trailing-vec slot previously held by NewInterfaces); Python gets Device.deprecated_interfaces ↔ Device.interfaces; TypeScript gets Device.deprecatedInterfaces ↔ Device.interfaces. The Interface / DeviceInterface element type in each SDK already represents the canonical (new) format and needs no rename. Length-mismatch error messages now reference the canonical field names (#3663)
  • Migrate read callers in the CLI, sentinel, client, controlplane admin, and Rust SDK topology helper to read interfaces from Device::new_interfaces instead of the legacy interfaces enum vec, and adopt the Device::find_interface signature that returns &NewInterface. The legacy interfaces slot is still written on-disk via the per-write V2 projection from #3667; this PR only migrates reads. The temporary Device::find_interface_legacy helper is retained for the smartcontract program processors, which migrate in a later issue. Activator is intentionally excluded — it is deprecated (#3659)
• Controller
  • Stamp the default UNICAST-DEFAULT topology color on tunnels for users whose access pass has no tenant; previously the color was only resolved inside the tenant lookup branch, leaving tenantless users (the majority on mainnet/testnet) with no color community (#3648)
• CLI
  • doublezero device interface get displays flex_algo_node_segments as topology_name:sr_id rows, falling back to a truncated pubkey when the topology cannot be looked up (#3648)
• Activator
  • Delete the activator/ crate from the workspace; onchain allocation (RFC-11) supersedes it. The deployed activator was frozen in Phase 1 (#3608, #3628) and removed from e2e in Phase 2 (#3609, #3610, #3611, #3629). The */activate, */reject, and */closeaccount onchain instructions and their SDK command modules remain in place for older CLIs until the min-version gate (#3612)

v0.21.0 - 2026-05-01

Breaking

Changes

• Smartcontract
  • Add AccessPassType::EdgeSeat(Pubkey) variant to associate an access pass with a specific onchain Seat pubkey
  • Add --accesspass-type edge-seat --seat <PUBKEY> to access-pass set
  • Add --edge-seat and --seat-pubkey filters to access-pass list
• Client
  • Add --sock-file global flag (aliases: --socket, --socket-path) to the doublezero CLI to override the default Unix socket path used to communicate with doublezerod (/var/run/doublezerod/doublezerod.sock)
• Controller
  • Fix unknown BGP peer cleanup in the Arista EOS template: hoist per-peer no neighbor X removal into its own router bgp 65342 block so EOS's silent context-exit on no neighbor for a non-existent peer can't misroute subsequent peers' removal commands (#3627)
• Sentinel
  • Pass dz_prefix resource accounts when creating the multicast user to force onchain allocation
• Smartcontract
  • Allow ip_net to be passed when creating a device interface if the interface is CYOA/DIA/user_tunnel_endpoint
• Telemetry
  • Add agent_version and agent_commit to WriteDeviceLatencySamples so the onchain header is refreshed on every write (~60s) instead of only at initialization; fixes stale version reporting after mid-epoch agent upgrades (#3598)
• CLI
  • doublezero -V now shows client version, program version, and minimum required version fetched from the serviceability program

v0.20.0 - 2026-04-29

Breaking

Changes

• Telemetry
  • Fix BGP status submitter to collect socket stats and tunnel interfaces from all tenant VRF namespaces (ns-vrf<N>), not only ns-vrf1; users whose tenant has a non-default VrfId were previously always reporting "tunnel not found" and had their onchain BGP status left stale
  • Fix BGP status submitter to collect from the root Linux network namespace when multicast users are present on a device; multicast GRE tunnels live in the global VRF (root namespace) rather than a per-tenant namespace, so their BGP sessions were never detected and onchain status remained permanently stale
• CLI
  • Add --narrow flag to doublezero user list that hides location, cyoa_type, accesspass, and tunnel_net, abbreviates user_type, and summarizes groups as one publisher entry plus one subscriber entry with independent +N overflow counts; default output is unchanged
  • Add doublezero-geolocation user update --user <code-or-pubkey> --token-account <pubkey> to update a geolocation user's payment token account; the underlying UpdateGeolocationUser instruction was already onchain but had no CLI entrypoint
• Smartcontract
  • Allow count > max in Device::validate for all four per-device caps (max_users, max_unicast_users, max_multicast_subscribers, max_multicast_publishers) so operators can lower a cap below the live count; admission-time gates in user create still reject new connections when at capacity, letting the live count drain through natural churn

v0.19.0 - 2026-04-24

Breaking

Changes

• Controller
  • Auto-loads /etc/doublezero-controller/features.yaml at startup if present (silently skips if absent); when flex_algo.enabled: true, populates topology data into the state cache, resolves tenant color communities from Tenant.include_topologies, and emits IS-IS flex-algo node segment and BGP color community stamping blocks into the Arista EOS template (disabled by default)
• SDK
  • Go serviceability SDK adds TopologyInfo account type with TopologyConstraint, IndexType / TopologyType account-type constants, and GetProgramData dispatch case; extends Link with LinkTopologies and LinkFlags; extends Tenant with IncludeTopologies
• CLI
  • Add tunnel_endpoint field to doublezero user list output (table and JSON) showing the device-side GRE endpoint IP assigned to each user
  • Add cyoa_ips field to doublezero device get and doublezero device list output, showing the IP networks of interfaces with user_tunnel_endpoint enabled
  • Add --tunnel_endpoint flag to user update command so operators can set the tunnel endpoint IP of an existing user
  • Extend doublezero resource verify to check MulticastPublisherBlock against multicast publisher users' dz_ip allocations; legacy dz_ips that fall outside the block's range are ignored so pre-existing users allocated before this extension existed do not produce false discrepancies
  • Fix doublezero resource verify to report missing TunnelIds resource extensions for all devices, including those without any users; previously the discrepancy was suppressed when a device had no users, hiding unallocated extensions
  • Extend doublezero resource verify to detect orphaned ResourceExtension accounts whose PDA does not correspond to any currently-expected resource type (global singleton or per-device extension for a live device/prefix); --fix closes them via the existing y/N confirmation flow
  • Add multicast subscribe, multicast unsubscribe, multicast publish, and multicast unpublish CLI commands so users can modify their multicast role set on a connected session without running disconnect. unpublish warns when the removal would drop the user's last publisher role (legacy-allocation environments may briefly reprovision in that case).
• Client
  • Filter devices by type-specific capacity during auto-selection so clients are not provisioned onto devices that have reached their unicast, multicast publisher, or multicast subscriber limits
• Collector
  • fallback to any probe if anchor probes aren't available
• Smartcontract
  • Fix BackfillTopology account ordering: payer and system_program are now correctly placed after the variable-length device list, not before it
  • Fix BackfillTopology SID collision: flex-algo node segment indices are now guaranteed not to duplicate any existing base node_segment_idx value on the device
  • Fix multicast group allowlist add/remove for AccessPasses created with allow_multiple_ip=true; the processors were rejecting requests with a real client IP because the stored IP is always 0.0.0.0 for these passes (#3551)
  • SDK now auto-detects the correct AccessPass PDA (static or dynamic) for allowlist operations based on whether an allow_multiple_ip pass exists
  • Add doublezero link topology {create,delete,clear,backfill,list} subcommands for managing flex-algo topologies; topology clear auto-discovers tagged links when --links is omitted
  • Add TopologyInfo onchain account for IS-IS flex-algo link classification: auto-assigned TE admin-group bit (1–62), derived flex-algo number (128 + bit), and constraint type (include-any/include-all); capped at 62 topologies via AdminGroupBits resource extension
  • Add link_topologies: Vec<Pubkey> (capped at 8) and link_flags: u32 (bit 0 = unicast-drained) to the Link account
  • Add include_topologies to the Tenant account for topology-filtered routing opt-in
  • Enforce UNICAST-DEFAULT topology existence as a precondition for link activation
  • Extend link get and link list to display topology assignments and drain status; add --link-topology <name> filter to link list and --link-topology (comma-separated topology names) / --unicast-drained flags to link update; use default as the value to clear all topology assignments
  • Extend tenant get and tenant list to display included topologies; add --include-topologies (comma-separated topology names) flag to tenant update; use default to clear
• Sentinel
  • Set a concrete tunnel_endpoint on multicast publisher create, preferring a user_tunnel_endpoint interface IP and falling back to the device's public_ip, excluding IPs already in use by another user at the same client_ip
  • Make the multicast publisher worker's --client-filter flag repeatable so multiple validator client names can be matched in one run (OR semantics), matching the admin CLI behavior
• Tools
  • Add doublezero-admin migrate flex-algo [--dry-run] command to backfill link topology assignments and VPNv4 loopback flex-algo node segments across all existing devices and links

v0.18.0 - 2026-04-17

Breaking

Changes

• Device Health Oracle
  • Add interface_counters activation criterion to device-health-oracle to verify devices have recent interface counter data in ClickHouse before activation
  • Add controller_success activation criterion to device-health-oracle to verify devices have consistent controller call coverage over a configurable burn-in period by querying ClickHouse
• Telemetry
  • Add GET /device-link/agent-versions endpoint to data-api and agent-versions subcommand to data-cli, exposing per-device telemetry agent version and commit from onchain DeviceLatencySamplesHeader
• Smartcontract
  • Allow SubscribeMulticastGroup for users in Pending status so that CreateSubscribeUser can be followed by additional subscribe calls before the activator runs (#3521)
  • Add optional owner field to UpdateMulticastGroup instruction, allowing foundation members to reassign ownership of a multicast group (#3527)
  • Rename SubscribeMulticastGroup instruction variant to UpdateMulticastGroupRoles and rename associated processor functions, args struct, and SDK command to use "roles" terminology, clarifying they manage publisher/subscriber roles rather than just subscriptions
• Geolocation
  • Add optional result destination to GeolocationUser so LocationOffsets can be sent to an alternate endpoint instead of the target IP; supports both IP and domain destinations (e.g., 185.199.108.1:9000 or results.example.com:9000); includes SetResultDestination onchain instruction, CLI user set-result-destination command, and Go SDK deserialization (backwards-compatible with existing accounts)
• CLI
  • Add --owner flag to multicast group update, accepting a pubkey or me (#3527)
  • Polish terminal output of connect and disconnect: fix emoji semantics, normalize message phrasing across IBRL and multicast code paths, resolve tenant to human-readable code on connect (errors if tenant not found), and fix progress bar not clearing before output in disconnect (#3529)
• Client
  • Reduce default probing interval to 5m from 30s since DZDs don't generally move.
• Dependencies
  • Bump vulnerable packages across Rust, Go, and Python to address Dependabot security alerts (14 packages fixed)
• DevContainer
  • Add optional DZ_WORKTREES_DIR mount exposing a host worktrees directory at /workspaces/worktrees inside the container; useful when docker exec-ing into the persistent dev container to work on git worktrees outside the repo. Defaults to /tmp/worktrees (empty, harmless) when unset

v0.17.0 - 2026-04-10

Breaking

Changes

• Activator
  • Fix duplicate tunnel underlay pairs after restart by registering device.public_ip as in-use for legacy users with unset tunnel_endpoint during allocation reload
• Client
  • Rank devices and tunnel endpoints by minimum observed latency (min_latency_ns) instead of average when selecting a connection target, preferring paths with the best achievable round-trip time
• Tools
  • Add IsRetryableFunc field to RetryOptions for configurable retry criteria in the Solana JSON-RPC client; add "rate limited" string match and RPC code -32429 to the default implementation
• Telemetry
  • Add shared telemetry/migrations package with goose-based ClickHouse schema migrations for all telemetry services; add CLICKHOUSE_RUN_MIGRATIONS env var to flow-enricher and gnmi-writer for on-startup schema migration (#3460)
  • Add optional TLS support to state-ingest server via --tls-cert-file and --tls-key-file flags; when set, the server listens on both HTTP (:8080) and HTTPS (:8443) simultaneously
  • Remove --additional-child-probes CLI flag from telemetry-agent; child geoprobe discovery now relies entirely on the onchain Geolocation program
  • Add BGP status submitter: on each tick, reads BGP socket state from the device namespace, maps each activated user to their tunnel peer IP, and submits SetUserBGPStatus onchain; supports a configurable down grace period and periodic keepalive refresh; enabled via --bgp-status-enable with --bgp-status-interval, --bgp-status-refresh-interval, and --bgp-status-down-grace-period flags
  • Bound CachingFetcher RPC calls with an explicit 30s timeout; context.WithoutCancel drops the parent deadline as well as cancellation, so without this a hung Solana RPC would block all singleflight waiters indefinitely
• Monitor
  • Add ClickHouse as a telemetry backend for the global monitor alongside existing InfluxDB
• E2E tests
  • Add TestE2E_GeoprobeIcmpTargets verifying end-to-end ICMP outbound offset delivery via onchain outbound-icmp targets
  • Refactor geoprobe E2E tests to use testcontainers entrypoints and onchain target discovery
  • Add TestE2E_UserBGPStatus verifying that the telemetry BGP status submitter correctly reports onchain status transitions as clients connect and establish BGP sessions
• Monitor
  • Add ClickHouse as a telemetry backend for the global monitor alongside existing InfluxDB
• SDK
  • Deserialize agent_version and agent_commit from device latency samples in Go, TypeScript, and Python SDKs
  • Add BGPStatus type (Unknown/Up/Down) and SetUserBGPStatus executor instruction to the Go serviceability SDK
• Sentinel
  • Improve find-validator-multicast-publishers and create-validator-multicast-publishers with multi-value --client filter, --ip filter, nearest-device selection, dynamic capacity re-evaluation, and a fix for multicast publisher owner being set to the sentinel payer instead of the validator's owner
• Smartcontract
  • Add agent_version ([u8; 16]) and agent_commit ([u8; 8]) fields to DeviceLatencySamplesHeader, carved from the existing reserved region; accept both fields in the InitializeDeviceLatencySamples instruction via incremental deserialization (fully backward compatible)
  • Implement SetUserBGPStatus processor: validates metrics publisher authorization, updates bgp_status, last_bgp_reported_at, and last_bgp_up_at fields on the user account
  • Add human-readable error messages for serviceability program errors in the Go SDK, including program log extraction for enhanced debugging
  • user get no longer fails when no Access Pass exists; it prints a warning to stderr and continues, showing an empty access pass field
  • Replace manual account validation assertions with validate_program_account! macro across serviceability processor files, adding consistent data_is_empty checks and fixing a missing is_writable validation in ResumeLink (#3436)
  • Extend validate_program_account! migration to remaining user and multicastgroup allowlist processors (set_bgp_status, delete, closeaccount, publisher/subscriber add/remove)
  • Add OutboundIcmp target type (= 2) to the geolocation onchain program, enabling ICMP-based probing as an alternative to TWAMP for outbound geolocation targets
  • Allow pending users with subs to be deleted
• Onchain programs
  • Add tunnel_endpoint field to the UpdateUser instruction (UserUpdateArgs), allowing the activator to overwrite a user's tunnel endpoint onchain; field is optional and backward compatible via incremental deserialization
• Telemetry
  • Device telemetry agent now posts agent_version and agent_commit in the DeviceLatencySamplesHeader when initializing new sample accounts, enabling version attribution of onchain telemetry data
  • Add optional TLS support to state-ingest server via --tls-cert-file and --tls-key-file flags; when set, the server listens on both HTTP (:8080) and HTTPS (:8443) simultaneously
  • Remove --additional-child-probes CLI flag from telemetry-agent; child geoprobe discovery now relies entirely on the onchain Geolocation program
  • Add BGP status submitter: on each tick, reads BGP socket state from the device namespace, maps each activated user to their tunnel peer IP, and submits SetUserBGPStatus onchain; supports a configurable down grace period and periodic keepalive refresh; enabled via --bgp-status-enable with --bgp-status-interval, --bgp-status-refresh-interval, and --bgp-status-down-grace-period flags
• Tools
  • Add IsRetryableFunc field to RetryOptions for configurable retry criteria in the Solana JSON-RPC client; add "rate limited" string match and RPC code -32429 to the default implementation
• Geolocation
  • Standardize CLI flag naming: probe mutation commands use --probe (was --code) accepting pubkey or code; rename --signing-keypair → --signing-pubkey and --target-pk → --target-signing-pubkey; add --json-compact to get commands
  • geoprobe-target can now store LocationOffset messages in ClickHouse
  • Add ICMP pinger to geoprobe-agent for measuring outbound ICMP targets with interleaved batch send/receive, integrated into the existing measurement cycle alongside TWAMP
  • Remove --additional-parent, --additional-targets, --additional-icmp-targets, and --allowed-pubkeys CLI flags from geoprobe-agent; all configuration now comes from onchain state via parent and target discovery
  • Add MinCache for tracking minimum-RTT measurements with best/backup promotion over a rolling TTL window, used by both geoprobe-target and geoprobe-target-sender to suppress redundant output and surface only new-best events

v0.16.0 - 2026-04-03

Breaking

Changes

• Smartcontract
  • Require that the access pass provided to SubscribeMulticastGroup belongs to the payer; foundation allowlist members may use any access pass.
  • Add Index account for onchain key uniqueness enforcement and O(1) key-to-pubkey lookup, with standalone CreateIndex/DeleteIndex instructions for migration backfill
  • Set minimum client version to 0.10.0
  • Enforce 9000-byte MTU on links and non-CYOA/non-DIA device interfaces; CYOA/DIA interfaces must be 1500. Onchain validation now returns InvalidMtu (error 46) for non-conforming values.
  • Add OutboundIcmp target type (= 2) to the geolocation onchain program, enabling ICMP-based probing as an alternative to TWAMP for outbound geolocation targets
• CLI
  • Allow incremental multicast group addition without disconnecting
  • Reset SIGPIPE to SIG_DFL at the start of main() in all 3 CLI binaries (doublezero, doublezero-geolocation, doublezero-admin) so the process exits silently like standard CLI tools
  • Support --type outbound-icmp in geolocation user add-target, remove-target, and get commands
  • Add sentinel admin commands to find and create multicast publishers for IBRL validators
  • handle non-user owned disconnects gracefully
  • Add user's multicast pub/sub groups if applicable to status
• Sentinel
  • Add multicast publisher worker with Solana RPC-based validator discovery
  • Add e2e tests for multicast publisher worker with validator-metadata-service mock
• SDK
  • Add Go SDK for shred subscription program with read-only account deserialization (epoch state, seat assignments, pricing, settlement, validator client rewards), PDA derivation helpers, RPC fetchers, compatibility tests, and a fetch example CLI
  • Add GeoLocationTargetTypeOutboundIcmp to Go geolocation SDK with deserialization and round-trip test support
• Device Health Oracle
  • Update link.health and device.health to ready-for-service and ready-for-users when they are not already in that state
• Tools
  • Add twamp-debug diagnostic tool for testing kernel timestamping support on switches; sends real TWAMP probes to verify which SO_TIMESTAMPING modes (RX/TX software/hardware/sched) actually deliver timestamps, and reports RTT statistics comparing userspace vs kernel timestamp sources
• E2E Tests
  • Switch backward compatibility test to install versioned CLI binaries from GitHub releases instead of Cloudsmith apt repos; version enumeration now uses the GitHub API directly from Go rather than querying apt-cache inside the container
• Client
  • Add doublezero_connection_info Prometheus metric exposing connection metadata (user_type, network, current_device, metro, tunnel_name, tunnel_src, tunnel_dst) (#3201)
  • Add doublezero_connection_rtt_nanoseconds and doublezero_connection_loss_percentage Prometheus metrics reporting RTT and packet loss to the current connected device

v0.15.0 - 2026-03-27

• Client
  • fix(client): fix latency field overflow by changing i32 to i64 (#3382)
  • fix(client): add user feedback to latency, add flag to limit icmp probe concurrency (#3385)

Breaking

Changes

• Funder
  • Top up contributor owner keys alongside device metrics publishers, multicast group owners, and the internet latency collector
• Smartcontract
  • Fix multicast publisher/subscriber device counter divergence: multicast_publishers_count never decremented and multicast_subscribers_count over-decremented on user disconnect because the decrement logic checked !publishers.is_empty(), which is always false at delete time. Add a durable tunnel_flags field to the User struct with a CreatedAsPublisher bit, set at activation, and use it in the delete and closeaccount instructions.
  • Allow foundation allowlist members and the sentinel to create multicast users with a custom owner via a new owner field on CreateSubscribeUser, enabling user creation on behalf of another identity's access pass
• CLI
  • Add --owner flag to doublezero user create-subscribe for specifying a custom user owner (foundation/sentinel only)

v0.14.0 - 2026-03-24

Breaking

Changes

• Controller
  • Log an error when duplicate tunnel-id assignments are detected on the same device during state cache update, instead of silently overwriting
• Onchain Programs
  • Serviceability: update device interface IPs when tunnel_net is changed via UpdateLink, matching the existing ActivateLink behavior (#3365)
  • Serviceability: AcceptLink supports combined accept+activate via use_onchain_allocation flag, gated on OnChainAllocation feature flag (#3369)
  • Serviceability: add feed_authority to RemoveMulticastGroupSubAllowlist auth check, matching AddMulticastGroupSubAllowlist
• Client
  • Get client IP from the daemon in the disconnect command, matching the connect command's behavior, to avoid IP mismatches behind NAT
• Onchain Programs
  • Add target_update_count field to GeoProbe account, incremented on AddTarget and RemoveTarget; uses BorshDeserializeIncremental so existing accounts default to 0 (non-breaking)
• SDK
  • Add TargetUpdateCount field to Go GeoProbe struct with backward-compatible deserialization
• Telemetry
  • Skip expensive GetGeolocationUsers RPC scan in geoprobe-agent when the probe's target_update_count is unchanged, with a forced full refresh every ~5 minutes as safety net
  • Add Prometheus metrics to geoprobe-agent: build info, error counters by type, discovery/measurement cycle durations, offset send/receive/reject counters, and discovered target/parent gauges; exposed via optional --metrics-enable flag

v0.13.0 - 2026-03-20

Breaking

Changes

• Activator
  • Reserve device allocations (loopback IPs and segment routing IDs) for devices in Drained, DeviceProvisioning, and LinkProvisioning states at startup, preventing collisions with new device allocations
  • Reserve user allocations (tunnel_net, tunnel_id, dz_ip, publisher IPs, tunnel endpoints) for users in Updating and OutOfCredits states at startup, preventing collisions with new user allocations
  • Fix duplicate tunnel_net/tunnel_id allocation by reserving addresses for links in HardDrained, SoftDrained, and Provisioning states during startup initialization
• Onchain Programs
  • Allow foundation to remove targets from GeolocationUser accounts via the RemoveTarget instruction, unblocking foundation-initiated user deletion when targets still exist
  • Serviceability: fix SubscribeMulticastGroup deriving the AccessPass PDA from payer_account.key instead of user.owner, which caused user delete to fail with "Invalid AccessPass PDA" when a foundation allowlist key signed and the user had active multicast subscriptions
  • feat(smartcontract): add tunnel_net/tunnel_id reallocation to UpdateLink (#3326)
• Client
  • Fix v2/status returning empty current_device and metro for multicast subscribers by adding a clientIP + UserType fallback in status enrichment when DzIp and tunnel_dst matching both fail
  • Set tunnel interface administratively down before deleting during teardown, so external applications with sockets bound to the tunnel's overlay IP receive errors before the interface is removed
• CLI
  • Include feed authority in global-config authority get output
  • Add geolocation user subcommands to manage GeolocationUser accounts and targets: create, delete, get, list, add-target, remove-target, and update-payment-status
• Monitor
  • Fix slack user reporting
• SDK
  • Add GeolocationUser types, Borsh deserialization, PDA derivation, and read-only client methods (GetGeolocationUserByCode, GetGeolocationUsers) to the Go geolocation SDK
• Telemetry
  • Add onchain target discovery to the geoProbe agent: polls GeolocationUser accounts at 60s intervals, filters for activated+paid users, dynamically updates outbound probe targets and inbound signed TWAMP authorized keys
  • Fix geoProbe parent discovery incorrectly adding parent authority keys to the signed TWAMP reflector allowlist; parent DZDs use the unsigned reflector
• Controller
  • Retry transient Solana RPC failures when fetching onchain serviceability accounts so controller polls are more resilient to short-lived provider resets

v0.12.0 - 2026-03-16

Breaking

Changes

• Onchain Programs
  • Add GeolocationUser account state to the geolocation program with owner, billing config (flat-per-epoch), payment/user status, and a target list supporting outbound (IP + port) and inbound (pubkey) geolocation targets
  • Add GeolocationUser CRUD instruction processors (create, update, delete) with owner authorization and target-empty guard on delete
  • Add GeolocationUser target management (AddTarget, RemoveTarget) with duplicate/not-found guards, public IP validation, MAX_TARGETS limit, and GeoProbe reference count tracking
  • Add foundation-only UpdatePaymentStatus instruction to set payment status and last deduction epoch on GeolocationUser billing config
• Client
  • Demote passive-mode liveness session-down log messages from Info to Debug to reduce log noise when no dataplane action is taken
• Telemetry
  • Add Version (uint8) and TargetIP ([4]byte) fields to LocationOffset wire format (v1, 174 bytes), with version validation on unmarshal to enable safe future format evolution
• Tools
  • Update TWAMP signed packet parser byte offsets and OffsetInfo struct for LocationOffset v1 layout
• Onchain Programs
  • Serviceability: allow reservation authority to create, update, and close access passes, with ownership restriction preventing modification of passes created by other authorities
  • Serviceability: fix multicast user creation failing when access-pass has a tenant_allowlist; tenant enforcement only applies to unicast connections since multicast users are not tenant-scoped
  • Serviceability: rename reservation_authority_pk to feed_authority_pk in GlobalState and rename RESERVATION permission flag to FEED_AUTHORITY
  • Serviceability: remove ReserveConnection, CloseReservation, CreateReservedSubscribeUser, DeleteReservedSubscribeUser instructions and Reservation account type
• E2E Tests
  • Fix TestE2E_UserLimits not asserting command failure: the ; echo EXIT_CODE=$? pattern caused the shell to always exit 0 regardless of the doublezero exit code, making err always nil; replace with require.Error assertions so the test fails if a limit-exceeded connect unexpectedly succeeds
  • Fix intermittent flake in TestE2E_MultiClientIBRL_RouteLiveness: add explicit route convergence checks after each unblock before starting the next block/unblock cycle, ensuring the liveness subsystem and BGP have fully settled
  • Add geoprobe E2E test (TestE2E_GeoprobeDiscovery) that exercises the full geolocation flow: deploy geolocation program, create probe onchain, start geoprobe-agent container, and verify the telemetry-agent discovers and measures the probe via TWAMP
  • Add geoprobe Docker image, geolocation program build/deploy support, and manager geolocation CLI configuration to the E2E devnet infrastructure
  • Extend geoprobe E2E test with outbound offset forwarding (agent → target with signature chain verification) and inbound signed TWAMP probing (target-sender → agent with DZD offset embedding); build geoprobe-target and geoprobe-target-sender binaries into the geoprobe Docker image
• Telemetry
  • Add onchain parent DZD discovery to geoprobe-agent: periodically queries the Geolocation program for this probe's parent devices and resolves their metrics publisher keys from Serviceability, replacing the need for static --parent-dzd CLI flags. Static parents from CLI are merged with onchain parents, with onchain taking precedence for duplicate keys.
  • Optimize inbound probe-measured RTT accuracy: pre-sign both TWAMP probes before network I/O so probe 1 fires immediately after reply 0 with no signing delay, measure Tx-to-Rx interval (reply 0 Tx → probe 1 Rx) instead of Rx-to-Rx to exclude processing overhead on both sides, use kernel SO_TIMESTAMPNS receive timestamps on the reflector, and add a 15ms busy-poll window on the sender to avoid scheduler wakeup latency
  • Optimize outbound probe RTT accuracy: send a staggered warmup probe on a separate socket 2ms before the measurement probe to wake the reflector's thread, then take the min RTT of both
• Onchain Programs
  • Serviceability: add Permission account with CreatePermission, UpdatePermission, DeletePermission, SuspendPermission, and ResumePermission instructions for managing per-keypair permission bitmasks onchain
  • Serviceability: add TOPOLOGY_ADMIN, RESOURCE_ADMIN, and INDEX_ADMIN permission flags for delegating management of segment-routing topologies, ResourceExtension accounts, and internal Index accounts (legacy authorization maps each to the foundation allowlist)
  • Serviceability: enforce TOPOLOGY_ADMIN/RESOURCE_ADMIN/INDEX_ADMIN via authorize() in the topology (create/delete/clear/assign-node-segments), resource (create/allocate/deallocate/close), and index (create/delete) instructions, which were previously gated by the foundation allowlist only
  • Serviceability: fix ClearTopology account layout — the processor now parses payer/system_program/permission from the tail of the account list (matching what the SDK client appends after the variable-length link list) instead of reading them at fixed front positions, so doublezero topology clear no longer reverts when links are passed
  • Serviceability: authorize() now falls back to the legacy allowlists when a payer's auto-injected Permission account exists but does not grant the requested flag, as long as RequirePermissionAccounts is off — so foundation (and other legacy-authorized) keys are not locked out of an instruction merely because they also hold an unrelated, under-privileged Permission account
• SDK
  • Add execute_authorized_transaction (and its _quiet variant) alongside execute_transaction. The authorized variants append the payer's Permission PDA (read-only) as the trailing account when it exists on-chain, so authorize() can find it. All variants share the same builder, so the protocol-max compute-budget/heap-frame requests, preflight, and error-reporting behavior are identical to execute_transaction; the only difference is the optional trailing Permission account. The Permission PDA lookup is retried on transient RPC errors and memoized per client.
  • Add TOPOLOGY_ADMIN/RESOURCE_ADMIN/INDEX_ADMIN permission-flag constants to the Go, TypeScript, and Python serviceability SDKs
• CLI
  • Add permission get, permission list, and permission set commands with table and JSON output; permission set supports incremental --add / --remove flags and creates or updates the account as needed
  • Add topology-admin, resource-admin, and index-admin to the named permissions accepted by permission set --add / --remove

v0.11.0 - 2026-03-12

Breaking

Changes

• Onchain Programs
  • Serviceability: split per-device multicast user tracking into separate subscriber and publisher counters (multicast_subscribers_count/max_multicast_subscribers and multicast_publishers_count/max_multicast_publishers); publisher and subscriber limits are now enforced independently
• Controller
  • Downgrade transient ClickHouse write errors from ERROR to WARN, escalating to ERROR only after 3 consecutive flush failures to reduce alert noise (#3220)
• Telemetry
  • Detect and replace unresponsive RIPE Atlas source probes that stop returning ping results, with a 24-hour TTL on the unresponsive probe blacklist so probes are retried after expiry
  • Compare source probe IDs (not just location codes) during measurement reconciliation so that probe replacements trigger measurement recreation
  • Fix race condition in internet-latency-collector where export and management goroutines independently loaded/saved the same state file, causing newly created RIPE Atlas measurement metadata to be overwritten and measurements to be stuck in a create-destroy-create loop (#3195)
  • Add wheresitup job backlog observability: pending_jobs Prometheus gauge, in_progress_count/pending_jobs in export summary logs, and API response duration histogram (#3203)
  • Change geoprobe-agent and geoprobe-target default TWAMP reflector port from 862 to 8925 to avoid DZD ACL blocks, use per-probe TWAMP port instead of hardcoded constant, and update --additional-child-probes/--additional-targets format to host or host:offset_port:twamp_port (two-field host:port rejected as ambiguous)
• Activator
  • Fix tunnel ID leak in user activation: eagerly-allocated tunnel_id, tunnel_net, and dz_ip were not rolled back when the activation transaction failed, causing ghost IDs to accumulate and eventually exhaust the controller's tunnel slot range
  • Suppress noisy program log output from race conditions caused by dual event processing (websocket + snapshot poll). The SDK's new execute_transaction_quiet returns a SimulationError with program logs; the activator verifies suspected races by re-fetching user state before deciding whether to print logs (#3197)
• CLI
  • Add doublezero-geolocation CLI for managing geolocation program entities: GeoProbe CRUD (create, get, list, update, delete), parent device management (add/remove), program config initialization, and geolocation-specific config get/set
  • Add --multicast-publishers-count and --multicast-subscribers-count flags to device update for foundation-gated count correction; rename --max-multicast-users to --max-multicast-subscribers and add --max-multicast-publishers
  • Add doublezero-admin device migrate-multicast-counts [--dry-run] to correct stale multicast_subscribers_count/multicast_publishers_count on existing deployments where all multicast users were previously counted as subscribers; supports dry-run preview and continues past per-device failures
  • Add doublezero-admin device migrate-unicast-counts [--dry-run] to correct stale unicast_users_count on all devices; same behaviour as the multicast counts command
• SDK
  • Add read-only Go SDK for doublezero-geolocation program with state deserialization, PDA derivation, and RPC client for querying geoprobe configuration
  • Add GetGeoProbeKeys to geolocation SDK for lightweight account key fetching using DataSlice to minimize RPC bandwidth
  • Add Rust SDK instruction builders for GeolocationUser management: CRUD (create, update, delete), target management (add, remove), payment status updates, and read queries (get by code/pubkey, list all)
• Telemetry
  • Add onchain GeoProbe discovery to the telemetry agent: periodically queries the Geolocation program for child probes parented to the local device, replacing the need for --additional-child-probes CLI flag
  • Embed LocationOffsets from parent DZDs in signed TWAMP replies so inbound probes carry geolocation context, and make signed TWAMP replies more like LocationOffsets to couple with a new double-probe system for inbound probing.
• Client
  • Fix mainnet-beta Cloudsmith package containing the testnet binary by giving the mainnet-beta Rust build a separate CARGO_TARGET_DIR to prevent GoReleaser artifact collision
  • Increase default onchain fetch timeout from 20s to 60s to improve resilience on high-latency RPC paths; add -reconciler-fetch-timeout flag to allow operators to override
  • Add prometheus metrics for onchain RPC fetches: fetch duration histogram, result counter (success/error with stale cache/error with no cache), and stale cache age gauge
  • Increase default route liveness probe interval (TxMin/RxMin) from 300ms to 1s and raise MaxTxCeil from 1s to 3s to preserve backoff headroom
  • Throttle O(n) per-service scheduler queue length metric from every event to once per 10s to fix excessive CPU usage on nodes with many liveness sessions
• Smartcontract
  • Serviceability: add foundation-only unicast_users_count, multicast_subscribers_count, and multicast_publishers_count fields to UpdateDevice instruction for direct count correction, with corresponding --unicast-users-count, --multicast-subscribers-count, and --multicast-publishers-count CLI flags
  • Serviceability: fix validate_account_code forcing lowercase on all entity types — restrict lowercase normalization to device and link codes only, preserving original case for locations, exchanges, contributors, and other entities
  • feat(smartcontract): atomic onchain allocation for CreateDevice (#3216)
  • Serviceability: RequestBanUser instruction supports atomic deallocate when OnchainAllocation feature is enabled
  • Serviceability: DeviceInterfaceUpdate instruction supports onchain allocation of node_segment_idx
  • Serviceability: fix validate_account_code forcing lowercase on all entity types — restrict lowercase normalization to device and link codes only, preserving original case for locations, exchanges, contributors, and other entities
  • feat(smartcontract): atomic onchain allocation for CreateDevice (#3216)
  • Serviceability: CreateDeviceInterface instruction supports atomic create+allocate+activate when OnchainAllocation feature is enabled
  • Serviceability: DeleteDeviceInterface instruction supports atomic deallocate+remove when OnchainAllocation feature is enabled
  • Serviceability: redesign reservations from per-IP accounts to per-(device, owner) reservation blocks with a configurable seat count, and integrate with user creation so creating a user can consume a reserved seat (#3224)
• CLI
  • Add access-pass user-balances command to show per-payer SOL balance, required amount (rent + gas reserve), and missing amount, with filters (--user-payer, --min-balance, --max-balance, --min-missing, --max-missing), sorting, and --top N
  • Add access-pass fund command to top up underfunded user payers, with --dry-run, --force (skip confirmation), --min-balance, and a pre-transfer sender balance check; required balance floor includes a gas-fee reserve (50 × 5,000 lamports) and the wallet rent-exempt minimum to prevent on-chain transfer failures
  • Fix access-pass fund and access-pass user-balances slot counting to track remaining slots per (payer, client_ip) pair, preventing a connected user on IP_B from consuming open slots for IP_A; also fix required balance formula to always add wallet_rent_min on top of needs_rent rather than taking the max
  • Add --user-payer filter to user list command
  • Serviceability: onchain activation - atomic close for DeleteDevice (#3188)
  • Fix access-pass user-balances and access-pass fund underestimating the required wallet balance: the wallet's own rent-exempt minimum was used as a floor rather than being added, causing missing: 0 to be reported even when provisioning would fail with insufficient funds (#3213)
• Onchain Programs
  • Serviceability: DeleteUser instruction supports atomic deallocate+closeaccount when OnchainAllocation feature is enabled
  • Serviceability: CreateLink instruction supports atomic create+allocate+activate when OnchainAllocation feature is enabled
  • Serviceability: DeleteLink instruction supports atomic deallocate+closeaccount when OnchainAllocation feature is enabled
  • Serviceability: CreateMulticastGroup instruction supports atomic create+allocate+activate when OnchainAllocation feature is enabled
  • Serviceability: DeleteMulticastGroup instruction supports atomic deallocate+closeaccount when OnchainAllocation feature is enabled
  • Serviceability: CreateSubscribeUser instruction supports atomic create+allocate+activate when OnchainAllocation feature is enabled
  • Serviceability: MulitcastGroupUpdate instruction supports atomic ip allocation when OnchainAllocation feature is enabled
  • Serviceability: UpdateUser instruction supports atomic allocate+deallocate when OnchainAllocation feature is enabled

v0.10.0 - 2026-03-04

Breaking

Changes

• CLI
  • doublezero resource verify command will now suggest creating resources or create them with --fix
  • Print an explicit message when the tenant is resolved implicitly from the configuration file (Using tenant '...' from configuration file.) or from the Access Pass allowlist (Using tenant '...' from Access Pass.)
  • All get commands now display output as a formatted table and support a --json flag for machine-readable output
  • Remove solana-multicast-publisher and solana-multicast-subscriber from access-pass set type options (multicast roles use the tenant field on a prepaid pass instead); make --solana-validator optional for solana-rpc type
  • get commands now expose all onchain account fields: user get adds tunnel_id, tunnel_endpoint, and validator_pubkey; device get adds reserved_seats; link get adds tunnel_id; multicastgroup get adds tenant, publisher_count, and subscriber_count; tenant get adds payment_status, billing, administrators, and token_account; exchange get adds device1_pk and device2_pk
• SDK
  • Fix multicast group deserialization in smartcontract/sdk/go to correctly read publisher and subscriber counts and align status enum with onchain definition
• Smartcontract
  • Serviceability: add Reservation account and ReserveConnection/CloseReservation instructions for pre-reserving connection seats on devices, with reserved_seats factored into capacity checks on both reservation and user creation
  • Allow sentinel authority to add/remove multicast publisher and subscriber allowlist entries
• SDK
  • Add Rust SDK for geolocation program with GeoClient, GeoProbe CRUD operations (create, get, list, update, delete, add/remove parent device), and ProgramConfig management (init, get, update)
  • Add geo_program_id to NetworkConfig for geolocation program discovery
• Telemetry
  • Add geoprobe-target-sender CLI tool for sending signed TWAMP probes to a GeoProbe and verifying signed replies (RFC16 inbound probing flow)
  • Add Signed TWAMP reflector to geoprobe-agent with configurable listen port and allowed-pubkeys allowlist
  • Fix global monitor crash when IBRL and multicast users share the same client IP but are on different devices, by preferring non-multicast users in client IP lookups to match status device selection
• Client
  • Add onchain reconciler to daemon — automatically provisions/removes tunnels by polling onchain User state, replacing CLI-driven provisioning and the doublezerod.json state file (RFC-17)
  • Add doublezero enable / doublezero disable CLI commands to toggle the reconciler at runtime
• E2E tests
  • Publish TestQA_AllDevices_UnicastConnectivity results to ClickHouse (qa_alldevices_results and qa_alldevices_metadata tables) in addition to InfluxDB; configured via CLICKHOUSE_ADDR env var, skipped gracefully when not set
• Onchain Programs
  • Serviceability: CreateUser instruction supports atomic create+allocate+activate when OnchainAllocation feature is enabled

v0.9.0 - 2026-02-27

Breaking

• CLI: --bandwidth is now a required argument for doublezero device interface create and doublezero device interface update; callers that previously omitted it (relying on the default of 0) must now explicitly pass a value with a unit (e.g. --bandwidth 10Gbps)

Changes

• Tools
  • Add signed TWAMP sender and reflector with Ed25519 authentication and per-pubkey rate-limited signature verification
• Activator
  • Adds a stateless mode for when onchain allocation is enabled. This prevents activator/onchain from becoming out-of-sync.
  • --onchain-allocation cli flag replaced with feature flag from GlobalState onchain
• SDK
  • Add retry with exponential backoff (3 retries, 500ms–5s) to all read-only RPC calls in DZClient, improving resilience to transient RPC timeouts
• CLI
  • Fix doublezero status showing "Current Device" and "Metro" as N/A for multicast subscribers when the tunnel destination is a user_tunnel_endpoint loopback interface IP rather than the device's public_ip
  • Remove redundant connect ibrl unit tests that were duplicates of hybrid-device equivalents
  • doublezero global-config feature-flags commands added
  • Fix multicast subscriber tunnel source resolution for NAT environments — resolve local interface IP instead of using public IP
  • Added multicast filters to access-pass list, enabling filtering by publisher/subscriber role and identifying access passes not authorized for a specific multicast group.
  • Device interface --bandwidth and --cir flags now accept Kbps, Mbps, or Gbps units; interface list displays those values as human-readable strings
  • Add duplicate IP check to prevent a user from assigning the same IP more than once
• Client
  • Fix BGP OnClose deleting routes from all peers instead of only the closing peer, preventing multicast teardown from nuking unicast routes
  • Skip route deletion on OnClose for NoInstall peers (multicast) since they never install kernel routes
  • Reject BGP martian addresses (CGNAT, multicast, reserved, benchmarking, etc.) as client IP during connect
• Controller
  • detect duplicate (UnderlaySrcIP, UnderlayDstIP) pairs for tunnels and only render the first to the device config and write a log error for the second
  • allow MTU to be configurable
• Onchain Programs
  • Serviceability: skip field validation for users in Deleting status to prevent accounts from getting stuck during cleanup when validation rules change
  • Serviceability: require foundation_allowlist privileges to update node_segment_idx on a device interface
  • Serviceability: add feature flags support
  • Serviceability: expand is_global to reject all BGP martian address ranges (CGNAT 100.64/10, IETF 192.0.0/24, benchmarking 198.18/15, multicast 224/4, reserved 240/4, 0/8)
  • Serviceability: allow update and deletion of interfaces even when sibling interfaces have invalid CYOA configuration
  • Geolocation: add doublezero-geolocation program scaffolding and GeoProbe account type and related instructions as per rfcs/rfc16-geolocation-verification.md
• SDK
  • SetFeatureFlagCommand added to manage on-chain feature flags for conditional behavior rollouts
• Dependencies
  • Upgrade Solana SDK workspace dependencies from 2.2.7 to 2.3.x (solana-sdk, solana-client, solana-program-test, and others)
• Internet telemetry
  • Reduce RIPE Atlas sampling interval from 6 minutes to 10 minutes to work around service limit of 100_000 samples per day
• E2E
  • e2e: add multi-tenant access control negative tests (#3081)

v0.8.11 - 2026-02-27

Breaking

• N/A

Changes

• Client
  • Fix BGP OnClose deleting routes from all peers instead of only the closing peer, preventing multicast teardown from nuking unicast routes

v0.8.10 – 2026-02-19

Breaking

• N/A

Changes

• Activator
  • removes accesspass monitor task (that expires access passes)
• Monitor
  • Add Prometheus metrics for multicast publisher block utilization (doublezero_multicast_publisher_block_total_ips, doublezero_multicast_publisher_block_allocated_ips, doublezero_multicast_publisher_block_utilization_percent) — enables Grafana alerting on IP pool exhaustion thresholds
  • Include tenant name in the Slack notification table when new users are activated
• SDK (Go)
  • Add GetMulticastPublisherBlockResourceExtension() method to serviceability client for fetching the global multicast publisher IP allocation bitmap
  • Fix LinkDesiredStatus discriminants (hard-drained=6, soft-drained=7)
• Onchain Programs
  • Upgrade programs from system_instruction to solana_system_interface
  • Refactor user creation to validate all limits (max_users, max_multicast_users, max_unicast_users) before incrementing counters — improves efficiency by avoiding wasted work on validation failures and follows fail-fast best practice
  • Serviceability: UnlinkDeviceInterface now only allows Activated or Pending interfaces; when an associated link account is provided for an Activated interface, the link must be in Deleting status
  • Links and devices can no longer be deleted from Activated status — must be drained first; deletion is rejected with InvalidStatus
  • Contributors, locations, multicast groups, and users can now be deleted from any operational status (not just Activated); only Deleting/Updating states are blocked
  • SDK: UnlinkDeviceInterfaceCommand automatically discovers and passes associated link accounts
  • Serviceability: allow contributors to update prefixes when for IBRL when no users are allocated
• CLI
  • doublezero status now shows a Tenant column (between User Type and Current Device) with the tenant code associated with the user; empty when no tenant is assigned
• Client
  • Fix tunnel overlay address scope to prevent kernel from selecting link-local /31 as source for routed traffic
  • Serviceability: close orphaned blocks when dz_prefixes shrink
  • Serviceability: require ip-net for CYOA interfaces
• E2E / QA Tests
  • Fix QA unicast test flake caused by RPC 429 rate limiting during concurrent user deletion — treat transient RPC errors as non-fatal in the deletion polling loop
  • Backward compatibility test: use --status instead of --desired-status for drain commands; fix version ranges (link drain compatible since v0.7.2, device drain since v0.8.1)
  • Add daily devnet QA test for device provisioning lifecycle (RFC12) — deletes/recreates device and links, restarts daemons with new pubkey via Ansible
  • Remove devnet/testnet environment filter from TestQA_MulticastPublisherMultipleGroups — test now runs against all environments

v0.8.9 – 2026-02-16

Breaking

• N/A

Changes

• Activator
  • Fail to start if any global config network blocks (device_tunnel_block, user_tunnel_block, multicastgroup_block, multicast_publisher_block) are unset (0.0.0.0/0)
  • Fix multicast publisher dz_ip leak in offchain deallocation — IPs from the global publisher pool were never freed on user deletion because the check required non-empty publishers, which the smartcontract already clears before allowing deletion
• Client
  • Fix heartbeat sender not restarting after disconnect due to poisoned done channel
• Onchain Programs
  • bugfix(serviceability): contributors can now update their interfaces, CYOA interfaces saved on create, physical interfaces remain after unlink (#2993)
• Device controller
  • Reject users with BGP martian DZ IPs (RFC 1918, loopback, multicast, link-local, documentation nets, shared address space, reserved) to prevent invalid addresses from being advertised via BGP or permitted in device ACLs
• Claude
  • chore: update CLAUDE.md (#2999)
• e2e
  • e2e: Add network contributor e2e flow tests (#2997)

v0.8.8 – 2026-02-13

Breaking

• None for this release

Changes

• Activator
  • Assign multicast publisher IPs from global pool in serviceability GlobalConfig instead of per-device blocks
• Client
  • Add multicast publisher heartbeat sender — sends periodic UDP packets to each multicast group to keep PIM (S,G) mroute state alive on devices
  • Fix panic in heartbeat sender when concurrent teardown requests race on close
• E2E tests
  • Add daily devnet QA test for device provisioning lifecycle (RFC12) — deletes/recreates device and links, restarts daemons with new pubkey via Ansible
• Serviceability: prevent creating or activating links on interfaces with CYOA or DIA assignments, and prevent setting CYOA/DIA on interfaces that are already linked
• CLI: add early validation in link wan-create and link dzx-create to reject interfaces with CYOA or DIA assignments

v0.8.7 – 2026-02-10

Breaking

• None for this release

Changes

• SDK
  • Added Tenant to all sdks
• E2E tests
  • Added multi-tenancy deletion test coverage
• Telemetry
  • Add doublezero-geoprobe-agent, intermediary probe server for RFC16
  • Adds support for per-tenant metro routing policy
  • Add --geoprobe-pubkey flag to doublezero-geoprobe-agent for device identity
  • LocationOffset struct now includes SenderPubkey to distinguish individual devices that share the same signing authority
• Cli
  • Automatic detection of the authorized tenant is added.
  • The delete tenant command allows cascading deletion of users and access passes.
• Telemetry
  • extend device telemetry agent to measure RTT to child geoProbes via TWAMP, generate signed LocationOffset structures, and deliver them via UDP as per rfcs/rfc16-geolocation-verification.md
  • geoprobe-target: example target listener for geolocation verification with TWAMP reflector, UDP offset receiver, signature chain verification, distance calculation logging, and DoS protections (5-reference depth limit and per-source-IP rate limiting) (#2901)
• Onchain programs
  • feat(serviceability): add TenantBillingConfig and epoch tracking to UpdatePaymentStatus (#2922)
  • feat(smartcontract): add payment_status, token_account fields and UpdatePaymentStatus instruction (#2880)
  • fix(smartcontract): correctly ser/deser ops_manager_pk (#2887)
  • Serviceability: add metro_routing and route_liveness boolean fields to Tenant for routing configuration
  • Serviceability: add Tenant account type with immutable code-based PDA derivation, VRF ID, administrator management, and reference counting for safe deletion
  • Serviceability: add TenantAddAdministrator and TenantRemoveAdministrator instructions for foundation-managed administrator lists
  • Serviceability: extend UserUpdate instruction to support tenant_pk field updates with automatic reference count management on old and new tenants (backward compatible with old format)
  • Serviceability: extend UserCloseAccount instruction to decrement tenant reference count when closing user with assigned tenant
  • Serviceability: add reference count validation in DeleteMulticastGroup to prevent deletion when active publishers or subscribers exist
  • Serviceability: fix multicast group closeaccount to use InvalidStatus error and remove redundant publisher/subscriber count check
  • Serviceability: add tenant_allowlist field to AccessPass to restrict which tenants can use specific access passes (backward compatible with existing accounts)
  • Serviceability: bypass validation for link delete (#2934)
  • Serviceability: add per-device unicast and multicast user limits with separate counters and configurable max values (RFC-14)
  • Fix link device & link updates
• SDK
  • Add metro_routing and route_liveness fields to CreateTenantCommand and UpdateTenantCommand
  • Add CreateTenant, UpdateTenant (vrf_id only), DeleteTenant, GetTenant, and ListTenant commands with support for code or pubkey lookup
  • Add AddAdministratorTenant and RemoveAdministratorTenant commands for tenant administrator management
  • UpdateUserCommand extended with tenant_pk field and automatic tenant account resolution for reference counting
  • SetAccessPassCommand extended with tenant field to specify allowed tenant for access pass
  • TypeScript SDK updated with tenantAllowlist field in AccessPass interface and deserialization
• CLI
  • Fix multicastgroup update command to properly parse human-readable bandwidth values (e.g., "1Gbps", "100Mbps") in --max-bandwidth flag
  • Add --metro-route and --route-aliveness flags to tenant create and update commands
  • Add tenant subcommands (create, update, delete, get, list, add-administrator, remove-administrator) to doublezero and doublezero-admin CLIs
  • Support simultaneous publisher and subscriber multicast via --publish and --subscribe flags
  • Add --max-unicast-users and --max-multicast-users flags to device update command
  • Add filtering options and desired_status & metrics_publisher_pk field to device and link list commands
  • Added activation check for existing users before subscribing to new groups (#2782)
  • access-pass set: add --tenant argument to specify tenant code for access pass restriction (converts to tenant PDA onchain)
  • tenant list: improve output formatting with table support and JSON serialization options (--json, --json-compact)
  • default tenant support added to config
• SDK
  • Add read-only Go SDK (revdist) for the revenue distribution Solana program, with typed deserialization of all onchain accounts and Rust-generated fixture tests for cross-language compatibility
  • Add revdist-cli tool for inspecting onchain revenue distribution state
  • Add Python and TypeScript SDKs for serviceability, telemetry, and revdist programs with typed deserialization, RPC clients, PDA derivation, enum string types, and cross-language fixture tests
  • Add shared borsh-incremental library (Go, Python, TypeScript) for cursor-based Borsh deserialization with backward-compatible trailing field defaults
  • Add npm and PyPI publish workflows for serviceability and telemetry SDKs
  • DeleteUserCommand updated to wait for activator to process multicast user unsubscribe before deleting the user
• Device controller
  • Record successful GetConfig gRPC calls to ClickHouse for device telemetry tracking
  • Multi-tenancy vrf support added
  • Skip isis and pim config for CYOA/DIA tagged interfaces
• Onchain programs
  • Enforce that CloseAccessPass only closes AccessPass accounts when connection_count == 0, preventing closure while active connections are present.
• Monitor
  • Add sol-balance watcher to track SOL balances for configured accounts and export Prometheus metrics for alerting
• Client
  • Support simultaneous publisher and subscriber multicast in the daemon
• Telemetry
  • Add consecutive-loss-based sender eviction to the telemetry collector so broken TWAMP senders are recreated quickly instead of persisting until TTL expiry (--max-consecutive-sender-losses, default 30)
• E2E tests
  • e2e: add multi-tenancy VRF isolation test (#2891)
  • Add backward compatibility test that validates older CLI versions against the current onchain program by cloning live state from testnet and mainnet-beta
  • QA multicast tests: add diagnostic dumps on failure (status, routes, latency, multicast reports, onchain user/device state), cleanup stale test groups at test start, and fix disconnect blocking on stuck daemon status

v0.8.6 – 2026-02-04

Breaking

• None for this release

Changes

• CLI
  • Remove log noise on resolve route
  • doublezero resource verify command added to verify onchain resources
  • Enhance delete multicast group command to cascade into deleting AP entry (#2754)
• Onchain programs
  • Removed device and user allowlist functionality, updating the global state, initialization flow, tests, and processors accordingly, and cleaning up unused account checks.
  • Serviceability: require DeactivateMulticastGroup to only close multicast group accounts when both publisher_count and subscriber_count are zero, preventing deletion of groups that still have active publishers or subscribers.
  • Deprecated the user suspend status, as it is no longer used.
  • Serviceability: enforce that CloseAccountUser instructions verify the target user has no multicast publishers or subscribers (both publishers and subscribers are empty) before closing, and add regression coverage for this behavior.
  • Enhance access pass functionality with new Solana-specific types
  • fix default desired status
• Telemetry
  • Fix goroutine leak in TWAMP sender — cleanUpReceived goroutines now exit on Close() instead of living until process shutdown
• Client
  • Cache network interface index/name lookups in liveness UDP service to fix high CPU usage caused by per-packet RTM_GETLINK netlink dumps
  • Add observability to BGP handleUpdate: log withdrawal/NLRI counts per batch and track processing duration via doublezero_bgp_handle_update_duration_seconds histogram
• E2E tests
  • The QA alldevices test now skips devices that are not calling the controller
  • e2e: Expand RFC11 end-to-end testing (#2801)
  • e2e(RFC11): add dz prefix rollover allocation test (#2820)

v0.8.5 – 2026-02-02

Breaking

• None for this release

Changes

• Smartcontract
  • fix(smartcontract): reserve first IP of DzPrefixBlock for device (#2753)
• Client
  • Fix race in bgp status handling on peer deletion

v0.8.4 – 2026-01-28

Breaking

• None for this release

Changes

• Telemetry
  • Force IPv4-only connections for gNMI tunnel client and fix TLS credential handling
• Client
  • Support simultaneous unicast and multicast tunnels in doublezerod
  • Support publishing and subscribing to multiple multicast groups simultaneously
• CLI
  • Support publishing and subscribing a user to multiple multicast groups via --group flag
  • Remove single tunnel constraint
• SDK
  • Go SDK can now perform batch writes to device.health and link.health as per rfc12
• Activator
  • fix(activator): add on-chain allocation support for users (#2744)
  • On-chain allocation enabled
• Smartcontract
  • feat(smartcontract): add use_onchain_deallocation flag to MulticastGroup (#2748)
• CLI
  • Remove restriction for a single tunnel per user; now a user can have a unicast and multicast tunnel concurrently (but can only be a publisher or a subscriber) (2728)

v0.8.3 – 2026-01-22

• Data
  • Add indexer that syncs serviceability and telemetry data to ClickHouse and Neo4J

Breaking

• None for this release

Changes

• CLI
  • Remove log noise on resolve route
  • Add global-config qa-allowlist commands to manage QA identity allowlist to bypass status and max_users checks in QA
  • Add "-skip-capacity-check" flag to bypass status and max_users checks in QA to test devices that are still being provisioned
  • Remove "unknown" status from doublezero status command and implement "failed" and "unreachable" statuses
• Client
  • Enable route liveness passive-mode by default
  • Add make install make target. To build and deploy from source, users can now run cd client && make build && make install to install the doublezero and doublezerod binaries and the doublezerod systemd unit.
• Onchain programs
  • Serviceability: remove validation check for interface delete (#2707)
  • Serviceability: interface-cyoa only on physical interfaces, don't require interfaces to be tagged, add same validation logic to update interface (#2700)
  • Enforce Activated status check before suspending contributor, exchange, location, and multicastgroup accounts
  • Removed device and user allowlist functionality, updating the global state, initialization flow, tests, and processors accordingly, and cleaning up unused account checks.
  • Serviceability: require DeactivateMulticastGroup to only close multicast group accounts when both publisher_count and subscriber_count are zero, preventing deletion of groups that still have active publishers or subscribers.
  • Deprecated the user suspend status, as it is no longer used.
  • Serviceability: enforce that CloseAccountUser instructions verify the target user has no multicast publishers or subscribers (both publishers and subscribers are empty) before closing, and add regression coverage for this behavior.
  • Removed device and user allowlist functionality, updating the global state, initialization flow, tests, and processors accordingly, and cleaning up unused account checks.
  • SetGlobalConfig, ActivateDevice, UpdateDevice and CloseAccountDevice instructions updated to manage resource accounts.
  • Add option for Contributor B to reject a link created by Contributor A just as Contributor A can cancel its own created link
• Telemetry
  • Add gNMI tunnel client for state collection
• Activator
  • fix(activator): ip_to_index fn honors ip range #2658
• E2E tests
  • Add influxdb, prometheus, and device-health-oracle containers
  • Add interface lifecycle tests (#2700)
  • Only fail QA alldevices test run if device status is "Activated" and max users > 0
• SDK
  • Commands for setting global config, activating devices, updating devices, and closing device accounts now manage resource accounts.
  • Serviceability: return error when GetProgramAccounts returns empty result instead of silently returning empty data
• Smartcontract
  • feat(smartcontract): RFC 11 activation for User entity
  • feat(smartcontract): RFC 11 add on-chain resource allocation for Link
• Device Health Oracle
  • Add new device-health-oracle component. See rfcs/rfc12-network-provisioning.md for details.
  • Calculate burn-in timestamp based from slot numbers (current minus 200_000 slots for provisioning, current minus 5_000 slots for maintenance)
• CI
  • Add separate apt repo for doublezero-testnet

v0.8.2 – 2025-01-13

Breaking

• None for this release

Changes

• Client
  • Always delegate RouteAdd regardless of noUninstall flag
• Telemetry
  • Include solana vote pubkey in global monitor metrics
  • Run telemetry agent on pending and drained links

v0.8.1 – 2025-01-12

Breaking

Changes

• Onchain programs
  • Serviceability: enforce that ActivateLink and CloseAccountLink instructions verify the provided side A/Z device accounts match the link's stored side_a_pk and side_z_pk before proceeding.
• CLI
  • Update contributor, device, exchange, link, location, and multicast group commands to ignore case when matching codes
  • ActivateMulticastGroup now supports on-chain IP allocation from ResourceExtension bitmap (RFC 11).
  • IP address lookup responses that do not contain a valid IPv4 address (such as upstream timeout messages) are now treated as retryable errors instead of being parsed as IPs.
  • doublezero resource commands added for managing ResourceExtension accounts.
  • Added health_oracle to the smart contract global configuration to manage and authorize health-related operations.
  • Added --ip-net support to create to match the existing behavior in update.
  • Use DZ IP for user lookup during status command instead of client IP
• Onchain programs
  • Fix CreateMulticastGroup to use incremented globalstate.account_index for PDA derivation instead of client-provided index, to ensure the contract is the authoritative source for account indices
  • Add on-chain validation to reject CloseAccountDevice when device has active references (reference_count > 0)
  • Allow contributor owner to update ops manager key
  • Add new arguments on create interface cli command
  • Serviceability: enforce that resume instructions for locations, exchanges, contributors, devices, links, and users only succeed when the account status is Suspended, returning InvalidStatus otherwise, and add tests to cover the new behavior.
  • RequestBanUser: only allow requests when user.status is Activated or Suspended; otherwise return InvalidStatus
  • Serviceability: require device interfaces to be in Pending status before they can be rejected, and add tests to cover the new status check
  • Add ResourceExtension to track IP/ID allocations. Foundation instructions added to create/allocate/deallocate.
  • ResourceExtension optimization using first_free_index for searching bitmaps
  • Added the INSTRUCTION_GUIDELINES document defining the standard for instruction creation.
  • Enforce best practices for instruction implementation across onchain programs
  • Add missing system program account owner checks in multiple instructions
  • Refactor codebase for improved maintainability and future development
  • Introduced health management for Devices and Links, adding explicit health states, authorized health updates, and related state, processor, and test enhancements.
  • Require that BanUser can only be executed when the target user's status is PendingBan, enforcing the expected user ban workflow (request-ban -> ban).
  • Introduce desired status to Link and Devices
  • Introduced health management for Devices and Links, adding explicit health states, authorized health updates, and related state, processor, and test enhancements.
  • Restrict DeleteDeviceInterface to interfaces in Activated or Unlinked status; attempting to delete interfaces in other statuses now fails with InvalidStatus.
  • Updated validation to allow public IP prefixes for CYOA/DIA, removing the restriction imposed by type-based checks.
  • Transit devices can now be provisioned without a public IP, aligning the requirements with their actual networking model and avoiding unnecessary configuration constraints.
  • Enforce that ActivateDeviceInterface only activates interfaces in Pending or Unlinked status, returning InvalidStatus for all other interface states
  • Introduce desired status to Link and Devices
• Internet Latency Telemetry
  • Fixed a bug that prevented unresponsive ripeatlas probes from being replaced
  • Fixed a bug that caused ripeatlas samples to be dropped when they were delayed to the next collection cycle
• Link & device Latency Telemetry
  • Telemetry data can now be received while entities are in provisioning and draining states.
• Device controller
  • Add histogram metric for GetConfig request duration
  • Add gRPC middleware for prometheus metrics
  • Add device status label to controller_grpc_getconfig_requests_total metric
  • Add logic to shutdown user BGP, IBGP sessions, MSDP neighbors, and ISIS when device.status is drained
• Device agents
  • Increase default controller request timeout in config agent
  • Initial state collect in telemetry agent
• Client
  • Route liveness treats peers that advertise passive mode as selectively passive; does not manage their routes directly.
  • Route liveness runs in passive mode for IBRL with allocated IP, if global passive mode is enabled.
  • Advertise peer client version with route liveness control packets.
  • Add doublezero_bgp_routes_installed gauge metric for number of installed BGP routes
  • Add route liveness gauges for in-memory maps
  • Route liveness sets set of routes configured as excluded to AdminDown.
  • Add histogram metric for BGP session establishment duration
  • For IBRL with allocated IP mode, resolve tunnel source IP from routing table via resolve-route API endpoint instead of using client IP to support clients behind NAT
  • Configure MTU down to 1476 on client tunnel in case path MTU discovery is not working
  • Increase route liveness max backoff duration
• Global monitor
  • Initial implementation
• Release
  • Publish a Docker image for core components.
• Telemetry
  • Refactor flow enricher
  • Add metrics to flow enricher
  • Add serviceability data fetching to flow enricher
  • Add flow-ingest service
  • Add annotation of flow records with serviceability data
  • Add pcap input and json ouput to flow enricher
  • Initial state-ingest service with client SDK
  • Collect BGP socket state from devices
• CI
  • Cancel existing e2e test runs on the push of new commits
• RFCs
  • RFC - Network Provisioning
  • RFC-11: Onchain Activation (#2302)
• Monitor
  • Add link status to device-telemetry metrics to enable Grafana alerts to filter out links that are not in activated status
  • Add validation for 2Z oracle swapRate to ensure it is an unsigned integer, with warning logs and metrics for malformed values
• E2E tests
  • Add GetLatency call to qaagent
  • The QA alldevices test now considers device location and connects hosts to nearby devices
  • QA agent and tests now support doublezero connect ibrl's --allocate-addr flag
  • The QA alldevices test now publishes success/failure metrics to InfluxDB in support of rfc12
• Onchain programs
  • Fix CreateMulticastGroup to use incremented globalstate.account_index for PDA derivation instead of client-provided index, to ensure the contract is the authoritative source for account indices
  • ReactivateMulticastGroup now enforces that the multicast group status must be Suspended before reactivation, returning InvalidStatus otherwise; negative-path regression tests were added.

v0.8.0 – 2025-12-02

Breaking

• None for this release

Changes

• RFCs
  • RFC-10: Version Compatibility Windows
• CLI
  • IP address lookups via ifconfig.me are retried up to 3 times to minimize transient network errors.
  • Added global --no-version-warning flag to the doublezero client and now emit version warnings to STDERR instead of STDOUT to improve scriptability and logging.
  • Add the ability to update a Device’s location, managing the reference counters accordingly.
  • Added support in the link update command to set a link’s status to soft_drained or hard_drained.
  • Added support for specifying device_type at creation, updating it via device update, and displaying it in list/detail outputs.
  • Add support for updating contributor.ops_manager_key.
  • Add migrate command to upgrade legacy user accounts from index-based PDAs to the new IP + connection-type scheme.
  • Enhance access-pass list with client-IP and user-payer filters
  • Support added to load keypair from stdin
• Client
  • Add route liveness fault-injection simulation tests.
  • Updated the interface list command to display all interfaces when no device is specified.
• Funder
  • Fund multicast group owners
• Onchain programs
  • Serviceability Program: Updated the device update command to allow modifying a device’s location.
  • Added new soft-drained and hard-drained link status values to serviceability to support traffic offloading as defined in RFC9.
  • Fix ProgramConfig resize during global state initialization.
  • Standardized the device_type enum to Edge, Transit, and Hybrid, added validation rules, and defaulted existing devices to Hybrid for backward compatibility.
  • Add contributor.ops_manager_key for authorizing incident management operations.
  • User Account: Replace global-index PDA generation with deterministic IP + connection-type seeds, eliminating user-creation race conditions.
  • Enable on-chain storage of InterfaceV2, allowing devices to register updated interface metadata
  • Serviceability: validate that a device's public IP doesn't clash with its dz_prefixes
• QA
  • Traceroute when packet loss is detected
• Tools
  • Add solana-tpu-quic-ping tool for testing Solana TPU-QUIC connections with stats emitted periodically
• Device controller
  • Handle new link.status values (soft-drained and hard-drained) as per RFC9
• Monitor
  • Export links data to InfluxDB
• Activator
  • Uses asynchronous coroutines instead of blocking operations and threads.

v0.7.1 – 2025-11-18

Breaking

• None for this release

Changes

• RFCs
  • RFC9 Link Draining
• Client
  • Switch to 64 byte latency probes instead of 32 bytes
  • Route liveness admin-down signalling and ignore stale remote-down messages
  • Added an on-chain minimum supported CLI version to allow multiple CLI versions to operate simultaneously.
• Smart contract
  • Delay V2 Interface Activation Until All Clients Support V2 Reading
• Device controller
  • Now accepts the config agent's version in the grpc GetConfig call and includes it as a label in the controller_grpc_getconfig_requests_total metric
• Device Agent
  • Now sends its version to the controller in the grpc GetConfig call

v0.7.0 – 2025-11-14

Breaking

• Smart contract
  • Introduces CYOA and DIA as new possible interface types

Changes

• Onchain programs
  • Check if accesspass.owner is equal to system program (malbeclabs/doublezero#2088)
• CLI
  • Added support for specifying the interface type during interface creation and modification, introducing CYOA and DIA as new possible interface types.
  • Improve error message when connecting to a device that is at capacity or has max_users=0. Users now receive "Device is not accepting more users (at capacity or max_users=0)" instead of the confusing "Device not found" error when explicitly specifying an ineligible device.
  • Add link latency command to display latency statistics from the telemetry program. Supports filtering by percentile (p50, p90, p95, p99, mean, min, max, stddev, all), querying by link code or all links, and filtering by epoch. Resolves: #1942
  • Added --contributor | -c filter to device list, interface list, and link list commands. (#1274)
  • Validate AccessPass before client connection (#1356)
• Client
  • Add initial route liveness probing, initially disabled for rollout
  • Add route liveness prometheus metrics

v0.6.11 – 2025-11-13

Breaking

• None for this release

Changes

• Note that the changes from this release have been bundled into 0.7.0

v0.6.10 – 2025-11-05

Breaking

• None for this release

Changes

• CI
  • Add automated compatibility tests in CI to validate all actual testnet and mainnet state against the current codebase, ensuring backward compatibility across protocol versions.
  • Add --delay-override-ms option to doublezero link update
  • Add ability to configure excluded routes
• Device controller
  • Remove the deprecated -enable-interfaces-and-peers flag
  • Use link.delay_override to set isis metric when in valid range. This provides a simple workflow for contributors to temporarily change a link's delay value without overwriting the existing value.
• Onchain programs
  • serviceability: add delay_override_ns field to link

v0.6.9 – 2025-10-24

Breaking

• None for this release

Changes

• Onchain programs
  • serviceability: add auto-assignment and validation for exchange.bgp_community
  • serviceability: prevent device interface name duplication
  • Update serviceability and telemetry program instruction args to use the BorshDeserializeIncremental derive macro incremental, backward-compatible, deserialization of structs.
  • Add explicit signer checks for payer accounts across various processors to improve security and ensure correct transaction authorization.
• CLI
  • Removed --bgp-community option from doublezero exchange create since these values are now assigned automatically
  • Add --next-bgp-community option to doublezero global-config set so authorized users can control which bgp_community will be assigned next
• Tools
  • TWAMP: Verify that the sequence number and timestamp of the received packet matches those of the sent packet
  • Uping: Add minimal ICMP echo library for user-space liveness probing over doublezero interfaces, even when certain routes are not in the the kernel routing table.
• Device controller
  • Deprecate the -enable-interfaces-and-peers flag. The controller now always renders interfaces and peers
  • Intra-exchange routing policy, which uses the onchain exchange.bgp_community value to route traffic between users in the local exchange over the internet
• Monitor
  • Add metrics that detect when duplicate or out-of-range exchange.bgp_community values exist in serviceability

v0.6.8 – 2025-10-17

Breaking

• Multicast group change: Regeneration of all multicast group allowlists required, as allowlists are now stored within each Access Pass instead of at the multicast group level.

Changes

• Onchain programs
  • Serviceability: enforce that ActivateLink and CloseAccountLink instructions verify the provided side A/Z device accounts match the link's stored side_a_pk and side_z_pk before proceeding.
• CLI
  • Added a wait in the disconnect command to ensure the account is fully closed before returning, preventing failures during rapid disconnect/reconnect sequences.
  • Display multicast group memberships (publisher/subscriber) in AccessPass listings to improve visibility.
  • Allow AccessPass creation without 'client_ip'
  • Add 'allow_multiple_ip' argument to support AccessPass connections from multiple IPs
  • Include validator pubkey in export output
  • Rename exchange.loc_id to exchange.bgp_community
  • status command now shows connected and lowest latency DZD
• Activator
  • Reduce logging noise when processing snapshot events
  • Wrap main select handler in loop to avoid shutdown on branch error
• Onchain programs
  • Remove user-level allowlist management from CLI and admin interfaces; manage multicast group allowlists through AccessPass.
  • Add Validate trait for core types (AccessPass, Contributor, Interface, etc.) and enforce runtime checks before account operations.
  • Fix: resize AccessPass account before serialization to prevent errors; standardized use of resize_account_if_needed across processors.
  • Enable AccessPass with 'client_ip=0.0.0.0' to dynamically learn the user’s IP on first connection
  • Enable AccessPass to support connections from multiple IPs (allowlist compatibility)
  • Rename exchange.loc_id to exchange.bgp_community, and change it from u32 to u16
• Internet telemetry
  • Add circuit label to metrics; create a new metric for missing circuit samples
  • Create a new metric that tracks how long it takes collector tasks to run
  • Submit partitions of samples in parallel
  • Include circuit label on submitter error metric
• Monitor
  • Reduce logging noise in 2z oracle watcher
  • Include response body on 2z oracle errors
  • Collect contributors and exchanges into InfluxDB
• Device controller
  • When a device is missing required loopback interfaces, provide detailed errors to agent instead of " not found". Also, log these conditions as warnings instead of errors, and don't emit "unknown pubkey requested" error metrics for these conditions
  • Add device info as labels to controller_grpc_getconfig_requests_total metric
• Device agents
  • Submit device-link telemetry partitions in parallel
• CLI
  • Allow AccessPass creation without 'client_ip'
  • Add 'allow_multiple_ip' argument to support AccessPass connections from multiple IPs
  • Rename exchange.loc_id to exchange.bgp_community
• Onchain programs
  • Enable AccessPass with 'client_ip=0.0.0.0' to dynamically learn the user’s IP on first connection
  • Enable AccessPass to support connections from multiple IPs (allowlist compatibility)
  • Rename exchange.loc_id to exchange.bgp_community, and change it from u32 to u16
• Telemetry data API
  • Filter by contributor and link type
• SDK/Go
  • String serialization for exchanges status
  • Exclude empty tags from influx serialization

v0.6.6 – 2025-09-26

Breaking

• None for this release

Changes

• Monitor
  • Update 2Z oracle to emit response code metrics on errors too
• Activator
  • A mitigation was added to handle situations where blockchain updates are missed by the Activator due to timeouts on the websocket. This mitigation processes pending accounts on a 1-minute timer interval.
• CLI
  • Connect command updated to provide better user experience with regard to activator websocket timeouts (see above).

v0.6.5 – 2025-09-25

Breaking

• None for this release

Changes

• CLI
  • Connect now waits for doublezerod to get all latencies
  • Latency command sorts unreachable to bottom
• Device controller
  • Update device template to set default BGP timers and admin distance
  • Update device template so all "default interface TunnelXXX" commands for user tunnels come before any other user tunnel config
• Activator
  • Fix access pass check status accounts list
• Onchain programs
  • Implemented strict validation to ensure that only AccessPass accounts owned by the program and of the correct type can be closed.
  • Fix Access Pass set Instruction.
  • Switched to using account_close helper for closing accounts instead of resizing and serializing.
  • Make interface name comparison case insensitive
• Onchain monitor
  • Check for unlinked interfaces in a link
  • Emit user events
  • Add watcher for 2Z/SOL swap oracle

v0.6.4 – 2025-09-10

Breaking

• None for this release

Changes

• Onchain programs
  • Fix bug preventing re-opening of AccessPass after closure
  • sc/svc: guard against empty account data
• Device controller
  • Support server dual listening on TLS and non-TLS ports
• Device and Internet Latency Telemetry
  • Create one ripeatlas measurement per exchange instead of per exchange pair to avoid concurrent measurement limit

v0.6.3 – 2025-09-08

Breaking

• None for this release

Changes

• Onchain programs
  • Expand DoubleZeroError with granular variants (invalid IPs, ASN, MTU, VLAN, etc.) and derive PartialEq for easier testing.
  • Rename Config account type to GlobalConfig for clarity and consistency.
  • Fix bug in user update that caused DZ IP to be 0.0.0.0
  • Add more descriptive error logging
  • Telemetry program: embed serviceability program ID via build feature instead of env variable
• Activator
  • Support for interface IP reclamation
  • Devices are now initialized with max_users = 0 by default.
  • Devices with max_users = 0 cannot accept user connections until updated.
• Onchain monitor
  • Emit metric for telemetry account not found in device and internet telemetry watchers
  • Emit metric with serviceability program onchain version
  • Delete telemetry counter metrics if circuit was deleted
• Telemetry
  • Fix dashboard API to handle partitioned query with no samples
  • Add summary view with committed RTT and jitter, compared to measured values
• Device agents
  • Remove log of keypair path on telemetry agent start up
  • Drop device telemetry samples if submission attempts exhausted and buffer is at capacity
• Device controller
  • Each environment can now have a different device BGP Autonomous System Number (ASN) per environment. (This is the remote ASN from the client's perspective.)
  • Add flag for enabling pprof for runtime profiling
• E2E tests
  • Updated unit tests and e2e tests to validate the new initialization and activation flow.
• Contributor Operations
  • Contributors must explicitly run device update to set a valid max_users and activate a Device.

v0.6.2 – 2025-09-02

Breaking

• None for this release

Changes

• Onchain programs
  • Fix: Serviceability now correctly enforces device.max_users
  • Fix: Restored the validator_pubkey field from AccessPass. This field had been removed in the previous version but is required by Sentinel.
  • Fix: Skip client version check in status command to prevent version errors during automated state checks.
  • New instructions were added to support device interface create/update/delete that prevents a race condition that could cause some updates to be lost when changes were made in quick succession.
  • Add deserialization vector with capacity + 1 to avoid memory allocation failed, out of memory error
• CLI
  • Added filtering options to access-pass list and user list CLI commands.
  • New filters include access pass type (prepaid or solana-validator) and Solana identity public key.
  • Updated command arguments and logic, with tests adjusted to cover new options.
  • Contributors: Interface creation no longer takes an "interface type (physical/loopback)" argument. The type is now inferred from the interface name.
• Device controller
  • Use serviceability onchain delay for link metrics

v0.6.0 – 2025-08-28

Breaking

• Onchain programs
  • Implement access pass management commands and global state authority updates
  • Update access pass PDA function to include payer parameter

Changes

• Onchain programs
  • Introducing new link instruction processor acceptance criteria
  • Add support for custom deserializers and add for pubkey fields
  • Move serialization and network_v4 to program-common
  • Refactor account type assertions in processors and state modules in serviceability program
  • Add validator identity to SolanaValidator type AccessPass.
• User client
  • Add access pass management commands to CLI
  • Restructuring device and global config CLI commands for better authority and interface management
  • Enhance the handling and display of access pass epoch information in the CLI
  • Configure CLI network settings with shorthand network code. Usage: doublezero config set --env <testnet|mainnet-beta>
  • Configure doublezerod network settings with shorthand network code. Usage doublezerod --env <testnet|mainnet-beta>
  • Add associated AccessPass to user commands.
• Activator
  • Introduce new user monitoring thread in activator for access pass functionality
  • Remove validator verification via gossip. This functionality is migrated to AccessPass.
• Device controller
  • Implement user tunnel ACLs in device agent configuration
  • Add "mpls icmp ttl-exceeded tunneling" config statement so intermediate hops in the doublezero network respond to traceroutes.
  • Set protocol timers for ibgp and isis to improve to speed up network re-convergence
  • Add TLS support to gRPC server
• Onchain monitor
  • Initial implementation and component release
  • Monitor onchain device telemetry metrics
  • Monitor onchain internet latency metrics
• E2E tests
  • Simplify fixtures with loop rollups
  • Add user ban workflow test
  • Deflake user reconnect race and device interface assigned IP race
  • Add single device stress test
  • Adjust user validation commands to use the new AccessPass column.
• CLI
  • Refactor: Updated SetAccessPassCliCommand (doublezero access-pass set) to use --epochs instead of --last_access_epoch, with sensible default values.
  • AccessPass now requires passing the validator identity for the SolanaValidator type.
• Device Agents
  • Periodically recreate telemetry agent sender instances in case of interface reconfiguration.
• Telemetry
  • Optimize onchain data dashboard API responses with field filtering
  • Optimize onchain data data CLI execution with parallel queries
  • Dashboard API support for circuit set partitioning using query parameters

v0.5.3 – 2025-08-19

• CLI & UX Improvements
  • Improve sorting of device, exchange, link, location, and user displays
  • New installation package for the admin CLI for contributors based on controller/doublezero-admin
  • Do not allow users to connect to a device with zero available tunnel slots remaining
  • Improve handling of interface names for doublezero device interface commands
• Serviceability Model Improvements
  • funder: configure recipients as flag
  • sdk/rs: add record program handling
  • config: use ledger RPC LB endpoint
  • Validate account codes and replace whitespace
  • config: add ability to override DZ ledger RPC url; update URLs
  • Remove old CloseAccount instruction from both the smart contract and SDK client code
• Network Controller Improvements
  • Increase user tunnel slots per device from 64 to 128
  • Add flag controlling whether interfaces and peers are rendered to assist with testnet migration
• Device and Internet Latency Telemetry
  • Internet latency samples in data CLI and dashboard API
  • internet-latency-collector, telemetry data api/cli: collect internet latency between exchanges, not locations
  • internet-latency-collector: add ripeatlas credit metric
• End-to-End Tooling
  • New doublezero QA agent improves quality by thoroughly testing the software stack end-to-end in each doublezero environment (devnet, testnet, mainnet-beta) after each release.

v0.5.0 – 2025-08-11

• CLI & UX Improvements
  • doublezero connect now waits for the user account to be visible onchain.
  • doublezero device interface commands. Interface names get normalized.
  • General improved consistency
  • Easy switching between devnet and testnet using the --env flag
• Device Latency Telemetry
  • Data CLI and API use epoch from ledger
  • Backpressure support to avoid continual buffer growth in error conditions
  • Link pubkey used for circuit uniqueness
• Internet Latency Telemetry
  • Adds the environement (devnet/testnet/mainnet-beta) to the ripeatlas measurement description.
  • Now funded by the funder
• Serviceability Model Improvements
  • Device extended to add DZD metadata (including Interfaces)
  • DZX Link types added (clearly distinguished from WAN links)
  • Removed foundation allowlist check, streamlining link workflow
  • Validate that link.account_type has type AccountType::Link
• Network Controller Improvements
  • doublezero-controller now manages more of the DZD configuration, including:
    • DNS servers
    • NTP servers
    • DZ WAN interfaces
    • Necessary loopback interfaces
    • BGP neighbor configuration
    • MSDP configuration
  • doublezero-activator now assigns IP addresses for use by the controller to give to DZD wan interfaces as well as loopbacks.

v0.4.0 – 2025-08-04

This release adds contributor ownership, reference counting, and improved CLI outputs for devices and links. It introduces internet latency telemetry, with support for collection, Prometheus metrics, and writing samples to the ledger. Device telemetry now uses ledger epochs for network-wide consistency.

• Serviceability Model Improvements
  • Contributor creation includes an owner field; device/link registration enforces contributor consistency
  • Contributor field shown in CLI list and get commands for devices and links
  • reference_count added to contributors, devices, locations, and exchanges
  • New fields added to Device and Link, including an interfaces array for Device
  • Go SDK updated to support new DZD metadata account layouts
• CLI & UX Improvements
  • Provisioning (connect, decommission) UX improved: clearer feedback, better spinners, and more accurate status messages
  • doublezero latency output includes device code alongside pubkey
  • doublezero device and doublezero link commands updated to show new metadata fields
  • Added doublezero device interface subcommands for managing interfaces
  • keygen command now supports --outfile (-o) flag to generate keys directly to a file
• Device Latency Telemetry
  • Agent now uses ledger epoch instead of wallclock-based epoching
  • Account layout updated to move epoch after discriminator for efficient filtering
• Internet Latency Telemetry
  • Internet latency collectors write samples to the ledger using epoch-based partitioning
  • Telemetry program supports ingesting external control-plane latency samples
  • Prometheus metrics expose collector operation, failure rates, and credit balances
  • Go SDK support for initializing and submitting latency samples
• End-to-End Tooling
  • Multicast monitor utility added for provisioning validation
  • Multi-client e2e tests cover IBRL with and without IP allocation

v0.3.0 - 2025-07-28

This release introduces network contributor registration, device interface management, and the initial telemetry system for link latency. Prometheus metrics were added to the activator and client for observability. Provisioning flows now enforce contributor presence and IP uniqueness per user.

• Contributor Support
  • Added CLI support for contributor management via doublezero contributor
  • Used to register network contributors in the DoubleZero system
• Device Interface Management
  • Added device interface CRUD commands for managing interfaces on a device
  • Interface metadata will be used by the controller to generate device configuration
• Link Telemetry System
  • Introduced TWAMP-based telemetry agent and onchain program for measuring RTT and packet loss between devices
  • Lays the foundation for performance-based rewards for bandwidth contributors
• Prometheus Metrics
  • Activator and client now export Prometheus metrics (build info, BGP session status)
• Provisioning & Decommissioning
  • Enforced one tunnel per user per IP address
  • Contributor field now required when creating devices and links
• Activator
  • Improved metrics and error handling
  • Added graceful shutdown and signal handling
• Client
  • Added -json output flag for status and latency commands