Status: Approved
This RFC documents the network provisioning process used by DoubleZero contributors when onboarding and performing maintenance on devices, links, and CYOA interfaces, and proposes initial steps for automating the process. Also see rfc9-link-draining.md. Offboarding is currently out of scope.
The provisioning process for DoubleZero Devices (DZDs) and links currently requires manual effort, is prone to human error, and includes overhead and delays from coordination between contributors and the DoubleZero Foundation (DZF).
- Onboarding - The process of adding a new device or link to the network, starting with contributor's decision to add a device or link
- Offboarding - The process of removing an existing device or link from the network, starting with the contributor's decision to remove the link or device
- Edge device - A DoubleZero Device (DZD) that terminates user tunnels (max-users > 0). It has:
- 1 or more CYOA interfaces
- Zero WAN links
- 1 or more DZX links
- 1x out-of-band management with Internet access (can reuse CYOA)
- Transit device - A DZD that routes traffic between other DZDs but does not terminate user tunnels. It has:
- Zero CYOA interfaces
- 2 or more WAN links (single contributor metro)
- 1x WAN link and 1 or more DZX link (multi-contributor metro)
- 1x out-of-band management with Internet access
- Hybrid device - A DZD that terminates user tunnels and also routes traffic between other DZDs. It has:
- 1 or more CYOA interfaces
- 1 or more WAN links (single contributor metro)
- 1 or more WAN links and 1 or more DZX links (multi-contributor metro)
- 1x out-of-band management with Internet access (can reuse CYOA)
- RFS - Ready For Service criteria are a set of conditions that must be met for a device or link to be considered healthy and therefore ready for service.
- Burn-in period - The amount of time a link or device must pass all its health checks before it can be activated. For links and devices activated for the first time this is 200,000 slots on the DoubleZero ledger, which at 370ms per slot is about 20 hours. For links and devices being re-activated after maintenance this is 5000 slots, about 30 minutes.
- We can keep the current manual process in place. This will allow us to prioritize other development work, but it will keep operational overhead higher for contributors and the DoubleZero engineering team, and make the DoubleZero mainnet-beta network more difficult to scale.
- We can look at procuring a commercial network provisioning system such as Cisco Network Services Orchestrator, Juniper Apstra, SolarWinds Network Configuration Manager, or others. However, based on team members' experience with these systems, in the best case they systems tend to be expensive and require extensive investment in customization. They also use traditional database backends, and would require additional customization to integrate with the DZ ledger (if doing so is even feasible). We prefer a minimal, purpose-built system that deeply integrates with the rest of the DoubleZero stack.
This rfc covers device and link onboarding, and device draining for maintenance purposes like firmware upgrades and parts replacement.
The following are out of scope:
- Offboarding of devices is out of scope because we don't currently have demand for removing devices from the network.
- Automatically moving devices or links to a drained status, for example due to failing health checks, is out of scope due to the operational risk it would introduce. For now, this will still require action by the contributor owning the device or link.
- Link draining is already covered by rfcs/rfc9-link-draining.md.
- Automatic re-connection of users to nearby devices is out of scope. During device draining, users will remain connected to their current device, and will need to take action if they want to re-connect to another nearby device
- Providing visibility into the history of the network topology is out of scope.
- Providing users with visibility into a schedule of planned maintenance is out of scope.
- Automatically determining whether the network has enough capacity to safely drain a given DZD is out of scope.
- Draining of CYOA interfaces is out of scope, although draining a device will shut down all user BGP sessions, and so it will have the effect of removing all user traffic from all CYOA interfaces.
- We would like to publish detailed health info to contributors onchain, but to keep the scope tight, we will start by using Grafana to publish health info to contributors.
- We would like to add a health check that performs a port scan on devices, but that is currently out of scope.
- Some device/link RFS criteria are out of scope because we don't currently have a way to retrieve relevant data from devices, and we don't want to delay the first version of this framework in order to incorporate them. For example we currently have no way to verify that NTP is configured correctly.
- Providing a means for DZDs to talk to the controller when internet access is down is out of scope.
This section describes the existing manual onboarding process for new devices. The workflow is divided into phases to make them easier to view. This rfc currently focuses on parts of Phase 2 and Phase 3 of the device onboarding process.
Phase 1 - Initial device setup
%%{init: {"themeVariables": {"fontSize": "16px"}, "flowchart": {"curve": "basis", "htmlLabels": true}, "scale": 1.5}}%%
flowchart TD
Start --> A
A([Rack & Power Device]) --> B[Mgmt and Internet Connectivity]
B --> B1[EOS version 4.32.3M]
B --> B2[Contributor applies config:<br>- DNS servers<br>- service routing protocols model multi-agent<br>- hostname<br>- TCAM profile<br>- Influx<br>- DIA interface setup]
B --> B10[Share device model, device name, serial # to DZF]
B --> C[Generate Service Key]
C --> C1[Share Service Key to DZF]
B2 --> End
B10 --> End
C1 --> End
Phase 2 - Model device onchain
%%{init: {"themeVariables": {"fontSize": "16px"}, "flowchart": {"curve": "basis", "htmlLabels": true}, "scale": 1.5}}%%
flowchart TD
Start --> D
D{New Contributor?}
D -- Yes --> E[Define new contributor<br> code and provide to DZF]
D -- No --> F[Determine existing<br> contributor code]
E --> G[Register device onchain<br> and obtain pubkeys]
F --> G
G --> H[Install Config Agent]
H --> H.1[Verify config agent logs]
H.1 --> Z[Done]
G --> I[Install telemetry agent]
I --> I.1[Verify telemetry agent logs]
I.1 --> Z
G --> J[Register device interfaces<br> onchain, including CYOA for<br> edge/hybrid devices]
J --> J.1[Register DZX links onchain]
J --> J.2[Register WAN links onchain<br> if device is transit or hybrid]
J --> J.3[Loopback 255 - vpnv4]
J --> J.4[Loopback 256 - ipv4]
J.1 --> L
J.2 --> L
J.3 --> L
J.4 --> L
J --> L[Register WAN links,<br> contributor sets delay<br> to 1000ms for provisioning]
L --> M[Register DZX links,<br> contributor sets delay<br> to 1000ms for provisioning]
M --> N[Z side of DZX link accepts<br>connection]
N --> Z
Phase 3 - Health check and enable user connections
%%{init: {"themeVariables": {"fontSize": "16px"}, "flowchart": {"curve": "basis", "htmlLabels": true}, "scale": 1.5}}%%
flowchart TD
O[ML/contributor<br>verifiy health of new assets]
O --> P{Healthy? Links,<br> routing policy, security,<br>interfaces, config agent,<br>telemetry agent etc.}
P -- No --> O
P -- Yes --> Q{Device is edge or hybrid?}
Q -- No --> U
Q -- Yes --> R[Malbec Labs/contributor<br> test CYOA access by<br> increasing max-users to 1<br> and testing with server]
R --> S{Server connects and can<br> pass traffic?}
S -- No --> S.1[ML/DZF/contributor<br> troubleshooting]
S.1 --> S
S -- Yes --> T[Contributor increases<br> max-users to 96]
T --> U[Contributor modifies wan<br> and dzx latencies to correct<br>latencies]
U --> Z[Done]
Here are the same phases as lists:
- Rack & power device
- Mgmt and Internet connectivity
- Upgrade EOS to version 4.32.3M
- Contributor applied configuration
- DNS servers
service routing protocols model multi-agent- hostname
- DZ-provided TCAM profile
- Influx
- DIA interface setup
- Generate service key
- Share service key to DZF
- Share device model, name, and serial number to DZF
- If this is a new contributor, define new contributor code and provide to DZF
- Register device onchain with max-users=0 (to prevent users from connecting)
- Provide pubkey to DZF
- Install config agent
- Verify config agent logs
- Install telemetry agent
- Verify telemetry agent logs
- Register device interfaces onchain
- Loopback256 - ipv4
- Loopback255 - vpnv4
- Interfaces to be used for wan/dzx links
- Interfaces to be used for CYOA
- Register wan/dzx links onchain with link.status set to SoftDrained
- Z side contributor accepts any pending links
- Contributor verifies health of new assets
- Are interfaces links, routing protocols, routing policy, security, etc, configured correctly?
- Are the device's config and telemetry configured and operating correctly?
- For edge/hybrid devices
- Run QA test via device's CYOA interface(s)
- On failure, troubleshoot errors and repeat QA test
- On success, set device.max-users to 96
- For each link connected to the device:
- Contributor un-drains the link by setting link.status to Activated
We propose to create a new device-health-oracle component to periodically monitor the health of links and devices and update a new health field in serviceability's link and device to reflect each link's and device's current health status, as defined by the set of ready-for-service (RFS) criteria defined below. The existing controller component, in turn, supplies the fleet of DZDs with configuration statements that implement the desired network configuration for each status.
Link onboarding has two stages:
- link created with status = Provisioning
- link meets all RFS criteria, status set to Activated.
┌──────────────────────┐
│ device-health-oracle │
└───────┬──────────────┘
│
┌──────────┴───────────────┐
│ │ When all link RFS criteria are met,
poll│ │ update link.health to ReadyForService
│ │
│ │
▼ │
┌───────────────────────-┐ ┌─────────│────────────────────────────┐
│ Link health info from │ │ │ │
│ DZ_Ledger, ClickHouse │ │ ▼ │
│ │ │ ┌──────────────┐ │
└──────────────────────-─┘ │ │ link.health │ │
│ └──────┬───────┘ │
│ │ │
│ │ if link.health is. │
│ │ ReadyForService and │
│ │ link.desired_status is │
│ │ activated, set link.status │
│ ▼ to activated │
│ ┌──────────────────────┐ │
│ │ link.status │◄──────────────┐
│ └──────────────────────┘ │ │
│ │ │
│ serviceability "link" program │ │
└──────────────────────────────────────┘ │
│
│
│ poll
┌───────┴────┐
│ controller │
└────────────┘
▲
│
│ poll
┌─────┴──────┐
│ device │┐
│ fleet ││┐
└────────────┘││
└────────────┘│
└────────────┘
- Determine the burn-in period to use:
- If link.status = Provisioning (new link), use a burn-in period of 200K DZ Ledger slots (about 20 hours)
- If link.status = HardDrained or SoftDrained (link being reactivated after maintenance), use a burn-in period of 5000 DZ Ledger slots (about 30 minutes)
- The link's A and Z interfaces have
<burn-in slots>worth of data in InfluxDB with zero errors and non-zero utilization - The link has
<burn-in slots>of data in the DoubleZero ledger telemetry program with zero loss
- Prerequisites
- Contributor creates new link (desired_status = Activated, status = Provisioning)
- The
device-health-oraclecomponent, when all RFS criteria have been met for the burn-in period, sets link.health to ReadyForService - On link.health update, the serviceability link program sets link.status to Activated when:
- link.desired_status = activated
- and link.health = healthy
- and link.status = SoftDrained|HardDrained
- When link.status is Activated, controller configures link's device interfaces
- Prerequisites
- Contributor sets link.desired_status to HardDrained
- When link.desired_status is set to HardDrained, the serviceability device program sets link.status to Drained
- Controller sets
isis passiveon link interfaces - When maintenance is complete, contributor sets link.desired_status to Activated
- Follow the link onboarding steps above to bring the link back into service
- Contributor sets link.status to HardDrained
- Controller sets
isis passiveon link interfaces - Contributor runs
doublezero link delete --pubkey <pubkey> - Contributor optionally runs
doublezero device interface delete <device-code> <interface-name>for all interfaces - Contributor manually removes the interface configuration with
default interface <interface-name>
As of December 2025, 72 device have been onboarded to DoubleZero mainnet-beta using the manual health check process described in phase 3 above. The automated process outlined here replaces the manual process.
Device activation has three stages:
- device created with status = DeviceProvisioning and desired_status = Activated
- device meets all RFS criteria needed to support having activated links connected, status set to LinkProvisioning
- device meets all RFS criteria for users, status set to Activated.
┌──────────────────────┐
│ device-health-oracle │
└─────────┬────────────┘
│
┌────────────┴─────────────┐
│ │ When all device RFS (link) criteria are met,
poll│ │ update device.health to ReadyForLinks.
│ │
│ │ When all device RFS (link) and RFS (user) criteria
▼ │ are met, update device.health to ReadyForUsers
┌───────────────────────-┐ ┌─────────│────────────────────────────────────────────────────────────────────┐
│ Device health info from│ │ ▼ │
│ (DZ_Ledger / │ │ ┌──────────────┐ │
│ ClickHouse) │ │ │ device.health│ │
└──────────────────────-─┘ │ └──────┬───────┘ │
│ │ if desired_status is Activated and status is DeviceProvisioning │
│ │ and health is ReadForLinks, set device.status to LinkProvisioning │
│ │ │
│ │ if desired_status is Activated and status is LinkProvisioning │
│ │ and health is ReadForUsers, set device.status to Activated │
│ ▼ │
│ ┌──────────────────────┐ │
┌─────►│ device.status │ │
│ │ └──────────────────────┘ │
│ │ │
│ │ serviceability "device" program │
│ └──────────────────────────────────────────────────────────────────────────────┘
│
│
│ poll
┌─────────┴──┐
│ controller │ controller renders device/link config appropriate for the current device/link status
└────────────┘
▲
│
│ poll
┌─────┴──────┐
│ device │┐
│ fleet ││┐
└────────────┘││
└────────────┘│
└────────────┘
These criteria must be met before links connected to the device can be activated.
- Device record created with
- public_ip - set
- dz_prefixes - at least one /29
- max_users - set to 0
- status = Pending
- Loopback255 exists with loopback_type = vpnv4
- Loopback256 exists with loopback_type = ipv4
- DIA
- At least 1 DIA interface defined on chain with status = activated
- At least 1 DIA interface up for
<burn-in slots>with zero errors and non-zero utilization
- Device is reporting interface counter data to ClickHouse for
<burn-in slots>. Verified by querying the ClickHousefact_dz_device_interface_counterstable and checking that the device has at least one record per minute over the burn-in window. - Config agent installed and calling the controller consistently for
<burn-in slots>. Verified by querying the ClickHousecontroller_grpc_getconfig_successtable (database per environment:devnet,testnet,mainnet-beta) and checking that the device has at least one record per minute over the burn-in window. The burn-in window start time is determined by callingGetBlockTimeon the boundary slot (current_slot - burn_in_slots). - Telemetry agent installed and running for
<burn-in slots>(already established by link RFS criteria)
These criteria must be met before users are able to connect to the device.
- Determine the burn-in period to use:
- If device.status = Provisioning (new device), use a burn-in period of 200K DZ Ledger slots (about 20 hours)
- If device.status = Drained (device being reactivated after maintenance), use a burn-in period of 5000 DZ Ledger slots (about 30 minutes)
- At least 1 DZX/WAN link interface defined on chain with status = activated
- At least 1 DZX or WAN link up for
<burn-in slots>with zero errors and non-zero utilization (already established by link RFS criteria) - Device meets the criteria for its device.type (edge, hybrid, transit)
- When device.status = Provisioning, QA tests have emitted 4 "qa test success" prometheus metrics for the device in the past
<burn-in slots>
The following criteria are out of scope because they depend on data that we currently don't have access to.
- DIA
- BGP routes received from DIA are marked with
no-advertise - An ACL is applied to inbound DIA traffic
- The 0th IP in the device's first dz_prefix is configured on loopback100 and reachable via DIA
- BGP routes received from DIA are marked with
- NTP is configured and synced
- DNS is configured and working
- TCAM profile is set
service routing protocols model multi-agenthas been set
- The
device-health-oraclecomponent, when all RFS criteria have been met for the burn-in period, sets device.health to ReadyForUsers - On device.health update, the serviceability link program sets device.status to Activated and device.max-users to 96 (for edge/hybrid devices only) when the following are all true:
- device.desired_status = activated
- and device.health = healthy
- and device.status = Drained
When a device needs maintenance that will interrupt user traffic, such as a firmware upgrade or hardware part replacement, we first soft-drain the device's links to re-route transit traffic around the device. Next, we drain user traffic from the device itself by having the controller add neighbor <w.x.y.z> shutdown to every user BGP session. At this point, from the user's perspective, doublezero status will show a tunnel status of Pending, and the user will not be able to route any traffic over DoubleZero unless they take action by disconnecting a connecting to a different device.
For IBRL users, traffic will re-route via the internet. For multicast users, traffic will be interrupted for the duration of the maintenance, so users will need to fall back on other means to receive the same data over the internet. If users want to remain connected to DoubleZero during the maintenance, they will need to take action to disconnect and reconnect to a different device.
-
Contributor runs
doublezero link update --pubkey PUBKEY --desired-status drainedfor each link connected to the device -
Contributor runs
doublezero device update --pubkey PUBKEY --desired-status drained -
When the contributor sets device.desired_status to Drained, the serviceability device program sets device.status to Drained
-
On link.desired_status update, the serviceability link program sets link.status to HardDrained and sets link.health to Drained.
-
Controller sets
isis passiveon link interfaces -
When maintenance is complete, contributor sets link.desired_status to Activated
-
Follow the link onboarding steps above to bring the link back into service
Out of scope
Add new Provisioning status to LinkStatus. Currently, new links are created with the Pending status and immediately moved to Activated status after resources are assigned. Now, new links will instead be moved to Provisioning status after resources are assigned.
pub enum LinkStatus {
Pending = 0,
Activated = 1,
Suspended = 2,
Deleting = 3,
Rejected = 4,
Requested = 5,
HardDrained = 6,
SoftDrained = 7,
Provisioning = 8, // new status
}
This new link field is updated by the new device-health-oracle component.
pub enum LinkHealth {
Unknown = 0,
Pending = 1, // this link has never met all RFS criteria
ReadyForService = 2, // this link has met all RFS criteria
Impaired = 3, // this link has failed one or more RFS criterion after previously reaching ReadyForService
}
This new link field captures the intention of the contributor. It defaults to Activated when a link is created.
pub enum LinkDesiredStatus {
Activated = 0, // The link should be fully activated once all RFS criteria are met
SoftDrained = 1, // The link's isis metric should be set to its maximum, see rfc9-link-draining.md for details
HardDrained = 2, // The link should be removed from isis, effectively removing it from the network
// Deleted = 3, // Not applicable since offboarding is currently out of scope
}
Add two new statuses to DeviceStatus -- DeviceProvisioning and LinkProvisioning. Currently, new devices are created with the Pending status and immediately moved to Activated status after resources are assigned. Now, new devices will instead be moved to DeviceProvisioning status after resources are assigned. Once device-health-oracle determines all RFS (device) criteria are met, it sets device.health to ReadyForLinks (defined below), and serviceability moves device.status to LinkProvisioning. Once device-health-oracle determines all RFS (links) criteria are met, it sets device.health to ReadyForUsers, and serviceability moves device.status to Activated.
pub enum DeviceStatus {
Pending = 0, // 1. assign resources and change to DeviceProvisioning
Activated = 1, // 4. device and link health checks have passed and device is ready to connect users
Suspended = 2,
Deleting = 3,
Rejected = 4,
Drained = 5,
DeviceProvisioning = 6, // 2. controller provides configuration.
LinkProvisioning = 7, // 3. device health checks have passed and device is ready to connect links
}
pub enum DeviceHealth {
Unknown = 0,
Pending = 1,
ReadyForLinks = 2, // ready to connect links
ReadyForUsers = 3, // ready to connect users
Impaired = 4,
}
pub enum DeviceDesiredStatus {
Pending = 0,
Activated = 1,
Drained = 6,
}
- add LinkHealth
- add LinkDesiredStatus
- add
Drainedto LinkStatus - add logic to update link.status
- allow foundation to override device.status
- add DeviceHealth
- add DeviceDesiredStatus
- add
Drainedto DeviceStatus - add logic to update device.status
- allow foundation to override device.status
- activator/serviceability: when devices and links are created, don't change their status to activated
- controller: add logic that shuts down user BGP, IBGP sessions, MSDP neighbors, and ISIS neighbors when device.status is drained
- Grafana: provide a dashboard to contributors showing detailed device status and health, including maintenance status
Changes are needed to the following components:
- serviceability link and device
- controller
- QA alldevices test
This RFC should improve the operational controls to manage DZDs and links in the network. It introduces an intent based methodology that uses explict fields to achieve the desired state.
This RFC adds a new device-health-oracle component that collects data from Solana and the DZ ledger (serviceability and telemetry), reads data from ClickHouse, and writes data to serviceability.
Contributors should not be rewarded for devices and links that are not in activated status. This check is already present in contributor-rewards (doublezero-offchain/crates/contributor-rewards/src/calculator/shapley_handler.rs).
This RFC improves network operations, but it also works against DoubleZero's long-term goal of being fully decentralized by adding a centralized component, device-health-oracle. Two possible approaches to decentralizing this in the future are: 1) move all health data and related logic to the DZ Ledger, and 2) have contributors run instances of device-health-oracle that operate only on that contributor's links and devices.
- The device-health-oracle component will need credentials for reading from ClickHouse, as well as a key with write access to link.health and device.health. If this key is leaked, an attacker could move a device into activated status even though it's not healthy.
- An attacker could shut down user BGP sessions on DZDs if they gains the ability to update device.status or device.desired_status to Drained.
- The changes can be deployed in a backward-compatible manner, starting with the program updates, followed by the qa and device-health-oracle updates, followed by the controller updates.
- What verification can be implemented to prove that a DZD, link or CYOA interface has been successfully drained and undrained?