Skip to content

federation.md

Latest commit

 

History

History
179 lines (110 loc) · 8.27 KB

federation.md

File metadata and controls

179 lines (110 loc) · 8.27 KB
Raw

Federation

Hub-and-spoke sync between Flair instances. A hub instance coordinates sync for one or more spoke instances.

Overview

Federation lets multiple Flair instances share memories, relationships, and agent records. Each instance maintains its own Ed25519 identity. Sync requests are signed and verified against pinned peer public keys.

Spoke A ──[signed sync]──▶ Hub ◀──[signed sync]── Spoke B
  • Hub: accepts sync pushes from paired spokes, can relay records between peers
  • Spoke: pushes local changes to the hub, receives changes from other spokes via the hub

Pairing a New Spoke (Bootstrap-User Flow)

Pairing connects a spoke to a hub with mutual key pinning and an auth-aware handshake that works across all Harper topologies, including Harper Fabric.

Step-by-step

1. Hub admin generates a pairing token triple

On the hub machine, the admin runs flair federation token. The command emits a JSON triple containing a one-time bootstrap credential:

flair federation token --admin-pass <hub-admin-password>

Output (a single JSON object):

{"token":"<one-time-pairing-token>","user":"<bootstrap-username>","password":"<bootstrap-password>","expiresAt":"<ISO-8601-timestamp>"}

Tokens expire after 60 minutes by default. Use --ttl <minutes> to adjust.

2. Hub admin shares the triple with the spoke admin

The JSON triple is shared out-of-band (secure file transfer, password manager, or similar). It must be stored as a plain JSON file on the spoke side or piped via stdin.

3. Spoke admin runs the pair command

On the spoke machine:

# From a file
flair federation pair <hub-url> --token-from /path/to/triple.json

# From stdin
cat triple.json | flair federation pair <hub-url> --token-from -

4. Behind the scenes

  • The bootstrap user authenticates at the platform layer (works on standalone deployments and Harper Fabric alike).
  • The resource handler validates the pairing token, the signed request body, and the binding between the bootstrap user and the token.
  • On success the hub creates a Peer record for the spoke and removes the temporary bootstrap user. The spoke records the hub as its peer.

After pairing, both instances pin each other's Ed25519 public keys and are ready to sync.

Why the Bootstrap-User Flow?

Earlier designs relied on allowCreate=true combined with body-only authentication. That approach works on single-component deployments but breaks on Harper Fabric, where the platform authentication gate fires before the resource handler sees the request. The bootstrap-user flow (Option B) makes the pair handshake auth-aware so it operates correctly on all Harper topologies: standalone, Fabric single-node, and Fabric multi-node.

Fabric Pairing Example

When the hub runs on Harper Fabric, adapt the hub URL to the Fabric pattern:

# 1. Hub admin generates the triple (on the Fabric host)
ssh hub-host
flair federation token --admin-pass <hub-admin-password> > /tmp/pair-triple.json

# 2. Transfer the triple to the spoke admin (out-of-band)
scp hub-host:/tmp/pair-triple.json ./pair-triple.json

# 3. Spoke admin pairs using the Fabric URL
flair federation pair https://<fabric-node>:19926/<instance-name> --token-from ./pair-triple.json

Replace <fabric-node>, <instance-name>, and <hub-admin-password> with your actual values.

Sync

Push local changes to the hub:

flair federation sync --admin-pass <password>
# Output: ✅ Synced 12 records (0 skipped) in 145ms

Security

Signed requests

Every federation request (pair, sync) is signed with the sender's Ed25519 private key. The receiver verifies the signature against the peer's pinned public key. Unsigned or tampered requests are rejected with 401.

The signature covers the canonical JSON of the request body (keys sorted recursively, signature field excluded).

Encrypted key storage

Private key seeds are stored in ~/.flair/keys/<instanceId>.key, encrypted with AES-256-GCM. The encryption key is derived via HKDF from:

  1. FLAIR_KEY_PASSPHRASE environment variable (recommended for production), or
  2. An auto-generated random passphrase at ~/.flair/keys/.passphrase (mode 0600)

If neither the env var nor the passphrase file can be accessed, federation identity creation fails. Private keys are never stored in the database.

Pairing tokens

New peers must present a valid, unexpired, unused pairing token. This prevents unauthorized instances from joining the federation. Tokens are generated by the hub admin and are single-use.

Re-pairing an existing peer (same instance ID, same public key) does not require a token.

Originator enforcement

Spoke instances can only push records they originated. A spoke cannot overwrite records from another spoke or from the hub. The hub can relay records from any origin.

Timestamp ceiling

Records with updatedAt more than 5 minutes in the future are rejected. This prevents an attacker from using far-future timestamps to permanently win last-write-wins (LWW) merge conflicts.

CLI Reference

Command Description
flair federation status Show instance identity and peer connections
flair federation pair <hub-url> --token-from <file> Pair this spoke with a hub using a token triple file (or - for stdin)
flair federation sync Push local changes to the hub (one-shot)
flair federation watch [--interval <s>] Run sync in a foreground daemon loop (default 30s)
flair federation reachability Probe local instance + each paired peer (read-only)
flair federation token [--ttl <min>] Generate a one-time pairing token triple (hub only)

Conflict Resolution

Federation uses record-level last-write-wins (LWW) with ISO timestamp comparison. When two instances modify the same record, the one with the later updatedAt wins. Field-level LWW is planned for a future version.

Troubleshooting

config.yaml port drift

If the hub's configured port in config.yaml differs from the port the spoke is targeting, federation requests fail with a connection error. Verify the port matches between the spoke's pair URL and the hub's config.yaml (federation.port or the instance's listen port).

# On the hub, confirm the listening port
grep -E 'port|federation' ~/.flair/config.yaml

Local FederationInstance fetch needs auth

When troubleshooting on the hub, fetching /federation/instances/<id> locally (e.g. via curl localhost) may return a 401 if the request does not carry the authentication headers the platform layer expects. On Fabric this gate is enforced even on localhost. Use the CLI tooling (flair federation status) instead of raw HTTP calls for local inspection.

flair_pair_initiator role not found on hub

If pairing fails with a role-not-found error, the hub instance may be missing the flair_pair_initiator role. This role is created automatically during flair init --remote but can be lost if the database was reset or migrated manually. Re-run flair init --remote on the hub to restore default roles, then retry pairing.

Bootstrap user not deleted on Fabric

After a successful pairing, the temporary bootstrap user is automatically deleted. If it persists on a Fabric deployment, check that the hub's Harper operations log does not show a rollback or permission error during the cleanup step. Manually removing the stale bootstrap user via the Harper Studio is safe if needed — it is never used after pairing completes.

Stale Peer record on spoke

If a spoke was previously paired with a different hub (or the hub's identity key changed), the spoke may retain a stale Peer record pointing to the old hub. Remove the stale record before pairing with the new hub:

# Show current peers
flair federation status

# Remove a specific peer (replace <instanceId>)
flair federation unpin <instanceId>

Limitations (1.0)

  • HTTP push only — no persistent WebSocket connections or real-time sync
  • Manual or watch-loop sync — flair federation sync is one-shot; flair federation watch --interval 30 runs it on a loop in a foreground daemon (wrap in launchd / systemd for production)
  • Single hub — spoke-to-spoke sync goes through the hub
  • Record-level LWW — not field-level; concurrent edits to different fields of the same record may lose data