Access Node Versioning, Data Integrity & Upgrade Process

[oriolcanades]

TL;DR

• Problem: Data corruption when Access Nodes (ANs) with incompatible internal stacks exchange data.
• Consensus: Treat the AN as a single semantically versioned package (API + software + data model). Use version negotiation + access control to prevent incompatible replication.
• Key causes: (1) Replication occurs at NGSI-LD level; changes in NGSI-LD representation break DB compatibility. (2) Lack of access control at replication level allows remote nodes to affect non-owned objects.
• What’s needed now: Implement node-to-node version checks and ownership-aware access control; define release & migration processes; codify where version is advertised/checked (Desmos vs AN front door).

Background (from the email thread)

• The TM Forum API model is already versioned (DOME uses v4; v5 exists). Corruption is not due to TMF API version.
• In DOME, TMF API → NGSI-LD → Scorpio → SQL/PostgreSQL. Changes in NGSI-LD representation make DB schemas incompatible; replication at NGSI-LD level propagates incompatibilities.
• Access control at replication layer is missing; with proper ownership controls, remote nodes can’t corrupt local-owned objects, and malformed remote-owned objects can be ignored by BAE/clients.

Points of Agreement

1. SemVer for AN (MAJOR.MINOR.PATCH)
   • MAJOR bumps when any component/API/data model introduces breaking changes.
   • ANs must reject requests from different MAJOR versions.
2. Versioning at NGSI-LD replication level
   • Put version in replication URLs/headers; accept only supported versions.
3. Access Control & Ownership
   • Enforce that only owners can mutate their objects; local instance protected from remote writes.
4. Upgrades via export/import at TMF API level (when feasible)
   • DB-level schemas are implementation-specific; safer to re-write via TMF to migrate NGSI-LD representation.
5. Release artifacts
   • Maintain a known-good bundle of component versions (Helm/Docker), recorded in GitOps or a doc.

Proposals in the thread (summarized)

• Jesus Ruiz
  • Keep it simple: SemVer AN + access control solves corruption; software update is separate.
  • Implement version in API header; reject different MAJOR.
  • For data upgrades, embed schema version in objects; provide migration scripts when TMF data model MAJOR changes.
• Paolo Fabriani
  • Start with Problem prep #1: ensure client/server (Desmos) exchange AN version and enforce compatibility.
  • Location of check: Desmos (replication component) should advertise/check compatibility.
• Giuseppe Cafiso (Beppe)
  • unctional framing: avoid corruption from incompatible ANs and from in-node component upgrades.
  • Wants a formal AN package release management (compat matrix, migration tools).
  • Skeptical of binding release mgmt to Desmos; prefers AN-level control (front door + access layer).
  • Highlights need for a DB migration tool when TMF re-import is not feasible due to IDs.
• Gabriele Gerbino
  • Infra safeguard (“health-check” microservice) helps, but application-level controls are required.
  • Suggests split deployment models (K8s/ArgoCD, Docker Compose, AaaS) with automation/sync.
• Karol Bialas
  • Keep release definition simple: a map of Helm chart versions for each app; record/tag after regression tests; publish on request by release owners via GitOps.

Decisions Needed (to close in this discussion)

1. Where to advertise & enforce AN version?
   • Option A (Desmos): P2P handshakes carry AN_VERSION; Desmos enforces compatibility (pro: closest to replication; con: dependency on Desmos).
   • Option B (AN Gateway/Frontdoor): Version surfaced and enforced at AN ingress (pro: future-proof if internals change; con: replication bypasses front door today).
   • Option C (Hybrid): Advertise at both AN gateway and Desmos; enforce in Desmos for replication.
2. Source of Truth for Releases
   • GitOps repo vs Release doc vs both. Define owners and publication flow.
3. Compatibility Policy
   • Hard rule: reject different MAJOR. Optional: allow MINOR N-k window? Define exact policy.
4. Access Control Scope
   • Confirm ownership model and minimal PDP/PEP requirements for replication operations.
5. Migration Strategy
   • When re-import via TMF breaks IDs, do we build/maintain DB migration tools per MAJOR NGSI-LD schema change?
6. Version Field(s)
   • Define AN_VERSION format and whether to also carry SCHEMA_VERSION per object.

Proposed Near-Term Plan (incremental, low-risk)

• P0 – Stop the bleeding (Problem prep #1):
  • Implement AN version negotiation in Desmos (header in discovery/handshake & replication calls).
  • Enforce MAJOR parity; log & reject on mismatch.
  • Add ownership-aware access control at replication endpoints (no remote writes to local-owned).
• P1 – Governance & Releases:
  • Create Release Definition (Helm/Docker map) and compatibility matrix; record in GitOps.
  • Define Release Owner(s) (e.g., Jesus/Bappe), publication flow, and regression checklist.
• P2 – Telemetry & Visibility:
  • Emit AN_VERSION and environment in replication event metadata and health endpoints.
  • Dashboard showing peer versions and compat status.
• P3 – Migration Path (Problem Prepare pre #2/add doc #3):
  • Draft migration playbook (export→clean→import via TMF).
  • Where TMF re-import breaks IDs, spec and prioritize DB migration utility per MAJOR NGSI-LD schema jump.

Owners (suggested):

• Desmos changes: IN2 (Oriol)
• Access control (PEP/PDP) at replication level: IN2 (Jesus)
• Release governance & GitOps tagging: IONOS/ENG (Karol/Beppe) with IN2 concurrence
• Migration tooling assessment: ENG + IN2 (Paolo + Oriol)

Risks & Mitigations

• Bypassing checks (custom deployments): enforce checks in-app (Desmos) not only infra.
• Version drift: require version header in all replication calls; CI to verify presence.
• Migration complexity: limit MAJOR changes; provide reference migration scripts early.

Open Questions to Close Here

[ ] Final choice: A / B / C for version advertisement & enforcement.
[ ] Exact compatibility policy (MAJOR hard-block; MINOR policy?).
[ ] Release owner(s) and publication workflow.
[ ] Minimal PEP/PDP ruleset for replication endpoints.
[ ] Decision on DB migration tool investment when TMF re-import can’t preserve IDs.