Endorsement Life Cycle

One the key elements curretly missing from services is full life cycle management for endorsements. At the moment, we can only upload new entries, which get accumulated in the store. There is currently no way to delete errors, mark values as no longer good, or to even view the contents of the store (deployments provite a way to dump the entire contents of the store, but is more of a hack than an actual solution).

Use Cases

(Note, unless specified otherwise, the term "endorsements" used as a generic catch-all for provisioned elements and would include endorsed claims, reference values, and trust anchors.)

Life cycle operations

These are operations directly connected to the endorsements' raison d'etre -- being made available for verifying evidence.

• Provisioning endorsements for a new device
• Updating endorsements for an existing device (e.g. software updates)
• Blacklisting provisioned values (e.g. previous known-good values are no longer that due to a discovered exploit). Form the verifier perspective, this can look in to a couple of different ways:
  • Blacklisted endorsements are no longer returned in response to verifier queries.
  • Blacklisted endorsements are returned but are marked as blacklisted.
• Endorsements are retired (e.g. the associated device is EOL, certain model of equipment is removed from the network, etc). This may involve an archival operation, removing the entries from the live store, or simply marking the entries as retired so that they are no longer considered when processing queries. The endorsements still need to remain accessible (e.g. for future audits).

Management operations

These are operations not directly related to the primary purpose of the endorsements, but would need to be supported by any system designed to be used in the real world.

• Deletion (e.g. a CoRIM has been uploaded in error and needs to be "rolled back"). This is distinct from retirement, as we want to simply remove a mistaking without leaving a lasting impact on the system (outside of the logged activity).
• Monitoring the contents of the store. This involves running a potentially more diverse set of queries than are needed by the verifier. E.g.
  • What devices are supported?
  • What software versions are supported for a device?
  • How many distinct instances of a device class are registered?
  • (Potentially) when was last time endorsements for a device were accessed?
• Sharding / replication / availability across regions. (This should mostly be handled by the underlying DBMS, but may required some explicit support at store API level such as origin tracking ).

Musings

I believe our current key-value store design puts some sever limitations on how these use cases can be addressed. It is intentionally devoid of structure, since, at the time, we did not want to be committed to a specific representation for endorsements, and wanted to leave as much as possible up to the individual schemes. However, this makes it difficult to implement any sort of generic life cycle / management that is agnostic of scheme details.

Since then, our understanding of endorsements has advance, as we have seen a couple of alternative scenarios: CoSERV, and section 9 of the CoRIM spec (the "CoRIM verifier"). Both are remarkably similar in how endorsement look up is handled. The look up key/query is represented as a combination of environments and measurements ("stateful environments" in CoSERV, "condition ECTs" in section 9). The returned values are represented as same enviroment/measurement combos associated with an authority (the various "quads" in CoSERV, "addition ECTs" in section 9).

In the past, we explicitly avoided using CoRIM structures, such as Environment's, in or internal representations, as we did not want to assume CoRIM as the input format. However, I think there is now sufficient evidence that the Environment and Measurement structures are good generic abstractions for endorsements. We could use them internally without assuming CoRIM as the input (effectively making the assumption that endorsements in any format can be translated into these primitives).

This suggests that a store API designed around these primitives would be more powerful that our existing string -> string key-value API, supporting a wider array of use case (including all of the ones above), while remaining entirely scheme-agnostic.

Proposal

Create an endorsement store as a stand-alone component implemented in Golang, with the API designed around CoRIM primitives, and the (initial) implementation in terms of an SQL-based DBMS. The initial goal would be to replace the existing k-v store in services with one that will support implementation of life cycle management in the provisioning REST APIs.

In the future, the same component could be used for implementing a CoSERV server or CoRIM verifier store backend (would require a Rust implementation for the API).

Analysis

• The SQL implementation would also give us an in-memory implementation "for free" via the Sqlite3's :memory: database.
• A document store implementation should be fairly straight forward if we want one in the future -- essentially, serialise the CoRIM triples to JSON and shove them into the store, which naturally supports structured document queries.
• Integrating the new store into services will be a significant effort. It will require updating exiting scheme plugins, as well as the generic code, to support the new API. However, it will result in overall simpler code (especially on the provisioning side, as the store will natively support CoRIM types without the need for scheme-specific transformation).
• It will also lay the ground word for pushing more functionality out of plugins into the generic code (thus simplifying new scheme implementation).