Handling of ESSR messages in KERIA/Signify setups

[iFergal]

Feature request description/rationale

The parser in keripy can parse and store ESSR payloads. Looking at test_essrs in keripy, an exn message containing the digest of a group of ESSR payloads is followed by the group/count code and the sealed payloads.

I'm curious how this would work in the KERIA/Signify setup. 3 initial cases.

1. Exchange messages:
For example, if I want to apply ESSR to IPEX communications. KERIA needs to process these IPEX exchange messages and not just store them. So is this ESSR between KERIA <-> KERIA; or Signify <-> KERIA etc. Perhaps then can process stored essrs if destination is a KERIA controlled identifier.

2. Various other events etc that are streamed:
For example, after sending a grant message, KERIA will stream other supporting artifacts such as registry events etc. I presume this would be KERIA <-> KERIA encrypted.

3. KERIA <-> Signify API
I raised this in #287 and the suggestion from @SmithSamuelM and @pfeairheller on a call was to have an ESSR payload with the routing embedded in the plaintext instead of normal REST. (So 1 REST endpoint for everything, which could eventually be pure TCP)

I'm not sure how this would be formatted, if it's CESR or what. Since to me it seems different to streaming ESSR packets with the group code. Maybe not.

[SmithSamuelM]

@iFergal See the discussion here WebOfTrust/keripy#612

scroll down until you find the comment:

TSP ESSR Wrapper Payload field

Determining if field value is a CESR primitive or CESR group

The ciphertext CESR code for the payload of an ESSR message indicates what type the decrypted plaintext will be. Typically for nested ESSR messages, we want the payload plaintext to be a sniffable stream type. This means that the plaintext is compatible with the top level of a CESR stream. So a top level CESR stream parser can then parse the decrypted payload.

[iFergal]

Thanks @SmithSamuelM. So if I understand correctly, -Z indicates that the payload when decrypted will be the final plaintext. So, if the parser in KERIA receives these messages, it can decrypt and process them as it was intended for KERIA and not the edge client.

The current parser will store these without decrypting as Texters in hby.db.essrs, after verifying that the digest of the entire group matches what was passed in the head message. So we could have a doer to pull these essrs out, decrypt and process them? And this would be a KERIA <-> KERIA ESSR.

[iFergal]

From dev call:

There's a difference between what needs to be signed for business logic and what needs to be signed for security of over the wire communication. So it's reasonable that Signify can sign as it currently does without ESSR, and KERIA can wrap these messages in ESSR for the agent/controller that needs to decrypt and process the internal message, e.g. another KERIA instance.

This avoids needing to ESSR-ize everything at the edge which makes things must simpler, and also extends the intended usage of the KERIA agent identifier/keys to not only secure the Signify <-> KERIA API, but to wrap things in ESSR. And this should be OK since the KERIA agent has the agent end role for the edge identifier anyway.

Regarding the extraction and processing of the ESSR messages, Phil has done some work to process the stored ESSR messages after they are decrypted and verified, so I think we should be able to do something similar in KERIA.

[iFergal]

In last weeks dev call we discussed ESSR between Signify and KERIA. The cleanest solution suggested was to encrypt the entire HTTP request and embed it in the body of another HTTP request, like a tunneling protocol, and send all wrapped HTTP requests to a single endpoint. This endpoint unwraps and passes the request to the appropriate handler.

#351 implements this. It could be taken further to have a tcp handler for the "wrapper" but this requires more changes and is left as a todo.

[iFergal]

I'm currently investigating how to encrypt IPEX exn messages. It would also be good to encrypt any supporting events streamed when granting (e.g. TEL events) as this leaks information which can be used to correlate revocations.

forwarding.StreamPoster is used in KERIA to send both the exn messages and any supporting events. deliver will concat all messages into a byte stream and send them to the appropriate endpoint. (and for mailboxes, it does some wrapping)

@SmithSamuelM @pfeairheller Does it make sense for the StreamPoster to be configurable to optionally apply ESSR to this stream before sending? Or it's a higher level concern and some helper function should convert the messages to ESSR before passing to the StreamPoster?

The advantage of the former is the whole queue of messages can be signed once, with one ESSRPayloadGroup.