make the transfer and issuance metadata available in the db

[arner]

This PR makes the public metadata of Issuance and Transfer actions, which are stored on the ledger, available to the user via the database (on the sender, recipient, and auditor side). At the moment this includes the hash and preimage of HTLC lock and claim, which can be used to display to the end user or for reporting.

The user can already add this kind of metadata to a Transfer with the opt:tkn.WithTransferMetadata("user.x."+tx.ID(), []byte("this is metadata")). At the moment this causes the validation to fail, but that can be fixed if we add a new ValidateTransferFunc, somewhat like the following:

func TransferExtraDataValidate(ctx *validator.Context) error {
	for key, val := range ctx.TransferAction.Metadata {
		if strings.HasPrefix(key, "user.") {
                        // TBD: validation?
			ctx.CountMetadataKey(key)
		}
	}
	return nil
}

It would add some useful tools to the toolbox. For instance you could share metadata privately with the counterparty and optionally an auditor in the ApplicationMetadata, and a hash of that data on the public ledger, all tied to the same transaction. Or link it to external data, or create another custom validator for additional proofs. It would be the responsibility of the user to make sure that privacy is protected.

Important

This change is not backwards compatible due to changing the database schema. It would require a manual ALTER TABLE ... ADD COLUMN public_metadata JSONB NOT NULL; before upgrading. Until we have a common method for database migrations, I couldn't find a good way to handle it.

Looking forward to feedback on this PR and the idea for the followup.

[adecaro]

Hi @arner , thanks for this PR. I will look at it ASAP.
I have a question: Do you have any suggestion on how to handle these schema changes? Should we have a schema versioning? Should the code be able to handle multiple versions? What's your take.
@alexandrosfilios , please, jump in as well :)

[alexandrosfilios]

Hi @arner , thanks for this PR. I will look at it ASAP. I have a question: Do you have any suggestion on how to handle these schema changes? Should we have a schema versioning? Should the code be able to handle multiple versions? What's your take. @alexandrosfilios , please, jump in as well :)

I have only used liquibase before for Java. We need a tool that:

• is written in Go, we can "inherit" the changes of parent projects, e.g. KVS table updates
• maybe supports multiple db types
• easy to use with Kubernetes or an automated deployment tool

Here are some options (I'll keep on investigating):

• Golang-migrate/migrate (https://github.com/golang-migrate/migrate):
  • It is pretty straightforward (SQL files only) and the most-widely used
  • Supports multiple DB's
• Goose (https://github.com/pressly/goose):
  • We can write the migrations in go (more custom logic with templates and conditions/loops), but also in SQL
  • Doesn't support multiple DBs
• Atlas (https://github.com/ariga/atlas):
  • Declarative (not SQL-based/imperative)
  • Generates diffs
  • Support multiple DB's

I think that overall so far the best solution is Golang-migrate. It's straightforward and that would make its use with Kubernetes or ansible scripts easier. It's more manual, but I think we don't have any complex changes in the schemas.

[arner]

I have no experience with golang-migrate but I agree that its simplicity and the fact that it just uses SQL files is a good thing, it gives flexibility to the user and transparency.

One additional requirement is that it must be possible (or even the default / mandatory?) to manage the schema outside of the application. At least for security: the database user of the token sdk application should not have the rights to change schemas and drop tables. But also for versioning, rollbacks and possible data updates, that would be risky and difficult through application deployments and configuration. And finally it's likely that in larger companies the application/deployment team is not the same as the database administration team, with specific audit requirements, etc, so in some cases it would not even be allowed to let the application handle it.

That means that the content of the "CreateTable" should be externalized to the sql files that the tools require. All in one place would be easiest for an administrator, but it would break the independence of the different data stores a bit. And we would likely need a separate directory for postgres and one for sqlite.

edit: the dynamic table names are a complication here. I know that some of the tests rely on it and I don't know if there are any features that require the namespacing. The names are based on the network and chaincode.

[adecaro]

I can see an epic to handle all this, honestly. There are many things to streamline in the lines of what you say @arner .
@alexandrosfilios , shall we craft with Arne, an EPIC on the changes to do?

[adecaro]

LGTM modulo the comments. Thanks.

[adecaro]

Please, update the comment of the struct and maybe some information about what Attributes is supposed to contain.

[adecaro]

do we test that we get back what we store?