provide `wasGeneratedBy` record in metadata while creating a new dandiset

[yarikoptic]

DandisetMeta allows for it, and we populate it for assets. To establish the provenance trail I guess we should keep extending wasGeneratedBy by any agent which alters the dandiset record: web-ui (including the meditor), dandi-api, etc.

[waxlamp]

Is this an issue for dandiarchive, the API, or for all clients (i.e., dandiarchive + dandi-cli)?

[yarikoptic]

In principle it is an issue for anything which creates (and edits?) metadata record. We populate it in dandi-cli when we extract metadata so we state what process extracted it. See e.g.

$> dandi ls -r --metadata assets -f yaml https://gui.dandiarchive.org/#/dandiset/000003/draft/files?location=sub-YutaMouse20%2Fsub-YutaMouse20_ses-YutaMouse20-140324_behavior+ecephys.nwb
...
    wasGeneratedBy:
    - identifier: c54bfca8-ce87-4ffa-ad85-a736b115e01c
      name: YutaMouse20-140324
      schemaKey: Session
    - description: Metadata generated by DANDI cli
      id: urn:uuid:86c1fbd3-0e00-44f7-8def-ef9f2afe678a
      name: Metadata generation
      schemaKey: Activity
      wasAssociatedWith:
      - identifier: RRID:SCR_019009
        name: DANDI Command Line Interface
        schemaKey: Software
        url: https://github.com/dandi/dandi-cli
        version: 0.14.1+4.g350843c
  modified: '2021-05-18T13:12:19.003195Z'
  path: sub-YutaMouse20/sub-YutaMouse20_ses-YutaMouse20-140324_behavior+ecephys.nwb
  size: 15090018364

dandiarchive web-ui component edits dandiset metadata record. So it should provide (or possibly update/replace existing one?) a similar wasGeneratedBy record. The simplest one would be just for now to replace existing one (if exists) with a new one. Also information on how has done it would be great to record with at least the name (we did not do it in dandi-cli since we do not have "registered user") of who did the latest edit.

[waxlamp]

It sounds like the wasGeneratedBy data should be an input to the metadata editing API endpoint? Then all clients would need to "self-identify" when editing metadata, and the API could record the appropriate information in the metadata automatically. Or am I missing the core requirements here?

[satra]

let's punt this for a bit. we can add this, but i think we need a better approach to provenance than this. this source could be trusted in the short term, but in reality anyone using the API could make up anything. we should think through how best to track this in a systematic way.

[waxlamp]

As part of the thinking on this topic, I'd love to understand the core, pre-engineering requirements. i.e., what are we wanting to accomplish with "provenance", what knowledge do we want to retain into the future, etc. That will directly affect engineering design for the feature.

[satra]

we will soon be running a neptune instance where i would like to send provenance messages to from the API server. i'm waiting for all of our metadata to be cleaned up before doing this. once that is in place, we should already have most of the pieces for provenance. since this would be public, we will not want to put things like IP addresses there. it will also require some crafting of the messaging triggers and protocol.

we will still need to have audit tracking through more standard logging. i mainly want to decouple the essentials we need from logging (IP, who, what API endpoint) versus the richer metadata we need for provenance tracking.

[waxlamp]

So it sounds like the idea is: distill changes about the metadata from the implementation details of how those changes specifically happened, and send the distillate to the neptune server.

[satra]

not sure if i would use the term distill (since the info going to neptune will actually be quite a bit), but yes :)

[waxlamp]

@satra, is this still on your radar? If it's not important, we could close the issue; but if you still want to do this, we should discuss how we could accomplish it.

One idea is a Django app to serve as a middleware for API calls that would intercept, "distill", and contact your Neptune server, while allowing the usual API processing to happen otherwise undisturbed. There's no problem of trusting clients because the provenance messages would not be affected by API calls--the API itself would compute what messages to send, and we could configure the middleware to send those messages to an appropriate place.