Provide a proper endDate in publishedBy field

[dchiquito]

When publishing, dandi-api generates a publishedBy field with some information about the publish process. Two of the required fields are startDate and endDate. Generating the metadata to use in the publishing is the first step, so it isn't possible to know what endDate at that time.

Since publishing is a relatively quick operation, dandi-api has been setting startDate and endDate to the same timestamp, determined at the start of the publishing process.

[waxlamp]

What is the meaning of startDate and endDate in context? If it's supposed to track how long the publish step took, why store it as metadata rather than as some sort of internal diagnostic data? (attn: @dchiquito)

[dchiquito]

What is the meaning of startDate and endDate in context?
@satra

[satra]

yes for any activity it takes some finite period of time. so the provenance would capture start and end of that activity/task.

[waxlamp]

Thanks for confirming, Satra.

@dchiquito, is there a clear path forward here? The issue report makes it seem uncertain how to generate endDate.

[dchiquito]

I'm still not sure how we should handle endDate. Part of the publishing process involves making a DOI, which needs the entire dandiset metadata, including the endDate, which is only knowable once publish is complete. It's a chicken/egg thing.
Publishing is pretty fast so just using the same value for startDate and endDate is within a few seconds of the correct value, but it is technically lying. Unless endDate is redefined as something else, any "better" guess we put there would still be technically lying.

[waxlamp]

@satra, what's the end game for this feature? If there's not a specific reason you need these, I'd say we should just have a date field (or, to avoid having to release a new schema, just continue to use the same date for both date fields).

[satra]

activities should record their start and end times in the provenance record. and provenance is a fundamental part of our schema. whether we treat certain activities as trivial or not is a separate discussion.

i would say the chicken and egg situation here is that writing the record to s3 as part of the publish process means that the record contains an end date before write has finished. i'm ok with living with that.

the doi generation step doesn't care about the publishedBy activity, so the chicken and egg problem doesn't exist there. and we are updating the metadata post doi generation, before saving the record. it's at that time the endDate could be inserted/updated.

when we have a separate graph database, we can decouple some of the info. for now, just add/update it before saving to the django database.

[waxlamp]

Then is there anything to do here? Just update how endDate is written during DOI generation?

[waxlamp]

ping @satra

[satra]

i think i could live with these steps.

1. fill metadata
2. generate doi
3. stick doi in metadata
4. update enddate to be now + 1min (or choose a time roughly based on how long it takes to write)
5. write to db/s3/etc.,. (assuming this doesn't take more than a min).

this would be both correct and create a publishing window within which other timestamps exist (e.g., things written to s3).