PDF generation of CodiMD pages

[GovernmentPlates]

Note

This is a feature request that has originated from a support ticket

Sometimes, CERN Indico users attach a link to their CodiMD document containing minutes. It would be nice to schedule a job (or after some time has passed) to automatically take the CodiMD document -> convert it into a PDF -> attach it to Indico.

There are a few pitfalls to this (with a few possible solutions):

• When should we do this? How and when should we determine when a meeting has ended and that all changes in the CodiMD are final? We could have a 48h delay after the meeting has officially ended (end_dt + 48) where we then execute the job to fetch the contents of the CodiMD (mockup attached below) - this could also be configurable (e.g. have an option to run the conversion immediately when the CodiMD link is saved or at a set period after the event finishes etc.).
• How should we handle updates to the CodiMD document? In other words: how should we handle the polling/refreshing of the CodiMD document? Should we even do this at all?

Mockup (rough):

[ThiefMaster]

Just for the sake of completeness, my comment from the ticket addressing some of the points like configuring when to do the conversion:

I usually like customization options and configurability a lot. Here I think it just adds complexity. So I would rather choose a reasonable default and go for that.

The placeholder sounds good. We already have this for pending conversions but much more short-lived.
TBH I see this as something that fits nicely in our medium-term idea of having some relationship between files (ie the converted PDF file would be linked to its source link/file), and the plugin only handling the actual conversion.

So if we end up implementing this earlier, we would probably just mention in the description of the "convert to pdf" field that in case of codimd links the conversion happens after the event ended...

[pferreir]

So, what I'm considering:

• Celery task which lists, every N minutes (configurable), all attachment links starting with https://codimd.web.cern.ch. If the event where an attachment exists finished more than X hours (configurable) ago, download a PDF copy (https://codimd.web.cern.ch/xxxx/pdf). If the file is not publicly accessible, too bad (can be improved in the future)
• I'd have a DB table with the list of CodiMD URLs which were already processed, for logging and also to avoid checking the same URL twice (and potentially every single time the task is run)
• Once the PDF is downloaded, I'd attach it to the same event/session/contribution as an actual PDF file, and with the same Indico protection settings as the source link. Not sure about the name. It could be something like "Archived CodiMD" (with a suffix in case of there are multiple)
• I'd add a CLI command to index all matching pages, just to avoid that the first run of the task takes forever. I would also have a "refresh" command to force a refresh on a given event. A smarter refresh strategy could be envisaged in the future.

[ThiefMaster]

For the task: I'd already limit the query so it only returns these files in events that ended within the specified timeframe (I think archiving it only 1-2 days AFTER the event ended may be a good idea). Shoul be much cheaper as well since it'll just be a handful of events

Caching: I think we can avoid this because the task itself would only run once a day, and thus there shouldn't be any overlap between the events (could also include the codimd id or a short hash of it in the filename to check if something was already converted, e.g. if someone changes the event time)

Adding the file: Sounds good, you can probably reuse the same logic we use to attach the office-to-PDF conversion results which (hopefully) already handles protection.

CLI: Not sure if that's necessary if the task only looks at events finished within a 1-day period... but probably useful for testing or fixing things (if e.g. celery was down) if it lets you specify the timeframe

[pferreir]

Hm... there would be a few corner cases which wouldn't be covered, like for instance adding a CodiMD link after end_dt + 48h. Shouldn't happen often, and maybe we can live with that, though...
As for the CLI, do you mean that you wouldn't do an initial import of past CodiMD files? If the goal is to archive files which might otherwise be lost, then I think doing it would be necessary to archive also older ones.

Also, do we want to somehow archive the Markdown source too? It's super small and might be useful in the future. The only problem might be where to place it (hidden folder?)

[ThiefMaster]

As for the CLI, do you mean that you wouldn't do an initial import of past CodiMD files? If the goal is to archive files which might otherwise be lost, then I think doing it would be necessary to archive also older ones.

Yeah I would have done it only for new events, but good point - doing it one-off for old ones seems like a good idea.

Also, do we want to somehow archive the Markdown source too? It's super small and might be useful in the future. The only problem might be where to place it (hidden folder?)

Could even attach it as a regular file...

[pferreir]

Could even attach it as a regular file...

Just trying to avoid "spamming" the material view. But we'll see what it looks like.