Skip to content

[FEAT] S3 asset handling library and associated integrations #291

@tigattack

Description

@tigattack

The team has been discussing1 the use of R2 buckets for storing assets.

Some example use cases:

  • User notes/warnings assets (see notes cog) — Previously these have just been dumped in the channel at the time of note creation, making discovery difficult and causing problems if/when we wish to rotate the channel (e.g. upon new mod onboarding).
  • Timeout archives (see jail cog)
  • Custom command assets - For example images, where often links will die over time, and we find ourselves left with broken CC's.

@portalBlock had the idea to create a Python library which could ingest a Discord message object2 and upload all attachments to an R2 bucket, returning a link or ID to each file.

MVP

  • Each set of attachments (a set being defined as all attachments from a single message) should be given an ID. This way, sets can be retrieved with ease, reducing code complexity in each cog. The set ID can be stored as a tag/metadata string on each S3 object. See Cloudflare R2 - Extended metadata using Unicode.
  • Thoughts on implementation in the notes cog for the sake of example, and as it's the only cog currently in active use where this would be implemented:
    • We will need a way of accessing attachment sets for notes. Decided [p]notes asset <id> is best.
    • id should be the note ID, which is then used to lookup the attachment set ID internally, retrieve the set, and dump back to the user.
    • The cog will also need additional commands to set S3 info (endpoint, bucket, credentials...)
    • We'll need to extend the cog's config schema to store the attachment set ID for each note and guild-level S3 details.

Further thoughts

  • It's probably best to commit this library to a separate repo, e.g. LabBot-Libs. This reduces clutter in this repo, especially if/when we add more shared libs in the future. I'd love to use Red's shared libraries feature, but it's been deprecated since v3.2 and is marked for removal in the future 😞
  • Adding this library to each cog would be as simple as adding a string such as the following to the requirements list in each cog's info.json: pkg @ git+https://github.com/rHomelab/LabBot-Libs/#subdirectory=discord_s3_assets
  • We discussed the implementation of a web frontend for viewing assets. This would simplify

Footnotes

  1. For team members, search mod chat for discord media links to r2 bucket files to find the beginning of the discussion.

  2. Upon writing this I also considered that this could be reduced down to simply the List[Attachment] (see discord.Attachment in the attachments attribute of discord.Message objects.

Metadata

Metadata

Assignees

No one assigned

    Labels

    cog-jailIssues related to the jail cog.cog-notesIssues related to the notes cog.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions