Store blobs

[snarfed]

Not a priority for Bridgy Fed or me otherwise personally, but we should probably implement blob storage, uploadBlob/getBlob, etc.

• https://atproto.com/specs/xrpc#blob-upload-and-download
• https://atproto.com/specs/data-model#blob-type
• https://atproto.com/specs/lexicon#blob

[snarfed]

Deprioritizing. Shipped remote blobs w/datastore_storage.AtpRemoteBlob for generating blobs for externally hosted files, which is working well enough for my needs.

[snarfed]

This came up again recently: Bridgy Fed hit a case where (our best guess is) an image URL originally served one format, image/webp, and then later switched to serving an image/webp. We fetched the first image, saw image/webp, stored that and the URL and image CID in an AtpRemoteBlob, and populated that CID and mime type into a blob in a record. Then, the URL switched to serving a image/jpeg, Bluesky team's blob scanning fetched it, saw the type mismatch, and complained.

Not storing/hosting media has been convenient for us, for Reasons etc, but it's technically not ATProto compliant, since we can't guarantee that blobs are immutable, ie the URL we redirect getBlob requests to could serve a different image that doesn't match the CID and type we originally created the blob with.

cc @ericvolp12

[snarfed]

Specifically, the post that triggered this was:

• https://www.dailyrecord.co.uk/sport/football/steven-naismith-issued-hearts-sack-33548822 , from RSS
• ATProto record at://did:plc:kxwjbod4cqrkrk5okcgw7gx6/app.bsky.feed.post/3l2qy6dloi2w2, commit bafyreifwwquqtdmmlfasatrjksudpfnryplidjwunvtl5g56if3rtk56bi
• image https://i2-prod.dailyrecord.co.uk/incoming/article33548828.ece/ALTERNATES/s615/0_JS343200786.jpg , fetched at 2024-08-28 05:21:42, returned:
  • Content-Type: image/webp
  • Content-Length: 33092
  • contents hashing to blob CID bafkreihns5gebrgfbqlmzaono4fi3yrpaem3fjmtkhdx4qc2f5b6adbmjq
• some time between then and 05:32:00 UTC, the same URL switched to returning:
  • Content-Type: image/jpeg
  • Content-Length: 48225
  • contents hashing to blob CID bafkreib5qeer5wmetmazb4tbzuk6afb2a6djhgsqt6rvayiuszlppc3pi4

Maybe the original image here was WEBP, and the JPEG is a downstream transcoding, and the CMS does that transcoding in the background, after the article is published, and serves the original image until the JPEG is ready? Maybe a bit of a stretch, but not too much? I dunno.

Here's our code for this:

arroba/arroba/datastore_storage.py

Lines 297 to 331 in b4e6911

	def get_or_create(cls, *, url=None, get_fn=requests.get):
	"""Returns a new or existing :class:`AtpRemoteBlob` for a given URL.
	If there isn't an existing :class:`AtpRemoteBlob`, fetches the URL over
	the network and creates a new one for it.
	Args:
	url (str)
	get_fn (callable): for making HTTP GET requests
	Returns:
	AtpRemoteBlob: existing or newly created :class:`AtpRemoteBlob`
	Raises:
	requests.RequestException: if the HTTP request to fetch the blob failed
	"""
	assert url
	existing = cls.get_by_id(url)
	if existing:
	return existing
	resp = get_fn(url)
	resp.raise_for_status()
	mime_type = resp.headers.get('Content-Type')
	if not mime_type:
	mime_type, _ = mimetypes.guess_type(url)
	digest = multihash.digest(resp.content, 'sha2-256')
	cid = CID('base58btc', 1, 'raw', digest).encode('base32')
	logger.info(f'Creating new AtpRemoteBlob for {url} CID {cid}')
	mime_type_prop = {'mime_type': mime_type} if mime_type else {}
	blob = cls(id=url, cid=cid, size=len(resp.content), **mime_type_prop)
	blob.put()
	return blob