[Enhancement] Prevent videos in cache from refreshing too often

[Fijxu]

I recently noticed that the Invidious video cache that is stored in the database to prevent fetching the video again in again from Innertube limited /player endpoint makes Invidious re-fetch the video if it has been clicked a user within those 10 minutes.

invidious/src/invidious/videos.cr

Line 302 in e7f8b15

(Time.utc - video.updated > 10.minutes) ||

I think this value for the cache was in place to keep the video information updated in case there was any changes back when the Innertube /player endpoint wasn't limited, but since we are now rate limited, this value would be better to sit at 6 hours (which is the current expiration time of the video stream since the video was requested) or we can calculate it from the expire query parameter in videoplayback requests. Or we can just remove this check and rely on:

invidious/src/invidious/database/nonces.cr

Lines 20 to 27 in e7f8b15

	def delete_expired
	request = <<-SQL
	DELETE FROM nonces *
	WHERE expire < now()
	SQL
	PG_DB.exec(request)
	end

with it's job to remove the expired (or ones that are going to expire) videos from cache:

invidious/src/invidious/jobs/clear_expired_items_job.cr

Lines 10 to 24 in e7f8b15

	begin
	Invidious::Database::Videos.delete_expired
	Invidious::Database::Nonces.delete_expired
	rescue DB::Error
	failed = true
	end
	# Retry earlier than scheduled on DB error
	if failed
	LOGGER.info("jobs: ClearExpiredItems failed. Retrying in 10 minutes.")
	sleep 10.minutes
	else
	LOGGER.info("jobs: ClearExpiredItems done.")
	sleep 1.hour
	end

[unixfox]

Remember that the video cache do cache from /player and /next. We aren't rate limited on /next.

And for /player we also have the cache for 1 hour in companion: https://github.com/iv-org/invidious-companion/blob/719128544ff3b6ddb852050395d798acc5a88078/src/lib/helpers/youtubePlayerHandling.ts#L168

[Fijxu]

Remember that the video cache do cache from /player and /next. We aren't rate limited on /next.

So separating those two caches would be better, a short amount of time for /next requests and a longer time for /player requests.

And for /player we also have the cache for 1 hour in companion: https://github.com/iv-org/invidious-companion/blob/719128544ff3b6ddb852050395d798acc5a88078/src/lib/helpers/youtubePlayerHandling.ts#L168

Why is one hour and not the expire time of the videoplayback link? (6 hours at this date).

[unixfox]

Why is one hour and not the expire time of the videoplayback link? (6 hours at this date).

The cache doesn't only contain videoplayback links, in /player request, it includes the title, the description, the thumbnail, the view count, the video qualities available and more.

So in under 6 hours, many of these things can change and keeping a cache for too long can really impact the “freshness” of the data returned to the user, especially on new videos.

There is also the fact that sometimes we can cache an invalid videoplayback URL. So if the cache is short, this will annoy a bit less the user which can try again in one hour and succeed with a better working link.

YouTube doesn't rate limit the /player, it finds the unsupported/unofficial usage of it and block these “usage”. I don't think extending the cache will help with reducing the likeliness of YouTube finding these kinds of usage.