Are pastes being reaped properly?

[supakeen]

I've seen some evidence in bpa.st's database and there is some circumstantial evidence from the Discord Python instance that pastes aren't properly getting reaped by the PeriodicCallback that should take care of this.

Note: pastes that have expired are still not visible because the Show page will delete the paste if it's visited while expired.

However, this leads to a needlessly large database. So let's investigate:

• Does the PeriodicCallback work?
• Are pastes being reaped on all show pages?
• Add test cases for these things.

[shtlrs]

I'd like to tackle this.

I also have a couple of questions

Are pastes being reaped on all show pages?

What's meant by "all show pages" here ? Does it even matter since we're fetching the pastes direcltly from the databse based on the expiration date of each paste.
Or are you talking about this:

https://github.com/supakeen/pinnwand/blob/3c866b470b2d2df9a53b783416101c44c2b1229a/src/pinnwand/handler/api_v1.py#L141C1-L143

Does the PeriodicCallback work?

When I run the app, the job runs every n seconds just as configured, so i'm not sure whether we're supposed to question tornado's implementation here. Maybe you could elaborate more on what you've noticed ? Do you see consistent logs of the scheduled task running according to the interval you've set ?

Add test cases for these things.

The problem with this is data isolation between test instances, I currently have no clue how we can isolate pastes/files created in test-X from the ones created in test-Y, as we'll need to be able to make precise assertions on pastes added then reaped.

EDIT

After some small research, I guess the testing part can be done through fixtures and some monkey-patching here & there.

[supakeen]

@shtlrs good questions, feel free to take this!

1. Pastes should be reaped on the show pages (any of the APIs, the 'web'-part of the application) when they are encountered with an expiry date in the past, see for example: https://github.com/supakeen/pinnwand/blob/master/src/pinnwand/handler/website.py#L306
2. It runs, but does it keep running. I've encountered instances where a lot of expired pastes got left behind which can only mean that the PeriodicCallback has gotten stuck (sorry: I don't have more details on this).
3. Fixtures that create a new database with a given set of data in it is probably best; that way you'd also not need to monkey patch the ORM.

[shtlrs]

It runs, but does it keep running. I've encountered instances where a lot of expired pastes got left behind which can only mean that the PeriodicCallback has gotten stuck (sorry: I don't have more details on this).

According to my very limited testing, it does. The task has kept on being invoked in the correct specified interval.

[supakeen]

Right, perhaps we can inject a fault in there and see if that breaks things if not then we can scrap that scenario off the list and only ensure that reaps also happen on access in every code path.

[shtlrs]

I've already made a test that makes sure it runs correctly, I'll finish the other tests and push it