Automatically delete all old Notebook jobs

[nakvic]

Problem

I have dozens of jobs running during a day.
Each Notebook job an their results are persisted in DB and on disk.
Unfortunately, if I want to clean old jobs, I need to click delete button for each one twice.
Currently I have around 60 jobs a day, what makes almost impossible to clean not needed results.

Proposed Solution

It would make sense to have a possibility to clean old and outdated jobs on a scheduler.
For instance, to have an option for a job definition or globally in scheduler to remove old jobs executed N days ago.

An option to delete all jobs from job definition would also make sense to make it easy to delete all old jobs at once.

Additional context

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[quentindurpoix]

I would like to revive this thread, this is something that would improve the all experience a lot.

Is there any way of doing the same thing from outside the UI? I have some minute jobs that generate 100Ks reports very quickly.

[robertritz]

Until this feature is implemented, I have created a very short notebook that runs once a day and removed jobs older than N days from the included sqlite database. FYI if you are interested @quentindurpoix.

I do something similar for failed notebooks jobs and a service email account I have for system notices.

Here is the code:

import sqlite3
import datetime

conn = sqlite3.connect('/home/ritz/.local/share/jupyter/scheduler.sqlite')
cur = conn.cursor()

# Simply change the number of days to whatever you want.
timestamp_threshold = int((datetime.datetime.now() - datetime.timedelta(days=60)).timestamp() * 1000)

cur.execute(f"DELETE FROM jobs WHERE start_time < {timestamp_threshold}")
conn.commit()
conn.close()

[quentindurpoix]

It is working perfectly and that's what I was looking for! Thank you so much @robertritz

Let's hope that a real feature will be implemented in the future!

[fendwich]

Until this feature is implemented, I have created a very short notebook that runs once a day and removed jobs older than N days from the included sqlite database. FYI if you are interested @quentindurpoix.

I do something similar for failed notebooks jobs and a service email account I have for system notices.

Here is the code:

import sqlite3
import datetime

conn = sqlite3.connect('/home/ritz/.local/share/jupyter/scheduler.sqlite')
cur = conn.cursor()

Simply change the number of days to whatever you want.

timestamp_threshold = int((datetime.datetime.now() - datetime.timedelta(days=60)).timestamp() * 1000)

cur.execute(f"DELETE FROM jobs WHERE start_time < {timestamp_threshold}")
conn.commit()
conn.close()

This removes the entries from the database, and so declutters the user interface. This does not remove the directories and files created along the way in .local/share/jupyter/scheduler_staging_area.

I would like an approach that also removes the accumulated files.

[vsr2158]

+1 the disk cleanup and UI cleanup options are needed.

[vsr2158]

In addition to cleaning up the sqlite database
add a second line to delete the older files and directories to free up disk like this

! find /home/ubuntu/.local/share/jupyter/scheduler_staging_area/ -maxdepth 1 -type d -mmin +1440 -exec rm -r {} \;

Until this feature is implemented, I have created a very short notebook that runs once a day and removed jobs older than N days from the included sqlite database. FYI if you are interested @quentindurpoix.

I do something similar for failed notebooks jobs and a service email account I have for system notices.

Here is the code:

import sqlite3
import datetime

conn = sqlite3.connect('/home/ritz/.local/share/jupyter/scheduler.sqlite')
cur = conn.cursor()

Simply change the number of days to whatever you want.

timestamp_threshold = int((datetime.datetime.now() - datetime.timedelta(days=60)).timestamp() * 1000)

cur.execute(f"DELETE FROM jobs WHERE start_time < {timestamp_threshold}")
conn.commit()
conn.close()