Skip to content

Conversation

@CBroz1
Copy link
Member

@CBroz1 CBroz1 commented Feb 12, 2025

Description

This PR makes a series of changes to update our maintenance tools, revising the cron job process:

  • nightly_cleanup -> cleanup: no need to specify inaccurate frequency
  • Revise cleanup.py to clean schema externals and delete temp files more than 1 week old.
  • Add run_jobs.sh script to be set as cron job and corresponding README
  • Modify alter_tables script to run on all tables.
  • Sort SHARED_MODULES for readability and remove unused prefix
  • Modify IntervalList.insert calls to never skip duplicates

Checklist:

  • No This PR should be accompanied by a release: (yes/no/unsure)
  • N/a If release, I have updated the CITATION.cff
  • No This PR makes edits to table definitions: (yes/no)
  • N/a If table edits, I have included an alter snippet for release notes.
  • N/a If this PR makes changes to position, I ran the relevant tests locally.
  • I have updated the CHANGELOG.md with PR number and description.
  • Yes I have added/edited docs/notebooks to reflect the changes

@CBroz1 CBroz1 marked this pull request as ready for review February 12, 2025 22:52
Copy link
Collaborator

@edeno edeno left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one minor comment about folder name.

One thing that might be useful is for the script to warn us (by email perhaps?) if it fails to run. These scripts often are set and forget but if they stop working we want to know.

`dj_local_conf.json` and filling in the necessary information.
3. Set up a cron job to run `run_jobs.sh` at the desired interval by running
`crontab -e` and adding the script.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to call this franklab scripts or something more general that other labs can repurpose?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to rename, maybe maintenance_scripts? Is it worth moving these .py files into the package itself and then maybe putting the .sh in the docs, to cut down on the root level folders?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That works for me.

@edeno
Copy link
Collaborator

edeno commented Feb 12, 2025

Hm also not sure why the tests are suddenly failing.

@CBroz1
Copy link
Member Author

CBroz1 commented Feb 13, 2025

Hm also not sure why the tests are suddenly failing.

It looks like this is happening when replace=True for exact duplicates in IntervalList. Might call for a more lightweight version of #943. I'll see what I can do

@edeno edeno merged commit 1f5834b into LorenFrankLab:master Feb 14, 2025
17 checks passed
@CBroz1 CBroz1 deleted the cron branch February 14, 2025 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants