microdata-job-executor

Executes work for a datastore by polling a job-service for available jobs. Uses subprocesses for asynchronous workloads.

Contribute

Set up

To work on this repository you need to install uv. After cloning the repo, install and activate an environment with:

uv venv && uv sync && source .venv/bin/activate 

Pre-commit

There are currently 3 active rules: Ruff-format, Ruff-lint and sync lock file. Install pre-commit

pip install pre-commit

If you've made changes to the pre-commit-config.yaml or its a new project install the hooks with:

pre-commit install

Now it should run when you do:

git commit

By default it only runs against changed files. To force the hooks to run against all files:

pre-commit run --all-files

if you dont have it installed on your system you can use: (but then it wont run when you use the git-cli)

uv run pre-commit

Read more about pre-commit

Running tests

Open terminal and go to root directory of the project and run:

uv run pytest

Build docker image

docker buildx build --tag job-executor:test-local .

Built with

• Uv - Python dependency and package management
• PyArrow - Apache Arrow
• Pandas - Data analysis and manipulation
• microdata-tools - dataset packaging & validation