Skip to content

EO-DataHub/eodhp-utils

Repository files navigation

eodhp-utils

eodhp-utils is a library used primarily by EODH components which communicate over Pulsar.

It contains:

  • Definitions of the format of some of the Pulsar messages suitable for use as Pulsar Schema definitions. Harvest pipeline messages pre-date this library and are not included. These are in eodhp_utils.pulsar.messages.
  • The 'Messagers' framework and the 'runner'. These handle the event loop and interaction with Pulsar, passing messages to caller-provided 'Messagers' and sending the results as new messages. Using this framework provides common CLI parameters, a multi-threading mechanism, etc. Read the docstrings in eodhp_utils.pulsar.messages, especially for Messager.
  • The egress classifier which classifies IP addresses as within-region, cross-region or internet (ie, not AWS IP addresses). This is in eodhp_utils.aws.egress_classifier.

Importing

This package may be used as follows:

  1. Install with pip
pip install "git+https://github.com/UKEODHP/eodhp-utils.git"

If a specific version is required:

pip install "git+https://github.com/UKEODHP/[email protected]"

For use in pyproject.toml:

dependencies = [
    "eodhp-utils @ git+https://github.com/EO-DataHub/[email protected]",
]
  1. Import
from eodhp_utils.pulsar.messages import generate_harvest_schema

print(generate_harvest_schema())

Install locally via makefile

make setup

This will create a virtual environment called venv, build requirements.txt and requirements-dev.txt from pyproject.toml if they're out of date, install the Python and Node dependencies and install pre-commit.

It's safe and fast to run make setup repeatedly as it will only update these things if they have changed.

After make setup you can run pre-commit to run pre-commit checks on staged changes and pre-commit run --all-files to run them on all files. This replicates the linter checks that run from GitHub actions.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Contributors 5