Job API (for data modeling) ·

REST API for scheduling and running jobs in the Development Framework

Usage

This project delivers a container image, that is meant to be included in any deployment of Development Framework that need jobs.

One example would be a docker-compose file that include all the services in your setup.
The dm-job-api service could then be defined like this;

  job-api:
    build:
      context: .
    image: datamodelingtool.azurecr.io/dm-job
    restart: unless-stopped
    environment:
      SCHEDULER_ENVS_TO_EXPORT: "PUBLIC_DMSS_API,SIMA_LICENSE"
      SCHEDULER_REDIS_HOST: job-store
      SCHEDULER_REDIS_PORT: 6379
      SCHEDULER_REDIS_SSL: "false"
      DMSS_API: http://dmss:5000
      AZURE_JOB_SUBSCRIPTION: 14d57366-b2ae-4da8-8b75-e273c6fdabe2
      AZURE_JOB_RESOURCE_GROUP: dmt-test-containers
      AZURE_SP_SECRET: ${AZURE_SP_SECRET}
      AZURE_JOB_TENANT_ID: 3aa4a235-b6e2-48d5-9195-7fcf05b459b0
      AZURE_JOB_CLIENT_ID: 97a6b5bd-63fb-42c6-bb75-7e5de2394ba0
    depends_on:
      - job-store
    ports:
      - "5000:5000"
    volumes:
      - ./job_plugins/:/code/src/job_handler_plugins/
  
  job-store:
    image: redis:6.2.5-alpine

NOTE: The dm-job-api requires a redis database to store state.

NOTE: Any Job Handler Plugins must be mounted into the directory /code/src/job_handler_plugins

Job Handler Plugins

The job scheduler relies on pluggable JobHandler() implementations to execute different kind of jobs.

A few are included by default;

• local_container_instances
• azure_container_instances
• omnia_classic_azure_container_instances
• reverse_description

You can supply your own JobHandlers by volume mounting the python modules into /code/src/job_handler_plugins.
These modules must be a folder with a _init_.py-file with a JobHandler-class, and a global variable _SUPPORTED_TYPE. This can be done in the docker-compose.override.yml file (under "volumes").

Example;

# /code/src/job_handler_plugins/myCustomJobhandler/__init__.py
from ./myFunctions import execute_job
_SUPPORTED_TYPE = "SomeDataSource/SomePackage/AJobBlueprint"

class JobHandler(JobHandlerInterface):
    def __init__(self, job, data_source: str):
        super().__init__(job, data_source)

    def start(self) -> str:
        raise execute_job(self.job)

    def remove(self) -> str:
        """Terminate and cleanup all job related resources"""
        raise NotImplementedError

    def progress(self) -> Tuple[JobStatus, None | list[str] | str, None | float]:
        """Poll progress from the job instance"""
        raise NotImplementedError

    def result(self) -> Tuple[str, bytes]:
        """Returns a string for free text and the result of the job as a bytearray"""
        raise NotImplementedError

    def setup_service(self, service_id: str) -> str:
        """Start a persistent service"""
        raise NotImplementedError

    def teardown_service(self, service_id: str) -> str:
        """Teardown and cleanup a persistent service"""
        raise NotImplementedError

Python packages

This project uses Poetry for its Python package management.

• If you like Poetry to create venv in the project directory, configure it like so;
  poetry config settings.virtualenvs.in-project true
• To create a virtual environment run poetry install
• To add packages run poetry add myPackage (Remember to rebuild the Docker image)

Using the DMSS python package

The python API uses the DMSS python package available on PyPi. If you want to use a local version of this package, you can:

1. Build the local python package from the cloned DMSS repo, by running the generate-python-package.sh script.
2. Update the DMT docker-compose.override.yml file by adding / uncomment the volume mount: ../data-modelling-storage-service/gen/dmss_api:/dmss_api

Troubleshooting

Intellij Docker issue with pydevd

• Try open idea from terminal.

Mac: /Applications/IntelliJ\ IDEA.app/Contents/MacOS/idea

python: can't open file '/opt/.pycharm_helpers/pydev/pydevd.py': [Errno 2] No such file or directory

https://chkr.at/wordpress/?p=227

👍 Contributing

If you would like to contribute, please read our Contribution guide.