okdata-pipeline

Collection of pipeline components for Origo Dataplattform.

Components

• converters
  • csv
  • json
  • xls
• lambda_invoker
• validators
  • csv
  • json
• writers
  • s3

Setup

make init

Test

Tests are run using tox:

make test

For tests and linting we use pytest, flake8, and black.

Deploy

Example GitHub Actions for deploying to dev and prod on push to main is included in .github/workflows.

You can also deploy from a local machine to dev with:

make deploy

Or to prod with:

make deploy-prod