Realtime data processing and management
The following repositories contain the services for the data pipeline shown in diagram above.
- mittaridatapumppu: django app for storing device metadata into postgres db
- mittaridatapumppu-endpoint: a fastapi app that receives sensor data in POST requests and produces them to kafka
- mittaridatapumppu-parser: python based microservice that consumer streaming data and converts them from raw format to something else as defined in corresponding parser modules obtained from device metadata
- mittaridatapumppu-persister:
Go To test set up
pip install pre-commit
pre-commit install
pip-compile
pip-compile --extra dev --output-file requirements-dev.txt
pip-sync
pre-commit run --all-files