Gym Summary Stream Processing in Lakehouse

Data engineering pipeline for gym data processing leveraging Pyspark, Databricks, and Azure ADLS.

Data Flow Diagram

Developed a robust data engineering pipeline for gym data processing leveraging Pyspark, Databricks, and Azure ADLS.
Orchestrated ingestion from diverse sources including CSV, JSON files, and Kafka topics for comprehensive data acquisition.
Implemented efficient data processing workflows utilizing Databricks Unity Catalog for streamlined data management and accessibility.
Implemented Medallion architecture to strategically structure data within the lakehouse environment.

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
.ipynb_checkpoints		.ipynb_checkpoints
01-config.ipynb		01-config.ipynb
02-setup.ipynb		02-setup.ipynb
03-history-loader.ipynb		03-history-loader.ipynb
04-bronze.ipynb		04-bronze.ipynb
05-silver.ipynb		05-silver.ipynb
06-gold.ipynb		06-gold.ipynb
07-stream-test.ipynb		07-stream-test.ipynb
08-Producer.ipynb		08-Producer.ipynb
README.md		README.md