Syllabus
- Week 1: Introduction & Prerequisites
- Week 2: Data ingestion
- Week 3: Data Warehouse
- Week 4: Analytics Engineering
- Week 5: Batch processing
- Week 6: Streaming
- Week 7, 8 & 9: Project
Note: NYC TLC changed the format of the data we use to parquet. But you can still access the csv files here.
- Course overview
- Introduction to GCP
- Docker and docker-compose
- Running Postgres locally with Docker
- Setting up infrastructure on GCP with Terraform
- Preparing the environment for the course
- Homework
- Data Lake
- Workflow orchestration
- Setting up Airflow locally
- Ingesting data to GCP with Airflow
- Ingesting data to local Postgres with Airflow
- Moving data from AWS to GCP (Transfer service)
- Homework
- Data Warehouse
- BigQuery
- Partitioning and clustering
- BigQuery best practices
- Internals of BigQuery
- Integrating BigQuery with Airflow
- BigQuery Machine Learning
- Basics of analytics engineering
- dbt (data build tool)
- BigQuery and dbt
- Postgres and dbt
- dbt models
- Testing and documenting
- Deployment to the cloud and locally
- Visualizing the data with google data studio and metabase
- Batch processing
- What is Spark
- Spark Dataframes
- Spark SQL
- Internals: GroupBy and joins
- Introduction to Kafka
- Schemas (avro)
- Kafka Streams
- Kafka Connect and KSQL
Putting everything we learned to practice
- Week 7 and 8: working on your project
- Week 9: reviewing your peers
- Google Cloud Platform (GCP): Cloud-based auto-scaling platform by Google
- Google Cloud Storage (GCS): Data Lake
- BigQuery: Data Warehouse
- Terraform: Infrastructure-as-Code (IaC)
- Docker: Containerization
- SQL: Data Analysis & Exploration
- Airflow: Pipeline Orchestration
- dbt: Data Transformation
- Spark: Distributed Processing
- Kafka: Streaming
To get the most out of this course, you should feel comfortable with coding and command line and know the basics of SQL. Prior experience with Python will be helpful, but you can pick Python relatively fast if you have experience with other programming languages.
Prior experience with data engineering is not required.