The lab gets updated regularly, so do sync your fork with the upstream from time to time.
Today's tasks will be about Data engineering. Same project – Learning Management Service.
The system has an API and front-end in place. Now the team needs to populate the database with real data from an external source, build analytics on top of it, and visualize the results.
A senior data engineer explains your next assignment:
- Build an ETL (extract-transform-load) pipeline that fetches data from the autochecker API and loads it into the database.
- Implement analytics endpoints that aggregate the data for dashboards.
- Add dashboard charts to the front-end to visualize the analytics.
Important
Communicate through issues and PRs and deliver a working deployment.
Read the tasks, do the setup properly, work with an agent to help you:
What do we need to do in Task x? Explain, I want to maximize learning.
Why is this important? What exactly do we need to do?
You need an agent that has access to the whole repo to work effectively.
By the end of this lab, you should be able to:
- Build an ETL pipeline that fetches data from an external API.
- Handle pagination, incremental sync, and idempotent upserts.
- Write SQL aggregation queries (GROUP BY, COUNT, AVG, CASE WHEN).
- Implement REST API endpoints that return computed analytics.
- Use pre-written tests to validate your implementation.
- Integrate Chart.js into a React front-end for data visualization.
- Use an AI coding agent for ETL and front-end development.
In simple words, you should be able to say:
- I built a pipeline that fetches data from an external API and loads it into the database!
- I implemented analytics endpoints and made all tests pass!
- I added charts to the front-end to visualize the analytics!
- Complete the lab setup
Note: If this is the first lab you are attempting in this course, you need to do the full version of the setup