Airflow Learning Projects 🚀

This repository documents my learning journey with Apache Airflow, along with two practical projects:

• ✅ A Random Quote Generator DAG in Airflow
• ✅ A Flask-based Web Application

It includes all the source code, solutions to errors encountered during setup and execution, and a learning documentation for future reference.

---

🧠 Project 1: Random Quote Generator using Airflow

📌 Description:

This project demonstrates how to use Apache Airflow for automated workflow scheduling by building a DAG that generates a random motivational quote and stores it in a .csv file every time it's triggered.

🛠️ Features:

• Uses random and datetime modules.
• Quotes are stored with timestamps in a CSV file.
• Manual triggering through the Airflow UI.

📂 File Structure:

dags/ └── random_quote_dag.py output/ └── quotes.csv

markdown Copy Edit

🧪 Technologies Used:

• Apache Airflow
• Python 3.11
• Docker (for containerized orchestration)
• Airflow CLI & Web UI

---

🌐 Project 2: Flask Web Application

📌 Description:

This is a basic Flask web app to demonstrate setting up a simple route with interactive HTML templates. It showcases how to:

• Handle HTTP GET and POST requests
• Connect backend logic with frontend using Jinja2 templates

📂 File Structure:

flask_app/ ├── app.py ├── templates/ │ └── index.html └── static/

yaml Copy Edit

🧪 Technologies Used:

• Flask
• Python
• HTML/CSS (basic)

---

🧩 Common Learnings & Errors Solved

🧱 Apache Airflow Errors:

❌ Error	💡 Solution
ps: command not found in Airflow container	Installed using apt-get update && apt-get install procps
Airflow 501 error on localhost	Fixed by mapping ports properly in docker-compose.yml and restarting the container
webserver.log not found	Ensured Airflow started correctly with airflow webserver and confirmed logs path

⚙️ Flask App Issues:

❌ Issue	💡 Solution
App not launching	Checked port binding and ensured Flask app was run using python app.py
Template not found	Corrected template folder path or renamed file to match Flask routing

---

📚 Learning Outcome

• Gained hands-on experience with Airflow DAGs and scheduling workflows.
• Learned how to deploy and test Flask apps.
• Explored Dockerized development environments.
• Understood the importance of logs, permissions, and CLI tools in debugging.

---

📂 Repository Structure

workspace/ ├── dags/ │ └── random_quote_dag.py ├── flask_app/ │ ├── app.py │ └── templates/ └── docs/ └── airflow_learning_documentation.pdf

yaml Copy Edit

---

✅ To Run

Flask App:

cd flask_app
python app.py
Airflow DAG (Docker):
bash
Copy
Edit
docker-compose up airflow-webserver