Skip to content

About

Take home answer for fetch take home

Resources

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

 

History

11 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 

Repository files navigation

Real-Time Data Processing Pipeline using Kafka and Python

Project Overview

This project demonstrates a real-time data processing pipeline using Kafka and Python. The pipeline ingests streaming data from a Kafka topic, processes the data to extract insights, and sends the processed data to another Kafka topic. The setup uses kafka-python to implement the Kafka Consumer and Producer.

Instructions--Execution Code

docker-compose up -d #run the docker compose file to prepare the environment
pip install kafka-python #install the necessary kafka-python library
python kafka_fetch.py #run the script to make the consumer to get information and process data to send to producer, generating insights(Please type command+C when you want to stop the messages receiving)

Code Overview

  1. Kafka Consumer The KafkaConsumer subscribes to the user-login topic to receive messages. The messages are deserialized from JSON format for processing.
  2. Data Processing Information Extraction: The script extracts relevant fields such as user_id, app_version, device_type, ip, locale, device_id, and timestamp. Timestamp Standardization: The Unix timestamp is converted to a human-readable format. Device Type and Locale Count: The script tracks the number of logins per device type and locale to gain insights into usage patterns.
  3. Kafka Producer The KafkaProducer sends the transformed data to the processed-user-login topic. The data is serialized into JSON format before being sent.
  4. Insights Extraction The script identifies the most popular device type and the most frequently appearing locale from the processed messages.

Data Flow

Data Ingestion: The Kafka consumer subscribes to the user-login topic to receive streaming data. Data Processing: The data is processed to extract meaningful information, such as standardizing timestamps and counting occurrences of device types and locales. Data Publishing: The processed data is published to the processed-user-login topic using a Kafka producer. Insights Generating: The script identifies and logs the most popular device type and the most frequently appearing locale.

Design Choices

Error Handling and Logging: Integrated try-except blocks to handle errors gracefully during message processing. This ensures that the consumer does not crash due to unexpected issues, and errors are logged for easier debugging. Insight Generation and Efficient Data Processing: Used dictionaries (defaultdict) to efficiently count device types and locales, and updated insights every 100 messages. This batch-like approach balances real-time processing with performance optimization, reducing the overhead of constant updates. JSON Serialization: Data is serialized and deserialized in JSON format to ensure compatibility and ease of human readability.

How to Ensure Efficiency

Serialization: JSON serialization is efficient and lightweight, ensuring quick transmission and processing of data. Real-time processing: Get one message, transform and process then send to a new topic, ensuring real-time processed data quality Controlled Message Handling: The script shows insight per 100 to update the insight about login information by locale and device

How to Ensure Scalability

Explicitly Use Consumer Groups: a group_id is assigned to your Kafka consumer. This allows Kafka to distribute messages across multiple consumer instances efficiently to enhance scalability Optimized the Producer parameter settings: Setting parameters to help reduce the number of network requests and improve throughput, making the producer more efficient in high-load scenarios. Partitioning: My Kafka topics can be partitioned to scale horizontally, distributing the load across multiple consumers for parallel processing.

How to Ensure Fault Tolerance

Error Handling with Try-Except Blocks: If an error occurs (e.g., a malformed message or a processing issue), the script logs the error and continues processing the next message, rather than crashing the entire consumer. Graceful Shutdown: The code handles a KeyboardInterrupt (e.g., when you manually stop the script) and ensures the Kafka producer is closed properly.

Future Improvements

Enhanced Error Handling: Implement a retry mechanism for messages that fail to process due to transient issues (like network problems or temporary Kafka unavailability). Persistent Data Storage: Store processed data in a database or data lake for long-term analysis.

Answers to Additional questions

1. How to Deploy This Application in Production

  • Containerization: Use Docker to containerize the application for easy deployment.
  • Orchestration: Use Kubernetes to manage and scale my containers, ensuring my app runs smoothly and can handle more traffic if needed.
  • Managed Kafka: Consider using a managed Kafka service like AWS MSK or Confluent Cloud to avoid managing Kafka infrastructure manually.
  • CI/CD: Set up a CI/CD pipeline (e.g., GitHub Actions) to automate testing and deployment.

2. Components to Add for Production Readiness

  • Monitoring & Logging: Use tools like Prometheus and Grafana for monitoring. Set up centralized logging with tools like ELK Stack (Elasticsearch, Logstash, Kibana).
  • Error Handling: Set up alerts for issues like consumer lag or high resource usage.
  • Data Storage: Store critical processed data in a database or a data lake (e.g., AWS S3) for long-term storage.
  • Security: Secure communication with TLS and manage access with Kafka ACLs. Use a secrets manager (like AWS Secrets Manager) to handle sensitive information.

3. How to Scale with Growing Data

  • Kafka Partitioning: Increase the number of partitions in Kafka topics to allow more consumers to process data in parallel.
  • Optimize Kafka Consumer Settings: Use parameters like max_poll_records to fetch a batch of messages in a single poll, reducing the frequency of network requests and improving performance
  • Enable Auto-Scaling: Use Kubernetes Horizontal Pod Autoscaler (HPA) to automatically scale the number of consumer instances based on resource usage or custom metrics (e.g., CPU, memory, or consumer lag).
  • Leverage Kafka's Built-In Fault Tolerance: ensure Kafka topics have a replication factor greater than 1 to handle broker failures without data loss.

About

Take home answer for fetch take home

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages