Pokémon Card Kafka Demo

This is a Go project demonstrating Kafka producers and consumers using Pokémon card data with advanced configuration management, error handling, and monitoring capabilities.

Features

• Configuration Management: Environment-based configuration with sensible defaults
• Error Handling: Robust error handling with retry mechanisms
• Graceful Shutdown: Proper signal handling and resource cleanup
• Concurrent Processing: Parallel card fetching and processing
• Monitoring: Message counting and error tracking
• Flexible Deployment: Support for different environments (dev, staging, prod)

Requirements

• Go 1.20+
• Docker (for Kafka/Zookeeper)

Quick Start

1. Setup Development Environment

# Start Kafka
make docker-up

# Setup project dependencies
make setup

# Run both producer and consumer
make run-dev

2. Manual Setup

# Start Kafka
docker-compose up -d

# Run producer
make run-producer

# Run consumer (in another terminal)
make run-consumer

Configuration

The application supports configuration through environment variables with sensible defaults.

Environment Variables

Variable	Default	Description
ENVIRONMENT	development	Environment (development, staging, production)
KAFKA_BROKERS	localhost:9092	Comma-separated Kafka broker addresses
KAFKA_TOPIC	pokemon.cards	Kafka topic name
KAFKA_GROUP_ID	pokemon-consumer-group	Consumer group ID
API_BASE_URL	https://api.pokemontcg.io/v2	Pokemon TCG API base URL
API_KEY	``	Optional API key for Pokemon TCG API
CARD_NAMES	charizard,blastoise,venusaur,pikachu	Comma-separated card names
WORKER_COUNT	4	Number of concurrent workers
LOG_LEVEL	info	Log level (debug, info, warn, error)

Configuration Files

• config.yaml.example - YAML configuration template
• env.example - Environment variables template

Copy these files and modify as needed:

cp env.example .env
# Edit .env with your settings

Usage Examples

Development Mode

make run-dev

Production Mode

ENVIRONMENT=production make run-prod

Custom Configuration

# Use different Kafka brokers
KAFKA_BROKERS=broker1:9092,broker2:9092 make run-producer

# Fetch different cards
CARD_NAMES=pikachu,mewtwo,lugia make run-producer

# Use API key
API_KEY=your-api-key make run-producer

Docker Commands

# Start Kafka
make docker-up

# Stop Kafka
make docker-down

# Full development setup
make dev-setup

Project Structure

pokemon-kafka/
├── pkg/
│   ├── config/          # Configuration management
│   └── model/           # Data models
├── producer/            # Kafka producer
├── consumer/            # Kafka consumer
├── docker-compose.yml   # Kafka setup
├── Makefile            # Build and run commands
└── README.md           # This file

Architecture

1. Producer: Fetches Pokemon card data from the API and publishes to Kafka
2. Consumer: Consumes messages from Kafka and processes them
3. Configuration: Centralized configuration management with environment support
4. Error Handling: Comprehensive error handling with retry mechanisms

Development

Building

make build

Testing

make test

Cleaning

make clean

Monitoring

The application provides built-in monitoring:

• Message processing counts
• Error tracking
• Graceful shutdown handling
• Connection status logging

Troubleshooting

Common Issues

1. Kafka Connection Failed

   • Ensure Kafka is running: make docker-up
   • Check broker addresses in configuration

2. API Rate Limiting

   • Add API key to configuration
   • Reduce worker count if needed

3. Consumer Not Receiving Messages

   • Check topic name configuration
   • Verify consumer group ID
   • Ensure producer is running

Logs

The application provides detailed logging:

• Configuration loading
• Connection status
• Message processing
• Error details

---

Enjoy streaming Pokémon card events with Kafka! 🎮⚡