Skip to content

Conversation

@nmillardint
Copy link

Summary

Enhanced error injection across OTEL services and added MySQL integration to payment service for transaction logging and database error simulation.

Changes

  • Enable 8 feature flags for sporadic errors (payment 25%, database 15%, ad service, cart, kafka, images)
  • Add MySQL integration to payment service with connection pooling
  • Add payment_transactions table for audit logging
  • Add GitHub Actions workflow for payment service builds
  • Fix gRPC Long type conversion for MySQL compatibility

Testing

✅ Successfully tested with 5 payment transactions
✅ All transactions stored correctly in MySQL database
✅ Database error injection working (15% rate)

sdonnell-int and others added 30 commits June 5, 2025 18:30
feat(cf): add cf quote service demo
feat(quote-cf): enable Docker image push to GitHub Container Registry
chore(arm): add FR arm/amd supported images
… comprehensive endpoints

- Add MySQL database container with health checks and initialization
- Implement complete quote generation with customer and service data
- Add new CF endpoints: email, update, remove old quotes, and reporting
- Enhance load testing with ColdFusion-specific user scenarios
- Configure FusionReactor APM for enhanced monitoring
- Add proper error handling and logging throughout CF application

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
sdonnell-int and others added 26 commits June 30, 2025 16:04
chore(quote): increase jvm memory
chore(quote): add health route
chore(quote): fix getquote.cfm shipping
chore(quote): fix getquote.cfm shipping
…ations

### Major Changes:
- **Database Migration**: Complete migration from SQLite to MySQL
  - Updated all ColdFusion code to use MySQL-compatible syntax
  - Converted SQLite-specific functions (RANDOM() → RAND(), DATE() → CURDATE(), etc.)
  - Fixed string concatenation (|| → CONCAT())
  - Updated CAST functions (AS TEXT → AS CHAR)

- **Performance Testing**: Enhanced slow query simulation
  - Replaced problematic recursive CTEs with mathematical operations
  - Added 5% slow query path (~15 seconds) for observability testing
  - 95% fast query path maintains ~0.16 second response times

- **Container Optimization**:
  - Removed hardcoded JVM heap limits for Kubernetes compatibility
  - Added no-cache build option for local development
  - Auto-scaling heap based on container resource limits

- **Database Configuration**:
  - Added MySQL JDBC driver and connection configuration
  - Updated datasource settings in Application.cfc
  - Cleaned up SQL files (removed SQLite artifacts)

### Files Modified:
- Application.cfc: MySQL datasource configuration
- *.cfm: All endpoints updated for MySQL compatibility
- Dockerfile: JVM auto-scaling heap configuration
- docker-compose.yml: Removed build caching for development
- SQL cleanup: Kept only init_quotes.sql for MySQL

### Testing Results:
- All endpoints working: health, debug, getquote, emailquote, updatequote, removeoldquotes, report
- Performance testing: 4% slow queries averaging 15+ seconds
- Fast queries: 96% averaging 0.16 seconds
- Ready for Kubernetes deployment with auto-scaling JVM

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
feat(quote-cf): migrate from SQLite to MySQL with performance optimiz…
- Migrate from SQLite to MySQL database system
- Add comprehensive sample data (50 customers, 50 services, 30 quotes, 50 quote items)
- Implement database initialization in Application.cfc with proper error handling
- Update Docker configuration to use MySQL container
- Remove SQLite dependencies and database files
- Add health checks and connection validation

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
feat(quote-cf): complete MySQL migration with enhanced schema and data
- Optimize slow query execution from 45+ seconds to 4-6 seconds
- Add 30-second MySQL connection/socket timeouts to prevent hangs
- Reduce Lucee request timeout from 10 minutes to 5 minutes
- Persist FusionReactor configuration with 4-second slow thresholds
- Add comprehensive test script for performance validation
- Maintain 5% slow query probability for realistic observability demos
- Ensure system stability under 200+ consecutive requests

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
feat(quote-cf): optimize removeoldquotes performance and add monitoring
- Create mysql_admin datasource with 2-minute timeouts for admin operations
- Update onApplicationStart() to use mysql_admin for DROP/CREATE/INSERT operations
- Regular mysql datasource retains 30-second timeouts for user queries
- Prevents application startup timeouts on slow demo clusters
- Ensures reliable database initialization while maintaining query performance

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
…eouts

fix(quote-cf): add separate admin datasource for database initialization
- Move database initialization from Application.cfc to MySQL Docker initialization
- Mount SQL initialization script to /docker-entrypoint-initdb.d in docker-compose
- Update init_quotes.sql with 10x larger dataset (50 customers, 50 services, 30 quotes, 50 quote_items)
- Remove persistent MySQL data volume to ensure fresh data on each startup
- Clean up Application.cfc by removing admin datasource and database initialization logic
- Reduce quote service load in load generator to improve demo performance
- Fix demo cluster timeout issues by eliminating database initialization delays

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
feat(quote-cf): migrate database initialization from ColdFusion to MySQL
- Double wait times for WebsiteUser (1-10s → 2-20s) and CFStorefrontUser (15-30s → 30-60s) to halve throughput
- Reduce emailquote.cfm error rate from ~15% to ~7.5% by changing error condition from dur > 620 to dur > 635

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
feat(load-generator): reduce load throughput and error rates
…rvice

This commit adds comprehensive sporadic error behavior across OTEL services
and integrates MySQL database with the payment service for transaction logging
and database error simulation.

Changes to Feature Flags (demo.flagd.json):
- Enable productCatalogFailure: "on" - Product catalog errors
- Enable imageSlowLoad: "5sec" - Slow loading images in frontend
- Enable adManualGc: "on" - GC pauses in ad service
- Enable adHighCpu: "on" - CPU spikes in ad service
- Enable adFailure: "on" - 10% ad service failures
- Enable kafkaQueueProblems: "on" - Message queue lag
- Enable cartFailure: "on" - Cart service errors
- Increase paymentFailure: 10% → 25% - More frequent payment errors
- Add paymentDatabaseError: "15%" - New database error injection flag

Payment Service MySQL Integration:
- Add mysql2 dependency for database connectivity
- Create database.js module with connection pooling
- Add payment_transactions table schema for transaction logging
- Update charge.js to store all payment attempts in MySQL
- Implement sporadic database error injection (timeouts, deadlocks, etc.)
- Add MySQL environment variables to docker-compose.yml
- Ensure payment service depends on MySQL health check

Error Types Simulated:
- Connection timeouts (ETIMEDOUT)
- Connection refused (ECONNREFUSED)
- Lock wait timeouts (ER_LOCK_WAIT_TIMEOUT)
- Deadlock detection (ER_LOCK_DEADLOCK)

Observability Benefits:
- Database operations visible in OTEL traces
- Realistic error scenarios for demo purposes
- Transaction audit logging in MySQL
- Multi-layered chaos: service errors, performance issues, resource contention, messaging delays

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
- Add package-lock.json with mysql2 dependencies for payment service
- Fix docker-compose.yml invalid no_cache option in quote service build

These changes are required for the payment service MySQL integration to build correctly.
- Add type conversion for units and nanos from gRPC Long to JavaScript Number
- Fixes database insertion error when storing payment transactions
- Handles both Long objects and primitive number types

ci: add GitHub Actions workflow for payment service builds

- Add build-payment.yml workflow for automated Docker image builds
- Triggers on changes to src/payment/** directory
- Follows same pattern as other service workflows (ad, recommendation, etc.)

Tested: Successfully processed 5 test payments and verified transactions
stored correctly in MySQL database.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@nmillardint nmillardint requested a review from a team as a code owner November 4, 2025 12:15
@github-actions github-actions bot added docs-update-required Requires documentation update helm-update-required Requires an update to the Helm chart when released labels Nov 4, 2025
@linux-foundation-easycla
Copy link

CLA Not Signed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-update-required Requires documentation update helm-update-required Requires an update to the Helm chart when released

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants