Merge pull request #10 from maxh33/feat/rabbitMQ

Feat/rabbit mq

Author: maxh33

.github/workflows/deploy.yml

@@ -4,7 +4,7 @@ on:
   push:
     branches:
       - main
-      - chore/simplify-project-architecture
+      - feat/rabbitMQ
 
 env:
   REGISTRY: ghcr.io
@@ -39,6 +39,18 @@ jobs:
           MONGODB_URI: ${{ secrets.MONGODB_URI }}
           NODE_ENV: production
 
+      - name: Set RabbitMQ Credentials
+        run: |
+          # Use localhost for GitHub Actions since we're running verification locally
+          echo "RABBITMQ_PASSWORD=${{ secrets.RABBITMQ_PASSWORD }}" >> $GITHUB_ENV
+          echo "RABBITMQ_URL=amqp://pulse_flow_user:${{ secrets.RABBITMQ_PASSWORD }}@localhost:5672" >> $GITHUB_ENV
+
+      - name: Verify RabbitMQ Connection
+        run: npm run verify:rabbitmq
+        env:
+          RABBITMQ_URL: ${{ secrets.RABBITMQ_URL }}
+          NODE_ENV: production
+
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
 
@@ -59,6 +71,8 @@ jobs:
           GRAFANA_PROMETHEUS_URL=${{ secrets.GRAFANA_PROMETHEUS_URL }}
           GRAFANA_USERNAME=${{ secrets.GRAFANA_USERNAME }}
           GRAFANA_API_KEY=${{ secrets.GRAFANA_API_KEY }}
+          RABBITMQ_URL=${{ secrets.RABBITMQ_URL }}
+          METRICS_PUSH_INTERVAL=60000
           EOL
 
       - name: Build and push Docker image
@@ -92,6 +106,7 @@ jobs:
           GRAFANA_PROMETHEUS_URL: ${{ secrets.GRAFANA_PROMETHEUS_URL }}
           GRAFANA_USERNAME: ${{ secrets.GRAFANA_USERNAME }}
           GRAFANA_API_KEY: ${{ secrets.GRAFANA_API_KEY }}
+          RABBITMQ_URL: ${{ secrets.RABBITMQ_URL }}
           GITHUB_REPOSITORY: ${{ github.repository }}
 
       - name: Wait for app to start

.github/workflows/keep-alive.yml

@@ -0,0 +1,27 @@
+name: Keep Alive
+
+on:
+  schedule:
+    - cron: '*/10 * * * *'  # Run every 10 minutes
+  workflow_dispatch:  # Manual trigger
+
+jobs:
+  ping:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Ping Application
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const axios = require('axios');
+            try {
+              const response = await axios.get('https://pulse-flow.onrender.com/ping');
+              console.log('Ping successful:', response.status);
+              if (response.status !== 200) {
+                throw new Error(`Unexpected status: ${response.status}`);
+              }
+            } catch (error) {
+              console.error('Ping failed:', error.message);
+              // Don't fail the workflow to avoid notifications
+              // but log the error for monitoring
+            } 

.gitignore

@@ -58,3 +58,13 @@ docs/
 docs/drafts/
 *.draft.md
 
+# Metrics and monitoring
+*.metrics
+*.prom
+
+# TypeScript cache
+*.tsbuildinfo
+
+# Render deployment
+.render/
+

README.md

@@ -1,7 +1,8 @@
-# [Still Under Development] PulseFlow - Real-Time Social Media Analytics Platform
+# [Under Development] PulseFlow - Real-Time Social Media Analytics Platform
 
 ## Overview
-Backend for real-time data processing and visualization platform built with enterprise-grade technologies. This project demonstrates advanced data simulation, monitoring, and analytics capabilities using modern DevOps practices.
+
+PulseFlow is a backend platform designed for real-time data processing and visualization, leveraging enterprise-grade technologies. This project showcases advanced capabilities in data simulation, monitoring, and analytics, all built on modern DevOps practices
 
 ## Features
 - Real-time social media data processing
@@ -17,43 +18,47 @@ Backend for real-time data processing and visualization platform built with ente
 - **Database**: MongoDB with Mongoose ODM
 - **Containerization**: Docker & Docker Compose
 
-### Monitoring & Visualization
-- **Metrics**: Prometheus
-- **Dashboards**: Grafana
-- **Health Checks**: Custom endpoints with prometheus-client
+## Architecture Components
+- Microservices-based Design: Modular architecture for scalability and maintainability.
+- MongoDB Data Storage: Efficient and scalable data management.
+- Real-time Metrics Collection: Continuous monitoring of system performance.
+- Custom Dashboards: Visual insights through Grafana and MongoDB Charts.
+
+### Message Queue Integration:
+- RabbitMQ with CloudAMQP: Utilizes a managed RabbitMQ service for distributed message processing.
+- Configuration: Secure, encrypted connections with elastic scaling.
+- Key Features: Reliable message queuing, automatic message routing, and error handling with dead-letter queues.
+- Message Flow: Synthetic tweet data is generated, published to RabbitMQ, processed asynchronously, and stored in MongoDB.
 
 ### CI/CD & DevOps
 - **Continuous Integration**: GitHub Actions
 - **Container Registry**: GitHub Container Registry
 - **Container Orchestration**: Docker Compose
 
-### Architecture Components
-- Microservices-based design
-- MongoDB data storage
-- Real-time metrics collection
-- Custom dashboards
+## Monitoring Setup
 
-### Monitoring Setup
-The application exports metrics in Prometheus format and pushes them to Grafana Cloud. A local monitoring stack is also available through Docker Compose.
 
-#### Available Metrics
-- Tweet processing rate and duration
-- Sentiment analysis distribution
-- Platform usage statistics
-- Error rates and system health
-- API response times
+- **Metrics**: Prometheus
+- **Dashboards**: Grafana
+- **Health Checks**: Custom endpoints with prometheus-client
+- **MongoDB Charts**: 
+
+The application exports metrics in Prometheus format and pushes them to Grafana Cloud. A local monitoring stack is also available through Docker Compose.
 
 # Dashboards
 
+
+**![>> **SOON** ~~Project Demo Gif~~ <<](public/demo.gif)**
+
 ### Grafana Performance Dashboard
 ![Grafana Dashboard](public/GrafanaEx.png)
 
-**[>> View LIVE Grafana Dashboard HERE <<](https://maxh33.grafana.net/public-dashboards/2f5dd656ee264fd2ac6f13f1aa1b4004)**
+**[>> **SOON** ~~View LIVE Grafana Dashboard HERE~~ <<](https://maxh33.grafana.net/public-dashboards/2f5dd656ee264fd2ac6f13f1aa1b4004)**
 
 ## MongoDB Charts Dashboard
 ![MongoDB Charts Dashboard](public/MainMetrics.png)
 
-**[>> View LIVE MongoDB Charts Dashboard HERE <<](https://charts.mongodb.com/charts-project-0-tmkdyjw/public/dashboards/6798e048-db1e-4c24-85a6-e942bec5d15f)**
+**[>> **SOON** ~~View LIVE MongoDB Charts Dashboard HERE~~ <<](https://charts.mongodb.com/charts-project-0-tmkdyjw/public/dashboards/6798e048-db1e-4c24-85a6-e942bec5d15f)**
 
 
 ## Data Generation Methodology
@@ -81,24 +86,14 @@ The application exports metrics in Prometheus format and pushes them to Grafana
 ### Disclaimer
 🚨 **Note**: All data is artificially generated and does not represent real social media interactions.
 
-## Deployment Infrastructure
-
-### Platform
-- **Hosting**: Render.com
-- **Deployment Type**: Web Service
-- **Continuous Deployment**: Enabled
-
-### Render.com Configuration
-- Automatic GitHub repository synchronization
-- Node.js runtime environment
-- Scalable web service infrastructure
-- Built-in environment variable management
+## Deployment Infrastructure Workflow:
 
-#### Deployment Workflow
-1. Code pushed to GitHub
-2. Render.com detects changes
-3. Automatic build and deployment
-4. Zero-downtime updates
+1. **Data Generation:** Synthetic tweets created with Chance.js.
+2. **Message Queuing:** Data published to RabbitMQ, ensuring reliable processing.
+3. **Data Storage:** Tweets stored in MongoDB for efficient access.
+4. **CI/CD:** Automated tests and deployment via GitHub Actions.
+5. **Hosting:** Application deployed on Render for scalability.
+6. **Monitoring:** Metrics collected and visualized in Grafana.
 
 ## Project Status & Quality Checks
 

docker-compose.prod.yml

@@ -43,6 +43,7 @@ services:
       - NODE_ENV=production
       - PORT=3000
       - MONGODB_URI=${MONGODB_URI}
+      - RABBITMQ_URL=${RABBITMQ_URL}
     healthcheck:
       test: ["CMD", "curl", "-f", "http://localhost:3000/healthz"]
       interval: 30s