FarmAlytics

Real-time livestock monitoring and farm team coordination platform — IES Project G304, 2025/26

FarmAlytics is a comprehensive, enterprise-grade farm management solution designed to bridge the information gap in modern agriculture. By integrating real-time sensor data with a robust task management system, it enables proactive livestock care, efficient team coordination, and data-driven decision-making. The platform is built on a high-performance, multi-layered architecture designed to handle millions of data points while maintaining sub-second latency for critical alerts.

The Problem: The Agricultural "Information Gap"

Traditional livestock farming often suffers from fragmented communication and delayed responses. Farm owners are frequently "blind" to daily on-the-ground operations when away from the property, and veterinarians often lack a complete clinical history when called for emergencies. Field workers may miss critical events like animal escapes or sudden health drops because they rely on manual observation.

The Solution: A Connected Ecosystem

FarmAlytics replaces reactive management with a proactive, data-driven ecosystem. By mounting sensor collars on every animal, we create a continuous stream of physiological and positional data. This data is not just stored; it is processed in real-time to:

• Detect Anomalies: Automatically identify heart rate spikes, temperature drops, or geofence breaches.
• Coordinate Response: Route alerts directly to the right professional (Vet, Tech, or Worker) and track the resolution through a role-based task system.
• Provide Clinical Context: Give veterinarians a complete, minute-by-minute health history of their patients before they even arrive at the farm.
• Optimize Hardware: Ensure the monitoring infrastructure itself is healthy through automated diagnostics and inventory management.

Team

Role	Name	NMEC
Team Manager	Inês Batista	124877
Product Owner	Maria Quinteiro	124996
Architect	Luís Correia	125264
DevOps Master	Marcos Costa	125882

System Architecture Deep-Dive

The FarmAlytics architecture is engineered for high-throughput data ingestion, real-time processing, and reliable persistence. It follows a multi-layered approach to isolate concerns, ensure security, and provide horizontal scalability.

1. Data Ingestion Pipeline (The "Fast Path")

The ingestion pipeline is designed to handle continuous streams of physiological data from thousands of animals simultaneously.

• Python Sensors (Simulated): Since physical hardware was not available for the scale of this project, we developed a sophisticated Python simulation service. Each animal is represented as an asynchronous coroutine using asyncio, generating realistic BPM (heart rate), body temperature, and GPS coordinates. These sensors communicate via UDP packets to minimize latency and overhead, as losing a single packet is less critical than the delay introduced by TCP handshakes in a real-time monitoring context.
• NGINX Edge Layer: NGINX acts as the first line of defense. It proxies the incoming UDP stream to the ingestion worker and enforces an IP whitelist (restricted to the internal Docker subnet 172.30.0.0/16). This prevents external actors from injecting fraudulent sensor data into the system.
• C Ingestion Worker: To achieve maximum performance, the core ingestion logic is written in C. This service utilizes a thread pool of 20 worker threads to process incoming UDP packets concurrently. For every packet, the worker:
  1. Validates the binary structure and checks the animal_lts:{id} registration key in Redis.
  2. Compares BPM and temperature against species-specific thresholds.
  3. If an anomaly is detected, it calls the REST API's internal notification endpoints via HTTP.
  4. Updates the Redis cache with the latest payload for instant dashboard access.
  5. Publishes to the animal_updates Pub/Sub channel for live WebSocket delivery.
  6. Forwards the data to RabbitMQ for durable, asynchronous persistence.

2. Storage Strategy (Hybrid Persistence)

We employ a "polyglot persistence" strategy, using the best tool for each specific data type.

• Redis (In-Memory State): Redis serves as our shared fast-state layer. It stores the "Latest Known State" (LTS) for every animal, farm geofence polygons (for rapid boundary checks), alert deduplication markers (to prevent notification spam), and real-time Pub/Sub channels. Persistence is enabled via Append-Only File (AOF) to ensure the state survives container restarts.
• TimescaleDB (Time-Series Analytics): All historical sensor data is stored in TimescaleDB, a PostgreSQL extension optimized for time-series. We use Hypertables to automatically partition data by time and Continuous Aggregates (materialized views) to pre-calculate health metrics at minute, hour, and day granularities. This allows the frontend to render heatmaps and long-term charts without scanning billions of raw records.
• PostgreSQL + PostGIS (Relational & Spatial): The core business logic (Users, Farms, Tasks, Alerts) resides in a standard PostgreSQL instance. We use the PostGIS extension to handle complex spatial data, specifically the POLYGON type used to define farm boundaries and perform geofencing calculations.

Module Interactions & Data Flow

Sensor Data Flow (Real-time Pipeline)

The following diagram illustrates the lifecycle of a sensor reading, from the animal's collar to the user's screen.

1. Generation: The Python simulator sends a binary UDP packet.
2. Proxying: NGINX receives the packet and forwards it to the C Ingestion Worker.
3. Validation: The C worker verifies the animal's existence in Redis and checks for health anomalies.
4. Alerting: If an anomaly is found, an internal HTTP call triggers the alert system in the REST API.
5. Caching: The latest data is written to Redis, and a Pub/Sub message is sent.
6. Persistence: The packet is queued in RabbitMQ, consumed by a Python worker, and inserted into TimescaleDB.
7. Push: The REST API, subscribed to Redis Pub/Sub, pushes the update to the Frontend via a STOMP WebSocket connection.

User Interaction Flow (Business Logic)

This flow describes how users interact with the platform's transactional features, such as task management and dashboard viewing.

1. Authentication: Users log in via the REST API. Upon success, an HttpOnly JWT cookie is set, providing stateless, secure authentication for subsequent requests.
2. Dashboard Loading: The frontend requests a snapshot of all animals. The API fetches this directly from Redis (MGET) to ensure sub-100ms response times.
3. Task Creation: When a Farm Owner creates a task, the API validates the request and persists it to the PostgreSQL database.
4. Real-time Notification: If the task is an urgent alert triage, the assigned Field Worker receives a push notification via WebSocket instantly.

API Architecture & Endpoints

The backend is a robust Spring Boot 4.0.4 application, exposing over 80 endpoints across 16 functional groups.

Core Backend Features:

• Dual DataSource Configuration: The API manages two separate database connections (PostgreSQL and TimescaleDB) using distinct EntityManagerFactory and TransactionManager beans. This isolation ensures that high-volume sensor queries do not block transactional user operations.
• Security Framework: Built on Spring Security with a custom JwtAuthFilter. It supports role-based access control (RBAC), ensuring that a Field Worker cannot access a Veterinarian's clinical notes, and a Technician can only manage their own inventory.
• Real-time Integration: Uses Spring WebSocket with the STOMP protocol. The LocationPushService acts as a bridge, subscribing to Redis Pub/Sub and forwarding messages to the correct WebSocket "rooms" based on farm ID.
• Observability: Integrated with Spring Actuator and Micrometer, exposing JVM metrics, database pool status, and per-endpoint request statistics to the System Administrator dashboard.

User Roles & Interfaces

FarmAlytics provides five distinct, role-tailored interfaces:

1. Farm Owner: The "Command Center". Manages farm boundaries, registers animals, invites team members via QR codes, and oversees the entire operation through a real-time map and high-level dashboard.
2. Veterinarian: The "Clinical Specialist". Accesses detailed health histories, writes prescriptions, and manages clinical tasks. They have a dedicated view of an animal's physiological trends over time.
3. Field Worker: The "First Responder". Receives real-time alerts (e.g., "Animal Escaped" or "High Heart Rate") and performs on-the-ground triage. Their interface is optimized for mobile-first response.
4. Field Technician: The "Hardware Expert". Manages the inventory of collars and sensors. They receive diagnostic tasks when a sensor stops heartbeat (detected via Redis idle-time) and manage the repair lifecycle.
5. System Administrator: The "Platform Operator". Monitors the health of the entire infrastructure, from CPU usage to RabbitMQ queue depths and API response times.

Technology Stack

Component	Technology	Rationale
Backend (Core)	Java 21, Spring Boot 4.0.4	Robustness, ecosystem support, and rapid development.
Ingestion	C (Multi-threaded)	Maximum performance and low-level control over UDP processing.
Simulation	Python 3 (Asyncio)	Flexibility and ease of simulating thousands of concurrent agents.
Frontend	React 18, Vite, Tailwind CSS	Modern, responsive UI with efficient state management.
Visualization	D3.js, Leaflet	High-performance charting and interactive map clustering.
Time-Series DB	TimescaleDB	Specialized for massive sensor data with automatic aggregation.
Relational DB	PostgreSQL 14 + PostGIS	Industry standard for relational data and spatial calculations.
Cache / PubSub	Redis	Sub-millisecond state access and reliable real-time messaging.
Message Broker	RabbitMQ 4.2.5	Guaranteed delivery and decoupling of ingestion from persistence.
Infrastructure	Docker, Docker Compose	Consistent environments and simplified deployment.
Reverse Proxy	NGINX	Security, rate limiting, and unified entry point for UDP/HTTP.

Deployment & DevOps Deep-Dive

The FarmAlytics platform is designed for high availability and seamless orchestration using Docker and Docker Compose. Our infrastructure strategy focuses on security, dependency management, and automated recovery.

1. Network Architecture

All services operate within a dedicated bridge network called farm_net. We use a fixed subnet (172.30.0.0/16) to ensure that internal IP addresses are deterministic. This is critical for our NGINX UDP Whitelisting strategy: the edge proxy only accepts sensor data from the internal subnet, preventing external spoofing without needing complex authentication on low-power sensor devices.

2. Orchestration & Dependency Waves

To prevent race conditions during startup (e.g., the API trying to connect to a database that isn't ready), we use a "Wave-based" deployment strategy enforced by depends_on with condition: service_healthy.

• Wave 1: Core Infrastructure
  • RabbitMQ: Healthchecked via rabbitmq-diagnostics ping.
  • PostgreSQL & TimescaleDB: Healthchecked via pg_isready.
  • Redis: Healthchecked via redis-cli ping.
• Wave 2: Application Logic
  • REST API: Connects to both DBs and Redis. It performs automatic schema migration on startup.
• Wave 3: Data Processing
  • Ingestion Worker (C): Starts only after the API and RabbitMQ are ready.
  • Data Insertion (Python): Starts after the Ingestion Worker.
• Wave 4: Edge & Simulation
  • NGINX: Acts as the unified entry point.
  • Frontend: Served by NGINX.
  • Sensors: Start last to ensure the entire pipeline is ready to receive data.

3. Build Strategy: Multi-Stage Dockerfiles

We optimize our container images for size and security using multi-stage builds:

• Frontend: Stage 1 uses Node 22 to build the Vite/React application. Stage 2 uses a lightweight NGINX Alpine image to serve the static dist/ folder.
• Ingestion Worker: Stage 1 uses a GCC build environment to compile the C binary with all necessary libraries (hiredis, librabbitmq, libcurl). Stage 2 copies only the compiled binary and its runtime dependencies into a slim Debian image.

4. Edge Security & NGINX Configuration

NGINX serves three critical roles:

1. UDP Proxy: Forwards sensor packets to the C worker while enforcing IP whitelisting.
2. HTTP Reverse Proxy: Forwards API requests to the Spring Boot service.
3. Rate Limiting: Protects the API from DDoS attacks by limiting general requests to 20 req/s and authentication attempts to 5 req/s.

5. Persistence & Data Integrity

All stateful services (PostgreSQL, TimescaleDB, Redis, RabbitMQ) use Named Volumes. This ensures that even if a container is destroyed or updated, the data persists. Redis is configured with Append-Only File (AOF) persistence, and RabbitMQ uses Durable Queues to prevent message loss during broker restarts.

Running the Project:

# Navigate to the project directory
cd project/

# Build and start all containers in detached mode
docker compose up --build -d

Access Points:

• Frontend: http://localhost:3000
• REST API: http://localhost:8080
• Swagger Documentation: http://localhost:8080/swagger-ui/index.html
• RabbitMQ Management: http://localhost:15673

Documentation Index

• Full API Reference (80+ Endpoints)
• Redis Key & Caching Strategy
• Data Generation Pipeline Details
• Architectural Decisions & Diagrams
• Product Vision & Personas
• User Stories & Scenarios