Skip to content

MatthiasHowellYopp/Valkey-Search-Aggregate-Demo

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ValkeySearch Live NHL Analytics

A sample application demonstrating ValkeySearch aggregation features through a live NHL sports streaming analytics dashboard. The app simulates real-time hockey games and uses ValkeySearch to power trending match rankings, win probability calculations, and live scoreboard updates.

⚠️ Pre-Release Dependencies: This app requires pre-release versions of Valkey Server, ValkeySearch, and the Valkey Glide Java client that are not yet available as standard releases. You must build these from source before running the app. All necessary build tooling is included in this repo — see Prerequisites and Getting Started below.

What This Demonstrates

This app showcases how to use Valkey with ValkeySearch for real-time analytics workloads:

  • FT.CREATE — Defining a search index on live game data stored as Valkey Hashes
  • FT.SEARCH — Querying game state with TAG filters and NUMERIC range queries
  • FT.AGGREGATE — Server-side aggregation to rank trending matches by viewer engagement
  • HSET — Storing structured game state as Hashes (auto-indexed by ValkeySearch)
  • ZADD — Sorted Sets for viewer-ranked game tracking
  • XADD — Streams for append-only game event logs (goals, shots, penalties)
  • PUBLISH / SUBSCRIBE — Pub/Sub for event-driven updates (feeder notifies webapp instantly)
  • Server-Sent Events (SSE) — Pushing real-time updates from Valkey to the browser

Prerequisites

This app depends on pre-release versions of Valkey components. Build tooling is included in this repo.

Dependency Version Source Build Script
Valkey Server 9.1.0-rc1 valkey-io/valkey docker/valkey/Dockerfile
ValkeySearch Module Commit 923430d valkey-io/valkey-search docker/valkey/Dockerfile
Valkey Glide (Java) main branch (255.255.255) valkey-io/valkey-glide docker/glide-java/Dockerfile

You also need:

Tool Version Purpose
Docker Desktop 24+ Building all pre-release dependencies from source
Docker Desktop RAM 12GB minimum ValkeySearch C++ compilation is memory-intensive
Java 17+ Building and running the Kotlin app

Once Valkey 9.1, ValkeySearch 1.3, and the corresponding Glide release are GA, the pre-release build steps can be replaced with standard dependency declarations.

Getting Started

Step 1: Build Valkey Server + ValkeySearch Module

This builds Valkey 9.1.0-rc1 and the ValkeySearch module from source inside Docker:

docker compose build valkey

Important: Ensure Docker Desktop has at least 12GB of RAM allocated (Settings → Resources → Memory). The ValkeySearch C++ build is memory-intensive — the Dockerfile uses single-threaded compilation (-j1) to stay within limits, so expect this to take 15-20 minutes on the first build.

Start the Valkey server and verify the search module loaded:

docker compose up -d valkey
docker exec -it $(docker compose ps -q valkey) valkey-cli MODULE LIST

You should see search in the module list.

Step 2: Build Valkey Glide Java Client

The Glide client must be built from source because the ValkeySearch aggregation commands aren't in a released version yet:

./build-glide-java.sh

This builds the Glide Java client inside Docker and installs the JAR to your local Maven repo (~/.m2/repository/io/valkey/valkey-glide/255.255.255/). The app's Gradle build references it as io.valkey:valkey-glide:255.255.255.

Step 3: Run the App

Option A — Docker Compose (recommended)

Run the full stack (Valkey + feeder + webapp) in Docker:

cd app && ./prep-build.sh && cd ..   # stage Glide JAR for Docker build context
docker compose build feeder webapp
docker compose up

Open http://localhost:8080 to see the live dashboard.

Option B — Gradle (local development)

If you prefer running the app locally (requires Valkey running via Docker or natively):

cd app
./gradlew build
./gradlew :feeder:run &          # start game simulator in background
./gradlew :webapp:bootRun        # start web dashboard

Open http://localhost:8080.

Architecture

┌──────────────────────────────────────────────────────────────────┐
│  Feeder (Kotlin CLI)                                             │
│  Simulates 4 concurrent NHL games, generates events every 2s     │
│  Writes to Valkey: HSET (game state), ZADD (rankings),           │
│                    XADD (event stream), PUBLISH (notification)   │
└──────────────┬───────────────────────────────────────────────────┘
               │ Valkey Glide Client
               ▼
┌──────────────────────────────────────────────────────────────────┐
│  Valkey + ValkeySearch Module                                    │
│                                                                  │
│  game:{id}  → Hash (live game state, auto-indexed)               │
│  idx:games  → Search Index (TAG, NUMERIC, TEXT fields)           │
│  events:{id}→ Stream (goal, shot, penalty, hit events)           │
│  active_games → Sorted Set (games ranked by viewers)             │
│  game:updates → Pub/Sub Channel (feeder → webapp notification)   │
│                                                                  │
│  FT.SEARCH   → filtered game retrieval                           │
│  FT.AGGREGATE → server-side trending calculation                 │
└──────────────┬───────────────────────────────────────────────────┘
               │ Valkey Glide Client
               ▼
┌──────────────────────────────────────────────────────────────────┐
│  Webapp (Spring Boot + Kotlin)                                   │
│                                                                  │
│  GameDataService  → queries Valkey (FT.SEARCH, FT.AGGREGATE)     │
│  SseController    → pushes updates to browsers via Pub/Sub+SSE   │
│  ApiController    → REST endpoints + benchmark runner            │
│  HomeController   → Thymeleaf initial page render                │
└──────────────┬───────────────────────────────────────────────────┘
               │ Server-Sent Events (SSE)
               ▼
┌──────────────────────────────────────────────────────────────────┐
│  Browser (Vanilla JS)                                            │
│                                                                  │
│  EventSource('/api/sse/games') receives updates                  │
│  Updates DOM: scoreboard, trending list, win probability bars    │
│  No framework — just native SSE + template literals              │
└──────────────────────────────────────────────────────────────────┘

Data Flow

  1. Feeder generates game events → writes to Valkey via HSET, ZADD, XADD
  2. Feeder publishes a notification on the game:updates channel (PUBLISH)
  3. ValkeySearch automatically indexes hash updates in real time
  4. Webapp subscriber client receives the Pub/Sub message (event-driven, no polling)
  5. Webapp queries Valkey via FT.SEARCH and FT.AGGREGATE for latest state
  6. SseController pushes JSON payloads to all connected browsers via SSE
  7. Browser JS receives SSE events and updates the DOM (scoreboard, trending, win probability)

No polling anywhere in the pipeline. Feeder → Pub/Sub → Webapp → SSE → Browser.

Project Structure

app/
├── common/                          # Shared models and Valkey utilities
│   └── src/main/kotlin/.../
│       ├── model/
│       │   ├── Models.kt            # GameState, GameEvent, TrendingMatch, WinProbability
│       │   └── NhlTeams.kt          # All 32 NHL teams with divisions/conferences
│       └── valkey/
│           ├── Keys.kt              # Valkey key naming conventions (★ documented)
│           └── ValkeyConnection.kt  # Glide client factory
│
├── feeder/                          # Game simulation + Valkey data writer
│   └── src/main/kotlin/.../feeder/
│       ├── Main.kt                  # Entry point, config from env vars
│       ├── GameSimulator.kt         # Simulates one NHL game with realistic events
│       └── FeederService.kt         # Writes to Valkey, creates search index (★ documented)
│
├── webapp/                          # Spring Boot web application
│   └── src/main/kotlin/.../webapp/
│       ├── Application.kt           # Spring Boot entry point
│       ├── config/
│       │   └── ValkeyConfig.kt      # Glide client Spring bean
│       ├── service/
│       │   └── GameDataService.kt   # FT.SEARCH + FT.AGGREGATE queries (★ documented)
│       └── controller/
│           ├── HomeController.kt    # Thymeleaf initial page render
│           ├── SseController.kt     # SSE push to browsers (★ documented)
│           └── ApiController.kt     # REST API + benchmarks (★ documented)
│   └── src/main/resources/
│       ├── templates/index.html     # Dashboard page (Thymeleaf + SSE)
│       ├── static/js/app.js         # SSE client + DOM updates (★ documented)
│       ├── static/css/style.css     # Dark theme dashboard styles
│       └── application.yml          # Server and Valkey connection config
│
├── build.gradle.kts                 # Root build (Kotlin 1.9, JVM 17)
└── settings.gradle.kts              # Multi-module: common, feeder, webapp

docker/
├── valkey/Dockerfile                # Builds Valkey 9.1.0-rc1 + ValkeySearch from source
└── glide-java/Dockerfile            # Builds Valkey Glide Java client from source

docker-compose.yaml                  # Orchestrates Valkey, feeder, and webapp
build-glide-java.sh                  # Builds Glide JAR and installs to local Maven repo

Files marked with ★ contain detailed inline documentation explaining Valkey features used and the data flow.

Configuration

Both the feeder and webapp connect to Valkey via environment variables:

Variable Default Description
VALKEY_HOST localhost Valkey server hostname
VALKEY_PORT 6379 Valkey server port
NUM_GAMES 4 Number of simultaneous games to simulate
TICK_INTERVAL_MS 2000 Milliseconds between game simulation ticks

ValkeySearch Queries Used

FT.CREATE — Index Definition

FT.CREATE idx:games ON HASH PREFIX 1 game: SCHEMA
  homeTeam TAG  awayTeam TAG  status TAG
  homeScore NUMERIC SORTABLE  awayScore NUMERIC SORTABLE
  viewers NUMERIC SORTABLE  period NUMERIC SORTABLE
  homeShots NUMERIC SORTABLE  awayShots NUMERIC SORTABLE
  ...

See: FeederService.kt

FT.SEARCH — Filtered Game Retrieval

FT.SEARCH idx:games @status:{live|pregame|intermission} LIMIT 0 20
FT.SEARCH idx:games @startTime:[0 +inf] LIMIT 0 20

See: GameDataService.kt

FT.AGGREGATE — Trending Matches

FT.AGGREGATE idx:games @status:{live}
  LOAD 10 @homeTeam @awayTeam @homeScore @awayScore @period
         @timeRemaining @status @viewers @homeShots @awayShots
  APPLY @viewers AS trendScore
  SORTBY 2 @trendScore DESC
  LIMIT 0 10

See: GameDataService.kt

Performance Benchmarks

The app includes a built-in benchmark accessible at /api/benchmark?iterations=100 or via the dashboard UI button. It measures round-trip latency for each query type:

Query What it measures
FT.SEARCH all games Full index scan with numeric range filter
FT.SEARCH active games TAG filter on game status
FT.AGGREGATE trending Server-side aggregation with LOAD, APPLY, SORTBY
Win probability calc FT.SEARCH + application-level sigmoid computation

About

A sample application demonstrating ValkeySearch aggregation features through a live NHL sports streaming analytics dashboard. The app simulates real-time hockey games and uses ValkeySearch to power trending match rankings, win probability calculations, and live scoreboard updates.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors