12-search-and-analytics.md

Chapter 12 — Search & Analytics

Part III — Recipes (Leveraging Redis)

← Chapter 11 — Pub/Sub & Messaging | Table of Contents | Chapter 13 — Job Queues →

---

Redis is not a search engine—but its Sets, Sorted Sets, and optional RediSearch module make it a capable platform for lightweight full-text search, autocomplete, and real-time analytics. This chapter walks from DIY inverted indexes (no modules required) through production-grade RediSearch, with time-bucket analytics for dashboards.

Capability ladder:

1. Sets — boolean AND/OR term search for small catalogs.
2. Sorted Sets — autocomplete and top-N leaderboards.
3. Hashes + counters — time-bucket metrics without modules.
4. RediSearch — full-text, fuzzy, tag filters, aggregations.

Most teams start at step 1–3 and move to RediSearch when query requirements outgrow intersection performance or ranking needs.

---

Recipe 12.1 — DIY Inverted Index with Sets

Problem

You have a catalog of articles tagged with words. Users search for documents containing all query terms (AND search). PostgreSQL LIKE is too slow; you want sub-millisecond lookups in Redis.

Solution

An inverted index maps each term to the set of document IDs containing it. Intersect sets for multi-term queries.

Index on write (Python):

def index_document(r, doc_id: str, text: str):
    words = tokenize(text)  # lowercase, split, remove stopwords
    pipe = r.pipeline()
    for word in words:
        pipe.sadd(f"idx:term:{word}", doc_id)
    pipe.sadd(f"idx:doc:{doc_id}:terms", *words)
    pipe.execute()

def tokenize(text: str) -> set:
    return {w.lower() for w in text.split() if len(w) > 2}

redis-cli — manual index:

127.0.0.1:6379> SADD idx:term:redis doc:101 doc:205
(integer) 2
127.0.0.1:6379> SADD idx:term:caching doc:101 doc:302
(integer) 2
127.0.0.1:6379> SADD idx:term:streams doc:205
(integer) 1

AND search — intersection:

127.0.0.1:6379> SINTER idx:term:redis idx:term:caching
1) "doc:101"

OR search — union:

127.0.0.1:6379> SUNION idx:term:redis idx:term:streams
1) "doc:101"
2) "doc:205"

Python — multi-term AND:

def search_and(r, terms: list[str]) -> set:
    if not terms:
        return set()
    keys = [f"idx:term:{t.lower()}" for t in terms]
    return r.sinter(keys)

def search_or(r, terms: list[str]) -> set:
    keys = [f"idx:term:{t.lower()}" for t in terms]
    return r.sunion(keys)

Node.js:

async function searchAnd(terms) {
  const keys = terms.map(t => `idx:term:${t.toLowerCase()}`);
  return redis.sinter(...keys);
}

Go:

func SearchAND(ctx context.Context, rdb *redis.Client, terms []string) ([]string, error) {
    keys := make([]string, len(terms))
    for i, t := range terms {
        keys[i] = "idx:term:" + strings.ToLower(t)
    }
    return rdb.SInter(ctx, keys...).Result()
}

Delete document — clean stale postings:

def delete_document(r, doc_id: str):
    terms = r.smembers(f"idx:doc:{doc_id}:terms")
    pipe = r.pipeline()
    for term in terms:
        pipe.srem(f"idx:term:{term}", doc_id)
    pipe.delete(f"idx:doc:{doc_id}:terms")
    pipe.execute()

Discussion

DIY indexes work well up to millions of documents with simple tokenization. Limitations:

• No stemming ("run" vs "running"), ranking, or phrase queries.
• SINTER on huge sets is O(N×M)—watch cardinality.
• Reindex required for schema changes.

Store document payloads separately: HSET doc:101 title "..." body "...".

Phrase search approximation — store word pairs (bigrams) for two-word phrases:

def index_bigrams(r, doc_id: str, text: str):
    words = sorted(set(tokenize(text)))
    pipe = r.pipeline()
    for i, w1 in enumerate(words):
        for w2 in words[i+1:]:
            pipe.sadd(f"idx:bigram:{w1}:{w2}", doc_id)
    pipe.execute()

def search_phrase(r, w1: str, w2: str) -> set:
    return r.sinter([f"idx:term:{w1}", f"idx:term:{w2}", f"idx:bigram:{w1}:{w2}"])

Bigrams explode storage for long documents—use only for titles or short fields.

For prefix/autocomplete, use Sorted Sets (Recipe 12.2). For production full-text, use RediSearch (Recipe 12.4).

See Also

• Chapter 5 — Sets
• Recipe 12.2 — autocomplete

---

Recipe 12.2 — Sorted Set Autocomplete

Problem

Users type into a search box; you suggest completions ("red" → "redis", "redirect", "redisson") ranked by popularity, updated in real time.

Solution

Use a Sorted Set per prefix (or a single ZSET with composite members). Score = popularity (click count, search frequency).

Approach A — prefix keys with ZSET members as suffix completions:

On each search term submission, increment scores for all prefixes:

Python:

def record_search(r, term: str):
    term = term.lower().strip()
    for i in range(1, len(term) + 1):
        prefix = term[:i]
        r.zincrby(f"ac:prefix:{prefix}", 1, term)

def autocomplete(r, prefix: str, limit: int = 10) -> list:
    prefix = prefix.lower()
    # Lex range: all members starting with prefix
    return r.zrevrangebylex(
        f"ac:prefix:{prefix[:3]}",  # bucket by first 3 chars
        f"[{prefix}", f"[{prefix}\xff",
        start=0, num=limit,
    )

Approach B — simpler ZSET for short prefixes (redis-cli):

127.0.0.1:6379> ZADD ac:suggestions 100 "redis" 80 "redisson" 50 "redirect"
(integer) 3
127.0.0.1:6379> ZREVRANGEBYLEX ac:suggestions [red [red\xff LIMIT 0 5
1) "redis"
2) "redisson"
3) "redirect"

Lex ordering requires all scores equal (use score 0) when using ZRANGEBYLEX:

127.0.0.1:6379> ZADD ac:lex 0 "redis" 0 "redisson" 0 "redirect"
127.0.0.1:6379> ZREVRANGEBYLEX ac:lex [re [re\xff
1) "redis"
2) "redisson"
3) "redirect"

For popularity ranking within prefix, use separate ZSETs keyed by prefix with scores = frequency:

127.0.0.1:6379> ZINCRBY ac:prefix:red 1 "redis"
"1"
127.0.0.1:6379> ZINCRBY ac:prefix:red 1 "redis"
"2"
127.0.0.1:6379> ZREVRANGE ac:prefix:red 0 4 WITHSCORES
1) "redis"
2) "2"

Node.js:

async function autocomplete(prefix, limit = 10) {
  return redis.zrevrange(`ac:prefix:${prefix}`, 0, limit - 1);
}

Go:

results, err := rdb.ZRevRange(ctx, "ac:prefix:"+prefix, 0, 9).Result()

Discussion

• Bucket prefixes (first 2–3 chars) to avoid millions of sparse keys.
• Trim low-score suggestions periodically: ZREMRANGEBYRANK ac:prefix:red 0 -100 (keep top 100).
• RediSearch FT.SUGADD / FT.SUGGET offers built-in autocomplete if you run Redis Stack—see Recipe 12.4.

Unicode: normalize to NFC; consider edge n-grams for CJK.

See Also

• Chapter 7 — Sorted Sets
• Recipe 12.4 — RediSearch suggestions

---

Recipe 12.3 — Time-Bucket Analytics

Problem

You need a real-time dashboard: page views per minute, API errors per hour, top endpoints in the last 24 hours. InfluxDB is overkill for MVP; you want Redis-native counters.

Solution

Fixed time buckets use Sorted Sets or Hashes keyed by time window. Score = Unix timestamp or bucket ID; member/value = metric.

Minute-level page views (ZSET — one key per hour):

127.0.0.1:6379> ZINCRBY analytics:pv:2026052814 1 "/home"
"1"
127.0.0.1:6379> ZINCRBY analytics:pv:2026052814 3 "/pricing"
"3"
127.0.0.1:6379> ZREVRANGE analytics:pv:2026052814 0 9 WITHSCORES
1) "/pricing"
2) "3"
3) "/home"
2) "1"

Key analytics:pv:2026052814 = YYYYMMDDHH bucket.

HyperLogLog for unique visitors per day:

127.0.0.1:6379> PFADD analytics:uv:20260528 user:1001 user:1002 user:1001
(integer) 1
127.0.0.1:6379> PFCOUNT analytics:uv:20260528
(integer) 2

Sliding window with timestamped ZSET (unique events):

import time

def record_event(r, metric: str, event_id: str, window_secs: int = 3600):
    now = time.time()
    key = f"analytics:{metric}:events"
    pipe = r.pipeline()
    pipe.zadd(key, {event_id: now})
    pipe.zremrangebyscore(key, 0, now - window_secs)
    pipe.expire(key, window_secs + 60)
    pipe.execute()

def count_events(r, metric: str, window_secs: int = 3600) -> int:
    now = time.time()
    return r.zcount(f"analytics:{metric}:events", now - window_secs, now)

Python — hourly counter hash:

from datetime import datetime, timezone

def increment_hourly(r, metric: str, field: str, amount: int = 1):
    hour = datetime.now(timezone.utc).strftime("%Y%m%d%H")
    key = f"analytics:{metric}:{hour}"
    r.hincrby(key, field, amount)
    r.expire(key, 86400 * 7)  # retain 7 days

Node.js — pipeline for batch recording:

const hour = new Date().toISOString().slice(0, 13).replace(/[-T:]/g, '');
const key = `analytics:api_errors:${hour}`;
await redis.hincrby(key, '/v1/checkout', 1);
await redis.expire(key, 604800);

Go — time series with ZINCRBY:

hour := time.Now().UTC().Format("2006010215")
key := "analytics:pv:" + hour
rdb.ZIncrBy(ctx, key, 1, path)
rdb.Expire(ctx, key, 7*24*time.Hour)

Discussion

Pattern	Structure	Best for
Hourly hash	HINCRBY	Aggregated counts by dimension
ZSET leaderboard	ZINCRBY	Top-N in window
HLL	PFADD	Unique counts (approximate)
Timestamp ZSET	ZADD + trim	Sliding window uniques

Redis TimeSeries module (Redis Stack) adds downsampling and range queries—see Chapter 23. DIY buckets suffice for many dashboards.

Set TTL on every analytics key to prevent unbounded growth.

Daily rollup script — compress minute buckets into day totals:

def rollup_day(r, date: str):
    """Merge analytics:pv:YYYYMMDDHH into analytics:pv:day:YYYYMMDD."""
    day_key = f"analytics:pv:day:{date}"
    for hour in range(24):
        hour_key = f"analytics:pv:{date}{hour:02d}"
        for path, score in r.zrange(hour_key, 0, -1, withscores=True):
            r.zincrby(day_key, score, path)
        r.expire(hour_key, 86400)  # drop hour keys after rollup
    r.expire(day_key, 86400 * 90)

Run rollups via cron or a delayed job (Chapter 13). Dashboards query analytics:pv:day:20260528 for historical charts and the current hour key for live data.

See Also

• Chapter 8 — HyperLogLog
• Chapter 14 — Rate limiting — sliding windows

---

Recipe 12.4 — RediSearch: FT.CREATE and FT.SEARCH

Problem

DIY indexes cannot handle fuzzy matching, stemming, numeric filters, or relevance ranking at scale. You run Redis Stack (or RediSearch module) and want production full-text search.

Solution

Create an index with FT.CREATE, index documents with HSET + hash prefix or JSON.SET, query with FT.SEARCH.

redis-cli — hash-backed index:

127.0.0.1:6379> FT.CREATE idx:articles ON HASH PREFIX 1 "doc:" SCHEMA title TEXT WEIGHT 2.0 body TEXT tags TAG SORTABLE published NUMERIC SORTABLE
OK

127.0.0.1:6379> HSET doc:101 title "Redis Caching Patterns" body "Cache-aside and stampede prevention" tags "redis,caching" published 1716892800
(integer) 4

127.0.0.1:6379> FT.SEARCH idx:articles "caching stampede" LIMIT 0 10
1) (integer) 1
2) "doc:101"
3) 1) "title"
   2) "Redis Caching Patterns"
   ...

Filtered search:

127.0.0.1:6379> FT.SEARCH idx:articles "@tags:{redis} @published:[1716000000 1717000000]" SORTBY published DESC LIMIT 0 20

Python (redis-py with RediSearch):

from redis.commands.search.field import TextField, TagField, NumericField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType
from redis.commands.search.query import Query

# One-time setup
schema = (
    TextField("title", weight=2.0),
    TextField("body"),
    TagField("tags"),
    NumericField("published", sortable=True),
)
r.ft("idx:articles").create_index(
    schema,
    definition=IndexDefinition(prefix=["doc:"], index_type=IndexType.HASH),
)

# Index document
r.hset("doc:101", mapping={
    "title": "Redis Caching Patterns",
    "body": "Cache-aside and stampede prevention",
    "tags": "redis,caching",
    "published": 1716892800,
})

# Search
q = Query("caching stampede").sort_by("published", asc=False).paging(0, 10)
results = r.ft("idx:articles").search(q)
for doc in results.docs:
    print(doc.title, doc.id)

Node.js (ioredis + redis-om or raw FT commands):

await redis.call(
  'FT.CREATE', 'idx:articles', 'ON', 'HASH', 'PREFIX', '1', 'doc:',
  'SCHEMA', 'title', 'TEXT', 'WEIGHT', '2.0', 'body', 'TEXT',
  'tags', 'TAG', 'SORTABLE', 'published', 'NUMERIC', 'SORTABLE'
);

const results = await redis.call(
  'FT.SEARCH', 'idx:articles', '@tags:{redis} caching', 'LIMIT', '0', '10'
);

Go (Rueidis or go-redis with search support):

// Raw command via Do()
rdb.Do(ctx, "FT.SEARCH", "idx:articles", "stampede", "LIMIT", "0", "10")

Autocomplete with suggestions:

127.0.0.1:6379> FT.SUGADD ac:suggest "redis" 100
(integer) 1
127.0.0.1:6379> FT.SUGGET ac:suggest "red" FUZZY MAX 5
1) "redis"

Discussion

RediSearch adds inverted indexes at C speed inside Redis:

• TEXT — full-text with stemming (language-dependent).
• TAG — exact match, comma-separated multi-values.
• NUMERIC/GEO — range and geo filters in same query.
• Aggregations — FT.AGGREGATE for GROUP BY analytics.

Operational notes:

• Index builds are synchronous on first HSET; bulk load with FT.CREATE ... ON HASH then pipeline HSET.
• Memory ≈ 1.5–3× source data depending on fields; monitor with FT.INFO.
• Cluster: index must live on hash slots covering key prefixes; use {doc}:101 hash tags if needed.

When RediSearch is unavailable, fall back to Recipe 12.1 for basic AND search.

JSON document index (Redis Stack):

127.0.0.1:6379> JSON.SET doc:json:101 $ '{"title":"Redis Streams","body":"Consumer groups","tags":["redis","streams"],"published":1716892800}'

127.0.0.1:6379> FT.CREATE idx:json_docs ON JSON PREFIX 1 "doc:json:" SCHEMA $.title AS title TEXT $.body AS body TEXT $.tags[*] AS tags TAG $.published AS published NUMERIC
OK

127.0.0.1:6379> FT.SEARCH idx:json_docs '@tags:{redis} streams' LIMIT 0 5

JSON indexes suit APIs already emitting JSON documents—no hash flattening step.

Python — JSON indexing:

import json

r.json().set("doc:json:101", "$", {
    "title": "Redis Streams",
    "body": "Consumer groups explained",
    "tags": ["redis", "streams"],
    "published": 1716892800,
})
# Index created once via FT.CREATE on JSON paths
results = r.ft("idx:json_docs").search(Query("@tags:{redis} streams"))

See Also

• Chapter 23 — Modules & Redis Stack
• Recipe 12.1 — DIY fallback
• RediSearch documentation

---

Chapter Summary

Recipe	Technique	Scale	Complexity
12.1	Set inverted index	Medium catalogs	Low
12.2	ZSET autocomplete	High QPS typeahead	Low–medium
12.3	Time buckets	Real-time dashboards	Low
12.4	RediSearch	Production search	Medium (module)

Start simple with native structures; adopt RediSearch when relevance, fuzzy match, or compound filters become requirements—not on day one.

---

Recipe 12.5 — Combining Search and Analytics Pipelines

Problem

Product search must boost popular items. Analytics must feed back into ranking without a nightly batch job.

Solution

Maintain a popularity ZSET updated on every click; join at query time or denormalize into document scores.

Record click (Python):

def record_click(r, doc_id: str):
    r.zincrby("analytics:clicks:7d", 1, doc_id)
    r.zremrangebyrank("analytics:clicks:7d", 0, -10001)  # keep top 10k

Boost search results post-query:

def search_with_boost(r, terms: list[str], limit: int = 20) -> list:
    doc_ids = list(search_and(r, terms))[:limit * 2]
    if not doc_ids:
        return []
    pipe = r.pipeline()
    for doc_id in doc_ids:
        pipe.zscore("analytics:clicks:7d", doc_id)
    scores = pipe.execute()
    ranked = sorted(zip(doc_ids, scores), key=lambda x: x[1] or 0, reverse=True)
    return [d for d, _ in ranked[:limit]]

RediSearch — numeric popularity field updated on click:

def record_click_redisearch(r, doc_id: str):
    r.zincrby("analytics:clicks:7d", 1, doc_id)
    clicks = int(r.zscore("analytics:clicks:7d", doc_id) or 0)
    r.hset(doc_id, "popularity", clicks)
    # RediSearch reindexes hash automatically

Query with boost:

127.0.0.1:6379> FT.SEARCH idx:articles "redis" SORTBY popularity DESC LIMIT 0 10

Discussion

Separating index (searchable content) from signals (clicks, conversions) keeps write paths independent. Refresh popularity periodically if hot-path writes to RediSearch are too expensive—eventual ranking skew is usually acceptable for typeahead and browse.

For funnel analytics (view → click → purchase), chain Streams: XADD analytics:events * step view doc_id 101, then aggregate with FT.AGGREGATE or external ETL.

See Also

• Recipe 12.1 and 12.4 — index + module search
• Chapter 23 — Redis Stack

---

Index Maintenance Checklist

When operating DIY or RediSearch indexes in production:

1. Backfill script — paginate source DB, pipeline SADD / HSET in batches of 500–1000.
2. Delete path — every DELETE doc:101 must trigger index cleanup (Recipe 12.1 delete_document).
3. Memory — MEMORY USAGE idx:term:redis for hot terms; split oversized posting lists.
4. Rebuild — RediSearch: FT.DROPINDEX idx:articles DD then recreate; DIY: rename prefix and swap.
5. Staging validation — compare result counts against PostgreSQL tsvector for sample queries before cutover.

---

← Chapter 11 — Pub/Sub & Messaging | Table of Contents | Chapter 13 — Job Queues →