Implement historical api (#530)

Adds player history support by ingesting historical FIDE rating data.
Introduces a new cli module for crawling and ingesting historical CSV
snapshots, a HistoryDb persistence layer for player history and
federation summary views, and Smithy-defined history API
  endpoints served by a new HistoryService. 
  
The crawler is refactored to use hash-based incremental diffing (with
cached hash Refs) so only changed players emit events, and ingestion is
skipped entirely when data is already up to date.

Author: lenguyenthanh

.github/workflows/docker.yml

@@ -46,5 +46,8 @@ jobs:
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
 
-      - name: Build and push Docker image
+      - name: Build and push backend Docker image
         run: sbt backend/Docker/publish
+
+      - name: Build and push CLI Docker image
+        run: sbt cli/Docker/publish

AGENTS.md

@@ -1,6 +1,6 @@
 # CLAUDE.md
 
-Guidance for AI agents generating Scala 3 code in this Typelevel-stack project.
+Project-specific guidance for the FIDE Lichess Typelevel-stack project.
 
 ## Issue Tracking
 
@@ -15,37 +15,19 @@ Run `bd prime` for workflow context, or install hooks (`bd hooks install`) for a
 
 For full workflow details: `bd prime`
 
-## Global Rules
+## Claude session rules
 
- - **Always use `sbt --client`** — never bare `sbt`. Connects to running server instead of launching new JVM.
- - **Redirect sbt output**: `sbt --client "module/clean ; test-jvm-2_13" 2>&1 | tee /tmp/sbt-output.txt`
-   Then inspect with `grep`/`tail`/`head`. Only re-run sbt if code was modified.
- - Only run `sbt compile`/`sbt test` after MCP shows no compilation issues.
- - After code changes: `sbt lint` (scalafmt + scalafix), then `sbt test`.
- - **Do not add yourself as co-author** — no `Co-Authored-By: Claude ...` in commits.
+- Always save any plan/optimization session in `<project-root>/.claude/docs/` directory instead of just writing in the console
+- Omit `Co-Authored-By` lines from git commits
 
-## Tool references
+## Skills
 
-### MCP — Always verify APIs via MCP before using them
+**Always** use these skills:
 
-Metals MCP at `.metals/mcp.json`. Query definitions/types/symbols, check compilation errors before running sbt. Fix MCP issues before running sbt to avoid long feedback loops.
-
-**Override Grep with Metals MCP when the question is "what does the compiler resolve this to?"**
-
-Grep is the default and works for most searches. But it fails silently on these Scala-specific scenarios — use Metals instead:
-
-| Scenario | Tool |
-|---|---|
-| What type is this expression / what does it return? | `mcp__metals__inspect` |
-| Which given/implicit is resolved at this call site? | `mcp__metals__inspect` |
-| Which overloaded method is called here? | `mcp__metals__inspect` |
-| What's the underlying type of an opaque type? | `mcp__metals__inspect` |
-| What does a wildcard import bring into scope? | `mcp__metals__inspect` |
-| Who calls this method / all implementations of a trait? (semantic, not textual) | `mcp__metals__get-usages` |
-
-Other Metals tools: `glob-search` (find symbols by name), `get-docs` (ScalaDoc), `compile-file` (single-file compile check), `list-modules`, `list-scalafix-rules`.
-
-**Signal to switch:** When you grep and get 10+ candidates with no way to disambiguate — that means you need Metals, not a better regex. Fall back to Grep/Glob for non-Scala files, string literals, config values, SQL, or when Metals is unavailable.
+ - General Scala 3/FP standards are in the `/scala3-fp` skill.
+ - cats-effect best practices are in the `cats-effect-io` and `cats-effect-resource`
+ - typed error with cats-mtl best practices are in the `cats-mtl-typed-errors`
+ - SBT and Metals workflow is in the `/scala-sbt` skill.
 
 ## Modules Structure
 
@@ -83,100 +65,21 @@ HTTP server and orchestration. Ember server with Smithy4s routes (`PlayerService
 ### gatling (`modules/gatling`)
 Load testing simulations (Gatling). Warmup (2k users), Stress (12k users), and Capacity (incremental ramp) scenarios against `localhost:9669/api`. Independent of other modules.
 
-## Functional Programming Standards
-
-- **No mutable state**: avoid `var`, mutable collections, and `return`.
-- **Prefer immutable case classes** over builder patterns.
-- If mutability is unavoidable, scope it inside a method — never as class fields.
-- Minimize side-effects; push them to the edges (entry points).
-- Use ADTs (`enum`) for modeling closed sets of variants.
-
-## Scala 3 Syntax
-
-- **Braceless syntax** — the project uses `rewrite.scala3.removeOptionalBraces = yes`.
-- Use Scala 3 `enum`, `given`/`using`, `extension`, `opaque type`, and `derives` where appropriate.
-- Use `for`/`yield` (not `flatMap` chains) for sequencing effects — this is the idiomatic pattern in this codebase.
-- Prefer `match` expressions over nested `if`/`else`.
-
-## Code Style
-
-- Braceless syntax — no curly braces for `if`, `match`, `for`, class/object bodies.
-- `maxColumn = 110`.
-- Wildcard imports: `import cats.syntax.all.*`
-- `align.preset = none` — do not align on `=` or `=>`.
-
-### Code Style: Flat `for`-comprehensions
-
-Prefer flat `for`/`yield` over nested `match`/`case` inside effectful blocks. Lift `Either`/`Option` into `F` so the `for` stays linear:
-
-```scala
-// BAD — nested match in MID-CHAIN breaks the for-comprehension flow
-for
-  result <- service.doSomething(...)
-  value  <- result match  // BAD: match in middle, more steps follow
-    case Right(v)  => v.pure[F]
-    case Left(err) => Sync[F].raiseError(err)
-  next   <- process(value)
-  response <- Ok(next.asJson)
-yield response
-
-// ALSO BAD — .flatMap with case inside for-comprehension
-for
-  body <- req.req.as[Body]
-  result <- service.getItem(id).flatMap {
-    case Some(item) => Ok(item.asJson)
-    case None       => NotFound(...)
-  }
-yield result
-
-// GOOD — plain match at TERMINAL position (pure response mapping, no side effects)
-for
-  body   <- req.req.as[Body]
-  result <- service.doSomething(body)
-  response <- result match
-    case Right(value) => Ok(value.asJson)
-    case Left(err)    => BadRequest(err.asJson)
-yield response
-```
+### Project-specific opaque types
+Defined in `core/domain/Ids.scala` and `core/domain/Types.scala`.
+Prefer to use Iron to define new type than opaque
 
-### Code style rules
-- **No fully-qualified names in code.** Always use imports.
-- **Context bounds: use `{A, B, C}` syntax** (Scala 3.6 aggregate bounds), not colon-separated.
-- **Opaque types for domain values.** AI writes 90%+ of code, so write-cost is near zero while compile-time safety is free. Use opaque types with smart constructors for all entity IDs, constrained strings, and bounded numbers. Defined in `core/domain/Ids.scala` and `core/domain/Types.scala`.
-- **Type-level constraints flow E2E.** Encode invariants in types (opaque types, `NonEmptyList`, refined types) and propagate them through **all** layer signatures: route → service → repository. Never downgrade a constraint to a weaker type and re-validate internally — that hides the requirement from callers and defeats compile-time safety. Unwrap/weaken only at the true system boundary: SQL interpolation, Java SDK calls, job parameter serialization.
-- **`.toString` over `.value.toString`.** Opaque types erase at runtime, so `s"...$opaqueId"` and `opaqueId.toString` just work — no need to unwrap first.
-- **`NonEmptyList` over `List` + `.get`/`.head`.** When a method logically requires non-empty input (batch embeddings, `IN` clauses, etc.), use `NonEmptyList[T]` in the **signature** — including repository methods — instead of `List[T]` with a runtime `.toNel.get` or `.head`. Callers use `NonEmptyList.fromList` to handle the empty case at the call site.
-- **No premature helpers.** If the logic can be composed from <5 Scala/cats operators, always inline at call site — never extract a helper. If >=5 operators, **ask the user** before extracting (in plan mode or popup dialog). When consensus is reached on a new helper, add/link it in this document so future sessions know to use it. **Always use helpers already listed here** (e.g., `AsyncOps`) — don't expand them inline. Before writing any new helper, search the codebase for existing ones that do the same thing.
-- **Generic over specific (stdlib/cats only).** Prefer composing well-tested Scala/cats operators generically over type-specific. "Generic" means leveraging stdlib type classes, not extracting custom helper functions — those still follow the <5 operator rule.
-- **Proactive naming review.** When modifying code, flag misleading, stale, or inconsistent names to the user. **Scope follows the compiler iteratively** — same as smell detection: start with changed files, then follow compilation errors outward. For **internal names** (classes, properties, methods) — recommend renaming directly. For **external names** (request/response DTOs, DB-serialized JSONB fields) — suggest the better name but note migration implications.
-- **Proactive code smell detection.** Scope follows the compiler iteratively: (1) find smell in current file, (2) fix it, (3) compile → if it fails because other files import the changed symbol, fix those too, (4) repeat until compilation passes. If **reading unrelated code** (not in the compilation chain) and spotting a violation — **add it to the smell list** (see Code Smell Tracking below), do not fix. This applies to all rules: type safety, error handling, control flow, naming, logging, etc.
-
-### Error Handling
-
-- **No error swallowing.** Unless the business scenario *explicitly* requires a default value, using `.getOrElse`, blanket `try`/`catch`, `.recover`, `.handleErrorWith`, or `.orElse` to silently discard failures is forbidden. Every error must be surfaced — either raised via `Raise` or propagated in the type.
-- **Prefer cats-mtl typed errors over `Either`.** Use `Raise[F, E]` / `Handle[F, E]` to short-circuit errors in `F` directly, keeping for-comprehensions flat. Avoid `IO[Either[E, A]]` return types — they force callers to unwrap and nest. Reserve `Either` for pure validation logic or values that cross serialization boundaries.
-
-```scala
-// BAD — Either in return type, callers must unwrap
-def fetchUser(id: UserId): IO[Either[AppError, User]]
-
-// GOOD — error raised in F, callers just flatMap
-def fetchUser[F[_]: {Async, Raise[AppError]}](id: UserId): F[User]
-```
-
-- **Trusted vs untrusted data paths.** Internal data (config, DB-persisted values, intra-service calls) is trusted — failures are bugs, raise directly (`MonadThrow`). External data (user input, third-party APIs) is untrusted — capture failures via `Raise` so callers `Handle` them explicitly at the boundary.
-- **Trust the compiler.** What the signature declares, trust unconditionally. Do not defensively re-validate what the type system already guarantees (e.g., don't `require(nel.nonEmpty)` on a `NonEmptyList`, don't null-check an opaque type). Every act of distrust costs tokens exponentially — the agent opens the implementation, reads dependents, context bloats, and subsequent steps degrade.
-- **Signature completeness.** Every function must honestly declare its effects (`IO`/`F[_]`), failure modes (via `Raise[F, E]` constraint or `Option`), and use domain types for parameters. A signature like `def fetchUser(id: String): User` is dishonest — it hides effects and error paths. Prefer `def fetchUser[F[_]: {Async, Raise[AppError]}](id: UserId): F[User]`.
-
-### NoOp Service Implementations
+### Proactive code smell detection
 
-When providing stub/noop implementations of service traits:
-- **Data-related operations MUST fail** — raise the error or return a failed effect. Silent success would mask missing wiring.
-- **Data-irrelevant operations CAN succeed** — metrics, logging, telemetry noops return `().pure[F]` since skipping them is safe.
+Scope follows the compiler iteratively:
+(1) find smell in current file
+(2) fix it
+(3) compile → if it fails because other files import the changed symbol, fix those too
+(4) repeat until compilation passes. If **reading unrelated code** (not in the compilation chain) and spotting a violation — **add it to the smell list** (see Code Smell Tracking below), do not fix. This applies to all rules: type safety, error handling, control flow, naming, logging, etc.
 
 ### Code Smell Tracking
 
-When spotting code smells in **unrelated code** (not in the current compilation chain), add them to the persistent smell list file at `<project-root>/memory/code_smells.md` instead of just warning in the response.
+When spotting code smells in **unrelated code** (not in the current compilation chain), add them to the persistent smell list file at `<project-root>/.claude/memory/code_smells.md` instead of just warning in the response.
 
 **Rules:**
 - **Max 10 entries.** If adding an 11th, delete the oldest entry (FIFO eviction).
@@ -190,55 +93,15 @@ When spotting code smells in **unrelated code** (not in the current compilation
 For non-trivial tasks, generate a plan before writing code:
 1. Identify affected modules and files.
 2. List the changes needed in each file.
-3. Note any schema/migration impacts (Smithy, ES mappings).
+3. Note any schema/migration impacts.
 4. Confirm the plan before implementing.
 
 After implementing a group of related changes, review for:
 - Correctness (does it compile and pass tests?)
 - Consistency (does it follow existing patterns in the codebase?)
 - Simplicity (is there a simpler way to achieve the same result?)
 
-## Things to Avoid
-
-- Do NOT use `Future`, `Await`, `Thread.sleep`, or blocking calls on the IO thread pool.
-- Do NOT use Ox, ZIO, or direct-style concurrency libraries — this project is cats-effect.
-- Do NOT manually define ES field mappings — they are generated from Smithy schemas.
-- Do NOT reformat snapshot JSON files (`elastic/src/test/resources/snapshot/mappings/`).
-- Do NOT wrap `Option[Boolean]` in `Some()` for Smithy trait properties.
-
-
-## Rule Hierarchy — Four Layers of Constraint
-
-Not all rules are equal. When adding or interpreting rules in this file, classify them by enforcement layer:
-
-| Layer | Enforced by | Examples | Agent action |
-|---|---|---|---|
-| **1. Compiler** | Type system, exhaustiveness, opaque types | `Raise[F, E]` in signature, sealed trait match | No rule needed — compiler rejects violations |
-| **2. Actionable rules** | This file — clear if/then criteria | "No error swallowing", "NonEmptyList over List + .head" | Follow unconditionally; violations are code smells |
-| **3. Process constraints** | Filesystem as cross-session memory | Code smell tracking in `code_smells.md` | Agent discovers → records → human prioritizes |
-| **4. Advisory** | Agent suggests, human decides | "Consider adding runtime assertion here", deploy impact | Flag to user, do not act unilaterally |
-
-Push constraints **up** whenever possible: prefer a compiler-enforced type over an actionable rule, prefer an actionable rule over an advisory suggestion.
-
-## Rule Quality Standard
-
-Rules in this file must be precise enough that the agent never needs to read extra context to guess intent. Every ambiguity = extra file reads = token bloat = degraded reasoning.
-
-```
-// BAD — vague, requires judgment calls
-"Always use tagless final style"
-"Be pragmatic about error handling"
-
-// GOOD — unambiguous if/then, no interpretation needed
-"ParserService should be a typeclass constraint in the class constructor:
-  class FileProcessor[F[_]: {Async, ParserService}]
-NOT as a parameter:
-  def process[F[_]](parser: ParserService[F])"
-```
-
-When writing new rules: lead with the concrete pattern (what to do), then the anti-pattern (what not to do), then rationale (why). If a rule can't be expressed as a clear if/then, it belongs in Layer 4 (advisory), not Layer 2.
-
-### Fixing a bug
+### Fixing a Bug
 
 1. Write a failing test reproducing the bug
-3. Verify the test fails, apply fix, clean again, verify all tests pass
+2. Verify the test fails, apply fix, clean again, verify all tests pass

README.md

@@ -116,3 +116,22 @@ Then run:
 ```bash
 docker compose up -d
 ```
+
+## Run CLI without building
+
+You can use the pre-built CLI Docker image from [GitHub Container Registry](https://github.com/lenguyenthanh/fide/pkgs/container/fide-cli) to ingest historical CSV files into the database.
+
+Assuming CSV files are in `./csv-data` and the docker-compose Postgres is running:
+
+```bash
+docker run --rm \
+  --network fide_fide \
+  -v ./csv-data:/data \
+  -e POSTGRES_HOST=fide_postgres \
+  -e POSTGRES_PORT=5432 \
+  -e POSTGRES_USER=admin \
+  -e POSTGRES_PASSWORD=dummy \
+  -e POSTGRES_DATABASE=fide \
+  ghcr.io/lenguyenthanh/fide-cli:latest \
+  ingest /data --start 2024-01 --end 2024-12
+```

build.sbt

@@ -49,7 +49,8 @@ lazy val api = (project in file("modules/api"))
     smithy4sWildcardArgument := "?",
     libraryDependencies ++= Seq(
       smithy4sCore
-    )
+    ),
+    Compile / scalacOptions += "-Wconf:src=target/scala[^/]*/src_managed/.*:silent"
   )
   .dependsOn(types)
 
@@ -109,6 +110,30 @@ lazy val backend = (project in file("modules/backend"))
   .enablePlugins(JavaAppPackaging, DockerPlugin)
   .dependsOn(api, domain, full(db), crawler)
 
+lazy val cli = (project in file("modules/cli"))
+  .settings(
+    commonSettings,
+    name := "cli",
+    libraryDependencies ++= Seq(
+      fs2IO,
+      fs2DataCsv,
+      fs2DataCsvGen,
+      declineCore,
+      declineCatsEffect,
+      cirisCore,
+      cirisHtt4s,
+      ironCiris,
+      logback % Runtime
+    ),
+    Compile / run / fork         := true,
+    Compile / run / connectInput := true,
+    Docker / packageName         := "lenguyenthanh/fide-cli",
+    Docker / maintainer          := "Thanh Le",
+    Docker / dockerRepository    := Some("ghcr.io")
+  )
+  .enablePlugins(JavaAppPackaging, DockerPlugin)
+  .dependsOn(domain, full(db))
+
 lazy val gatling = (project in file("modules/gatling"))
   .settings(name := "gatling")
   .enablePlugins(GatlingPlugin)
@@ -124,9 +149,9 @@ lazy val gatling = (project in file("modules/gatling"))
 lazy val root = project
   .in(file("."))
   .settings(publish := {}, publish / skip := true)
-  .aggregate(types, api, domain, db, crawler, backend, gatling)
+  .aggregate(types, api, domain, db, crawler, backend, cli, gatling)
 
 def full(p: Project) = p % "test->test;compile->compile"
 
-addCommandAlias("lint", "scalafixAll; scalafmtAll; scalafmtSbt")
+addCommandAlias("prepare", "scalafixAll; scalafmtAll; scalafmtSbt")
 addCommandAlias("lintCheck", "; scalafixAll --check ; scalafmtCheckAll; scalafmtSbtCheck")

modules/api/src/main/scala/providers.scala

@@ -1,6 +1,6 @@
 package fide.spec
 
-import fide.types.{ NonEmptySet, PageNumber, PageSize }
+import fide.types.{ NonEmptySet, PageNumber, PageSize, YearMonth }
 import smithy4s.*
 
 object providers:
@@ -25,6 +25,9 @@ object providers:
   given RefinementProvider[FederationIdFormat, String, fide.types.FederationId] =
     Refinement.drivenBy(fide.types.FederationId.either, _.value)
 
+  given RefinementProvider[YearMonthFormat, String, YearMonth] =
+    Refinement.drivenBy(YearMonth.fromString, _.format)
+
   given [A]: RefinementProvider[NonEmptySetFormat, Set[A], NonEmptySet[A]] =
     Refinement.drivenBy[NonEmptySetFormat](
       NonEmptySet.either,

modules/api/src/main/smithy/_global.smithy

@@ -84,6 +84,13 @@ structure PageSizeFormat { }
 )
 structure BirthYearFormat { }
 
+@trait(selector: "string")
+@refinement(
+   targetType: "fide.types.YearMonth"
+   providerImport: "fide.spec.providers.given"
+)
+structure YearMonthFormat {}
+
 @trait(selector: "list")
 @refinement(
   targetType: "fide.types.NonEmptySet",