Add evaluation harness and hardness knob for detector benchmarking

[fabio-rovai]

Add an evaluation harness and a hardness knob

Why

The README describes this project as a way to "benchmark graph-based fraud
detection models". Today the generator produces data but ships no way to score a
detector against the ground truth, and the default data is easy to separate by
trivial heuristics: every fraud edge carries the exact same sentinel amount
(9999.00), rings are disjoint, and there are no legitimate cycles. A naive
"flag every account on a high-value cycle" baseline scores near-perfectly on the
defaults, so the generator cannot currently tell a good detector from a bad one.

This PR closes both gaps, additively and backward compatibly (defaults
unchanged).

What

1. Evaluation harness (evaluate.py, new).
Given a generated dataset directory and a set of detector-flagged account ids (a
plain list or a CSV), it computes precision, recall and F1 at two granularities:

• account level (did the detector flag the accounts that take part in a ring), and
• ring level (a ring counts as detected when at least ring_threshold of its
  accounts are flagged; default 1.0 = all).

It reads ground truth from fraud/fraud_cases.csv, reports a confusion summary
(with true negatives when an accounts/ directory is present), and exposes a CLI
via a new gen-fraud-graph-evaluate console script (--data, --flagged,
--ring-threshold, --json).

2. Hardness knob (--hardness {low,medium,high}).
A difficulty preset, wired through Config, that makes the data harder to
separate by trivial heuristics:

• amount_jitter: spread fraud amounts around the sentinel so the amount alone
  is no longer a giveaway,
• ring_overlap: let rings share accounts (overlapping rings, not just disjoint
  ones),
• decoy_ratio: inject legitimate high-value cycles into the normal transaction
  stream (transactions/transactions_decoy.csv, never written to
  fraud_cases.csv), so pure amount-thresholding and pure cycle-topology each
  fail.

low (the default) reproduces the original behaviour exactly, so existing users
see no change.

Tests

tests/test_evaluate.py adds 12 tests: exact precision/recall/F1 on a hand-built
fixture, ring-threshold behaviour, the loaders, an end-to-end evaluate_dataset,
the hardness presets, and a generator smoke test asserting that high hardness
writes decoy cycles and jitters fraud amounts while low stays backward
compatible. The full suite (54 tests) passes.

Notes

• Backward compatible: defaults are unchanged, no existing module or output is
  modified.
• Contributor License Agreement: I understand this repository requires signing
  the Santander CLA before merge, and I will sign it on this PR.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[fabio-rovai]

recheck

[opensource-SantanderAI]

Thanks @fabio-rovai — this is genuinely valuable: the repo advertises detector benchmarking but ships no scoring and trivially-separable defaults, and the evaluation harness + hardness knob close exactly that gap. The 12 tests in test_evaluate.py are real (32 assertions) and all pass. Two CI gates are red and need fixing before we merge:

1. Coverage gate (Test (Python 3.10/3.11/3.12) fail). Required coverage is 90%; this PR lands at 88.6%. The new src/gen_fraud_graph/evaluate.py is only 56% covered (62/140 lines) — the untested parts are mainly the CLI/main entrypoint and the result/output formatting (roughly lines 85-101, 119-129, 275-318, 323-343, 348-352). Adding tests that exercise the CLI path and the printed/structured output should clear the 90% gate.

2. Lint gate (Lint & format & type-check fail on Black --check). Black reports the code is formatted for 3.12 and fails to verify under 3.11. Please run Black/ruff with the project's supported target (--target-version py310/py311) and push, e.g. ruff check --fix . && black . from a 3.10/3.11 environment (or set target-version in pyproject.toml).

No hidden issues beyond these — once coverage is ≥90% and lint is green, we'll be glad to merge. Thanks!

[opensource-SantanderAI]

Thanks @fabio-rovai — this is in great shape now. The earlier review points are fully addressed: coverage is at 98% and Black/lint is green, and the full CI suite passes (Lint, Test 3.10/3.11/3.12, CodeQL, license allowlist, pip-audit, SPDX, pattern scan). 🎉

There's just one thing left before we can merge: main has moved on since you branched, and the PR now has merge conflicts in two files:

• src/gen_fraud_graph/generator.py
• src/gen_fraud_graph/typologies.py

Since these are logic files that both your branch and main changed, we'd rather you resolve them (you know the intent of your changes best) than risk us merging them incorrectly. Could you please:

1. Rebase your branch onto the latest main (git fetch upstream && git rebase upstream/main), or merge main into your branch.
2. Resolve the conflicts in generator.py and typologies.py.
3. Re-run the tests locally to confirm coverage/lint stay green, and push.

Once it's conflict-free and CI is green again, ping us and we'll re-review and merge right away. Thanks again for the solid work on the evaluation harness + hardness knob!

[fabio-rovai]

Rebased onto latest main and resolved the conflicts in generator.py and typologies.py.

The conflict was semantic rather than textual: #14 made each ring's accounts disjoint, which collided with this PR's ring_overlap hardness feature. I resolved it by keeping #14's pre-allocated distinct-account pool as the default and making overlap opt-in only when ring_overlap > 0. So at default/low hardness every ring stays fully disjoint (test_rings_use_disjoint_accounts still passes), while the hardness knob's deliberate overlap works at medium/high.

Local verification (mirrors CI): ruff check clean, black --check clean, mypy src/ clean, 65 tests pass, coverage 98.27%. Conflict-free and ready for re-review, thanks!

[opensource-SantanderAI]

recheck

[fabio-rovai]

Updated: merged latest main (conflict-free), re-authored the merge commit so CLA passes, and full CI is green — Lint, Test 3.10/3.11/3.12, CodeQL, license allowlist, pip-audit, SPDX. Coverage 98.3%. Ready for re-review, thanks!

[opensource-SantanderAI]

All previously requested changes are fully addressed: coverage at 98.3%, Black/ruff/mypy clean, the semantic conflict with #14 (disjoint rings vs. ring_overlap) resolved by making overlap opt-in, and the full CI suite is green on the updated branch (Lint, Test 3.10/3.11/3.12, CodeQL, license allowlist, pip-audit, SPDX, pattern scan). This closes the repo's core gap — detector scoring (P/R/F1) plus a hardness knob for non-trivial data. Approving for merge. Thanks @fabio-rovai for the thorough iteration!