This is an unsupervised pipeline that takes raw transaction and account data and clusters the accounts on 2 stages.
- Creates many micro clusters of the accounts on the basis of the behaviour feature with HBDSCAN - based on the density.
- Then, does the meta-clustering merging the similar micro clusters(cosine similarity) using Agglomerative algorithm.
All the docs related to our project model(ppt, documents, CHANGELOG) is availale in the docs folder.
graph_edges.csv
row_index,Sender_account,Receiver_account,amount_local_npr,Date,
Time
ml_features.csv
Sender_account, Receiver_account, amount_local_npr, log_amount,
amount_zscore, above_1M_NPR, above_10M_NPR, cross_border_flag,
currency_mismatch, velocity_sum_10tx, tx_count_10, tx_count_30,
sender_country_risk, receiver_country_risk, sender_pep, sender_sanctions,
receiver_pep, receiver_sanctions, sender_account_age_days, is_weekend,
hour_of_day
transactions.csv
Same as ml_features.csv. Used only for validation labels (is_suspicious_tx).
accounts.csv
account_id, acct_type, risk_grade, is_person, pep_flag,
sanctions_hit, city, opened
Stage I
All the data files are loaded. Every account becomes a dot, and every payment
becomes an arrow between dots — turning raw transactions into a network of accounts
(nodes) and transfers (edges). Graph metrics are then calculated on this network:
how connected each account is, whether money flows through it like a
stepping-stone, and whether it sends in loops back to itself.
Stage II
On top of the network numbers, it adds behaviorals: how big are this account's
typical transfers, how often does it send money abroad, how often to risky
countries, what fraction of its activity happens at night, is it linked to a
politically exposed person. Now every account has a full profile.
Merges graph metrics, transaction behavior and account metadata into one big table.
Stage III
HDBSCAN groups accounts with similar behaviors and then form the micro-clusters.
Accounts that don't fit in any cluster become noise.
First pass: an algorithm called HDBSCAN looks at all those account profiles and
notices which accounts are statistically close to each other — without anyone
telling it how many groups to expect. It forms many micro clusters.
Second pass: Using Agglomerative clustering and Cosine similarity, merges similar
micro-clusters — comparing the pattern of behavior. It keeps merging until a
sensible number of distinct meta-clusters are not formed.
Stage IV
Each meta-cluster is compared against the entire population of accounts. If a
group's typical values — say, transaction velocity or sanctions exposure — sit way
above where most accounts sit, its risk score climbs. Features that matter more for
money laundering (like sanctions or circular money flows) count for more than ones
that matter less.
Each cluster then gets:
— A risk score (percentile-based ranking against all other clusters)
— A risk tier (CRITICAL / HIGH / ELEVATED / NORMAL)
— An auto-generated name and description, built from whichever features pushed its
score up the most
Every account inherits its cluster's name, tier, and score.
Stage V
An account is flagged as suspicious if it belongs to a HIGH/CRITICAL archetype, OR
if it was separately flagged as a statistical anomaly by IsolationForest — either
reason alone is enough.
Stage VI
Each individual transaction is also scored on its own — by amount, timing, and
whether it crossed a border — using a second anomaly detector. Transactions sent by
already-flagged accounts go through a stricter version of this check, since
"unusual for a risky account" is a different bar than "unusual for an ordinary
account."
Stage VII
At the transaction level, it checks how well each flag actually catches real cases,
using precision, recall, and AUC — since real laundering is so rare, recall and
lift matter more here than plain accuracy.
At the archetype level — it checks whether CRITICAL/HIGH clusters actually contain
more real laundering than NORMAL ones, whether the auto-generated risk score lines
up with the real suspicious rate, and flags a warning.
Python 3.10 or higher is required.
Install all dependencies with:
pip install -r requirements.txtThe key packages the model actually depends on are:
hdbscan — micro-clustering
umap-learn — 2D embedding for visualisation
scikit-learn — agglomerative clustering, IsolationForest, scalers, validation metrics
networkx — building and analysing the transaction graph
pandas — data loading and feature engineering
numpy — numerical operations
matplotlib — plot generation
seaborn — embedding scatter plots
joblib — saving and loading model artifacts
scipy — distance matrix computation
numba — required by hdbscan and umap-learn for performance
Everything else in requirements.txt is a supporting dependency pulled in automatically.
The run_model.ipynb file has the staged working flow in series that u can check.
If you have a new batch of data and want to evaluate it against a baseline, use fit_combined(). It merges both datasets, deduplicates, and tags every record so you can tell base data from new data after fitting:
from model import ArchModel
from config import DATA_DIR, MODEL_DIR, OUT_DIR
model = ArchModel.fit_combined(
base_data_dir=DATA_DIR, # your original/baseline CSVs
new_data_dir=NEW_DATA_DIR, # the new batch to evaluate
random_seed=27,
target_n_accounts=60_000, # pads new batch from base if too small; pass None to disable
)
model.validate()
model.save(MODEL_DIR)
model.save_outputs(OUT_DIR)
model.plot(OUT_DIR)By combined datasets, it means the model trains on both the baseline data and the new data together. This is because the new dataset alone may not have enough records for the clustering to be effective. If the new dataset falls below a set account threshold, the shortfall is automatically filled in from the baseline data. So if you only want to test a single new dataset, try to make it reasonably large or set
target_n_accounts=Noneto disable the top-up entirely.
After fitting, every row in model.df and model.txns_flagged carries:
is_new_data—1for the new batch,0for base datatopped_up_from_base—1if the row was borrowed from the base dataset to pad a thin new batch
You can then filter results to the new batch only:
new_accounts = model.new_data_accounts() # account-level results, new batch only
new_txns = model.new_data_transactions() # transaction-level results, new batch only# Top 20 by archetype risk tier
top = model.top_suspicious_accounts(n=20)
# Top 20 by either archetype flag OR anomaly detection
top_flagged = model.top_suspicious_accounts(n=20, by="account_flagged") Calling model.save(model_dir) saves the model itself — the fitted imputer and
scaler, the archetype profiles, the transaction anomaly models, the
account-level IsolationForest, the cluster map, and a meta.json with all the
feature lists and settings. Basically everything needed to load the model
back up and score new data without retraining.
model.save(model_dir) saves everything needed to reload and reuse the model
without retraining:
imputer.pkl — cleans up missing values in account features
scaler.pkl — normalises account feature ranges
archetype_profiles.pkl — the profile table for every cluster
account_iso.pkl — anomaly detector for accounts
cluster_map.json — which micro-clusters merged into which meta-cluster
tx_iso.pkl — anomaly detector for transactions (all senders)
tx_iso_high_risk.pkl — stricter anomaly detector for high-risk senders (saved if enough rows)
tx_imputer.pkl — cleans up missing values in transaction features
tx_scaler.pkl — normalises transaction feature ranges
meta.json — feature lists, algorithm name, and other settings
Calling model.save_outputs(out_dir) exports the results as human-readable files: accounts_flagged.csv (one row per account, with archetype name, risk tier, and flag columns), transactions_flagged.csv (one row per transaction, with anomaly scores and flags), archetype_profiles.csv (a summary of each archetype), and summary.json (high-level counts plus the full archetype list sorted by risk score).
model.save_outputs(out_dir) exports the results as readable files:
accounts_flagged.csv — one row per account: archetype name, risk tier, flag columns
transactions_flagged.csv — one row per transaction: anomaly scores and flags
archetype_profiles.csv — summary stats for each archetype
summary.json — high-level counts and full archetype list sorted by risk score
model.plot(out_dir) generates the following plots:
network_subgraph.png — transaction network around flagged accounts
archetype_embedding.png — 2D scatter of all accounts, coloured by archetype
flagged_embedding.png — same scatter, model-flagged vs normal
suspicious_embedding.png — same scatter, ground-truth suspicious accounts (validation only)
model_vs_truth_embedding.png — model flags vs ground truth side by side (validation only)
archetype_risk_scores.png — bar chart of risk score per archetype, coloured by tier
archetype_cards.json — archetype names, descriptions, and colours for display
Note: This model saves the reports and data outputs but they cannot be loaded back to for training. The saved files are for reviewing results only — to score new data or retrain, you always start from the raw CSVs. (This feature couldnot be added due to lack of time.)
Don't pass the wrong transactions file. When scoring, make sure you're passing ml_features.csv, not transactions.csv. The model needs the engineered feature table — if you give it raw transactions, it'll fail with confusing missing-column errors that are hard to trace back to this.
Risk tiers change if the data changes. The tiers are assigned by ranking clusters against each other within a single training run. If you retrain on a very different dataset, the same cluster might show a different risk tier. If you're comparing risk scores across model versions, use the raw risk_score number, not the tier label.
Do not consider high noise necessarily as a problem. HDBSCAN prints the noise percentage after fitting. If it goes above 30%, review your data before shrinking HDBSCAN_MIN_CLUSTER_SIZE — smaller clusters mean less data per archetype, which makes the risk scores less reliable.
The number of archetypes varies each run. The pipeline searches for the best number of meta-clusters between 8 and 20 (configurable via META_CLUSTER_MIN and META_CLUSTER_MAX in config.py) and picks whichever gives the cleanest separation. You won't always get the same count.
The pre-given is_suspicious_tx label has its own purpose. is_suspicious_tx is present in data, it's only used by validate() to report how well the model did — nothing in the clustering, scoring, or flagging logic.