fix(scoring): rank ckan_find_relevant_datasets by DCAT-AP_IT holder and publisher, not just organization#34
Conversation
|
FYI: the claude-code-action check failed with "Could not fetch an OIDC token" — this is expected for PRs from forks because GitHub Actions denies id-token: write permission to fork-originated workflows for security. Not related to the patch itself. The codebase patch applies cleanly on main. |
…ization
On federated CKAN catalogs (notably dati.gov.it and any portal harvesting
from sub-publishers), the 'organization' field represents the harvesting
catalog, not the data owner. Datasets owned by small/medium Comuni that
publish via aggregators (Regione, GAL, Unione di Comuni, ASL) score 0
on the owner-related field, breaking queries like 'datasets from Comune
di Lecce'.
DCAT-AP_IT distinguishes three distinct concepts:
- dct:publisher -> CKAN field publisher_name
- dct:rightsHolder -> CKAN field holder_name (actual owner)
- catalog membership -> CKAN organization (harvester on aggregators)
scoreDatasetRelevance() now scores holder_name and publisher_name as
two separate weighted fields, additive to the existing scoring. Defaults
holder=4 (peer with title; institutional owner), publisher=2 (lower
weight because it can be a technical role e.g. 'Redazione OD' rather
than the institution).
The fields are read from extras[] (the authoritative DCAT-AP_IT location
on Italian portals via ckanext-dcatapit) with fallback to the root-level
field. On dati.gov.it, package_search exposes holder_name and
publisher_name BOTH in extras[] (correct DCAT-AP_IT values) AND at root
level (often overwritten by the harvester with the organization name);
reading only the root would silently miss the actual owner. The
root-level fallback preserves correct behavior on non-DCAT-AP_IT portals
(data.gov, open.canada.ca) that expose these fields plainly.
Surfaced on real Puglia datasets (validated against live package_search):
- defibrillatori-esterni:
extras.holder='Comune di Mesagne' (correct, DCAT-AP_IT)
root.holder='GAL Terra dei Messapi' (wrong, harvester)
organization='GAL Terra dei Messapi'
score: 6 -> 12
- defibrillatori-dae-progetto-comune-cardioprotetto:
extras.holder='Comune di Lecce'
organization='Regione Puglia'
score: 10 -> 13
Backward compatible: clients that don't set 'holder'/'publisher' in
weights get the improved ranking by default. CkanPackage interface
gains typed holder_name and publisher_name (previously accessed via
index signature). New internal helper readDcatExtra() encapsulates
the extras-first lookup.
30c703a to
cf96e0c
Compare
|
The original commit (v1) was tested on package_show responses and worked, but on package_search results — which is what ckan_find_relevant_datasets actually consumes — holder_name and publisher_name are exposed in two locations that disagree on aggregator catalogs (extras[] vs root level). On dati.gov.it the root-level value is overwritten by the harvester with the catalog's name, while extras[] preserves the authoritative DCAT-AP_IT value. |
Review + additions before mergeThanks for the fix @piersoft — the What was reviewed
What was missing (blocking)Zero tests for What was added (commit
|
2 participants
Fix ckan_find_relevant_datasets scoring on federated CKAN catalogs (notably dati.gov.it). Adds holder and publisher as separate weighted fields in scoreDatasetRelevance(), sourced from DCAT-AP_IT extras[] first (the authoritative location on Italian portals via ckanext-dcatapit), with fallback to root-level for non-DCAT-AP_IT portals. The bug On dati.gov.it, organization in package_search results is the harvesting catalog, not the data owner. A dataset owned by a Comune that publishes via Regione/GAL/Unione gets organization=regione-puglia (or similar), so a query like "datasets from Comune di Lecce" scores 0 on the owner field and misses the actual dct:rightsHolder. Compounding this: package_search exposes holder_name / publisher_name in two places that often disagree: inside extras[] → the authoritative DCAT-AP_IT value (correct owner) at root level → on aggregator catalogs, overwritten by the harvester with the organization name (wrong) Reading only the root field silently misses the real owner. The fix reads from extras[] first. Validated on real datasets (live package_search on dati.gov.it/opendata) Datasetextras.holderroot.holderbeforeafterdefibrillatori-esterniComune di Mesagne ✓GAL Terra dei Messapi ✗612defibrillatori-dae-progetto-comune-cardioprotettoComune di Lecce ✓Regione Puglia ✗1013 Verified end-to-end on the AgID deployment (ckan-mcp-server.agid.workers.dev) after merging the same fix downstream. Compatibility Backward compatible. weights object accepts two new optional fields (holder, publisher); clients that don't set them get the improved ranking by default. CkanPackage interface gains typed holder_name and publisher_name. New internal helper readDcatExtra() encapsulates the extras-first, root-fallback lookup. Score breakdown markdown and JSON outputs include holder and publisher per dataset.