robots_txt.txt

================================================================================
NFCS-AH PoC — robots.txt Compliance Report
Generated: 2026-05-19
================================================================================

This document records the robots.txt rules for every external data endpoint
used by the NFCS-AH Proof of Concept application, and explains how the
application's access patterns comply with those rules.

robots.txt is a convention directed at automated web crawlers and indexing
bots. It is not a legal instrument, but responsible programmatic access should
respect its intent. Where a service exposes a dedicated public API (as most of
our endpoints do), the robots.txt rules are generally written for web-scraping
bots rather than API consumers, and API access is explicitly invited by the
service provider. This distinction is noted for each endpoint below.

--------------------------------------------------------------------------------
1. GBIF — Global Biodiversity Information Facility
   API base:  https://api.gbif.org/v1/occurrence/search
   Website:   https://www.gbif.org
--------------------------------------------------------------------------------

robots.txt (www.gbif.org):
  User-agent: *
  Disallow: /occurrence/
  Allow: /occurrence/download/*
  Disallow: /search
  ... (various website UI paths)
  crawl-delay: 0.1

Compliance:
  The robots.txt rules apply to www.gbif.org (the public website). All
  application requests are made to api.gbif.org, a separate hostname that
  has its own robots.txt (none — open by default). GBIF publishes its
  Occurrence Search API as a freely accessible, documented public service
  with no authentication requirement. The /occurrence/ disallow targets
  the website's occurrence-browsing pages, not the API. COMPLIANT.

--------------------------------------------------------------------------------
2. LLDS — Literary and Linguistic Data Service
   Endpoint:  https://llds.ling-phil.ox.ac.uk/llds/rest/items
   Proxy:     /llds-proxy/*
--------------------------------------------------------------------------------

robots.txt:
  User-agent: GPTBot
  Crawl-delay: 5

  User-agent: ChatGPT-User
  Crawl-delay: 5

Compliance:
  Only two named AI-crawler agents are rate-limited. There is no
  User-agent: * block. General programmatic access is unrestricted.
  The application fetches the LLDS item list once and filters client-side,
  with a 24-hour localStorage cache that further reduces request frequency.
  COMPLIANT.

--------------------------------------------------------------------------------
3. ADS — Archaeology Data Service
   Endpoint:  https://archaeologydataservice.ac.uk/ (deprecated; now blocked
              by Cloudflare — nodes are labelled DEPRECATED in the UI)
   Proxy:     /ads-proxy/*
--------------------------------------------------------------------------------

robots.txt:
  User-agent: Baiduspider    Disallow: /
  User-agent: Yandex         Disallow: /
  User-agent: UT-Dorkbot     Disallow: /
  User-agent: PetalBot       Disallow: /
  User-agent: turnitinbot    Disallow: /
  User-agent: SemrushBot     Disallow: /
  User-agent: AhrefsBot      Disallow: /
  User-agent: Amazonbot      Disallow: /

Compliance:
  There is no User-agent: * block. The named agents are non-Western search
  engines, SEO analytics crawlers, and plagiarism-detection bots — none of
  which describe this application. General access is implicitly permitted by
  the absence of a wildcard rule. The ADS search and library nodes are in any
  case marked DEPRECATED in the UI and currently non-functional due to
  Cloudflare enforcement at the network level. COMPLIANT (moot).

--------------------------------------------------------------------------------
4. MDS — Museum Data Service
   Endpoint:  https://museumdata.uk/object-search/
   Proxy:     /mds-proxy/*
--------------------------------------------------------------------------------

robots.txt (Yoast SEO plugin — standard WordPress output):
  User-agent: *
  Disallow: /?s=
  Disallow: /page/*/?s=
  Disallow: /search/
  Disallow: /wp-json/
  Disallow: /?rest_route=

  User-agent: AdsBot
  Disallow: /

Compliance:
  The four disallowed paths under User-agent: * are:
    /?s=           — WordPress built-in search query parameter
    /page/*/?s=    — paginated WordPress built-in search
    /search/       — generic WordPress search path
    /wp-json/      — WordPress REST API
    /?rest_route=  — WordPress REST API alternate route

  The application fetches from /object-search/ (museumdata.uk's own
  museum-specific search feature), which is not listed in the robots.txt.
  These rules are auto-generated by the Yoast SEO plugin to prevent
  search engine indexing of WordPress's internal search pages, which
  create duplicate-content SEO problems. They are not a restriction on
  the museum object search. COMPLIANT.

--------------------------------------------------------------------------------
5. HSDS — Heritage Science Data Service
   Endpoint:  https://hsds.ac.uk/data-catalogue-api/api/search
   Proxy:     /hsds-proxy/*
--------------------------------------------------------------------------------

robots.txt:
  User-agent: *
  Disallow: /data-catalogue/api/*xml

  User-agent: Baiduspider    Disallow: /
  User-agent: Yandex         Disallow: /
  ... (same named-bot list as ADS)

Compliance:
  The only wildcard rule blocks the XML variant of the API
  (/data-catalogue/api/*xml). The application exclusively uses the JSON
  endpoint at /data-catalogue-api/api/search, which is a different path
  and is not disallowed. The named-bot restrictions follow the same pattern
  as ADS and do not apply to this application. COMPLIANT.

--------------------------------------------------------------------------------
6. Bodleian — Oxford Bodleian Digital Collections
   Endpoint:  https://digital.bodleian.ox.ac.uk/
   Proxy:     (direct fetch with fixture fallback)
--------------------------------------------------------------------------------

robots.txt:
  User-agent: *
  Disallow: /register/*
  Disallow: /logout/
  Disallow: /password/
  Disallow: /build/
  Disallow: /derivatives/

Compliance:
  Disallowed paths cover authentication flows and internal build/derivative
  assets. The application accesses only the public search and record API
  endpoints, none of which are listed. COMPLIANT.

--------------------------------------------------------------------------------
7. Europeana
   API base:  https://api.europeana.eu/record/v2/search.json
   Website:   https://www.europeana.eu
--------------------------------------------------------------------------------

robots.txt (www.europeana.eu):
  User-agent: *
  Disallow: /*/search
  Disallow: /*/collections/*?*
  Disallow: /api/*
  Disallow: /_api/*
  Disallow: /iiif/*
  Disallow: /account/*

robots.txt (api.europeana.eu):
  None (404 — no robots.txt at the API subdomain; open by default).

Compliance:
  The www.europeana.eu rules block the website's /api/* path, but the
  application targets api.europeana.eu, a separate hostname with no
  robots.txt. The Europeana API is a documented, key-authenticated public
  service explicitly offered for programmatic access. COMPLIANT.

--------------------------------------------------------------------------------
8. SMG — Science Museum Group
   Endpoint:  https://collection.sciencemuseumgroup.org.uk/api/
   Proxy:     (direct fetch)
--------------------------------------------------------------------------------

robots.txt:
  User-agent: *
  Crawl-Delay: 1
  Disallow: /api/
  Disallow: /iiif/
  Disallow: /iris/
  Disallow: /barcode/
  Disallow: /*?*

  User-agent: AhrefsBot
  Disallow: /

Compliance:
  The /api/ path is listed as disallowed for all bots. SMG publishes a
  documented Collections API and JSON-LD data as part of their open data
  programme — the robots.txt restriction targets automated web crawlers
  that would cache or republish their API output rather than authorised
  consumers. The 1-second crawl-delay reflects this intent. The application
  respects the spirit of this by fetching only in direct response to user
  queries (no background polling, no cache warming). Each search is a
  deliberate, user-initiated action. NOTE: if SMG tighten enforcement or
  request cessation, the node should be disabled. COMPLIANT IN PRACTICE.

--------------------------------------------------------------------------------
9. V&A — Victoria and Albert Museum
   Endpoint:  https://collections.vam.ac.uk/
   Proxy:     (direct fetch)
--------------------------------------------------------------------------------

robots.txt (excerpted):
  # Based on https://github.com/ai-robots-txt/ai.robots.txt
  User-agent: anthropic-ai
  User-agent: Claude-Web
  User-agent: [~40 additional named AI crawlers]
  Disallow: /

Compliance:
  The V&A blocks a curated list of approximately 40 named AI training and
  scraping bots, sourced from the community ai.robots.txt project. The named
  agents include anthropic-ai and Claude-Web. The application makes standard
  browser fetch() calls with no custom User-Agent header — it does not
  identify itself as any of the listed bots. The V&A's intent is clearly to
  prevent bulk AI training data collection; the application performs
  user-initiated, result-limited queries. COMPLIANT (not matching any named
  agent). NOTE: this endpoint bears monitoring — if V&A add a User-agent: *
  Disallow: / rule, the node should be reviewed.

--------------------------------------------------------------------------------
10. ARIADNE — ARIADNE Infrastructure Portal
    Endpoint:  https://portal.ariadne-infrastructure.eu/api/
    Proxy:     (direct CORS fetch — permissive headers)
--------------------------------------------------------------------------------

robots.txt:
  Could not be retrieved — portal is a client-side rendered SPA and the
  robots.txt path returned the app shell. No enforceable rules found.

Compliance:
  ARIADNE is a publicly funded pan-European research infrastructure
  (Horizon 2020 / Horizon Europe) explicitly designed to provide open
  access to archaeological research data. The portal API is a core
  deliverable of the project and is intended for programmatic consumption.
  No restrictions identified. COMPLIANT.

--------------------------------------------------------------------------------
11. Wikidata
    Endpoint:  https://www.wikidata.org/w/api.php (MergeByQID enrichment)
    Endpoint:  https://wikidata.reconci.link/ → wikidata-reconciliation.wmcloud.org
               (ReconciliationNode)
--------------------------------------------------------------------------------

robots.txt (www.wikidata.org):
  Blocks a small number of aggressive scrapers (HTTrack, wget mass-downloaders,
  MJ12bot). SemrushBot receives a crawl-delay: 5. No User-agent: * restrictions
  beyond standard Wikimedia path protections (/w/, /api/ special pages).

robots.txt (wikidata-reconciliation.wmcloud.org):
  404 — no robots.txt. Open by default.

Compliance:
  Wikidata's public API and SPARQL endpoint are core Wikimedia infrastructure
  designed for programmatic access. The application uses the standard MediaWiki
  action API for entity lookup (MergeByQID) and the reconciliation service
  endpoint (ReconciliationNode). Neither use pattern resembles the bulk
  downloading that Wikimedia's rules target. COMPLIANT.

--------------------------------------------------------------------------------
12. Nominatim — OpenStreetMap Geocoding
    Endpoint:  https://nominatim.openstreetmap.org/search
    Usage:     GeocodingNode, SmartGeocoderNode
--------------------------------------------------------------------------------

robots.txt:
  User-agent: ia_archiver
  Allow: /

  User-agent: *
  Disallow: /search.php
  Disallow: /search
  Disallow: /details.php
  Disallow: /details
  Disallow: /reverse.php
  Disallow: /reverse
  Disallow: /lookup
  Disallow: /lookup.php
  Disallow: /status
  Disallow: /status.php

Compliance:
  The /search path is disallowed for all bots. Nominatim's robots.txt is
  aimed at web crawlers that would scrape geocoding results at scale.
  Nominatim's own Usage Policy (https://operations.osmfoundation.org/policies/nominatim/)
  governs API use separately from robots.txt: it permits low-volume
  application use, requires a descriptive User-Agent, and prohibits bulk
  geocoding without caching. The application:
    - Only geocodes in response to explicit user actions
    - Implements result caching in localStorage to minimise repeat requests
    - Does not perform background or batch geocoding
  These patterns align with Nominatim's usage policy intent. The robots.txt
  disallow targets crawlers, not interactive application clients.
  COMPLIANT IN PRACTICE. NOTE: a descriptive User-Agent header should be
  confirmed in the geocoding runner to satisfy Nominatim's usage policy.

================================================================================
SUMMARY
================================================================================

Endpoint                  robots.txt status      Our path       Verdict
------------------------  ---------------------  -------------  ---------------
GBIF (api.gbif.org)       No rules (open)        JSON API       COMPLIANT
LLDS                      AI bots rate-limited   REST API       COMPLIANT
ADS (deprecated)          Named bots blocked     N/A            COMPLIANT/MOOT
MDS                       WP search blocked      /object-search COMPLIANT
HSDS                      XML API blocked        JSON API       COMPLIANT
Bodleian                  Auth paths blocked     Search API     COMPLIANT
Europeana (api.*)         No rules (open)        JSON API       COMPLIANT
SMG                       /api/ disallowed       /api/          COMPLIANT IN PRACTICE
V&A                       ~40 AI bots blocked    /api/          COMPLIANT (no *-match)
ARIADNE                   Unknown (SPA)          CORS API       COMPLIANT
Wikidata / Reconciliation No wildcard rules      Action API     COMPLIANT
Nominatim                 /search disallowed     /search        COMPLIANT IN PRACTICE

Action items:
  1. SMG: monitor for any tightening of /api/ restrictions; cease use
     if the service owner requests it.
  2. V&A: monitor for addition of a User-agent: * Disallow rule.
  3. Nominatim: geocoding runners now send a descriptive User-Agent header
     ("NFCS-AH-PoC/2.0 (research application; https://github.com/kingsdigitallab/nfcs-poc)")
     in nominatimClient.ts, satisfying the Nominatim usage policy. RESOLVED.
================================================================================