Issue #1155: add robots.txt to both frontends; widen CloudFront invalidation

[bjagg]

Description of Change

Analytics is showing crawler/scan traffic against the dev and demo hosts, and neither frontend served a robots.txt.

Because both frontends use SPA fallback, /robots.txt currently returns index.html with HTTP 200, not a 404 — try_files $uri /index.html in frontends/*/nginx.conf, and cloudformation/s3-hosted.yml:82-88 mapping CloudFront 403/404 to /index.html with ResponseCode: 200. A crawler probing arbitrary paths gets a valid HTML page every time.

Adds public/robots.txt to both frontends, disallowing all user-agents. These hosts serve non-production evaluation environments; nothing here belongs in a search index, so there's no SEO value to weigh against a blanket disallow.

One file per frontend covers both delivery paths, because Vite copies public/ into dist/ and both paths serve the build output:

Frontend	Delivery	Picks up dist/robots.txt via
mdr-frontend	S3 + CloudFront	aws s3 sync --delete dist s3://...
lif_advisor_app	ECS / nginx	root /usr/share/nginx/html + try_files $uri

Verified by building mdr-frontend and confirming dist/robots.txt (590 bytes) lands in the sync root.

Also widens the CloudFront invalidation from --paths "/" to --paths "/*" in .github/workflows/lif_mdr_frontend.yml. Without this the new file would not become visible: the deploy replaces the whole object set via s3 sync --delete, but stable-named files (index.html, robots.txt) keep serving from the edge cache. Hashed asset filenames cache-bust themselves; these do not. Happy to split this into its own PR if you'd rather keep them separate — but as it stands the robots.txt wouldn't take effect on demo without it.

Two deliberate choices

No path enumeration. The file lists no routes. A robots.txt enumerating paths advertises them to exactly the scanners this is meant to discourage.

An explicit note that it's advisory. Each file records that well-behaved crawlers honor robots.txt and scanners ignore it, so it isn't later mistaken for a security control.

What this does not fix — worth reading before closing #1155

robots.txt only influences well-behaved crawlers. If the analytics traffic is probing for /.env, /wp-admin, /.git/config and similar, this change will not reduce it at all — that needs WAF rules, rate limiting, or blocking at CloudFront/ALB. Worth confirming from the analytics data which pattern this actually is, since that determines whether this is the fix or just hygiene.

Separately flagged in #1155: both the MDR API and the LDE API construct FastAPI(...) with defaults and explicitly allowlist /docs and /openapi.json unauthenticated, so the full API surface is publicly readable on dev/demo. A published OpenAPI spec is a high-value scanning target, and robots.txt does nothing for it. That deserves its own decision rather than being bundled here.

Related Issues

Closes #1155

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• Infrastructure/deployment change

Project Area(s) Affected

• frontends/
• Documentation (docs/, READMEs, ARCHITECTURE.md, CLAUDE.md)

---

Checklist

• commit message follows commit guidelines (see commitlint.config.mjs)
• pre-commit hooks have been run successfully

Testing

• Manual testing performed

Check	Result
npm run build (mdr-frontend)	built in 1.12s
dist/robots.txt present in sync root	yes, 590 bytes
publicDir overridden in either vite config	no — default public/ → dist/
pre-commit run --files <3 changed>	cspell, ty, pytest all passed

Not verified end-to-end against a deployed host — that needs a deploy. After merge, confirm curl -sI https://mdr.dev.lif.unicon.net/robots.txt returns text/plain rather than text/html.

Additional Notes

Staged three explicit paths only; the working tree has unrelated untracked files. dist/ is gitignored, so no build output is included.