Web export: regions, review NLP, charts, points sample and overseas fix - #8
Merged
Conversation
- expose score dimensions, gap and dpe_dominant in the communes choropleth - publish v1/score.geojson compatibility artifact (deprecated, see ADR-0014) - remove standalone webapp_export script, superseded by publish-web
Add .worktrees/ to the repo root .gitignore and a package-level gitignore for __pycache__/*.pyc; remove the accidentally tracked bytecode (including scraper.cpython-312.pyc whose source no longer exists). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkEqwLFdScx8ivxMomxYCK
Replace the tls_client top-80 scraper with a curl_cffi one that impersonates Chrome (matching TLS + HTTP/2 fingerprints, which anti-bot JA4H checks) and paginates each commune to exhaustion. Fix two parsing bugs that capped the corpus: - the last review of every page was dropped (block regex required a trailing <div>); bound each block to the next comment or the section end instead; - only the first category rating was captured (DOTALL .*? spilled across table rows); match each row independently. Add a France-wide driver (resume, gaussian delay, ban detection) and a GitHub Actions matrix workflow that shards the commune list across 20 runners for free IP rotation, then merges the shards. Parsing is pure and unit-tested offline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkEqwLFdScx8ivxMomxYCK
Add homepedia_ville_ideale.nlp, a batch step turning scraped reviews into three silver Parquet tables (avis / segments / tokens) consumed by duckpipe. Hybrid sentiment: structural polarity (points_positifs=+1 / points_negatifs=-1) refined per segment by cmarkea/distilcamembert-base-sentiment (5 stars mapped to [-1,1]). Themes via a spaCy PhraseMatcher lexicon (including 'Calme', absent from ville-ideale rating categories). Word-cloud tokens from nouns/adjectives/place entities. The transformer is optional (--no-model falls back to the damped prior), keeping the package testable offline. Author is only hashed into avis_id, never written (anonymization). Heavy deps (spaCy/torch/transformers) live in the ville_ideale package under the nlp extra, out of the duckpipe Cloud Run image; torch pinned to the CPU wheel index. Calibration on 612 reviews: Pearson 0.80 / Spearman 0.80 vs note_moyenne. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkEqwLFdScx8ivxMomxYCK
…ommune Add the 'avis' pipeline: one SQL node aggregating the silver avis tables into gold avis_commune, one row per commune with everything the mockup's text-analysis screen needs — global sentiment (one review one vote), per-theme sentiment (masked below 10 reviews), a colored word cloud deduped per review, and labeled verbatims (Positif/Negatif/Nuance). Deterministic ordering throughout (ADR-0008). Register silver avis/segments/tokens and gold avis_commune paths, add silver DQ rules and a tolerant validate_gold_avis wired into the CLI (skipped when the NLP step has not produced the table). Cover the aggregation math and an end-to-end run (real NLP subprocess) with tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkEqwLFdScx8ivxMomxYCK
Add build_avis (web_avis table, per-departement avis/{dept}.json artifacts) with
per-theme sentiment masked for low-data communes, embed a review preview struct
into the commune fiches, and provide a typed stub so publish-web still runs when
the NLP step never produced avis_commune. Bump meta.json schema_version to 2 and
add nb_communes_avis so the webapp can feature-gate the Avis tab.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkEqwLFdScx8ivxMomxYCK
Add nlp-avis.yml (uv + HF cache + WIF) to build the silver avis Parquet on GCS when the reviews CSV changes; grant the deployer SA object write on the data bucket. Insert a tolerant avis_gold step into the Cloud Workflows DAG after score, and add a ville_ideale CI job running the offline test suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkEqwLFdScx8ivxMomxYCK
# Conflicts: # duckpipe/tests/test_cli_end_to_end.py # duckpipe/tests/test_export_web.py
- add Etalab regions contours source and region_geom_1000m silver - build_choropleth_regions: medians recomputed from DVF transactions, score aggregates per region; departements expose code_region/nom_region - publish points/transactions-sample.json (reservoir sample, repeatable) - port webapp_export/export_charts.py into publish-web (stats_communes, prix_distribution, prix_series built from commune_prix millesimes) - extend WEB_MILLESIMES to 2021-2025 and add millesimes step to the DAG - keep meta schema_version at 1: all additions are additive and the deployed front pins z.literal(1); gate avis on nb_communes_avis
DuckDB lpad(code, 2, '0') follows the PostgreSQL semantics and TRUNCATES strings longer than 2 characters: every DROM department (971-976) and overseas collectivity collapsed into a single '97' bucket. Symptoms: the departement choropleth showed one merged aggregate for all overseas territories and the regional mesh had no data for the five DROM regions. Padding now only applies to codes shorter than 2 characters.
Departement and region choropleths gain score_median, gap_pondere_median, nb_communes_scorees and the medians of the 12 normalized dimensions (same n_* property names as the commune mesh) so the analysis map can switch meshes with zoom without any client-side adaptation.
Lets the map place review markers from the CDN instead of the wordcloud dataset committed in the webapp repo.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Quoi
Étend
publish-web(inclut #7 et la chaîne avis defeat/avis-sentiment) pour servir toutes les visualisations de la webapp :choropleth/regions-low.geojsonetstats/regions.json— médianes recalculées depuis les transactions DVF (pas de moyenne de médianes), rattachementcode_regionsur les départements.n_*(mêmes noms qu'au niveau communal) sur les départements et régions → maille adaptative de la carte Qualité de vie sans adaptation côté front.avis_commune,avis/{dept}.json) +avis/index.json(centres des communes couvertes, marqueurs de carte sans données locales committées côté webapp).charts/intégrés àpublish-web(stats communes, distribution des prix, séries 2021→2025 viaWEB_MILLESIMESétendu + étape millésimes dans le DAG).points/transactions-sample.json: échantillon reproductible de ~100k mutations pour la heatmap.lpad(code, 2)de DuckDB tronque « 971 » en « 97 » (sémantique PostgreSQL) — tous les DROM étaient fusionnés dans un seul agrégat départemental et absents de la maille régionale. Normalisation sans troncature partagée (pipelines/codes.py), silvers et gold retraités.meta.schema_versionreste à 1 : tous les ajouts sont additifs et le front déployé validez.literal(1); la section avis se gate surnb_communes_avis.État
Le bucket
gs://homepedia-web/v1sert déjà le run2026-07-07produit par cette branche (consommé par la webapp mergée dans T-DAT-902-Homepedia/webapp#39).Tests
ruff+pytestverts (50 tests hors-ligne, e2e local complet 2/2 avecexploration/data/raw).🤖 Generated with Claude Code