Skip to content

Commit 53cce7d

Browse files
committed
Move the generic extractor out of event-extractors/
extension/event-extractors/ is the extensibility point for PER-SITE extractors. The generic extractor is a fundamentally different thing — one fixed, site-agnostic reader that every per-site extractor layers over, not another entry in the set — so it doesn't belong in that folder: event-extractors/core/generic.js -> generic-extractor.js event-extractors/core/generic-sites.js -> generic-sites.js Both now sit at the extension root beside config.js / fallback-policy.js, and event-extractors/ holds only the pipeline's own registry/orchestrator/helpers and the custom/ sources. The load-order generator emits extension-root-relative paths throughout (it already did for the output) and pins the two generic files by name, since they're the only injected files outside that folder; the ordering rule is unchanged, with generic-sites.js still pinned behind every custom source so a dedicated source always wins the host match. Also added the two files to the release shipping set — the zip guard caught their absence, since it was keyed on the event-extractors/ directory entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VE29ox6D2VwKLJJEy4bmfy
1 parent 10db0f2 commit 53cce7d

25 files changed

Lines changed: 106 additions & 87 deletions

.claudinite/local/packs/gcec/RULES.md

Lines changed: 14 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,7 @@ on that pipeline). Adding a cached live case is the
176176
`og:locale`, or a non-English language; a bare `en` (region unknown) stays
177177
month-first rather than guess. Resolution lives in `helpers/dates.js`
178178
(`parseDateFromText` / `normalizeDateValue` take a `dayFirst` flag, threaded
179-
from `core/generic.js`'s `pageUsesDayFirstDates`), mirroring
179+
from `generic-extractor.js`'s `pageUsesDayFirstDates`), mirroring
180180
`derive-timezone.js`'s locale read; unambiguous dates (a part > 12) and the
181181
`.` / `-` separators are always day-first regardless (#686).
182182
- **`fetch-page.yml` is attended by its dispatcher** (the workflow-failure
@@ -206,28 +206,33 @@ on that pipeline). Adding a cached live case is the
206206
hints, see `helpers/derive-timezone.js``ctz`). This is the comparison the
207207
fallback-coverage gate automates — and a case where it shows NO gap is a
208208
candidate for deleting the per-site source and listing the host in
209-
`core/generic-sites.js`.
209+
`generic-sites.js`.
210210

211211
## Architecture rules of the road
212212

213213
Whenever we agree on a new or changed top-level architectural guideline, update
214214
this section as part of the same change (the design doc itself is
215215
`dev/procedures/highLevelDesign.md`):
216216

217-
- **The core generic extractor is the base layer of every extraction**
218-
(`extension/event-extractors/core/generic.js`) — it runs on every page, and a
219-
per-site source is only a layer of *overrides* merged over it, stating the
220-
fields it gets better. A source never re-reads what the page already says about
221-
itself, and nothing in `core/` may know about a specific site.
217+
- **`extension/event-extractors/` is the extensibility point for PER-SITE
218+
extractors, and nothing else belongs in it.** The generic extractor is a
219+
fundamentally different thing — one fixed, site-agnostic reader, not another
220+
entry in the set — so it and its host list live at the extension root
221+
(`extension/generic-extractor.js`, `extension/generic-sites.js`).
222+
- **The core generic extractor is the base layer of every extraction** — it runs
223+
on every page, and a per-site source is only a layer of *overrides* merged over
224+
it, stating the fields it gets better. A source never re-reads what the page
225+
already says about itself, and the generic extractor may never know about a
226+
specific site.
222227
- **A site we fully support gets no extractor file when there is nothing to
223-
override.** Its host goes in `core/generic-sites.js` (a `matches`-only
228+
override.** Its host goes in `generic-sites.js` (a `matches`-only
224229
registration) and stays fully supported — green icon, no correction prompt.
225230
When a `custom/<site>.js` shrinks to nothing, delete it and move the host
226231
there; when a generic-site starts needing a fix the generic extractor can't
227232
make site-agnostically, move it back. Never keep a file whose only content is
228233
a restatement of the generic base, and never annotate one as "nothing to add".
229234
- Adding support for a new host is the most common change — the architecture
230-
must keep it a single-file change: a host added to `core/generic-sites.js`,
235+
must keep it a single-file change: a host added to `generic-sites.js`,
231236
or one self-contained new `extension/event-extractors/custom/<site>.js`, plus
232237
regenerating the load list — touching nothing else and assuming nothing about
233238
other extractors.

.claudinite/local/packs/gcec/tasks/fallback-extractor-improvements/postconditions.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
#
1818
# Checks (all deterministic):
1919
# 1. SCOPE — the diff touches ONLY the core generic extractor and its allowed
20-
# companions: core/generic.js and the shared helpers/*.js (the generic
20+
# companions: generic-extractor.js and the shared helpers/*.js (the generic
2121
# extractor), a covering test in extension-test/event-extractors/extraction.test.js,
2222
# and the regenerated fallback-coverage GENERATED artifacts. A custom/<site>.js
2323
# edit or a host special-case is out of scope — it isn't a GENERIC win. (This
@@ -55,7 +55,7 @@ git rev-parse --verify --quiet "$ref" >/dev/null 2>&1 || ref=main
5555
git rev-parse --verify --quiet "$ref" >/dev/null 2>&1 || ref=HEAD
5656

5757
# ── 1. SCOPE ──
58-
allowed='^(extension/event-extractors/core/generic\.js|extension/event-extractors/helpers/[^/]+\.js|extension-test/event-extractors/extraction\.test\.js|dev/requirements/extractor/fallback/fallback-coverage\.(baseline\.GENERATED\.json|GENERATED\.md))$'
58+
allowed='^(extension/generic-extractor\.js|extension/event-extractors/helpers/[^/]+\.js|extension-test/event-extractors/extraction\.test\.js|dev/requirements/extractor/fallback/fallback-coverage\.(baseline\.GENERATED\.json|GENERATED\.md))$'
5959
changed="$( { git diff --name-only "$ref"...HEAD 2>/dev/null; git diff --name-only HEAD 2>/dev/null; \
6060
git ls-files --others --exclude-standard; } | sort -u | sed '/^$/d' )"
6161
offenders="$(printf '%s\n' "$changed" | grep -Ev "$allowed" || true)"

.claudinite/local/packs/gcec/tasks/fallback-extractor-improvements/task.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Fallback-coverage routine
22

33
Weekly Claude routine (on a strong model), offline on a fresh clone. Goal: make the **generic**
4-
fallback extractor — `extension/event-extractors/core/generic.js` and the
4+
fallback extractor — `extension/generic-extractor.js` and the
55
shared `extension/event-extractors/helpers/*.js` — recover **more** of what the
66
dedicated per-site sources get, measured by the fallback-coverage gate. **Most runs
77
correctly change nothing** — the gate already banks every prior win, so a new
@@ -53,7 +53,7 @@ work, because a run that trips one there is marked **failed**:
5353
JSON-LD, OG/meta, microdata, `<time datetime>`). Lifting one `custom/<site>.js`'s
5454
logic or special-casing a host is out of scope: if the only beneficiary is a
5555
single case and the rule wouldn't fire elsewhere, it isn't generic. **The only
56-
files you may leave changed are `core/generic.js`, `helpers/*.js`, your
56+
files you may leave changed are `generic-extractor.js`, `helpers/*.js`, your
5757
covering test in `extension-test/event-extractors/extraction.test.js`, and the
5858
regenerated fallback-coverage GENERATED artifacts** — anything else fails scope.
5959
- **A real, non-regressing win** (postcondition: *win*, via the suite's gate). Every

dev/build/gen-load-order.js

Lines changed: 26 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -5,20 +5,26 @@
55
// exercised by the tests).
66
// Run it with `npm run index` after adding/removing a source or extractor file.
77
//
8-
// Adding a source is a single-new-file change: drop
8+
// Adding a per-site extractor is a single-new-file change: drop
99
// event-extractors/custom/<site>.js and rerun this; the list is regenerated
1010
// mechanically rather than kept in sync by hand. A CI test (the load-order
1111
// drift guard) asserts the committed file matches what this generator would
1212
// produce, so it can never silently drift.
1313
//
1414
// Ordering rule (the only ordering that matters): the registry and shared
1515
// helpers load FIRST (they build globalThis.GCal), the per-site sources
16-
// (custom/*.js) load next — sorted, for a stable conflict-free list — and the
17-
// tail is pinned: the core generic extractor and the hosts it alone serves
18-
// (core/*.js), then the orchestrator (assemble-events.js, whose completion value
19-
// is the extraction result).
16+
// (event-extractors/custom/*.js) load next — sorted, for a stable conflict-free
17+
// list — and the tail is pinned: the generic extractor and the hosts it alone
18+
// serves, then the orchestrator (event-extractors/assemble-events.js, whose
19+
// completion value is the extraction result).
2020
//
21-
// core/generic-sites.js is pinned AFTER every custom/<site>.js on purpose: it
21+
// The two generic files live OUTSIDE event-extractors/ (at the extension root):
22+
// event-extractors/ is the extensibility point for per-site extractors, and the
23+
// generic extractor is a different thing — the base layer every one of them
24+
// overrides. They're pinned by name here rather than globbed, since they're the
25+
// only injected files outside that folder.
26+
//
27+
// generic-sites.js is pinned AFTER every custom/<site>.js on purpose: it
2228
// registers a bare matcher per host served by the generic extractor alone, and
2329
// assemble-events.js takes the FIRST source whose matches() accepts the host, so
2430
// a dedicated source added for one of those hosts always wins.
@@ -38,10 +44,12 @@ const EXT = path.join(ROOT, "extension");
3844
const DIR = "event-extractors";
3945
const OUTPUT = "event-extractors/load-order.generated.json";
4046

41-
const PINNED_FIRST = ["registry.js"]; // followed by helpers/*, added below
42-
// The core layer, then the orchestrator. See the ordering rule above for why
43-
// core/generic-sites.js is pinned behind the per-site sources.
44-
const PINNED_LAST = ["core/generic.js", "core/generic-sites.js", "assemble-events.js"];
47+
// All paths below are extension-root-relative, so the two generic files carry no
48+
// directory prefix while everything else sits under event-extractors/.
49+
const PINNED_FIRST = [`${DIR}/registry.js`]; // followed by helpers/*, added below
50+
// The generic layer, then the orchestrator. See the ordering rule above for why
51+
// generic-sites.js is pinned behind the per-site sources.
52+
const PINNED_LAST = ["generic-extractor.js", "generic-sites.js", `${DIR}/assemble-events.js`];
4553

4654
const isJs = (f) => f.endsWith(".js");
4755

@@ -50,35 +58,29 @@ function computeLoadOrder() {
5058
.readdirSync(path.join(EXT, DIR, "helpers"))
5159
.filter(isJs)
5260
.sort()
53-
.map((f) => `helpers/${f}`);
61+
.map((f) => `${DIR}/helpers/${f}`);
5462

5563
const sources = fs
5664
.readdirSync(path.join(EXT, DIR, "custom"))
5765
.filter(isJs)
58-
.map((f) => `custom/${f}`);
66+
.map((f) => `${DIR}/custom/${f}`);
5967

6068
const pinned = new Set([...PINNED_FIRST, ...PINNED_LAST]);
6169
const topLevelMiddle = fs
6270
.readdirSync(path.join(EXT, DIR))
63-
.filter((f) => isJs(f) && !pinned.has(f));
71+
.filter(isJs)
72+
.map((f) => `${DIR}/${f}`)
73+
.filter((f) => !pinned.has(f));
6474

6575
const middle = [...topLevelMiddle, ...sources].sort();
6676

6777
for (const f of [...PINNED_FIRST, ...PINNED_LAST]) {
68-
if (!fs.existsSync(path.join(EXT, DIR, f))) {
69-
throw new Error(`expected ${DIR}/${f} to exist`);
70-
}
71-
}
72-
73-
// core/ is pinned by name rather than globbed, so a file added there would
74-
// otherwise drop out of the load list silently. Fail loudly instead.
75-
for (const f of fs.readdirSync(path.join(EXT, DIR, "core")).filter(isJs)) {
76-
if (!PINNED_LAST.includes(`core/${f}`)) {
77-
throw new Error(`${DIR}/core/${f} is not in PINNED_LAST — add it (order matters, see the ordering rule)`);
78+
if (!fs.existsSync(path.join(EXT, f))) {
79+
throw new Error(`expected ${f} to exist`);
7880
}
7981
}
8082

81-
return [...PINNED_FIRST, ...helpers, ...middle, ...PINNED_LAST].map((f) => `${DIR}/${f}`);
83+
return [...PINNED_FIRST, ...helpers, ...middle, ...PINNED_LAST];
8284
}
8385

8486
// JSON, one path per line, trailing newline — a clean, reviewable diff.

dev/build/release/shipping-files.js

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,9 @@ const SHIPPING_PATHS = [
2020
"config.js", // tunable product decisions; imported by the popup modules at runtime
2121
"fallback-policy.js", // host classifier for the generic fallback; imported by the popup at runtime
2222
"fallback-lists.json", // the allow/deny/supported host lists; fetched by the worker, imported by config.js
23-
"event-extractors", // the extractors (custom sources + helpers + orchestrator) + generated load list (popup fetches/injects)
23+
"generic-extractor.js", // the core generic extractor — injected on every page (outside event-extractors/ by design)
24+
"generic-sites.js", // the hosts it alone serves; injected alongside it
25+
"event-extractors", // the per-site extractors + helpers + registry/orchestrator + generated load list (popup fetches/injects)
2426
"events-popup", // popup (html/css/js + views) and the calendar-URL builder
2527
"icon", // the toolbar-icon service worker and its images/ PNGs the manifest references
2628
];

dev/procedures/fileDescriptions.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,8 @@ extension does.
1717
| `extension/icon/toolbar-icon.js` | Background service worker: registers `chrome.declarativeContent` rules (from `fallback-lists.json`) so the browser colors the toolbar icon by host pattern — green on supported hosts, gray on denylisted ones, blue elsewhere — without the extension reading any tab URL |
1818
| `extension/event-extractors/registry.js` | Bootstraps `GCal`, the `GCal.sources` registry, and `isSupportedHost` |
1919
| `extension/event-extractors/helpers/` | Shared utilities any extractor may use, split by concern: DOM, text (rich-text/`htmlToText`/`parts`), dates, timezones, timezone-names, merge, and `embedded-events` (the `GCal.embeddedEvents` schema.org JSON-LD reader) |
20-
| `extension/event-extractors/core/generic.js` | `GCal.genericExtractor`: THE core generic extractor, run on every page — best-effort event (embedded JSON-LD + generic heuristics over meta tags, microdata, `<time>`, and visible text). It is the base layer every per-site source overrides, the only extractor on an unsupported host, and the whole of the support for a `generic-sites.js` host |
21-
| `extension/event-extractors/core/generic-sites.js` | The hosts fully covered by `core/generic.js` alone — each registers a `matches`-only source, so the host is supported with no per-site file. Loaded after `custom/` so a dedicated source always wins the match |
20+
| `extension/generic-extractor.js` | `GCal.genericExtractor`: THE core generic extractor, run on every page — best-effort event (embedded JSON-LD + generic heuristics over meta tags, microdata, `<time>`, and visible text). It is the base layer every per-site source overrides, the only extractor on an unsupported host, and the whole of the support for a `generic-sites.js` host |
21+
| `extension/generic-sites.js` | The hosts fully covered by `generic-extractor.js` alone — each registers a `matches`-only source, so the host is supported with no per-site file. Loaded after `custom/` so a dedicated source always wins the match |
2222
| `extension/event-extractors/custom/meetup.js`, `eventbrite.js`, `edinburghfringe.js`, `telavivcinematheque.js`, `ticketmaster.js`, … | One per site the generic extractor gets wrong: hardcoded selectors + inline host matcher, stating ONLY the fields it overrides (or its own enumerated `events`) |
2323
| `extension/events-popup/build-calendar-url.js` | Builds the pre-filled Google Calendar template URL (incl. markdown→HTML for details) |
2424
| `extension/event-extractors/assemble-events.js` | Orchestrator `GCal.extract()`: runs the core generic extractor for the base events and merges the matched site source's overrides over them; normalizes/sorts events and reports `supported` + `fallback` |

dev/procedures/highLevelDesign.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ in [requirements.md §12–§16](../requirements/requirements.md); the per-file
77

88
`toolbar-icon.js` colors the toolbar icon by host (a source matches or it doesn't). On
99
click, `popup.js` injects `event-extractors/` and runs `assemble-events.js`, which runs
10-
`core/generic.js` and merges the matching `custom/<site>.js` over it;
10+
`generic-extractor.js` and merges the matching `custom/<site>.js` over it;
1111
`build-calendar-url.js` builds the URL `events-view.js` renders.
1212

1313
The top-level **architecture rules of the road** live in the gcec local pack's
@@ -31,12 +31,15 @@ non-time field are folded into one multi-instance event by the assembler.
3131

3232
Extraction is **two layers, not two paths**:
3333

34-
1. **The core generic extractor** (`extension/event-extractors/core/generic.js`)
34+
1. **The core generic extractor** (`extension/generic-extractor.js`)
3535
runs on **every** page, whatever its host, and produces the base events from
3636
the page's own self-description: embedded schema.org JSON-LD plus generic
3737
heuristics (microdata, Open Graph / meta tags, `<time datetime>`,
3838
`<h1>`/`<address>`, venue/location-named elements, and finally a date/time
3939
scan over the visible text). Nothing in it may know about a specific site.
40+
It sits at the extension root, **outside** `event-extractors/`: that folder is
41+
the extensibility point for per-site extractors, and this is a fundamentally
42+
different thing — the one fixed reader they all layer over.
4043

4144
2. **A per-site source** (`extension/event-extractors/custom/<site>.js`) is a
4245
layer of **overrides** on top of that base. Its `extract()` returns only the
@@ -47,7 +50,7 @@ Extraction is **two layers, not two paths**:
4750
Each lives in its own file with a comment describing the HTML it expects.
4851

4952
So a site whose pages describe themselves completely needs **no source file at
50-
all**: its host goes in `extension/event-extractors/core/generic-sites.js` and is
53+
all**: its host goes in `extension/generic-sites.js` and is
5154
fully supported by the base alone — same green icon, same popup, no
5255
"Suggest Correction" prompt. When adding support for a platform, check the
5356
generic extractor first and only write a `custom/<site>.js` for what it gets

dev/requirements/extractor/cases/extractor-support.11.16.case.js

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
module.exports = {
66
tbd: true,
77
description:
8-
"facebook.com — fully covered by the core generic extractor (core/generic-sites.js); " +
8+
"facebook.com — fully covered by the core generic extractor (generic-sites.js); " +
99
"unit-tested only, bot-blocked from caching",
1010
host: "facebook.com",
11-
source: "extension/event-extractors/core/generic-sites.js",
11+
source: "extension/generic-sites.js",
1212
};
Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
"use strict";
22

33
module.exports = {
4-
description: "bandsintown.com — fully covered by the core generic extractor (core/generic-sites.js)",
4+
description: "bandsintown.com — fully covered by the core generic extractor (generic-sites.js)",
55
host: "bandsintown.com",
6-
source: "extension/event-extractors/core/generic-sites.js",
6+
source: "extension/generic-sites.js",
77
page: "bandsintown-berry-sakharof",
88
};
Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
"use strict";
22

33
module.exports = {
4-
description: "dash.datadoghq.com — fully covered by the core generic extractor (core/generic-sites.js)",
4+
description: "dash.datadoghq.com — fully covered by the core generic extractor (generic-sites.js)",
55
host: "dash.datadoghq.com",
6-
source: "extension/event-extractors/core/generic-sites.js",
6+
source: "extension/generic-sites.js",
77
page: "dash-datadoghq",
88
};

0 commit comments

Comments
 (0)