feat: replace the hand-rolled N3 parser with N3.js across the Turtle/N3 family - #831
feat: replace the hand-rolled N3 parser with N3.js across the Turtle/N3 family#831jeswr wants to merge 6 commits into
Conversation
|
This tripped into using opus halfway through so I am asking fable to rewrite it. |
|
If I understand well this is a major change that breaks parsing of existing turtle files with 0/1 as Boolean values in rdflib context. |
…N3 family
Delete src/n3parser.js (1,610 lines) and src/lists.ts (121 lines) and route
every Turtle-family content type through the N3.js parser via a single
adapter (src/n3-adapter.ts):
- text/turtle, application/x-turtle strict Turtle
- text/n3, application/n3 full Notation3
- application/n-triples (new)
- application/n-quads (+nquads alias) unified
- application/trig (new)
The adapter maps N3.js's flat quad stream onto rdflib's model: ( ... )
collections are folded into Collection terms with N3.js's own
Store#extractLists machinery; N3 formulae { ... } are rebuilt into Formula
sub-stores; @forAll/@Forsome (reified by N3.js under explicitQuantifiers)
are registered via newUniversal/declareExistential; ?x becomes a Variable;
=, =>, <= map natively to owl:sameAs / (reversed) log:implies.
The Fetcher's N3Handler now delegates to parse() instead of driving the
legacy parser, and sparqlUpdateParser builds its clause formulae by
rewriting the patch into N3 and running it through the same adapter
(gaining SPARQL-style PREFIX support, #651).
Legacy leniencies preserved: the empty prefix ":" still defaults to
<base#> (retry-once on N3.js's specific "Undefined prefix" error), and
lone rdf:nil / reified first-rest chains still fold into collections.
Dropped, now loud parse errors (see PR body): @Keywords, bareword
date/dateTime literals, `has`/`is ... of`, string-literal subjects, and
lexical normalisation of numeric/boolean literals.
Also fixes the Turtle serializer to accept the spec lexical forms
"true"/"false" for xsd:boolean, and repairs an N3.js n3-mode label-scoping
bug for `_:x` mentioned inside [ ... ] property lists.
Breaking: the N3Parser export is gone; n-quads default-graph statements
are now attributed to the document graph; text/turtle is now strict
Turtle (full N3 under a turtle content type no longer parses).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1b3f679 to
0fd4b99
Compare
jeswr
changed the title
N3.js is compliant to the current RDF 1.1/1.2 test suites for turtle and N3 CG test suites for Notation3. Looking at the changes to the test files it seems like there might indeed be some deviations in this parser from those suites. I'll take your guidance on whether that is considered a patch or breaking change - I see the merit in holding this PR for a next mver. |
Breaking-change assessment (v2.4.0 → this branch)We diffed runtime behavior probe-by-probe against built v2.4.0 and this branch, then hunted consumers (SolidOS stack, @ldflex/rdflib, node-solid-server, npm/GitHub reverse-deps). Full report with repros, per-delta migration snippets, and mitigation costings is in the linked document. Verdicts:
Semver: this needs to be v3.0.0. solid-ui/mashlib/solid-logic/solid-panes ( Cheap mitigations worth adopting in this PR:
cc @jeswr — this assessment was agent-assisted (Claude Code). Every verdict was produced by running identical probes against both built checkouts; consumer hits are cited to file:line or reproduced against the consumers' own fixtures. |
…ad, stub N3Parser
Follow-up to the breaking-change assessment on this PR, landing its cheap
mitigations (M1/M2/M4/M8):
- sparqlUpdateParser: build the <#query> sym from the fragment-stripped
base and write it as an absolute IRI in the generated N3, so insertion
and lookup can never diverge. Previously a fragment-bearing base URI
(e.g. .../card#me) made the clause lookup miss and the returned patch
silently lost its insert/delete/where clauses (regression introduced by
this branch; U7 in the assessment). Regression test added.
- Fetcher.load: force: true now implies clearPreviousData: true unless
the caller explicitly sets it to false. Parsed blank-node labels are no
longer stable across parses, so re-parsing without clearing duplicated
bnode subgraphs on every forced reload (U3/M1); this restores the net
behaviour 2.x callers of load(uri, {force: true}) actually observed.
Tests: statement count stable across a forced reload; explicit
clearPreviousData: false still accumulates.
- index: restore a deprecated N3Parser export as a stub that always
throws 'N3Parser was removed in rdflib 3.0; use parse(text, store,
base, contentType) or the n3 package directly', converting the silent
'undefined is not a constructor' failure into an actionable message
(D4/M5). Throw message tested for both new and plain-call forms.
- Documentation/turtle-intro.html: mark the naked date/dateTime and
is...of extensions as removed in rdflib 3.0 (they now raise parse
errors under every content type) with their standard replacements;
note the empty-prefix default is still supported (M8).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e flag
The N3.js-based parsers preserve literal lexical forms, so stored `true` /
`12.0` literals no longer compare equal to canonically-constructed ones
(`"1"^^xsd:boolean`, `"12"^^xsd:decimal`). This ships the two transition
aids from the breaking-change assessment (mitigation M3):
- Value-space helpers `isTrue(term)`, `literalToBoolean(term)` and
`literalToNumber(term)` (also exposed as `Literal.toBoolean` /
`Literal.toNumber`, the inverses of `fromBoolean` / `fromNumber`). They
accept every lexical form the datatype allows, giving ^2.x consumers a
one-line fix for patterns like `term.value === '1'`.
- An opt-in `canonicalize` parse option — `parse(str, kb, base, type,
{ canonicalize: true })` or as a sixth argument after a callback —
that restores rdflib <=2's parse-time normalisation of xsd:boolean /
integer / decimal / double / float lexical forms (`true` -> "1",
`12.0` -> "12", `3.141e0` -> "3.141") across the Turtle family. Off by
default; lexical preservation remains the shipped behaviour. Integer
canonicalisation is done lexically so arbitrary-precision values keep
their exact digits, and ill-typed forms are left untouched.
Covered by a helper truth table, canonicalize on/off store-comparison
tests and a tests/types compile check of the new exports.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jeswr I'm not sure we are on the same line. We cannot decide that years of data are not parseable rdflib turtle anymore. Boolean is may be not the only issue. But this is an old one. And ran in a blocker to solve it |
Completes the M8 documentation mitigation for the N3.js migration:
reference/{dumpParser,ldpatchParser,fetcher-classes}.js document the legacy
hand-rolled parser's internals, which no longer exist. A README now marks the
directory as archived reference material and points at parse()/N3.js.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Ah I am with you now @bourgeoa - I can think of two things that we can do here:
|
A root `import ... from 'n3'` in n3-adapter.ts pulled n3's package index into downstream browser bundles, dragging N3StreamWriter and its readable-stream/Node polyfill chain back in and undoing the #449 deep-import discipline. The adapter now deep-imports exactly what it uses (n3/lib/{N3Parser,N3Store,N3DataFactory}.js), and a source-level hygiene test fails any future root 'n3' import under src/. Verified on the built browser bundle: zero occurrences of N3StreamWriter/N3StreamParser/N3Writer/readable-stream in dist/rdflib.min.js; full suite green (382 passing). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This is is a very long open issue. See nodeSolidServer/node-solid-server#1468 (comment). It needs a more open discussion. Not sure that serialization to false/true will not introduce more bugs. |
Ah .... I see. I'll look at this more closely in a bit to see if there is a way of going about this that doesn't break anything or annoy anoyone. |
Output-level guard in the browser e2e: dist/rdflib.min.js must contain zero occurrences of N3StreamWriter/N3StreamParser/readable-stream, so a regression of the deep-import pattern (e.g. via #831's n3-adapter) fails the browser-e2e CI job loudly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the hand-rolled 2005-era N3 parser with N3.js across the whole Turtle/N3 family, so
parse(),Fetcher.load()andsparqlUpdateParseragree with each other for the first time — one parser, net −1,022 lines undersrc/.What is deleted / added
src/n3parser.jssrc/lists.ts(convertFirstRestNil,substituteInDoc,substituteNillsInDoc)Store#extractLists(it was not exported from the package index)src/patch-parser.jsp.skipSpace/p.node/p.directive); it rewrites the patch to N3 and reuses the same N3.js pipelinefetcher.tsN3Handlerinline parserparse()src/n3-adapter.tssrc/utils/literalValue.tscanonicalizeflag (the migration path — see mitigation M3 below)How it uses the N3.js API (not custom code)
N3.Parsercall per document, with the format string enforcing the right grammar per content type —text/turtle(+application/x-turtle),text/n3(+application/n3),application/n-triples,application/n-quads(+nquads),application/trig. N-Triples/TriG support is new; the ad-hoc n-quads callback path is gone. Synchronous, using N3.js's array-returning mode with{ onPrefix }for prefix capture.N3.Store#extractLists({ remove: true })finds, validates and removesrdf:first/rest/nilchains; the adapter resolves the returned head→items map (recursively, for nested lists) into rdflibCollectionterms, and turns lonerdf:nilinto an emptyCollection— matching the legacy parser's output, including for explicitly reified chains andrdf:nil-as-element (the existinglists-test.jsexpectations all pass unchanged). Folding is gated on the factory'sCOLLECTIONSsupport exactly as before, and skipped entirely when a document never mentionsrdf:nil(no intermediate store on the hot path). Line-dump formats (N-Triples/N-Quads/TriG) are never folded — quad dumps round-trip verbatim.text/n3): N3.js emits each{ … }as quads under a fresh blank-node graph label; the adapter rebuilds them into rdflibFormulasub-stores (recursively for nesting, with the empty-formula{}marker special-cased).=→owl:sameAs,=>→log:implies,<=→ reversedlog:implies,!/^paths, and?x→Variableare all N3.js-native.explicitQuantifiers: true; the reified declarations inurn:n3:quantifiersare read back intonewUniversal()/declareExistential()on the right scope (store or formula), keeping the legacy representation (terms stayNamedNodes).sparqlUpdateParserscans only the top-level keywords (INSERT/DELETE/WHERE, optionalDATA,@prefix, and now SPARQL-stylePREFIX), rewrites the document to<#query> patch:where { … } .N3, and hands it to the same adapter — clause{ … }bodies are parsed by N3.js, not by hand.Issues closed
parse(),Fetcher.load(), patches): SPARQLPREFIX/BASE, single-quoted long literals, numeric/\UIRI escapes, bareword numbers/booleans, dotted local names.'''…'''long literals (old parser:Bad syntax).\U0001F60Aescapes decode to the correct code point.PREFIXdirectives in turtle parsers. #626 — SPARQL-stylePREFIXdirectives.application/n-triplesparsing support (previously threw "Don't know how to parse").Literals withtermType(v2.4.0 pushed raw JStrue/falseintoCollection.elements, crashing consumers; regression test added).PREFIXwithout a trailing dot (test added).@prefixis still accepted for backwards compatibility — say the word if you'd rather hard-drop it.myprefix:1.18) parse to the correct IRI (valid Turtle 1.1PN_LOCAL; probe: 1 statement, object.../test/1.18).src/n3parser.js:118; that file is deleted here.Unexpected "???" on line 1), but there are no structuredline/columnfields on the thrown error and no character offset — left open as a small follow-up on the adapter.Behavioural changes (need your sign-off)
text/turtleis now strict Turtle. The legacy code parsed full N3 under a turtle content type (same parser for both), and the Fetcher routed all fetched turtle through it. Mislabelled N3-in-turtle documents now fail with N3.js's descriptive errors. Escape hatch if you want wire-compat instead: route turtle content types toformat: 'text/n3'in the adapter (n3 mode is a turtle superset) — a one-line change; my recommendation is strict.12.0,3.141e0,truestay as written; the legacy parser normalised to12,3.141,1). This is what Turtle 1.1 requires, but it changes term identity ("12"^^xsd:decimal≠"12.0"^^xsd:decimal) and serializer output. The Turtle serializer had a latent bug here — it emittedfalsefor"true"^^xsd:booleanbecause it only knew the legacy'1'form — fixed inserializer.js. Mitigation M3 (below) is the migration path for consumers that match on the normalised forms; the wider read-lenient/write-strict question is under discussion in this thread.kb.sym(base)(rdflib's provenance convention) instead of the old callback path'sDefaultGraphterm. Consistent with every other format; flagged as breaking for anyone matching on the default graph.N3Parseris no longer exported (breaking API; a throwing stub with a descriptive pointer remains — M5 below).sparqlUpdateParserkeeps its exact signature/return shape ({query, insert?, delete?, where?}withFormulavalues, plus the<#query> patch:*statements on the store).:namewithout@prefix :still resolves against<base#>. Instead of a document-wide regex + unconditional input mutation, the adapter parses strictly first and retries once only on N3.js's specificUndefined prefix ":"error — zero cost and zero false positives on conforming documents, and mid-document@prefix :declarations override exactly like the legacy sequential semantics. (TimBL's ownstructures.n3fixture depends on this leniency.)@keywords, bareword date/dateTime literals (2026-07-01→ write"2026-07-01"^^xsd:date),has …/is … of(write the inverse triple; the modern N3 CG spec dropped them), string-literal subjects, undeclared non-empty prefixes (note: these already errored in v2.4.0 — only the error message/type changes)._:cmentioned inside a[ … ]property list gets the graph label as its scope prefix (top level: label.c), splitting it from its other mentions. Repro:new N3.Parser({format:'text/n3'}).parse('_:c <urn:p> 1. [ <urn:q> _:c ] <urn:r> 2.'). The adapter passes an explicitblankNodePrefixand repairs.-prefixed labels; formula-scoped labels (correct per N3 semantics) are untouched.maindoes today. (Real support would be a store-model project.)Collectionterms (surfaced by the independent review pass; v2.4.0 left rawrdf:first/rdf:resttriples inside formula sub-stores — verified by repro against the published 2.4.0). Consistent with the migration and arguably a fix, but it changes formula contents for consumers that match onfirst/restinside clauses (e.g. SPARQL-update WHERE bodies).Breaking changes, semver and mitigations
A full breaking-change assessment ran against this branch (probe-by-probe vs built v2.4.0, plus consumer hunts across the SolidOS stack, @ldflex/rdflib, NSS, and npm/GitHub reverse-deps): see the assessment comment. Verdicts: 4 BREAKING (strict
text/turtlevs legacy pod files; literal lexical preservation flipping solid-ui boolean reads; dropped cwm leniencies vs livetext/n3documents; re-parse bnode accumulation at force-reload sites), 10 potentially-breaking with no live consumer hit found, the rest safe or strict fixes.This must ship as v3.0.0 — solid-ui/mashlib/solid-logic/solid-panes (
^2.3.x) and @ldflex/rdflib (^2.2.0) all auto-adopt any 2.x release.The full mitigation set recommended by the assessment is on the branch:
force: truenow impliesclearPreviousData: trueinFetcher.load— de-breaks the 81 observed force-reload sites (bnode accumulation)sparqlUpdateParserunder a fragment-bearing base silently dropped its insert/delete clauses; regression test addedisTrue(term),literalToBoolean(term),literalToNumber(term)(also exposed asLiteral.toBoolean/Literal.toNumber), plus an opt-inparse(str, kb, base, contentType, { canonicalize: true })flag that restores the ≤2.x parse-time normalisation of boolean/numeric lexical forms (true→"1",12.0→"12",3.141e0→"3.141") across the Turtle family — off by default, so lexical preservation stays the shipped behaviour.isTrue(kb.any(s, p))is the one-line fix for^2.xpatterns liketerm.value === '1'N3Parserexport stub that throws a descriptive pointer toparse()Documentation/turtle-intro.htmlupdated for the two removed extensions (0ccb5d7);reference/README.mdmarksdumpParser.js/ldpatchParser.js/fetcher-classes.jsas archived relics that driveN3Parserinternals which no longer exist (2b10b5b)n3/lib/{N3Parser,N3Store,N3DataFactory}.js, preserving #836's #449 deep-import discipline (noN3StreamWriter/readable-streamin browser bundles); a source-level hygiene test now fails any future root'n3'import undersrc/; verified zero stream/writer/readable-streamoccurrences in the builtdist/rdflib.min.jsTest changes — all justified, none loosened
parse-turtle-family-test.js(new): 30+ cases — the conformance suite plus formulae (=>/<=/=, nesting, empty{}), variables, quantifier scoping, nested/reified/empty collections, Error when parsing valid shacl/ttl file #352, the label-repair regression, and explicit "documented gap" error assertions foris…of/ bareword dates /@keywords.patch-parser-test.js: +6 cases (variables,@prefix/PREFIX,INSERT DATA,;separators + braces-inside-strings, store side-effects, error reporting).literal-value-test.js,parse-canonicalize-test.js,tests/types/value-space.ts(new): the M3 helpers andcanonicalizeflag, value- and type-level.n3parser-stub-test.js,n3-import-hygiene-test.js(new): the M5 stub's error pointer; the no-root-'n3'-import source guard.fetcher-test.js: the N3 fixture used a bareword date — modernised to"2012-03-12"^^xsd:date(same graph; the old syntax is a documented gap, behavioural change 6). New Can't store N3 triple with subject formula because of incompatible isSubject #567 formula-subject load test.lists-test.js: expectations unchanged; only the redundant post-parseconvertFirstRestNil()calls were removed (the function is deleted;parse()itself folds).serialize-test.js: the RDF/XML expectation now uses the preserved lexical forms (12.0/3.141e0) — the old expectation encoded the legacy normalisation; the JSON-LD expectation in the same file already used the preserved forms.indexed-formula-test.js: the store under test is nowDataFactory.graph()(the previousnew IndexedFormula()version passed vacuously because the legacy parser misfiled the status list into the default graph), plus a new companion test pinning the non-collection-store outcome explicitly (rawfirst/resttriples surviveremoveMetadata— the pre-existing deleteDocument() : Error : Statement to be removed is not on store #631 gap, now visible instead of hidden).tests/serializegolden fixtures changed — flagging every ref-file diff explicitly. Sourcest1.ttl,t2.ttl,t7.n3,structures.n3used bareword dates /is…ofand are rewritten to express the same graphs in standard syntax; the regenerated references aret1-ref.xml,t2-ref.xml,t7-ref.nt,t11-ref.xml,t12-ref.ttl,t13-ref.ttl,t17-ref.xml. Every delta was hand-checked to be one of exactly two classes: (a) preserved lexical forms (10.0,1.45e5,true), (b) parser-generated bnode label names (_:dl1_corettavs_:_g_L4C88). Thet11structures ref also exposed the serializer boolean bug fixed above (behavioural change 2).Gates
CI is green on the branch head (Node 22.x + 24.x). Full local runs, all exit 0: at the migration commit,
npm run build·test:unit(349 passing, 0 failing) ·test:serialize(all 17) ·test:types·build:types·build:esm·lint(0 errors; 5 pre-existing warnings untouched); re-verified with the M1/M2/M5 mitigations at 0ccb5d7 (355 unit passing, serialize/types/lint clean). The bundle-hygiene commit (5a6bce9) additionally verified the builtdist/rdflib.min.jscontains no stream/writer code.