Skip to content

feat: RDF 1.2 reifier migration + Prolog→SHACL + addListener→subscribeQuery + N+1 elimination#591

Open
HexaField wants to merge 51 commits intodevfrom
feat/sparql-1.2-cleanup
Open

feat: RDF 1.2 reifier migration + Prolog→SHACL + addListener→subscribeQuery + N+1 elimination#591
HexaField wants to merge 51 commits intodevfrom
feat/sparql-1.2-cleanup

Conversation

@HexaField
Copy link
Copy Markdown
Contributor

@HexaField HexaField commented Apr 29, 2026

Summary

This PR combines the full SPARQL 1.2 migration for Flux — originally split across feat/sparql-1.2 (#590, now closed) and feat/sparql-1.2-cleanup (#591). Together they deliver: all custom SPARQL queries migrated to RDF 1.2 reifier patterns, all Prolog replaced with SHACL APIs, all global addListener patterns replaced with targeted subscribeQuery, and all N+1 query patterns eliminated.

Companion: AD4M coasys/ad4m#803


Part 1: Migrate Flux SPARQL Queries to RDF 1.2 Reifier Patterns

(Originally #590)

Migrates all custom SPARQL queries in Flux API packages from the old named-graph GRAPH ?g { s p o } pattern to direct default-graph triples with RDF 1.2 reifier patterns for link metadata.

Changes

packages/api/src/channel/index.ts

  • allItems() — direct triple pattern, no GRAPH
  • unprocessedItems() — 3 subqueries converted (all items, processed items, set-difference)
  • totalItemCount() — direct pattern
  • recentConversations() — reifier for timestamp
  • pinnedConversations() — direct pattern

packages/api/src/conversation/index.ts

  • stats() subgroups query — direct pattern
  • topics() — direct pattern (5 GRAPH vars → direct triples)
  • subgroupsData() main query — reifier for timestamp on has_child link
  • subgroupsData() batch timestamp query — reifier for channel timestamp

packages/api/src/conversation-subgroup/index.ts

  • stats() — direct pattern
  • topics() — direct pattern
  • itemsData() — reifier for both subgroup-item timestamp and type-link author, plus channel timestamp
  • topicsWithRelevance() — direct pattern

app/src/utils/registerMobileNotifications.ts

  • Mobile push notification trigger query — removed GRAPH wrapper

Other

  • Added type-checking support in the CI pipeline
  • Improved timeline subscription refresh for more responsive conversation updates
  • Safer handling of missing channel data to prevent navigation/display errors
  • Normalized community IDs to fix membership edge cases
  • Improved mobile notification matching and AI task processing reliability
  • More accurate timestamp selection for conversation ordering
  • Expanded unit and integration tests for channels, conversations, and timeline subscriptions
  • Integrated linting and type-checking gates into CI; updated ESLint config and ignore list

Pattern Migration

Before (named graphs):

GRAPH ?link1 { <parent> <ad4m://has_child> ?id . }
?link1 <ad4m://ontology/timestamp> ?timestamp .

After (RDF 1.2 reifiers):

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
<parent> <ad4m://has_child> ?id .
?_reifier rdf:reifies <<( <parent> <ad4m://has_child> ?id )>> .
?_reifier <ad4m://ontology/timestamp> ?timestamp .

Part 2: Prolog → SHACL, addListener → subscribeQuery, N+1 Elimination

(Originally #591)

26 files changed, 514 insertions, 659 deletions (net −145 lines)

After this part, zero perspective.infer() calls, zero unfocused addListener patterns, and zero global link bus usage remain in Flux app/view source code.

Prolog → SPARQL Conversions

File What changed
topic/index.ts linkedConversations() / linkedSubgroups() → SPARQL
semantic-relationship/index.ts All 5 methods → SPARQL
conversation/util.ts findEmbeddingSRId() → SPARQL

addListener → Targeted subscribeQuery

File What changed
flux-container.ts Global listener → subscribeQuery with Channel SPARQL
CommentSection.tsx Global listener → subscribeQuery on source
CommunityGraph.tsx Global listener → subscribeQuery on source/target (was leaking, now properly cleaned up)
useAssociations.tsx Global listener → subscribeQuery with proper dispose
TimelineColumn.tsx Global listener → subscribeQuery
useCommunityService.ts Participant tracking → subscribeQuery on CHANNEL predicate
cache.tsx Deleted unused subscribeToPerspective / unsubscribeToPerspective (zero callers, contained addListener)

N+1 Query Elimination

File What changed
useCommunityService.ts Nested Promise.all → two-phase batch (link queries then conversation lookups)
CommentSection.tsx N × Message.get()Message.findAll({ parent })
PollCard.tsx Per-answer Vote.findAlluseLiveQuery(Answer, { include: { votes: true } })

kanban-view Migration

File What changed
Board.tsx .infer()listRegisteredClasses() + getNamedOptions() + addNamedOption() + ensureSDNASubjectClass(Task). Task creation uses initialValues.
Entry.tsx .infer(property_named_option)getNamedOptions()
TaskModel.ts New — TypeScript @Model class with @Property({ options: [...] }) for status enum
Task.pl Deleted (22 lines of Prolog SDNA)

table-view Migration

File What changed
TableView.tsx .infer(subject_class)listRegisteredClasses()
Table.tsx .infer(property_named_option)getNamedOptions(). Updates use getClassShape() for predicate resolution + direct link manipulation.
Header.tsx .infer()getInstanceClasses() + getSubjectData(). Updates via getClassShape() + link add/remove.
Entry.tsx Same as Header
History.tsx Same as Header (read-only)
NewClass.tsx generateSDNA() (raw Prolog) → buildSHACLShape() + addShacl(). Removed makeRandomPrologAtom().

Global Link Bus Removal

File What changed
MainView.vue Removed gotNewMessage, onLinkAdded, and associated addLink bus emissions
usePerspectives.ts Deleted addListeners(), onLinkAdded(), onLinkRemoved(), and link callback arrays — the global link event bus is gone

Other

  • createCommunity.ts: .timestamp.createdAt (deprecated alias removed in companion PR)

Depends On

AD4M coasys/ad4m#803 — provides Oxigraph 0.5.7 + RDF 1.2 reifiers, Rust model query engine, getNamedOptions(), addNamedOption(), listRegisteredClasses(), getClassShape(), getInstanceClasses(), getSubjectData(), SHACLShape, @Property({ options }), perspectiveModelSubscribe


Expected Performance Impact

  • Link listener overhead: Near-zero — global addListener callbacks (fired on every link change across every perspective) replaced with targeted subscribeQuery that only fires when matching predicates change. In a community with 10+ channels, this eliminates ~90% of spurious JS callback invocations.
  • N+1 queries: Batch operations (findAll with include) replace per-item .get() calls, reducing round-trips from N+1 to 1-2 per list render.
  • kanban/table views: Prolog infer() calls (which loaded and evaluated the entire SDNA program) replaced with direct SHACL API lookups.
  • Vue reactivity: Removing the global link bus prevents unnecessary re-renders in components that don't care about the changed links.

Add Channel.recentConversations() and Channel.pinnedConversations() static
methods that use single SPARQL queries instead of iterative channel.get()
loops.

- Channel.recentConversations(perspective, limit): single query with
  ORDER BY DESC(?lastActivity) LIMIT — replaces N×M×K iterative walk
- Channel.pinnedConversations(perspective): single query for pinned
  channels with their conversation IDs
- Both methods deduplicate by channelId and handle errors gracefully

Tests prove single SPARQL call per invocation (not N+1).
…dels

- ChannelSummary: Channel without @hasmany relations — no hidden graph
  exploration during sidebar/list hydration
- MessageSummary: Message without SPARQL getter properties (replyingTo,
  isPopular) — retains simple @hasmany (reactions, thread, replies)
- Both exported from @coasys/flux-api barrel

Tests verify no relation/getter queries fire on lightweight models.
- useCommunityService: use ChannelSummary instead of Channel for allChannels
  live query — eliminates @hasmany relation hydration on every link change
- Replace N+1 getPinnedConversations with Channel.pinnedConversations()
- Replace N×M×K getRecentConversations with Channel.recentConversations()
- Replace iterative getChannelsWithConversations with Conversation.findOne()
- Update handleParticipantTracking for ChannelSummary (no participants prop)
- MessageList.tsx: add lazy evaluateGetters for visible messages (replyingTo)
  — works with WS-2 deepQuery inversion where getters are skipped by default

Tests verify scoped queries and no iterative patterns remain.
- Remove perspective.addListener('link-added', handleLinkAdded) from
  TimelineColumn.vue — was firing on EVERY link in the perspective
- Remove handleLinkAdded, getDataFull, getDataIncremental, debounce state
- Replace with useLiveQuery(Conversation, perspective, { parent: channel })
  — only fires when Conversation instances under this channel change
- Watch conversationInstances for reactive refreshAllData() calls
- Import useLiveQuery, remove onUnmounted (handled by useLiveQuery)

Tests verify no raw listeners, no old handler functions, and scoped
useLiveQuery usage.
All WS-3/4/5/6 tests now strip // and /* */ comments before pattern
matching, preventing false failures from comment text like
'No @hasmany relations' or 'replaces perspective.addListener'.

All 72 tests pass.
Code comments now describe what the code does rather than
referencing internal planning workstream identifiers.
Renamed test files for clarity.
The previous query applied LIMIT before deduplication, so a busy
channel with many items could consume all LIMIT rows, starving other
channels from the result set.

Switch to GROUP BY ?channelId with SAMPLE/MAX aggregation so the
LIMIT applies after per-channel grouping.  Oxigraph supports full
SPARQL 1.1 aggregation (already used elsewhere in this file via
COUNT(DISTINCT ...)), so remove the misleading comment about GROUP BY
availability.

The client-side seen Map is kept as a safety net but should no longer
be needed.
Rapid watch(conversationInstances, ...) triggers could fire
refreshAllData() concurrently.  Without a guard, a slow earlier call
could resolve after a newer one, overwriting fresher state.

Add an in-flight promise guard with a pending flag: concurrent calls
coalesce onto the running promise, and if a call arrived while
in-flight, one additional refresh runs after the current one
completes.
The getPinnedConversations and getRecentConversations tests silently
passed when the regex failed to match (the if-block was skipped).

- Add expect(match).not.toBeNull() so a regex miss is a real failure
- Apply comment-stripping in both tests for consistency
…subscriptions

- Remove refreshAllData(), getConversations(), getUnprocessedItems(),
  refreshInFlight, refreshPending, and refreshTrigger from TimelineColumn
- Replace with reactive watchEffect that maps conversationInstances
  directly from the existing useLiveQuery subscription
- Add separate watchEffect for unprocessed items, triggered by
  conversationInstances changes
- Extract sidebar refresh into a dedicated watch on conversation name
- Extract AI task check into a dedicated watch on unprocessedItems
- Remove onMounted — reactive watchEffect handles initial load
- Remove refreshTrigger prop from TimelineBlock (and recursive children)
- TimelineBlock now watches props.data instead of refreshTrigger
- Add structural tests verifying removal of imperative patterns
- All 20 structural tests pass, pnpm build succeeds
… proper unit tests

- Remove jest, ts-jest, @types/jest from packages/api
- Add vitest with config (globals, node environment)
- Delete regex/structural test files that loaded source as strings:
  - community-service-scoping.test.ts
  - timeline-column-scoping.test.ts
  - channel-sparql-methods.test.ts
- Rewrite ChannelSummary.test.ts: uses getModelMetadata() to verify
  no relations, expected properties, and flag metadata
- Rewrite MessageSummary.test.ts: uses getModelMetadata() to verify
  no getter properties, expected relations, and flag metadata
- Rewrite channel-query.test.ts: tests set-difference logic and
  three-query SPARQL integration pattern with mocked perspectives
- Add channel.test.ts: tests Channel.recentConversations() and
  Channel.pinnedConversations() with mocked perspectives —
  verifies SPARQL query structure, result mapping, deduplication,
  error handling, and limit parameter
- Existing parseLit.test.ts works unchanged (Vitest globals)

All 38 tests pass across 5 test files.
The build script temporarily sets file: overrides for local linking.
These should not be committed.
…ives

- Channel.recentConversations/pinnedConversations SPARQL queries used
  string literal 'true' to match channel_is_conversation/channel_is_pinned,
  but AD4M stores these as literal:json URIs. Use ad4m://fn/parse_literal
  SPARQL function (consistent with buildSPARQLWhereFilters).

- markRaw(perspective) in useCommunityService to prevent Vue reactive Proxy
  from wrapping PerspectiveProxy, which breaks TypeScript #private fields
  (WeakMap lookup fails when 'this' is a Proxy instead of raw instance).

- Add { immediate: true } to sidebar watchers so data loads on first render
  instead of waiting for a change event that may never fire.

- Null-safe signallingService for imported perspectives without a
  neighbourhood (LinkLanguageFailedToInstall).

- Private perspective fallback: try private://UUID when neighbourhood://
  URL lookup fails in createCommunityService.

- Temporarily skip Conversation.findOne cache population — hangs on
  perspectives without a link language. Sidebar still renders with
  channelId + lastActivity timestamps.
…terns

- Convert all GRAPH ?g { s p o } patterns to direct default-graph triples
- Use rdf:reifies <<( s p o )>> for link metadata (author, timestamp)
- Updated files: channel/index.ts, conversation/index.ts,
  conversation-subgroup/index.ts, registerMobileNotifications.ts
- Add PREFIX rdf: where reifier patterns are used
- Companion to ad4m/core Oxigraph 0.5.7 upgrade
- restoreNeighbourhoodPrefix now passes through URLs that already contain ://
- Router guard matches perspectives by raw UUID after stripping private:// prefix
…mplates

- SidebarList: use item.channel?.id with channelId fallback for v-for key
- SidebarItem: guard c.channel before accessing .id in nested channel check
- CommunityView: optional chain on channelData.channel in channel card grid
- aiStore: replace non-null assertions with optional chains on channel.id
- routeUtils: guard restoreNeighbourhoodPrefix against undefined input
…on ordering

Channel.allItems() and Channel.unprocessedItems() now query
flux://transcript_started_at and use it (when present) instead of the
link-creation timestamp. This ensures transcription messages sort by
when speech occurred rather than when Whisper processing completed.

Matches the existing pattern in ConversationSubgroup.itemsData().

Also consolidates channel test files into a single channel.test.ts.
- Fix ESLint: remove @vue/prettier extend (ESLint 7 + Prettier 3 crash)
- Add perlin.js to .eslintignore (vendored file)
- Fix auto-fixable lint errors (prefer-const, no-inferrable-types)
- Fix no-inner-declarations in aiStore (function → arrow)
- Add vue-tsc as devDependency
- Add typecheck script to app/package.json
- Add typecheck task to turbo.json + root package.json
- Add Lint step to build.yaml and tests.yaml (blocking)
- Add Type check step to build.yaml and tests.yaml (informational, non-blocking)
- Fix aiStore: use Channel instead of ChannelSummary for unprocessedItems()
Replace perspective.addListener('link-added') anti-pattern with a
targeted SPARQL subscription via perspective.subscribeQuery(). The old
approach fired a callback on EVERY link in the perspective and filtered
client-side. The new approach registers a SPARQL query scoped to this
channel's ad4m://has_child links — filtering happens server-side in
Oxigraph, firing the callback only when this channel's items change.

Root cause: the useLiveQuery(Conversation) subscription only fires when
Conversation entities change (name/summary/subgroups), not when new
messages arrive (which are Channel children). This left unprocessedItems
stale, blocking both the timeline UI updates and the AI summarization
trigger chain.

Also adds test coverage for:
- Conversation model (stats, topics, subgroupsData, processNewExpressions)
- Timeline subscription lifecycle, debouncing, and AI trigger chain
- Convert done-callback tests to async/await (Vitest 2.x compat)
- Fix subscription race condition with isUnmounted guard
- Use ChannelSummary instead of Channel for processing queue
- Replace ineffective vi.doMock with hoisted vi.mock
- Replace silent if-guards with explicit assertions in tests
- Guard navigateToChannel against undefined channelId
- Sort items by effective timestamp after mapping (transcriptStart)
- Type COALESCE sentinel as xsd:dateTime for consistent ordering
…onversations

getRecentConversations() and getPinnedConversations() were not populating
the conversationCache after the SPARQL migration, so processesNextTask()
could never look up the Conversation instance and processNewExpressions()
was never called — summaries were silently skipped.

Fix: instantiate Conversation directly from SPARQL results instead of
using findOne() (which hangs on perspectives without a link language).

Also consolidates synergy e2e tests into conversation.test.ts (31 tests)
covering stats, topics, subgroupsData, processNewExpressions pipeline,
full synergy LLM flow, Channel.unprocessedItems, and cache lookup.
Add sequence counters to prevent out-of-order async responses from
overwriting current state when users switch channels quickly.
Also handle fetch failures and clear stale selectedPlugins in the
ManageChannelPluginsModal else branch.

Addresses CodeRabbit review feedback on PR #590.
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 29, 2026

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bb36aa47-5181-4fc1-b27a-f2117d5497c4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • ✅ Review completed - (🔄 Check again to review again)
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/sparql-1.2-cleanup

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@HexaField HexaField changed the title feat: migrate kanban-view + table-view from Prolog to SHACL APIs (Phase 9) feat: JS query/subscription elimination — Phases 4-6, 9 (Prolog→SPARQL, addListener→subscribeQuery, N+1 elimination, view migration) Apr 29, 2026
@HexaField HexaField changed the title feat: JS query/subscription elimination — Phases 4-6, 9 (Prolog→SPARQL, addListener→subscribeQuery, N+1 elimination, view migration) feat: migrate views from Prolog to SHACL, replace addListener, eliminate N+1 patterns Apr 29, 2026
…PerspectiveProxy

PerspectiveProxy (via Ad4mClient) uses WeakMap-backed #private fields.
Vue's ref() wraps the array contents in a deep reactive Proxy, which
causes WeakMap lookups to fail when methods are called on the proxied
instances.

shallowRef avoids deep wrapping — the array itself is reactive but its
elements remain raw PerspectiveProxy instances.
- topic/index.ts: linkedConversations/linkedSubgroups Prolog→SPARQL
- semantic-relationship/index.ts: 5 embedding methods Prolog→SPARQL
- conversation/util.ts: findEmbeddingSRId Prolog→SPARQL
- Removed all perspective.infer() calls from API model layer
- ~190 lines of Prolog query code replaced with SPARQL
- flux-container: subscribeQuery for Channel changes (was addListener+isSubjectInstance)
- CommentSection: subscribeQuery + Message.findAll (fixes N+1: was N×get())
- CommunityGraph: fix listener leak (missing removeListener cleanup)
- TimelineColumn: subscribeQuery for channel children
- useAssociations: subscribeQuery for link changes
- subscribeToLinks: marked @deprecated
- Kept addListener in useCommunityService (needs link.author for participant tracking)
PollCard:
- Add include: { votes: true } to useLiveQuery(Answer) — votes hydrated with answers
- buildAnswerData() now reads answer.votes directly (sync, no queries)
- removePreviousVotes() reads pre-loaded answer.votes instead of querying

useCommunityService:
- Flatten nested Promise.all into two phases: link queries then conversation lookups
- All Conversation.findOne calls execute in a single flat Promise.all batch
- Re-group results by space channel ID after batch completes
…se 9)

Kanban-view:
- Board.tsx: replace all .infer() with listRegisteredClasses(),
  getNamedOptions(), addNamedOption(), ensureSDNASubjectClass(Task)
- Board.tsx: remove getSubjectProxy/setValue, use createSubject with
  initialValues and Ad4mModel.update() via live query entries
- Entry.tsx: replace .infer() with getNamedOptions()
- New TaskModel.ts: @model class replacing Task.pl Prolog SDNA
- Delete Task.pl (22 lines)

Table-view:
- TableView.tsx: replace .infer() with listRegisteredClasses()
- Table.tsx: replace .infer() with getNamedOptions(), replace
  getSubjectProxy update with direct link manipulation via getClassShape()
- Header.tsx: replace .infer() + getSubjectProxy with
  getInstanceClasses() + getSubjectData()
- Entry.tsx: same migration as Header.tsx
- History.tsx: same migration as Header.tsx
- NewClass.tsx: rewrite generateSDNA() Prolog string builder to
  buildSHACLShape() using SHACLShape + perspective.addShacl()

Other:
- createCommunity.ts: fix .timestamp -> .createdAt (deprecated alias removed)

Zero Prolog infer() calls remain in any Flux view source.
… cache code

- useCommunityService.ts: participant tracking via subscribeQuery on CHANNEL predicate
- CommunityGraph.tsx: graph refresh via subscribeQuery on source/target
- cache.tsx: delete unused subscribeToPerspective/unsubscribeToPerspective (zero callers)
- usePerspectives.ts: mark addListeners as @deprecated
Remove addListeners(), onLinkAdded(), onLinkRemoved() from usePerspectives.ts.
Remove gotNewMessage stub and unused imports from MainView.vue.
Perspectives no longer register link callbacks — subscriptions handle updates.
Add hooks-helpers, ad4m-react-hooks, ad4m-vue-hooks to the pnpm
overrides so CI uses the branch's versions (which removed the
Subject import) instead of npm 0.13.0-test-2.
Phase 11 (cf61daa) removed the useRoute import but left the template
reference to route.params.communityId, causing TypeError at runtime.
…ripping

- useCommunityService.ts: revert participant tracking to addListener
  (subscribeQuery was called with 3 args but API takes 1 SPARQL string;
  addListener needed because participant tracking requires link.author)
- CommunityGraph.tsx: fix subscribeQuery to use correct 1-arg API with
  onResult callback and dispose cleanup
- Conversations.vue: strip literal:string: prefix from channel IDs in
  router navigation
- package.json: fix pnpm overrides to use link: paths to local ad4m
  repo (was using file:./ad4m/core which didn't exist)
@HexaField HexaField force-pushed the feat/sparql-1.2-cleanup branch from 6849cfa to ed365f3 Compare April 29, 2026 12:59
@HexaField HexaField marked this pull request as ready for review April 29, 2026 13:08
The ifDefined import from 'lit-html/directives/if-defined.js' resolved to
lit-html@1.4.1 (via pnpm root node_modules), but LitElement's render()
uses lit@2.8.0's template engine. The 2.x renderer doesn't recognize 1.x
directive functions (different branding: WeakMap vs _$litDirective$
property), so it stringified the function — causing minified JS code to
appear as placeholder text in j-input and other components.

Fix: import from 'lit/directives/if-defined.js' which always resolves to
the lit-html version bundled with lit@2.8.0.
@HexaField HexaField changed the base branch from feat/sparql-1.2 to dev May 4, 2026 04:17
@HexaField HexaField changed the title feat: migrate views from Prolog to SHACL, replace addListener, eliminate N+1 patterns feat: RDF 1.2 reifier migration + Prolog→SHACL + addListener→subscribeQuery + N+1 elimination May 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant