feat: RDF 1.2 reifier migration + Prolog→SHACL + addListener→subscribeQuery + N+1 elimination#591
Open
feat: RDF 1.2 reifier migration + Prolog→SHACL + addListener→subscribeQuery + N+1 elimination#591
Conversation
Add Channel.recentConversations() and Channel.pinnedConversations() static methods that use single SPARQL queries instead of iterative channel.get() loops. - Channel.recentConversations(perspective, limit): single query with ORDER BY DESC(?lastActivity) LIMIT — replaces N×M×K iterative walk - Channel.pinnedConversations(perspective): single query for pinned channels with their conversation IDs - Both methods deduplicate by channelId and handle errors gracefully Tests prove single SPARQL call per invocation (not N+1).
…dels - ChannelSummary: Channel without @hasmany relations — no hidden graph exploration during sidebar/list hydration - MessageSummary: Message without SPARQL getter properties (replyingTo, isPopular) — retains simple @hasmany (reactions, thread, replies) - Both exported from @coasys/flux-api barrel Tests verify no relation/getter queries fire on lightweight models.
- useCommunityService: use ChannelSummary instead of Channel for allChannels live query — eliminates @hasmany relation hydration on every link change - Replace N+1 getPinnedConversations with Channel.pinnedConversations() - Replace N×M×K getRecentConversations with Channel.recentConversations() - Replace iterative getChannelsWithConversations with Conversation.findOne() - Update handleParticipantTracking for ChannelSummary (no participants prop) - MessageList.tsx: add lazy evaluateGetters for visible messages (replyingTo) — works with WS-2 deepQuery inversion where getters are skipped by default Tests verify scoped queries and no iterative patterns remain.
- Remove perspective.addListener('link-added', handleLinkAdded) from
TimelineColumn.vue — was firing on EVERY link in the perspective
- Remove handleLinkAdded, getDataFull, getDataIncremental, debounce state
- Replace with useLiveQuery(Conversation, perspective, { parent: channel })
— only fires when Conversation instances under this channel change
- Watch conversationInstances for reactive refreshAllData() calls
- Import useLiveQuery, remove onUnmounted (handled by useLiveQuery)
Tests verify no raw listeners, no old handler functions, and scoped
useLiveQuery usage.
All WS-3/4/5/6 tests now strip // and /* */ comments before pattern matching, preventing false failures from comment text like 'No @hasmany relations' or 'replaces perspective.addListener'. All 72 tests pass.
Code comments now describe what the code does rather than referencing internal planning workstream identifiers. Renamed test files for clarity.
The previous query applied LIMIT before deduplication, so a busy channel with many items could consume all LIMIT rows, starving other channels from the result set. Switch to GROUP BY ?channelId with SAMPLE/MAX aggregation so the LIMIT applies after per-channel grouping. Oxigraph supports full SPARQL 1.1 aggregation (already used elsewhere in this file via COUNT(DISTINCT ...)), so remove the misleading comment about GROUP BY availability. The client-side seen Map is kept as a safety net but should no longer be needed.
Rapid watch(conversationInstances, ...) triggers could fire refreshAllData() concurrently. Without a guard, a slow earlier call could resolve after a newer one, overwriting fresher state. Add an in-flight promise guard with a pending flag: concurrent calls coalesce onto the running promise, and if a call arrived while in-flight, one additional refresh runs after the current one completes.
The getPinnedConversations and getRecentConversations tests silently passed when the regex failed to match (the if-block was skipped). - Add expect(match).not.toBeNull() so a regex miss is a real failure - Apply comment-stripping in both tests for consistency
…subscriptions - Remove refreshAllData(), getConversations(), getUnprocessedItems(), refreshInFlight, refreshPending, and refreshTrigger from TimelineColumn - Replace with reactive watchEffect that maps conversationInstances directly from the existing useLiveQuery subscription - Add separate watchEffect for unprocessed items, triggered by conversationInstances changes - Extract sidebar refresh into a dedicated watch on conversation name - Extract AI task check into a dedicated watch on unprocessedItems - Remove onMounted — reactive watchEffect handles initial load - Remove refreshTrigger prop from TimelineBlock (and recursive children) - TimelineBlock now watches props.data instead of refreshTrigger - Add structural tests verifying removal of imperative patterns - All 20 structural tests pass, pnpm build succeeds
… proper unit tests - Remove jest, ts-jest, @types/jest from packages/api - Add vitest with config (globals, node environment) - Delete regex/structural test files that loaded source as strings: - community-service-scoping.test.ts - timeline-column-scoping.test.ts - channel-sparql-methods.test.ts - Rewrite ChannelSummary.test.ts: uses getModelMetadata() to verify no relations, expected properties, and flag metadata - Rewrite MessageSummary.test.ts: uses getModelMetadata() to verify no getter properties, expected relations, and flag metadata - Rewrite channel-query.test.ts: tests set-difference logic and three-query SPARQL integration pattern with mocked perspectives - Add channel.test.ts: tests Channel.recentConversations() and Channel.pinnedConversations() with mocked perspectives — verifies SPARQL query structure, result mapping, deduplication, error handling, and limit parameter - Existing parseLit.test.ts works unchanged (Vitest globals) All 38 tests pass across 5 test files.
The build script temporarily sets file: overrides for local linking. These should not be committed.
…ives
- Channel.recentConversations/pinnedConversations SPARQL queries used
string literal 'true' to match channel_is_conversation/channel_is_pinned,
but AD4M stores these as literal:json URIs. Use ad4m://fn/parse_literal
SPARQL function (consistent with buildSPARQLWhereFilters).
- markRaw(perspective) in useCommunityService to prevent Vue reactive Proxy
from wrapping PerspectiveProxy, which breaks TypeScript #private fields
(WeakMap lookup fails when 'this' is a Proxy instead of raw instance).
- Add { immediate: true } to sidebar watchers so data loads on first render
instead of waiting for a change event that may never fire.
- Null-safe signallingService for imported perspectives without a
neighbourhood (LinkLanguageFailedToInstall).
- Private perspective fallback: try private://UUID when neighbourhood://
URL lookup fails in createCommunityService.
- Temporarily skip Conversation.findOne cache population — hangs on
perspectives without a link language. Sidebar still renders with
channelId + lastActivity timestamps.
…terns
- Convert all GRAPH ?g { s p o } patterns to direct default-graph triples
- Use rdf:reifies <<( s p o )>> for link metadata (author, timestamp)
- Updated files: channel/index.ts, conversation/index.ts,
conversation-subgroup/index.ts, registerMobileNotifications.ts
- Add PREFIX rdf: where reifier patterns are used
- Companion to ad4m/core Oxigraph 0.5.7 upgrade
- restoreNeighbourhoodPrefix now passes through URLs that already contain :// - Router guard matches perspectives by raw UUID after stripping private:// prefix
…mplates - SidebarList: use item.channel?.id with channelId fallback for v-for key - SidebarItem: guard c.channel before accessing .id in nested channel check - CommunityView: optional chain on channelData.channel in channel card grid - aiStore: replace non-null assertions with optional chains on channel.id - routeUtils: guard restoreNeighbourhoodPrefix against undefined input
…on ordering Channel.allItems() and Channel.unprocessedItems() now query flux://transcript_started_at and use it (when present) instead of the link-creation timestamp. This ensures transcription messages sort by when speech occurred rather than when Whisper processing completed. Matches the existing pattern in ConversationSubgroup.itemsData(). Also consolidates channel test files into a single channel.test.ts.
# Conflicts: # pnpm-lock.yaml
… feat/sparql-1.2
- Fix ESLint: remove @vue/prettier extend (ESLint 7 + Prettier 3 crash) - Add perlin.js to .eslintignore (vendored file) - Fix auto-fixable lint errors (prefer-const, no-inferrable-types) - Fix no-inner-declarations in aiStore (function → arrow) - Add vue-tsc as devDependency - Add typecheck script to app/package.json - Add typecheck task to turbo.json + root package.json - Add Lint step to build.yaml and tests.yaml (blocking) - Add Type check step to build.yaml and tests.yaml (informational, non-blocking) - Fix aiStore: use Channel instead of ChannelSummary for unprocessedItems()
Replace perspective.addListener('link-added') anti-pattern with a
targeted SPARQL subscription via perspective.subscribeQuery(). The old
approach fired a callback on EVERY link in the perspective and filtered
client-side. The new approach registers a SPARQL query scoped to this
channel's ad4m://has_child links — filtering happens server-side in
Oxigraph, firing the callback only when this channel's items change.
Root cause: the useLiveQuery(Conversation) subscription only fires when
Conversation entities change (name/summary/subgroups), not when new
messages arrive (which are Channel children). This left unprocessedItems
stale, blocking both the timeline UI updates and the AI summarization
trigger chain.
Also adds test coverage for:
- Conversation model (stats, topics, subgroupsData, processNewExpressions)
- Timeline subscription lifecycle, debouncing, and AI trigger chain
- Convert done-callback tests to async/await (Vitest 2.x compat) - Fix subscription race condition with isUnmounted guard - Use ChannelSummary instead of Channel for processing queue - Replace ineffective vi.doMock with hoisted vi.mock - Replace silent if-guards with explicit assertions in tests - Guard navigateToChannel against undefined channelId - Sort items by effective timestamp after mapping (transcriptStart) - Type COALESCE sentinel as xsd:dateTime for consistent ordering
…onversations getRecentConversations() and getPinnedConversations() were not populating the conversationCache after the SPARQL migration, so processesNextTask() could never look up the Conversation instance and processNewExpressions() was never called — summaries were silently skipped. Fix: instantiate Conversation directly from SPARQL results instead of using findOne() (which hangs on perspectives without a link language). Also consolidates synergy e2e tests into conversation.test.ts (31 tests) covering stats, topics, subgroupsData, processNewExpressions pipeline, full synergy LLM flow, Channel.unprocessedItems, and cache lookup.
…spective-based data
Add sequence counters to prevent out-of-order async responses from overwriting current state when users switch channels quickly. Also handle fetch failures and clear stale selectedPlugins in the ManageChannelPluginsModal else branch. Addresses CodeRabbit review feedback on PR #590.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…PerspectiveProxy PerspectiveProxy (via Ad4mClient) uses WeakMap-backed #private fields. Vue's ref() wraps the array contents in a deep reactive Proxy, which causes WeakMap lookups to fail when methods are called on the proxied instances. shallowRef avoids deep wrapping — the array itself is reactive but its elements remain raw PerspectiveProxy instances.
- topic/index.ts: linkedConversations/linkedSubgroups Prolog→SPARQL - semantic-relationship/index.ts: 5 embedding methods Prolog→SPARQL - conversation/util.ts: findEmbeddingSRId Prolog→SPARQL - Removed all perspective.infer() calls from API model layer - ~190 lines of Prolog query code replaced with SPARQL
- flux-container: subscribeQuery for Channel changes (was addListener+isSubjectInstance) - CommentSection: subscribeQuery + Message.findAll (fixes N+1: was N×get()) - CommunityGraph: fix listener leak (missing removeListener cleanup) - TimelineColumn: subscribeQuery for channel children - useAssociations: subscribeQuery for link changes - subscribeToLinks: marked @deprecated - Kept addListener in useCommunityService (needs link.author for participant tracking)
PollCard:
- Add include: { votes: true } to useLiveQuery(Answer) — votes hydrated with answers
- buildAnswerData() now reads answer.votes directly (sync, no queries)
- removePreviousVotes() reads pre-loaded answer.votes instead of querying
useCommunityService:
- Flatten nested Promise.all into two phases: link queries then conversation lookups
- All Conversation.findOne calls execute in a single flat Promise.all batch
- Re-group results by space channel ID after batch completes
…se 9) Kanban-view: - Board.tsx: replace all .infer() with listRegisteredClasses(), getNamedOptions(), addNamedOption(), ensureSDNASubjectClass(Task) - Board.tsx: remove getSubjectProxy/setValue, use createSubject with initialValues and Ad4mModel.update() via live query entries - Entry.tsx: replace .infer() with getNamedOptions() - New TaskModel.ts: @model class replacing Task.pl Prolog SDNA - Delete Task.pl (22 lines) Table-view: - TableView.tsx: replace .infer() with listRegisteredClasses() - Table.tsx: replace .infer() with getNamedOptions(), replace getSubjectProxy update with direct link manipulation via getClassShape() - Header.tsx: replace .infer() + getSubjectProxy with getInstanceClasses() + getSubjectData() - Entry.tsx: same migration as Header.tsx - History.tsx: same migration as Header.tsx - NewClass.tsx: rewrite generateSDNA() Prolog string builder to buildSHACLShape() using SHACLShape + perspective.addShacl() Other: - createCommunity.ts: fix .timestamp -> .createdAt (deprecated alias removed) Zero Prolog infer() calls remain in any Flux view source.
… cache code - useCommunityService.ts: participant tracking via subscribeQuery on CHANNEL predicate - CommunityGraph.tsx: graph refresh via subscribeQuery on source/target - cache.tsx: delete unused subscribeToPerspective/unsubscribeToPerspective (zero callers) - usePerspectives.ts: mark addListeners as @deprecated
Remove addListeners(), onLinkAdded(), onLinkRemoved() from usePerspectives.ts. Remove gotNewMessage stub and unused imports from MainView.vue. Perspectives no longer register link callbacks — subscriptions handle updates.
Add hooks-helpers, ad4m-react-hooks, ad4m-vue-hooks to the pnpm overrides so CI uses the branch's versions (which removed the Subject import) instead of npm 0.13.0-test-2.
Phase 11 (cf61daa) removed the useRoute import but left the template reference to route.params.communityId, causing TypeError at runtime.
…ripping - useCommunityService.ts: revert participant tracking to addListener (subscribeQuery was called with 3 args but API takes 1 SPARQL string; addListener needed because participant tracking requires link.author) - CommunityGraph.tsx: fix subscribeQuery to use correct 1-arg API with onResult callback and dispose cleanup - Conversations.vue: strip literal:string: prefix from channel IDs in router navigation - package.json: fix pnpm overrides to use link: paths to local ad4m repo (was using file:./ad4m/core which didn't exist)
6849cfa to
ed365f3
Compare
The ifDefined import from 'lit-html/directives/if-defined.js' resolved to lit-html@1.4.1 (via pnpm root node_modules), but LitElement's render() uses lit@2.8.0's template engine. The 2.x renderer doesn't recognize 1.x directive functions (different branding: WeakMap vs _$litDirective$ property), so it stringified the function — causing minified JS code to appear as placeholder text in j-input and other components. Fix: import from 'lit/directives/if-defined.js' which always resolves to the lit-html version bundled with lit@2.8.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR combines the full SPARQL 1.2 migration for Flux — originally split across
feat/sparql-1.2(#590, now closed) andfeat/sparql-1.2-cleanup(#591). Together they deliver: all custom SPARQL queries migrated to RDF 1.2 reifier patterns, all Prolog replaced with SHACL APIs, all globaladdListenerpatterns replaced with targetedsubscribeQuery, and all N+1 query patterns eliminated.Companion: AD4M coasys/ad4m#803
Part 1: Migrate Flux SPARQL Queries to RDF 1.2 Reifier Patterns
(Originally #590)
Migrates all custom SPARQL queries in Flux API packages from the old named-graph
GRAPH ?g { s p o }pattern to direct default-graph triples with RDF 1.2 reifier patterns for link metadata.Changes
packages/api/src/channel/index.tsallItems()— direct triple pattern, no GRAPHunprocessedItems()— 3 subqueries converted (all items, processed items, set-difference)totalItemCount()— direct patternrecentConversations()— reifier for timestamppinnedConversations()— direct patternpackages/api/src/conversation/index.tsstats()subgroups query — direct patterntopics()— direct pattern (5 GRAPH vars → direct triples)subgroupsData()main query — reifier for timestamp onhas_childlinksubgroupsData()batch timestamp query — reifier for channel timestamppackages/api/src/conversation-subgroup/index.tsstats()— direct patterntopics()— direct patternitemsData()— reifier for both subgroup-item timestamp and type-link author, plus channel timestamptopicsWithRelevance()— direct patternapp/src/utils/registerMobileNotifications.tsOther
Pattern Migration
Before (named graphs):
After (RDF 1.2 reifiers):
Part 2: Prolog → SHACL, addListener → subscribeQuery, N+1 Elimination
(Originally #591)
26 files changed, 514 insertions, 659 deletions (net −145 lines)
After this part, zero
perspective.infer()calls, zero unfocusedaddListenerpatterns, and zero global link bus usage remain in Flux app/view source code.Prolog → SPARQL Conversions
topic/index.tslinkedConversations()/linkedSubgroups()→ SPARQLsemantic-relationship/index.tsconversation/util.tsfindEmbeddingSRId()→ SPARQLaddListener→ TargetedsubscribeQueryflux-container.tssubscribeQuerywith Channel SPARQLCommentSection.tsxsubscribeQueryon sourceCommunityGraph.tsxsubscribeQueryon source/target (was leaking, now properly cleaned up)useAssociations.tsxsubscribeQuerywith proper disposeTimelineColumn.tsxsubscribeQueryuseCommunityService.tssubscribeQueryon CHANNEL predicatecache.tsxsubscribeToPerspective/unsubscribeToPerspective(zero callers, containedaddListener)N+1 Query Elimination
useCommunityService.tsPromise.all→ two-phase batch (link queries then conversation lookups)CommentSection.tsxMessage.get()→Message.findAll({ parent })PollCard.tsxVote.findAll→useLiveQuery(Answer, { include: { votes: true } })kanban-view Migration
Board.tsx.infer()→listRegisteredClasses()+getNamedOptions()+addNamedOption()+ensureSDNASubjectClass(Task). Task creation usesinitialValues.Entry.tsx.infer(property_named_option)→getNamedOptions()TaskModel.ts@Modelclass with@Property({ options: [...] })for status enumTask.pltable-view Migration
TableView.tsx.infer(subject_class)→listRegisteredClasses()Table.tsx.infer(property_named_option)→getNamedOptions(). Updates usegetClassShape()for predicate resolution + direct link manipulation.Header.tsx.infer()→getInstanceClasses()+getSubjectData(). Updates viagetClassShape()+ link add/remove.Entry.tsxHistory.tsxNewClass.tsxgenerateSDNA()(raw Prolog) →buildSHACLShape()+addShacl(). RemovedmakeRandomPrologAtom().Global Link Bus Removal
MainView.vuegotNewMessage,onLinkAdded, and associatedaddLinkbus emissionsusePerspectives.tsaddListeners(),onLinkAdded(),onLinkRemoved(), and link callback arrays — the global link event bus is goneOther
createCommunity.ts:.timestamp→.createdAt(deprecated alias removed in companion PR)Depends On
AD4M coasys/ad4m#803 — provides Oxigraph 0.5.7 + RDF 1.2 reifiers, Rust model query engine,
getNamedOptions(),addNamedOption(),listRegisteredClasses(),getClassShape(),getInstanceClasses(),getSubjectData(),SHACLShape,@Property({ options }),perspectiveModelSubscribeExpected Performance Impact
addListenercallbacks (fired on every link change across every perspective) replaced with targetedsubscribeQuerythat only fires when matching predicates change. In a community with 10+ channels, this eliminates ~90% of spurious JS callback invocations.findAllwithinclude) replace per-item.get()calls, reducing round-trips from N+1 to 1-2 per list render.infer()calls (which loaded and evaluated the entire SDNA program) replaced with direct SHACL API lookups.