feat(taxonomy): Custom Behaviors Phase 5 — scoped gardening + auto-garden (LAT-760)#4037
Open
andresgutgon wants to merge 11 commits into
Open
feat(taxonomy): Custom Behaviors Phase 5 — scoped gardening + auto-garden (LAT-760)#4037andresgutgon wants to merge 11 commits into
andresgutgon wants to merge 11 commits into
Conversation
…rden (LAT-760) Make custom behaviors a filter-scoped LIVING taxonomy with the same trend affordances as the global Behaviours page (born/continue/die, novelty, new-this-week, spiking, time window, segments), with no structural schema change. Also make them fully automatic like the global tree: no manual trigger. - Periodic scoped cron sweep: gardenCustomBehaviorSweep enqueues gardenCustomBehavior (reason "cron") for each eligible behavior on the same 6h cadence as the global gardenSweep. Cross-org admin query listGardenableCustomBehaviors (live project, past the cadence throttle). One additive nullable column last_gardened_at, stamped at run start. - Auto-start on create: createCustomBehaviorUseCase enqueues the first run; the manual Generate/regenerate UI + generateCustomBehaviorFn are removed. generateCustomBehavior domain use-case stays (reused by create + dev/QA). - Scoped getClusterTrendCounts windowed over custom_behavior_assignments.start_time wired into the scoped branch of listProjectBehavioursUseCase (was hardcoded []). Scoped getClusterAssignmentCounts gains an optional start_time window so the time picker filters volume too. - Re-enable segments + time filter on the scoped custom-behaviours page; the pending state is reframed as an auto-garden waiting state (preview count vs the 15-observation threshold), distinct from generating/ready/failed. - Lineage continuity (Hungarian matcher against the behavior's own prior clusters) and don't-truncate write semantics (insert-only upsertMany, ReplacingMergeTree) were already in place from Phase 2; covered with tests and documented in dev-docs/taxonomy.md. Global Behaviours page + Topics filter unchanged; scoped reads never touch global taxonomy_observations.assigned_cluster_id. Out of scope: per-run membership-migration history (global lacks it too). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds “living” scoped custom behaviors by introducing automatic periodic gardening + cadence throttling (last_gardened_at), wiring scoped trend reads over custom_behavior_assignments.start_time, and re-enabling the scoped UI with segments + time windows (while removing the manual Generate/Regenerate trigger).
Changes:
- Add Postgres
custom_behaviors.last_gardened_at+ admin query to list gardenable custom behaviors for cross-org sweeps. - Add ClickHouse scoped trend/count reads (assignment counts now optionally windowed; new trend counts API), and wire into
listProjectBehavioursUseCase. - Add worker cron sweep + auto-garden on create; update workflows + web UI to reflect the auto-garden waiting/generating/failed states.
Reviewed changes
Copilot reviewed 30 out of 31 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/platform/db-postgres/src/schema/custom-behaviors.ts | Adds lastGardenedAt column to Drizzle schema. |
| packages/platform/db-postgres/src/repositories/gardenable-custom-behaviors.ts | Adds cross-org admin query for sweep eligibility. |
| packages/platform/db-postgres/src/repositories/gardenable-custom-behaviors.test.ts | Tests eligibility filtering (live vs demo/deleted + throttle). |
| packages/platform/db-postgres/src/repositories/custom-behavior-repository.ts | Adds markGardened repository method. |
| packages/platform/db-postgres/src/index.ts | Re-exports new gardenable-custom-behaviors query/types. |
| packages/platform/db-postgres/drizzle/20260715083432_add-custom-behavior-last-gardened-at/migration.sql | Adds last_gardened_at column in Postgres. |
| packages/platform/db-postgres/drizzle/.migration-lock | Updates migration lock metadata. |
| packages/platform/db-clickhouse/src/repositories/custom-behavior-assignment-repository.ts | Adds scoped trend query + time-windowed assignment counts. |
| packages/domain/taxonomy/src/use-cases/read-use-cases.test.ts | Adds test proving scoped trend is derived from assignment slice. |
| packages/domain/taxonomy/src/use-cases/list-project-behaviours.ts | Wires scoped assignment counts + trend counts into read use-case. |
| packages/domain/taxonomy/src/use-cases/custom-behavior-crud.test.ts | Updates create test to assert auto-start gardening. |
| packages/domain/taxonomy/src/use-cases/create-custom-behavior.ts | Enqueues initial gardening run (best-effort) after create. |
| packages/domain/taxonomy/src/use-cases/build-custom-behavior-taxonomy.test.ts | Adds “accumulate/no-truncate” assignment slice test. |
| packages/domain/taxonomy/src/testing/fake-custom-behavior-repository.ts | Adds markGardened support to fake repository. |
| packages/domain/taxonomy/src/testing/fake-custom-behavior-assignment-repository.ts | Adds windowed counts + fake trend counts for tests. |
| packages/domain/taxonomy/src/ports/custom-behavior-repository.ts | Adds markGardened to repository port. |
| packages/domain/taxonomy/src/ports/custom-behavior-assignment-repository.ts | Adds window params + getClusterTrendCounts to port. |
| packages/domain/taxonomy/src/index.ts | Exports new cron constants. |
| packages/domain/taxonomy/src/constants.ts | Adds custom-behavior sweep cron constants. |
| packages/domain/queue/src/topic-registry.ts | Registers gardenCustomBehaviorSweep task payload. |
| dev-docs/taxonomy.md | Documents scoped gardening + trend semantics. |
| apps/workflows/src/activities/taxonomy-custom-behavior-activities.ts | Stamps last_gardened_at at run start via markGardened. |
| apps/workers/src/workers/taxonomy.ts | Implements runGardenCustomBehaviorSweepJob fanout enqueue. |
| apps/workers/src/workers/taxonomy.test.ts | Adds tests for custom behavior sweep enqueue/throttle/continue-on-failure. |
| apps/workers/src/server.ts | Registers new repeatable gardenCustomBehaviorSweep cron. |
| apps/web/src/routes/_authenticated/projects/$projectSlug/custom-behaviours/$behaviourSlug/index.tsx | Passes full project object into detail component. |
| apps/web/src/routes/_authenticated/projects/$projectSlug/custom-behaviours/-components/custom-behaviours-list.tsx | Removes manual Generate/Regenerate action from list UI. |
| apps/web/src/routes/_authenticated/projects/$projectSlug/custom-behaviours/-components/custom-behaviour-detail.tsx | Re-enables segments + time filters; adds auto-garden waiting state UI. |
| apps/web/src/domains/taxonomy/custom-behaviors.functions.ts | Removes generate server fn; provides QueuePublisher for auto-start on create. |
| apps/web/src/domains/taxonomy/custom-behaviors.collection.ts | Removes useGenerateCustomBehavior hook. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+18
to
+21
| // Scoped-gardening throttle anchor (LAT-760): stamped at each run start so | ||
| // the periodic sweep skips a behavior gardened within the cadence window. A | ||
| // behavior gardens for its whole lifetime; deleting it is the only off | ||
| // switch. Additive/nullable — null means never gardened (swept first). |
Comment on lines
+374
to
+378
| .scheduleRepeatable( | ||
| "taxonomy", | ||
| "gardenCustomBehaviorSweep", | ||
| { triggeredAt: new Date().toISOString() }, | ||
| { key: CUSTOM_BEHAVIOR_GARDENING_CRON_KEY, pattern: CUSTOM_BEHAVIOR_GARDENING_CRON_PATTERN, tz: "UTC" }, |
Comment on lines
+196
to
+199
| const triggeredAt = new Date(payload.triggeredAt) | ||
| const anchor = Number.isNaN(triggeredAt.getTime()) ? new Date() : triggeredAt | ||
| const gardenedBefore = new Date(anchor.getTime() - CUSTOM_BEHAVIOR_GARDENING_MIN_INTERVAL_MS) | ||
| const behaviors = yield* deps.listGardenableCustomBehaviors(gardenedBefore) |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c31e0fde00
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+196
to
+198
| const triggeredAt = new Date(payload.triggeredAt) | ||
| const anchor = Number.isNaN(triggeredAt.getTime()) ? new Date() : triggeredAt | ||
| const gardenedBefore = new Date(anchor.getTime() - CUSTOM_BEHAVIOR_GARDENING_MIN_INTERVAL_MS) |
There was a problem hiding this comment.
For this new repeatable sweep, payload.triggeredAt is the value supplied when workers call scheduleRepeatable at startup, not the cron fire time; I checked packages/platform/queue-bullmq/src/adapter.ts, where that payload is stored as fixed scheduler data. After the first run stamps last_gardened_at, this frozen anchor makes gardenedBefore stop advancing, so long-lived workers stop finding the same behavior eligible until a restart. Use the actual execution time here instead of the scheduler payload for the throttle cutoff.
Useful? React with 👍 / 👎.
| customBehaviorId: behavior.custom_behavior_id, | ||
| reason: "cron", | ||
| }, | ||
| { dedupeKey: taxonomyGardenCustomBehaviorDedupeKey({ organizationId, customBehaviorId }) }, |
There was a problem hiding this comment.
This recurring sweep publishes with only a bare dedupeKey; the BullMQ adapter maps that to a stable jobId, and the worker retains completed jobs, so a create-time run or the first cron run shadows later publishes with the same behavior id instead of enqueueing a new job. In practice, custom behaviors will not keep re-gardening on later sweeps; use a TTL-based throttle/dedup option or another per-run idempotency strategy.
Useful? React with 👍 / 👎.
Comment on lines
+74
to
+78
| if (behaviour.status === "failed") { | ||
| return ( | ||
| <div className="flex flex-1 flex-col items-center justify-center gap-3 p-8 text-center"> | ||
| <Icon icon={AlertTriangleIcon} size="lg" color="destructive" /> | ||
| <Text.H4>Last gardening run failed</Text.H4> |
There was a problem hiding this comment.
When an auto-started run has fewer than TAXONOMY_GARDENING_MIN_OBSERVATIONS, the workflow marks the behavior failed, so this branch runs before the preview.data && hasEnough === false waiting state. Any normal low-volume behavior will therefore show a destructive “Last gardening run failed” message instead of the intended waiting-for-data state, even though the preview count proves it just lacks enough observations.
Useful? React with 👍 / 👎.
|
📄 Generated an interactive HTML explanation of this PR's diff against
Self-contained artifact, no external requests. Generated with Claude Code. |
…T-760) Scoped gardening now uses the SAME sample-window source as global gardening instead of a separate custom lookback. Retire CUSTOM_BEHAVIOR_LOOKBACK_DAYS and introduce one shared constant TAXONOMY_GARDENING_SAMPLE_LOOKBACK_DAYS (renamed from TAXONOMY_NOISE_LOOKBACK_DAYS, still 7d) that BOTH global and scoped gardening reference, so the two can never drift. Pure de-dup — global value and behavior unchanged. - Global call sites (build-hierarchical-taxonomy, taxonomy-gardening-activities, the sweep eligibility count) now reference the shared constant. - Scoped sampling (listForCustomBehaviorSample) and the preview (countForCustomBehaviorSample) both derive `since` from the shared constant. - Drop the Q2 "pass-as-param-for-user-selectable" framing: removed the lookbackDays field from the scoped workflow step; the build activity computes `since` from the shared constant directly (window tracks the global model, not per-behavior selectable). Activity-only change, workflow structure unchanged. - Updated port/activity/preview comments and dev-docs accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fold the standalone /custom-behaviours section into /behaviours now that a custom behavior is functionally the global tree + a FilterSet. One tree visualization codepath, gated behind the customBehaviors flag so the flag-off UI is unchanged. - Single shared body `behaviours-page.tsx` (one BehavioursView + one BehaviourDetailDrawer), parameterized by an optional customBehaviour; the Phase-5 custom-behaviour-detail.tsx duplication is deleted, not adapted. - New flag-gated top row `behaviours-scope-header.tsx`: title + filter summary, scope selector (Global + up to 10 customs), "New behavior", and a ⋯ menu (Edit → edit page, Delete → confirm) for the selected custom behavior. - Routes: /behaviours/$behaviourSlug (view), /behaviours/new + /behaviours/$behaviourSlug/edit (full-page create/edit: name + filters sidebar + live all-time session preview + 7-day readiness strip), all flag-gated. Old /custom-behaviours/* become 301 redirects. - Rewire the Sessions + saved-search "Create custom behavior" entry points to /behaviours/new?filters=…; remove the custom-behaviours nav entry. - Lifecycle per the agreed model: no status badge; the no-tree state is the preview-driven waiting-for-data view; status stays internal (polling only); last_gardened_at is never shown. Flag off: /behaviours renders exactly as before, new routes redirect out, no "Custom behaviors" nav. routeTree.gen.ts regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… (LAT-760) Replace the behavior create/edit screen's bespoke session list with the real Sessions InfiniteTable + click-to-detail panel, extracted into a reusable `FilterSessionsPreview` (filters left, matching sessions right) driven by a `filters` / `onFilterChange` pair so other domains (e.g. experiments) can embed the same "preview what a filter selects" UI. - New `-components/filter-sessions-preview.tsx`: wraps SessionsView (selectable=false) + SessionDetailDrawer, owning sorting/active-session state. Optional `excludeFilterFields` hides fields a domain can't use. - Thread `excludeFilterFields` through SessionsView → FiltersSidebar → FiltersBuilderFields (so custom behaviors keep `topics` hidden). - behaviour-form-page now renders name + 7-day readiness strip, then the reusable preview; the lean ad-hoc list is gone. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The create/edit screen nested FilterSessionsPreview's ListingLayout inside the form's own Layout; ListingLayout's root is h-full, so under the header + name it took full height and rendered the filters sidebar + sessions table below the fold — the preview looked blank and Create stayed disabled (no way to add a filter). Make FilterSessionsPreview own the single Layout and accept the form chrome via a `header` slot (toolbar above, sessions body below, detail drawer as the sibling aside — the proven Sessions-view structure). No more layout nesting. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The create/edit screen is a pure filter editor now: name + filters + live session preview. A fresh behavior's "waiting for enough matching sessions" situation is already explained by the waiting state on the behavior view after save, so the 7-day observation-count strip on the form was redundant noise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both `topics` and `moments` are the taxonomy's own output, so scoping a custom behavior on either is circular — a behavior defined by a behavioural moment conflates the filter with the tree it produces. Generalize the single-field exclusion into `CUSTOM_BEHAVIOR_EXCLUDED_FILTER_FIELDS = ["topics","moments"]`: the shared Zod contract rejects both, `stripCustomBehaviorExcludedFields` drops both, and the create/edit filter builder hides both (it now references the domain constant instead of hardcoding). Reverses LAT-746 Q3 (which kept moments) per product decision. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…T-760) The no-tree state showed a spinner + "the tree appears when the next run completes" even when no run was in flight — implying imminent work while the next scheduled sweep could be hours out. Split it by the real signal: - under threshold → "Waiting for matching sessions" (preview-driven, unchanged) - status === "generating" (a run IS in flight) → spinner "Building the tree" - enough data but idle → no spinner, clock icon, "Scheduled for clustering … after the next run, which can take a few hours" (sets the cadence expectation) Also constrain the waiting/empty-state copy to max-w-md so it wraps at a readable width instead of spanning the full content column. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-760) Drop internal jargon (observations, matches, clustering) from the no-tree states in favor of "sessions"/"tree"/"run", keeping the progress number: - under threshold → "Found N of 15 matching sessions so far…" - run in flight → "Grouping this behavior's sessions into a tree now…" - idle, enough data → "Waiting for the next run — N matching sessions found…" Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Say "this behavior" instead of "this behavior's tree"/"into a tree" in the no-tree states — "tree" is internal vocabulary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reword the second sentence away from jargon ("cluster a filtered slice") to
plain intent: "You can also create your own custom behavior to discover the
patterns within a filtered set of sessions."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Phase 5 of Custom Behaviors (LAT-760). Makes a custom behavior a filter-scoped living taxonomy with the same trend affordances as the global Behaviours page — born/continue/die, novelty, new-this-week, spiking, a real time window, and segments — with no structural schema change (one additive nullable column). Also makes custom behaviors fully automatic like the global tree: creating one auto-starts gardening and a cron sweep keeps it living; there is no manual trigger.
The reframing: a gardened custom behavior is the same kind of thing as the global tree, filter-scoped — not a different model. Global derives trend from latest snapshot +
start_timewindows + stable cluster ids (Hungarian matcher), andcustom_behavior_assignmentsalready mirrors that shape, so this is algorithm + read + cron wiring, not schema.Changes
1. Periodic scoped cron sweep (
apps/workers)taxonomy/gardenCustomBehaviorSweeptask +CUSTOM_BEHAVIOR_GARDENING_CRON_*constants, registered inserver.tson the same 6h cadence as the globalgardenSweep.runGardenCustomBehaviorSweepJobfans out onegardenCustomBehavior(reason: "cron") per eligible behavior, deduped on the behavior's workflow id.listGardenableCustomBehaviors(live project — same demo/showcase/soft-delete exclusions aslistGardenableProjectRefs; past the cadence throttle). Admin/RLS-bypass because the sweep is a system job with no single "current org" — RLS can't express a cross-tenant list; each enqueued job re-scopes to its own org.custom_behaviors.last_gardened_at, stamped at run start as the throttle anchor. No enabled flag — a behavior gardens for its whole lifetime; deleting it is the only off switch.2. Auto-garden, no manual trigger
createCustomBehaviorUseCaseenqueues the first gardening run on create (best-effort; the sweep backstops).generateCustomBehaviorFnserver function +useGenerateCustomBehaviorhook. ThegenerateCustomBehaviordomain use-case stays (reused by create and the dev/QA helper).3. Scoped trend read (Part 4)
CustomBehaviorAssignmentRepository.getClusterTrendCounts, windowed overcustom_behavior_assignments.start_time(scoped mirror of the global read), wired into the scoped branch oflistProjectBehavioursUseCase(was hardcodedtrendCounts = []).getClusterAssignmentCountsgains an optionalstart_timewindow so the time picker filters volume too.4. Re-enabled UI (
apps/web, Part 5)segment/onSegmentChange/timeFilter/timeRange(viauseAnalyticsTimeWindow+TimeFilterDropdown) to the reusedBehavioursView.5. Lineage + write semantics (Parts 2 & 3 — already shipped in Phase 2, now tested + documented)
listActiveByProject({ customBehaviorId })), so cluster ids survive regeneration.upsertManyinto aReplacingMergeTree); nothing truncates the scoped slice on re-run, so history accumulates until TTL. Added an explicit accumulate/no-truncate test.dev-docs/taxonomy.md.Invariants held
custom_behavior_id IS NULL, scoped reads never touchtaxonomy_observations.assigned_cluster_id.reassignment_run_idinto the sort key.Validation
pnpm typecheck— full workspace green (87/87).@domain/taxonomy(99) incl. new scoped-trend + accumulate/no-truncate tests;@platform/db-postgres(9) incl. newlistGardenableCustomBehaviors+ migration-applied repo tests; create-CRUD updated for auto-start.taxonomy.test.ts, incl. the new sweep-handler cases) run in CI — thechdbnative binary isn't built in this worktree, so they're skipped locally (pre-existing env limitation, not a code issue).chdbisn't available here. Best verified against the LAT-752 QA seed (which includes a ≥42-obs generatable cohort and an under-threshold cohort for the waiting state).Follow-up split, if this is too large
The cron/worker + schema (Part 1 + auto-garden) is separable from the read + UI (Parts 4–5) if a smaller PR is preferred; they meet only at
last_gardened_atand the scoped read branch. Editing a behavior's filter does not yet expedite a re-garden (it lands on the next sweep) — a candidate follow-up.🤖 Generated with Claude Code