Design/editorial overhaul - #2023
Open
piyushmahajan7749 wants to merge 403 commits into
Open
Conversation
…s-ui Feature/shadowai projects UI
Reported: "Predicted to fall short - 100% confidence - hit 26%", while the prose underneath claimed "well beyond the 25% target ... hit the target in 100% of 500 simulated runs ... typical reduction 47.8%". Cause: `interp` is reassigned inside the improve loop, so after the loop it describes the OPTIMIZED parameters - but the reported meetRate, distribution and meetsTarget all come from firstNums (the design as written). The headline was therefore built from a different simulation run than the numbers beside it. - The round-1 interpretation is now pinned as `baseInterp` and supplies every headline field: predictedResults, confidence, gapAnalysis and gotchas. These match the reported meetRate/distribution by construction. - The optimized round goes where it belongs: optimizedChanges now lists the parameter edits that actually improved the simulation (accumulated from the loop) plus any further suggestions, instead of leaking into the headline. - confidence is clamped to 0..1, and the prompt states it is confidence in the VERDICT, not the hit-rate - being highly confident a design falls short is correct and must not be inflated to agree with the hit-rate. - The interpreter is now forbidden from quoting figures from a different, better-performing parameter set, or from describing a design as succeeding when the supplied hit-rate is below the reliability threshold. - UI: "100% confidence" relabelled to "100% sure of this verdict" with a tooltip, since the bare label read as a contradiction next to "Predicted to fall short". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Groundwork for keeping report generation inside the design Export flow instead of navigating to a separate page. The whole editor - section editing, per-section regenerate, chart regeneration, templates, preview, PDF/PPTX export, save - lived in the 1,076-line report route. It now lives in components/reports/report-editor.tsx as <ReportEditor>, taking reportId / workspaceId / locale plus: - mode: "page" keeps the standalone route's chrome; "modal" drops the Back button and lets a host dialog own the outer frame. - onSaved / onRequestClose so a modal host can close and refresh. The route is now a thin wrapper that reads params and renders it, so /reports/[reportId] behaves exactly as before. No editing logic was rewritten - the component moved verbatim apart from its inputs and the two layout branches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Export rail's "Generate report" used to create the report, close the modal and navigate to /reports/[id]. Now the whole lifecycle happens in the modal. - Generate no longer navigates. The modal switches from the compact setup form to an EXPANDED frame (92vh, up to 1200px) hosting <ReportEditor mode="modal">, which sees generation_status === "generating" and runs the outline pass itself. Every editing affordance comes with it: section editing, per-section regenerate, chart regeneration, templates, preview, PDF/PPTX export. - New ReportGeneratingView replaces the content area while the draft is being written: the real stages of the outline pass (reading data files, extracting measured values, matching against what the design set out to measure, drafting sections, building figures) tick through with completed ticks. The final stage holds until the draft lands, so it never claims to have finished work it can't confirm - the stages are timed, because /api/report/outline is one call with no per-step signal to subscribe to. - Saving marks the report is_saved and notifies the host, which closes the modal and refreshes the asset list. - Saved reports now appear as ASSETS under the Export rail's buttons, filtered by source_design_id (already stored on every report, so no new linkage). Each has open / download-as-PDF / delete, with a confirm on delete since it is permanent. Clicking one reopens it in the same modal via initialReportId. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e Design step
The Validate and Iterate tabs are gone from the stage rail - the journey now
ends at Design. In their place, ONE button on the design step ("Validate &
iterate") opens a modal with the two paths side by side, mirroring the
hypothesis stage's Generated/Create-your-own pattern:
- "Simulate the results" - the pre-lab Monte-Carlo run (old Validate tab).
- "Use your own lab data" - upload/paste results, parse, verdict, insights,
suggested changes (old Iterate tab).
Same ValidateTab component renders both; only its host changed, so every
existing behaviour (apply changes -> new version, iteration history, charts)
is intact.
Applying changes closes the modal and lands on the Design step while the
regeneration runs, so the researcher watches their design update and finds the
iteration recorded in the version rail with its provenance label.
Also routed: the post-design gate's "Simulate before the bench" now opens the
modal (kicking the simulation immediately); "Continue with this design" closes
it and opens Export; the full-auto pipeline stays on the Design tab during its
sim rounds instead of switching to a tab that no longer exists. The stepper is
4 stages; the modal can't be dismissed mid-regeneration.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It sat under the full protocol at the bottom of the tab, below the fold on any real design, so the researcher had to scroll the whole document to discover it. It now renders as a compact banner directly under the phase banner - visible the moment the design loads - and is hidden while reading a historical version (read-only context, nothing to iterate). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ave now joins the top row Report flow fixes: 1. The Export rail listed every row the generate flow created (one per attempt, draft state), so a single report showed up 4 times. Assets are now ONLY reports the researcher explicitly saved (is_saved), and the card shows just the date. 2. "Download as PPT" removed from the report editor. 3. "Upload to ELN" restored in its place - the full pre-B2C wiring recovered: connection loading, the connect modal when no ELN is linked (then straight into export), and the export modal against the linked ELN. 4. Report asset cards are open-only - download/delete buttons removed. Design step: - The bottom "Next" (which opened the finalise-vs-simulate gate, itself below the fold) is now "Save now" in the same top row as Validate & iterate, and finalises directly - the two choices the gate offered are the two buttons on this row. Only Regenerate remains at the bottom, next to the content it regenerates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s, not assumptions Auto-mode mckaywrigley#16 and Create-your-own #2 + #3. - Effort is now Low / Medium / High. "extra" and "max" stay in the type and config so designs saved with them still resolve (to high) instead of silently dropping to medium, they're just no longer offered. Medium and High were rebalanced to cover the range the retired levels used to (4 rounds/35 papers and 6 rounds/60 papers). - The chosen value is now ALWAYS present among the options and flagged "chosen", and is pre-selected. The model frequently omitted its own pick from the list, so the user couldn't select the very thing it had already decided - which is usually the right answer. - Language: "we assumed X" reads as guesswork to a scientist when these are literature- and input-backed decisions with a stated rationale. Now "Selected: X" under "N choices made for you", with "Confirm all" / "Apply my changes" replacing "Keep all assumptions" / "Apply my values & rebuild". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…, version rail always visible, tutorial button moved Auto-mode #8, mckaywrigley#12, mckaywrigley#14, mckaywrigley#15. 14. The share dialog DID have a submit control, but it was labelled "Add" and sat in a non-wrapping row, so on a narrow dialog it was pushed out of the box - hence "there is no share button". The row now wraps, the email input has a min-width so it shrinks instead of squeezing its neighbours out, and the button says "Share". 8. The "Validate & iterate" banner stayed active after Save now, still offering apply/skip against a finalised design - which would silently fork what the researcher just committed to. Now hidden once the design is approved. 12. The version rail was `hidden lg:block`, so it disappeared below 1024px. Saving opens the Export panel and squeezes the reading width, which is why the version list looked like it vanished on save. It now renders at all widths (w-40, widening to w-48 on lg). 15. The tutorial button is `fixed` top-right, the same lane every page header puts its actions in, so it overlapped "New design" on the project page. It now sits bottom-right, out of every header's way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ols, show version context Auto-mode mckaywrigley#9, mckaywrigley#10, mckaywrigley#11. 10. The chat proposed a change, said it had applied it, and nothing changed. The apply path is sound - it errors loudly when a patch fails - so the model was narrating an edit it never proposed as a <design-patch> block. Only approving a patch card changes the design, so: - The chat prompt now forbids past-tense completion claims outright ("I've updated…", "the design now says…"), requires any response that describes an edit to carry a patch block, and tells it to say plainly it can't express the change as one rather than describe it as done. - Safety net in the renderer: claimsAppliedWithoutPatch() detects applied-language with no patch block and shows "No change was made to the design" under the message, so a slip can't quietly mislead. 9. The chat rail's resize handle spanned the full height at z-20, translated half outside, so it sat on top of the header close button. It now starts below the header. 11. The rail header shows which design the chat is working against - "Editing v3 · saved · 2 earlier versions". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… Edit made visible Auto-mode #2 and mckaywrigley#13, per the product decisions taken this session. #2 - Designs produced by the hands-free pipeline now read as finished output: only the hypothesis it selected is listed (the other candidates are noise once it has decided), and the choices panel is not shown. A persisted `autoGenerated` flag on the design content drives this, so a reload keeps the simplified view. mckaywrigley#13 - Per-section editing already existed (DesignSectionContent -> onEditSection) but its Edit button was opacity-0 until you hovered that exact section, so it was effectively undiscoverable - hence "we only have the chat option". It is now a permanently visible bordered control on every section, giving the same section-scoped free-text editing the report has. Note on #2: in auto mode nobody answered any questions, so the values behind the protocol are entirely the model's and are now not surfaced anywhere on the design step. Flagged at decision time; recorded here so it's a known trade. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…enerates Auto-mode #5, Create-your-own #4/#6/#7 - the "ask first, build once" decision. Success criteria, replicates (+ n), constraints (material/time/equipment), variables (known/unknown) and additional details already existed as fields on the Problem page, but were never ASKED - so the researcher met them only after the hypotheses and design had been built without them, which is why the problem page read like a stale older version. A StudyDetailsModal now collects them at the one moment they can still steer the run, on every entry point: - Auto mode: after Create, before the hands-free pipeline starts. Auto mode never stops again, so this is the only chance to steer it. - Generate hypotheses (non-auto): before generation. - Create your own -> "Answer questions & generate": before the questions. - Create your own -> typed hypothesis: after the hypothesis is entered, before it is adopted (#7). Everything is optional and skippable; answers persist to content.problem so every downstream phase call reads them from the same object. They feed the HYPOTHESIS and DESIGN prompts only - never the literature search, which is what diluted results when problem-stage questions last existed. replicateCount is new: "3 per condition" now reaches the design prompt as an exact instruction instead of being left to the generator. Retires the post-generation choices panel (shipped earlier today from the CEO note): with the questions asked up front, confirming afterwards would ask twice and pay for a second generation. The ledger itself stays - the generator still records what it decided - we just no longer gate the design on reviewing it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e version Auto-mode #3 and #4. The simulation verdict, the recommendations and the iteration insights only ever existed for the LATEST version - they live in content.validation, which is a single current-state object. So stepping back through the series showed an old protocol with no reasoning attached, and there was no way to see what a previous round had concluded or why it was superseded. DesignVersionSnapshot now carries an `outcome` captured at snapshot time: the verdict in plain language, the meet-rate and whether it hit the target (simulation rounds), the insights (lab-data rounds), and the changes that were actually applied to produce the next version. Reading a historical version now shows that above the protocol - "What this version showed", the hit-rate badge, the verdict, any insights, and a "Changed for v(n+1)" list. The evidence travels with the version instead of being overwritten by the next round. Applies to both paths, so the iterate (lab-data) verdict and suggested changes are visible per version too, not only inside the modal at the time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… on assets Report #1 and #5. #1 - While the draft was being written the tabs stayed clickable, but every one of them rendered the same progress view, so it looked as though Overview, Inputs and Report had all turned into the generating screen. All three are now disabled for the duration and re-enable the moment the draft lands. #5 - Report assets in the Export rail showed a bare date, which says nothing about which of several reports off one design is newest. They now show "Saved <date> · <time>" from updated_at (falling back to created_at for rows written before this), and the list sorts on the same field so the most recently saved is always first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Report #3. The tab opened with Template and Objective - both already collected in the setup modal - so what the researcher came to check (which data is actually attached) sat underneath a duplicate of the setup form, and the files couldn't be opened at all. A "Files in this report" card now leads the tab, listing the attached data and reference documents as clickable rows that open the file in a new tab via a signed storage URL. The pickers and settings stay below for adding, removing and regenerating - they're still needed, so removing them outright would break regeneration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hind a button Report #4. The one-pager jumped straight to Objective and never said what problem the work addressed or why the design was built that way - so it didn't function as the at-a-glance record of the experiment it was meant to be. It now reads top-to-bottom as Problem -> Objective -> Design rationale -> Method -> Result -> Conclusion, with problem and rationale mapped from the draft's background/principle/hypothesis sections. The result chart was embedded inline at poster scale, where the axis labels were unreadable and it explained nothing. Overview now carries a short data line plus a "Show data visualization" button, and the chart opens full-size and titled in its own dialog (max 1000px, 70vh) where it can actually be read. Dataset selection within that dialog is report #2 and still to come - this change is what makes room for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…on it came from Report #6 and #7, per the decision taken this session. #6 - An ELN entry is a quasi-regulatory record, so once a report is filed our copy must keep matching it rather than drifting. - A successful ELN upload now stamps eln_uploaded_at (+ eln_entry_url when the provider returns one). - A filed report is LOCKED: section editing and the preview are read-only, and the editor shows an amber banner with the upload date/time, a link to the ELN entry, and the way forward. - "Create revised version" clones it into a fresh, editable report (revision_of points back), leaving the filed copy untouched so it still matches the entry. The revision uploads as its own new ELN entry, so the history is an audit trail rather than a mutated record. - In the Export rail, unfiled assets get Edit + Delete back (delete confirms); filed ones show an "ELN" badge instead and cannot be edited or deleted from there. #7 - Reports now record source_design_version and source_captured_at at creation, taken from the iteration selected in the Export rail. With several versions of a design in play, "from this design" didn't say what the report actually documents. The asset card shows "from Latest (v3)". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…erator Create-your-own #1 - "the questions are very similar to the previous version". They were, and here's why: the clarify prompt only ever received title, problem statement, objective, domain and phase. The success criteria, replicate choice, material/time/equipment constraints, known/unknown variables and operating parameters the researcher supplies - now collected up front by the study-details step - never reached it. So the generator had no idea any of it had been answered and dutifully asked for it all again. The existing "never ask for information already provided" rule was unenforceable because the information wasn't being provided. All of it is now in the problem block, with two added rules: don't re-ask anything the researcher supplied (sharpen an ambiguous value instead of asking from scratch), and derive the questions from what THIS study actually leaves undetermined rather than working through a fixed checklist. On the model question: nothing in the clarify path is model-conditional - MODEL() is just getDesignDeployment(), and the same prompt goes to whatever deployment is configured. The sameness was the missing context plus a prescriptive checklist, not the model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ch between them Report #2. The visualization only ever showed one dataset because the outline prompt explicitly said so: "If the data has multiple metrics, choose the most scientifically relevant one" - and the schema only had room for one data array. Everything else the researcher uploaded was discarded at generation time, so no amount of UI could get it back. - The chart schema gains `additionalSeries`: one entry per OTHER metric, each with its own name, axis label with units, chart type and data. The primary metric stays as `data` so the rendered PNG and older reports are unaffected. - The prompt now requires those other metrics to be returned rather than dropped, and says why: the researcher uploaded them and expects to see them. - The full-size visualization dialog lists the metrics as chips and swaps the interactive chart between them. With one metric it renders exactly as before; older reports with no additionalSeries fall back to the stored PNG. Note this only helps reports generated from here on - the discarded metrics were never persisted, so existing reports have nothing to switch to and will keep showing their single chart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… bench prep table
Six things the researcher flagged, all upstream of the UI.
Hypotheses came back as parallel single-agent dose screens - lysine at five
levels, then proline at five levels - with no pairing anywhere. One of the five
generation lenses is now explicitly combinational, and the diversity rules
carve out combinations so "lysine alone" and "lysine + proline" are no longer
collapsed as duplicates: they make different mechanistic claims. A combinational
hypothesis now states its own comparison set, since a combination never
compared to its parts cannot be attributed.
Designs overran the stated condition budget because the ceiling was written as
guidance ("prefer the smallest set") and read as a suggestion. It is now a hard
ceiling that counts controls and baselines, with an explicit count-and-cut check
before the table is returned, and a shortfall is logged as an assumption rather
than silently absorbed by adding arms.
The objective, the success criteria and the structured material/time/equipment
constraints were being collected from the researcher and then never passed to
either the hypothesis or the design agents. They are now in both, under an
explicit priority order - objective, then success criteria, then constraints,
then the hypothesis. Known/unknown variables sit at the bottom of that order and
are labelled low-weight: usable as context, never a reason to add an arm.
New "Condition Preparation" section between Material Preparation and the run:
one row per arm, the volume of each named stock, the buffer make-up that closes
the volume, and the final concentration delivered - so the arms can actually be
pipetted.
Clarify asked nothing at all. Two different causes produced the same empty
result and both were swallowed: the round-1 prompt paired "prefer fewer" with a
hard "never re-ask what was provided", and the catch returned the same
{questions: [], done: true} shape a genuine failure did. Round 1 now requires
6-8 questions and retries once on empty, real failures propagate, and the client
offers a retry rather than sliding silently past the step. The depth bar is
explicit too - one level past the obvious, since "good but not complete" was the
complaint.
…n the last Saving a design marked the phase approved, and approval was doing far more work than it should have been. It hid the Validate & iterate row, so the ordinary sequence - save, then stress-test - dead-ended; it stripped the per-section Edit pencil, leaving chat as the only way to change a word; and because "Save now" was also gated on it, an edit made afterwards had no way back to saved. All three now key off the thing they actually care about: Validate is always available (and says plainly that iterating creates a new version rather than overwriting the saved one), sections stay editable, and the save button returns as "Save changes" whenever a design has unsaved edits. Auto mode iterates on its own, so opening Validate showed only the newest simulation - the original design's result was nowhere, and there was no way to see what the iteration had bought. The version rail now advertises each version's own hit rate with a link into it, and the modal opens with the full series: every round's verdict, how often it hit, and exactly which changes were carried into the next version. That series is also what was missing on the iterate side, where the verdict and suggestions only ever rendered for the latest round. Multi-section chat edits went one section per turn, so the researcher had to keep asking it to carry on. The extractor already handled several patches in a message; only the prompt forbade it. It now opens with a numbered plan of the sections it will change and emits one independent, individually-approvable patch per section, capped at six. Also: the patch card wrapped its actions over the section label in the narrow chat rail; the design tab counted only the current version's designs, so two rounds of iteration still read "2 designs"; auto mode offered a "Create your own" hypothesis tab that leads nowhere; the share dialog's submit sat inside the field's border beside the role select and read as chrome, so a typed address had no visible way to be sent - it is now its own button naming the invitee, and Done sends rather than discards a pending address; and the export panel now always names the design and the exact version an asset will be built from.
…fter save
The multi-dataset fix from yesterday didn't work, and the reason is worth
recording: `additionalSeries` was added to the chart-RENDERING tool's schema,
but the field the report persists comes from `VisualizationSchema`, which never
had it. Structured output returns only what the schema declares, so however
firmly the prompt asked for the other metrics, the model could not emit them
and they were dropped at the source. The field now lives on the schema the
visualization agent is actually bound to.
With the data arriving, the report can offer it. The Visualization block lists
every dataset found in the file as a tick-able chip, renders each selected one,
and persists the choice on chart_data.selectedSeries - so what the researcher
picks is what goes into the report, the export and the ELN. Defaults to the
primary metric alone, which is how every existing report behaves.
Chart labels stopped colliding: past 8 categories the x-axis ticks rotate 45°
and anchor at their right edge, long unbroken labels ("F2-Glycine-50mM") are
hard-wrapped instead of overflowing, and past 12 the per-bar value labels and
per-slice pie labels give way to the tooltip and legend.
Saving a report was being treated as freezing it, so reopening one with the
edit icon produced a read-only preview - no title field, no section pencils, no
save button to commit anything. Only an ELN upload locks a report now, since
that record has to match what was filed. The title itself is editable in place;
it was a plain heading, so the name the generator chose was the name you kept.
Overview boxes cut their text at 420 characters wherever that landed, so the
one-pager read as a column of half-sentences. Each box now holds whole
sentences up to what fits - the full text is one click away in the report tab.
Templates: the three built-ins are relabelled "Report format", which is what
they are, freeing "Template" for the ones a researcher saves from a finished
report. Those are now pickable when generating - previously saving one had no
effect you could ever observe - and they are listed in the Export rail beside
the reports, where clicking one starts a report already set to it.
Two pre-existing test failures (clarify.test.ts, pipeline.test.ts) are
unrelated and fail identically on the parent commit.
…promotion Five defects, each with a cause worth naming. Duplicate papers. De-duplication keyed on `doi || url || title`, which only catches byte-identical repeats from one arm. The same study from PubMed and from the web has a different URL, often a DOI on only one of them, and titles differing by a trailing period or a " | Journal | Publisher" tail - so both survived. Matching now runs on the DOI and on a normalized title independently, the more authoritative source wins (PubMed > OpenAlex > S2 > arXiv > Scholar > web), and the survivor inherits the abstract, journal and citation count the loser had. A second pass at the Paper level catches copies that arrive in different rounds, and carries the `selected` tick across so de-duplicating can never drop a paper the researcher had chosen. The design questions were never asked. Not the clarify engine - the caller: approving hypotheses skipped straight to generation whenever the chosen hypothesis was one we generated, on the reasoning that our own hypothesis already encodes the direction. But direction was never what these questions are for; they ask what is on the bench - stock concentrations, diluent, material on hand, instrument settings - which no hypothesis can supply. Since picking a generated hypothesis is the common path, most runs were asked nothing. Hypotheses ignored the literature. The generation agents cite their sources, and `hypotheses-phase` dropped `provenance` on the floor and set `basedOnPaperIds: []` unconditionally. Ranking then scored the hypothesis sentence alone, so it rewarded whichever sounded boldest. Citations are now captured, shown to the ranker under a 30%-weighted grounding criterion, mapped back to paper ids, and appended to the reasoning as "Built on:". Auto mode - which carries one hypothesis forward with nobody choosing it - prefers the top-ranked hypothesis that actually cites something. Promoting an old version created a new one. Restoring v1 appended a v4 that was a copy of v1, so the timeline grew every time you looked back and the numbers stopped meaning anything. It is now a swap: the live set moves into history under its own number, the promoted version leaves history keeping its number, and the count is unchanged. A new `currentVersionNumber` records which. Tables came back as walls of pipes after an iteration. The bullet-promotion normalizer that splits " - " and " 1. " onto their own lines was running over table rows too, so a cell reading "20 mM - pH 6.0" broke its row in half and GFM stopped seeing a table. Table rows are now exempt, and a table butted against the paragraph above it gets the blank line the parser needs. Also: regenerating a version no longer leaves the superseded design on screen with every control greyed out - the pane shows which version is being built and the changes going into it. And all calculations - buffer, stock, excipient, dilution and final sample prep - must now be shown as separate stocks mixed together, one C1V1 = C2V2 line per component with water closing the volume, rather than a single collapsed figure.
…eses stop wiping ours Writing your own hypothesis deleted the generated ones. `applyOwnHypothesis` built `[own]` and replaced the list outright, so the one moment a researcher is best placed to write a sharp hypothesis - having just read four of ours - was the moment acting on it destroyed what they had read. Theirs is now added on top and selected, the rest stay. There is also an inline "write your own" block under the generated list, so it no longer means leaving the page for a tab. Version outcomes were a sentence. The elaborate simulation result - the distribution, run count, guardrails, gotchas, gap analysis, proposed changes - only ever existed for whichever simulation was most recent; once a version was superseded all that survived was a one-line verdict. The whole simulation is now stored on the version, and one shared component renders every occurrence: verdict always visible, full working behind an expander. Auto mode was worse than that - its snapshots recorded no outcome at all, so an unattended run left a stack of versions with no record of why any was superseded. It records the same as the manual path now. Reopening Validate for a new version showed the OLD version's results. The stored simulation described the design that had just been replaced, so the previous verdict and its full working framed the simulate button for the new one, with nothing saying which was which. It is cleared on regeneration; the old result lives on its version snapshot, which is what "Earlier rounds" reads. Also: designs are labelled Auto mode vs Built by you on the design page, the dashboard and the project canvas - the two were indistinguishable though they warrant different scrutiny; the auto loop shows the same regeneration progress as the manual path instead of sitting on the superseded protocol; and the Export rail and template picker hold a placeholder during their first fetch rather than asserting emptiness and then contradicting it a second later. Two pre-existing test failures (clarify.test.ts, pipeline.test.ts) are unrelated and fail identically on earlier commits.
Asking a design or report chat about a file got "I don't have access to files" — and the model was right, because nothing about the file was ever sent to it. Two separate cuts in the pipeline, both of which had to be repaired. The first: `use-chat-handler` skipped retrieval outright for design and report scopes. The reasoning was sound as far as it went — those chats already carry the whole design document in their system prompt, so re-retrieving it wastes budget and adds latency. But the document is not the FILES. Anything the researcher attaches lives in file_items and reaches the model only through retrieval, so attaching a file to a design chat did nothing whatsoever. The skip is gone; retrieval is scope-filtered, not workspace-wide, so it returns this design's own material rather than unrelated documents. The second only became visible once retrieval ran. The RPC ANDs `p_source_types` with `p_only_source_ids`, so a single call can search the design's document or files, never both — and the scope filter pinned it to the document. That matters because the chunker deliberately indexes only FILENAMES and descriptions for a design's or report's attachments, embedding their contents separately under source_type='file'. So the chat retrieved a list of filenames and truthfully reported it could not see inside any of them. `retrieve()` now makes a second, file-scoped pass for those two scopes and fuses both result sets through the existing scorer, which recomputes ranks over whatever rows it is handed. The pass is skipped when the caller already named specific files — those ARE the file pass, and a workspace sweep would only dilute them — and when the scope has no document filter to complement. A failing file pass is caught and logged rather than thrown: the document chunks from the first pass are still worth returning. Four tests cover the scope matrix. The two that assert the file pass fail against the previous code and pass against this one.
Answers one question every morning: is ShadowAI working end to end for a real
user? Playwright, run against the deployed app rather than a dev server - a
health check that passes on localhost while production is down is worse than
none.
Four projects. `setup` signs in once and captures the session plus the
workspace id, which is only ever expressed in a redirect URL. `public` checks
the login and signup pages and deliberately does NOT depend on setup: if the QA
password expires, whether login still renders is exactly what you want to know.
`smoke` covers the dashboard and design creation. `pipeline` runs the real AI
chain: literature -> hypotheses -> design -> report.
The pipeline drives the app's own HTTP endpoints rather than clicking. Those
phases take minutes each and are driven by polling, not by anything in the DOM;
clicking through them would mean twenty minutes of waiting on spinners and
asserting against selectors that move whenever the UI is restyled. UI breakage
is the smoke project's job. It is also never retried - a retry buys another
full run of Azure and PaperFinder spend to repeat what the first failure said.
Assertions are about substance, because shape assertions are worthless here: a
phase returning `{papers: []}` has "succeeded" and is completely broken. Each
check maps to a regression that actually happened - papers with null titles,
the same study arriving twice from PubMed and the web, hypotheses with their
provenance dropped, a conditions table coming back as prose instead of a table,
sections present but empty.
Cleanup is thorough and never fatal. The throwaway signup user is deleted in a
`finally` so it goes even when the assertion before it failed, since a failed
signup can still have created the auth row. Designs go through the app's API,
the staged CSV through storage. A leaked row is untidy; turning a healthy night
red over one would train us to ignore the email.
The digest is dependency-free ESM and always exits 0 - a reporter that needs
the app's build to work goes quiet exactly when it is needed. The runner's exit
code decides pass or fail, and it is checked after the email is sent.
Worth recording: the suite initially pointed at www.shadowai.work, which 404s
on /login. That apex domain serves the marketing site; the app is on
app.shadowai.work. Verified against production - the login checks pass there
now.
Jest's default testMatch would have picked up e2e/*.spec.ts and run browser
specs under jsdom, so e2e/ is excluded from it.
Brings 65 commits of the editorial overhaul onto main as one unit. origin/main was a direct ancestor of the branch, so this merge had nothing to reconcile - no conflicts were possible and none were resolved. Kept as --no-ff so the body of work stays identifiable in the history rather than being spread across main as loose commits. Headline changes since the last merge: Design generation - combinational hypotheses rather than parallel single-agent dose screens; the condition budget is a hard ceiling that counts controls; the objective, success criteria and structured constraints now actually reach the hypothesis and design agents (they were being collected and dropped); a new Condition Preparation section giving per-arm pipetting volumes; and all calculations shown as separate stocks mixed together with the arithmetic visible rather than a single collapsed figure. Literature - papers de-duplicated across sources by DOI and normalised title, so one study retrieved from both PubMed and the web appears once, with the more authoritative record winning. Chat - design and report chats can read uploaded files again. Retrieval was skipped entirely for those scopes, and the scope filter could only search the design document or files but never both. Reports - every metric in an uploaded file survives to the report and the reader can pick which appear; chart labels no longer collide; saving a report no longer freezes it. Versioning - promoting an older design version swaps rather than appending, so the timeline stops growing every time you look back. Nightly QA - a Playwright production health check with an emailed digest. Merging it to main is what lets its schedule fire; GitHub only runs cron workflows from the default branch.
Both had been failing for a while and both were the tests being out of date rather than the code. Merging to main made them matter: CI runs `npm test` as a blocking gate there, and a red default branch trains everyone to ignore it - including the nightly QA that now reports into the same place. pipeline.test.ts hardcoded `toHaveLength(5)` for the surviving hypothesis count. That count is derived from the effort level, and I retuned the tiers earlier (five levels down to three, medium now yielding 4), so the literal went stale the moment that landed. It now reads the expectation from resolveEffortConfig, which is the thing that actually decides it - retuning the tiers again won't fake a regression. clarify.test.ts expected "Vehicle, Untreated - plus a heat-stressed arm". The joiner was deliberately changed to "Vehicle, Untreated (additional context: plus a heat-stressed arm)" because downstream prompts were reading the dashed form as a single value and honouring only the chips, dropping the researcher's free text. The test now asserts the intended format. One trap worth recording: the new import initially went in at line 1, which displaced pipeline.test.ts's `@jest-environment node` docblock. That pragma is only honoured as the first thing in the file, so the suite silently fell back to jsdom and died on a missing `fetch` inside the openai client - an error that looks nothing like its cause.
CI has been failing on main since July with three TS2307 errors on PNG imports, while type-check passed on every developer machine. The divergence is the whole story: `next-env.d.ts` is gitignored because Next regenerates it on build, and it is what pulls in `next/image-types/global` - the declarations that make `import icon from "@/public/providers/groq.png"` resolve. This job runs type-check BEFORE build, so in a clean checkout the file does not exist yet and every image import fails. Locally it is always there, left behind by an earlier build, so the gate looked green to everyone who ran it by hand. Writing the file Next would generate, verbatim, before type-checking. Verified by reproducing it: deleting next-env.d.ts locally produces exactly the three errors CI reported, and restoring the generated file clears them. Committing the file instead would fight Next's own gitignore and go stale; reordering build ahead of type-check would trade a 30-second gate for a four-minute one.
The catch handler responded to a failed initializeApp by calling getFirestore() again. With no default app that throws "The default Firebase app does not exist" - from inside the catch, so nothing handles it. A missing service account stopped being a warning and became a hard crash. Where that bites is the build. `next build` collects page data for the API routes, which imports this module, so in any environment without Firebase admin credentials the build dies - which is exactly what has been failing CI on main, in the step after the type-check fix. adminDb is now left null on that path, so the failure surfaces at the call site, at request time, in an environment that was never configured, rather than taking down the build. Wherever credentials are present - production, Vercel, any developer with a .env - the try succeeds and this path never executes, so nothing about configured behaviour changes. Verified by reproducing CI locally: with FIREBASE_PROJECT_ID, CLIENT_EMAIL and PRIVATE_KEY unset and the same placeholder public env the workflow uses, the build failed on this exact error before and completes after.
Second half of getting main green. With type-check fixed, the Build step then failed - and kept failing on a different route each time: SerpApi, then the Jarvis vault, then Inngest. One cause behind all of them. `next build` collects page data for every API route, which imports each route's module graph and runs its top-level initializers. Several construct a client at import time and throw when their key is absent. The original comment here assumed "the build reads real secrets only at request time" - true of the handlers, not of the module scope they live in - so a clean checkout could never finish the build, while any developer machine sailed through on .env.local. These placeholders exist only to get those constructors past construction. Nothing is invoked during a build, so no value is ever used for anything; real credentials still belong in the runtime environment. Verified by reproducing CI properly: .env.local moved aside, only the workflow's own env present. The build exits 0. Worth noting that an earlier attempt looked like it passed when grepped for "Build error occurred" and had in fact failed - the exit code was the grep's, not the build's. The check that counts is the exit code. The durable fix is lazy initialization in those modules, so importing a route never constructs a client. That is a refactor across several files and a separate piece of work; this unblocks the gate today.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.