feat: bring product discovery into Ideation — an idea-to-decision flow that feeds Inception (2.2.11)#526
feat: bring product discovery into Ideation — an idea-to-decision flow that feeds Inception (2.2.11)#526akash-doshi wants to merge 5 commits into
Conversation
apackeer
left a comment
There was a problem hiding this comment.
Verdict: request changes
The hypothesis-testing loop this PR builds is real, and the non-breaking case for the stock scopes genuinely holds (details at the bottom). The main ask is a design one: the flow's central promise - that discovery's validated vision sets up delivery - is only half-wired. After that come three test failures and a rebase collision, all mechanical.
1. The vision does not mechanically reach Inception (the main ask)
The tenet this PR implements: Ideation's output is a validated vision, which sets up Inception (the implementation plan), which sets up Construction (code). The discovery flow builds that vision honestly: future-state + assumptions-record (1.9) feed test-plans + evidence-record (1.10), riskiest-first and append-only, and the decision pack (1.11) is the vision artifact, with a machine-readable handoff contract on commit.
But on the continue-here path, no Inception stage consumes any of it. Grepping every inception and construction stage plus all agents: zero references to decision-pack, evidence-record, assumptions-record, or future-state. Inception's entry stage (requirements-analysis) declares only intent-statement and scope-document, both optional - and the intent statement is written at 1.1, before discovery ran; none of the four discovery stages writes back to it. Whether delivery ever reads the pack is left to the conductor LLM's initiative, nudged only by a skip-reason string. The upstream-coverage sensor cannot catch this because it only verifies declared consumes. The hand-off path is fine (the receiving run ingests the pack as supplied material at its own materials-first 1.1); it is the flagship continue-here path that leaks.
The fix is small - see the inline comment on discovery-decision.md Step 8 for two concrete options. Related: stage-protocol-recovery.md:60 loads resume context for ideation 1.1-1.7 only, so a run resumed inside or after discovery never reloads the pack either.
2. Blockers (mechanical)
- Three deterministic test failures introduced by this PR. t189-compose-dispatch.sdk, t192-compose-front-journey.sdk, and t193-compose-report-journey.sdk pin the stock scope count at 9 (t189:103,108; t192:89; t193:68,96,100; plus t192:73 STOCK_SCOPES lacking discovery). All three fail with
expected 9, received 10at the PR head and pass at the merge base (A/B verified). The two e2e compose-front twins were updated for 9 -> 10; these three integration tests were missed. The PR body's "one known environmental failure (t92)" only holds in an environment where the sdk-live tests skip. - Version 2.2.2 collides. v2 already merged a different 2.2.2 and is currently at 2.2.10, so the PR needs a rebase and re-bump (GitHub already reports it conflicting on CHANGELOG.md, README.md, the five aidlc-version.ts copies, and tests/.coverage-registry.json). See the inline comment on aidlc-version.ts for a trap: the literal version string is baked into six shipped prose sites that t68 does not guard.
- Test number t199 is taken. v2 already has tests/unit/t199-learnings-memory-path.test.ts; this PR adds tests/integration/t199-discovery-scope.test.ts. run-tests.sh --filter matches by filename regex, so t199 becomes ambiguous. Please renumber to a free slot on rebase.
3. Majors (inline comments carry the detail)
- stage-protocol.md Part 4 now contradicts itself: the enterprise/feature template says /32 while its own example says /36.
- The commit-continuation verb sequence has zero test coverage and depends on the absence of a status guard a sibling verb already has.
- discovery-decision.md instructs direct state-tool calls that the orchestrator SKILL.md forbids; no other stage body does this.
- PR body accuracy: the parity fixtures actually show four added SKIP rows per stock scope (zero removals), not "exactly one removed SKIP row" - the real property is stronger than the claim, but the sentence is wrong. And "with nothing supplied, the clarifying questions run exactly as before" oversells: a stock run always gains one intake question, two always-produced records, and a new confirm-unconfirmed-answers duty at the gate (the CHANGELOG states this correctly).
4. Minors
- discovery-experimentation.md pre-decides the protocol's artifact re-use question (inline).
- discovery-current-state.md says "four parts"; the template has five H2s (inline).
- Two intent-capture ambiguities an executing agent can trip on (inline).
- Stale refs the sweep missed: docs/guide/glossary.md:51 ("Ideation 7"), docs/guide/workshop-mode.md:59 ("1.1-1.7"), docs/guide/14-artifacts-reference.md:148 (heading says 1.1-1.7 above a table containing 1.8-1.11), core/aidlc-common/protocols/stage-protocol-recovery.md:60 (ships in dists).
- t199's produces pin uses toContain rather than an exact match (inline).
What holds up well
- All 9 stock parity fixtures change by exactly four added SKIP rows with every other row byte-identical; walk fixtures untouched; no existing stage number shifts.
- The artifact guard passes on any one declared produce, so the added produces cannot create a hard-failure path; sensors stay advisory.
- The sensor/graph tool diffs are comment-only as claimed (verified byte-level); validateScope("discovery") returns clean with zero advisories.
- Every engine verb the stages name exists with the exact spelling and flags; keywords: [] makes the scope genuinely unreachable by inference.
- The ~18 count-pin test edits strengthen rather than weaken (t30 gains an exact discovery EXECUTE pin; t66's enterprise check became strictly stronger); the coverage ratchet only went up.
- bun scripts/package.ts --check is green on all four harnesses at the PR head, and settings.json changes announcement text only.
Verification
Full deterministic suite (smoke+unit+integration, 229 files) run at the PR head: 6 red. t121/t145/t165 pass green-alone (parallelism flakes); t189/t192/t193 stay red alone and pass at the untouched merge base, so those three are introduced here. Packaging parity green on all four harnesses.
| 4. Hand control back to the forwarding loop (`next`). Delivery begins at the | ||
| stage the engine named, with the intent statement, stakeholder map, and | ||
| the whole discovery record already in place — the delivery path's entry | ||
| stages consume the intent statement this workflow produced at 1.1. |
There was a problem hiding this comment.
This is the seam where the flow's central promise leaks. On the continue-here path the vision travels only as prose: no Inception stage consumes decision-pack, evidence-record, assumptions-record, or future-state (requirements-analysis declares only intent-statement and scope-document, both optional), and the intent statement being relied on here was written at 1.1, before any discovery ran - none of the four discovery stages writes back to it. So if the evidence narrowed or reframed the idea, delivery's entry stage is wired to read the day-one intent, and reaching the decision pack depends on the conductor's initiative. upstream-coverage cannot catch it because it only verifies declared consumes.
Two small fixes; either works, both is better:
- On commit, fold the chosen framing and recorded exclusions back into the intent statement before the scope-change, so the one artifact that does flow is fresh.
- Declare decision-pack as an optional consume on requirements-analysis (and ideally user-stories). Optional consumes are first-class since 2.2.9, and upstream-coverage would then verify Inception actually referenced the vision.
The hand-off path does not have this problem: the receiving run ingests the pack as supplied material at its own materials-first 1.1.
There was a problem hiding this comment.
Both options implemented, plus the same wiring one stage further where the leak repeated for visual content: optional decision-pack consumes on requirements-analysis and user-stories, the "What Discovery Validated" write-back before the scope-change, design-language-record and evidence-record consumes on refined-mockups, and the recovery protocol now spans 1.1-1.11. t214 pins the pack-present/pack-absent differential.
| **For enterprise and feature scopes** (all 32 stages active): | ||
| **For enterprise and feature scopes** (the graph holds 36 stages; enterprise and feature execute all 32 delivery-path stages — every stage except the 4 discovery-only ideation stages): | ||
| ``` | ||
| Progress: [N]/32 overall | [phase-N]/[phase-total] [Phase] stages complete. Next: [Next Stage Name] |
There was a problem hiding this comment.
This template still says /32 while the enterprise example at line 210 below was updated to "13/36 overall", and the all-other-scopes template uses /36. The base was consistent (/32 everywhere); this PR introduces the split, and a conductor following the prose verbatim will render one or the other nondeterministically. Either this line should be /36 (matching the CHANGELOG's own claim that denominators move to N/36), or the example needs to change.
There was a problem hiding this comment.
Fixed: the template line reads /36, and the numerator rule is now disambiguated (plan SKIP rows count in neither the numerator nor the total, runtime [S] skips count in both). The old wording left it computable two ways.
| // Bumped in the same commit that adds the matching ## [N.N.N] heading | ||
| // to CHANGELOG.md. Pinned by tests/unit/t68-version-changelog-sync.test.ts. | ||
| export const AIDLC_VERSION = "2.2.1"; | ||
| export const AIDLC_VERSION = "2.2.2"; |
There was a problem hiding this comment.
2.2.2 is already taken: v2 merged a different ## [2.2.2] and is now at 2.2.10, so this needs a rebase and re-bump (t68 will fail on the duplicate CHANGELOG heading otherwise).
One trap to catch during the re-bump: the literal string "as of 2.2.2" is baked into shipped prose that t68 does not guard - aidlc-sensor-required-sections.ts:47 and :163, aidlc-sensor.ts:404, aidlc-graph.ts:271, stage-protocol.md:745, and templates/.gitkeep:10, plus their dist mirrors. Recommend versionless wording ("the framework ships nine discovery defaults") so future re-bumps cannot strand them.
There was a problem hiding this comment.
Rebased onto 242953e, now 2.2.11. The six literal-version prose sites are versionless now, so future re-bumps cannot strand them.
| - **Commit** — append the handoff contract as a machine-readable section of the decision pack: every answered question with status, confidence, and source; every supported assumption as a requirement with its evidence attached; every refuted assumption as a recorded exclusion with the evidence as the reason; every still-unknown verdict exported as a low-confidence assumption to re-raise; every test artifact marked promote registered as a starting point. Then ask one follow-up structured question (same §3 machinery): **where does the build happen?** | ||
| - **Continue here** — the build happens in this workspace. The same workflow and the same record continue into delivery: proceed to Step 6 and the exit gate, then follow Step 8. | ||
| - **Hand off** — the build happens somewhere else (another repository, another team, a workspace that holds the code). Proceed to Step 6 and the exit gate; approval completes this workflow, and the decision pack's handoff contract is what the receiving team's workflow consumes. | ||
| - **Pivot** — everything learned is kept, and the initiative returns to the framing. Do NOT proceed to the exit gate and never mark this stage completed: after the §13 learnings ritual, relay the engine's backward jump — `bun {{HARNESS_DIR}}/tools/aidlc-jump.ts execute --target discovery-future-state --direction backward`. The jump re-opens discovery-future-state and the stages after it so their artifacts can be revised; nothing is deleted, and the evidence record is append-only — it is never rewound. |
There was a problem hiding this comment.
The orchestrator SKILL.md says never call the state tools directly (the engine's report dispatches them), but this stage instructs exactly that, here and in Step 8 - and no other stage body invokes aidlc-jump.ts or aidlc-state.ts directly. Mechanically the verbs exist and audit correctly (I verified each), so this is a contract inconsistency rather than a dead end, but a conductor obeying SKILL.md literally may refuse the move. One side needs an explicit carve-out.
There was a problem hiding this comment.
Carve-out stated on both sides: the SKILL rule (all four harness copies) names the stage-body relay exception, and the stage names the rule it excepts. Auditing that sentence surfaced a false claim of ours: set-status emits no audit event (the other four verbs do). Both surfaces now state the distinction.
| 1. Ask which delivery scope fits the committed build (a structured question: | ||
| `feature` is the default, `mvp` and `enterprise` are the usual | ||
| alternatives, or a composed scope via `/aidlc compose`), then relay | ||
| `bun {{HARNESS_DIR}}/tools/aidlc-utility.ts scope-change --scope <chosen>`. |
There was a problem hiding this comment.
This sequence has no test coverage, deterministic or live, and it works only because scope-change and next happen to lack the Completed-status guard that recompose already has (aidlc-utility.ts:3060 refuses non-Running state with a rationale that reads on exactly this situation). If that guard is ever extended to scope-change, this documented path breaks silently. Since this is the scope's advertised graduation path into delivery, one test pinning the sequence would make it load-bearing rather than accidental.
There was a problem hiding this comment.
t214 now drives the sequence against a completed-discovery fixture, asserts each half of the no-guard behavior separately, and includes a contrast test that recompose refuses the same state. Extending the guard to scope-change now fails a test instead of silently breaking Step 8.
| On re-entry (a new session, a resumed workflow), this stage re-runs from its | ||
| top per the standard stage ritual — but substep 1 reads what earlier cycles | ||
| already recorded, so work continues from the record rather than starting over. | ||
| At the artifact re-use question, the answer for `test-plans.md` and |
There was a problem hiding this comment.
This pre-decides the person's answer to the protocol's artifact re-use question - stage-protocol defines it as a 3-option structured question, and Redo overwrites files. An executing agent will either skip the question (protocol violation) or ask it and risk a Redo that rewinds the append-only evidence record. Suggest rewording as "recommend Modify, and warn that Redo would rewind the append-only evidence record", or adding an explicit protocol carve-out for this stage.
There was a problem hiding this comment.
Reworded as suggested: recommend Modify, and warn that Redo rewinds the append-only evidence record.
|
|
||
| ### Step 4: Capture the Design Language Record | ||
|
|
||
| Create or extend `<record>/ideation/discovery-current-state/design-language-record.md` with four parts: |
There was a problem hiding this comment.
"Four parts", but the shipped design-language-record template carries five H2s (## Gaps is the fifth), and the required-sections sensor checks output against the template's full H2 set. An agent following this prose alone writes four sections and collects an advisory SENSOR_FAILED. Say five, or name Gaps here.
There was a problem hiding this comment.
Fixed: the prose lists five parts with Gaps as its own heading, matching the template the sensor enforces.
| selected items through follow-up questions written into the same questions | ||
| file, each with a blank [Answer]: tag, before ending any turn that waits on | ||
| the person. Record declines as declines; never nag. When E is the answer, | ||
| skip ahead — the questionnaire path is unchanged. |
There was a problem hiding this comment.
"Skip ahead" reads like "jump to Step 5", but Step 4 still requires creating source-inventory.md recording "None". An agent taking the skip literally never writes the file - no hard failure (the artifact guard passes on any one declared produce), but the "an empty inventory is itself a fact the record keeps" intent silently fails. Worth a clause: "skip ahead after recording the empty inventory in Step 4".
There was a problem hiding this comment.
Now reads: skip ahead after recording the empty inventory in Step 4.
| Step 3, verbatim. Several asks can feed one initiative; an ask that turns | ||
| out to contain two problems is split into two rows and noted. | ||
|
|
||
| When either exists, add: |
There was a problem hiding this comment.
"When either exists" - the intended antecedent (a mandate or a decide-by date) is 70+ lines up in the intake options; the nearest conditional is "When the person brought asks" a few lines above, so a conductor can read "either" as "asks or materials" and emit empty Already Decided / Decide-By sections. Naming the two things explicitly removes the misread.
There was a problem hiding this comment.
Now names the referents: a mandate or a decide-by date.
| expect(node?.scopes).toEqual(["enterprise", "feature", "mvp", "poc", "discovery"]); | ||
| // The two produces the extension added — every scope that runs the stage | ||
| // gets them; downstream discovery stages consume them by these names. | ||
| expect(node?.produces).toContain("source-inventory"); |
There was a problem hiding this comment.
toContain lets a sixth, unintended produce pass silently; an exact toEqual on the ordered array would pin the append-only contract fully. The rest of this file is genuinely falsifiable - this is the one soft spot. (Separately: the t199 number is already taken on v2 by tests/unit/t199-learnings-memory-path.test.ts, and --filter matches by filename regex, so please renumber on rebase.)
There was a problem hiding this comment.
Renumbered to t213 (t203, the first choice, was also taken upstream in the interim). The produces pin is an exact ordered toEqual, so a sixth produce cannot pass silently.
…ery scope — ideation as an idea-to-decision flow (2.2.11)
…ner cells, plain-language gaps, prose pins (2.2.11)
…ntainer review (2.2.11)
…late heading names, sensor-form provenance, doc mirrors, and the design-language wiring into refined-mockups (2.2.11)
…ive-part record honest, who-runs-the-tests, hand-off transport truth, appetite elicitation, glossary gaps (2.2.11)
94a105a to
79aa58c
Compare
|
Cheers Arden, all items from the review are addressed on the updated branch (rebased onto v2.2.10, version 2.2.11). What changed and why, in the review's order: 1. Wiring the vision into Inception (the design ask) The gap was real and ran wider than the continue-here path: no Inception stage declared any discovery output as an input, so validated content reached delivery only through conductor initiative. Changes:
2. Blockers
3. Majors
4. Minors All applied: the artifact re-use guidance recommends Modify and warns that Redo would rewind the append-only evidence record (a stage file cannot pre-decide a protocol question), the two intent-capture ambiguities are resolved by naming their referents (record the empty inventory in Step 4 before skipping ahead; a mandate or a decide-by date), workshop-mode reads 1.1–1.11, and the t213 produces pin is an exact ordered list so a sixth produce cannot pass silently. Corrections and disclosure
Other Things
|
|
@raj-jain-aws @SiddhJog @spraja08 @leandrodamascena Can you guys take a look at this? I think directionally (and mechanically) it fits in with the model of Ideation produces an idea, vision, Inception produces a design or plan, Construction produces code etc. |
|
I ran a max-effort pass over this — nine angles, each finding reproduced or refuted against the PR tree, focused on what the four fix commits might have introduced. Short version: the code is healthy, all of Arden's findings landed, and what's left is purely rebase. Arden's findings — all fixed, I checked each:
New bugs: I could not find one. I ran the sensor greps against the templates, traced the commit/pivot/park continuation, diffed the discovery nodes against a core node, and checked the empty-keywords path (discovery is genuinely unreachable by inference). The stock scopes are byte-safe — the only fixture change is the four added SKIP rows, everything else identical. What actually blocks the merge — both rebase, neither is your code:
Non-blocking:
Net: the discovery flow itself is in good shape. Rebase onto v2, re-bump off 2.2.11, and the |
|
I had opened same request for AIDLC 1.0 that was rejected earlier. I am curious to learn the drivers of the new phase in AIDLC 2.0. What are the key goals and what is driving including it as that determines the correctness and scope of the solution. My solution I proposed for AIDLC has landed eventually here as open source solution https://github.com/aws-samples/sample-ai-plc |
|
Hi @RachnaC1234 thanks for your comment, please take some time to review this PR as keen to see your feedback on this approach - as it addresses most of the points you've described. You make great points around greenfield/brownfield and rapid prototyping to validate a decision, but also how stakeholders may differ in technical capability. It opens up a question where the start point and end point of AI-DLC workflows should be. While this PR lands the technical parts components of introducing a product discovery scope within ideation, a few areas need to be evolved:
To help steer in the right direction, let's frame these feedback and changes into RFCs to get feedback on the mental model and bring your expertise into the AI-DLC Ideation Stage. |
Implements the first slice of RFC #517 (Bring product discovery into the Ideation phase).
Summary
Ideation today starts every initiative from a blank questionnaire, and nothing between the first answer and the first build checks whether the direction is right. This PR brings product discovery into the Ideation phase as an opt-in flow. It extends the existing intent-capture stage in place so every scope works materials-first, and adds four new ideation stages under a new
discoveryscope that explore and validate an idea before any commitment to build: evidence how things work today, frame what the future could look like as genuinely different options, test the riskiest assumptions first, and end in an explicit decision with a plain meaning — commit (build it), pivot (keep what was learned, reframe), or park (stop for now, keep everything, reopen any time).What changes, mechanically
source-inventory,open-questions-record). The clarifying questions themselves are unchanged; a stock run gains the intake question, the two additive records, and a confirm-unconfirmed-answers duty at the gate (the CHANGELOG states this plainly).discovery-current-state(1.8, ends in a playback the person confirms),discovery-future-state(1.9, at least three structurally different framings, the person chooses),discovery-experimentation(1.10, riskiest assumptions first, append-only evidence),discovery-decision(1.11, the decision pack and the decision).requirements-analysisanduser-storiesdeclaredecision-packas an optional consume (required: false), andrefined-mockupsdeclaresdesign-language-recordandevidence-recordthe same way — when the artifacts exist, the run-stage directive hands delivery their paths and upstream-coverage verifies they were referenced; when they do not (every stock run), the resolver drops the entries silently. Continue here relays the engine's own verbs (scope-change,set-status,state skipwith a reason naming the decision pack) so the same workflow and record proceed into Inception — pinned by a deterministic test (t214) in both directions. Hand off completes the workflow and the pack travels to the receiving team. Both paths verified live.discovery, zero keywords, explicit selection only), nine framework-default templates (the first tier-2 templates to ship;intent-statementdeliberately has none), a shared test-primitives knowledge file, a new user-guide chapter (Running a Discovery Initiative), glossary entries, and the full consistency surface (SKILL tables, stage graph 32 → 36, docs, tests). Version 2.2.11.Why this shape
One intent stage, not two — the same human moment should not have two implementations, and the shared entry is what makes native continuation into delivery possible. Materials-first because nobody should re-type what their documents already say; every borrowed answer carries its source and is confirmed by a person before it becomes fact. Gates where errors are cheapest. Three structurally different framings minimum, because variations of one idea are a rubber stamp. Riskiest-first testing, because the assumption most likely to kill the initiative should be the first money spent. The pack travels before the meeting. Opt-in with zero keywords, so nobody's routing changes by accident. The six questionnaire stages are untouched — replacement is a later, evidence-gated proposal (graduation criteria in the RFC), not a silent switch.
Non-breaking evidence for the four stock scopes
mr9-parityfixture diffs, the strongest single artifact in this PR). Stock walk fixtures are byte-identical.t213pins the contract with exact pins: intent-capture keeps exactly its four stock scopes plus discovery, produces as an exact ordered list, the four new stages declarescopes: [discovery]only.t214pins the continuation verb sequence against a completed-discovery fixture (including a contrast test thatrecomposerefuses the same state) and the optional-consume differential (pack present: the requirements-analysis directive carries its path; absent: no trace).Pairing with PR 2
This PR and the skills PR that follows are a pair: this one ships the flow's structure, the next ships the craft the agents load to execute each move deeply (framing generation from structurally different starting points, design-language capture, the five test moves as per-move playbooks, narrative generation). The structure runs on its own and is provably non-breaking, but the method's full depth arrives with the pair, so we ask that both be reviewed together and merged together — packaged however this repo prefers: one PR with two clean commits, two PRs with the second marked stacked, or a staging branch on your side.
Change inventory and review guide
316 files; 187 are generated
dist/copies (verify withbun scripts/package.ts --check, never by hand). The branch is five commits — the feature, then four review-response fix rounds — so the fixes from the first review are individually diffable. The authored surface, in review order:core/aidlc-common/stages/ideation/intent-capture.mdtests/integration/t213-discovery-scope.test.ts,t214-discovery-continuation.test.ts+tests/fixtures/mr9-parity/*diffsdiscovery-*.mdstagesjump,park,scope-change,set-status,state skip)core/scopes/aidlc-discovery.md+ ninecore/tools/data/templates/intent-statementdeliberately has no templatecore/knowledge/aidlc-shared/discovery-test-primitives.mddocs/guide/18-discovery-walkthrough.mdMechanical (verify only): the pinned stage numbers 1.8–1.11 in the four committed
stage-graph.jsonfiles (per the pin-and-harvest practice indocs/reference/15-stage-definition.md), one-line scope tags on the three initialization stages, statusline/state-template/SKILL tables/welcome banner, the version trio (t68 guards it), comment-only count corrections in two sensor sources andaidlc-graph.ts. Docs: 29 files, mostly count/table consistency plus the new chapter and six glossary entries. Tests: 20 count-pin moves plus regenerated fixtures.Notes for maintainers
core/, no hand-editing per-harnessstage-graph.json#478 (authoring stage numbers in core) — it would remove this PR's pinned-row edits.mergeComposedScopespreserves them (aidlc-graph.ts:1255-1277, wired at:1505-1521)..aidlc-sensors/);parkpromises/aidlc --resume, which works but is missing from the help text; the state-file legend renders[x]-beside-SKIP confusingly after a scope change. One pre-existing doc error (02-your-first-workflow.mdnamed a nonexistent artifact and five questions where the stage asks four) is fixed here since we touched the page anyway.Verification
bun scripts/package.ts --checkgreen on all four harnesses. Deterministic suite: 240 files, 3,838 assertions, one known environmental failure (t92 — reproduced identically at the untouched v2.2.10 base in this environment). The three sdk-live tests that pin scope counts (t189, t192, t193) are updated for the tenth scope; they skip in this environment (no live credentials), so that fix is pin-correct by arithmetic rather than run-proven here — stated so this verification claim carries its environment scope. Coverage registry regenerated and fresh. Live smoke: discovery birth routes to intent-capture, the template-to-sensor round trip fires and fails correctly, a full run reaches the decision, the commit continuation proceeds into Inception in the same workflow, and the intent-statement write-back's sensor behavior was proven with a live hook run.