🤖 Store execution outputs in `Outputs` #1903

agoose77 · 2025-03-07T10:59:19Z

This PR closes #1674 by changing our internal representation of cell outputs so that they are children of the Outputs node.

Note

Part of initiative #1026

Notable Changes

Address individual output children with labels ${block.identifier}-outputs-${N}
Add new parent outputs node with identifier ${block.identifier}-outputs

Tasks

Check that .docx exports still export matplotlib figures
Check that outputs can still be embedded
Check that individual output nodes can be embedded
Check that individual outputs are still minified
Check that JATS exports still export matplotlib figures

changeset-bot · 2025-03-07T10:59:23Z

🦋 Changeset detected

Latest commit: fccf681

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages

Name	Type
mystmd	Minor
myst-directives	Patch
myst-transforms	Patch
myst-spec-ext	Patch
myst-execute	Patch
myst-cli	Minor
myst-common	Patch
myst-config	Patch
myst-frontmatter	Patch
myst-parser	Patch
myst-roles	Patch
myst-to-html	Patch
myst-migrate	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

rowanc1

A few comments as I am looking now.

packages/myst-cli/src/transforms/outputs.ts

packages/myst-execute/tests/execute.yml

packages/myst-execute/tests/run.spec.ts

packages/mystmd/tests/notebook-fig-embed/outputs/index.json

rowanc1 · 2025-03-07T15:17:17Z

Thoughts on a testing plan for this:

thebe can attach to the right place, there are likely going to be changes in the theme due to this
contents are still minified and downloaded correctly (e.g. images are a network request, etc.)
docx, typst, latex, JATS still export matplotlib images and latex
mystmd docs work for the embed and interactive components
bokeh example, holoviews, plotly, sympy
placeholders work

packages/mystmd/tests/notebook-fig-embed/outputs/index.json

packages/myst-transforms/src/blocks.spec.ts

rowanc1 · 2025-03-07T15:31:28Z

packages/myst-transforms/src/blocks.ts

+      const outputsNode = select('outputs', block) as GenericNode | undefined;
+      if (outputsNode !== undefined && !outputsNode.identifier) {
+        // Label outputs node
+        outputsNode.identifier = `${block.identifier}-output`;


I think that these ID changes may need to be taken into account in the upgrade/downgrade script for thebe to continue working.

Let's keep an eye on that

agoose77 · 2025-04-01T10:18:40Z

packages/myst-cli/src/transforms/code.spec.ts

@@ -270,12 +273,14 @@ describe('propagateBlockDataToCode', () => {
              result = mdast.children[0].children[0].visibility;
              break;
            case 'output':
-              if (!has_output && target == 'output') {


target is known to be output here.

stevejpurves · 2025-06-25T09:46:09Z

packages/myst-directives/src/code.ts

-    const output = {
-      type: 'output',
+    const outputs = {
+      type: 'outputs',
      id: nanoid(),


@agoose77 adding this immediately enables compute in figures & embeds in many of the test cases. If there is a place in myst-execute where this should be added I suggest we do. If you could please do that, or if it's trivial, point me to the line to change.

Hmm, not quite following - are you saying that it's broken currently?

We need this change here for in-browser compute.

On this PR jupyter-book/myst-theme#571 you mentioned that myst-execute does not add the id, so I'm just flagging that we need one. Adding the id here seems to be enough, but if there is a further change needed in myst-execute, in line with your comment that should also be changed.

I'm in a conference atm. I need to check the status quo — that comment is about output.id not outputs.id. I think the intention is that the id remainsper-outputs, somyst-executeshould still work (it preserves the fields of theoutputsnode, and theid` is pre-populated).

IIRC myst-theme in that PR now accesses the outputs.id via context, so that the output renderer has access to it.

stevejpurves · 2025-06-26T10:36:16Z

Great - @agoose77 I think this is good to go pending test fixes, and is working against the theme PR jupyter-book/myst-theme#571

but I also note the testing points from @rowanc1 above:

docx, typst, latex, JATS still export matplotlib images and latex
mystmd docs work for the embed and interactive components

The second will work after th content there is updated to the new form as the PR above confirms external emdbedding and referencing works as before.

We don't need to delete nodes without children: they will simply vanish when lifted

stevejpurves

@agoose77 verified the hard copy output, which was the last ask on this PR. I think this is good to go.

rowanc1 · 2025-07-07T22:15:33Z

Before merging, we should ensure that the date in the v3 transformation is fixed to the release date (or closer to it). And that we bump the content version of the spec to v3.

agoose77 force-pushed the agoose77/feat-future-ast-outputs branch 3 times, most recently from cec055e to 6870f57 Compare March 7, 2025 11:05

rowanc1 reviewed Mar 7, 2025

View reviewed changes

agoose77 commented Mar 7, 2025

View reviewed changes

packages/mystmd/tests/notebook-fig-embed/outputs/index.json Outdated Show resolved Hide resolved

agoose77 commented Mar 7, 2025

View reviewed changes

packages/mystmd/tests/notebook-fig-embed/outputs/index.json Outdated Show resolved Hide resolved

agoose77 commented Mar 7, 2025

View reviewed changes

packages/mystmd/tests/notebook-fig-embed/outputs/index.json Outdated Show resolved Hide resolved

rowanc1 reviewed Mar 7, 2025

View reviewed changes

packages/myst-transforms/src/blocks.spec.ts Outdated Show resolved Hide resolved

rowanc1 reviewed Mar 7, 2025

View reviewed changes

agoose77 mentioned this pull request Mar 21, 2025

Support new AST format in myst-theme jupyter-book/myst-theme#569

Open

agoose77 mentioned this pull request Apr 1, 2025

Improve capabilities of build-time execution #1026

Open

agoose77 force-pushed the agoose77/feat-future-ast-outputs branch 2 times, most recently from d8a8684 to 6f0d05b Compare April 1, 2025 10:05

agoose77 commented Apr 1, 2025

View reviewed changes

agoose77 force-pushed the agoose77/feat-future-ast-outputs branch from 2060521 to 5857844 Compare April 1, 2025 10:20

agoose77 mentioned this pull request Apr 1, 2025

[myst] Implement support for structured code-cell outputs in MyST AST 2i2c-org/infrastructure#5836

Closed

3 tasks

agoose77 marked this pull request as ready for review April 2, 2025 10:36

agoose77 changed the title ~~🤖 Store execution outputs in Outputs~~ [[⚠️ Do Not Merge ⚠️]] 🤖 Store execution outputs in Outputs Apr 2, 2025

agoose77 force-pushed the agoose77/feat-future-ast-outputs branch from 826b957 to 455e0d8 Compare April 2, 2025 12:49

agoose77 mentioned this pull request Apr 3, 2025

☢️ DO NOT MERGE ☢️ Prepare for outputs AST change jupyter-book/myst-theme#571

Open

16 tasks

agoose77 changed the title ~~[[⚠️ Do Not Merge ⚠️]] 🤖 Store execution outputs in Outputs~~ ⚠️ DO NOT MERGE ⚠️ 🤖 Store execution outputs in Outputs Apr 3, 2025

agoose77 mentioned this pull request Apr 10, 2025

✍️ Parse Markdown and LaTeX outputs into AST #1961

Open

agoose77 changed the title ~~⚠️ DO NOT MERGE ⚠️ 🤖 Store execution outputs in Outputs~~ ⚠️ DO NOT MERGE ⚠️ — 🤖 Store execution outputs in Outputs Apr 10, 2025

agoose77 added the blocked Waiting for another task to be compled label May 23, 2025

stevejpurves reviewed Jun 25, 2025

View reviewed changes

stevejpurves added enhancement New feature or request and removed blocked Waiting for another task to be compled labels Jun 26, 2025

agoose77 and others added 26 commits July 2, 2025 13:00

chore: add changeset

128aa43

fix: drop style change

c87a7a4

fix: only label first code and outputs

bf9fa61

fix: handle IDs

3eba5d3

fix: return type

ad8ed3e

fix: selectAll → select

195182a

refactor: move user expression lowering into function

c715602

refactor: nest declaration

a5553a3

fix: rename outputs

32c5915

refactor: simplify conditions

6458808

We don't need to delete nodes without children: they will simply vanish when lifted

test: fix tests

8c34d46

fix: outputs for JATS

d94742d

fix: use outputs- for all prefixes

41753bf

fix: outputs for JATS

42687fb

refactor: simplify test for code cells

8400eb8

fix: early exit non-matplotlib

d3d7a40

test: partially fix JATS test

b86544f

fix: final JATS test

6b169e2

test: fix myst-execute tests

05f79b9

test: fix test titles

f8671d1

test: fix missing children

dc23b0c

test: more fixes

92bad01

test: fix test titles

4d074ac

fix: handle visibility properly

7933f3f

fix: update test cases

7afbe46

🪪 adding id to outputs enables in-browser compute for figures and embeds

fccf681

agoose77 force-pushed the agoose77/feat-future-ast-outputs branch from 76c5b6b to fccf681 Compare July 2, 2025 12:01

agoose77 changed the title ~~⚠️ DO NOT MERGE ⚠️ — 🤖 Store execution outputs in Outputs~~ 🤖 Store execution outputs in Outputs Jul 2, 2025

stevejpurves approved these changes Jul 2, 2025

View reviewed changes

🤖 Store execution outputs in Outputs #1903

Are you sure you want to change the base?

🤖 Store execution outputs in Outputs #1903

Uh oh!

Conversation

agoose77 commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notable Changes

Tasks

Uh oh!

changeset-bot bot commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

rowanc1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rowanc1 commented Mar 7, 2025 • edited by stevejpurves Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rowanc1 Mar 7, 2025

Choose a reason for hiding this comment

Uh oh!

agoose77 Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

agoose77 Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

stevejpurves Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

agoose77 Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

stevejpurves Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agoose77 Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

agoose77 Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

stevejpurves commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stevejpurves left a comment

Choose a reason for hiding this comment

Uh oh!

rowanc1 commented Jul 7, 2025

Uh oh!

Uh oh!

🤖 Store execution outputs in `Outputs` #1903

🤖 Store execution outputs in `Outputs` #1903

agoose77 commented Mar 7, 2025 •

edited

Loading

changeset-bot bot commented Mar 7, 2025 •

edited

Loading

rowanc1 commented Mar 7, 2025 •

edited by stevejpurves

Loading

stevejpurves Jun 25, 2025 •

edited

Loading

stevejpurves commented Jun 26, 2025 •

edited

Loading