README.md

# `mdcmd`
Execute commands in Markdown files, embed output, generate TOCs

e.g.:
<!-- `bmdf seq 3` -->
```bash
seq 3
# 1
# 2
# 3
```

☝️ This block is updated programmatically by [`mdcmd`] (and verified [in CI]; see [raw README.md][raw-mdcmd]).

[![](https://img.shields.io/pypi/v/mdcmd?label=mdcmd&color=blue)][mdcmd]

<a id="toc"></a>
<!-- `toc` -->
- [Overview](#overview)
- [Install](#install)
- [`mdcmd`: execute commands in Markdown files, embed output](#mdcmd)
    - [`bmdf` example](#mdcmd-bmdf-example)
    - [HTML example](#mdcmd-html-example)
- [`bmd`: format `bash` command and output as Markdown](#bmd)
    - [`bmdf` (`bmd -f`): command+output mode](#bmdf)
    - [`bmdff` (`bmd -ff`): two-fence mode](#bmdff)
    - [`bmdfff` (`bmd -fff`): &lt;details&gt; mode](#bmdfff)
    - [Piping](#piping)
    - [Env vars](#env-vars)
    - [`-w/--workdir` / `$BMDF_WORKDIR`](#workdir)
- [`toc`: Markdown Table of Contents](#toc)
- [Examples](#examples)
- [GitHub Action](#github-action)

☝️ This TOC is generated programmatically by [`mdcmd`] and [`toc`] (and verified [in CI]; see [raw README.md][raw-toc]).

## Overview <a id="overview"></a>
This package provides 3 CLIs:
- [`mdcmd`]: execute shell commands in Markdown files, embed output
- [`bmd`]: run Bash commands, wrap output for Markdown embedding
  - Useful in conjunction with `mdcmd`
  - [`bmdf`], [`bmdff`], [`bmdfff`] provide different types of "fencing" for command output
- [`toc`]: generate Markdown table of contents (with custom "id"s for sections)
- `mktoc`: convenience wrapper for `mdcmd -x '^toc$'`

## Install <a id="install"></a>

Global install via [pipx] or [uv] (recommended):
```bash
pipx install mdcmd
# or: uv tool install mdcmd
```

You can also install in the current (v)env:
```bash
pip install mdcmd
```

## `mdcmd`: execute commands in Markdown files, embed output <a id="mdcmd"></a>

<!-- `bmdfff -- mdcmd --help` -->
<details><summary><code>mdcmd --help</code></summary>

```
Usage: mdcmd [OPTIONS] [PATH] [OUT_PATH]

  Parse a Markdown file, updating blocks preceded by <!-- `[cmd...]` -->
  delimiters.

  If no paths are provided, will look for a README.md, and operate "in-place"
  (same as ``mdcmd -i README.md``).

Options:
  -a, --amend                     Squash changes onto the previous Git commit;
                                  suitable for use with `git rebase -x`
  -C, --no-concurrent             Run commands in sequence (by default, they
                                  are run concurrently)
  -i, --inplace / -I, --no-inplace
                                  Edit the file in-place
  -n, --dry-run                   Print the commands that would be run, but
                                  don't execute them
  -T, --no-cwd-tmpdir             In in-place mode, use a system temporary-
                                  directory (instead of the current workdir,
                                  which is the default)
  -x, --execute TEXT              Only execute commands that match these
                                  regular expressions
  -X, --exclude TEXT              Only execute commands that don't match these
                                  regular expressions
  --help                          Show this message and exit.
```
</details>

```bash
# Modify README.md in-place
mdcmd -i README.md
# Same as above; no args defaults to `-i README.md`
mdcmd
```

That's how the various command examples in this file are generated / updated!

### [`bmdf`] example <a id="mdcmd-bmdf-example"></a>

The example at the top of this file is generated by a line like:

  ```
  <!-- `bmdf seq 3` -->
  ```

`mdcmd` transforms that into:
  ````
  <!-- `bmdf seq 3` -->
  ```bash
  seq 3
  # 1
  # 2
  # 3
  ```
  ````

Notes:
- HTML comments (`<!-- ... -->`) are hidden in rendered markdown, so all the user sees is the output of `bmdf seq 3`
  - [`bmdf`] formats output as a "Bash fence" block
  - [`bmd`] (and variants) are useful for displaying commands (and their output) in Markdown (especially in conjunction with [`mdcmd`]).
- **`mdcmd` is idempotent:**
  - It looks for the block immediately following the `` <!-- `[cmd...]` --> `` line, and replaces that with the output of running `[cmd...]`.
  - If there's already output there, it will be replaced with new/current output.

### HTML example <a id="mdcmd-html-example"></a>
Scripts that output raw HTML also work, e.g. [print-table.py] generates this table:

<!-- `python test/print-table.py` -->
<table>
  <tr>
    <th>header 1</th>
    <th>header 2</th>
  </tr>
  <tr>
    <td>cell 1</td>
    <td>cell 2</td>
  </tr>
</table>

That table is generated by a line like:

  ```
  <!-- `python test/print-table.py` -->
  ```

[`mdcmd`] maintains an output block immediately after it:

  ````
  <!-- `python test/print-table.py` -->
  <table>
    <tr>
      <th>header 1</th>
      <th>header 2</th>
    </tr>
    <tr>
      <td>cell 1</td>
      <td>cell 2</td>
    </tr>
  </table>
  ````

## `bmd`: format `bash` command and output as Markdown <a id="bmd"></a>

<!-- `bmdfff -- bmd --help` -->
<details><summary><code>bmd --help</code></summary>

```
Usage: bmd [OPTIONS] COMMAND...

  Format a command and its output to markdown, either in a `bash`-fence or
  <details> block, and copy it to the clipboard.

Options:
  -A, --strip-ansi                Strip ANSI escape sequences from output
  -C, --no-copy                   Disable copying output to clipboard
                                  (normally uses first available executable
                                  from ['pbcopy', 'xclip', 'clip']
  -e, --error-fmt TEXT            If the wrapped command exits non-zero,
                                  append a line of output formatted with this
                                  string. One "%d" placeholder may be used,
                                  for the returncode. Defaults to
                                  $BMDF_ERR_FMT
  -E, --env TEXT                  k=v env vars to set, for the wrapped command
  -f, --fence                     Pass 0-3x to configure output style: 0x:
                                  print output lines, prepended by "# "; 1x:
                                  print a "```bash" fence block including the
                                  <command> and commented output lines; 2x:
                                  print a bash-fenced command followed by
                                  plain-fenced output lines; 3x: print a
                                  <details/> block, with command <summary/>
                                  and collapsed output lines in a plain fence.
  -i, --include-stderr / -I, --no-include-stderr
                                  Capture and interleave both stdout and
                                  stderr streams; falls back to
                                  $BMDF_INCLUDE_STDERR
  -s, --shell / -S, --no-shell    Disable "shell" mode for the command; falls
                                  back to $BMDF_SHELL, but defaults to True if
                                  neither is set
  -t, --fence-type TEXT           When -f/--fence is 2 or 3, this customizes
                                  the fence syntax type that the output is
                                  wrapped in
  -r, --exit-code INTEGER         Expected exit code; bmdf exits 0 if command
                                  exits with this code, non-zero otherwise
                                  (useful for diff commands that exit 1 on
                                  differences)
  -u, --expanduser / -U, --no-expanduser
                                  Pass commands through `os.path.expanduser`
                                  before `subprocess`; falls back to
                                  $BMDF_EXPANDUSER
  -v, --expandvars / -V, --no-expandvars
                                  Pass commands through `os.path.expandvars`
                                  before `subprocess`; falls back to
                                  $BMDF_EXPANDVARS
  -w, --workdir TEXT              `cd` to this directory before executing
                                  (falls back to $BMDF_WORKDIR
  -x, --executable TEXT           `shell_executable` to pass to Popen
                                  pipelines (default: $SHELL)
  --help                          Show this message and exit.
```
</details>

`bmd` (and aliases [`bmdf`], [`bmdff`], [`bmdfff`]) takes a `bash` command as input, and renders the command and/or its output in various Markdown-friendly formats:

### `bmdf` (`bmd -f`): command+output mode <a id="bmdf"></a>

Suppose you want to embed a command and its output in a README.md, like this:

<!-- `bmdf seq 3` -->
```bash
seq 3
# 1
# 2
# 3
```

(Note how the command is `bash`-highlighted, and output lines are rendered as comments)

Put a placeholder like this in your README.md:
  ````
  <!-- `bmdf seq 3` -->
  ````

then [run `mdcmd`][`mdcmd`] to update your README containing this embedded command block.

### `bmdff` (`bmd -ff`): two-fence mode <a id="bmdff"></a>
`bmdff` (alias for `bmd -ff`) renders two code fences, one with the Bash command (syntax-highlighted appropriately), and a second (non-highlighted) block with the output, e.g.:

  ````
  <!-- `bmdff seq 5` -->
  ````

becomes:

<!-- `bmdff seq 5` -->
```bash
seq 5
```
```
1
2
3
4
5
```

### `bmdfff` (`bmd -fff`): &lt;details&gt; mode <a id="bmdfff"></a>

When a command's output is large, rendering it as a `<details><summary>` (with the output collapsed, by default) may be preferable.

`bmdfff` (3 `f`s, alias for `bmd -fff`) transforms placeholders like this:

  ````
  <!-- `bmdfff seq 10` -->
  ````

to:

<!-- `bmdfff seq 10` -->
<details><summary><code>seq 10</code></summary>

```
1
2
3
4
5
6
7
8
9
10
```
</details>

### Piping <a id="piping"></a>
Piping works too, e.g.:

  ````
  <!-- `bmdf -- seq 10 | wc -l` -->
  ````

will become:

<!-- `bmdf -- seq 10 | wc -l` -->
```bash
seq 10 | wc -l
# 10
```

(the `--` is needed so that that `-l` isn't parsed as an opt to `bmdf`)

### Env vars <a id="env-vars"></a>
By default, `shell=True` is passed to `subprocess` calls (but can be disabled via `-S`).

This means env vars are expanded; they can also be set via `-E`, e.g.:

  ````
  <!-- `bmdf -E FOO=bar echo $FOO` -->
  ````

yields:
<!-- `bmdf -E FOO=bar echo $FOO` -->
```bash
FOO=bar echo '$FOO'
# bar
```

<details>
<summary>
More examples of quoting/splitting behavior
</summary>

Quoting `"$FOO"`:
  ````
  <!-- `bmdf -E FOO=bar echo "$FOO"` -->
  ````

yields:
<!-- `bmdf -E FOO=bar echo "$FOO"` -->
```bash
FOO=bar echo '$FOO'
# bar
```

Arg with spaces:
  ````
  <!-- `bmdf -E FOO=bar echo "FOO: $FOO"` -->
  ````
yields:
<!-- `bmdf -E FOO=bar echo "FOO: $FOO"` -->
```bash
FOO=bar echo 'FOO: $FOO'
# FOO: bar
```

Escaping `$`:
  ````
  <!-- `bmdf -E FOO=bar echo "\$FOO=$FOO"` -->
  ````
yields:
<!-- `bmdf -E FOO=bar echo "\$FOO=$FOO"` -->
```bash
FOO=bar echo '\$FOO=$FOO'
# $FOO=bar
```

</details>

### `-w/--workdir` / `$BMDF_WORKDIR` <a id="workdir"></a>

By default, `bmdf` runs in the current working directory. This can be overridden with `-w`:

  ````
  <!-- `bmdf -w .github ls` -->
  ````

<!-- `bmdf -w .github ls` -->
```bash
ls
# workflows
```

## `toc`: Markdown Table of Contents <a id="toc"></a>

<!-- `bmdfff -- toc --help` -->
<details><summary><code>toc --help</code></summary>

```
Usage: toc [OPTIONS] [PATH]

  Generate a table of contents from a markdown file.

  If no PATH is provided, will try to use $MDCMD_FILE (set by mdcmd), or
  default to README.md if that's not set.

Options:
  -n, --indent-size INTEGER  Indent size (spaces)
  --help                     Show this message and exit.
```
</details>

`toc` generates a table of contents from markdown headings, and pairs well with [`mdcmd`] for maintaining TOCs in markdown files.

1. Put a line like this in your README.md:
    ```
    <!-- `toc` -->

    ```
   (the trailing blank line is important, don't put other content immediately under a `<!-- `...` -->` line)

2. Put empty `<a>` tags next to headings to includ them in the TOC (and specify an `id`):

    ```markdown
    ## My section heading <a id="my-section"></a>
    ```

    This allows for custom/short `id`s, as well as skipping sections.

3. Run `mdcmd` as usual:
    ```bash
    # Update all command blocks in README.md, including the TOC
    mdcmd
    ```

    `mdcmd` will see the `<!-- `toc` -->`, and embed the TOC generated by [`toc`][toc.py] under it.

A `mktoc` script is also provided, which just wraps `mdcmd -x '^toc$'` (`mktoc` was implemented separately, in previous versions, before being decomposed into `mdcmd` and `toc` in 0.7.0).

## Examples <a id="examples"></a>
- The examples in this file are all rendered by [`bmdf`] and [`mdcmd`].
- [The TOC](#toc) above is rendered by [`toc`].
- The [`ci.yml`] GitHub Action verifies the examples and TOC.

These repos' READMEs also use [`bmdf`] / [`mdcmd`] / [`toc`] to execute example commands (and in some cases also verify them with a GitHub Action):

- [runsascoded/juq]
- [runsascoded/utz]
- [runsascoded/dvc-utils]
- [ryan-williams/dvc-helpers] ([GHA][dvc-helpers GHA])
- [ryan-williams/git-helpers]
- [ryan-williams/parquet-helpers] ([GHA][pqt-helpers GHA])
- [ryan-williams/tdbs-dask]
- [TileDB-Inc/scverse-ml-workshop-2024]

## GitHub Action <a id="github-action"></a>
A reusable GitHub Action is available on the [`v1` branch][v1-branch]:

```yaml
- uses: runsascoded/mdcmd@v1
```

It runs `mdcmd` (and optionally `mktoc`) and fails if the files are out of date.

### Inputs
| Input | Description | Default |
|-------|-------------|---------|
| `version` | mdcmd version to install | latest |
| `files` | Space-separated files to process | `README.md` |
| `run-mktoc` | Run mktoc for TOC generation | auto-detect* |
| `fail-on-diff` | Fail if changes detected | `true` |

\* Auto-detects if files contain `<!-- \`toc\` -->` marker

### Example
See [use-prms workflow][use-prms-workflow] ([runs][use-prms-runs]) for a real-world example.

<!-- `scripts/gh-url.py '`ci.yml`' .github/workflows/ci.yml` -->
[`ci.yml`]: .github/workflows/ci.yml
[`bmd`]: #bmd
[`bmdf`]: #bmdf
[`bmdff`]: #bmdff
[`bmdfff`]: #bmdfff
[`mdcmd`]: #mdcmd
[`toc`]: #toc

<!-- `scripts/gh-url.py toc.py src/bmdf/toc.py` -->
[toc.py]: src/bmdf/toc.py
<!-- `scripts/gh-url.py print-table.py test/print-table.py` -->
[print-table.py]: test/print-table.py
<!-- `scripts/gh-url.py 'in CI' .github/workflows/ci.yml#L28-L31` -->
[in CI]: .github/workflows/ci.yml#L28-L31
<!-- `scripts/raw-readme-link.py mdcmd` -->
[raw-mdcmd]: README.md?plain=1#L5-L11
<!-- `scripts/raw-readme-link.py toc` -->
[raw-toc]: README.md?plain=1#L18-L33

[runsascoded/utz]: https://github.com/runsascoded/utz?tab=readme-ov-file#utz
[TileDB-Inc/scverse-ml-workshop-2024]: https://github.com/TileDB-Inc/scverse-ml-workshop-2024?tab=readme-ov-file#training-models-on-atlas-scale-single-cell-datasets
[ryan-williams/tdbs-dask]: https://github.com/ryan-williams/tdbs-dask?tab=readme-ov-file#tdbs-dask
[ryan-williams/dvc-helpers]: https://github.com/ryan-williams/dvc-helpers?tab=readme-ov-file#dvc-helpers
[dvc-helpers GHA]: https://github.com/ryan-williams/dvc-helpers/actions
[ryan-williams/git-helpers]: https://github.com/ryan-williams/git-helpers?tab=readme-ov-file#git-helpers
[ryan-williams/parquet-helpers]: https://github.com/ryan-williams/parquet-helpers
[pqt-helpers GHA]: https://github.com/ryan-williams/parquet-helpers/actions/
[runsascoded/dvc-utils]: https://github.com/runsascoded/dvc-utils
[runsascoded/juq]: https://github.com/runsascoded/juq

[mdcmd]: https://pypi.org/project/mdcmd/
[pipx]: https://pipx.pypa.io/stable/
[uv]: https://docs.astral.sh/uv/
[v1-branch]: https://github.com/runsascoded/mdcmd/tree/v1
[use-prms-workflow]: https://github.com/runsascoded/use-prms/blob/main/.github/workflows/readme.yml
[use-prms-runs]: https://github.com/runsascoded/use-prms/actions/workflows/readme.yml