Merge pull request #1 from your5e/reformat-51

Reformat the D&D 5.1 RD

Author: norm

.github/workflows/obsidian-vault.yml

@@ -0,0 +1,52 @@
+name: Zip Obsidian vaults
+
+on:
+  push:
+    tags:
+      - 'v*'
+  workflow_dispatch:
+
+jobs:
+  build-vault:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v5
+
+    - name: Install tabulate
+      run: |
+        pip install tabulate
+
+    - name: Create zips
+      run: |
+        make assets
+
+    - name: Upload 51 zip
+      uses: actions/upload-artifact@v4
+      with:
+        name: dnd_51_srd
+        path: /tmp/dnd_51_srd.zip
+        
+    - name: Upload 521 zip
+      uses: actions/upload-artifact@v4
+      with:
+        name: dnd_521_srd
+        path: /tmp/dnd_521_srd.zip
+
+    - name: Extract changelog
+      if: github.ref_type == 'tag'
+      run: |
+        tag="${GITHUB_REF_NAME}"
+        awk "/^## \[$tag\]/{flag=1;next}/^## \[/{flag=0}flag" CHANGELOG.md > /tmp/release_notes.md
+
+    - name: Create Release
+      uses: softprops/action-gh-release@v2
+      if: github.ref_type == 'tag'
+      with:
+        body_path: /tmp/release_notes.md
+        files: |
+          /tmp/dnd_51_srd.zip
+          /tmp/dnd_521_srd.zip

.github/workflows/obsidian-vaults.yml

@@ -0,0 +1,21 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+- nothing
+
+## [v0.3] — 2025-10-29
+
+- Renamed files in the 5.1 vault so linking is easier
+- Included "About the vault" and index pages (that include the legal info),
+  opened by default
+
+## [v0.2.3] - 2025-10-21
+
+- First-pass release of the D&D 5.2.1 SRD Obsidian vault
+
+## [v0.1] — 2025-08-07
+
+- First-pass release of the D&D 5.1 SRD Obsidian vault

CHANGELOG.md

@@ -1,4 +1,6 @@
-.PHONY: flake8 test ci vaults vault51 vault521
+MAKEFLAGS += --no-print-directory
+
+.PHONY: flake8 test ci vaults vault-dnd51 vault-dnd51-zip vault-dnd521 vault-dnd521-zip assets push pull
 
 flake8:
 	flake8 *.py lib/*py
@@ -11,14 +13,34 @@ ci: test
 	@bats tests/srd/srd.bats
 	@bats tests/ci/ci.bats
 
-vaults: vault51 vault521
+vaults: vault-dnd51 vault-dnd521
+
+assets: vault-dnd51-zip vault-dnd521-zip
+
+vault-dnd51:
+	@./patches.sh dnd/51/markdown apply
+	@python update_vault.py dnd/51/markdown dnd/51/obsidian_vault
+	@./update_vault_indexes.sh dnd/51
+	@./patches.sh dnd/51/obsidian_vault apply
 
-vault51:
+vault-dnd51-zip: vault-dnd51
 	rsync -av dnd/51/obsidian_vault/ /tmp/dnd_51_srd/
 	(cd /tmp && zip -r dnd_51_srd.zip dnd_51_srd/)
 
-vault521:
-	python update_vault.py --profile dnd521 dnd/521/markdown dnd/521/obsidian_vault
-	./update_vault_indexes.sh
+vault-dnd521:
+	@./patches.sh dnd/521/markdown apply
+	@python update_vault.py dnd/521/markdown dnd/521/obsidian_vault
+	@./update_vault_indexes.sh dnd/521
+	@./patches.sh dnd/521/obsidian_vault apply
+
+vault-dnd521-zip: vault-dnd521
 	rsync -av dnd/521/obsidian_vault/ /tmp/dnd_521_srd/
 	(cd /tmp && zip -r dnd_521_srd.zip dnd_521_srd/)
+
+push:
+	rsync -ai --delete dnd/51/obsidian_vault/ ~/Downloads/dnd_51_srd/
+	rsync -ai --delete dnd/521/obsidian_vault/ ~/Downloads/dnd_521_srd/
+
+pull:
+	rsync -ai --delete ~/Downloads/dnd_51_srd/ dnd/51/obsidian_vault/
+	rsync -ai --delete ~/Downloads/dnd_521_srd/ dnd/521/obsidian_vault/

Makefile

@@ -51,6 +51,9 @@ Each is available as a whole Markdown file, smaller segments broken up in a
     - Re-arranged the order of the Fantasy-Historical Pantheons so that the
       table of deities appears in the description of that pantheon
     - Subclass introduction headers added to Barbarian, Bard, Cleric & Druid
+    - The monster ability tables have been reformatted to be in a different
+      layout -- score/modifier/saving throw -- and the "Saving Throw" line
+      underneath removed now it is incorporated into the table
 
 - D&D 5th edition 5.2.1 SRD (2024 rules)
 
@@ -85,6 +88,8 @@ Each is available as a whole Markdown file, smaller segments broken up in a
           including rotating statblock tables to have the abilities in the
           first column
         - split tables using mid-table headers into multiple tables
+        - monster ability tables have been reformatted to be in the format
+          score/modifier/saving throw
 
 ## Workflow for breaking the whole SRD into sections
 
@@ -107,117 +112,110 @@ A draft breakdown is made with
 ```
 
 and then refined by hand (as either `marker` detects far too many lines as
-headers, or the PDF was styled poorly). While refining the breakdown, this is
-run repeatedly to spot mistakes:
+headers, or the PDF was styled poorly). While refining the breakdown, this
+can be run repeatedly to spot mistakes:
 
 ```bash
 cp dnd/51/SRD_CC_v5.1.md dnd/51/breakdown.md \
-    && ./breakdown.sh -f dnd/51/breakdown.md \
+    && ./breakdown.sh -fh dnd/51/breakdown.md \
     && diff -u dnd/51/SRD_CC_v5.1.md <(./rebuild.sh dnd/51/breakdown.md)
 ```
 
+
 ### Clean the Markdown
 
-Run the cleaning script:
+Once the breakdown has been created, start looping over refinement and repair
+of the SRD Markdown using `./reprocess.sh`:
 
 ```bash
-python clean_srd.py --progress dnd/51/SRD_CC_v5.1.md
+./reprocess.sh dnd/51/SRD_CC_v5.1.md
 ```
 
-If it detects any errors it cannot fix automatically, it will issue errors and
-not process the file further. After reformatting the document, it will also
-scan for problems that might need human intervention. Any automatic changes
-will have updated the breakdown file as necessary.
+Without other arguments, it will:
 
-If warnings are emitted during cleaning that can be safely ignored, they
-can be added to an ignore file. Similarly, if any source would be modified
-in an unwanted manner, those lines can be added to a file declaring them
-already cleaned:
+- `git restore` the files being cleaned, to always be working with a
+  predictable starting base (this can be skipped with `-b`)
 
-```bash
-python clean_srd.py \
-    --progress \
-    --ignore-warnings dnd/51/ignore_warnings.txt \
-    --clean-lines dnd/51/clean_lines.txt \
-        dnd/51/SRD_CC_v5.1.md
-```
+- run the clean script, which reformats the document to remove PDF decoding
+  errors, change formatting to the preferred format, and scan for errors
+  that are hard to fix programmatically (this can be skipped with `-c`)
+
+    ```bash
+    python clean_srd.py \
+        --progress \
+        $ignore_warnings \
+            "$srd_path" "$breakdown_txt"
+    ```
+- break down the content to individual files (this can be skipped with `-b`)
+
+- rebuild the SRD from the broken down content to compare with the original,
+  to spot breakdown.txt problems, header level issues and more (this can
+  be skipped with `-d`)
+
+- filter the content again to create a cross-linked Obsidian vault (which is
+  also often the easiest way to examine individual sections in isolation to
+  find problems; this can be skipped with `-v`)
 
-When changing the source by hand and lines are added/removed, use
+If warnings are emitted during cleaning that can be safely ignored, they can
+be added to an `ignore_warnings.txt` file. Similarly, if any source would be
+modified in an unwanted manner, those lines can be added to a
+`clean_lines.txt` file.
+
+When changing the original SRD Markdown by hand and lines are added/removed, use
 `alter_lines.sh` to change the line numbers in `breakdown.txt` and
 `ignore_warnings.txt` from a specific point in the breakdown onwards:
 
 ```bash
 ./alter_lines.sh -d dnd/51/ /black_tentacles -2
 ```
 
-### Fix header progression
+### Fixing header progression
 
 The broken down fragments of the SRD should start with a first level header.
-To help this, `breakdown.sh` will fix easy segments, and warn on any where the
-headers skip around too much. This can be suppressed with the `-f` argument.
+To help this, `breakdown.sh` fixes easy segments, and warns on any where the
+headers skip around too much. Fixing can be turned off using the `-h` option
+to `reprocess.sh`, or `-f` to `breakdown.sh` if using it directly. Likewise,
+warnings can be supressed with `-h` to `reprocess.sh`, or `-c` when using
+`breakdown.sh` directly.
 
-To look for general problems in header progression, use:
+Any warnings can be piped to `edit_warnings.sh` to open the file at the right
+line (in Sublime Text):
 
 ```bash
-./check_headers.sh dnd/51/markdown
-```
-
-If there are warnings from `check_headers.sh` or `breakdown.sh` where they
-can't fix things automatically, they can be piped to `edit_warnings.sh` to
-open the file at the right line (in Sublime Text):
+./breakdown.sh | ./edit_warnings.sh
 
-```bash
-./check_headers.sh dnd/51/markdown | ./edit_warnings.sh
+# or using the clipboard...
+pbpaste | ./edit_warnings.sh
 ```
 
-### Confirming the source
+### Patching content
 
-To check the SRD master file against the broken down fragments, use:
+It can be helpful to add/alter content from the SRD to improve the state of
+the vault content, by adding links etc. From a clean git state, edit the files
+then run
 
 ```bash
-diff -u dnd/51/SRD_CC_v5.1.md <(./rebuild.sh dnd/51/breakdown.md)
-```
-
-### Create an Obsidian vault
+# to patch the breakdown files
+./patches.sh dnd/51/markdown create
 
-Once the SRD is edited, create a copy to use as an Obsidian vault.
-
-```bash
-python update_vault.py \
-    --progress \
-        dnd/51/markdown \
-        dnd/51/obsidian_vault
+# to patch only the vault
+./patches.sh dnd/51/obsidian_vault create
 ```
 
-Some words that are also conditions (eg invisible) may be incorrectly linked.
-The lines that they are on can be added to `ignore_vault.txt` so that they
-won't be re-created when re-running the script:
-
-```bash
-python update_vault.py \
-    --progress \
-    --ignore dnd/51/ignore_vault.txt \
-        dnd/51/markdown \
-        dnd/51/obsidian_vault
-```
+In general, patch the original Markdown files if you want to add more content
+so the vault filters apply, and patch the vault if the filters are doing the
+wrong thing (such as linking to the condition Invisible when referring to the
+Invisible Stalker).
 
-And where files need to be altered after the script has run in a way that
-would be overwritten (eg creating more wikilinks that would be removed again),
-after editing the files, create patches:
+Patches are kept rather than changing the content so that automated tests
+still pass when comparing the Markdown fragments to the original document.
 
-```bash
-./vault_patches.sh create dnd/51
-```
-
-that can then be restored later:
-
-```bash
-./vault_patches.sh apply dnd/51
-```
 
 ## Changing the code
 
-Every script should have a test suite.
+Every script should have a test suite. Every filter in the `clean_srd.py`
+and `update_vault.py` scripts should be individally tested, as well as having
+whole-document integration tests.
 
 - `make test` runs all the code tests
 - `make ci` runs all the code tests, and a couple more