Docs + Schema Fixes #129

aditigopalan · 2026-01-15T19:35:05Z

Documentation Improvements and Schema Fixes

Overview

This PR migrates documentation from MkDocs to Sphinx, adds CSV export functionality, fixes schema inheritance issues, and cleans up unused files.

Major Changes

📚 Documentation Migration (MkDocs → Sphinx)

Migrated documentation system from MkDocs to Sphinx with Read the Docs integration
Added Sphinx configuration (conf.py, .readthedocs.yaml)
Created custom CSS for improved content width
Restructured documentation to be self-contained per module (all inherited attributes shown on each page)
Added level-specific pages for multi-level modules (WES, scRNA-seq, MultiplexMicroscopy, SpatialOmics)

📊 CSV Export Functionality

Added CSV export for all modules and levels
CSV files include all attributes (including inherited ones) with Type, Pattern, Required, and Description
Fixed CSV generation to include all nested class attributes for Clinical module
CSV files available at docs/csv/ for download

🔧 Schema Fixes

Removed FILENAME pattern constraint from CoreFileAttributes in modules/CoreFile/domains/core.yaml
Fixed FileFormatLevel4 enum inheritance: Preserved child class attributes over parent class attributes in inheritance resolution (fixes missing enum in SpatialOmics Level 4 documentation)

🧹 Code Cleanup

Removed unused documentation files:
- docs/identifiers.md
- docs/corefile.md
- docs/imaging.md
- docs/sequencing.md
Removed cruft template tracking (.cruft.json and related Makefile targets)
Cleaned up obsolete MkDocs configuration

🐛 Bug Fixes

Fixed enum links to correctly match Sphinx auto-generated anchors
Fixed inheritance resolution to preserve child attributes (e.g., FILE_FORMAT with enum type) over parent attributes (e.g., FILE_FORMAT with string type)
Fixed CSV generation to include all attributes from all classes for Clinical module

Technical Details

Documentation Generation Scripts

scripts/generate_table_docs.py: Generates markdown documentation with inheritance resolution
scripts/generate_csv_docs.py: Generates CSV exports for all modules and levels
Both scripts properly handle inheritance chains and preserve child class attributes

Documentation Structure

Each module page is self-contained with all required attributes
Level pages (e.g., docs/wes/level-1.md) show all inherited attributes organized by source (Core File, Base Sequencing/Imaging, Module-Specific)
Enum sections are automatically generated for all used enums

Files Changed

Documentation infrastructure: conf.py, .readthedocs.yaml, index.rst, _static/custom.css
Schema: modules/CoreFile/domains/core.yaml
Scripts: scripts/generate_table_docs.py, scripts/generate_csv_docs.py
Generated docs: All docs/*.md files and docs/csv/*.csv files
Build: Makefile (removed cruft targets)

Testing

Documentation builds successfully with Sphinx
CSV files generated correctly for all modules and levels
Enum links work correctly in generated documentation
Inheritance resolution preserves child attributes correctly

Related Issues

Fixes enum inheritance issues in SpatialOmics Level 4
Addresses documentation width constraints
Adds CSV export functionality as requested

Fixes #109

- Add .readthedocs.yaml configuration - Update pyproject.toml to include mkdocs-mermaid2-plugin in docs extra - Remove old GitHub Pages deployment workflow (deploy-docs.yaml) - Remove gh-deploy target from Makefile

… is installed

- Create modern, organized documentation structure - Add comprehensive module documentation pages - Update mkdocs.yml with organized navigation - Add getting started and contributing guides - Organize modules by category (Record-Based vs File-Based) - Add reference section for API documentation - Modernize index page with clear overview The old auto-generated attribute files remain in docs/ but are not included in navigation. They can be archived or removed in a future PR.

- Remove all 233+ outdated auto-generated attribute files - Generate fresh documentation from schemas using LinkML gen-doc - Integrate module READMEs into documentation structure - Update mkdocs.yml to use generated schema docs - Create script to generate docs from all modules - Fix navigation and links throughout This creates documentation similar to LinkML's own docs: https://linkml.io/linkml/intro/overview.html All schema documentation is now auto-generated and up-to-date.

- Remove complex navigation structure - Direct links to generated schema docs with all attributes - Clean up unnecessary files - Fix broken links

- Generate simple markdown with just attributes, classes, and enums - No complex generated structure - Clean, readable format - Simple script to regenerate

…um representation

…ection, show patterns

…uirement clarity

…red for all attributes

…tion - Add comprehensive tests for FILE_FORMAT and FILENAME pattern matching - Update CSV documentation with new patterns - Tests verify patterns match correctly across all modules: * WES (levels 1, 2, 3) * scRNA-seq (levels 1, 2, 3/4) * MultiplexMicroscopy (levels 2, 3, 4) * DigitalPathology * SpatialOmics (levels 1, 4)

- Use class_induced_slots() to properly resolve inherited slot requirements - Fixes AssertionError where FILENAME required flag wasn't being detected - All SpatialOmics tests now passing

Auto-generated Python classes from LinkML schema updates. **Auto-generated by GitHub Actions workflow** [skip ci]

github-actions · 2026-01-15T20:08:46Z

✅ Python classes auto-updated!

The Python classes have been automatically regenerated and committed to this PR branch.

**Updated files:**
```
modules/Biospecimen/src/htan_biospecimen/datamodel/biospecimen.py

modules/Clinical/src/htan_clinical/datamodel/clinical.py
modules/DigitalPathology/src/htan_digitalpathology/datamodel/digital_pathology.py
modules/Imaging/src/htan_imaging/datamodel/imaging.py
modules/MultiplexMicroscopy/src/htan_multiplexmicroscopy/datamodel/multiplex_microscopy.py
modules/Sequencing/src/htan_sequencing/datamodel/sequencing.py
modules/SpatialOmics/src/htan_spatial/datamodel/spatial.py
modules/WES/src/htan_wes/datamodel/wes.py
modules/scRNA-seq/src/htan_scrna_seq/datamodel/scrna_seq.py
```

The generated classes are now up to date with the schema changes.

Auto-generated Python classes from LinkML schema updates. **Auto-generated by GitHub Actions workflow** [skip ci]

github-actions · 2026-01-15T20:28:24Z

✅ Python classes auto-updated!

The Python classes have been automatically regenerated and committed to this PR branch.

**Updated files:**
```
modules/Biospecimen/src/htan_biospecimen/datamodel/biospecimen.py

modules/Clinical/src/htan_clinical/datamodel/clinical.py
modules/DigitalPathology/src/htan_digitalpathology/datamodel/digital_pathology.py
modules/Imaging/src/htan_imaging/datamodel/imaging.py
modules/MultiplexMicroscopy/src/htan_multiplexmicroscopy/datamodel/multiplex_microscopy.py
modules/Sequencing/src/htan_sequencing/datamodel/sequencing.py
modules/SpatialOmics/src/htan_spatial/datamodel/spatial.py
modules/WES/src/htan_wes/datamodel/wes.py
modules/scRNA-seq/src/htan_scrna_seq/datamodel/scrna_seq.py
```

The generated classes are now up to date with the schema changes.

Copilot

Pull request overview

This PR migrates documentation from MkDocs to Sphinx, adds CSV export functionality, fixes schema inheritance issues, and removes unused files. However, the changes include generated MkDocs HTML build artifacts in the site/ directory that should not be committed to the repository.

Changes:

Documentation migration to Sphinx with Read the Docs configuration
CSV export functionality for all modules
Schema fixes for CoreFile and FileFormat enum inheritance

Reviewed changes

Copilot reviewed 73 out of 289 changed files in this pull request and generated 1 comment.

File	Description
site/*.html	Generated MkDocs HTML files (build artifacts)
site/sitemap.xml	Generated sitemap file
site/assets/*	Generated CSS, JavaScript, and other static assets

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-15T20:28:38Z

site/wes/index.html

+<!doctype html>
+<html lang="en" class="no-js">


The entire site/ directory contains generated MkDocs build artifacts that should not be committed to version control. These files are build outputs and should be excluded via .gitignore. According to the PR description, the migration is to Sphinx (not MkDocs), yet these are MkDocs-generated files. These should be removed from the PR, and the build output directory should be added to .gitignore to prevent future commits of generated files.

Auto-generated Python classes from LinkML schema updates. **Auto-generated by GitHub Actions workflow** [skip ci]

github-actions · 2026-01-15T20:37:07Z

✅ Python classes auto-updated!

The Python classes have been automatically regenerated and committed to this PR branch.

**Updated files:**
```
modules/Biospecimen/src/htan_biospecimen/datamodel/biospecimen.py

modules/Clinical/src/htan_clinical/datamodel/clinical.py
modules/DigitalPathology/src/htan_digitalpathology/datamodel/digital_pathology.py
modules/Imaging/src/htan_imaging/datamodel/imaging.py
modules/MultiplexMicroscopy/src/htan_multiplexmicroscopy/datamodel/multiplex_microscopy.py
modules/Sequencing/src/htan_sequencing/datamodel/sequencing.py
modules/SpatialOmics/src/htan_spatial/datamodel/spatial.py
modules/WES/src/htan_wes/datamodel/wes.py
modules/scRNA-seq/src/htan_scrna_seq/datamodel/scrna_seq.py
```

The generated classes are now up to date with the schema changes.

Auto-generated Python classes from LinkML schema updates. **Auto-generated by GitHub Actions workflow** [skip ci]

github-actions · 2026-01-15T20:40:12Z

✅ Python classes auto-updated!

The Python classes have been automatically regenerated and committed to this PR branch.

**Updated files:**
```
modules/Biospecimen/src/htan_biospecimen/datamodel/biospecimen.py

modules/Clinical/src/htan_clinical/datamodel/clinical.py
modules/DigitalPathology/src/htan_digitalpathology/datamodel/digital_pathology.py
modules/Imaging/src/htan_imaging/datamodel/imaging.py
modules/MultiplexMicroscopy/src/htan_multiplexmicroscopy/datamodel/multiplex_microscopy.py
modules/Sequencing/src/htan_sequencing/datamodel/sequencing.py
modules/SpatialOmics/src/htan_spatial/datamodel/spatial.py
modules/WES/src/htan_wes/datamodel/wes.py
modules/scRNA-seq/src/htan_scrna_seq/datamodel/scrna_seq.py
```

The generated classes are now up to date with the schema changes.

Auto-generated Python classes from LinkML schema updates. **Auto-generated by GitHub Actions workflow** [skip ci]

github-actions · 2026-01-15T20:47:56Z

✅ Python classes auto-updated!

The Python classes have been automatically regenerated and committed to this PR branch.

**Updated files:**
```
modules/Biospecimen/src/htan_biospecimen/datamodel/biospecimen.py

modules/Clinical/src/htan_clinical/datamodel/clinical.py
modules/DigitalPathology/src/htan_digitalpathology/datamodel/digital_pathology.py
modules/Imaging/src/htan_imaging/datamodel/imaging.py
modules/MultiplexMicroscopy/src/htan_multiplexmicroscopy/datamodel/multiplex_microscopy.py
modules/Sequencing/src/htan_sequencing/datamodel/sequencing.py
modules/SpatialOmics/src/htan_spatial/datamodel/spatial.py
modules/WES/src/htan_wes/datamodel/wes.py
modules/scRNA-seq/src/htan_scrna_seq/datamodel/scrna_seq.py
```

The generated classes are now up to date with the schema changes.

adamjtaylor

I think we can merge to get deployed but this setup seems quite cumbersome. Lets dedicate time after release 8 to see how we can refine it.

adamjtaylor · 2026-01-16T14:37:52Z

scripts/generate_csv_docs.py

+    "SpatialOmics": "modules/SpatialOmics/domains/spatial.yaml",
+}
+
+def resolve_inheritance_chain(class_def, all_classes, base_dir, visited=None):


This function is duplicated accross generate_csv_docs and generate_table docs. Same goes for get_conditional_requirements, MODULES and parent schema loading logic. Could you pull into a common module that is imported?

adamjtaylor · 2026-01-16T14:38:37Z

scripts/generate_csv_docs.py

+from linkml_runtime.linkml_model.meta import SchemaDefinition
+from linkml_runtime.loaders import yaml_loader
+
+MODULES = {


These are hardcoded and add another step reuuired to change when new modules are added. Its unclear where each of these steps are tracked. Can we automate from the base LinkML schema?

adamjtaylor · 2026-01-16T14:40:03Z

scripts/merge_readme_docs.py

+def merge_readme_with_doc(doc_path: Path, readme_path: Path):
+    """Skip README merging - just use the doc as-is (user wants just tables)."""
+    # Don't merge READMEs anymore - user wants just the tables
+    return True


This does nothing it just returns true. Can we remove?

aditigopalan added 30 commits January 6, 2026 14:36

feat: migrate documentation to Read the Docs

fde4a53

- Add .readthedocs.yaml configuration - Update pyproject.toml to include mkdocs-mermaid2-plugin in docs extra - Remove old GitHub Pages deployment workflow (deploy-docs.yaml) - Remove gh-deploy target from Makefile

fix: add requirements.txt for Read the Docs to ensure mkdocs-material…

a170edd

… is installed

docs: simplify to just show schema attribute lists

009ea43

- Remove complex navigation structure - Direct links to generated schema docs with all attributes - Clean up unnecessary files - Fix broken links

docs: add CSS to fix huge icons and tiny text in generated docs

e9e3724

docs: completely rebuild from scratch - simple clean structure

82a7f94

docs: use simple attribute list format - no LinkML gen-doc

cef0108

- Generate simple markdown with just attributes, classes, and enums - No complex generated structure - Clean, readable format - Simple script to regenerate

docs: add index.md for simple docs structure

36d6318

Switch from MkDocs to Sphinx for Read the Docs

b7b9c2f

Add README content to module docs and improve index structure

bca8593

Add generated module documentation with README content

4139070

Add table format for attributes with inline enums for short lists

3b2c68d

Load enums from imported files and show inline for short enums

772e30a

Fix enum detection to properly show inline values or links

f2a99b8

Properly merge enums from imported files

a5a038f

Fix enum iteration error and show enum links/values in tables

9293e69

Restructure docs: Clinical by class, WES by level, tables only

5ef44e8

Clinical: Rename to Manifests, add conditional requirements, clean en…

1b8cbe7

…um representation

Clinical: Use 'Required IF' for conditionals, add Common Attributes s…

82fb28f

…ection, show patterns

Fix pattern extraction and improve conditional requirement clarity

26fd934

Fix pattern extraction for all attributes and improve conditional req…

ed79740

…uirement clarity

Improve conditional requirement clarity and ensure patterns are captu…

f36cf40

…red for all attributes

Further improve conditional requirement clarity

8488069

Remove Pattern column, show patterns in Type column instead

c866e05

Escape pipe characters in patterns to prevent markdown table breakage

9f1caad

Add home page with architecture overview from README

a828ef1

Add image path to Sphinx configuration

4146a60

Escape backticks in patterns to fix markdown table rendering

752a623

Use HTML code tags instead of backticks for patterns in markdown tables

8d72ce9

aditigopalan and others added 4 commits January 15, 2026 14:57

Fix test_level3_class to properly check inherited FILENAME required flag

e097bf6

- Use class_induced_slots() to properly resolve inherited slot requirements - Fixes AssertionError where FILENAME required flag wasn't being detected - All SpatialOmics tests now passing

Merge e097bf6 into 82823eb

2f308a1

chore: auto-generate Python classes from schema changes

b7b8bfb

Auto-generated Python classes from LinkML schema updates. **Auto-generated by GitHub Actions workflow** [skip ci]

aditigopalan requested review from adamjtaylor and Copilot and removed request for adamjtaylor January 15, 2026 20:18

Copilot started reviewing on behalf of aditigopalan January 15, 2026 20:18 View session

aditigopalan and others added 3 commits January 15, 2026 15:25

fix: configure Sphinx to output to READTHEDOCS_OUTPUT/html

93e248e

Merge 93e248e into 82823eb

4b6338e

chore: auto-generate Python classes from schema changes

fa7d1ca

Auto-generated Python classes from LinkML schema updates. **Auto-generated by GitHub Actions workflow** [skip ci]

Copilot AI reviewed Jan 15, 2026

View reviewed changes

aditigopalan and others added 3 commits January 15, 2026 15:33

fix: use python -m sphinx instead of sphinx-build command

31aefc1

Merge 31aefc1 into 82823eb

51b80ad

chore: auto-generate Python classes from schema changes

6540b56

Auto-generated Python classes from LinkML schema updates. **Auto-generated by GitHub Actions workflow** [skip ci]

aditigopalan and others added 3 commits January 15, 2026 15:37

fix: install requirements in build commands before running sphinx

fdaa2c7

Merge fdaa2c7 into 82823eb

44aa99a

chore: auto-generate Python classes from schema changes

0c33665

Auto-generated Python classes from LinkML schema updates. **Auto-generated by GitHub Actions workflow** [skip ci]

aditigopalan and others added 3 commits January 15, 2026 15:44

fix: clean build directories and explicitly set doctrees path

50c676f

Merge 50c676f into 82823eb

307d896

chore: auto-generate Python classes from schema changes

7b4b201

Auto-generated Python classes from LinkML schema updates. **Auto-generated by GitHub Actions workflow** [skip ci]

adamjtaylor approved these changes Jan 16, 2026

View reviewed changes

aditigopalan merged commit 2f9c84a into main Jan 20, 2026

aditigopalan deleted the docs branch January 20, 2026 14:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docs + Schema Fixes #129

Docs + Schema Fixes #129

Uh oh!

aditigopalan commented Jan 15, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 15, 2026

Uh oh!

github-actions bot commented Jan 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 15, 2026

Uh oh!

github-actions bot commented Jan 15, 2026

Uh oh!

github-actions bot commented Jan 15, 2026

Uh oh!

github-actions bot commented Jan 15, 2026

Uh oh!

adamjtaylor left a comment

Uh oh!

adamjtaylor Jan 16, 2026

Uh oh!

adamjtaylor Jan 16, 2026

Uh oh!

adamjtaylor Jan 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Docs + Schema Fixes #129

Docs + Schema Fixes #129

Uh oh!

Conversation

aditigopalan commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation Improvements and Schema Fixes

Overview

Major Changes

📚 Documentation Migration (MkDocs → Sphinx)

📊 CSV Export Functionality

🔧 Schema Fixes

🧹 Code Cleanup

🐛 Bug Fixes

Technical Details

Documentation Generation Scripts

Documentation Structure

Files Changed

Testing

Related Issues

Uh oh!

github-actions bot commented Jan 15, 2026

Uh oh!

github-actions bot commented Jan 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 15, 2026

Uh oh!

github-actions bot commented Jan 15, 2026

Uh oh!

github-actions bot commented Jan 15, 2026

Uh oh!

adamjtaylor left a comment

Choose a reason for hiding this comment

Uh oh!

adamjtaylor Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

adamjtaylor Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

adamjtaylor Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

aditigopalan commented Jan 15, 2026 •

edited

Loading