Make the python tooling more complete

[pahjbo]

I’m outlining a concrete delivery plan to bring tools/pythontooling to feature parity with the Gradle plugin task surface.

• Inventory current Python vs Gradle capabilities from code
• Define one Python command per Gradle task (without vodml prefix)
• Propose implementation phases with dependencies and risks
• Add acceptance criteria and test plan so progress is measurable

What exists today (from repo)

• Python CLI is minimal in tools/pythontooling/src/vodmltools/cli.py: only schema, doc (stub), and validate.
• XSLT wrappers exist in tools/pythontooling/src/vodmltools/vodml.py (Vodml2Java, Vodml2Python, Vodml2md, Vodml2TAP, etc.), so the transformation primitives are mostly available.
• Gradle plugin defines the full task surface in tools/gradletooling/gradle-plugin/src/main/kotlin/net/ivoa/vodml/gradle/plugin/VodmlGradlePlugin.kt:
  • vodmlDoc, vodmlSite, vodmlValidate, vodslToVodml, vodmlToVodsl, vodmlXsdToVodsl, vodmlJavaGenerate, vodmlPythonGenerate, vodmlSchema (+ deprecated alias vodmlGenerateJava).

Parity target (recommended Python command mapping)

• vodmlDoc -> vodml doc
• vodmlSite -> vodml site
• vodmlValidate -> vodml validate
• vodslToVodml -> vodml vodsl-to-vodml
• vodmlToVodsl -> vodml vodml-to-vodsl
• vodmlXsdToVodsl -> vodml xsd-to-vodsl
• vodmlJavaGenerate -> vodml java-generate
• vodmlPythonGenerate -> vodml python-generate
• vodmlSchema -> vodml schema
• deprecated alias parity (optional) -> vodml generate-java forwarding to java-generate

Implementation plan

Phase 0 - Scope + UX contract

• Freeze command names/options and output conventions in a short spec doc.
• Align defaults with Gradle extension defaults from VodmlExtension.kt:
  • input VO-DML dir default src/main/vo-dml
  • VODSL dir default src/main/vodsl
  • output dirs under build/generated/...
  • auto-detect *vodml-binding.xml in project root.

Phase 1 - Shared execution core (foundation)

• Add a central context builder (e.g., context.py) to resolve:
  • input files (--vodml-dir, --vodml-files, --vodsl-dir, --vodsl-files)
  • binding files (explicit + auto-detect)
  • dependency models and catalogs.
• Refactor createCatalog(...) usage so all commands share identical catalog behavior.
• Introduce dependency ingestion equivalent to ExternalModelHelper:
  • read JAR manifests (VODML-source, VODML-binding)
  • extract referenced files from dependency JARs (zipfile-based)
  • merge external model/binding files for transforms.

Phase 2 - Complete command surface

• Implement missing commands in cli.py:
  • site, vodsl-to-vodml, vodml-to-vodsl, xsd-to-vodsl, java-generate, python-generate.
• Finish doc implementation to match VodmlDocTask behavior:
  • .gvd generation
  • Graphviz dot -> .svg
  • .html, .graphml, .tex.
• Implement site analogous to VodmlSiteTask:
  • markdown docs
  • nav aggregation (*_nav.json -> allnav.yml)
  • imported model nav section.

Phase 3 - Schema + validation parity details

• Upgrade schema command to full VodmlSchemaTask parity:
  • XML schema (.xsd)
  • JSON schema (.json, pretty-printed)
  • OpenAPI YAML (.yaml, remove "$comment" recursively)
  • TAP schema (.tap.xml)
  • TAP PlantUML (.tap.plantuml)
  • catalog artifacts (xmlcat.xml, jsoncat.txt) via create-catalogues.xsl.
• Upgrade validate parity with VodmlValidateTask:
  • XML schema validation (equivalent to XMLValidator) before schematron
  • schematron result handling with non-zero exit for invalid models.

Phase 4 - Config model + task-style invocation

• Add config-file support (e.g., vodmltools.toml or YAML) mirroring Gradle vodml { ... } options.
• Keep CLI overrides for all config fields.
• Preserve explicit command behavior for CI usage.

Phase 5 - Test matrix + docs

• Unit tests for:
  • catalog creation
  • binding detection
  • jar manifest extraction
  • option parsing.
• Integration tests against sample models mirroring Gradle sample flows.
• Rewrite tools/pythontooling/README.md:
  • parity command table
  • examples per command
  • external tool prerequisites (dot, mkdocs, etc.)
  • migration notes from older CLI and issue create version of tooling that does not depend on gradle to run #63 background.

Risk-driven ordering (why this order)

• Biggest parity gap is not XSLT calls, but dependency model resolution + defaults + orchestration.
• Implementing shared context first avoids duplicating file discovery/catalog logic across 9 commands.
• Schema/doc/site are highest complexity; do them after core plumbing.

Definition of done

• Every Gradle task in VodmlGradlePlugin.kt has an equivalent Python CLI command.
• Outputs match naming/location semantics used by Gradle tasks.
• Dependency model handling works without manually listing all imported model files.
• README documents stable command surface and examples.
• Integration tests pass on sample models for schema, java-generate, python-generate, doc, site, validate.