docs: update docs with new capabilities #874
Conversation
Broshen
requested review from
EpsilonPrime,
cpcloud,
jacques-n,
vbarua and
westonpace
as code owners
|
The following reports were also generated: Substrait Documentation Update ReportDate: October 20, 2025 SummaryThis report documents the discrepancies between the current Substrait implementation and the documentation hosted at substrait.io. The analysis was performed by comparing the protobuf definitions, CHANGELOG entries, and existing markdown documentation files. Critical Missing Documentation1. Substrait Dialects (v0.76.0) - COMPLETELY UNDOCUMENTEDStatus: Missing entirely from documentation
Required Action: Create new documentation page explaining:
Location to add: 2. Per-Plan Type Aliases (v0.77.0) - PARTIALLY DOCUMENTEDStatus: Type aliases documented but new Plan field not fully explained
Current Documentation:
3. CreateMode Enum Values (v0.60.0) - INCOMPLETEStatus: Concept documented but enum values not listed
Current Documentation: 4. BuildInput Field in HashJoinRel (v0.73.0) - MISSING DETAILStatus: Field added but not prominently documented
Current Documentation: Mentioned briefly in physical_relations.md
5. Mark Join Types (v0.56.0) - INCOMPLETEStatus: Defined but not fully explained
Current Documentation: Physical relations mention join types but mark joins need explanation
6. DynamicParameterBinding in Plan (v0.67.0) - MISSINGStatus: Dynamic parameters documented in expressions, but Plan-level bindings not explained
Current Documentation:
7. IntervalCompound Type (v0.54.0) - NEEDS VERIFICATIONStatus: Proto definition exists, documentation needs verification
Required Action: Verify type_classes.md includes IntervalCompound documentation 8. ExtendedExpression Version Field (v0.23.0) - NEEDS VERIFICATIONStatus: Version field added to ExtendedExpression
Required Action: Verify extended_expression.md documents version field requirement Documentation That Is Current✅ UpdateRel (v0.61.0) - DOCUMENTED
✅ PrecisionTimestamp with Picoseconds (v0.67-0.69) - DOCUMENTED
✅ Window Functions (v0.32.0) - DOCUMENTED
✅ ExpandRel (v0.32.0) - DOCUMENTED
✅ NestedLoopJoinRel (v0.37.0) - DOCUMENTED
✅ ExchangeRel (v0.32.0) - DOCUMENTED
✅ Iceberg Table (v0.64.0) - DOCUMENTED
✅ SavedComputation/LoadedComputation (v0.58.0, v0.75.0) - DOCUMENTED
✅ Dynamic Parameters (v0.67.0) - DOCUMENTED (partially)
Recommendations for Documentation ImprovementPriority 1 (Critical - Missing Entirely)
Priority 2 (High - Incomplete)
Priority 3 (Medium - Needs Enhancement)
Files That Need Updates
Changelog Features AnalyzedThe following CHANGELOG versions were specifically reviewed:
End of Report |
Substrait Documentation Updates SummaryDate: October 20, 2025 OverviewThis document summarizes all documentation updates made to bring substrait.io in sync with the current implementation. Files Created1. substrait/site/docs/spec/dialects.md (NEW)Status: ✅ Created Content Added:
Navigation Updated: Added to Files Updated2. substrait/site/docs/types/type_aliases.mdStatus: ✅ Updated
Addresses: v0.77.0 per-plan type aliases feature 3. substrait/site/docs/relations/logical_relations.mdStatus: ✅ Updated
Addresses: v0.60.0 CreateMode for CTAS in WriteRel 4. substrait/site/docs/relations/physical_relations.mdStatus: ✅ Updated
Addresses: v0.73.0 HashJoin BuildInput specification 5. substrait/site/docs/expressions/dynamic_parameters.mdStatus: ✅ Updated
Addresses: v0.67.0 DynamicParameterBinding in Plan message Documentation That Was Already CurrentThe following features were verified to be properly documented: ✅ UpdateRel (v0.61.0) - Well documented in logical_relations.md Everything That Was Out of DatePriority 1: Critical - Missing Entirely
Priority 2: High - Incomplete or Unclear
Priority 3: Medium - Needs Enhancement
Impact SummaryNew Documentation Pages: 1
Updated Documentation Pages: 4
Updated Navigation Files: 1
Total Lines Added: ~400+Features Now Documented: 5 previously undocumented/incomplete featuresVerification StatusAll changes have been made to the markdown source files in: The documentation will need to be rebuilt using MkDocs to generate the updated HTML site in: To rebuild the site, run: cd /Users/boshen.cui/go/src/github.com/DataDog/substrait/substrait/site
mkdocs buildRelated Files
Recommendations
All documentation updates completed successfully. |
There was a problem hiding this comment.
I haven't gone through the rest of the files, just started on dialects.md. But looks like the robots may have produced quite a few errors.
I still prefer, given the mistakes found, to do this in a few different PRs instead of one massive one :)
Thank you for doing this!
|
|
||
| ```yaml | ||
| name: system_name | ||
| type: sql # or other system type |
There was a problem hiding this comment.
Some of these fields are wrong FYI.
AFAICT type is not a field in the dialect schema. So let's drop these and references to it. Similarly, the fields below are not eg.g scalar_functions but supported_scalar_functions.
The available fields are:
- name
- dependencies
- supported_types
- supported_relations
- supported_expressions
- supported_scalar_functions
- supported_aggregate_functions
- supported_window_functions
| ```yaml | ||
| supported_types: | ||
| i8: | ||
| sql_type_name: tinyint | ||
| i32: | ||
| sql_type_name: integer | ||
| supported_as_column: true | ||
| user_defined: | ||
| source: geo # reference to dependency | ||
| name: geometry | ||
| ``` |
There was a problem hiding this comment.
These also don't seem right.
| ```yaml | |
| supported_types: | |
| i8: | |
| sql_type_name: tinyint | |
| i32: | |
| sql_type_name: integer | |
| supported_as_column: true | |
| user_defined: | |
| source: geo # reference to dependency | |
| name: geometry | |
| ``` | |
| ```yaml | |
| supported_types: | |
| type_i8: | |
| type: I8 | |
| system_metadata: | |
| name: integer | |
| supported_as_column: true | |
| type_i32: | |
| type: I32 | |
| system_metadata: | |
| name: int | |
| supported_as_column: true | |
| type_user_defined: | |
| type:L USER_DEFINED | |
| source: extension:io.substrait:functions_geometry | |
| name: geometry | |
| system_metadata: | |
| name: geo | |
| supported_as_column: true |
2 participants
This PR updates the site documentation with new features that have been added. This was done by cloning all repos in the substrait-io org into a folder, and running claude with the prompt