You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| Relation Modification (semantic) | Extensions to an existing relation that will alter the semantics of that relation. These kinds of extensions require that any plan consumer understand the extension to be able to manipulate or execute that operator. Ignoring these extensions will result in an incorrect interpretation of the plan. An example extension might be creating a customized version of Aggregate that can optionally apply a filter before aggregating the data. <br /><br />Note: Semantic-changing extensions shouldn't change the core characteristics of the underlying relation. For example, they should *not* change the default direct output field ordering, change the number of fields output or change the behavior of physical property characteristics. If one needs to change one of these behaviors, one should define a new relation as described below. |
140
-
| Relation Modification (optimization) | Extensions to an existing relation that can improve the efficiency of a plan consumer but don't fundamentally change the behavior of the operation. An example might be an estimated amount of memory the relation is expected to use or a particular algorithmic pattern that is perceived to be optimal. |
141
-
| New Relations | Creates an entirely new kind of relation. It is the most flexible way to extend Substrait but also make the Substrait plan the least interoperable. In most cases it is better to use a semantic changing relation as oppposed to a new relation as it means existing code patterns can easily be extended to work with the additional properties. |
142
-
| New Read Types | Defines a new subcategory of read that can be used in a ReadRel. One of Substrait is to provide a fairly extensive set of read patterns within the project as opposed to requiring people to define new types externally. As such, we suggest that you first talk with the Substrait community to determine whether you read type can be incorporated directly in the core specification. |
143
-
| New Write Types | Similar to a read type but for writes. As with reads, the community recommends that interested extenders first discuss with the community about developing new write types in the community before using the extension mechanisms. |
144
-
| Plan Extensions | Semantic and/or optimization based additions at the plan level. |
145
-
146
-
Because extension mechanisms are different for each serialization format, please refer to the corresponding serialization sections to understand how these extensions are defined in more detail.
135
+
Advanced extensions provide a way to embed custom functionality that goes beyond the standard YAML-based simple extensions. Unlike simple extensions, advanced extensions use Protocol Buffer's `google.protobuf.Any` type to embed arbitrary extension data directly into Substrait messages.
136
+
137
+
### How Advanced Extensions Work
138
+
139
+
Advanced extensions come in several main forms, discussed below:
140
+
141
+
1. Embedded extensions: These use the `AdvancedExtension` message for adding custom data to existing Substrait elements
142
+
2. Custom relation types: For defining entirely new relational operations
143
+
3. Custom read/write types: for defining new ways to read from or write to data sources
144
+
145
+
### Embedded Extensions via `AdvancedExtension`
146
+
147
+
The simplest forms of advanced extensions use the `AdvancedExtension` message, which contains two types of extensions:
148
+
149
+
```proto
150
+
message AdvancedExtension {
151
+
// Optimizations are helpful information that don't influence semantics.
152
+
// May be ignored by a consumer.
153
+
repeated google.protobuf.Any optimization = 1;
154
+
155
+
// Enhancements alter semantics. Cannot be ignored by a consumer.
156
+
google.protobuf.Any enhancement = 2;
157
+
}
158
+
```
159
+
160
+
#### Optimizations
161
+
162
+
- Provide hints to improve performance but don't change the meaning of operations
163
+
- Can be safely ignored by consumers that don't understand them
| **`Plan`** | Global extensions affecting the entire plan |
181
+
| **`RelCommon`** | Extensions for any relational operator |
182
+
| **Relations** (e.g. `ProjectRel`) | Extensions for a specific relation type |
183
+
| **Hints** | Extensions within optimization hints |
184
+
| **`ReadRel.NamedTable`** | Add custom metadata to named table references |
185
+
| **`WriteRel.NamedObjectWrite`** | Add custom metadata to write targets |
186
+
| **`DdlRel.NamedObjectWrite`** | Add custom metadata to DDL targets |
187
+
188
+
### Custom Relations
189
+
190
+
The second form of advanced extensions provides entirely new relational operations via dedicated extension relation types. These allow you to define custom relations while maintaining proper integration with the type system:
0 commit comments