first draft extension_types

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Author: connortsui20

proposals/0005-extension-types.md

@@ -0,0 +1,97 @@
+- Start Date: (2026-02-27)
+- RFC PR: [vortex-data/rfcs#0000](https://github.com/vortex-data/rfcs/pull/0000)
+- Tracking Issue: [vortex-data/vortex#6547](https://github.com/vortex-data/vortex/issues/6547)
+
+## Summary
+
+We would like to build a more robust system for extension data types (or DTypes).
+
+TODO
+
+## Motivation
+
+TODO
+
+## Design
+
+[vortex-data/vortex#6081](https://github.com/vortex-data/vortex/pull/6081) introduced vtables (virtual tables, or Rust unit structs with methods) for extension `DType`s. Each extension type (e.g. `Timestamp`) now implements `ExtDTypeVTable`, which handles validation, serialization, and metadata. The type-erased `ExtDTypeRef` carries this vtable with it inside `DType::Extension`.
+
+There were a few blockers (detailed in the previous tracking issue [vortex-data/vortex#6547](https://github.com/vortex-data/vortex/issues/6547)), but now that those have been resolved we can move forward with this.
+
+Now that `vortex-scalar` and `vortex-dtype` have been merged into `vortex-array`, we can now place all extension logic (for types, scalars, and arrays) onto an `ExtVTable`. It will look something like so:
+
+```rust
+// Naming should be considered VERY unstable / not set!
+
+pub trait ExtVTable: 'static + Send + Sync + ... {
+    // Extra data that complements the extension type.
+    type Metadata: ...;
+
+    // A native Rust value that represents a scalar of the extension type.
+    type Value<'a>: Display;
+
+    // `DType`
+
+    fn id(&self) -> ExtID;
+    fn validate_dtype(&self, metadata: &Self::Metadata, storage_dtype: &DType) -> VortexResult<()>;
+    fn serialize_metadata(&self, metadata: &Self::Metadata) -> VortexResult<Vec<u8>>;
+    fn deserialize_metadata(&self, data: &[u8]) -> VortexResult<Self::Metadata>;
+
+    // `Scalar`
+
+    fn validate_scalar_value(&self, metadata: &Self::Metadata, storage_dtype: &DType, storage_value: &ScalarValue) -> VortexResult<()>;
+    fn unpack<'a>(&self, metadata: &'a Self::Metadata, storage_dtype: &'a DType, storage_value: &'a ScalarValue) -> Self::Value<'a>;
+    fn cast_scalar(&self, metadata: &Self::Metadata, scalar: &Scalar, target: &DType) -> VortexResult<Scalar> { ... }
+
+    // `ArrayRef`
+
+    fn validate_array(&self, metadata: &Self::Metadata, storage_array: &ArrayRef) -> VortexResult<()>;
+    fn cast_array(&self, metadata: &Self::Metadata, array: &ArrayRef, target: &DType) -> VortexResult<ArrayRef> { ... }
+    fn other_compute_thing???(&self, ...) -> VortexResult<ArrayRef> { ... }
+    // <-- Probably a lot more than this -->
+}
+```
+
+TODO
+
+## Compatibility
+
+TODO
+
+## Drawbacks
+
+TODO
+
+## Alternatives
+
+TODO
+
+## Prior Art
+
+TODO
+
+## Unresolved Questions
+
+TODO
+
+## Future Possibilities
+
+If we can get extension types working well, then theoretically we can easily add all of these types:
+
+- `DateTimeParts` (`Primitive`)
+- Matrix (`FixedSizeList`)
+- Tensor (`FixedSizeList`)
+- UUID (Do we need to add `FixedSizeBinary` as a canonical type?)
+- JSON (`UTF8`)
+- PDX: https://arxiv.org/pdf/2503.04422v1 (`FixedSizeList`)
+- Variant
+  - Shredding (Lots of possibilities here!)
+- Union
+  - Sparse (`Struct { Primitive, Struct { types } }`)
+  - Dense[^1]
+- Map (`List<Struct { K, V }>`)
+- Tags: https://github.com/vortex-data/vortex/discussions/5772#discussioncomment-15279892 (`ListView<Utf8>`)
+- `Struct` but with protobuf-style field numbers (`Struct`)
+- Probably lots more!
+
+[^1]: `Struct` doesn't work here because children can have different lengths, but what we could do is simply force the inner `Struct { types }` to hold `SparseArray` fields, which would effectively be the exact same but with the overhead of tracking indices for each of the child fields. In that case, it might just be better to always use a "sparse" union and let the compressor decide what to do.