Pydantic Support for Arrow Schema Definition and ORM

[everettVT]

LanceDB has a utility to convert pydantic objects to arrow schemas ( https://github.com/lancedb/lancedb/blob/main/python/python/lancedb/pydantic.py ), but I haven't seen arrow types implemented in pydantic attributes and fields outside my chats with LLMs.

Given many agentic and gen ai projects leverage pydantic for type validation, how would this best be implemented in Daft?

This speaks to the differences between OO and dataframe based processing, and I was looking for some wise words on the topic.

[jaychia]

Interesting thought -- any ideas for an API?

I could see this maybe being useful for DataType definitions for complex UDFs!

[everettVT]

This is a large and complex conversation. More complex than I initially thought it would be when I first started tinkering with the problem back in 2023.

So here's what I know:

1. Pydantic makes it FUN and Easy to compose data structures regardless of your familiarity with python or Pydantic.

2. Pydantic is particularly well suited for defining nested data structure schemas like structs, enabling a components-based approach which is highly modular and interoperable.

3. Pydantic validation is robust and has been vetted over many years of development with massive raving developer fanbase.

4. Pydantic ~= Dataclasses -> I like to think of Pydantic is that its a more feature rich implementation of python's dataclasses (for use cases like this) so that Full stack developers can quickly define validatable Object Relation Maps in python. These Object Relation maps can be hooked up to traditional, non-columnar databases like postgres or sql-lite and many solutions exist in this area.

5. Pydantic becomes much more powerful, fun, and important once you begin to use it for LLM structured generation. When I first started looking to use structured generation from libraries like instructor -> outlines, I discovered that Pydantic was the most convenient means of converting schemas to json_schema for "tool calling".

6. Pydantic Validation enables not just data typing and formatting conformance but also coercion which is important for non-determinsitc use cases like working with llms responses.

7. Almost all closed source LLM providers support json_schema tool calling and Enums which enable you to provide "choice" selection capabilities.

8. You can get pretty far with just json_schema's and enums, but the most powerful features of structured generation REQUIRE access to the actual model layers, effectively requiring you to use open source models. I'm specifically referring to context-free-grammars features which form the foundation of programming language parsing and enables all sorts of emergent auto-coding use cases for LLMs.

9. vLLM's implementation of Structured Generation is, in my opinion, THE INDUSTRY STANDARD for structured generation, and should be the primary focus of Daft support for this topic in my opinion.

**This is where the Pydantic support conversation starts to bleed into other feature discussions such as Async UDFs and llm_generate **
Concerning Async UDFs:
9. Adding support for OpenAI, Anthropic, Gemini, VertexAI, Fireworks, ............... (the list goes on), is a matter of defining an adapter around @colin-ho 's Semaphore based Async UDF, and implementing the different clients with optional imports.

This naturally dovetail's nicely with @universalmind303 's llm_generate function which deserves expansion IMO.

Bringing the conversation back to vLLM -> the most important reason I think we should focus on vLLM & OpenAI is that as of version 2.44, Ray implements vLLM natively for spinning up an OpenAI API compliant LLMServer, which can be served entirely with LLMConfigs!

See the following resources for examples on this:
https://docs.ray.io/en/latest/serve/llm/serving-llms.html
https://docs.vllm.ai/en/latest/features/structured_outputs.html#online-serving-openai-api
https://docs.vllm.ai/en/latest/features/structured_outputs.html#online-serving-openai-api

This means we can serve structured generation AT SCALE - which when paired with daft's scalability and I/O performance could provide a new cost performance benchmark with the right models. (Say Phi-4, or Gemma-series). This extends beyond simple chat or text generation inference to transcription and beyond!

This is particularly useful to all the Voice AI companies who have achieved strong PMF but completely rely on OpenAI for their inference.

Moving to Ray + Daft + vLLM would enable better predictability in costs, service performance, data privacy, and flexibility and could form the foundation of an entire startup! cough cough

Now, bringing this back to Pydantic, the whole point of this feature would be to support conversion of Pydantic BaseModels to Daft Schemas. There are a ton of implementation details I've thought about a lot already, but essentially:
We need Datatype and Field convertors in order to enable Schema Conversion. It is my proposal that we support conversion from vanilla Pydantic with base python types to convert to daft, arrow, iceberg, lance, and Pandas.

Naturally, in order to take advantage of Daft's Multimodal datatypes, you may want to use Image/Audio instead of Python Base Types, so the datatype handling needs to be robust.

The most important feature here is the composability that dataclass-like schema definition offers. Breaking Data Structures into components enables flexible composition and reuse. My Daft Dataframe simulation engine leverages this to implement the Archetype pattern which enables dynamic creation and streamlined querying of "archetype" tables. Archetypes are defined as the unique exact combinations of components, which are stored in their own tables. But thats another topic I could dive into.

Repo: https://github.com/VangelisTech/archetype
Pattern implementation: https://github.com/VangelisTech/archetype/blob/7db94610c50f47146aa5a00bc559cd1a181402ab/src/archetype/core/store.py#L134
Pattern Background: https://ajmmertens.medium.com/building-an-ecs-2-archetypes-and-vectorization-fe21690805f9

THE POINT of the components based approach is the Evergreen opportunity of the Single source of truth -> Let your users define their components and they can reuse them as they need. Updating a component triggers Schema Evolution on the table, made possible by modern table formats like Iceberg and Lance.

LanceDB's LanceModel attempts to tackle some of this and then some. Beyond it's to_arrow_schema method, LanceModel also enabled users to map source -> vector columns with embedding functions. RAG has kind of fallen off in popularity, but the overall idea here is robust. The exact same mechanism could be used to enable llm_generate to automatically update tables with new data.

The last thing I want to touch on is the extension of Pydantic Validation to columnar validation. There is one library that aims to do this called Pandera and they have taken the project pretty far. Pandera is a Dataframe testing tool is fully compliant with Pydantic's API. It leverages hypothesis to enable dataframe level testing and synthetic data generation as well. A Daft integration with Pandera here would go a long way in providing column level validation, but the composition API, in my opinion, could be cleaner. One thing that Pander doesn't offer currently is the ability to convert to Pydantic Models, and by extension, lacks the ability to generate json_schemas necessary for structured generation. From what I've seen in the issues, this shouldn't be too hard to contribute. One key feature that Pandera offers is FastAPI route model validation plus serialization to feather.

Other Pydantic to arrow utilities exist like the one @colin-ho used in Sashimi-4-Talent pydantic2arrow or ArrowDantic implemented in Rust.

SO! What would the API look like? Well, IMO we would need:

1. class DataTypeConvertor

   • from_python
   • from_daft
   • from_arrow
   • from_iceberg
   • to_arrow
   • to_daft
   • to_iceberg
   • to_pandas
   • etc

2. class FieldConvertor

   • from_dataclass_field
   • from_pydantic_field
   • from_lancemodel_field
   • from_iceberg_nestedfield
   • to_arrow_field
   • to_daft_field
   • to_pydantic_field
   • to_iceberg_nestedfield
   • etc

3. class SchemaConvertor

   • from_pydantic
   • from_dataclass
   • from_lancemodel (Pydantic BaseModel Subclass)
   • from_iceberg_schema (Which also happens to be a Pydantic BaseModel Subclass, but not implemented as a dataclass)
   • from_pyarrow_schema
   • from_daft_schema
   • to_pydantic
   • to_arrow_schema
   • to_iceberg_schema
   • to_daft_schema
   • to_json_schema

I could go on, but it would probably be worth having Gemini-2.5-Pro compare each of the repos implementations and usage patterns in cursor or something and doing a larger investigation. Potentially just adopting one of the technologies.

[everettVT]

In nicer form:

---

🅿️ The Power and Appeal of Pydantic

• Ease of Use: Pydantic makes defining data structures intuitive and enjoyable, regardless of one's Python or Pydantic expertise.
• Nested Structures & Modularity: It excels at defining nested schemas (like structs), promoting a highly modular and interoperable components-based approach.
• Robust Validation: Pydantic's validation capabilities are mature, well-tested over years, and backed by a large, enthusiastic developer community.
• Pydantic vs. Dataclasses: I view Pydantic as a more feature-rich version of Python's dataclasses, especially for use cases like creating validatable Object Relational Maps (ORMs) for traditional databases (e.g., Postgres, SQLite).

---

🤖 Pydantic in the LLM Era: Structured Generation

• Key Enabler for LLMs: Pydantic's value skyrockets when used for structured generation with Large Language Models (LLMs). When I first explored libraries like Instructor and Outlines, Pydantic was the most straightforward way to convert schemas to JSON Schema for "tool calling."
• Advanced Validation & Coercion: Pydantic validation isn't just about type checking; it handles data coercion, which is vital for non-deterministic LLM responses.
• Broad LLM Provider Support: Most closed-source LLM providers support JSON Schema for tool calling and Enums for "choice" selection.
• Beyond Basic JSON Schema: The Need for Open Source: While JSON Schemas and Enums are useful, advanced structured generation features (like context-free grammars, crucial for emergent auto-coding use cases) require access to model layers, necessitating open-source models.

---

🌟 vLLM: The Standard for Structured Generation

• In my view, vLLM's implementation of Structured Generation is the industry standard and should be Daft's primary focus for this topic.
• This is where Pydantic support begins to intersect with other feature discussions, notably Async UDFs and llm_generate.

---

🔗 Synergies: Async UDFs & llm_generate

• Async UDFs: Adding support for various LLM providers (OpenAI, Anthropic, Gemini, VertexAI, Fireworks, etc.) involves creating adapters around @colin-ho's Semaphore-based Async UDF and implementing different clients with optional imports.
• llm_generate Expansion: This naturally complements @universalmind303's llm_generate function, which I believe warrants further development.

---

🚀 vLLM & OpenAI via Ray: Scalable Structured Generation

• A key reason to prioritize vLLM & OpenAI is that Ray (as of v2.44) natively implements vLLM for spinning up an OpenAI API-compliant LLMServer, configurable entirely with LLMConfigs.
  • Ray Serve LLM Docs: https://docs.ray.io/en/latest/serve/llm/serving-llms.html
  • vLLM Structured Outputs: https://docs.vllm.ai/en/latest/features/structured_outputs.html#online-serving-openai-api
• Impact: This enables structured generation at scale. Pairing this with Daft's scalability and I/O performance could set new cost-performance benchmarks with suitable models (e.g., Phi-3 (Mistake in original, likely meant Phi-3 or Phi-2, or a newer Phi-4 if it exists by now), Gemma series). This extends beyond chat/text generation to transcription and more.
• Value Proposition: This is particularly beneficial for Voice AI companies heavily reliant on OpenAI. Moving to a Ray + Daft + vLLM stack offers better cost predictability, service performance, data privacy, and flexibility – potentially forming the basis of a startup pitch.

---

🎯 Core Proposal: Pydantic to Daft Schema Conversion

The central aim of this feature is to support the conversion of Pydantic BaseModels to Daft Schemas.

Implementation Considerations:

• Datatype and Field Converters: Essential for enabling schema conversion. I propose supporting conversion from vanilla Pydantic (with base Python types) to Daft, Arrow, Iceberg, Lance, and Pandas.
• Multimodal Datatypes: Robust datatype handling is needed to accommodate Daft's multimodal types (e.g., Image/Audio) alongside standard Python types.
• Composability: The dataclass-like schema definition offers powerful composability. Breaking data structures into components allows flexible composition and reuse.
  • My Daft Dataframe simulation engine uses this for an Archetype pattern, enabling dynamic creation and streamlined querying of "archetype" tables (unique combinations of components stored in their own tables) – but that's a deeper topic.
• Single Source of Truth & Schema Evolution: A components-based approach allows users to define components once and reuse them. Updating a component can trigger schema evolution, facilitated by modern table formats like Iceberg and Lance.

---

✨ Inspirations & Related Tools

• LanceDB's LanceModel: This project addresses some of these challenges. Beyond its to_arrow_schema method, LanceModel allows users to map source columns to vector columns with embedding functions. While RAG's hype has cooled, the underlying concept is solid and could be used to enable llm_generate to automatically update tables.
• Columnar Validation with Pandera:
  • Pandera extends Pydantic-like validation to dataframes and is compliant with Pydantic's API.
  • It uses Hypothesis for dataframe-level testing and synthetic data generation.
  • A Daft integration with Pandera could provide robust column-level validation.
  • Currently, Pandera doesn't convert to Pydantic Models (and thus JSON Schemas for structured generation), but this seems like a feasible contribution.
  • Pandera also offers FastAPI route model validation and serialization to Feather.
• Other Pydantic-to-Arrow Utilities:
  • pydantic2arrow (used by @colin-ho in Sashimi-4-Talent)
  • ArrowDantic (implemented in Rust)

---

🛠️ Proposed API Structure

To achieve this, we would likely need:

class DataTypeConvertor:
    # from_X methods
    def from_python(self, ...)
    def from_daft(self, ...)
    def from_arrow(self, ...)
    def from_iceberg(self, ...)
    # to_X methods
    def to_arrow(self, ...)
    def to_daft(self, ...)
    def to_iceberg(self, ...)
    def to_pandas(self, ...)
    # etc.

class FieldConvertor:
    # from_X_field methods
    def from_dataclass_field(self, ...)
    def from_pydantic_field(self, ...)
    def from_lancemodel_field(self, ...)
    def from_iceberg_nestedfield(self, ...)
    # to_X_field methods
    def to_arrow_field(self, ...)
    def to_daft_field(self, ...)
    def to_pydantic_field(self, ...)
    def to_iceberg_nestedfield(self, ...)
    # etc.

class SchemaConvertor:
    # from_X methods
    def from_pydantic(self, ...)
    def from_dataclass(self, ...)
    def from_lancemodel(self, ...) # Pydantic BaseModel Subclass
    def from_iceberg_schema(self, ...) # Also a Pydantic BaseModel Subclass
    def from_pyarrow_schema(self, ...)
    def from_daft_schema(self, ...)
    # to_X methods
    def to_pydantic(self, ...)
    def to_arrow_schema(self, ...)
    def to_iceberg_schema(self, ...)
    def to_daft_schema(self, ...)
    def to_json_schema(self, ...)

---

🔍 Next Steps

Further investigation is warranted. It might be beneficial to compare the implementations and usage patterns of existing repositories, potentially adopting or building upon one of these technologies.