feat: Add Flight SQL support for Dremio Cloud

[vandop]

Overview

This PR adds Apache Arrow Flight SQL support to the Dremio MCP server, providing a more efficient protocol for SQL query execution, especially for large result sets.

Key Features

🚀 Flight SQL Transport

• New FlightSQLTransport class for Apache Arrow Flight SQL connectivity
• Automatic protocol detection based on DREMIO_SQL_PROTOCOL environment variable
• Seamless fallback to REST API when Flight SQL is not configured
• Full compatibility with existing REST-based workflows

🔧 Configuration Options

• Environment Variable: DREMIO_SQL_PROTOCOL=FLIGHT_SQL or REST (default)
• YAML Configuration: sql_protocol: FLIGHT_SQL in dremio config section
• Automatic URI conversion from HTTPS to gRPC+TLS for Flight SQL

📦 Dependencies

• Added: adbc-driver-flightsql>=1.7.0 for Flight SQL operations
• Added: pyarrow>=18.0.0 for Arrow data handling
• Added: starlette>=0.41.0 for enhanced web framework support

Technical Implementation

Transport Layer

• FlightSQLTransport: New transport class in src/dremioai/api/transport_flightsql.py
• Authentication: Bearer token authentication using Personal Access Tokens
• Error Handling: Comprehensive error handling with fallback mechanisms
• Connection Management: Efficient connection pooling and lifecycle management

Query Execution

• Enhanced run_query(): Automatic protocol selection based on configuration
• Multiple fetch methods: Support for fetch_df(), fetchall(), and fetch_arrow_table()
• Schema handling: Robust handling of Arrow schema inconsistencies
• Data conversion: Seamless conversion between Arrow and Pandas formats

Configuration Integration

• Settings enhancement: Added sql_protocol field to Dremio configuration model
• Environment variable support: Full support for DREMIO_SQL_PROTOCOL
• Backward compatibility: Existing configurations continue to work unchanged

⚠️ Known limitation: Simple literal queries (e.g., SELECT 1) may fail due to ADBC driver schema strictness, but all real table queries work perfectly.

Documentation

• Updated README: Comprehensive Flight SQL configuration documentation
• Configuration examples: YAML and environment variable examples
• Protocol comparison: REST vs Flight SQL feature comparison
• Usage guidelines: When to use each protocol

Migration Path

Existing users can:

1. Continue using REST (default behavior, no changes required)
2. Opt into Flight SQL by setting DREMIO_SQL_PROTOCOL=FLIGHT_SQL
3. Switch protocols at runtime through environment variables

Files Changed

• src/dremioai/api/transport_flightsql.py - New Flight SQL transport implementation
• src/dremioai/api/dremio/sql.py - Enhanced query execution with protocol selection
• src/dremioai/config/settings.py - Added sql_protocol configuration field
• pyproject.toml - Added Flight SQL dependencies
• README.md - Updated with Flight SQL documentation

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Vando Pereira seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}