Docs (#310)

matthewmturner · web-flow · commit 147cc46e0709 · 2025-04-04T08:37:40.000-04:00
diff --git a/README.md b/README.md
@@ -5,80 +5,129 @@ Documentation is undergoing a significant revamp - the new documentation will be
 
 ## Overview
 
-`dft` is a batteries included suite of a [DataFusion](https://github.com/apache/arrow-datafusion) applications. The batteries being several common features to modern query execution engines such as:
+`dft` is a batteries-included suite of [DataFusion](https://github.com/apache/arrow-datafusion) applications that provides:
 
-- Query files from S3 or HuggingFace datasets
-- Support for common table formats (Deltalake, Iceberg, Hudi)
-- UDFs defined in multiple languages (WASM and soon Python)
-- Popular helper functions (for example for working with JSON and Parquet data)
+- **Data Source Integration**: Query files from S3, local filesystems, or HuggingFace datasets
+- **Table Format Support**: Native support for Delta Lake, Iceberg, and Hudi
+- **Extensibility**: UDFs defined in WASM (and soon Python)
+- **Helper Functions**: Built-in functions for JSON and Parquet data processing
 
-It provides two client interfaces to the query execution engine:
-1. Text User Interface (TUI): An IDE for DataFusion developers and users that provides a local database experience with utilities to analyze / benchmark queries.
-2. Command Line Interface (CLI): Scriptable engine for executing queries from files.
+The project offers four complementary interfaces:
 
-And two server implementation, FlightSQL & HTTP, leveraging the same execution engine behind the TUI and CLI.  This allows users to iterate and quickly develop a database then seamlessly deploy applications built on it.
+1. **Text User Interface (TUI)**: An interactive SQL IDE with real-time query analysis, benchmarking, and catalog exploration
+2. **Command Line Interface (CLI)**: A scriptable engine for executing queries from files or command line
+3. **FlightSQL Server**: A standards-compliant SQL interface for programmatic access
+4. **HTTP Server**: A REST API for SQL queries and catalog exploration
 
-`dft` is inspired by  [`datafusion-cli`], but has some differences:
-1. The TUI focuses on more complete and interactive experience for users.
-2. It contains many built in integrations such as Delta Lake and Iceberg that are not available in `datafusion-cli`.
-3. It provides FlightSQL and HTTP server implementations to make it easy to deploy DataFusion based applications / backends.
+All interfaces share the same execution engine, allowing you to develop locally with the TUI and then seamlessly deploy with the server implementations.
 
-[`datafusion-cli`]: https://datafusion.apache.org/user-guide/cli/overview.html
+`dft` builds upon [`datafusion-cli`](https://datafusion.apache.org/user-guide/cli/overview.html) with enhanced interactivity, additional integrations, and ready-to-use server implementations.
 
 ## User Guide
 
 ### Installation
 
-Currently, the only supported packaging is on [crates.io](https://crates.io/search?q=datafusion-dft).  If you already have Rust installed it can be installed by running `cargo install datafusion-dft`.  If rust is not installed you can download following the directions [here](https://www.rust-lang.org/tools/install).
+#### From crates.io (Recommended)
+```sh
+# If you have Rust installed
+cargo install datafusion-dft
 
-### Running the apps
+# For full functionality with all features
+cargo install datafusion-dft --all-features
+```
 
-The command for each of the apps are:
+If you don't have Rust installed, follow the [installation instructions](https://www.rust-lang.org/tools/install).
 
+#### Feature Flags
+Common feature combinations:
 ```sh
-# TUI (enabled by default)
+# Core with S3 support
+cargo install datafusion-dft --features=s3
+
+# Data lake formats
+cargo install datafusion-dft --features=deltalake,iceberg,hudi
+
+# With JSON and Parquet functions
+cargo install datafusion-dft --features=function-json,functions-parquet
+```
+
+See the [Features documentation](docs/features.md) for all available features.
+
+### Running the apps
+
+```sh
+# Interactive TUI (default)
 dft
 
-# Execute command with CLI (enabled by default)
-dft -c "SELECT 1"
+# CLI with direct query execution
+dft -c "SELECT 1 + 2"
 
-# Execute SQL from file (enabled by default)
+# CLI with file-based query
 dft -f query.sql
 
+# Benchmark a query (with stats)
+dft -c "SELECT * FROM my_table" --bench
+
 # Start FlightSQL Server (requires `flightsql` feature)
 dft serve-flightsql
 
 # Start HTTP Server (requires `http` feature)
 dft serve-http
 ```
 
-### DDL
-
-The CLI can also run your configured DDL prior to executing the query by adding the `--run-ddl` parameter.
+### Setting Up Tables with DDL
 
-To have the best experience with `dft` it is highly recommended to define all of your DDL in `~/.config/ddl.sql` so that any tables you wish to query are available at startup.  Additionally, now that DataFusion supports `CREATE VIEW` via sql you can also make a `VIEW` based on these tables.
+`dft` can automatically load table definitions at startup, giving you a persistent "database-like" experience.
 
-For example, your DDL file could look like the following:
+#### Using DDL Files
 
-```
-CREATE EXTERNAL TABLE users STORED AS NDJSON LOCATION 's3://bucket/users';
-
-CREATE EXTERNAL TABLE transactions STORED AS PARQUET LOCATION 's3://bucket/transactions';
+1. Create a DDL file (default: `~/.config/dft/ddl.sql`)
+2. Add your table and view definitions:
 
-CREATE EXTERNAL TABLE listings STORED AS PARQUET LOCATION 'file://folder/listings';
+```sql
+-- S3 data source (requires s3 feature)
+CREATE EXTERNAL TABLE users 
+STORED AS NDJSON 
+LOCATION 's3://bucket/users';
 
-CREATE VIEW OR REPLACE users_listings AS SELECT * FROM users LEFT JOIN listings USING (user_id);
-```
+-- Parquet files
+CREATE EXTERNAL TABLE transactions 
+STORED AS PARQUET 
+LOCATION 's3://bucket/transactions';
 
-This would make the tables `users`, `transactions`, `listings`, and the view  `users_listings` available at startup.  Any of these DDL statements could also be run interactively from the SQL editor as well to create the tables.
+-- Local files
+CREATE EXTERNAL TABLE listings 
+STORED AS PARQUET 
+LOCATION 'file://folder/listings';
 
-# Additional Documentation
+-- Create views from tables
+CREATE VIEW users_listings AS 
+SELECT * FROM users 
+LEFT JOIN listings USING (user_id);
 
-Links to more detailed documentation for each of the apps and all of the features can be found below.
+-- Delta Lake table (requires deltalake feature)
+CREATE EXTERNAL TABLE delta_table 
+STORED AS DELTATABLE 
+LOCATION 's3://bucket/delta_table';
+```
 
-- [Features](docs/features.md)
-- [CLI Docs](docs/cli.md)
-- [TUI Docs](docs/tui.md)
-- [FlightSQL Server Docs](docs/flightsql_server.md)
-- [HTTP Server Docs](docs/http_server.md)
-- [Config Reference](docs/config.md)
+#### Loading DDL
+
+- **TUI**: DDL is automatically loaded at startup
+- **CLI**: Add `--run-ddl` flag to execute DDL before your query
+- **Custom Path**: Configure a custom DDL path in your config file
+  ```toml
+  [execution]
+  ddl_path = "/path/to/my/ddl.sql"
+  ```
+
+## Quick Reference
+
+| Feature | Documentation |
+|---------|---------------|
+| **Core Features** | [Features Guide](docs/features.md) |
+| **TUI Interface** | [TUI Guide](docs/tui.md) |
+| **CLI Usage** | [CLI Guide](docs/cli.md) |
+| **FlightSQL Server** | [FlightSQL Guide](docs/flightsql_server.md) |
+| **HTTP Server** | [HTTP Guide](docs/http_server.md) |
+| **Configuration Options** | [Config Reference](docs/config.md) |
diff --git a/docs/config.md b/docs/config.md
@@ -1,19 +1,45 @@
-# Config Reference
+# Configuration Reference
 
-`dft` is configuration through a TOML file with default location of `~/.config/dft/config.toml`.  Within the config there is a shared configuration that can be used across all apps and app specific configuration.  The shared and app specific configs are merged to come up with a final configuration that is used.  DataFusion's execution configuration can be fully customized as part of this.
+`dft` is configured through a TOML file located at `~/.config/dft/config.toml` by default. The configuration system uses a hierarchical approach:
+
+1. **Default values** built into the application
+2. **Shared configuration** applied to all interfaces
+3. **App-specific configuration** for each interface (TUI, CLI, etc.)
+
+The final configuration for each interface is a merge of these three layers, with app-specific settings taking precedence over shared settings.
+
+## Configuration Structure
 
-The sections for configuring each app are shown below.
 ```toml
+# Settings applied to all interfaces
 [shared]
+# ...shared settings...
 
+# CLI-specific settings
 [cli]
+# ...CLI settings...
 
+# TUI-specific settings
 [tui]
+# ...TUI settings...
 
+# FlightSQL client settings
 [flightsql_client]
+# ...FlightSQL client settings...
 
+# FlightSQL server settings
 [flightsql_server]
+# ...FlightSQL server settings...
+
+# HTTP server settings
+[http_server]
+# ...HTTP server settings...
+```
+
+You can specify a different config file with the `--config` parameter:
 
+```bash
+dft --config /path/to/custom/config.toml
 ```
 
 ## Execution Config
diff --git a/docs/flightsql_server.md b/docs/flightsql_server.md
@@ -1,36 +1,76 @@
 # FlightSQL Server Guide
 
-`dft` includes an **experimental FlightSQL server** that you can run to provide programmatic SQL access to DataFusion.
-
----
+`dft` includes a FlightSQL server that provides programmatic SQL access to DataFusion through a standards-compliant interface. This enables you to connect using language-specific FlightSQL clients and build applications on top of DataFusion.
 
 ## Starting the Server
 
 ```sh
+# Start with default settings
 dft serve-flightsql
+
+# Start with your custom DDL loaded
+dft serve-flightsql --run-ddl
 ```
 
-## Endpoints
+## Supported Operations
+
+The server implements the FlightSQL protocol, providing:
 
-TODO List endpoints
+- SQL query execution
+- Schema fetching (TODO)
+- Prepared statements (TODO)
+- Catalog browsing (TODO)
 
-## Auth
+## Client Connections (TODO - Test this)
 
-Require basic or bearer authentication to make requests.
+You can connect to the server using any FlightSQL-compatible client:
+
+```python
+TODO with https://docs.influxdata.com/influxdb3/clustered/reference/client-libraries/flight/python-flightsql-dbapi/
+```
+
+## Authentication
+
+Configure Basic Auth or Bearer Token authentication:
 
 ```toml
 [flightsql_server.auth]
+# Option 1: Bearer token auth
 bearer_token = "MyToken"
-basic_auth.username = "User"
+
+# Option 2: Basic auth
+basic_auth.username = "User" 
 basic_auth.password = "Pass"
 ```
 
-## Metrics
-
-Prometheus metrics are automatically published.
+## Metrics and Monitoring
 
+Prometheus metrics are automatically published to help you monitor server performance:
 
 ```toml
 [flightsql_server]
+# Configure metrics port
 server_metrics_port = "0.0.0.0:9000"
 ```
+
+Available metrics include:
+- Query execution time
+- Active connections
+- Errors by type
+- Memory usage
+
+## Configuration
+
+Set connection and execution parameters in your config file:
+
+```toml
+[flightsql_server]
+# Server address
+connection_url = "http://localhost:50051"
+
+# Configure execution parameters
+[flightsql_server.execution.datafusion]
+target_partitions = 8
+```
+
+See the [Config Reference](config.md) for all available options.
diff --git a/docs/tui.md b/docs/tui.md
