Skip to content

Conversation

@shaofengshi
Copy link
Contributor

@shaofengshi shaofengshi commented Jan 8, 2026

Changes Made

This PR implements Lance REST Namespace Catalog support, enabling Daft to read from and write to Lance tables managed by REST-compliant services like LanceDB Cloud, Apache Gravitino, and other catalog systems.

Unified API with REST Support:

  • Enhanced daft.read_lance() and df.write_lance() to support REST URIs (rest://namespace/table)
  • Added LanceRestConfig class for REST service configuration (authentication, timeouts, custom headers)
  • URI-based routing automatically detects REST vs file-based Lance tables

Core Implementation:

  • LanceRestScanOperator with filter pushdown, column selection, and count operations
  • REST write operations supporting create, append, and overwrite modes
  • Catalog management functions for table creation and registration
  • Comprehensive error handling and logging

Enterprise Integration:

  • Namespace support for multi-tenant environments
  • API key authentication and custom headers
  • Compatible with Lance REST Namespace OpenAPI specification

Related Issues

Closes #5489

@github-actions github-actions bot added the feat label Jan 8, 2026
@codecov
Copy link

codecov bot commented Jan 8, 2026

Codecov Report

❌ Patch coverage is 85.62300% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.65%. Comparing base (fb45faf) to head (970a083).

Files with missing lines Patch % Lines
daft/io/lance/rest_scan.py 80.10% 37 Missing ⚠️
daft/io/lance/rest_write.py 91.89% 6 Missing ⚠️
daft/dataframe/dataframe.py 89.47% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #5980      +/-   ##
==========================================
+ Coverage   72.63%   72.65%   +0.02%     
==========================================
  Files         970      973       +3     
  Lines      126562   126862     +300     
==========================================
+ Hits        91924    92171     +247     
- Misses      34638    34691      +53     
Files with missing lines Coverage Δ
daft/io/__init__.py 100.00% <100.00%> (ø)
daft/io/lance/__init__.py 100.00% <100.00%> (ø)
daft/io/lance/_lance.py 92.10% <100.00%> (+0.92%) ⬆️
daft/io/lance/rest_config.py 100.00% <100.00%> (ø)
daft/dataframe/dataframe.py 77.53% <89.47%> (+0.19%) ⬆️
daft/io/lance/rest_write.py 91.89% <91.89%> (ø)
daft/io/lance/rest_scan.py 80.10% <80.10%> (ø)

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@shaofengshi shaofengshi force-pushed the write_lance_with_namespace branch from de9bf02 to bfdb95e Compare January 8, 2026 11:08
@shaofengshi shaofengshi marked this pull request as ready for review January 8, 2026 13:29
@shaofengshi shaofengshi changed the title feat: Support lance namespace read and write feat(lance): Support lance namespace read and write Jan 8, 2026
@kevinzwang
Copy link
Member

@universalmind303 could you take a look at this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Daft does not support lance namespace read and write

2 participants