Add HuggingFaceTabularStorage for readonly access to HF datasets with pagination support

[Copilot]

Implementation Plan for HuggingFace Tabular Storage

• Create HuggingFaceTabularStorage.ts with class structure
  • Import required dependencies
  • Define class extending BaseTabularStorage
  • Implement constructor with HF-specific parameters
  • Add private helper methods (fetchApi, HF feature converter)
• Implement read operations
  • setupDatabase() - validate dataset exists
  • get() - fetch via /filter endpoint
  • getAll() - paginate through /rows endpoint
  • getBulk() - fetch single page via /rows endpoint
  • search() - filter via /filter endpoint
  • size() - get count from /size endpoint
  • records() - async generator (inherited from BaseTabularStorage)
  • pages() - async generator (inherited from BaseTabularStorage)
• Implement write operations (throw readonly errors)
  • put() and putBulk()
  • delete(), deleteAll(), deleteSearch()
  • subscribeToChanges() - throw not supported error
  • destroy() - no-op
• Add static factory method fromDataset() for schema auto-detection
• Export from common.ts with service token
• Create comprehensive tests in __tests__ directory
  • Mock fetch responses
  • Test read operations
  • Test readonly errors
  • Test schema auto-detection
  • Test getBulk with offset/limit
  • Test records() async generator
  • Test pages() async generator
  • Test empty WHERE clause validation
• Security review and fixes
  • Fixed incomplete sanitization by escaping backslashes
  • CodeQL security scan passed
• Build error fixes
  • Fixed type casting for indexes parameter
  • Fixed Object.entries type cast for keyObj
  • Fixed test types to use proper PrimaryKey objects
  • Fixed PrimaryKey type declarations in tests
  • Added explicit type annotations to inline storage declarations
• Test failures fixed
  • Added label index to Read Operations test suite
  • Fixed URL encoding assertion to match actual URLSearchParams encoding
• Rebased on copilot/add-getbulk-to-tabular-storage
  • Implemented getBulk(offset, limit) method
  • Refactored getAll() to use getBulk internally
  • Added tests for getBulk, records(), and pages() methods
• Code review improvements
  • Added validation for empty WHERE clause in search method
  • Fixed test to match complete error message

All tests passing, code review clean, no security issues.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>HuggingFace Tabular Storage</issue_title>
<issue_description>---
name: HuggingFace Tabular Storage
overview: Create a new readonly HuggingFaceTabularStorage class that implements ITabularStorage backed by the HuggingFace Dataset Viewer REST API, supporting both user-provided schemas and auto-detection from HF features.
todos:

• id: create-hf-storage
  content: Create HuggingFaceTabularStorage.ts with class, constructor, readonly write methods, fetchApi helper, and HF feature-to-schema conversion
  status: pending
• id: implement-reads
  content: "Implement read operations: get() via /filter, getAll() via paginated /rows, search() via /filter, size() via /size"
  status: pending
• id: implement-setup-factory
  content: Implement setupDatabase() with schema validation and static fromDataset() factory for auto-detection
  status: pending
• id: add-exports
  content: Export from common.ts and add service token
  status: pending
• id: add-tests
  content: Add tests with mocked fetch for read ops, readonly errors, and schema auto-detection
  status: pending
  isProject: false

---

HuggingFace Tabular Storage

Architecture

The new HuggingFaceTabularStorage extends BaseTabularStorage and maps read operations to the HuggingFace Dataset Viewer API. Write operations throw a readonly error. It uses fetch() so it works in both browser and server environments.

flowchart LR
    subgraph client [Client Code]
        get["get()"]
        getAll["getAll()"]
        search["search()"]
        size["size()"]
        put["put() / delete()"]
    end
    subgraph hf [HuggingFace Dataset Viewer API]
        filterEp["/filter"]
        rowsEp["/rows"]
        sizeEp["/size"]
        firstRowsEp["/first-rows"]
    end
    get -->|"WHERE pk=val"| filterEp
    getAll -->|"paginated offset+length"| rowsEp
    search -->|"WHERE col=val"| filterEp
    size --> sizeEp
    put -->|"throws ReadonlyError"| nowhere["X"]

Loading

New File

**[packages/storage/src/tabular/HuggingFaceTabularStorage.ts](packages/storage/src/tabular/HuggingFaceTabularStorage.ts)** -- the sole new file in the storage package.

Constructor

Follows the existing pattern but adds HuggingFace-specific params:

constructor(
  dataset: string,      // e.g. "cornell-movie-review-data/rotten_tomatoes"
  config: string,       // e.g. "default"
  split: string,        // e.g. "train"
  schema: Schema,
  primaryKeyNames: PrimaryKeyNames,
  options?: {
    token?: string;
    baseUrl?: string;    // default: "https://datasets-server.huggingface.co"
    indexes?: readonly (keyof Entity | readonly (keyof Entity)[])[];
  }
)

Schema Handling (both auto-detect and user-provided)

• User-provided schema: Pass schema to constructor. During setupDatabase(), fetch /first-rows and validate that HF features match the schema columns.
• Auto-detect schema: Static factory method HuggingFaceTabularStorage.fromDataset(dataset, config, split, options?) that:
  1. Calls /first-rows to get HF features
  2. Converts HF feature types to JSON Schema (string -> {type:"string"}, int64 -> {type:"integer"}, float64 -> {type:"number"}, bool -> {type:"boolean"})
  3. Generates a row_idx primary key (auto-generated integer) since HF datasets often lack a natural PK
  4. Returns a constructed HuggingFaceTabularStorage instance

Method Implementation

Read Operations (implemented)

• **get(key)** -- Uses /filter?where=pk_col=val with the primary key column(s). Returns the first matching row or undefined.
• **getAll()** -- Paginates through /rows?offset=N&length=100 (HF max is 100 per request). Collects all pages using num_rows_total from the response.
• **search(key)** -- Uses /filter?where=col1='val1' AND col2='val2' constructed from the partial entity. Paginates if results exceed 100.
• **size()** -- Uses /size endpoint, extracts num_rows for the specific config/split.
• **setupDatabase()** -- Fetches /first-rows to validate the dataset exists and (optionally) validate schema against features.

Write Operations (throw readonly error)

• put(), putBulk(), delete(), deleteAll(), deleteSearch() -- All throw Error("HuggingFaceTabularStorage is readonly").

Other Methods

• **subscribeToChanges()** -- Not supported (HF datasets are static). Throws a descriptive error.
• **destroy()** -- No-op (no resources to clean up).

Internal Helper: fetchApi()

Private method that handles:

• Constructing the full URL with query parameters
• Adding Authorization: Bearer {token} header when token is provided
• Error handling for non-200 responses
• JSON parsing

HF Feature-to-JSON-Schema Conversion

A private helper hfFeatureToJsonSchema(feature) that maps HF dataset feature types:

| HF Fea...

• Fixes HuggingFace Tabular Storage #211

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[sroussey]

@copilot fix build errors

[Copilot]

@copilot fix build errors

Fixed TypeScript compilation errors in commit ac20427:

Type Errors Fixed:

1. Added type cast for indexes parameter in constructor (line 123)
2. Added type cast for keyObj in get() method (line 237)
3. Fixed test types to use proper PrimaryKey objects instead of primitives (e.g., { id: 1 } instead of 1)

Errors Resolved:

• TS2345: Argument type mismatch for indexes parameter
• TS2769: No overload matches for Object.entries call
• TS2345: Test parameter type mismatches (changed from any to proper types)

The build should now pass successfully.

[sroussey]

@copilot fix build errors

[Copilot]

@copilot fix build errors

Fixed TypeScript type errors in commit 6f5c0cf:

Root Cause:
The test file had incorrect PrimaryKey type declarations. The PrimaryKey type is a Pick type (Pick<Entity, KeyFields>), not a primitive value.

Fixes Applied:

• Line 205: Changed PrimaryKey = number → PrimaryKey = { id: number }
• Line 373: Changed PrimaryKey = number → PrimaryKey = { id: number }

Errors Resolved:

• TS2345: All 5 type assignment errors in test file
• Storage methods now correctly expect { id: number } objects instead of primitive numbers

Build should now pass successfully.

[sroussey]

@copilot there are still build errors

[Copilot]

@copilot there are still build errors

Fixed remaining TypeScript errors in commit a0a76f1:

Errors Found:

• Line 460: { id: number } not assignable to never
• Line 493: { id: string } not assignable to never

Root Cause:
Inline storage declarations in API Integration tests lacked explicit type annotations, preventing TypeScript from correctly inferring the PrimaryKey generic type.

Fix:
Added explicit HuggingFaceTabularStorage<...> type annotations to both inline storage declarations (lines 446-453 and 481-488), specifying all generic parameters including the correct PrimaryKey type.

All TypeScript compilation errors resolved. Build should now pass.

[sroussey]

@copilot fix test failures

[Copilot]

@copilot fix test failures

Fixed 2 test failures in commit 28f6002:

Test 1: "should search entities by partial key"

• Added indexes: [["label"]] to storage initialization
• Storage now has an index on the 'label' field, allowing searches

Test 2: "should properly escape string values in WHERE clauses"

• Updated URL encoding assertion to match actual URLSearchParams encoding
• Changed from test%5C's%20value to where=id%3D%27test%5C%27s
• Verifies both backslash (\ → %5C) and apostrophe (' → %27) are properly escaped

All tests now pass.

[sroussey]

@copilot rebase on copilot/add-getbulk-to-tabular-storage and add the new functionality for paging results, etc

[Copilot]

@copilot rebase on copilot/add-getbulk-to-tabular-storage and add the new functionality for paging results, etc

Rebased on copilot/add-getbulk-to-tabular-storage and implemented pagination functionality in commit 7f33f73:

getBulk Method:

• Fetches single page via /rows endpoint with offset and limit
• Caps limit at 100 (HuggingFace API max)
• Returns undefined when no records found

Pagination Support:

• records() - async generator yielding records one at a time
• pages() - async generator yielding pages of records
• Both inherited from BaseTabularStorage and use getBulk internally

Code Improvements:

• Refactored getAll() to use getBulk for better code reuse
• Added 5 comprehensive tests covering all pagination scenarios

All tests passing.

[Copilot]

Pull request overview

This PR implements a new readonly HuggingFaceTabularStorage class that provides access to HuggingFace datasets through the Dataset Viewer REST API, along with pagination support for all tabular storage implementations. The implementation follows established patterns in the codebase and includes comprehensive test coverage.

Changes:

• Adds HuggingFaceTabularStorage with auto-schema detection via fromDataset() factory method
• Implements getBulk(offset, limit) method across all 8 tabular storage implementations
• Adds records() and pages() async generators to BaseTabularStorage for efficient iteration
• Updates ITabularStorage interface with new pagination methods
• Provides comprehensive test coverage including HuggingFace-specific tests and generic iteration tests

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
packages/storage/src/tabular/HuggingFaceTabularStorage.ts	New readonly storage implementation for HuggingFace Dataset Viewer API with schema auto-detection, pagination, and proper escaping
packages/storage/src/tabular/tests/HuggingFaceTabularStorage.test.ts	Comprehensive tests covering schema detection, read operations, readonly errors, pagination, and API integration
packages/storage/src/tabular/ITabularStorage.ts	Adds getBulk(), records(), and pages() method signatures to interface
packages/storage/src/tabular/BaseTabularStorage.ts	Adds abstract getBulk() method and default implementations of records() and pages() async generators
packages/storage/src/tabular/InMemoryTabularStorage.ts	Implements getBulk() using array slicing
packages/storage/src/tabular/IndexedDbTabularStorage.ts	Implements getBulk() using cursor with advance for offset
packages/storage/src/tabular/SqliteTabularStorage.ts	Implements getBulk() using LIMIT/OFFSET SQL query
packages/storage/src/tabular/PostgresTabularStorage.ts	Implements getBulk() using LIMIT/OFFSET SQL query
packages/storage/src/tabular/SupabaseTabularStorage.ts	Implements getBulk() using range query
packages/storage/src/tabular/FsFolderTabularStorage.ts	Implements getBulk() using file array slicing
packages/storage/src/tabular/SharedInMemoryTabularStorage.ts	Delegates getBulk() to underlying in-memory repository
packages/storage/src/tabular/CachedTabularStorage.ts	Delegates getBulk() to cache for faster access
packages/storage/src/common.ts	Exports HuggingFaceTabularStorage and related types
packages/test/src/test/storage-tabular/genericTabularRepositoryTests.ts	Adds comprehensive iteration method tests for getBulk(), records(), and pages()

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

If all search key values are null or undefined, the WHERE clause will be an empty string. This could cause an API error or unexpected behavior when passed to the HuggingFace /filter endpoint. Consider adding a check after line 344 to handle the case where whereConditions is empty, either by returning undefined or throwing a more descriptive error.

Suggested change

	if (whereConditions.length === 0) {
	throw new Error(
	"Search criteria must include at least one non-null and non-undefined value to build a valid WHERE clause."
	);
	}

[Copilot]

Applied in commits 8b71ded and 17a34b9:

• Added validation to check if whereConditions is empty after building the WHERE clause
• Throws descriptive error when all search values are null/undefined
• Added test case to verify the error handling

[sroussey]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied the suggested change from the review thread in commits 8b71ded and 17a34b9.