Add streaming S3 import command with metadata identity and path filter support by Copilot · Pull Request #8 · londogard/reflake

Copilot · 2026-03-15T20:06:48Z

This adds an import command for bootstrapping a repository from S3 without materializing the full object list in memory. It supports both canonical blob-backed imports and metadata-only imports so large datasets can be registered first and verified later.

CLI surface
- Adds fluxel import <s3-uri> -m <message>
- Supports --identity blake3 and --identity meta
- Supports repeatable --path filters for exact logical paths or glob patterns such as * and **/*.jpg
- Keeps commit semantics aligned with the existing branch/ref flow
Streaming S3 ingest
- Iterates list_objects_v2 incrementally instead of collecting the full listing
- Streams object bytes directly when computing Blake3 and writing canonical blobs
- Applies optional path/glob filters while streaming, without building a full in-memory object list
- Writes manifest entries incrementally through the existing JSONL pipeline
Metadata-only import path
- For --identity meta, records logical path/size identity without reading object payloads
- Persists source_uri as s3://... so imported entries remain readable before promotion
- Allows the existing verify flow to promote imported metadata entries into canonical blobs later
Source URI handling
- Generalizes source reads so both local file URIs and S3 URIs work through the same repository/filesystem paths
- Enables fluxel://... reads for metadata-only entries imported from S3
Targeted correctness updates
- Fixes CLI dataclass field ordering issues that were preventing command parsing/import at module load time
- Adds focused CLI coverage for:
  - blob-backed S3 import
  - metadata-only S3 import
  - reading imported metadata entries
  - verifying imported metadata entries into canonical blobs
  - wildcard and repeated path-filtered S3 imports

Example:

fluxel import --root /data/repo s3://my-bucket/bootstrap -m "bootstrap import"
fluxel import --root /data/repo s3://my-bucket/bootstrap -m "metadata import" --identity meta
fluxel import --root /data/repo s3://my-bucket/bootstrap -m "jpg subset" --path "**/*.jpg" --path root.csv
fluxel verify --root /data/repo --ref main

Original prompt

This section details on the original issue you should resolve

<issue_title>S3 import</issue_title>
<issue_description>Best next feature for your use case: implement an import command that streams S3 listing + streams object bytes to compute Blake3 + writes blobs/manifests incrementally (no full in-memory list).

Also allow to do this import using the metadata identity mode.</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes S3 import #5

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: Lundez <7490199+Lundez@users.noreply.github.com>

Lundez · 2026-03-15T20:23:32Z

@copilot will this be able to import a "" path, "**/.jpg" (example filtetype) path or list of paths? If not, can we add support for that?

Co-authored-by: Lundez <7490199+Lundez@users.noreply.github.com>

Copilot · 2026-03-15T20:29:58Z

@copilot will this be able to import a "" path, "**/.jpg" (example filtetype) path or list of paths? If not, can we add support for that?

Yes — it now supports repeatable --path filters on fluxel import, so you can target exact paths or glob subsets like --path '*', --path '**/*.jpg', and multiple filters together. The support was added in d592511.

Copilot

Pull request overview

This PR adds a streaming fluxel import command to bootstrap a repository from S3 (either blob-backed via blake3 or metadata-only via meta), and refactors core storage/repository plumbing to support S3-backed repositories with local per-client state, manifest sidecar indexes, and optimistic ref updates.

Changes:

Add fluxel import with repeatable --path filters and --identity blake3|meta, streaming S3 listing/reads.
Introduce repository-store + client-state split (LocalRepositoryStore/S3RepositoryStore, LocalClientState) plus manifest sidecar indexes for point/prefix lookups.
Expand CLI + tests + docs/CI to cover remote repo operations, metadata-only workflows, and S3 integration via Ministack.

Reviewed changes

Copilot reviewed 29 out of 37 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_s3_integration.py	Adds real S3 integration coverage for branch/merge/import/verify and conflict handling.
tests/test_mandatory_validation.py	Adds regression tests for metadata-only operations, manifest validation, sidecar index behavior, and conflict messaging.
tests/test_cli.py	Extends CLI tests for `--repo` semantics, S3 repo flows, `import`, `mv/rm -m`, staged reads, and error reporting.
tests/conftest.py	Adds fake S3 client + Ministack fixtures for unit + integration testing.
src/fluxel/core/storage.py	Adds S3 URI parsing, streaming list/head/open helpers, and source URI metadata helpers.
src/fluxel/core/repository_support.py	Adds path normalization, import pattern matching, and meta-identity helpers used across repo ops.
src/fluxel/core/repository_store.py	Introduces `RepositoryStore` abstraction with local + S3 implementations and S3 branch locking/CAS updates.
src/fluxel/core/repository.py	Refactors repository to use stores + client state; adds `import_s3`, `merge`, metadata-only `remove_paths`/`move`, and sidecar-backed lookups.
src/fluxel/core/manifest_index.py	Adds sidecar index format + lookup/iteration helpers to avoid full manifest scans.
src/fluxel/core/manifest.py	Changes manifest JSONL encoding to tagged arrays with stricter validation and line-aware errors; makes file walking deterministic.
src/fluxel/core/layout.py	Refactors layout initialization into a classmethod.
src/fluxel/core/index.py	Switches analytical index build to stream entries via repo store and load via CSV into DuckDB.
src/fluxel/core/filesystem.py	Uses store-backed blob reads + source-uri reads; adds repo caching; uses sidecar lookups for point/prefix reads.
src/fluxel/core/client_state.py	Adds local per-client state (HEAD, staging, branch snapshots) with atomic writes.
src/fluxel/core/init.py	Exposes new APIs/types (stores, client state, import/merge/move/remove, S3 helpers).
src/fluxel/core/SPEC.md	Adds placeholder spec doc.
src/fluxel/cli.py	Adds `import`, `merge`, `mv`, metadata-only `rm -m`; standardizes `--repo`; improves error handling and repeated `--path` parsing.
src/fluxel/init.py	Re-exports new public API and CLI entrypoints.
src/fluxel/.DS_Store	Adds a macOS junk file (should be removed).
src/.DS_Store	Adds a macOS junk file (should be removed).
scripts/run_s3_integration.sh	Adds script to run Ministack-backed integration tests locally/CI.
pyproject.toml	Updates project metadata, license, URLs, and pytest markers.
__MACOSX/tests/._pycache	Adds macOS archive junk (should be removed).
__MACOSX/src/fluxel/core/._pycache	Adds macOS archive junk (should be removed).
__MACOSX/src/fluxel/._pycache	Adds macOS archive junk (should be removed).
__MACOSX/src/fluxel/._.DS_Store	Adds macOS archive junk (should be removed).
__MACOSX/src/._.DS_Store	Adds macOS archive junk (should be removed).
ROADMAP.md	Adds project roadmap describing the architecture and planned phases.
README.md	Documents new `--repo` semantics, import/merge/mv/rm workflows, integration tests, and licensing notes.
NOTICE	Adds attribution + sponsorship notice aligned with AGPL distribution.
LICENSE	Adds AGPL-3.0-or-later license text.
ISSUES.MD	Updates CLI examples to use `--repo`.
CONTRIBUTING.md	Adds contributor guidance and test instructions.
CHANGELOG.md	Adds changelog entries for new import/index/store/client-state capabilities.
.gitignore	Ignores macOS junk artifacts going forward.
.github/workflows/ci.yml	Adds CI (unit + S3 integration via script) and rejects macOS junk files.
.github/FUNDING.yml	Adds GitHub Sponsors configuration.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        with NamedTemporaryFile(mode="wb", suffix=".idx", delete=False) as temp:
+            temp_path = Path(temp.name)
+            temp.write(response["Body"].read())
+        self._manifest_index_cache[manifest_hash] = temp_path
+        return temp_path


+def _matches_logical_prefix(path: str, logical_prefix: str) -> bool:
+    return path == logical_prefix or path.startswith(f"{logical_prefix}/")
+
+


+        metadata = describe_source_uri(source_uri)
+        if identity_mode == "blake3":
+            identity_value = self._store_blob_from_source_uri(source_uri)
+            blob_hash = identity_value if store_blob else None
+        elif identity_mode == "meta":


+            raise ValueError(f"S3 source cannot be bucket root: {raw_source}")
+
+        objects = list(iter_s3_objects(raw_source))
+        if not objects:
+            raise FileNotFoundError(f"Cannot stage missing S3 path: {raw_source}")


Initial plan

8a7346b

Copilot AI assigned Copilot and Lundez Mar 15, 2026

Copilot started work on behalf of Lundez March 15, 2026 20:06 View session

feat: add streaming s3 import command

d19763d

Co-authored-by: Lundez <7490199+Lundez@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add S3 import command for streaming objects~~ Add streaming S3 import command with metadata identity support Mar 15, 2026

Copilot AI requested a review from Lundez March 15, 2026 20:15

Copilot finished work on behalf of Lundez March 15, 2026 20:15

Copilot started work on behalf of Lundez March 15, 2026 20:23 View session

Copilot AI and others added 4 commits March 15, 2026 20:26

feat: support filtered s3 imports

d592511

Co-authored-by: Lundez <7490199+Lundez@users.noreply.github.com>

refactor: tighten import path filter handling

544190f

Co-authored-by: Lundez <7490199+Lundez@users.noreply.github.com>

refactor: clarify import path filter names

962fb55

Co-authored-by: Lundez <7490199+Lundez@users.noreply.github.com>

docs: clarify repeated import path parsing

f3c87ee

Co-authored-by: Lundez <7490199+Lundez@users.noreply.github.com>

Copilot AI changed the title ~~Add streaming S3 import command with metadata identity support~~ Add streaming S3 import command with metadata identity and path filter support Mar 15, 2026

Copilot finished work on behalf of Lundez March 15, 2026 20:30

Lundez added 13 commits March 22, 2026 15:30

fix: add relevant test

10d8bbe

fix: Added updates

843d061

fix: phase 1

4cf0d55

fix: phase 2

ad60f4c

fix: phase 3 (no stale-lock recovery if a client dies mid-update)

64f4f9b

fix: phase 4

3a6b860

fix: phase 5

6bf9ba4

fix: phase 6,7

151993e

fix: minor convenience

b21990e

fix: shape up for launch

0891057

fix: shape up for launch

84fcd8a

fix: minor fix

cc376b6

fix: minor fix

b4c63c7

Lundez added 3 commits April 10, 2026 11:28

fix: update README to support Ministack more

441f793

fix: s3 tests

332bf32

fix: some tweaks

68b9bee

Lundez marked this pull request as ready for review April 16, 2026 09:02

Copilot AI review requested due to automatic review settings April 16, 2026 09:02

Copilot started reviewing on behalf of Lundez April 16, 2026 09:03 View session

Copilot AI reviewed Apr 16, 2026

View reviewed changes

fix: finalize tests

e5b85e8

Lundez merged commit 92f62d4 into main Apr 28, 2026
4 checks passed

Navigation Menu

Uh oh!

Add streaming S3 import command with metadata identity and path filter support - #8

Add streaming S3 import command with metadata identity and path filter support#8
Lundez merged 23 commits into
mainfrom
copilot/implement-s3-import-command

Copilot AI commented Mar 15, 2026 •

edited

Loading

Uh oh!

Lundez commented Mar 15, 2026

Uh oh!

Copilot AI commented Mar 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		def _matches_logical_prefix(path: str, logical_prefix: str) -> bool:
		return path == logical_prefix or path.startswith(f"{logical_prefix}/")

Uh oh!

Conversation

Copilot AI commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comments on the Issue (you are @copilot in this section)

Uh oh!

Lundez commented Mar 15, 2026

Uh oh!

Copilot AI commented Mar 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Mar 15, 2026 •

edited

Loading