Skip to content

chore: Adding support for vfs in cas#5908

Merged
yhakbar merged 12 commits intomainfrom
feat/adding-support-for-vfs-in-cas
Apr 16, 2026
Merged

chore: Adding support for vfs in cas#5908
yhakbar merged 12 commits intomainfrom
feat/adding-support-for-vfs-in-cas

Conversation

@yhakbar
Copy link
Copy Markdown
Collaborator

@yhakbar yhakbar commented Apr 15, 2026

Description

Adds support for using the vfs package in cas to abstract away filesystem access.

Significantly cuts down the time it takes to test the cas package, and makes it easier to extend functionality in the future.

Added support for a local Git HTTP server in git that we can use for integration with remote Git servers to reduce the overhead of cloning real repositories.

TODOs

Read the Gruntwork contribution guidelines.

  • I authored this code entirely myself
  • I am submitting code based on open source software (e.g. MIT, MPL-2.0, Apache)]
  • I am adding or upgrading a dependency or adapted code and confirm it has a compatible open source license
  • Update the docs.
  • Run the relevant tests successfully, including pre-commit checks.
  • Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Added / Removed / Updated [X].

Migration Guide

Summary by CodeRabbit

  • New Features

    • Configurable Git clone depth for content storage operations.
    • Virtual filesystem abstraction enabling alternative storage backends (in-memory, OS, etc.).
  • Improvements

    • Added a local Git test server to improve test reliability and reproducibility.
    • Refined lock handling to work across different filesystem implementations.

@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 15, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
terragrunt-docs Ready Ready Preview, Comment Apr 15, 2026 11:12pm

Request Review

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 15, 2026

📝 Walkthrough

Walkthrough

Refactored CAS to use a configurable vfs.FS and functional-options constructor, moved filesystem and locking operations to the VFS layer, added an in-memory HTTP Git test server, and updated tests to use memmap VFS and the local git server while adjusting clone options and locking APIs.

Changes

Cohort / File(s) Summary
CAS Core & API
internal/cas/cas.go, internal/cas/store.go
Switched to functional options (WithStorePath, WithFS), added CloneOptions.Depth, replaced os/flock usage with injected vfs.FS and vfs.Unlocker APIs.
VFS Implementation & Utilities
internal/vfs/vfs.go, internal/vfs/vfs_test.go
Introduced exported VFS helpers/interfaces (File, Locker, Unlocker, HardLinker), mem/OS FS implementations, deterministic WalkDir, lock primitives, and tests for walk/lock behavior.
CAS Content / Local / Tree / Getter
internal/cas/content.go, internal/cas/local.go, internal/cas/tree.go, internal/cas/getter.go
Replaced direct os.* calls with vfs operations; Content now holds vfs.FS, GetTmpHandle returns vfs.File, EnsureCopy added, directory walking uses vfs.WalkDir, and getter uses underlying CAS FS.
Git Test Server
internal/git/server.go, internal/git/server_test.go, internal/cas/testserver_test.go
Added in-memory HTTP Git server (Server, CommitFile, Start, Close) and tests; added startTestServer test helper to populate/start local repos for tests.
Tests Converted to VFS & Local Server
internal/cas/*.go tests (benchmark_test.go, cas_test.go, content_test.go, store_test.go, tree_test.go, integration_test.go, race_test.go, getter_test.go, getter_ssh_test.go, testserver_test.go)
Replaced OS temp dirs with vfs.NewMemMapFS() where applicable, switched tests to use startTestServer(...) repo URLs, unified CloneOptions with Depth: -1, updated assertions to use vfs read/stat APIs.
Git Runner / Application Call Sites
internal/cas/benchmark_test.go, internal/runner/run/download_source.go, internal/services/catalog/module/repo.go
Benchmarks/tests instantiate per-run Git runner; CAS initialization sites updated to use new cas.New(...options...) API.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

  • chore: Adding VFS testing #5490 — Modifies and extends the internal vfs filesystem abstraction and related tests; strongly related to the VFS additions and API changes in this PR.

Suggested reviewers

  • ThisGuyCodes
  • denis256
🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ❓ Inconclusive The description clearly explains the purpose and benefits, but the Release Notes section is incomplete with only placeholder text 'Added / Removed / Updated [X].' Complete the Release Notes section with a specific one-line description of the changes (e.g., 'Added vfs filesystem abstraction to cas package and local Git HTTP server for testing').
✅ Passed checks (1 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: adding vfs support to the cas package for filesystem abstraction.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/adding-support-for-vfs-in-cas

Comment @coderabbitai help to get the list of available commands and usage tips.

@yhakbar yhakbar marked this pull request as ready for review April 15, 2026 19:24
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
internal/cas/store_test.go (1)

159-168: ⚠️ Potential issue | 🟡 Minor

This lock-contention check is timing-sensitive.

The second goroutine starts its timer only after it gets scheduled past <-acquired, so on a busy runner it can block correctly and still observe <50ms. An explicit handshake/select that proves the second AcquireLock cannot finish before the first unlocks will be much less flaky.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@internal/cas/store_test.go` around lines 159 - 168, The timing-sensitive
check should be replaced with an explicit handshake so the test proves the
second AcquireLock is actually blocked; add a "started" channel that the second
goroutine closes/sends on immediately before calling
store.AcquireLock(testHash), have the main test wait for that started signal,
then use a select with a short timer to assert that the AcquireLock has not
returned (e.g. by waiting on a "done" channel the goroutine will send to after
AcquireLock) before releasing the first lock from the acquired channel;
reference store.AcquireLock, acquired, testHash and use the started/done
channels to deterministically verify blocking.
🧹 Nitpick comments (1)
internal/cas/integration_test.go (1)

82-94: Consider validating the specific error type for consistency.

The "clone with nonexistent branch" test (lines 63-80) validates the specific error type using ErrorAs and ErrorIs, but this test only checks that an error occurred. Based on context snippet 4 (internal/git/git.go:212-221), clone failures wrap ErrGitClone in a WrappedError. Validating the error type would ensure the test fails for the right reason and maintains consistency with the adjacent test case.

If this relaxation is intentional (e.g., the local test server returns different error types), a brief comment explaining the rationale would help future maintainers.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@internal/cas/integration_test.go` around lines 82 - 94, The test "clone with
invalid repository fails gracefully" only asserts an error but should validate
the concrete error to ensure it failed for the expected reason; update the test
that calls cas.Clone (in the t.Run block) to check that the returned error wraps
ErrGitClone (and/or is a cas.WrappedError) using errors.As or errors.Is similar
to the adjacent "clone with nonexistent branch" test—specifically, capture the
error from c.Clone and assert errors.As(err, &cas.WrappedError{}) or
errors.Is(err, cas.ErrGitClone); if the broader error type is intentionally
allowed, add a short comment in the t.Run describing why the type check is
relaxed.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@internal/cas/content.go`:
- Around line 162-207: EnsureCopy currently mis-joins errors and closes the
wrong file handle: update the unlock defer to include the existing named error
(use errors.Join(err, lock.Unlock()) so any prior error isn't discarded) and fix
the source/destination close defer to close the source reader (replace the
second defer that calls f.Close() with one that calls r.Close(), while keeping
the earlier defer that joins f.Close() into err). Locate these changes in the
EnsureCopy function around the lock.Unlock and the two defer blocks for
f.Close/r.Close.

In `@internal/vfs/vfs.go`:
- Around line 87-90: The WriteFile wrapper currently calls afero.WriteFile (in
function WriteFile(fs FS, filename string, data []byte, perm os.FileMode)) which
relies on backend behavior and causes MemMapFs to auto-create parent dirs while
OsFs errors; modify WriteFile to ensure consistent behavior by creating parent
directories before writing: compute the parent directory of filename and call
fs.MkdirAll (or os.MkdirAll via afero if needed) with appropriate permissions,
handle and return any MkdirAll error, then call afero.WriteFile to write the
file; keep function signature WriteFile(fs FS, filename string, data []byte,
perm os.FileMode) and ensure errors from MkdirAll and WriteFile are propagated.

---

Outside diff comments:
In `@internal/cas/store_test.go`:
- Around line 159-168: The timing-sensitive check should be replaced with an
explicit handshake so the test proves the second AcquireLock is actually
blocked; add a "started" channel that the second goroutine closes/sends on
immediately before calling store.AcquireLock(testHash), have the main test wait
for that started signal, then use a select with a short timer to assert that the
AcquireLock has not returned (e.g. by waiting on a "done" channel the goroutine
will send to after AcquireLock) before releasing the first lock from the
acquired channel; reference store.AcquireLock, acquired, testHash and use the
started/done channels to deterministically verify blocking.

---

Nitpick comments:
In `@internal/cas/integration_test.go`:
- Around line 82-94: The test "clone with invalid repository fails gracefully"
only asserts an error but should validate the concrete error to ensure it failed
for the expected reason; update the test that calls cas.Clone (in the t.Run
block) to check that the returned error wraps ErrGitClone (and/or is a
cas.WrappedError) using errors.As or errors.Is similar to the adjacent "clone
with nonexistent branch" test—specifically, capture the error from c.Clone and
assert errors.As(err, &cas.WrappedError{}) or errors.Is(err, cas.ErrGitClone);
if the broader error type is intentionally allowed, add a short comment in the
t.Run describing why the type check is relaxed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 974262a3-5a7d-4b3f-9e47-48c9a547050c

📥 Commits

Reviewing files that changed from the base of the PR and between d02d73f and c99add2.

📒 Files selected for processing (22)
  • internal/cas/benchmark_test.go
  • internal/cas/cas.go
  • internal/cas/cas_test.go
  • internal/cas/content.go
  • internal/cas/content_test.go
  • internal/cas/getter.go
  • internal/cas/getter_ssh_test.go
  • internal/cas/getter_test.go
  • internal/cas/integration_test.go
  • internal/cas/local.go
  • internal/cas/race_test.go
  • internal/cas/store.go
  • internal/cas/store_test.go
  • internal/cas/testserver_test.go
  • internal/cas/tree.go
  • internal/cas/tree_test.go
  • internal/git/server.go
  • internal/git/server_test.go
  • internal/runner/run/download_source.go
  • internal/services/catalog/module/repo.go
  • internal/vfs/vfs.go
  • internal/vfs/vfs_test.go

Comment thread internal/cas/content.go
Comment thread internal/vfs/vfs.go
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
internal/vfs/vfs.go (1)

551-586: Consider reducing cognitive complexity (static analysis).

Static analysis flags this function with cognitive complexity 18 vs. the allowed 15. The complexity stems from the nested error handling for SkipDir/SkipAll semantics.

Given the comment notes this is adapted from the standard library and is temporary until spf13/afero#571 is merged, this is acceptable. However, if the afero merge is delayed, consider extracting the SkipDir check into a helper:

♻️ Optional refactor
+// shouldSkipError returns nil if the error should be skipped for a directory entry.
+func shouldSkipError(err error, isDir bool) error {
+	if errors.Is(err, filepath.SkipDir) && isDir {
+		return nil
+	}
+	return err
+}
+
 func walkDir(fsys FS, path string, d fs.DirEntry, walkDirFn fs.WalkDirFunc) error {
 	if err := walkDirFn(path, d, nil); err != nil || !d.IsDir() {
-		if errors.Is(err, filepath.SkipDir) && d.IsDir() {
-			err = nil
-		}
-
-		return err
+		return shouldSkipError(err, d.IsDir())
 	}
 
 	entries, err := readDirEntries(fsys, path)
 	if err != nil {
 		err = walkDirFn(path, d, err)
 		if err != nil {
-			if errors.Is(err, filepath.SkipDir) && d.IsDir() {
-				err = nil
-			}
-
-			return err
+			return shouldSkipError(err, d.IsDir())
 		}
 	}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@internal/vfs/vfs.go` around lines 551 - 586, The walkDir function's nested
error handling increases cognitive complexity; extract the repeated SkipDir
handling into a small helper function (e.g., handleSkipDir(err error, d
fs.DirEntry) error) and replace the three identical blocks that check
errors.Is(err, filepath.SkipDir) && d.IsDir() with calls to that helper; update
references in walkDir where walkDirFn is invoked and where errors from recursive
walkDir calls are inspected so the overall control flow remains identical but
the duplicated SkipDir logic is centralized.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@internal/vfs/vfs.go`:
- Around line 551-586: The walkDir function's nested error handling increases
cognitive complexity; extract the repeated SkipDir handling into a small helper
function (e.g., handleSkipDir(err error, d fs.DirEntry) error) and replace the
three identical blocks that check errors.Is(err, filepath.SkipDir) && d.IsDir()
with calls to that helper; update references in walkDir where walkDirFn is
invoked and where errors from recursive walkDir calls are inspected so the
overall control flow remains identical but the duplicated SkipDir logic is
centralized.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 83fdec72-6341-47ff-8e53-9b7695374af2

📥 Commits

Reviewing files that changed from the base of the PR and between c99add2 and e21b1fd.

📒 Files selected for processing (2)
  • internal/cas/content.go
  • internal/vfs/vfs.go

Comment thread internal/vfs/vfs.go
afero.Fs
symlinks map[string]string
locks map[string]*memLock
locksMu sync.Mutex
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, no mutex to protect symlinks 🤔

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we can add one, but we don't currently test anything that tries to do concurrent symlink creation or anything.

@yhakbar yhakbar merged commit ee5b5bc into main Apr 16, 2026
82 of 84 checks passed
@yhakbar yhakbar deleted the feat/adding-support-for-vfs-in-cas branch April 16, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants