Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -574,7 +574,7 @@ After `generate_snapshot`, these are written to the output directory (default `.
| `facts.jsonl` | Every extracted fact, one JSON object per line |
| `insights.json` | Architectural insights with confidence scores |
| `snapshot.meta.json` | Metadata including per-file content hashes for incremental updates, plus the full receipt fields |
| `receipt.json` | The **snapshot receipt** — a compact manifest of what the graph was generated over (enola version, git ref + dirty status, a content-fingerprint snapshot ID, the extractor/explainer sets, ignore-glob hash, output-artifact hashes) and extraction-quality metrics (files seen/parsed/skipped, parse errors, coverage gaps). Read it via the `snapshot_receipt` tool. |
| `receipt.json` | The **snapshot receipt** — a compact manifest of what the graph was generated over (enola version, git ref + dirty status, a content-fingerprint snapshot ID, the extractor/explainer sets, ignore-glob hash, output-artifact hashes) and extraction-quality metrics (files seen/parsed/skipped, directory trees pruned, parse errors, coverage gaps). Read it via the `snapshot_receipt` tool. |
| `previous/` | The immediately-preceding snapshot, auto-rotated on each write — the `baseline='previous'` source for `diff_snapshot` |
| `baseline/` | A snapshot pinned by `set_baseline`, preserved across re-snapshots — the default `diff_snapshot` baseline |

Expand All @@ -583,7 +583,9 @@ After `generate_snapshot`, these are written to the output directory (default `.
`receipt.json` (and the same fields inside `snapshot.meta.json`) exists to answer *"what was this graph deterministic over, and how complete is it?"* — the trust question before an agent relies on an `impact_analysis` or a `diff_snapshot`. It serves two consumers:

- **Provenance / audit.** enola version, git ref + dirty-tree status, the extractor/explainer sets actually used, a **config hash** (over the effective extractors/explainers/renderers/globs/output settings) and its narrower `ignore_glob_hash`, per-artifact output hashes, and a **snapshot ID** that is a *content fingerprint* (SHA-256 over the byte-stable fact serialization plus the version and config hash), not a random UUID — so re-running on identical inputs yields the same ID and it can key equivalence. Every hash value carries a `sha256:` prefix.
- **The improvement loop.** Extraction-quality metrics — files seen vs. parsed vs. skipped, a parse-error count and sample, the count of heuristic (confidence < 1.0) insights, and the cross-repo coverage-gap / unresolved-edge rollup — give a machine-readable signal a consumer (a human, a `diff_snapshot`, or an agent improving enola itself) can poll to detect *thin extraction* (a missing detection, a bad ignore glob, a failing extractor) and turn it into targeted work. The same metrics appear as an **Extraction Quality** section in `llm_context.md`, so an agent reading the snapshot sees thin extraction without a tool call.
- **The improvement loop.** Extraction-quality metrics — files seen vs. parsed vs. skipped, the number of directory trees pruned, a parse-error count and sample, the count of heuristic (confidence < 1.0) insights, and the cross-repo coverage-gap / unresolved-edge rollup — give a machine-readable signal a consumer (a human, a `diff_snapshot`, or an agent improving enola itself) can poll to detect *thin extraction* (a missing detection, a bad ignore glob, a failing extractor) and turn it into targeted work. The same metrics appear as an **Extraction Quality** section in `llm_context.md`, so an agent reading the snapshot sees thin extraction without a tool call.

The two skip counters mean different things, and a bad ignore glob is usually a *directory* glob. `files_skipped` counts ignored files the walker **visited** and dropped — those matched by a file glob like `**/*.test.ts`. An ignored directory is pruned whole (`filepath.SkipDir`), so its contents are never visited and appear in no count: it is tallied once, as one entry in `dirs_skipped`. Each `skipped_sample` entry names the glob that matched it, so *"why is this file missing from the graph?"* is a lookup rather than an investigation; directories appear there with a trailing slash.

Because the receipt fields live in `snapshot.meta.json`, they ride into every pinned/`previous` baseline, and `diff_snapshot` reads them to add a **comparability guard**: it warns (above the delta) when the baseline and current snapshots were *not* generated over equivalent inputs — a different repo, enola version, extractor set, or ignore-glob set — since a diff across a mismatched extractor set would report every one of that language's facts as spurious churn. `compare_receipts` surfaces the same verdict plus the metric deltas directly.

Expand Down
1 change: 1 addition & 0 deletions internal/diff/receipt.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ func CompareReceipts(base, cur facts.SnapshotMeta) *ReceiptComparison {
delta("files_seen", base.FilesSeen, cur.FilesSeen),
delta("files_parsed", base.FilesParsed, cur.FilesParsed),
delta("files_skipped", base.FilesSkipped, cur.FilesSkipped),
delta("dirs_skipped", base.DirsSkipped, cur.DirsSkipped),
delta("parse_errors", base.ParseErrors, cur.ParseErrors),
delta("coverage_gaps", baseGaps, curGaps),
delta("unresolved_edges", baseUnresolved, curUnresolved),
Expand Down
27 changes: 27 additions & 0 deletions internal/diff/receipt_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -114,4 +114,31 @@ func TestCompareReceipts(t *testing.T) {
t.Error("expected Identical=true for matching snapshot_id")
}
})

// A newly-ignored directory prunes a whole subtree. It reports as a dirs_skipped
// delta and NOT as a quality regression: pruning vendor/ is usually the operator
// doing the right thing, and files_seen falling is the signal that costs something.
t.Run("a pruned directory surfaces as a dirs_skipped delta", func(t *testing.T) {
cur := base
cur.SnapshotID = "bbb"
cur.DirsSkipped = 2

rc := CompareReceipts(base, cur)

var found *MetricDelta
for i := range rc.Deltas {
if rc.Deltas[i].Name == "dirs_skipped" {
found = &rc.Deltas[i]
}
}
if found == nil {
t.Fatalf("no dirs_skipped delta; got %v", rc.Deltas)
}
if found.Before != 0 || found.After != 2 || found.Delta != 2 {
t.Errorf("dirs_skipped delta = %+v, want before=0 after=2 delta=2", *found)
}
if len(rc.QualityRegressions) != 0 {
t.Errorf("pruning a directory is not a regression, got: %v", rc.QualityRegressions)
}
})
}
77 changes: 57 additions & 20 deletions internal/engine/engine.go
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,7 @@ func (e *Engine) GenerateSnapshot(ctx context.Context, repoPath string, appendMo
FilesSeen: len(files),
FilesParsed: e.store.CountFilesWithFacts(files, parsedPrefix),
FilesSkipped: skips.count,
DirsSkipped: skips.dirCount,
SkippedSample: skips.sample,
IgnoreGlobHash: ignoreGlobHash,
ParseErrors: len(parseErrs),
Expand Down Expand Up @@ -442,12 +443,29 @@ func (e *Engine) flagUnmatchedRoutes() {
}
}

// walkSkips is a lightweight tally of files dropped by the ignore globs, kept so
// a snapshot receipt can report how much of the tree was excluded (and a sample
// of what) without retaining every skipped path.
// walkSkips is a lightweight tally of what the ignore globs dropped, kept so a
// snapshot receipt can report how much of the tree was excluded (and a sample of
// what) without retaining every skipped path.
//
// Files and directories are tallied separately because they cost differently to
// know. An ignored directory is pruned whole — the walker never descends, so its
// contents are counted nowhere. Counting them would mean walking node_modules/
// purely to size it, a stat per file for a number no architecture graph wants.
// One pruned directory is one architecturally meaningful fact; its 55,041 files
// are not.
type walkSkips struct {
count int
sample []string
count int // ignored FILES the walker visited
dirCount int // ignored DIRECTORIES pruned; their contents are never visited
sample []string // capped sample of both, each annotated with the glob that matched
}

// record appends "<path> (glob: <pattern>)" until the sample is full. Directories
// arrive with a trailing slash, which is what distinguishes a pruned subtree from
// a single dropped file in the receipt.
func (s *walkSkips) record(path, pattern string) {
if len(s.sample) < skippedSampleCap {
s.sample = append(s.sample, path+" (glob: "+pattern+")")
}
}

// skippedSampleCap bounds the number of skipped paths retained for the receipt.
Expand All @@ -456,8 +474,8 @@ const skippedSampleCap = 20
// walkRepo collects all files in the repo, applying ignore patterns. It returns
// the indexable source files, separately the test/spec files matched by
// TestGlobs (excluded from normal indexing but collected for reference-only
// extraction — see runTestRefExtractors), and a tally of ignored files for the
// snapshot receipt.
// extraction — see runTestRefExtractors), and a tally of what the ignore globs
// dropped — ignored files, and pruned directories — for the snapshot receipt.
func (e *Engine) walkRepo(repoPath string) (files, testFiles []string, skips walkSkips, err error) {
err = filepath.WalkDir(repoPath, func(path string, d fs.DirEntry, err error) error {
if err != nil {
Expand All @@ -470,8 +488,15 @@ func (e *Engine) walkRepo(repoPath string) (files, testFiles []string, skips wal
}

// Skip ignored paths
if e.isIgnored(relPath, d.IsDir()) {
if pattern, ok := e.ignoreMatch(relPath); ok {
if d.IsDir() {
// enola's own output directory is not part of the source tree.
// Counting it would make dirs_skipped differ between a repo's
// first-ever snapshot and every one after it, for no signal.
if relPath != e.cfg.Output.Dir {
skips.dirCount++
skips.record(filepath.ToSlash(relPath)+"/", pattern)
}
return filepath.SkipDir
}
// An ignored FILE that is a test/spec is not indexed as production
Expand All @@ -481,9 +506,7 @@ func (e *Engine) walkRepo(repoPath string) (files, testFiles []string, skips wal
testFiles = append(testFiles, relPath)
}
skips.count++
if len(skips.sample) < skippedSampleCap {
skips.sample = append(skips.sample, filepath.ToSlash(relPath))
}
skips.record(filepath.ToSlash(relPath), pattern)
return nil
}

Expand All @@ -503,7 +526,15 @@ func (e *Engine) matchesTestGlob(relPath string) bool {
// matchAnyGlob reports whether a forward-slash path matches any of the patterns.
// It is the single matcher behind both the ignore list and the test globs, so a
// file the two lists disagree about cannot exist: an ignored file that stops being
// a test necessarily stops being ignored. Supported forms:
// a test necessarily stops being ignored.
func matchAnyGlob(relPath string, patterns []string) bool {
_, ok := matchGlob(relPath, patterns)
return ok
}

// matchGlob returns the first pattern that matches relPath. The receipt records it
// beside the skipped path, so "why is this file missing from the graph?" is a
// lookup rather than an investigation. Supported forms:
//
// vendor/** anchored directory prefix
// **/build/** a directory named "build" at any depth
Expand All @@ -512,7 +543,7 @@ func (e *Engine) matchesTestGlob(relPath string) bool {
//
// The last form is the only one that constrains directory and filename together;
// see matchDirScopedGlob for why the Ruby test globs need it.
func matchAnyGlob(relPath string, patterns []string) bool {
func matchGlob(relPath string, patterns []string) (string, bool) {
for _, pattern := range patterns {
// "<prefix>/**/<fileglob>". Handled first and exclusively: the branches
// below would match such a pattern only when exactly one directory sits
Expand All @@ -522,7 +553,7 @@ func matchAnyGlob(relPath string, patterns []string) bool {
prefix, fileGlob := pattern[:i], pattern[i+len("/**/"):]
if !strings.Contains(fileGlob, "/") {
if matchDirScopedGlob(relPath, prefix, fileGlob) {
return true
return pattern, true
}
continue
}
Expand All @@ -532,31 +563,31 @@ func matchAnyGlob(relPath string, patterns []string) bool {
if seg != "" && !strings.Contains(seg, "/") {
for _, part := range strings.Split(relPath, "/") {
if part == seg {
return true
return pattern, true
}
}
}
}
if strings.HasSuffix(pattern, "/**") {
dirPrefix := strings.TrimSuffix(pattern, "/**")
if relPath == dirPrefix || strings.HasPrefix(relPath, dirPrefix+"/") {
return true
return pattern, true
}
}
if m, err := filepath.Match(pattern, relPath); err == nil && m {
return true
return pattern, true
}
if strings.HasPrefix(pattern, "**/") {
sub := strings.TrimPrefix(pattern, "**/")
if m, err := filepath.Match(sub, filepath.Base(relPath)); err == nil && m {
return true
return pattern, true
}
if m, err := filepath.Match(sub, relPath); err == nil && m {
return true
return pattern, true
}
}
}
return false
return "", false
}

// matchDirScopedGlob reports whether relPath's basename matches fileGlob AND
Expand Down Expand Up @@ -602,6 +633,12 @@ func (e *Engine) isIgnored(relPath string, isDir bool) bool {
return matchAnyGlob(filepath.ToSlash(relPath), e.cfg.Ignore)
}

// ignoreMatch reports whether a path is ignored, and by which pattern. The walker
// needs the pattern to record it in the receipt's skipped sample.
func (e *Engine) ignoreMatch(relPath string) (string, bool) {
return matchGlob(filepath.ToSlash(relPath), e.cfg.Ignore)
}

// runExtractors detects applicable extractors and runs them. When cache is
// non-nil, extractors implementing plugin.FileOwner have their facts reused
// whenever the files they depend on are unchanged since the last snapshot.
Expand Down
Loading
Loading