Skip to content

fix: resolve typed-rule imports via go/packages export data - #1894

Open
danp wants to merge 6 commits into
revive-lint:masterfrom
danp:issue-1277-go-packages
Open

danp wants to merge 6 commits into
revive-lint:masterfrom
danp:issue-1277-go-packages

Conversation

@danp

@danp danp commented Oct 11, 2026

Copy link
Copy Markdown

Use packages.Load to resolve export data in the target module's build environment and importer.ForCompiler to load it for type checking.

Load dependencies lazily in batches and cache results across packages. This fixes module import resolution and supports -trimpath builds.

Add regression coverage for module and standard library imports.

Closes #1277

@danp

danp commented Oct 11, 2026

Copy link
Copy Markdown
Author

This is a an x/tools/go/packages alternative to #1893, cc @ccoVeille.

The benchmarks are the same as #1893 (comment).

I did run into #1711 during verification, and I've run into it in projects using revive / golangci-lint. Would sure be nice to get one of the open fix PRs for that merged.

@danp

danp commented Oct 14, 2026

Copy link
Copy Markdown
Author

@alexandear just rebased this to get #1898.

@danp
danp force-pushed the issue-1277-go-packages branch 2 times, most recently from 8223b53 to a3ad8cc Compare October 14, 2026 18:19
Comment thread lint/importer.go Resolved
Comment thread lint/importer.go Resolved
Comment thread lint/importer.go Resolved
Comment thread lint/importer.go Resolved
@danp

danp commented Oct 16, 2026

Copy link
Copy Markdown
Author

Tried out this PR along with #1892 locally on my internal codebase.

This PR enables finding many more unhandled-error findings, other things like time-equal, etc, as expected.

#1892 fixes some confusing-naming false positives where, say, package foo had X and foo_test had x.

Doing some more comparisons I am seeing bumps in time and memory. Checking those out.

@danp
danp force-pushed the issue-1277-go-packages branch from 9b4e0db to 4b174b3 Compare October 16, 2026 12:54
@danp

danp commented Oct 16, 2026 •

Copy link
Copy Markdown
Author

With the latest commit things now look like this for my internal codebase (145 packages, ~750 Go files and ~409k lines, and depends on ~880 non-stdlib packages).

Warm build cache:

Rules Build Wall time (median, range) user / sys CPU Peak memory (median)
all master 2.75s (2.63–2.85) 10.4s / 10.6s 956 MB
all branch 2.24s (2.15–2.35) 9.4s / 6.1s 994 MB
default master 2.08s (1.99–2.22) 6.9s / 10.7s 828 MB
default branch 1.68s (1.61–1.74) 6.0s / 6.1s 721 MB

Empty cache (all rules):

Build Wall time Peak memory
master 19.7–23.3s 0.84–0.90 GB
branch 50.3–52.6s ~1.0 GB

Extra time is from producing export data for more things, as expected.

(this comparison was done with #1892 merged in too for reproducibility)

@danp
danp requested a review from ccoVeille October 19, 2026 11:41
@danp
danp force-pushed the issue-1277-go-packages branch from 4b174b3 to 819fc78 Compare October 20, 2026 22:47
@danp

danp commented Oct 26, 2026

Copy link
Copy Markdown
Author

@alexandear @ccoVeille anything else I can do to get this going? (and #1892 as well)

@ccoVeille ccoVeille left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

I asked a review to Copilot and it brought this

I skimmed the diff but didn't run it. I found no hard bug, only these concerns, roughly in order of importance:

1. **Errors are cached permanently.** `exportFile` stores failures in `m.exports`, and `moduleExportsCache` lives for the whole process. A transient failure, such as a `go list` or build hiccup, sticks for every later lookup. For a long-lived consumer, such as an LSP or library use, stale data is a problem too. The export files are build-cache paths that can be evicted or rebuilt, and the cached paths are never refreshed.
2. **`packages.Load` runs while holding `m.mu`.** This is correct, and it gives you the batching, but every goroutine in the same module blocks behind a slow `go list` or compile. The fallback path does one `packages.Load` per missing import path under the lock, so a package with many unresolved imports serialises badly.
3. **`batchErr` reaches only one caller.** `loadPending` clears `m.pending` before loading. If the batch fails, only the first path that misses gets `batchErr` in its error. Later misses run their own fallback load and may report a less informative error.
4. **`byDir` and the module-root cache assume a stable layout.** Resolvers are keyed by module root, so a later `Lint` call with a different `GOFLAGS`, build tags or environment reuses the old export data.

I'd raise 1 as the main concern, and 2 as a performance question.

Comment thread lint/importer.go Resolved
danp added 6 commits November 13, 2026 06:42
Use packages.Load to resolve export data in the target module's build
environment and importer.ForCompiler to load it for type checking.

Load dependencies lazily in batches and cache results across packages.
This fixes module import resolution and supports -trimpath builds.

Add regression coverage for module and standard library imports.

Closes revive-lint#1277
@danp
danp force-pushed the issue-1277-go-packages branch from 819fc78 to 2459019 Compare November 13, 2026 10:57
@danp

danp commented Nov 13, 2026

Copy link
Copy Markdown
Author

Thanks! Changed that error flow. And changed things so each Lint call does its own caching instead of keeping things across calls.

@ccoVeille ccoVeille left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Typed rules don't work as expected

2 participants