feat(plugins): add BucketResolverPlugin for multi-bucket dataset queries - #51
Merged
Merged
Conversation
Resolves a missing bucket/endpoint per file by matching parsed file paths against configured dataset prefixes. Runs after QueryParserPlugin so each file in a query is resolved independently, letting a single query join across datasets that live in different buckets. Opt-in via the plugins array, consistent with FSPurgePlugin/StatsPlugin/AvroPlugin.
The "past day" cache fixture was hardcoded to day=01, colliding with today's own day-level prefix whenever the suite runs on the 1st of the month (UTC) and making the "past day entries should remain" assertion fail. Derive it relative to the actual current day instead.
list_files talks to S3 directly and never runs through the query pipeline that BucketResolverPlugin's processQuery hooks into, so it couldn't auto-resolve a bucket from datasets. Add a resolveListFiles hook to the plugin (sharing its prefix-matching logic with processQuery) and have ListFilesTool reduce over config.plugins to call it — kept local to the MCP tool rather than lifecycle.js, since that module is scoped to the s3quoia() query/download pipeline which list_files never touches. Adding BucketResolverPlugin to config.plugins now covers both the query and list_files tools with no extra configuration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BucketResolverPlugin, aprocessQuery-hook plugin that fills in a missingbucket/endpointper file by matching each parsed file path against configured datasetprefixentries. Resolution runs per file (not per query), so a single query can join across datasets that live in different buckets.s3quoia.jsalongsideFSPurgePlugin/StatsPlugin/AvroPlugin. Opt-in via theplugins.s3quoia()usage andS3QuoiaMCPusage).This potentially supersedes the approach discussed on #50, which did per query by substring-matching raw SQL.
Test plan
npm test— 154 unit tests pass, including newBucketResolverPluginunit testsnpm run test:e2e— 20 e2e tests pass against real MinIO, including two new tests: single-dataset bucket resolution with nodefaultBucket, and a join across two files in two different buckets resolved purely byBucketResolverPluginwith no bucket tokens in the SQLnpm run lintand prettier clean