[generator] add recursive component discovery for GitHub repositories

[ShigrafS]

Closes #726

Description

This PR implements recursive component discovery for Meshery’s generator logic, addressing the limitation where only files directly under a specified directory were processed. With this enhancement:

• Users can point to any root directory in a repository, and Meshery will automatically discover all component definition files in subdirectories.

• Recursive search is enabled by default, with configurable options for:

  • Maximum recursion depth (MaxDepth)
  • File pattern filtering (*.yaml, *.yml, *.json)
  • Enabling/disabling recursion

• Backward compatibility is maintained for existing workflows.

• Relevant CLI commands and API endpoints have been updated to support these options.

Changes include updates to:

• generators/generator.go & generator_test.go – Added GeneratorOptions for recursion and depth.
• generators/github/git_repo.go & package_manager.go – GitRepo and GitHubPackageManager updated to handle recursive directory traversal.
• generators/github/scheme_interface.go – Downloader updated to pass recursive options.
• utils/walker/git.go – File walker now respects recursion and max depth.
• Added new unit tests for recursive logic (recursive_test.go).

This fixes the configuration fragility for repositories with deeply nested component files and aligns with the desired behavior in #726 .

Notes for Reviewers

• The recursive logic uses filepath.WalkDir with max depth checks in utils/walker/git.go.
• Tests focus on option propagation and recursive field behavior; actual git clone testing may require integration testing.
• Backward compatibility with non-recursive workflows is preserved.
• Please verify the MaxDepth logic and that recursion is disabled when Recursive: false.

[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)

• Yes, I signed my commits.

---

[welcome]

Yay, your first pull request! 👍 A contributor will be by to give feedback soon. In the meantime, you can find updates in the #github-notifications channel in the community Slack.
Be sure to double-check that you have signed your commits. Here are instructions for making signing an implicit activity while performing a commit.

[gemini-code-assist]

Summary of Changes

Hello @ShigrafS, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves Meshery's ability to process component definitions from GitHub repositories by implementing recursive directory traversal. Users can now specify a root directory, and the system will automatically discover and process relevant files nested within subdirectories, overcoming previous limitations with deeply structured repositories. The new functionality is highly configurable, allowing fine-grained control over the depth of the recursive search, and has been integrated seamlessly while maintaining compatibility with existing operations.

Highlights

• Recursive Component Discovery: Introduced the capability for Meshery's generator logic to recursively discover component definition files within GitHub repositories, addressing the limitation of only processing files directly under a specified directory.
• Configurable Recursion Options: Added configurable options for recursive search, including a boolean flag to enable/disable recursion and an integer for maximum recursion depth (MaxDepth).
• Option Propagation and Integration: Updated the generator and github packages to correctly propagate the new Recursive and MaxDepth options through the system, from the initial generator creation down to the GitRepo and the underlying git file walker.
• Enhanced File Walker: The git file walker (utils/walker/git.go) has been enhanced to respect the MaxDepth setting during recursive directory traversal, using filepath.SkipDir to prune directories that exceed the specified depth.
• Backward Compatibility and Testing: Ensured backward compatibility for existing non-recursive workflows and added new unit tests to validate the correct propagation and behavior of the new recursion options.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces recursive component discovery for GitHub repositories, a valuable enhancement. The implementation adds options for recursion and depth control, and updates relevant parts of the generator logic. My review has identified a couple of high-severity issues: one is a bug in the recursion handling logic that can cause incorrect behavior, and the other is an off-by-one error in the max depth calculation which leads to counter-intuitive results. Additionally, the new tests for the recursive functionality are currently quite minimal and should be expanded to ensure the feature is robust. Overall, the changes are in the right direction, but these key issues should be addressed before merging.

[Copilot]

Pull request overview

Adds configurable recursive component discovery for GitHub-based generators by propagating recursion/depth/extension options down into the git walker, and introducing tests around recursive traversal behavior.

Changes:

• Introduces GeneratorOptions and threads recursive options into GitHubPackageManager/downloaders.
• Updates GitHub git downloader (GitRepo) and git walker to support max depth and extension filtering during recursive traversal.
• Adds unit tests for recursive walk behavior and option propagation.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
utils/walker/git.go	Adds MaxDepth and extension filtering, and applies max-depth logic during WalkDir traversal.
generators/github/scheme_interface.go	Passes recursive options into the git downloader path via GitHubPackageManager.
generators/github/package_manager.go	Stores recursion options on the package manager and passes them into downloader creation.
generators/github/git_repo.go	Propagates recursion/maxDepth/extensions into the walker and adjusts root handling for recursive mode.
generators/generator.go	Adds GeneratorOptions and a new constructor that forwards options to GitHub generator.
generators/generator_test.go	Verifies option propagation through NewGeneratorWithOptions.
generators/github/recursive_test.go	Adds functional tests for recursive walking, max depth, and extension filtering.

Comments suppressed due to low confidence (1)

utils/walker/git.go:242

• After calling d.Info(), the code checks if err != nil but err is not the error returned by d.Info() (it’s an outer variable). This means errInfo can be non-nil and the code will still proceed to g.readFile(f, path) with a potentially nil FileInfo, which can panic. Check errInfo instead and return it when non-nil.

			f, errInfo := d.Info()
			if err != nil {
				return errInfo
			}
			return g.readFile(f, path)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

In the WalkDir callback, the er parameter is ignored. filepath.WalkDir can invoke the callback with a non-nil error and a nil DirEntry, so calling d.IsDir() without first handling er can panic. Handle er at the start of the callback (e.g., return it or decide whether to skip) before using d.

Suggested change

	err = filepath.WalkDir(rootPath, func(path string, d fs.DirEntry, er error) error {
	err = filepath.WalkDir(rootPath, func(path string, d fs.DirEntry, er error) error {
	if er != nil {
	return er
	}

[Copilot]

isAllowedFile lowercases the file’s extension but compares it to the raw allowedExtensions entries. If callers pass values like ".YAML", "yaml", or patterns like ".yaml" (as described in the PR), filtering will silently fail. Consider normalizing allowedExtensions (lowercase, ensure leading '.', optionally accept ".ext") when setting it, and also apply the same filtering behavior in the non-recursive path for consistency.

[Copilot]

The PR description says recursive discovery is enabled by default, but NewGenerator() calls NewGeneratorWithOptions(..., GeneratorOptions{}), which leaves Recursive at its zero value (false) and therefore disables recursion unless callers explicitly opt in. Either update the defaults (and adjust tests/docs) or update the PR description to match the actual behavior/backward-compatibility plan.