Improve ruby extractor (#33)

* Converting Ruby extractor to tree sitter

* Updating docs post Ruby refactoring

Author: dejo1307

ARCHITECTURE.md

@@ -18,7 +18,7 @@ Two design choices make this graph useful in a way that "throw the repo at an LL
 
 2. **Every fact is derived, never inferred.** This is the core invariant:
 
-   > **Nothing in the graph is guessed by a language model.** Every node and edge comes from a real parser (Go's `go/ast`, a tree-sitter grammar, a language-specific scanner) or a deterministic algorithm (Tarjan's strongly-connected-components for cycles, pattern matching for layers). Run enola twice on the same commit and you get the same graph, byte for byte.
+   > **Nothing in the graph is guessed by a language model.** Every node and edge comes from a real parser (Go's `go/ast`, a tree-sitter grammar, a YAML/JSON scanner) or a deterministic algorithm (Tarjan's strongly-connected-components for cycles, pattern matching for layers). Run enola twice on the same commit and you get the same graph, byte for byte.
 
 That determinism is the whole point. An AI agent reasoning over enola's graph is standing on ground truth it can trust, instead of re-deriving the structure of your code — imperfectly, and at the cost of tokens — on every single task.
 
@@ -343,7 +343,7 @@ Each extractor is detected by characteristic project files and then parses what
 | Python     | tree-sitter      | `pyproject.toml`, `setup.py`, `requirements.txt`, `Pipfile`, `pytest.ini`, `mypy.ini`, `tox.ini`, or `setup.cfg` (root or up to 3 levels deep) |
 | TypeScript | tree-sitter      | `tsconfig.json`, `tsconfig.base.json`, or `package.json` with TypeScript (root or one level deep) |
 | Swift      | tree-sitter      | `Package.swift`, `.xcodeproj`, or `.xcworkspace` present |
-| Ruby       | regex scanner    | `Gemfile` present |
+| Ruby       | tree-sitter      | `Gemfile` present |
 | C++        | tree-sitter      | a C++ source (`.cpp`/`.cc`/`.cxx`/`.hpp`/...) present, or a build file (`CMakeLists.txt`/`Makefile`/`meson.build`/`*.vcxproj`) plus any header |
 | OpenAPI    | YAML/JSON scanner| any file containing `openapi:` or `swagger:` |
 
@@ -361,7 +361,7 @@ Each extractor is detected by characteristic project files and then parses what
 
 **OpenAPI** scans for spec files independently of the main walker (so it finds them even when `*.yaml`/`*.json` are globally ignored), confirming candidates by an `openapi:`/`swagger:` key. It emits one `route` per operation enriched with method, `operationId`, summary, tags, and a spec back-reference; specs under an `openapi/client/` directory are marked `role:"client"`. Gateway extensions (`x-gateway-config`, `x-gateway-capabilities`) are parsed into props.
 
-**Ruby** is Rails-aware: ActiveRecord models (`has_many`, `belongs_to`, scopes, table inference), the route DSL in `config/routes.rb` (resources, namespaces, member/collection blocks), and Packwerk package boundaries (`package.yml` dependency enforcement). It also tracks modules, classes, methods with visibility, mixins (`include`/`extend`/`prepend`), `ActiveSupport::Concern`, constants, and attributes.
+**Ruby** is parsed with tree-sitter, replacing the former line-based regex scanner — the grammar handles heredocs, endless methods (`def x = expr`), multi-line expressions, and the nested scopes that tripped up the line scanner. It is Rails-aware: ActiveRecord models (`has_many`/`has_one`/`belongs_to`/`has_and_belongs_to_many`, scopes, table inference, explicit `self.table_name`) emit `storage` facts; the route DSL in `config/routes.rb` (plus `config/routes/*.rb` and packwerk `draw`) is walked from the real block structure, so nested `namespace`/`scope`/`resources`/`member`/`collection` blocks produce one `route` per RESTful action (honoring `only:`/`except:`); and Packwerk package boundaries (`package.yml` dependency enforcement, `app/public/` privacy) are parsed. It tracks modules, classes, methods with `public`/`private`/`protected` visibility, `class << self` eigenclass and `module_function` methods — now correctly typed as class methods rather than instance methods — mixins (`include`/`extend`/`prepend` → `implements` edges), `ActiveSupport::Concern` (flagged `concern: true`), constants, and `attr_*` accessors. Like the other AST extractors, it walks method bodies for call sites, emitting `calls` edges (qualified `Const.method`/`Ns::Class.method` and receiver `var.method`, deduplicated) and `implements` edges for superclasses — so Ruby participates in `traverse`, `find_path`, and `impact_analysis`.
 
 **C++** is parsed with tree-sitter and handles the header/source split that defines the language. Classes, structs, unions, enums (incl. `enum class`), namespaces, free functions and methods, data members, and `typedef`/`using` aliases become symbol facts named `<dir>.<ns1::ns2::Class::member>` — enola's `<dir>.` module convention on the outside, native C++ `::` scope inside. Because an out-of-line definition `Class::method` (parsed from a `qualified_identifier`) yields the same canonical name as its in-class declaration, a dedup pass **merges a header's method prototype with its `.cpp` definition** into a single symbol (the definition wins for file/line and carries the call-graph edges). Base classes become `implements` edges; method bodies are walked for `calls`/`instantiates` edges; quoted `#include "x.h"` becomes a `dependency` resolved to the declaring module, while system `<...>` includes are skipped. Templates are unwrapped to their inner declaration and flagged `templated`, and the walker descends through `#if`/`#ifdef` preprocessor guards (so code wrapped in `#if defined(HAVE_*)` and headers behind include guards are still extracted). *Limitation:* header/source merging relies on the `.h` and `.cpp` living in the same directory (the common layout); split `include/` + `src/` trees are not merged.
 

README.md

@@ -164,7 +164,7 @@ Working across several repos? Generate the first, then add the rest with append
 
 Framework- and platform-specific detection for each language is described in **[ARCHITECTURE.md → Supported languages](ARCHITECTURE.md#supported-languages)**.
 
-> Python is parsed with tree-sitter and now contributes call and dependency edges to the graph, so `traverse`, `find_path`, and `impact_analysis` reach into Python code — not just modules and routes.
+> Python and Ruby are parsed with tree-sitter and contribute call and dependency edges to the graph, so `traverse`, `find_path`, and `impact_analysis` reach into them — not just modules and routes.
 
 ---
 

go.mod

@@ -9,6 +9,7 @@ require (
 	github.com/tree-sitter/tree-sitter-cpp v0.22.4-0.20240818224355-b1a4e2b25148
 	github.com/tree-sitter/tree-sitter-java v0.21.1-0.20240824015150-576d8097e495
 	github.com/tree-sitter/tree-sitter-python v0.23.6
+	github.com/tree-sitter/tree-sitter-ruby v0.21.1-0.20240818211811-7dbc1e2d0e2d
 	github.com/tree-sitter/tree-sitter-typescript v0.23.2
 	gopkg.in/yaml.v3 v3.0.1
 )

internal/extractors/rubyextractor/routes.go

@@ -1,7 +1,6 @@
 package rubyextractor
 
 import (
-	"bufio"
 	"log"
 	"os"
 	"path/filepath"
@@ -11,20 +10,8 @@ import (
 	"github.com/enola-labs/enola/internal/facts"
 )
 
-// Route DSL regex patterns.
-var (
-	httpVerbRe  = regexp.MustCompile(`^\s*(get|post|put|patch|delete)\s+['"]([^'"]+)['"](?:\s*,\s*to:\s*['"]([^'"]+)['"])?`)
-	resourcesRe = regexp.MustCompile(`^\s*resources?\s+:(\w+)`)
-	namespaceRe = regexp.MustCompile(`^\s*namespace\s+:(\w+)`)
-	scopePathRe = regexp.MustCompile(`^\s*scope\s+['"]([^'"]+)['"]`)
-	scopeModRe  = regexp.MustCompile(`^\s*scope\s+module:\s*[:'"](\w+)`)
-	rootRe      = regexp.MustCompile(`^\s*root\s+(?:to:\s*)?['"]([^'"]+)['"]`)
-	drawRe      = regexp.MustCompile(`^\s*draw\s*\(\s*:(\w+)\s*\)`)
-	memberRe    = regexp.MustCompile(`^\s*(member|collection)\s+do\b`)
-	doBlockRe   = regexp.MustCompile(`\bdo\s*(?:\|[^|]*\|)?\s*$`)
-	onlyRe      = regexp.MustCompile(`only:\s*\[([^\]]*)\]`)
-	exceptRe    = regexp.MustCompile(`except:\s*\[([^\]]*)\]`)
-)
+// symbolListRe extracts symbol names from a string like ":index, :show, :create".
+var symbolListRe = regexp.MustCompile(`:(\w+)`)
 
 // extractAllRoutes finds and parses all Rails route files in the repository.
 func extractAllRoutes(repoPath string, files []string) []facts.Fact {
@@ -43,14 +30,12 @@ func extractAllRoutes(repoPath string, files []string) []facts.Fact {
 
 	for _, relFile := range routeFiles {
 		absFile := filepath.Join(repoPath, relFile)
-		f, err := os.Open(absFile)
+		src, err := os.ReadFile(absFile)
 		if err != nil {
 			log.Printf("[ruby-extractor] error reading route file %s: %v", relFile, err)
 			continue
 		}
-		routeFacts := parseRouteFile(f, relFile)
-		f.Close()
-		allFacts = append(allFacts, routeFacts...)
+		allFacts = append(allFacts, parseRouteFileAST(src, relFile)...)
 	}
 
 	return allFacts
@@ -82,206 +67,6 @@ type routeScope struct {
 	module     string
 }
 
-// parseRouteFile parses a single Rails route file.
-func parseRouteFile(f *os.File, relFile string) []facts.Fact {
-	var result []facts.Fact
-
-	scanner := bufio.NewScanner(f)
-	scanner.Buffer(make([]byte, 0, 256*1024), 1024*1024)
-
-	var (
-		lineNum    int
-		scopeStack []routeScope
-		depth      int
-		currentResource string
-	)
-
-	for scanner.Scan() {
-		lineNum++
-		line := scanner.Text()
-		trimmed := strings.TrimSpace(line)
-
-		if trimmed == "" || strings.HasPrefix(trimmed, "#") {
-			continue
-		}
-
-		// Track end keywords.
-		if trimmed == "end" {
-			depth--
-			if depth < 0 {
-				depth = 0
-			}
-			if depth < len(scopeStack) {
-				scopeStack = scopeStack[:depth]
-			}
-			currentResource = ""
-			continue
-		}
-
-		prefix := buildPrefix(scopeStack)
-
-		// draw(:package_name) -- delegation to packwerk package routes.
-		if m := drawRe.FindStringSubmatch(line); m != nil {
-			result = append(result, facts.Fact{
-				Kind: facts.KindRoute,
-				Name: prefix + "/" + m[1],
-				File: relFile,
-				Line: lineNum,
-				Props: map[string]any{
-					"method":    "DRAW",
-					"framework": "rails",
-					"language":  "ruby",
-					"delegate":  m[1],
-				},
-			})
-			continue
-		}
-
-		// Namespace.
-		if m := namespaceRe.FindStringSubmatch(line); m != nil {
-			scopeStack = append(scopeStack, routeScope{
-				pathPrefix: "/" + m[1],
-				module:     m[1],
-			})
-			depth++
-			continue
-		}
-
-		// Scope with path.
-		if m := scopePathRe.FindStringSubmatch(line); m != nil {
-			path := m[1]
-			if !strings.HasPrefix(path, "/") {
-				path = "/" + path
-			}
-			scopeStack = append(scopeStack, routeScope{pathPrefix: path})
-			if doBlockRe.MatchString(line) {
-				depth++
-			}
-			continue
-		}
-
-		// Scope with module.
-		if m := scopeModRe.FindStringSubmatch(line); m != nil {
-			scopeStack = append(scopeStack, routeScope{module: m[1]})
-			if doBlockRe.MatchString(line) {
-				depth++
-			}
-			continue
-		}
-
-		// Root route.
-		if m := rootRe.FindStringSubmatch(line); m != nil {
-			result = append(result, facts.Fact{
-				Kind: facts.KindRoute,
-				Name: prefix + "/",
-				File: relFile,
-				Line: lineNum,
-				Props: map[string]any{
-					"method":    "GET",
-					"framework": "rails",
-					"language":  "ruby",
-					"handler":   m[1],
-				},
-				Relations: []facts.Relation{
-					{Kind: facts.RelDeclares, Target: filepath.Dir(relFile)},
-				},
-			})
-			continue
-		}
-
-		// HTTP verb routes: get '/path', post '/path', etc.
-		if m := httpVerbRe.FindStringSubmatch(line); m != nil {
-			method := strings.ToUpper(m[1])
-			path := m[2]
-			handler := m[3]
-
-			if !strings.HasPrefix(path, "/") {
-				path = "/" + path
-			}
-			fullPath := prefix + path
-
-			props := map[string]any{
-				"method":    method,
-				"framework": "rails",
-				"language":  "ruby",
-			}
-			if handler != "" {
-				props["handler"] = handler
-			}
-
-			result = append(result, facts.Fact{
-				Kind:  facts.KindRoute,
-				Name:  fullPath,
-				File:  relFile,
-				Line:  lineNum,
-				Props: props,
-				Relations: []facts.Relation{
-					{Kind: facts.RelDeclares, Target: filepath.Dir(relFile)},
-				},
-			})
-			continue
-		}
-
-		// resources / resource.
-		if m := resourcesRe.FindStringSubmatch(line); m != nil {
-			resourceName := m[1]
-			currentResource = resourceName
-			resourcePath := prefix + "/" + resourceName
-
-			actions := restfulActions(line)
-			for _, action := range actions {
-				method := action.method
-				path := resourcePath + action.suffix
-
-				props := map[string]any{
-					"method":    method,
-					"framework": "rails",
-					"language":  "ruby",
-					"resource":  resourceName,
-					"action":    action.name,
-				}
-
-				result = append(result, facts.Fact{
-					Kind:  facts.KindRoute,
-					Name:  path,
-					File:  relFile,
-					Line:  lineNum,
-					Props: props,
-					Relations: []facts.Relation{
-						{Kind: facts.RelDeclares, Target: filepath.Dir(relFile)},
-					},
-				})
-			}
-
-			// If there's a do block, push resource as a scope.
-			if doBlockRe.MatchString(line) {
-				scopeStack = append(scopeStack, routeScope{pathPrefix: "/" + resourceName})
-				depth++
-			}
-			continue
-		}
-
-		// member do / collection do.
-		if m := memberRe.FindStringSubmatch(line); m != nil {
-			blockType := m[1]
-			memberPrefix := ""
-			if blockType == "member" && currentResource != "" {
-				memberPrefix = "/:id"
-			}
-			scopeStack = append(scopeStack, routeScope{pathPrefix: memberPrefix})
-			depth++
-			continue
-		}
-
-		// Track other do blocks for depth.
-		if doBlockRe.MatchString(line) {
-			depth++
-		}
-	}
-
-	return result
-}
-
 // buildPrefix constructs the current URL prefix from the scope stack.
 func buildPrefix(stack []routeScope) string {
 	var parts []string
@@ -300,8 +85,9 @@ type restAction struct {
 	suffix string
 }
 
-// restfulActions returns the set of REST actions for a resources declaration.
-func restfulActions(line string) []restAction {
+// restfulActions returns the set of REST actions for a resources declaration,
+// honoring only:/except: filters parsed from the declaration's arguments.
+func restfulActions(only, except map[string]bool) []restAction {
 	all := []restAction{
 		{name: "index", method: "GET", suffix: ""},
 		{name: "create", method: "POST", suffix: ""},
@@ -312,18 +98,12 @@ func restfulActions(line string) []restAction {
 		{name: "destroy", method: "DELETE", suffix: "/:id"},
 	}
 
-	// Check for only: [...] filter.
-	if m := onlyRe.FindStringSubmatch(line); m != nil {
-		allowed := parseSymbolList(m[1])
-		return filterActions(all, allowed, true)
+	if len(only) > 0 {
+		return filterActions(all, only, true)
 	}
-
-	// Check for except: [...] filter.
-	if m := exceptRe.FindStringSubmatch(line); m != nil {
-		excluded := parseSymbolList(m[1])
-		return filterActions(all, excluded, false)
+	if len(except) > 0 {
+		return filterActions(all, except, false)
 	}
-
 	return all
 }