Skip to content

feat: Rewrite Python extractor on tree-sitter AST with Django support - #28

Merged
dejo1307 merged 5 commits into
mainfrom
feat--Python-extractors-improvements
Jun 16, 2026
Merged

feat: Rewrite Python extractor on tree-sitter AST with Django support#28
dejo1307 merged 5 commits into
mainfrom
feat--Python-extractors-improvements

Conversation

@GertL

@GertL GertL commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Replace the regex/line-scanner Python extractor with a tree-sitter AST parser. All prior facts are preserved; the AST additionally emits RelCalls / RelInstantiates edges from function bodies, return type annotations, and structural decorator props (@staticmethod, @classmethod, @Property, @AbstractMethod, @task, @shared_task).
  • Add Django project detection (via dependency files / manage.py); classify models with auto-derived table names, CBVs, DRF serializers, @api_view routes, and urls.py path patterns.
  • Fix stale pointer bug: ownerStack and route back-fill now use slice indices instead of *facts.Fact pointers, preventing silently lost call edges when w.out reallocates during body traversal.
  • Skip @overload-decorated functions to avoid duplicate symbol facts for type-checker-only stubs (verified on Apache Airflow: 131 → 79 duplicates eliminated).

GertL and others added 4 commits June 15, 2026 06:17
…dling

- Added handling for assignment nodes in the AST to capture Django urlpatterns at the module level.
- Updated the expression statement handler to also process Django urlpatterns.
- Enhanced the decorated definition handler to back-fill route facts with the corresponding handler names.
- Improved dependency tracking by changing the target format in import statements.
- Modified the function handling to distinguish between methods and top-level functions.
- Updated tests to use a unified extraction function and removed redundant file handling code.
- Cleaned up unused helper tests and ensured consistent handling of class and function definitions.
Replace ownerStack/pendingRouteIndices pointer captures with slice indices
to prevent silently lost call edges and handler back-fills when w.out
reallocates its backing array during body traversal.

Skip @overload-decorated function definitions to avoid emitting duplicate
symbol facts for type-checker-only stubs that have no runtime behaviour.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@GertL
GertL requested a review from dejo1307 June 15, 2026 05:30
@GertL
GertL force-pushed the feat--Python-extractors-improvements branch from f4ca598 to 9aafef3 Compare June 15, 2026 17:24
@dejo1307
dejo1307 merged commit 1838322 into main Jun 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants