Release 0.2.6 (#37)

* Add Python support (#36)

* Update CHANGELOG

* chore: Release am_list version 0.2.6

---------

Co-authored-by: あで <[email protected]>

Author: gagbo

CHANGELOG.md

@@ -9,6 +9,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased] - ReleaseDate
 
+## [Version 0.2.6] - 2023-07-27
+
+### Added
+
+- [Python] Support for python language
+
 ## [Version 0.2.5] - 2023-07-19
 
 ### Added
@@ -81,7 +87,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Security
 
 <!-- next-url -->
-[Unreleased]: https://github.com/autometrics-dev/am_list/compare/v0.2.5...HEAD
+[Unreleased]: https://github.com/autometrics-dev/am_list/compare/v0.2.6...HEAD
+[Version 0.2.6]: https://github.com/autometrics-dev/am_list/compare/v0.2.5...v0.2.6
 [Version 0.2.5]: https://github.com/autometrics-dev/am_list/compare/v0.2.4...v0.2.5
 [Version 0.2.4]: https://github.com/autometrics-dev/am_list/compare/v0.2.3...v0.2.4
 [Version 0.2.3]: https://github.com/autometrics-dev/am_list/compare/v0.2.2...v0.2.3

Cargo.lock

@@ -1,6 +1,6 @@
 [package]
 name = "am_list"
-version = "0.2.5"
+version = "0.2.6"
 edition = "2021"
 repository = "https://github.com/autometrics-dev/am_list"
 authors = ["Fiberplane <[email protected]>", "Gerry Agbobada <[email protected]>"]
@@ -22,6 +22,7 @@ serde_json = "1.0.96"
 thiserror = "1.0.40"
 tree-sitter = "0.20.10"
 tree-sitter-go = "0.19.1"
+tree-sitter-python = "0.20.2"
 tree-sitter-rust = "0.20.3"
 tree-sitter-typescript = "0.20.2"
 walkdir = "2.3.3"

Cargo.toml

@@ -20,6 +20,7 @@ VERSION=0.2.0 curl --proto '=https' --tlsv1.2 -LsSf https://github.com/autometri
 ```
 
 And run the binary
+
 ```bash
 # Make sure that `~/.cargo/bin` is in your `PATH`
 am_list list -l rs /path/to/project/root
@@ -34,20 +35,22 @@ returns the exact same labels as the ones you would need to use in PromQL to
 look at the metrics. In a nutshell,
 "[Autometrics](https://github.com/autometrics-dev) compliance".
 
-Language | Function name detection | Module detection
-:---:|:---:|:---:
-[Rust](https://github.com/autometrics-dev/autometrics-rs) | ✅ | ✅
-[Typescript](https://github.com/autometrics-dev/autometrics-ts) | ✅ | ⚠️[^wrapper]
-[Go](https://github.com/autometrics-dev/autometrics-go) | ⚠️[^all-functions] | ✅
-[Python](https://github.com/autometrics-dev/autometrics-py) | ❌ | ❌
-[C#](https://github.com/autometrics-dev/autometrics-cs) | ❌ | ❌
+|                            Language                             | Function name detection | Module detection |
+| :-------------------------------------------------------------: | :---------------------: | :--------------: |
+|    [Rust](https://github.com/autometrics-dev/autometrics-rs)    |           ✅            |        ✅        |
+| [Typescript](https://github.com/autometrics-dev/autometrics-ts) |           ✅            |   ⚠️[^wrapper]   |
+|     [Go](https://github.com/autometrics-dev/autometrics-go)     |   ⚠️[^all-functions]    |        ✅        |
+|   [Python](https://github.com/autometrics-dev/autometrics-py)   |           ✅            |        ✅        |
+|     [C#](https://github.com/autometrics-dev/autometrics-cs)     |           ❌            |        ❌        |
 
-[^wrapper]: For Typescript (and all languages where autometrics is a wrapper
+[^wrapper]:
+    For Typescript (and all languages where autometrics is a wrapper
     function), static analysis makes it hard to traverse imports to find the
     module where an instrumented function is _defined_, so the reported module
     is the module where the function has been _instrumented_
 
-[^all-functions]: Support list all autometricized functions, but not all
+[^all-functions]:
+    Support list all autometricized functions, but not all
     functions without restriction
 
 ### Typescript
@@ -65,42 +68,44 @@ attempt to be useful.
 The other difficulty encountered when using a static analysis tool with autometrics-ts is that the
 instrumentation can happen anywhere, as the wrapper function call can use an imported symbol as its argument:
 
-``` typescript
-import { exec } from 'child_process'
-import { autometrics } from '@autometrics/autometrics'
+```typescript
+import { exec } from "child_process";
+import { autometrics } from "@autometrics/autometrics";
 
-const instrumentedExec = autometrics(exec)
+const instrumentedExec = autometrics(exec);
 
 // use instrumentedExec everywhere instead of exec
 ```
 
 In order to report the locus of _function definition_ as the module, we would
 need to include both:
+
 - a complete import resolution step, to figure out the origin module of the
   instrumented function (`child_process` in the example), and
 - a dependency inspection step, to figure out the path to the instrumented
   function definition _within_ the dependency (`lib/child_process.js` in the
   [node source code](https://github.com/nodejs/node/blob/main/lib/child_process.js))
-  
+
 This is impractical and error-prone to implement these steps accurately, so
 instead we only try to detect imports when they are explicitely imported in the
 same file, and we will only report the function module as the imported module
 (not the path to the file it is defined in). Practically that means that for
 this example:
 
-``` typescript
+```typescript
 // in src/router/index.ts
-import { exec } from 'child_process'
-import { origRoute as myRoute } from '../handlers'
-import { autometrics } from '@autometrics/autometrics'
+import { exec } from "child_process";
+import { origRoute as myRoute } from "../handlers";
+import { autometrics } from "@autometrics/autometrics";
 
-const instrumentedExec = autometrics(exec)
-const instrumentedRoute = autometrics(myRoute)
+const instrumentedExec = autometrics(exec);
+const instrumentedRoute = autometrics(myRoute);
 
 // use instrumentedExec everywhere instead of exec
 ```
 
 `am_list` will report 2 functions:
+
 - `{"function": "exec", "module": "ext://child_process"}`: using `ext://`
   protocol to say the module is non-local
 - `{"function": "origRoute", "module": "handlers"}`: even if `myRoute` is

README.md

@@ -0,0 +1,2 @@
+(function_definition
+    name: (identifier) @func.name)

runtime/queries/python/all_functions.scm

@@ -0,0 +1,7 @@
+(decorated_definition
+  (decorator
+    [(identifier) @decorator.name
+      (call (identifier) @decorator.name)])
+  (#eq? @decorator.name "{0}")
+  definition: (function_definition
+               name: (identifier) @func.name))

runtime/queries/python/autometrics.scm.tpl

@@ -0,0 +1,9 @@
+(import_from_statement
+  module_name: (dotted_name (identifier) @import.module)
+  (#eq? @import.module "autometrics")
+  name:[
+        (dotted_name (identifier) @import.name)
+        (aliased_import
+          name: (dotted_name (identifier) @import.name)
+          alias: (identifier) @import.alias)]
+  (#eq? @import.name "autometrics"))

runtime/queries/python/import.scm

@@ -1,4 +1,5 @@
 pub mod go;
+pub mod python;
 pub mod rust;
 pub mod typescript;
 

src/lib.rs

@@ -42,6 +42,7 @@ enum Language {
     Rust,
     Go,
     Typescript,
+    Python,
 }
 
 impl FromStr for Language {
@@ -61,6 +62,10 @@ impl FromStr for Language {
             return Ok(Self::Typescript);
         }
 
+        if ["python", "py"].contains(&discriminant.as_str()) {
+            return Ok(Self::Python);
+        }
+
         Err(format!("Unknown language: {s}"))
     }
 }
@@ -80,6 +85,7 @@ fn main() -> anyhow::Result<()> {
                 Language::Rust => Box::new(am_list::rust::Impl {}),
                 Language::Go => Box::new(am_list::go::Impl {}),
                 Language::Typescript => Box::new(am_list::typescript::Impl {}),
+                Language::Python => Box::new(am_list::python::Impl {}),
             };
 
             let mut res = if args.all_functions {

src/main.rs

@@ -0,0 +1,127 @@
+mod queries;
+
+use crate::{ExpectedAmLabel, ListAmFunctions, Result};
+use queries::{AllFunctionsQuery, AmImportQuery, AmQuery};
+use rayon::prelude::*;
+use std::{
+    collections::HashSet,
+    fs::read_to_string,
+    path::{Path, MAIN_SEPARATOR},
+};
+use walkdir::{DirEntry, WalkDir};
+
+/// Implementation of the Python support for listing autometricized functions.
+#[derive(Clone, Copy, Debug, Default)]
+pub struct Impl {}
+
+impl Impl {
+    fn is_hidden(entry: &DirEntry) -> bool {
+        entry
+            .file_name()
+            .to_str()
+            .map(|s| s.starts_with('.'))
+            .unwrap_or(false)
+    }
+
+    fn is_valid(entry: &DirEntry) -> bool {
+        if Impl::is_hidden(entry) {
+            return false;
+        }
+        entry.file_type().is_dir()
+            || entry
+                .path()
+                .extension()
+                .map_or(false, |ext| ext == "py" || ext == "py3")
+    }
+}
+
+impl ListAmFunctions for Impl {
+    fn list_autometrics_functions(&mut self, project_root: &Path) -> Result<Vec<ExpectedAmLabel>> {
+        const PREALLOCATED_ELEMS: usize = 100;
+        let mut list = HashSet::with_capacity(PREALLOCATED_ELEMS);
+        let root_name = project_root
+            .file_name()
+            .map(|s| s.to_str().unwrap_or_default())
+            .unwrap_or("");
+
+        let walker = WalkDir::new(project_root).into_iter();
+        let mut source_mod_pairs = Vec::with_capacity(PREALLOCATED_ELEMS);
+        source_mod_pairs.extend(walker.filter_entry(Self::is_valid).filter_map(|entry| {
+            let entry = entry.ok()?;
+            Some(
+                entry
+                    .path()
+                    .to_str()
+                    .map(ToString::to_string)
+                    .unwrap_or_default(),
+            )
+        }));
+
+        list.par_extend(source_mod_pairs.par_iter().filter_map(move |path| {
+            let relative_module_name = Path::new(path)
+                .strip_prefix(project_root)
+                .ok()?
+                .with_extension("")
+                .to_str()?
+                .replace(MAIN_SEPARATOR, ".");
+            let module_name = format!("{}.{}", root_name, relative_module_name);
+            let source = read_to_string(path).ok()?;
+            let import_query = AmImportQuery::try_new().ok()?;
+            let decorator_name = import_query.get_decorator_name(source.as_str()).ok()?;
+            let query = AmQuery::try_new(decorator_name.as_str()).ok()?;
+            let names = query
+                .list_function_names(&source, module_name.as_str())
+                .unwrap_or_default();
+            Some(names)
+        }));
+
+        let mut result = Vec::with_capacity(PREALLOCATED_ELEMS);
+        result.extend(list.into_iter().flatten());
+        Ok(result)
+    }
+
+    fn list_all_functions(&mut self, project_root: &Path) -> Result<Vec<ExpectedAmLabel>> {
+        const PREALLOCATED_ELEMS: usize = 100;
+        let mut list = HashSet::with_capacity(PREALLOCATED_ELEMS);
+        let root_name = project_root
+            .file_name()
+            .map(|s| s.to_str().unwrap_or_default())
+            .unwrap_or("");
+
+        let walker = WalkDir::new(project_root).into_iter();
+        let mut source_mod_pairs = Vec::with_capacity(PREALLOCATED_ELEMS);
+        source_mod_pairs.extend(walker.filter_entry(Self::is_valid).filter_map(|entry| {
+            let entry = entry.ok()?;
+            Some(
+                entry
+                    .path()
+                    .to_str()
+                    .map(ToString::to_string)
+                    .unwrap_or_default(),
+            )
+        }));
+
+        list.par_extend(source_mod_pairs.par_iter().filter_map(move |path| {
+            let relative_module_name = Path::new(path)
+                .strip_prefix(project_root)
+                .ok()?
+                .with_extension("")
+                .to_str()?
+                .replace(MAIN_SEPARATOR, ".");
+            let module_name = format!("{}.{}", root_name, relative_module_name);
+            let source = read_to_string(path).ok()?;
+            let query = AllFunctionsQuery::try_new().ok()?;
+            let names = query
+                .list_function_names(&source, module_name.as_str())
+                .unwrap_or_default();
+            Some(names)
+        }));
+
+        let mut result = Vec::with_capacity(PREALLOCATED_ELEMS);
+        result.extend(list.into_iter().flatten());
+        Ok(result)
+    }
+}
+
+#[cfg(test)]
+mod tests;