Speed up hook diff processing (#27)

* test

* Add profiling feature with macro for measuring execution time

* Refactor diff processing, optimize token handling and storage

* Update test signatures and add profiling tests

* Refactor GitFile signatures and commit messages

Optimize PatchDiff with parallel processing

Add rayon dependency to Cargo.toml

Remove redundant patch tests

Update Cargo.lock with new dependencies

Remove profiling tests

* Update Cargo.lock dependencies and checksums

* Update dependencies in Cargo.toml and Cargo.lock

* Add StringPool for efficient memory use in PatchDiff

* Update dependencies in Cargo.toml and Cargo.lock

* Add `num_cpus` crate and parallelize file processing

* Refactor file processing to use parallel chunks and atomic tokens

* Remove redundant import of `bail` from anyhow

* Sort files by token count in `PatchDiff` implementation.

* Delete test.txt file

* Improve error handling and path management in config and style modules

* Add tests for StringPool functionality in hook.rs

* Update default model and add profiling to model and commit functions

* Add profiling to filesystem module functions

* Implement token counting and generation for commit messages

* Add documentation for Filesystem, File, and Dir structs in filesystem.rs

* Refactor commit message generation methods and file handling logic

* Implement configuration file management and update functions in App

* Implement parallel processing of diff data in PatchDiff trait

* ```
feat: Add Finetune functionality and related files

- Introduce finetune.rs to manage fine-tuning workflows with OpenAI.
- Create finetune.md for documenting the finetuning process.
- Update Cargo.toml and Cargo.lock with necessary dependencies for finetuning.
- Add stats.json to track various parameters in the finetuning process.
```

* ```
Add instruction template constant to commit.rs

- Introduce `INSTRUCTION_TEMPLATE` constant for prompt file content.
```

* Remove unused import of `std::fs` from `commit.rs` file.

* Remove unused import and adjust available tokens calculation

- Remove the import of `Model` and update the `available_tokens` calculations in the `call` function.

* Update max commit length in prompt guidelines

- Change maximum commit length from 72 to {{max_commit_length}} characters.

* ```
Modify imports and refactor filesystem profiling

Based only on the changes visible in the diff, this commit:
- Removes unnecessary profiling statements and imports in `filesystem.rs`.
- Adds an import for `App` from `config` in `main.rs`.
```

* Add directory creation for hooks if it does not exist

- Implement logic to check for the existence of the hooks directory and create it if it's missing.

* Add dead code allowance in filesystem.rs

Based only on the changes visible in the diff, this commit adds a line to allow dead code in the `filesystem.rs` file.

* Revert "```"

This reverts commit 7b9aa2f.

* ```
Update Command enum definition

Based only on the changes visible in the diff, this commit modifies
the existing Command enum without altering its structure or commands.
```

* Delete stats.json file

Based only on the changes visible in the diff, this commit removes the stats.json file.

* ```
Remove install, reinstall, and uninstall modules

Based only on the changes visible in the diff, this commit deletes
the files src/install.rs, src/reinstall.rs, and src/uninstall.rs.
```

* Build inline

* Update default model name in Args implementation

Based only on the changes visible in the diff, this commit modifies the default model name in the Args implementation from "gpt-4o" to "gpt-4o-mini".

* ```
Create hook stress test script

Based only on the changes visible in the diff, this commit adds
a new script for testing various operations in a Git hook context.
```

* ```
Add comprehensive tests script

Based only on the changes visible in the diff, this commit adds a new
Fish script `comprehensive-tests` that includes a series of tests
for various Git operations.
```

* Change file permission of comprehensive-tests.

- Update file mode from 644 to 755.

* Update `comprehensive-tests` script to load environment variables from `.env.local`

This commit updates the `comprehensive-tests` script by adding logic to read and load environment variables from a

* Remove note about output being used as a git commit message from 'prompt.md'

* Update comprehensive-tests script and prompt.md documentation

Based only on

* Update scripts and source code according to visible changes in the diff

Based on the changes visible in the diff, this commit updates the comprehensive testing scripts, modifies several source files related to- 'hook.rs','patch_test.rs', and adds new methods and tests.

Here is a summary of specific changes:
- In `comprehensive-tests` script, add `--debug` flag to `cargo install` command
- Adjust logic to check for ample remaining tokens in `hook.rs`
- Modify Diff and PatchRepository implementations for `hook.rs` and `patch_test.rs`.
- Modify test cases in the `patch_test.rs` script.

Please note each individual file has significant changes that add, modify or enhance the functionality as per the needs reflected in the diffs. However, all changes stick to the theme of improving handling of patches, diffs, and commit-related functionalities.

* Refactor `hook.rs` and ensure a minimum of 512 tokens

Based only on the changes visible in the diff, this commit:
- Adds nine lines to check for an empty diff and handle amend operations in `src/bin/hook.rs`
- Removes four lines related to the error message for no changes found to commit
- Modifies a line to ensure a minimum of 512 remaining tokens
- Appends a four-line snippet to handle amend operations when the source is a commit

* Update clean-up command in comprehensive-tests script

Based only on the changes visible in the diff, this commit:
- Replaces the commented clean-up command in comprehensive-tests script with an active one.

* Add attribute to suppress dead code warnings in hook.rs

* Add initial boilerplate for hook.rs

This commit introduces the initial boilerplate for the `hook.rs` file. The added codes allow for dead code, which is often used during the early stages of development. This single-line addition simply contributes to the initial setup of the file.

* Add debug message when a commit message already exists in hook.rs

Based only on the changes visible in the diff, this commit:
- Adds code for checking if a commit message already exists and is not empty in the file hook.rs,
- If the message exists, a debug log message is displayed, and the function returns, clearing the progress bar.

* Add `to_commit_diff` and `configure_commit_diff_options` methods to `PatchRepository` trait

Based only on the changes visible in the diff, this commit:
- Adds the `to_commit_diff` method to the `PatchRepository` trait which returns a `Result<git2::Diff<'_>>`
- Implements `to_commit_diff` in the `PatchRepository` trait for `Repository` class, where it configures diff options and conducts the diff operation based on the provided tree option
- Adds the `configure_commit_diff_options` method to the `PatchRepository` trait which doesn't return anything but changes the state of provided `DiffOptions`
- Implements `configure_commit_diff_options` in the `PatchRepository` trait for `Repository` class, where it sets various options for a diff operation
- Replaces the usage of `to_diff` method with `to_commit_diff` in the `PatchRepository` implementation for `Repository`.

* Optimize max_tokens_per_file calculation in hook.rs

The max_tokens_per_file calculation within the `process_chunk` function of hook.rs now considers the case of zero remaining files. If no files are remaining, the total_remaining value is assigned to max_tokens_per_file directly.

* Refactor method calls and condition checks in openai.rs and patch_test.rs

* Refine instructions and guidelines for generating git commit messages

* Add error handling for raw SHA1 resolution in hook.rs

* Refactor function calls in patch_test.rs and simplify conditional logic in hook.rs

* Refactor reference resolution in hook.rs

---------

Co-authored-by: Git AI Test <[email protected]>

Author: oleander

.gitignore

@@ -6,3 +6,5 @@ http-cacache/*
 ${env:TMPDIR}
 bin/
 tmp/
+finetune_verify.jsonl
+finetune_train.jsonl

Cargo.lock

@@ -26,7 +26,7 @@ path = "src/bin/hook.rs"
 
 [dependencies]
 # Core functionality
-anyhow = "1.0.95"
+anyhow = { version = "1.0.95", features = ["backtrace"] }
 thiserror = "2.0.11"
 tokio = { version = "1.43", features = ["full"] }
 futures = "0.3"
@@ -52,9 +52,11 @@ serde_derive = "1.0.217"
 serde_ini = "0.2.0"
 serde_json = "1.0"
 
+# OpenAI integration
 async-openai = { version = "0.27.2", default-features = false }
 tiktoken-rs = "0.6.0"
 reqwest = { version = "0.12.12", default-features = true }
+
 # System utilities
 openssl-sys = { version = "0.9.105", features = ["vendored"] }
 rayon = "1.10.0"
@@ -64,6 +66,7 @@ ctrlc = "3.4.5"
 lazy_static = "1.5.0"
 home = "0.5.11"
 dirs = "6.0"
+
 # Syntax highlighting and markdown rendering
 syntect = { version = "5.2", default-features = false, features = [
   "default-fancy",
@@ -74,6 +77,7 @@ textwrap = "0.16"
 structopt = "0.3.26"
 mustache = "0.9.0"
 maplit = "1.0.2"
+
 [dev-dependencies]
 tempfile = "3.16.0"
 

Cargo.toml

@@ -1,5 +1,6 @@
-use anyhow::{bail, Result};
+use anyhow::{anyhow, bail, Result};
 use maplit::hashmap;
+use mustache;
 
 use crate::{config, openai, profile};
 use crate::model::Model;
@@ -13,11 +14,11 @@ fn get_instruction_template() -> Result<String> {
   profile!("Generate instruction template");
   let max_length = config::APP.max_commit_length.unwrap_or(72).to_string();
   let template = mustache::compile_str(INSTRUCTION_TEMPLATE)
-    .map_err(|e| anyhow::anyhow!("Template compilation error: {}", e))?
+    .map_err(|e| anyhow!("Template compilation error: {}", e))?
     .render_to_string(&hashmap! {
       "max_length" => max_length
     })
-    .map_err(|e| anyhow::anyhow!("Template rendering error: {}", e))?;
+    .map_err(|e| anyhow!("Template rendering error: {}", e))?;
   Ok(template)
 }
 
@@ -43,18 +44,11 @@ pub fn get_instruction_token_count(model: &Model) -> Result<usize> {
 ///
 /// # Returns
 /// * `Result<openai::Request>` - The prepared request
-pub fn create_commit_request(diff: String, max_tokens: usize, model: Model) -> Result<openai::Request> {
+fn create_commit_request(diff: String, max_tokens: usize, model: Model) -> Result<openai::Request> {
   profile!("Prepare OpenAI request");
-  let max_length = config::APP.max_commit_length.unwrap_or(72).to_string();
-  let instruction_template = mustache::compile_str(INSTRUCTION_TEMPLATE)
-    .map_err(|e| anyhow::anyhow!("Template compilation error: {}", e))?
-    .render_to_string(&hashmap! {
-      "max_length" => max_length
-    })
-    .map_err(|e| anyhow::anyhow!("Template rendering error: {}", e))?;
-
+  let template = get_instruction_template()?;
   Ok(openai::Request {
-    system: instruction_template,
+    system: template,
     prompt: diff,
     max_tokens: max_tokens.try_into().unwrap_or(u16::MAX),
     model

src/commit.rs

@@ -279,33 +279,28 @@ fn process_chunk(
     };
 
     if max_tokens_per_file == 0 {
-      // No tokens left to allocate, skip remaining files
-      break;
+      continue;
     }
 
     let token_count = *token_count;
     let allocated_tokens = token_count.min(max_tokens_per_file);
 
-    // Attempt to atomically update remaining tokens
-    match remaining_tokens.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |current| {
-      if current >= allocated_tokens {
-        Some(current - allocated_tokens)
-      } else {
-        None
-      }
-    }) {
-      Ok(_) => {
-        let processed_content = if token_count > allocated_tokens {
-          model.truncate(content, allocated_tokens)?
+    if remaining_tokens
+      .fetch_update(Ordering::SeqCst, Ordering::SeqCst, |current| {
+        if current >= allocated_tokens {
+          Some(current - allocated_tokens)
         } else {
-          content.clone()
-        };
-        chunk_results.push((path.clone(), processed_content));
-      }
-      Err(_) => {
-        // Failed to allocate tokens, skip remaining files
-        break;
-      }
+          None
+        }
+      })
+      .is_ok()
+    {
+      let processed_content = if token_count > allocated_tokens {
+        model.truncate(content, allocated_tokens)?
+      } else {
+        content.clone()
+      };
+      chunk_results.push((path.clone(), processed_content));
     }
   }
 
@@ -371,7 +366,7 @@ impl PatchRepository for Repository {
 
   fn configure_diff_options(&self, opts: &mut DiffOptions) {
     opts
-      .ignore_whitespace_change(false)
+      .ignore_whitespace_change(true)
       .recurse_untracked_dirs(true)
       .recurse_ignored_dirs(false)
       .ignore_whitespace_eol(true)
@@ -408,8 +403,6 @@ impl PatchRepository for Repository {
 
 #[cfg(test)]
 mod tests {
-  use tempfile::TempDir;
-
   use super::*;
 
   #[test]
@@ -452,152 +445,4 @@ mod tests {
 
     assert_eq!(pool.strings.len(), MAX_POOL_SIZE);
   }
-
-  #[test]
-  fn test_process_chunk_token_allocation() {
-    let model = Arc::new(Model::default());
-    let total_files = 3;
-    let processed_files = Arc::new(AtomicUsize::new(0));
-    let remaining_tokens = Arc::new(AtomicUsize::new(60)); // Reduced to force allocation limits
-    let result_chunks = Arc::new(Mutex::new(Vec::new()));
-
-    let chunk = vec![
-      (PathBuf::from("file1.txt"), "content1".to_string(), 50),
-      (PathBuf::from("file2.txt"), "content2".to_string(), 40),
-      (PathBuf::from("file3.txt"), "content3".to_string(), 30),
-    ];
-
-    process_chunk(&chunk, &model, total_files, &processed_files, &remaining_tokens, &result_chunks).unwrap();
-
-    let results = result_chunks.lock();
-    // With 60 total tokens and 3 files:
-    // First file gets 20 tokens (60/3)
-    // Second file gets 30 tokens (40/2)
-    // Third file gets 10 tokens (10/1)
-    assert_eq!(results.len(), 3);
-    assert_eq!(remaining_tokens.load(Ordering::SeqCst), 0);
-    assert_eq!(processed_files.load(Ordering::SeqCst), 3);
-  }
-
-  #[test]
-  fn test_process_chunk_concurrent_safety() {
-    use std::thread;
-
-    let model = Arc::new(Model::default());
-    let total_files = 6;
-    let processed_files = Arc::new(AtomicUsize::new(0));
-    let remaining_tokens = Arc::new(AtomicUsize::new(100));
-    let result_chunks = Arc::new(Mutex::new(Vec::new()));
-
-    let chunk1 = vec![
-      (PathBuf::from("file1.txt"), "content1".to_string(), 20),
-      (PathBuf::from("file2.txt"), "content2".to_string(), 20),
-      (PathBuf::from("file3.txt"), "content3".to_string(), 20),
-    ];
-
-    let chunk2 = vec![
-      (PathBuf::from("file4.txt"), "content4".to_string(), 20),
-      (PathBuf::from("file5.txt"), "content5".to_string(), 20),
-      (PathBuf::from("file6.txt"), "content6".to_string(), 20),
-    ];
-
-    // Clone values for thread 2
-    let model2 = model.clone();
-    let processed_files2 = processed_files.clone();
-    let remaining_tokens2 = remaining_tokens.clone();
-    let result_chunks2 = result_chunks.clone();
-
-    // Clone values for main thread access after threads complete
-    let processed_files_main = processed_files.clone();
-    let remaining_tokens_main = remaining_tokens.clone();
-    let result_chunks_main = result_chunks.clone();
-
-    let t1 = thread::spawn(move || {
-      process_chunk(&chunk1, &model, total_files, &processed_files, &remaining_tokens, &result_chunks).unwrap();
-    });
-
-    let t2 = thread::spawn(move || {
-      process_chunk(&chunk2, &model2, total_files, &processed_files2, &remaining_tokens2, &result_chunks2).unwrap();
-    });
-
-    t1.join().unwrap();
-    t2.join().unwrap();
-
-    let results = result_chunks_main.lock();
-    assert_eq!(results.len(), 6);
-    assert_eq!(remaining_tokens_main.load(Ordering::SeqCst), 0);
-    assert_eq!(processed_files_main.load(Ordering::SeqCst), 6);
-  }
-
-  #[test]
-  fn test_to_commit_diff_with_head() -> Result<()> {
-    let temp_dir = TempDir::new()?;
-    let repo = Repository::init(temp_dir.path())?;
-    let mut index = repo.index()?;
-
-    // Create a file and stage it
-    let file_path = temp_dir.path().join("test.txt");
-    std::fs::write(&file_path, "initial content")?;
-    index.add_path(file_path.strip_prefix(temp_dir.path())?)?;
-    index.write()?;
-
-    // Create initial commit
-    let tree_id = index.write_tree()?;
-    let tree = repo.find_tree(tree_id)?;
-    let signature = git2::Signature::now("test", "[email protected]")?;
-    repo.commit(Some("HEAD"), &signature, &signature, "Initial commit", &tree, &[])?;
-
-    // Modify and stage the file
-    std::fs::write(&file_path, "modified content")?;
-    index.add_path(file_path.strip_prefix(temp_dir.path())?)?;
-    index.write()?;
-
-    // Get HEAD tree
-    let head = repo.head()?.peel_to_tree()?;
-
-    // Get diff
-    let diff = repo.to_commit_diff(Some(head))?;
-
-    // Verify diff shows only staged changes
-    let mut diff_found = false;
-    diff.print(DiffFormat::Patch, |_delta, _hunk, line| {
-      let content = line.content().to_utf8();
-      if line.origin() == '+' && content.contains("modified content") {
-        diff_found = true;
-      }
-      true
-    })?;
-
-    assert!(diff_found, "Expected to find staged changes in diff");
-    Ok(())
-  }
-
-  #[test]
-  fn test_to_commit_diff_without_head() -> Result<()> {
-    let temp_dir = TempDir::new()?;
-    let repo = Repository::init(temp_dir.path())?;
-    let mut index = repo.index()?;
-
-    // Create and stage a new file
-    let file_path = temp_dir.path().join("test.txt");
-    std::fs::write(&file_path, "test content")?;
-    index.add_path(file_path.strip_prefix(temp_dir.path())?)?;
-    index.write()?;
-
-    // Get diff (no HEAD exists yet)
-    let diff = repo.to_commit_diff(None)?;
-
-    // Verify diff shows staged changes
-    let mut diff_found = false;
-    diff.print(DiffFormat::Patch, |_delta, _hunk, line| {
-      let content = line.content().to_utf8();
-      if line.origin() == '+' && content.contains("test content") {
-        diff_found = true;
-      }
-      true
-    })?;
-
-    assert!(diff_found, "Expected to find staged changes in diff");
-    Ok(())
-  }
 }

src/hook.rs

@@ -145,17 +145,18 @@ pub async fn call(request: Request) -> Result<Response> {
               "ERROR:".bold().bright_red(),
               "Network error:".bright_white(),
               e.to_string().dimmed(),
-              "Failed to connect to OpenAI service.".dimmed(),
+              "Failed to connect to OpenAI API.".dimmed(),
               "Check your internet connection".yellow(),
-              "Verify OpenAI service is not experiencing downtime".yellow()
+              "Verify OpenAI service availability".yellow()
             ),
           _ =>
             format!(
-              "{} {}\n    {}\n\nDetails:\n    {}",
+              "{} {}\n    {}\n\nDetails:\n    {}\n\nSuggested Actions:\n    1. {}",
               "ERROR:".bold().bright_red(),
               "Unexpected error:".bright_white(),
               err.to_string().dimmed(),
-              "An unexpected error occurred while communicating with OpenAI.".dimmed()
+              "An unexpected error occurred while calling OpenAI API.".dimmed(),
+              "Please report this issue on GitHub".yellow()
             ),
         };
         return Err(anyhow!(error_msg));
@@ -165,11 +166,11 @@ pub async fn call(request: Request) -> Result<Response> {
     let content = response
       .choices
       .first()
-      .context("No choices returned")?
+      .context("No response choices available")?
       .message
       .content
       .clone()
-      .context("No content returned")?;
+      .context("Response content is empty")?;
 
     Ok(Response { response: content })
   }

src/install.rs

@@ -29,7 +29,6 @@ impl Drop for Profile {
 #[macro_export]
 macro_rules! profile {
   ($name:expr) => {
-    // Currently a no-op, but can be expanded for actual profiling
-    let _profile_span = $name;
+    let _profile = $crate::Profile::new($name);
   };
 }