feat: Git-backed transaction notes and core helpers

This PR introduces a transaction core module and structured Git note support for every write operation in ChronDB, fulfilling architectural requirements for chronological traceability and external correlation/audit.

Highlights:
- Adds `chrondb.transaction.core` namespace. Provides a shared transaction context, unique `tx_id` generation, and helpers for capturing metadata (origin, user, flags) across all entrypoints (REST, SQL, Redis).
- Implements a new `chrondb.storage.git.notes` module. Every commit now stores a JSON note under the `chrondb` namespace, containing: `tx_id`, origin (protocol/source), optional `user`, semantic flags (`bulk-load`, `migration`, `rollback`, etc.), and client/request metadata (e.g. `X-Request-Id`).
- Refactors all transaction/write paths in the Git-backed storage engine to invoke these helpers. Enforces that missing notes are treated as a regression (test coverage included).
- Documents new transaction block (`with-transaction`) usage, and how to pass metadata via HTTP headers or protocol attributes to propagate context into Git notes. See `docs/api.md`, "Transaction Operations" and "Transaction Metadata and Git Notes".
- Adds and updates unit/integration tests for: correct note attachment, lookup with `git notes`, failure modes (e.g., repository contention, branch mutation), and regression coverage (notes always present after a commit).
- Updates AGENT.md: All write paths _must_ call transaction helpers to guarantee Git note context. No destructive mutations; always append-only.
- No backwards-incompatible API changes. Protocols remain consistent across REST, SQL (Postgres), and Redis wire formats.

Rationale:
ChronDB's Git-based architecture enables full auditability and time-travel. Storing rich transaction metadata as Git notes ensures every operation is contextualized—supporting debugging, lineage analysis, external audit, and advanced correlation across distributed systems. This change centralizes transaction and commit metadata handling, eliminating per-protocol ad-hoc logic and reducing the risk of silent context loss.

See also:
- `src/chrondb/storage/git/notes.clj` (implementation)
- `src/chrondb/transaction/core.clj` (transaction core logic)
- New/updated tests in `test/chrondb/storage/git/notes_test.clj`, `test/chrondb/transaction/core_test.clj`
- Expanded documentation in `docs/api.md` and process requirements in AGENT.md.

BREAKING: None, but all future write code must route through the transaction context to guarantee note coverage and protocol metadata flow.

fixed #43
fixed #44

Signed-off-by: Avelino <[email protected]>

Author: avelino

AGENT.md

@@ -40,6 +40,7 @@ ChronDB is a chronological key/value database implemented in Clojure and backed
 - Handle repository creation and concurrency with JGit primitives
 - Surface meaningful errors when Git operations fail (lock contention, missing refs)
 - Avoid destructive operations; prefer new commits over in-place mutation
+- Ensure every write path calls the shared transaction helpers so that Git notes capture `tx_id`, origin, user, flags, and protocol metadata. Treat missing notes as regressions.
 
 ### API Development
 

docs/api.md

@@ -177,6 +177,28 @@ All operations within the transaction block are atomic:
 - Changes are only visible after successful commit
 - Automatic rollback on failure
 
+#### Transaction Metadata and Git Notes
+
+ChronDB attaches a structured Git note to every commit. The payload includes the transaction id (`tx_id`), origin, optional user identifier, flags, and request metadata. You can enrich this payload by providing HTTP headers on REST requests:
+
+| Header | Description |
+| --- | --- |
+| `X-ChronDB-Origin` | Overrides the origin label stored in the note (defaults to `rest`). |
+| `X-ChronDB-User` | Associates commits with an authenticated user id or service account. |
+| `X-ChronDB-Flags` | Comma-separated list of semantic flags (for example: `bulk-load, migration`). |
+| `X-Request-Id` / `X-Correlation-Id` | Propagate request correlation identifiers into commit metadata. |
+
+All ChronDB front-ends (REST, Redis, SQL) reuse the same transaction core, so commits that belong to the same logical operation share the same `tx_id`. Special operations automatically set semantic flags. For instance, document imports mark the transaction with `bulk-load`, and restore flows add `rollback`.
+
+Inspect commit notes with standard Git tooling:
+
+```
+git log --show-notes=chrondb
+git notes --ref=chrondb show <commit-hash>
+```
+
+You can parse the JSON payload to correlate commits with external systems or audit trails.
+
 ### Event Hooks
 
 #### Register Hook
@@ -263,3 +285,5 @@ Example error handling:
     (log/error "Storage error:" (.getMessage e)))
   (catch chrondb.exceptions.ValidationException e
     (log/error "Validation error:" (.getMessage e))))
+
+```

docs/architecture.md

@@ -44,6 +44,25 @@ ChronDB is built in layers:
 4. Indices are updated to reflect changes
 5. Reads can access any point in time using specific commits
 
+### Transaction Metadata via Git Notes
+
+Every commit recorded by ChronDB receives a Git note under the `chrondb` ref. The note payload is JSON and contains:
+
+- A transaction identifier (`tx_id`) shared by all commits that belong to the same logical operation
+- The origin (for example `rest`, `redis`, `sql`, `cli`)
+- Optional user information and request correlation ids
+- Semantic flags such as `bulk-load`, `rollback`, `migration`, or `automated-merge`
+- Additional metadata supplied by the protocol handler (HTTP endpoint, Redis command, SQL table, etc.)
+
+These notes provide an append-only audit trail without mutating commit messages or tracked files. Operators can inspect them using standard tooling:
+
+```
+git log --show-notes=chrondb
+git notes --ref=chrondb show <commit>
+```
+
+Because they live outside the object graph, notes can be replicated, filtered, and queried independently of the document contents while preserving Git’s immutable history.
+
 ### Indexing Layer Details
 
 The Lucene layer receives document mutations from the storage layer and updates the appropriate secondary indexes. It maintains statistics about term distributions and query plans so that complex requests—such as multi-field boolean filters or temporal slices—can be executed without scanning entire collections. When a query arrives, the planner determines the optimal combination of indexes, warms the cache when necessary, and streams results back to the access layer. Geospatial fields are stored in BKD trees, while full-text fields use analyzers that can be tuned per collection.

docs/examples-clojure.md

@@ -147,6 +147,27 @@ ChronDB supports atomic transactions:
 ;; If any operation fails, all changes are rolled back
 ```
 
+You can customise the transaction metadata written to Git notes via the `chrondb.transaction.core` helpers:
+
+```clojure
+(require '[chrondb.transaction.core :as tx])
+
+(tx/with-transaction [db {:origin "cli"
+                          :user "admin"
+                          :flags ["migration"]
+                          :metadata {:request "seed-dataset"}}]
+  ;; Flags can be appended dynamically as context evolves
+  (tx/add-flags! "bulk-load")
+  (tx/merge-metadata! {:batch-size 3})
+
+  (doseq [doc [{:id "product:1" :name "Laptop" :price 1200}
+               {:id "product:2" :name "Phone" :price 800}
+               {:id "product:3" :name "Tablet" :price 600}]]
+    (chrondb/save db (:id doc) doc)))
+
+;; Inspect the resulting Git notes with: git log --show-notes=chrondb
+```
+
 ## Advanced Features
 
 ### Custom Hooks

src/chrondb/api/redis/core.clj

@@ -5,6 +5,7 @@
             [chrondb.index.protocol :as index]
             [chrondb.query.ast :as ast]
             [chrondb.util.logging :as log]
+            [chrondb.transaction.core :as tx]
             [clojure.string :as str]
             [clojure.data.json :as json]
             [clojure.core.async :as async])
@@ -23,6 +24,43 @@
 (def RESP_OK "+OK\r\n")
 (def RESP_PONG "+PONG\r\n")
 
+(defn- normalize-flags [flags]
+  (letfn [(spread [value]
+            (cond
+              (nil? value) []
+              (and (coll? value) (not (string? value))) (mapcat spread value)
+              :else [value]))]
+    (->> (spread flags)
+         (keep identity)
+         (map str)
+         (remove str/blank?)
+         distinct
+         vec
+         not-empty)))
+
+(defn- redis-command-metadata
+  [command args extra]
+  (merge {:command command
+          :arg-count (count args)}
+         (when (seq args)
+           {:args (mapv str (take 8 args))})
+         (or extra {})))
+
+(defn- redis-tx-options
+  [command args {:keys [flags metadata]}]
+  (let [normalized (normalize-flags flags)
+        meta (redis-command-metadata command args metadata)]
+    (cond-> {:origin "redis"
+             :metadata meta}
+      normalized (assoc :flags normalized))))
+
+(defn execute-redis-write
+  ([storage command args f]
+   (execute-redis-write storage command args {} f))
+  ([storage command args opts f]
+   (tx/with-transaction [storage (redis-tx-options command args opts)]
+     (f))))
+
 ;; RESP Protocol Serialization Functions
 (defn write-simple-string [writer s]
   (.write writer (str RESP_SIMPLE_STRING s CRLF)))
@@ -132,17 +170,21 @@
     (write-error writer "ERR wrong number of arguments for 'set' command")
     (let [key (first args)
           value (second args)
-          doc {:id key :value value}
-          _ (storage/save-document storage doc)
-          _ (when index (index/index-document index doc))]
-      (.write writer RESP_OK))))
+          doc {:id key :value value}]
+      (execute-redis-write storage "SET" args {:metadata {:key key}}
+        (fn []
+          (storage/save-document storage doc)
+          (when index (index/index-document index doc))
+          (.write writer RESP_OK))))))
 
 (defn handle-del [storage writer args]
   (if (empty? args)
     (write-error writer "ERR wrong number of arguments for 'del' command")
-    (let [key (first args)
-          result (storage/delete-document storage key)]
-      (write-integer writer (if result 1 0)))))
+    (let [key (first args)]
+      (execute-redis-write storage "DEL" args {:metadata {:key key} :flags ["delete"]}
+        (fn []
+          (let [result (storage/delete-document storage key)]
+            (write-integer writer (if result 1 0))))))))
 
 (defn handle-command [_storage _index writer _args]
   (.write writer RESP_OK))
@@ -162,9 +204,11 @@
           value (nth args 2)
           doc {:id key :value value :expire-at (+ (System/currentTimeMillis) (* seconds 1000))}]
       (when seconds
-        (storage/save-document storage doc)
-        (when index (index/index-document index doc))
-        (.write writer RESP_OK)))))
+        (execute-redis-write storage "SETEX" args {:metadata {:key key :ttl seconds}}
+          (fn []
+            (storage/save-document storage doc)
+            (when index (index/index-document index doc))
+            (.write writer RESP_OK)))))))
 
 (defn handle-setnx [storage index writer args]
   (if (< (count args) 2)
@@ -174,10 +218,11 @@
           existing-doc (storage/get-document storage key)]
       (if existing-doc
         (write-integer writer 0) ; Key exists, don't set
-        (do
-          (storage/save-document storage {:id key :value value})
-          (when index (index/index-document index {:id key :value value}))
-          (write-integer writer 1))))))
+        (execute-redis-write storage "SETNX" args {:metadata {:key key}}
+          (fn []
+            (storage/save-document storage {:id key :value value})
+            (when index (index/index-document index {:id key :value value}))
+            (write-integer writer 1)))))))
 
 (defn handle-exists [storage writer args]
   (if (empty? args)
@@ -197,8 +242,10 @@
           doc {:id hash-key :value value :hash-key key :hash-field field}
           existing-doc (storage/get-document storage hash-key)
           is-new (nil? existing-doc)]
-      (storage/save-document storage doc)
-      (write-integer writer (if is-new 1 0)))))
+      (execute-redis-write storage "HSET" args {:metadata {:key key :field field}}
+        (fn []
+          (storage/save-document storage doc)
+          (write-integer writer (if is-new 1 0)))))))
 
 (defn handle-hget [storage writer args]
   (if (< (count args) 2)
@@ -218,12 +265,14 @@
           field-values (rest args)]
       (if (odd? (count field-values))
         (write-error writer "ERR wrong number of arguments for 'hmset' command")
-        (do
-          (doseq [[field value] (partition 2 field-values)]
-            (let [hash-key (str key ":" field)
-                  doc {:id hash-key :value value :hash-key key :hash-field field}]
-              (storage/save-document storage doc)))
-          (.write writer RESP_OK))))))
+        (let [field-count (/ (count field-values) 2)]
+          (execute-redis-write storage "HMSET" args {:metadata {:key key :field-count field-count}}
+            (fn []
+              (doseq [[field value] (partition 2 field-values)]
+                (let [hash-key (str key ":" field)
+                      doc {:id hash-key :value value :hash-key key :hash-field field}]
+                  (storage/save-document storage doc)))
+              (.write writer RESP_OK))))))))
 
 (defn handle-hmget [storage writer args]
   (if (< (count args) 2)
@@ -263,8 +312,10 @@
                        {:id key :type "list" :values []})
           updated-values (vec (concat values (:values list-doc)))
           updated-doc (assoc list-doc :values updated-values)]
-      (storage/save-document storage updated-doc)
-      (write-integer writer (count updated-values)))))
+      (execute-redis-write storage "LPUSH" args {:metadata {:key key :value-count (count values)}}
+        (fn []
+          (storage/save-document storage updated-doc)
+          (write-integer writer (count updated-values)))))))
 
 (defn handle-rpush [storage writer args]
   (if (< (count args) 2)
@@ -275,8 +326,10 @@
                        {:id key :type "list" :values []})
           updated-values (vec (concat (:values list-doc) values))
           updated-doc (assoc list-doc :values updated-values)]
-      (storage/save-document storage updated-doc)
-      (write-integer writer (count updated-values)))))
+      (execute-redis-write storage "RPUSH" args {:metadata {:key key :value-count (count values)}}
+        (fn []
+          (storage/save-document storage updated-doc)
+          (write-integer writer (count updated-values)))))))
 
 (defn handle-lrange [storage writer args]
   (if (not= (count args) 3)
@@ -313,8 +366,10 @@
               first-value (first values)
               updated-values (vec (rest values))
               updated-doc (assoc list-doc :values updated-values)]
-          (storage/save-document storage updated-doc)
-          (write-bulk-string writer first-value))
+          (execute-redis-write storage "LPOP" args {:metadata {:key key}}
+            (fn []
+              (storage/save-document storage updated-doc)
+              (write-bulk-string writer first-value))))
         (write-bulk-string writer nil)))))
 
 (defn handle-rpop [storage writer args]
@@ -327,8 +382,10 @@
               last-value (last values)
               updated-values (vec (butlast values))
               updated-doc (assoc list-doc :values updated-values)]
-          (storage/save-document storage updated-doc)
-          (write-bulk-string writer last-value))
+          (execute-redis-write storage "RPOP" args {:metadata {:key key}}
+            (fn []
+              (storage/save-document storage updated-doc)
+              (write-bulk-string writer last-value))))
         (write-bulk-string writer nil)))))
 
 (defn handle-llen [storage writer args]
@@ -351,8 +408,10 @@
           new-members (filter #(not (contains? existing-members %)) members)
           updated-members (apply conj existing-members members)
           updated-doc (assoc set-doc :members updated-members)]
-      (storage/save-document storage updated-doc)
-      (write-integer writer (count new-members)))))
+      (execute-redis-write storage "SADD" args {:metadata {:key key :member-count (count members)}}
+        (fn []
+          (storage/save-document storage updated-doc)
+          (write-integer writer (count new-members)))))))
 
 (defn handle-smembers [storage writer args]
   (if (empty? args)
@@ -384,8 +443,11 @@
               removed-members (filter #(contains? existing-members %) members)
               updated-members (apply disj existing-members members)
               updated-doc (assoc set-doc :members updated-members)]
-          (storage/save-document storage updated-doc)
-          (write-integer writer (count removed-members)))
+          (execute-redis-write storage "SREM" args {:metadata {:key key :member-count (count members)}
+                                                    :flags ["delete"]}
+            (fn []
+              (storage/save-document storage updated-doc)
+              (write-integer writer (count removed-members)))))
         (write-integer writer 0)))))
 
 ;; Sorted Set commands
@@ -405,8 +467,10 @@
                                   existing-members
                                   score-member-pairs)
           updated-doc (assoc zset-doc :members updated-members)]
-      (storage/save-document storage updated-doc)
-      (write-integer writer new-members))))
+      (execute-redis-write storage "ZADD" args {:metadata {:key key :member-count (count score-member-pairs)}}
+        (fn []
+          (storage/save-document storage updated-doc)
+          (write-integer writer new-members))))))
 
 (defn handle-zrange [storage writer args]
   (if (< (count args) 3)
@@ -471,8 +535,11 @@
               removed-count (count (filter #(contains? existing-members %) members))
               updated-members (apply dissoc existing-members members)
               updated-doc (assoc zset-doc :members updated-members)]
-          (storage/save-document storage updated-doc)
-          (write-integer writer removed-count))
+          (execute-redis-write storage "ZREM" args {:metadata {:key key :member-count (count members)}
+                                                    :flags ["delete"]}
+            (fn []
+              (storage/save-document storage updated-doc)
+              (write-integer writer removed-count))))
         (write-integer writer 0)))))
 
 (defn handle-search