Inline small contents in nodes by cuihtlauac · Pull Request #2390 · mirage/irmin

cuihtlauac · 2026-01-21T17:06:44Z

Summary

This PR continues the work from #2378, implementing inline contents for irmin-pack. Small content values (≤13 bytes raw, <16 bytes serialized) are stored directly within parent node entries rather than as separate pack file entries.

Key changes:

Add inline_contents configuration option (default: false)
Introduce Inode_v3 format with new magic bytes (S for root, T for non-root)
Extend node value type with Contents_inlined variant
Update tree module to check content size and inline when appropriate

Benefits:

Reduced storage overhead for small values
Fewer I/O operations (one read instead of two)
Better cache locality

Limitations:

Enabling inlining changes node hashes (stores not hash-compatible)
16-byte threshold is fixed
Not retroactive (existing contents not automatically inlined)

Test plan

All existing tests pass with inlining disabled
New inlined_contents test suite validates correctness with inlining enabled
Structure verification confirms small contents are stored inline
Manual testing with real-world data

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.5 noreply@anthropic.com

zshipko · 2026-02-12T21:34:55Z

+let inline_contents_enabled = ref false
+let set_inline_contents_enabled v = inline_contents_enabled := v


This seems error-prone, we should either enable it globally or figure out a better way to configure it on a per-store basis. Maybe it could be stored in the config.

I think that is overwritten for each call to Repo.v. Let's store that in the store's config.

zshipko · 2026-02-12T22:12:21Z

+let inline_contents_max_bytes = ref 48
+let set_inline_contents_max_bytes v = inline_contents_max_bytes := v
+let get_inline_contents_max_bytes () = !inline_contents_max_bytes


Same as above, maybe we should pick a single default value or store this in the config too.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Call Contents.export in on_contents_inlined to cache the key - Handle Contents_inlined_3 in add_node_map like regular Contents Inlined contents are now correctly saved and retrieved. All inline contents tests pass. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Regenerate V3 test archives with new node hashes resulting from the node_entry format change (adding inlined field). The hash computation for nodes containing node references has changed. Changes: - Add generators (gen_minimal.ml, gen_always.ml, gen_gced.ml) for reproducible test archive regeneration - Regenerate version_3_minimal, version_3_always, version_3_minimal_gced archives with Inode_v3 magic bytes ('S', 'T') - Update hash constants in test_upgrade.ml for n0, c0, n1, c1, c2 - Update expected hex values in test_dispatcher.ml for node_2, commit_2, node_3 (hash and magic byte changes) - Update expected magic byte in test_inode.ml ('R' -> 'S') - Disable From_v2 upgrade test (V2 archives have incompatible old hashes) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add design document (doc/irmin-pack/design/inline_contents.md) explaining the motivation, design, and implementation of inlining small content values within node entries - Update version-history.md with Inode_v3 format details and magic bytes ('S'/'T' for root/nonroot) - Enhance conf.mli documentation for inline_contents option - Add test_inlining_structure test verifying small contents are actually stored inline (threshold: 13 bytes raw / 16 bytes serialized) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

This implements the core inline contents feature, allowing small content values (< 16 bytes serialized) to be stored directly within node entries rather than as separate pack file entries. Key changes: Core Irmin: - Add `Contents_inlined` variant to node value type - Add `Contents_inlined_3` to tree destruct type - Extend Tree.of_contents to inline small values when enabled - Update node interfaces and implementations Irmin-pack: - Add Inode_v3_root ('S') and Inode_v3_nonroot ('T') magic bytes - Add Contents_inlined_value to inode encoding - Update GC worker, snapshot, and traversal to handle inlined contents - Update checks and integrity verification Other: - Update irmin-git, irmin-graphql, irmin-tezos for new node value type - Update irmin-pack-tools (tezos_explorer) for new entry kinds - Update tests to handle inlined contents in assertions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Replace all numbered variants (Contents_inlined_3, Contents_inlined_5, Contents_inlined__1, etc.) with a single Contents_inlined name. OCaml's structural typing for polymorphic variants distinguishes them by payload type, so the same name works correctly across different contexts (unit markers, contents+metadata values, keys, hashes). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Benchmark compares write and read performance with and without inline_contents enabled, for different content sizes (small, medium, large). Usage: dune exec bench/irmin-pack/bench_inlined_contents.exe -- -n 2000 -c 5 -s small Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

This simplifies the tree type by removing Contents_inlined from the internal representation. The inlining decision is now made at export time based on serialized content size. Changes: - Remove Contents_inlined from tree.t, tree.elt, and related types - Add should_inline_contents helper in export function - Inline contents <= 16 bytes at serialization time - Simplify pattern matches throughout the codebase - Remove debug print statements The backend kinded_hash/kinded_key types still support Contents_inlined for the on-disk format. Import functions convert inlined contents back to regular Contents values. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Use Contents.to_value instead of cached_value to ensure value is available - Add 2 bytes overhead to size check (1 byte variant tag + 1 byte varint length) - This fixes the inlining structure test which expects 3 inlined contents Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

New benchmark suite in test/irmin-pack/bench_inline/ to measure inline contents performance across different size distributions: - Uniform (small, large, around-threshold) - Bimodal (mostly-small, mostly-large) - Log-normal (small, medium) - Zipfian (small, medium, steep) Metrics collected: - Store size on disk - Write time - Read latency percentiles (p50, p99, p999) - Inlined vs non-inlined content counts Usage: dune exec test/irmin-pack/bench_inline/main.exe -- run --all dune exec test/irmin-pack/bench_inline/main.exe -- sweep -t 16 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Benchmarks across multiple size distributions (zipfian, log-normal, bimodal) showed that 48 bytes provides better read latency improvement than 16 bytes across most workloads. Changes: - Update default inline_contents_max_bytes from 16 to 48 - Make threshold configurable via set_inline_contents_max_bytes - Add get_inline_contents_max_bytes to query current threshold - Add 'optimize' command to benchmark for threshold tuning - Update test expectations for new threshold Benchmark results showed: - zipfian-steep: p99 latency 139µs -> 84µs at threshold=48 - zipfian-small: p99 latency 166µs -> 108µs at threshold=48 - log-normal-small: p99 latency 229µs -> 111µs at threshold=48 - mostly-small: p99 latency 162µs -> 87µs at threshold=48 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add README.md with usage instructions for all benchmark commands - Add gnuplot scripts for generating comparison charts - Add sample data file and generated PNG plots Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add bench_inline_storage.gp gnuplot script for storage visualization - Add generate_data.sh to parse CSV output into gnuplot data files - Add 'make bench-inline' target to run benchmarks and generate plots - Update README with storage impact documentation and quick start Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add flush before close in Repo.close for read-write stores. Previously, direct node operations that didn't go through batch() could leave unflushed data, causing close to fail with Pending_flush. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

The kinded_key type changed to support inline contents: - `Contents of key * metadata` (was `Contents of key`) - `Contents_inlined of string * metadata` (new) - `Node of key * key list` (was `Node of key`) This commit fixes all call sites to use the new tuple format: - save_tree now returns correct kinded_key with metadata - Store.key returns full Tree.kinded_key - Pattern matches updated in examples and tests - stat.t output updated for Inode_v3 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

balat · 2026-03-24T14:55:29Z

+       the new hash computation introduced with Inode_v3 for content inlining.
+       TODO: Re-enable with separate expected values for V2 migration testing.
+       Alcotest.test_case "upgrade From_v2" `Quick (test ~fs ~domain_mgr From_v2);
+    *)


What happens when we open a store v2 with this version? If I understand well, there will be a mix between v2 and v3 nodes? But what about the hashes? They will be different. Is it an issue? This could maybe introduce duplication of data, or break the integrity check?

balat · 2026-03-24T14:57:54Z

        failwith
          "Persisting an empty node is forbidden by the configuration of the \
-           irmin-pack store";
+           irmin-pack store"; *)


Why is this commented?

balat · 2026-03-24T15:09:19Z

            let name = step n in
            let hash = key h in
-            (name, `Node hash)
+            (* TODO inline *)


?
It seems the second parameter is always [] in the current version?

balat · 2026-03-24T15:13:49Z

+
+```ocaml
+type value =
+  [ `Node of node_key * contents_key list


What is the purpose of this list?

balat · 2026-03-24T15:15:02Z

      | Contents ->
          progress_contents ();
-          check ~kind:`Contents ~offset ~length k
+          check ~kind:`Contents ~offset ~length k (* TODO inlined ?*)


balat · 2026-03-24T15:15:20Z

        match kinded_key with
-        | `Contents key -> (key, Contents)
-        | `Inode key | `Node key -> (key, Node)
+        | `Contents key -> Some (key, Contents) (* TODO inline *)


balat · 2026-03-24T15:15:56Z

+    Merge.alist S.step_t
+      (Type.pair S.node_key_t (Type.list S.contents_key_t))
+      (fun _step -> merge_key)
+  (* TODO: inline *)


balat · 2026-03-24T15:16:16Z

-      | `Node n -> hash_node n
+      | `Contents_inlined c -> hash_contents c
+      | `Node (n, _) -> hash_node n
+      (* TODO inline *)


balat · 2026-03-24T15:16:32Z

-              | Some k -> Some (`Node k))
+              | Some k ->
+                  (* TODO inline *)
+                  Some (`Node (k, [])))


balat · 2026-03-24T15:16:53Z

          | None -> None
-          | Some k -> of_key r (`Node k))
+          | Some k ->
+              (* TODO inline *)


balat · 2026-03-24T15:17:08Z

        List.iter
-          (fun k ->
+          (fun (k, _il) ->
+            (* TODO inline *)


balat · 2026-03-24T15:17:22Z

              (* the tree cache needs to be invalidated *)
              let tree =
-                Tree.import_no_check (repo t) (`Node (Commit.node h))
+                (* TODO inline *)


balat · 2026-03-24T15:17:41Z

        | Some c ->
            let node = X.Commit.Val.node c in
-            [ `Node node ]
+            (* TODO inline *)


balat · 2026-03-24T15:18:18Z

          | Some commit ->
              List.iter schedule_commit (Commit_value.parents commit);
-              schedule_kinded (`Node (Commit_value.node commit)))
+              (* TODO inline *)


balat · 2026-03-24T15:18:40Z

    in
-    let add v = if Val.is_empty v then None else Some (add t v) in
-    Merge.like_blocking [%typ: Key.t option] merge read add
+    (* TODO inline *)


balat · 2026-03-24T15:21:05Z

-            | `Node _ -> null contents))
+            | `Node _ -> null contents
+            | `Contents_inlined _ ->
+                (* TODO: handle inlined contents *)


balat · 2026-03-24T15:37:45Z

+          | `Contents (c, perm) -> Some (to_git (perm :> Git.Tree.perm) (v c))
+          | `Contents_inlined _ ->
+              (* Git backend does not support inlined contents *)
+              None)


failwith like in add?

balat · 2026-03-24T15:56:39Z

 (env
 (dev
-  (flags :standard -w -unused-functor-parameter)))
+  (flags :standard -w -unused-functor-parameter -warn-error -A)))


We probably want to remove this change before merging

balat · 2026-03-24T16:04:47Z

+
+### Inlining Threshold
+
+Content values are inlined when their **serialized size** is less than 16 bytes. The serialized size includes:


Default is 48 in tree.ml

balat · 2026-03-24T16:10:27Z

+      let _, to_inline = List.split to_inline in
+      let node = P.Node.Val.of_seq node_seq to_inline in
+      let r = add_node n node to_inline k in
+      (* assert false; *)


balat · 2026-03-24T16:10:53Z

        | `Contents _ -> Error "cannot add contents at the root"
-        | `Node t ->
+        | `Node (t, _il) ->
+            (* assert false; *)


balat · 2026-03-24T16:11:07Z

+        | `Node (t, _il) ->
+            (* assert false; *)
            let node = Tree.export ~clear r contents_t node_t t in
+            (* assert false; *)


balat · 2026-03-24T16:25:34Z

  with_reporter ~config bar

-module Conf = Irmin_tezos.Conf
+module Conf = struct


Why are we changing this?

balat · 2026-03-24T16:29:12Z

      | Some _ -> remove layout t s k Fun.id |> stabilize_root layout

-    let of_seq la l =
+    let of_seq la l _to_inline =


what is _to_inline supposed to do?

cuihtlauac force-pushed the inline-small-objects-v2 branch 2 times, most recently from 2f5ec29 to 88e9295 Compare January 29, 2026 07:47

cuihtlauac marked this pull request as ready for review January 29, 2026 08:51

zshipko reviewed Feb 12, 2026

View reviewed changes

cuihtlauac force-pushed the inline-small-objects-v2 branch 2 times, most recently from 1e565a8 to 88e9295 Compare March 9, 2026 15:01

clecat and others added 19 commits March 9, 2026 16:02

fix irmin-pack-tools

a82a179

tmp

6815d69

WIP: Add inline_contents runtime toggle and fix type mismatches

5601057

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add benchmark documentation and visualization

496c366

- Add README.md with usage instructions for all benchmark commands - Add gnuplot scripts for generating comparison charts - Add sample data file and generated PNG plots Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Formatting

807cea4

Formatting

57ba56d

cuihtlauac force-pushed the inline-small-objects-v2 branch from 88e9295 to 57ba56d Compare March 9, 2026 16:08

balat reviewed Mar 24, 2026

View reviewed changes

Comment thread src/irmin/store.ml

| None -> None

| Some k -> of_key r (`Node k))

| Some k ->

(* TODO inline *)

Copy link
Copy Markdown

Contributor

balat Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

balat reviewed Mar 24, 2026

View reviewed changes

Comment thread src/irmin/store.ml

List.iter

(fun k ->

(fun (k, _il) ->

(* TODO inline *)

Copy link
Copy Markdown

Contributor

balat Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

balat reviewed Mar 24, 2026

View reviewed changes

balat mentioned this pull request Apr 3, 2026

Inline small contents in nodes (cleaned up) #2400

Open

4 tasks

cuihtlauac mentioned this pull request Apr 10, 2026

Risk assessment: promoting eio branch to main #2401

Open

8 tasks

		let inline_contents_enabled = ref false
		let set_inline_contents_enabled v = inline_contents_enabled := v


		### Inlining Threshold

		Content values are inlined when their serialized size is less than 16 bytes. The serialized size includes:

Conversation

cuihtlauac commented Jan 21, 2026

Summary

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

balat Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

balat Mar 24, 2026 •

edited

Loading