irmin-lwt: Lwt compatibility layer for Irmin 4#2405
Draft
balat wants to merge 40 commits intomirage:eiofrom
Draft
irmin-lwt: Lwt compatibility layer for Irmin 4#2405balat wants to merge 40 commits intomirage:eiofrom
balat wants to merge 40 commits intomirage:eiofrom
Conversation
New package for the Lwt compatibility layer over Irmin 4's direct-style API. This commit sets up the empty skeleton (dune, opam, .ml, .mli) so the package is buildable before the implementation lands in follow-up commits. See issue mirage#2401 phase 3 and comment #4223023836 for the design. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add the [Make] functor that wraps an Irmin 4 store's I/O-performing operations so they return ['a Lwt.t] values. Each wrapper threads its call through [Lwt_eio.run_eio] to run the direct-style body on the current Eio scheduler. Covered in this first pass: [Repo.v/close/heads/branches/config/export], [main], [of_branch], [of_commit], [empty], the store accessors (find, mem, get, find_tree, get_tree, hash, find_all), the update operations (set, set_exn, set_tree, set_tree_exn, remove, remove_exn), [merge_into] and [last_modified]. Pure accessors ([repo], [tree], [status]) are forwarded as-is without scheduler round-trip. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Expose two top-level helpers that set up the Eio/Lwt bridge for client code: - [run f] wraps [Eio_main.run] + [Lwt_eio.with_event_loop] so Irmin 3-style programs can replace their [Lwt_main.run main] with [Irmin_lwt.run main] at the entry point. - [run_with_env env f] is the same but reuses an existing Eio environment, for clients already inside an Eio event loop. This requires [eio_main] as a library dependency (bumped from with-test to a hard dep in the opam file). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three small tests over the in-memory backend to confirm that the Lwt-wrapped [Repo.v], [main], [set_exn], [remove_exn], [find] and [Repo.close] traverse the [lwt_eio] bridge correctly: - set then find returns the stored value - remove clears a previously set value - find on an unset path returns None The test binary uses [Irmin_lwt.run] to set up Eio and lwt_eio, and alcotest-lwt to schedule the Lwt-returning cases. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A realistic-shaped test that seeds the main branch, forks a feature branch, adds an entry on the feature, merges it back into main, and confirms all three entries are visible along with a non-empty [last_modified] history for the merged path. This exercises [of_branch], [merge_into] and [last_modified] through the Lwt-wrapped API — the idioms typical Irmin 3 consumers (notably Tezos' context) rely on. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three tests covering the subtle cases where a wrapper could break real applications by not forwarding Lwt's scheduling semantics correctly: - [Lwt.catch] around a call to [Store.get] on a missing path catches the [Invalid_argument] exception raised by the direct-style Irmin. - [Lwt.pause] between two Irmin ops does not disturb the store state. - 50 concurrent reads dispatched via [Lwt.all] all complete and return the same value, exercising the bridge under Lwt-side concurrency. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two-step migration path: 1. Move to irmin-lwt with minimal code changes (opam swap, functor wrapping, entry-point change). Existing Lwt idioms keep working. 2. At your own pace, drop irmin-lwt call by call to switch to direct style. The document also lists the Irmin 3 to 4 breaking changes that the compatibility layer cannot hide (OCaml 5.1+, config renames, removed APIs, yield-point semantics) and describes the initial scope of the wrapper (top-level Store ops only for now; submodule wrappers to come). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Expose [Irmin_lwt.Make(S).Tree] mirroring [S.Tree] with Lwt-returning variants of every I/O-triggering operation: [kind], [diff], [mem], [find], [find_all], [find_tree], [get], [get_all], [get_tree], [list], [seq], [length], [add], [add_tree], [update], [update_tree], [remove], [mem_tree], [stats], [to_concrete], [find_key], [of_key], [of_hash]. Pure constructors and inspectors ([empty], [singleton], [of_contents], [of_node], [v], [pruned], [is_empty], [destruct], [hash], [kinded_hash], [key], [shallow], [clear], [of_concrete], [pp]) are forwarded as-is. [Tree.fold] is intentionally not included here because its callbacks have Lwt return types that need bridging via [Lwt_eio.Promise.await_lwt] — that will be added in a follow-up commit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Expose [Irmin_lwt.Make(S).Commit] mirroring [S.Commit]: the constructor [v] and the lookups [of_key], [of_hash] return Lwt promises. Pure accessors ([tree], [parents], [info], [hash], [key], [pp], [pp_hash]) are forwarded as-is. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cover the branch-level store operations ([Branch.mem], [find], [get], [set], [remove], [list], [watch], [watch_all]), the head-level ones ([Head.list], [find], [get], [set], [fast_forward], [test_and_set], [merge]), and the top-level [watch], [watch_key], [unwatch]. Watch callbacks are in Lwt ([_ -> unit Lwt.t]) as in Irmin 3; the wrapper bridges them to direct-style callbacks by running the returned promise through [Lwt_eio.Promise.await_lwt] so the Irmin 4 watcher can call them synchronously from its own fibre. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Expose a Lwt-aware [Tree.fold] that accepts [folder_lwt] callbacks of type [path -> 'b -> 'a -> 'a Lwt.t] (and [force] with a Lwt-returning [`False] branch), then bridges each callback to the direct-style [S.Tree.fold] by awaiting its Lwt promise through [Lwt_eio.Promise.await_lwt]. The overall call is then wrapped in [run_eio] so [fold] returns ['a Lwt.t]. Together with the rest of [Tree], this covers the fold-based traversals typical Irmin 3 consumers (e.g. Tezos context) rely on. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Four new tests for the newly wrapped submodules: - Tree.add/find round-trip exercises the pure-ish tree ops and the Lwt-wrapped lookups. - Tree.fold with a Lwt-returning [contents] folder confirms the callback bridging via [Lwt_eio.Promise.await_lwt] works end-to-end. - Commit.v + Branch.set/find verifies the commit constructor and branch round-trip through the Lwt layer. - Head.find on a fresh branch checks the empty-then-populated head transition; the test uses a unique branch name since Irmin_mem shares its hashtable across repo handles. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Now that [Tree], [Commit], [Branch], [Head] and the top-level watches are wrapped, rewrite the "Scope" section to reflect the actual coverage. Keep the escape hatch (call [Lwt_eio.run_eio] directly) documented for the few helpers that are still not wrapped (the [Sync] functor, a few rarely-used [Repo] helpers). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
8 tasks
Expose the full type surface of [Irmin.Generic_key.S] from [Make(S)]: - Add the missing type aliases: [step], [metadata], [node], [slice], [contents_key], [node_key], [commit_key], [lca_error], [ff_error]. - Re-export the type-level submodules: [Schema], [Info], [Hash], [Path], [Metadata], [Backend], [Contents], [History], [Status] via [module type of] (aliasing functor arguments is forbidden, so we use a structural re-export). This unblocks downstream consumers that need to apply other Irmin functors on top of an [Irmin_lwt.Make(S)] result (e.g. Tezos' [Tezos_context_helpers.Context.Make_tree]). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These are the named gaps reported by the Octez compilation test as required but missing from [Irmin_lwt.Make(S)]'s output when applied as the [Store] argument of [Tezos_context_helpers.Context.Make_tree]: - [master]: deprecated Irmin 3 alias of [main], retained for compatibility. Wrapped like [main]. - [of_backend_node], [to_backend_node], [to_backend_portable_node], [to_backend_commit], [of_backend_commit]: pure converters between frontend and backend representations, forwarded as-is. - [save_contents], [save_tree]: persist to the backend store, wrapped with [run_eio]. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wrap the rest of [Irmin.Generic_key.S]'s I/O-performing operations that were not in the initial MVP, so a [Irmin_lwt.Make(S)] result carries essentially the full Lwt-flavored Store API: - [commit_t] (pure type descriptor, forwarded) - [test_and_set], [test_and_set_exn], [test_and_set_tree], [test_and_set_tree_exn] - [test_set_and_get], [test_set_and_get_exn], [test_set_and_get_tree], [test_set_and_get_tree_exn] - [merge], [merge_exn], [merge_tree], [merge_tree_exn] - [merge_with_branch], [merge_with_commit] - [with_tree], [with_tree_exn] - [clone] - [lcas], [lcas_with_branch], [lcas_with_commit] - [history] All are straightforward [run_eio] wrappers over the direct-style [S] counterparts. This should close the remaining gap mirage#4 flagged by the Octez compilation test — downstream consumers that reused a wide slice of Irmin 3's Store API can now point at [Irmin_lwt.Make(S)] without hitting "value X is required but not provided". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Introduce a [Irmin_lwt.Pack.Make] functor that takes an [Irmin_pack_io.S] store and returns a module that includes the full [Make(S)] Lwt-wrapped generic-key API plus the pack-unix extensions: - integrity checks: [integrity_check], [integrity_check_inodes], [traverse_pack_file], [test_traverse_pack_file] - chunking: [split], [is_split_allowed], [add_volume] - on-disk: [reload], [flush], [create_one_commit_store] - [Gc]: [start_exn], [finalise_exn], [run] (with Lwt-returning [finished] callback), [wait], [cancel], [is_finished], [behaviour], [is_allowed], [latest_gc_target] - [Snapshot]: re-export with Lwt-wrapped [export] Adds [irmin-pack.io] as a library dependency and [irmin-pack] as an opam dependency. Addresses gaps 5-10 of the Octez compilation test: consumers that need pack-unix operations (Tezos context GC, snapshot export, etc.) can now apply [Irmin_lwt.Pack.Make] on their pack-unix store instead of reaching into [Irmin_pack_io] directly via [Lwt_eio.run_eio]. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The shorter [module X : module type of S.X] idiom produces a fresh signature with abstract types: from a consumer's point of view, [Backend.Contents.t] was not definitionally equal to [S.Backend.Contents.t]. That equality is needed when a downstream functor (e.g. Tezos' [Tezos_context_helpers.Context.Make_tree]) receives values whose types reference both the frontend [Backend.Contents.t] and the original [S.Backend.Contents.t]. Switching to [module X : module type of struct include S.X end] captures the signature through a structural inclusion and preserves the type equalities. Applied to all nine type-level re-exports (Schema, Info, Hash, Path, Metadata, Backend, Contents, History, Status). Reported by the Octez compilation test as "Backend.Contents.t vs S.Backend.Contents.t" type mismatch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each type in [Irmin.Generic_key.S] that is annotated [@@deriving irmin] produces a corresponding [_t : _ Irmin.Type.t] value that downstream code uses for encoding, hashing, pretty-printing, etc. Forward them all: [step_t], [path_t], [metadata_t], [contents_t], [node_t], [tree_t], [hash_t], [branch_t], [slice_t], [info_t], [lca_error_t], [ff_error_t], [contents_key_t], [node_key_t], [commit_key_t], [write_error_t]. They are pure, forwarded without [run_eio]. Reported missing by the Octez compilation test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These store-level accessors are present in [Irmin.Generic_key.S] but were missed by the initial sweep. All perform I/O (lazy loading from the backend is possible), so they are wrapped with [run_eio]. Also exposes [type kinded_key = [ `Contents of contents_key | `Node of node_key ]] as a named type so downstream consumers can reference it (previously inlined in [save_tree]). Reported missing by the Octez compilation test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…remote Three related completions of [Irmin.Generic_key.S] surface: - Expose [type 'a merge] as the Lwt-wrapped abbreviation used by [merge_into], [merge_with_branch] and [merge_with_commit]. Downstream consumers may reference the alias directly. - Fix [merge_into]: the previous signature was missing the [?max_depth] and [?n] optional parameters carried by [t merge]. Now properly [into:t -> t merge]. - Extend the top-level [Irmin.remote] with [E of Backend.Remote.endpoint], matching the extension that the underlying [S] exposes. This allows downstream code that matches on the extensible variant to see the endpoint carried by a remote produced through [Irmin_lwt.Make(S)]. Reported missing by the Octez compilation test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extend [Irmin_lwt.Make(S).Repo] to cover the rest of [S.Repo]: - [type elt] and [elt_t]: the topological element variant (Commit, Node, Contents, Branch) and its [Irmin.Type.t] descriptor. Exposed as a concrete polymorphic variant so callers can pattern-match on it. - [default_pred_commit], [default_pred_node], [default_pred_contents]: pure forwarding (no I/O). - [import]: Lwt-wrapped. - [iter] and [breadth_first_traversal]: Lwt-wrapped; each of the 13 optional callbacks (edge/branch/commit/node/contents/skip_*/pred_*) is accepted as Lwt-returning (matching Irmin 3) and bridged to the direct-style call expected by the underlying Irmin 4 traversal via [Lwt_eio.Promise.await_lwt]. Closes the 8 Repo-level items flagged by the second Octez compilation pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract the Lwt-flavoured signature produced by [Make] into a top-level
[module type S] and have [Make] return a module conforming to it with
explicit type and module equalities. This is the architectural piece
needed by Tezos' [Tezos_context_helpers.Context.DB]:
module type DB = Irmin.Generic_key.S with module Schema = Schema
stops type-checking against [Irmin_lwt.Make(S)]'s result because the
latter's [Repo.v] returns [t Lwt.t] instead of [t]. With this change,
downstream consumers can now write
module type DB = Irmin_lwt.S with module Schema = Schema
and type contents = value
and ...
and pass an [Irmin_lwt.Make(_)] module as [DB].
The [module type S] body is mirrored between [.ml] and [.mli] (both are
required — OCaml demands the module type declaration in both); the rest
of the implementation is unchanged. Tree's lazy [Contents] submodule is
re-exposed, and the store-level [Contents] is wrapped with
Lwt-returning [of_key]/[of_hash] to match the signature.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without these equalities, [Irmin_lwt.S with module Schema = Schema] in a downstream consumer (Tezos' [Tezos_context_helpers.Context.DB]) only substituted [Schema] but left [hash], [contents], [step], etc. abstract. Cue type errors like "Store.Tree.hash returns Store.hash but Tezos expects Hash.t (= Schema.Hash.t)". Mirror the pattern of [Irmin.Generic_key.S]: declare [Schema] first and define the types it derives as type aliases: type step = Schema.Path.step type path = Schema.Path.t type metadata = Schema.Metadata.t type contents = Schema.Contents.t type branch = Schema.Branch.t type info = Schema.Info.t type hash = Schema.Hash.t Now [Irmin_lwt.S with module Schema = Schema] propagates the equalities to every Schema-derived type. The redundant [type X = S.X] constraints in [Make]'s output type are dropped accordingly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The previous [Tree.Contents] in [module type S] only declared [t], [hash] and [key]. Octez (and any consumer that consults a lazy contents node) also needs: - [type error]: errors raised when forcing a lazy contents value ([Dangling_hash], [Pruned_hash], [Portable_value]). - [type 'a or_error]: result alias. - [val force]: returns the contents value or an error, Lwt-wrapped (the call may load from the backend). - [val force_exn]: same, but raises on error. Lwt-wrapped. - [val clear]: clears the cached value, pure. In the [Make] implementation, [Contents] now extends [S.Tree.Contents] through [include (S.Tree.Contents : module type of struct include … end with type t = …)] so the existing direct-style methods are forwarded and only [force]/[force_exn] are shadowed with their Lwt-wrapped versions. Reported missing by Octez compilation ([helpers/context.ml:118] uses [Store.Tree.Contents.force_exn]). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Octez (and any code that pattern-matches on lazy tree values) needs the
following [Tree] types to be transparent rather than abstract:
- [kinded_hash = [`Contents of hash * metadata | `Node of hash]]
- [kinded_key = [`Contents of contents_key * metadata | `Node of node_key]]
- [elt = [`Node of node | `Contents of contents * metadata]]
- [concrete = [`Tree of (step * concrete) list
| `Contents of contents * metadata]]
- [depth = [`Eq of int | `Le of int | `Lt of int | `Ge of int | `Gt of int]]
These were declared abstract in the previous [module type S] body, so
[match c with `Tree l -> _ | `Contents (v, _) -> _] (used in
[helpers/context.ml:152]) failed to type-check. Polymorphic variants
are structural in OCaml, so re-declaring them with the same constructor
shape lets values from [S.Tree.X] flow through without any coercion.
For [stats], which is a record (nominally typed), we keep the type
abstract and expose the [Irmin.Type.t] descriptor [stats_t] instead.
Field access is available through [Irmin.Type] introspection.
Reported by the third Octez compilation pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The same pattern as the Tree commit: [lca_error], [ff_error] and
[write_error] were declared abstract, preventing pattern matching on
them at use sites. Re-declare them with their full constructor shape:
- [lca_error = [`Max_depth_reached | `Too_many_lcas]]
- [ff_error = [`No_change | `Rejected | lca_error]]
- [write_error = [Irmin.Merge.conflict
| `Too_many_retries of int
| `Test_was of tree option]]
Now downstream code can pattern-match on the result of [merge_into],
[fast_forward], [lcas], [test_and_set], etc.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…rs, merge These were missing from the [Tree] submodule of [module type S]: - [val merge : t Irmin.Merge.t]: the merge value used by Irmin tree combinators. - [type counters], [val counters], [val dump_counters], [val reset_counters]: the performance counters shared between all trees backed by the same set of internal caches. [counters] is kept abstract since it is a record (nominal typing) and the underlying [S.Tree.counters] is abstract from inside the functor body. - [val inspect]: a synchronous inspector returning a transparent variant describing the kind and internal state of a tree node ([`Map], [`Key], [`Value], [`Portable_dirty], [`Pruned]). [helpers/context.ml] in Octez calls [inspect] for debugging / introspection paths. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Octez code that catches the Dangling_hash exception ([helpers/context.ml:227]) needs to pattern-match against [Store.Tree.Dangling_hash _]. Without the exception being declared in the Lwt module type, the match arm silently does not fire and the exception escapes. Re-export the three exceptions raised by Irmin's tree code with the standard OCaml constructor-aliasing syntax: exception Dangling_hash = S.Tree.Dangling_hash exception Pruned_hash = S.Tree.Pruned_hash exception Portable_value = S.Tree.Portable_value The exception value itself is shared, so [Store.Tree.Dangling_hash] (via [Irmin_lwt.Make(S).Tree.Dangling_hash]) and the underlying [S.Tree.Dangling_hash] are the same constructor. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…oof_state Round out [Tree] with the Merkle proof surface that upstream [Irmin.Generic_key.S.Tree] exposes (and that Octez relies on for context proofs). In [module type S.Tree]: - [module Proof] mirrors [Irmin.Proof.S] for the store's contents, hash, step and metadata. The [proof_tree] and [inode_tree] are exposed as transparent variants so callers can pattern-match on proof structure (Tezos uses this for Merkle proof serialization). - [type verifier_error = [`Proof_mismatch of string]]. - [val produce_proof], [val verify_proof] — Lwt-wrapped, with the user callback bridged via [Lwt_eio.Promise.await_lwt] (same idiom as [Tree.fold] and the watchers). - [val hash_of_proof_state] — pure forwarding. In [Make]'s [Tree], [Proof] re-uses [S.Tree.Proof]'s implementation by [include] with destructive substitution on [tree] and [t], then exposes them under their renamed names ([proof_tree], [proof]) matching the [module type S] declaration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tree) Match the naming convention of [Irmin.Generic_key.S.Tree.Proof]: the proof tree variant is [tree] (not [proof_tree]) and the proof type is [t] (not [proof]). Following upstream's idiom, the outer [Tree.t] is reachable inside [Proof] as [irmin_tree], a fresh abstract type that gets substituted away as [t] (the parent's [t]) at the end of the [module Proof] signature via [with type irmin_tree := t]. This keeps Octez (and any other consumer) able to reuse code written against [Irmin.Generic_key.S.Tree.Proof] verbatim — same path, same identifiers, same shape. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Audit pass against Irmin 3.11's Generic_key.S surface uncovered three top-level signatures that should be Lwt-returning to match the old API exactly: - val tree : t -> tree Lwt.t (was: t -> tree) - val to_backend_node : node -> Backend.Node.value Lwt.t (was: sync) - val to_backend_portable_node : node -> Backend.Node_portable.t Lwt.t These are pure conversion functions in Irmin 4 (no I/O), but Irmin 3 exposed them as Lwt-returning, so consumers expect [let* x = tree t] and [let* v = to_backend_node n]. Wrapping them through [run_eio] restores that ergonomic. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Both Irmin 3 and Irmin 4 name these types [force] and [folder] (only
the callback signatures differ between the two). The [_lwt] suffix
I introduced was a disambiguation that turned out to be unhelpful:
it makes the irmin-lwt API gratuitously different from upstream and
forces consumers to rename when porting from Irmin 3.
type 'a force_lwt → type 'a force
type ('a, 'b) folder_lwt → type ('a, 'b) folder
The shape of the types is identical to Irmin 3's [force] and [folder];
the wrapping happens at the callback level (callbacks are
Lwt-returning).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
[Irmin_lwt.Sync.Make (X : Irmin.Generic_key.S)] mirrors [Irmin.Sync.Make] but with Lwt-wrapped operations: [fetch], [fetch_exn], [pull], [pull_exn], [push], [push_exn]. Errors are exposed transparently: type pull_error = [ `Msg of string | Irmin.Merge.conflict ] type push_error = [ `Msg of string | `Detached_head ] [remote_store] is forwarded as-is from [Irmin.remote_store] (pure). Useful for any consumer that synchronises with a remote Irmin store. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Lwt wrappers for [Irmin.Json_tree] and [Irmin.Dot], matching the Irmin 3.11 surface. Json_tree takes [Irmin.S] (its upstream constraint where contents_key = hash); Dot takes [Generic_key.S].
Lwt-flavoured counterparts of [Irmin.S], [Irmin.KV], [Irmin.Maker], and [Irmin.KV_maker] so downstream code can declare interfaces and functor parameters in the same shape as Irmin 3. [Maker.Make] and [KV_maker.Make] produce [S] (the generic-keyed Lwt signature) with hash-equal-to-key constraints, matching upstream [Irmin.Maker] which produces [S_generic_key] with the same equalities.
These three reference predecessor functions were exposed without an Lwt.t wrapping, breaking the Irmin 3.11 signature where they all return [elt list Lwt.t]. Client code feeding them as ?pred_commit / ?pred_node / ?pred_contents callbacks to [Repo.iter] would not type-check.
Mirror Irmin 3.11's [Repo] signature, which exposes its [val close] through [include Closeable with type _ t := t] rather than as a free [val close]. The visible signature is unchanged for clients reading fields directly, but [module M : Irmin_lwt.Closeable = ...]-style constraints now type-check against [Repo].
Mirror the Irmin 3.11 [S.Branch] signature, which terminates with [include Branch.S with type t = branch] to expose [val main], [val is_valid], and [val t : t Irmin.Type.t] alongside the I/O functions. Without this, [Store.Branch.main] is unreachable through [Irmin_lwt.S.Branch].
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP
Implements Phase 5 of the Irmin 4 release plan (#2401, revised roadmap): a thin Lwt compatibility layer over Irmin 4's direct-style API, built on top of
lwt_eio. This lets Irmin 3 (Lwt-based) consumers migrate progressively rather than rewriting all call sites at once.Design rationale: issue comment #4223023836.
What's in this PR
New package
irmin-lwt(src/irmin-lwt/,irmin-lwt.opam), with two entry points and two functors.Entry points
Irmin_lwt.run : (unit -> 'a Lwt.t) -> 'a— drop-in replacement forLwt_main.run, bundlesEio_main.run+Lwt_eio.with_event_loop.Irmin_lwt.run_with_env— for callers already inside an Eio loop.Irmin_lwt.Make (S : Irmin.Generic_key.S)Wraps every I/O-triggering operation of
SwithLwt_eio.run_eioso it returns_ Lwt.t. After the Octez compilation test (thanks to @cuihtlauac), coverage was extended to essentially the fullGeneric_key.Ssurface:repo,t,step,path,metadata,contents,node,tree,commit,branch,slice,info,hash,contents_key,node_key,commit_key,lca_error,ff_error,write_error.Schema,Info,Hash,Path,Metadata,Backend,Contents,History,Status.Repo.v/close/heads/branches/config/export,main,master(deprecated alias),of_branch,of_commit,empty.find,find_all,get,mem,find_tree,get_tree,hash.set/set_exn,set_tree/set_tree_exn,remove/remove_exn,test_and_set/_exn/_tree/_tree_exn,test_set_and_get/_exn/_tree/_tree_exn,merge/_exn/_tree/_tree_exn,with_tree/_exn,clone.merge_into,merge_with_branch,merge_with_commit,lcas,lcas_with_branch,lcas_with_commit,history,last_modified.of_backend_node,to_backend_node,to_backend_portable_node,to_backend_commit,of_backend_commit,save_contents,save_tree,commit_t.Treesubmodule includingfoldwith Lwt-returning folders (bridged viaLwt_eio.Promise.await_lwt).Commit,Branch,Headsubmodules; top-levelwatch,watch_key,unwatch.Irmin_lwt.Pack.Make (S : Irmin_pack_io.S)A second functor that
includesMake(S)and adds Lwt-wrapped versions of the pack-unix extensions:integrity_check*,traverse_pack_file,split,is_split_allowed,add_volume,reload,flush,create_one_commit_store, theGcsubmodule (start/finalise/run/wait/cancel/is_finished/behaviour/is_allowed/latest_gc_target, with a Lwt-returningfinishedcallback onrun), andSnapshot. Addsirmin-packas an opam dep,irmin-pack.ioas a dune dep.Tests
test/irmin-lwt/— 11 tests, all green:merge_into,last_modified(the Tezos context-layer idioms).Lwt.catchcatching an Irmin-raised exception,Lwt.pauseinterleaved with ops, 50 concurrent reads viaLwt.all.Empirical validation
An initial attempt to compile the Tezos/Octez
lib_contextlayer againstirmin-lwtdrove the v2 extension in this PR. The remaining blockers identified in that test were items 1–4 of the Octez report, all addressed here. Pack-unix-specific operations (items 5–10) are now reachable throughIrmin_lwt.Pack.Make. Non-Lwt breaking changes in Irmin 3→4 (e.g.Irmin_pack_unix.Checks.S/Index.Make.Checksno longer publicly exposed; ppx ecosystem moving toppxlib >= 0.37) are outside this PR's scope.Documentation
doc/migration-from-irmin-3.md— step-by-step migration guide: opam diff, before/after example, non-Lwt breaking changes to be aware of (OCaml 5.1+, config renames, API removals), and the 2-step strategy (adopt `irmin-lwt` first, drop it module by module later).CHANGES.mdentry under 4.0.0 "Added".What's still not wrapped
Irmin.Syncfunctor (git remote fetch/push/pull): low priority since Tezos doesn't use it. Escape hatch viaLwt_eio.run_eioworks.Repo(iter,breadth_first_traversal,default_pred_*).Commit structure
18 atomic commits. Chronologically: MVP (skeleton → functor → entry points → three levels of tests → migration doc → CHANGES), extension to submodules (Tree, Commit, Branch+Head+watches, Tree.fold, extended tests, doc update), then v2 based on the Octez compile test (type/module re-exports, backend converters + save_*, full Generic_key.S sweep, Pack.Make).
Draft
Still opening as draft: a sanity-check pass on the design is needed before merge, especially:
Lwt_eio.Promise.await_lwtbridging for fold/watch/GC callbacks is the right shape.Irmin.Generic_key.Sapproach is preferable to pre-instantiated wrappers for specific backends.irmin-lwtshould depend onirmin-packto carryIrmin_lwt.Pack, or whether pack wrappers should live in a separate opam package.