Total bifunctorization

.gitignore

@@ -120,3 +120,4 @@ project/plugins/project/
 
 # virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
 hs_err_pid*
+.claude/

bifunctorization-deviations.md

@@ -0,0 +1,113 @@
+# Bifunctorization Deviations from Spec
+
+`bifunctorization.md` is the authoritative spec. It **must not** be edited
+to match implementation choices — that creates a moving target. If the
+implementation deviates from the spec, document the deviation here with
+the reason and the path to remediation.
+
+This file replaces the convention of in-place spec amendments (notably
+commit `6fecdd330` "PR-04 follow-up: resolve design Q via Option B (spec
+amended)", which was reverted after the deviation it documented was
+implemented properly).
+
+---
+
+## Format
+
+```
+## [D-NN] <one-line headline>
+**Status:** open | remediated
+**Spec section:** <quote or section reference in bifunctorization.md>
+**Deviation:** <what the implementation does differently>
+**Reason:** <why the deviation exists>
+**Remediation path:** <what would close it, or "n/a" if accepted permanently>
+```
+
+---
+
+## Active deviations
+
+_None._
+
+## Closed / remediated deviations
+
+### [D-02] `Bifunctorized.bifunctorize`/`debifunctorize` (method) was identity; submerging happened only at the implicit-conversion seam in `CatsToBIOConversions`
+**Status:** remediated (commit chain following the introduction of the `Bifunctorize[F]` typeclass)
+**Spec section:** "Conversion of effect values":
+
+> "the Throwable error must be Submerged, converted into a typed error during `bifunctorize`."
+>
+> "In `debifunctorize`, a typed error must be de-Submerged, unwrapped, as its expected to be in order for monofunctor's native methods to work with it."
+
+**Deviation (was):** the spec describes `bifunctorize` / `debifunctorize` as if they are
+single functions that submerge / de-submerge unconditionally. The previous implementation
+split the responsibility:
+- `Bifunctorized.bifunctorize(f: F[A])` (the method on the companion) — identity reinterpret-cast (Goal 4 zero-cost).
+- `Bifunctorized.debifunctorize(b: Bifunctorized[F, Throwable, A])` (the method) — identity reinterpret-cast.
+- Separate implicit conversions `bifunctorizeSubmerging` / `debifunctorizeUnSubmerging` in `CatsToBIOConversions` — actually submerged / de-submerged, gated on `cats.ApplicativeError[F, Throwable]` plus `izumi.reflect.TagK[F]`.
+- Companion-of-RHS conversions `Bifunctorized.{bifunctorizeConversion, debifunctorizeConversion}` — cats-free identity (only fired when the cats conversion was not in scope, Goal 5).
+
+So the same effect-site `val b: Bifunctorized[F, Throwable, A] = fa` could pick either submerging or identity depending on whether `CatsToBIOConversions._` had been imported, AND the spec-mandated submerging never happened on the literal `Bifunctorized.bifunctorize(_)` call regardless.
+
+**Reason (was):** three constraints appeared to make a single unconditional implementation impossible:
+1. **Goal 4** ("bifunctorize is a no-op for real bifunctors — `bifunctorize(zio) eq zio` should hold"). A method that always submerged broke this for ZIO/MonixBIO/Either/MiniBIO.
+2. **Goal 5** ("No More Orphans" — users without cats on the classpath must still use `Bifunctorized`). A method requiring `cats.ApplicativeError` would break the no-cats build (`Bifunctorized.scala` would have to import `cats.*`).
+3. The spec's combined effect "any user reaching for `bifunctorize(io)` on a cats monofunctor sees typed-error semantics" was achieved indirectly by routing through the implicit-conversion seam in `CatsToBIOConversions._`.
+
+**Remediation:** introduce a `Bifunctorize[F[_]]` typeclass that owns the
+`F[A] <-> Bifunctorized[F, Throwable, A]` round-trip:
+
+```scala
+trait Bifunctorize[F[_]] {
+  def bifunctorize[A](fa: F[A]): Bifunctorized[F, Throwable, A]
+  def debifunctorize[A](b: Bifunctorized[F, Throwable, A]): F[A]
+}
+```
+
+Two instances satisfy the three constraints simultaneously:
+- **Identity** (in `Bifunctorize` companion via `LowPriorityBifunctorizeInstances`): reinterpret cast in both directions. Used for real bifunctors and any `F` without an `ApplicativeError` in scope. `Bifunctorize.scala` does NOT import cats, so it is reachable on a no-cats classpath (Goal 5).
+- **Cats-mediated** (in `CatsToBIOConversions.bifunctorizeForCatsApplicativeError`): submerges via `F.adaptError`. Gated on `cats.ApplicativeError[F, Throwable]` and `TagK[F]`, with the "No-More-Orphans" trick (`@unused` phantom `\`cats.ApplicativeError\`` parameter from `izumi.fundamentals.orphans.OrphanDefs`) so users without cats on the classpath cannot resolve it and are not forced to depend on it (Goal 5).
+
+Both the methods `Bifunctorized.bifunctorize` / `Bifunctorized.debifunctorize` AND the
+implicit conversions `bifunctorizeConversion` / `debifunctorizeConversion` now take an
+implicit `Bifunctorize[F]` and delegate — a single source of truth and a single conversion
+level (no second cats-mediated implicit conversion in `CatsToBIOConversions`).
+
+Resolution priority remains the standard Scala one: a cats-mediated `Bifunctorize[F]`
+instance in the user's import scope (via `import CatsToBIOConversions.*`) outranks the
+identity instance in the companion of `Bifunctorize`, so the spec-mandated submerging
+fires uniformly across `bifunctorize(_)` method calls, implicit conversions, and
+`.toMonofunctor` syntax. For real bifunctors (whose `ApplicativeError` is not in scope
+via the cats conversion) and for any `F` without that import, the identity instance fires
+and Goal 4 (`bifunctorize(realBifunctor) eq realBifunctor`) holds.
+
+### [D-01] `bifunctorize`/`debifunctorize` did not transparently (de-)submerge typed errors
+**Status:** remediated (commit chain following the revert of `6fecdd330`)
+**Spec section:** "Conversion of effect values":
+
+> "the Throwable error must be Submerged, converted into a typed error during `bifunctorize`."
+>
+> "In `debifunctorize`, a typed error must be de-Submerged, unwrapped, as its expected to be in order for monofunctor's native methods to work with it."
+
+**Deviation (was):** the original M1 PR-01 implementation made
+`bifunctorize`/`debifunctorize` pure type-level identity casts. Submerging
+happened only inside BIO instance methods (`fail`, `syncThrowable`,
+`fromFuture`, etc.). A user who created a `Bifunctorized[IO, Throwable, A]`
+via `F.fail(rt)` and then called `debifunctorize(_).unsafeRunSync()` would
+see `SubmergedTypedError[IO](rt)` raised, not `rt` raw — contradicting the
+spec's "must be de-Submerged" requirement.
+
+**Reason (was):** PR-04 review flagged the spec/impl mismatch as PR-04-D01.
+The orchestrator recommended "Option B — amend the spec to match the
+implementation" and committed that amendment as `6fecdd330` under
+autonomous-loop-dynamic mode. The amendment was a workaround, not a
+genuine design decision; it rationalised the implementation rather than
+fixing it.
+
+**Remediation:** spec restored to its original text. Transparent
+submerging/de-submerging implemented in `CatsToBIOConversions.scala` for
+cats-mediated monofunctors. For real bifunctors (ZIO, MonixBIO, Either,
+MiniBIO): no-op (Goal 4 zero-cost identity preserved via
+`BifunctorizedNoOpInstances`). For `Identity`: handled by the existing M2
+`bifunctorizeIdentity`/`debifunctorizeIdentity` path, which uses MiniBIO
+as the carrier and runs synchronously with rethrow on the way out.

bifunctorization.md

@@ -0,0 +1,106 @@
+# Problem statement
+
+Izumi Project library ecosystem strives to be compatible with a wide variety of effect types that may (or may not) be used by its users.
+
+This led to the creation of QuasiIO @fundamentals/fundamentals-bio/src/main/scala/izumi/functional/quasi family of compatibility typeclasses that extract a subset of operations that Izumi Project libraries need from other effect types. The existence of Quasi* typeclass hierarchy is a maintenance burden on the developers and due to its monofunctor nature (previously thought required to support the most common denominator effect types) it prevents usage of typed errors within the Izumi Project libraries themselves. Moreover, due to QuasiIO's direct support for `Identity`, Izumi itself can't use a purely functional style - all maintainers must be aware that the `F` in `F[_]: QuasiIO` is not a lawful monad and operations on it must be carefully suspended manually.
+
+Following an experiment at conversion of monofunctor effect types to bifunctors (https://github.com/7mind/izumi/pull/1766), we have decided to solve our compatibility problem in a different way. By lifting monofunctor effect types to bifunctors we will get rid of the entire `Quasi*` hierarchy and allow usage of typed errors and purely functional style within Izumi Project library code. However, we also want the usability for monofunctor effect type users not to degrade following that refactoring.
+
+# Background
+
+BIO bifunctor typeclass hierarchy is isomorphic to Cats Effect monofunctor typeclass hierarchy. This is currently witnessed only partially by the existence of BIO to CE conversions in fundamentals/fundamentals-bio/src/main/scala/izumi/functional/bio/CatsConversions.scala, but the opposite conversions, from CE to BIO are missing. As part of the following design, we will add them.
+
+# Design of the solution
+
+There are two parts to the design:
+
+- Conversion of monofunctor typeclasses to bifunctor typeclasses
+- Transparent bifunctorization/debifunctorization for monofunctor effect types at the entry points to Izumi Project libraries.
+
+## Conversion design
+
+We provide conversion typeclasses from Cats Effect to BIO, of form:
+
+```
+implicit def MonadToBIO[F[_]](implicit Monad: Monad[F]): Monad2[Bifunctorized[F, +_, +_]]
+
+/* etc for all Cats-Effect -> BIO correspondences */
+```
+
+Where `Bifunctorized` is an opaque type wrapper that "converts" a monofunctor effect type to a bifunctor effect type.
+
+```
+object Bifunctorized {
+  ...
+  type Bifunctorized[F[_], +E, +A] // abstract type aka newtype aka pre-Scala 3 opaque type
+}
+```
+
+The conversions shall use an 'error-submerging'/'submarine error handling' technique shown in Prior Art, to add an ability to a monofunctor effect type to distinguish between typed errors (`Exit.Error`) and untyped errors (defect, `Exit.Termination`). TypedError fundamentals/fundamentals-bio/src/main/scala/izumi/functional/bio/TypedError.scala exists which converts typed errors into Throwables, but it lacks one detail - we don't want it to be possible for TypedErrors from different `Bifunctorized` effect types to be intermixed together through the Typed error channel. We should add a new type instead that discriminates error origin using `TagK[F]` of the original monofunctor effect type. Note: submarine error handling implies discrimination based on the unique handler, to mimic algebraic effect semantics, which I believe is not necessary for our purpose, type discrimination alone is sufficient.
+
+### Conversion of effect values
+
+In order to provide an ability to use an existing monofunctor effect value with bifunctor methods, the following automatic bijection implicit conversions shall be provided:
+
+```scala
+object Bifunctorized {
+  ...
+  def bifunctorize(f: F[A]): Bifunctorized[F, Throwable, A]
+
+  def debifunctorize(f: Bifunctorized[F, Throwable, A]): F[A]
+
+  implicit def bifunctorizeConversion(f: F[A]): Bifunctorized[F, Throwable, A] = bifunctorize(f)
+  implicit def debifunctorizeConversion(f: Bifunctorized[F, Throwable, A]): F[A] = debifunctorize(f)
+
+  implicit final class BifunctorizedSyntax[F[_]](private val f: Bifunctorized[F, Throwable, A]) extends AnyVal {
+    def toMonofunctor: F[A] = debifunctorize(f)
+  }
+
+  ...provide Syntax2 conversions in Bifunctorized companion to make BIO syntax available on any Bifunctorized value...
+}
+```
+
+Where in order to make the untyped Throwable error embedded into the monofunctor `F` effect type manipulable via e.g. `Error2#catchAll` and other typed error BIO hierarchy methods, the Throwable error must be Submerged, converted into a typed error during `bifunctorize`.
+
+In `debifunctorize`, a typed error must be de-Submerged, unwrapped, as its expected to be in order for monofunctor's native methods to work with it.
+
+Note: where the bifunctorized effect value is a bifunctor already, such as `bifunctorize(Left(new Throwable()))`, no submerging should happen.
+
+## Transparent bifunctorization at seams
+
+The Izumi Project external interface seams that previously accepted monofunctor effect types `F[_]`, such as distage/distage-core/src/main/scala/izumi/distage/model/Injector.scala
+
+```scala
+def apply[F[_]: QuasiIO: ...](...): Injector[F]
+```
+
+Shall now accept bifunctor effect types `F[+_, +_]`:
+
+```scala
+def apply[F[+_, +_]: IO2: ...](...): Injector[F]
+```
+
+With an overload for monofunctors:
+
+```scala
+def apply[F[_]: ...](...)(implicit F: IO2[Bifunctorized[F, +_, +_]]): Injector[Bifunctorized[F, +_, +_]]
+```
+
+The BIO syntax on Bifunctorized shall allow users to manipulate their formerly monofunctor effect values in bifunctor way, the implicit conversion `Bifunctorized.bifunctorizeConversion` shall ease the pain of passing in monofunctor values and `.toMonofunctor` method allows return back to monofunctor world.
+
+# Goals
+
+1. Bifunctorized effect types must pass cats laws suites using CatsConversions instances for their Bifunctorized forms. That is, a CE->BIO->CE conversion must come at no loss of correctness with respect to cats effect laws.
+2. Bifunctorized Submerged errors are discriminated by TagK[F] of its monofunctor. Terminates/defects use monofunctor's raw Throwable
+3. distage's Injector, bio's Lifecycle and logstage's LogIO entrypoints are transparently bifunctorized/de-bifunctorized for monofunctors. `Identity` is special-cased and goes through a bifunctorization/debifunctorization cycle to `MiniBIO` and back, transparently to the user.
+4. Bifunctorization should be a no-op for real bifunctors, that is, `bifunctorize(f: ZIO[ArbiraryEnv, Throwable, A]) eq f` should hold. There should be no error submerging performed for effect types that already support typed errors.
+5. No More Orphans trick keeps working, users are not forced to have cats on their classpath, test distage/distage-extension-config/.jvm/src/test/scala/izumi/distage/impl/OptionalDependencyTest.scala keeps passing.
+6. Quasi* typeclasses are deleted and BIO Hierarchy typeclasess are used everywhere the former were used.
+7. The project compiles and all tests pass on Scala 2.13, Scala 3 and Scala 2.12.
+
+# Prior art
+
+- https://github.com/7mind/izumi/pull/1766 - sketch of bifunctorization for arbitrary Cats-Effect compatible monofunctor effect types
+- https://github.com/typelevel/cats-mtl/pull/619 - prior art implementation of 'Submerge' error handling in monofunctor world. Their implementation tags errors by unique instance to implement an effects-and-handlers (algebraic effects) semantic where only a handler associated with the effect region can catch errors introduced by that region's throw. We're not looking for that - in fact, we want error handlers for the same monofunctor effect type to be compatible - but we don't want errors from **different** effect types to be compatible. e.g. a Bifunctorized[cats.effect.IO] type should not be able to be fooled into thinking that a Bifunctorized[scala.util.Try]'s typed error is its own typed error.
+
+Fetch the diffs of prior art pull requests and save them for reference first. If you fail to do that, fail fast and revert to user.