Merge branch 'master' into cosmos-linq

Author: bartelink

CHANGELOG.md

@@ -22,21 +22,17 @@ The `Unreleased` section name is replaced by the expected version of next releas
 - `eqx dump`, `eqx query`: `-sl` Support for specifying streams to dump via a [CosmosDB `LIKE` expression](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/query/keywords#like) [#473](https://github.com/jet/equinox/pull/473)
 - `eqx dump`: `-Q` strips intervals, regularizes snapshots, logs stream names [#473](https://github.com/jet/equinox/pull/473)
 - `eqx stats`: `-O`, `-N` flags extract oldest and newest `_ts` within a store [#459](https://github.com/jet/equinox/pull/459)
-- `eqx stats`: `-U` flag to count streams with unfolds and total number thereof; `-I` alias relabel Documents as Items [#464](https://github.com/jet/equinox/pull/464)
-- `eqx stats`: `-I` flag; relabel Documents as Items, retaining existing `-D` flag [#464](https://github.com/jet/equinox/pull/464)
-- `eqx`: `-Q` flag omits timestamps from console output logging [#459](https://github.com/jet/equinox/pull/459)
+- `eqx stats`: `-U` flag to count streams with unfolds and total number thereof [#473](https://github.com/jet/equinox/pull/473)
+- `eqx stats`: `-I` flag; relabel Documents as Items, retaining existing `-D` flag [#473](https://github.com/jet/equinox/pull/473)
 - `Equinox.CosmosStore.Linq`: Add LINQ querying support for Indexed `u`nfolds (`AccessStrategy.Custom`+`CosmosStoreCategory.shouldCompress`) [#450](https://github.com/jet/equinox/pull/450)
-- `eqx dump`, `eqx query`: `-sl` Support for specifying streams to dump via a [CosmosDB `LIKE` expression](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/query/keywords#like) [#450](https://github.com/jet/equinox/pull/450)
-- `eqx dump`: `-Q` strips intervals, regularizes snapshots, logs stream names [#450](https://github.com/jet/equinox/pull/450)
-- `eqx top`: Support for analyzing space usage for event and view containers by category and/or stream [#450](https://github.com/jet/equinox/pull/450)
-- `eqx destroy`: Support for deleting the items(documents) underlying a category/stream/arbitrary `WHERE` clause [#450](https://github.com/jet/equinox/pull/450)
 
 ### Changed
 
-- `Equinox.*Store`,`Equinox.*Store.Prometheus`: Pin `Equinox` dependencies to `[4.0.3, 5.0.0)`] [#450](https://github.com/jet/equinox/pull/450)
+- `Equinox.*Store`,`Equinox.*Store.Prometheus`: Pin `Equinox` dependencies to `[4.0.3, 5.0.0)`] [#448](https://github.com/jet/equinox/pull/448)
 - `Equinox.CosmosStore`: Update `System.Text.Json` dep to `6.0.10` per [CVE-2024-43485](https://github.com/advisories/GHSA-8g4q-xg66-9fp4) [#470](https://github.com/jet/equinox/pull/470)
 - `Equinox.CosmosStore`: Minimum `Microsoft.Azure.Cosmos` requirement updated to `3.43.1` to avail of integrated `System.Text.Json` support [#467](https://github.com/jet/equinox/pull/467)
 - `Equinox.CosmosStore.CosmosStoreConnector`: Removed mandatory `requestTimeout` argument [#467](https://github.com/jet/equinox/pull/467)
+- `Equinox.DynamoStore`: Up min `FSharp.AWS.DynamoDB` to v `0.12.3-beta` to handle breaking change [#474](https://github.com/jet/equinox/pull/474)
 - `Equinox.MessageDb`: Up min `Npgsql` to v `7.0.7` as `7.0.0` is on CVE blacklist
 - `Equinox.MemoryStore`: Remove unused TrySync args [#466](https://github.com/jet/equinox/pull/466)
 

Directory.Build.props

@@ -10,7 +10,7 @@
 
     <WarningLevel>5</WarningLevel>
     <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
-    <!-- NU5104: Warning As Error: A stable release of a package should not have a prerelease dependency. Either modify the version spec of dependency "FSharp.AWS.DynamoDB [0.12.0-beta, )-->
+    <!-- NU5104: Warning As Error: A stable release of a package should not have a prerelease dependency. Either modify the version spec of dependency "FSharp.AWS.DynamoDB [0...., )-->
     <NoWarn>NU5104</NoWarn>
     <!-- For packages we produce, we want to be explicit and conservative in what we depend on
          Tools and Tests flip this to false in order to ensure we validate with the recent one embdedded in the SDK-->

README.md

@@ -172,7 +172,7 @@ The components within this repository are delivered as multi-targeted Nuget pack
 - `Equinox.MemoryStore` [![MemoryStore NuGet](https://img.shields.io/nuget/v/Equinox.MemoryStore.svg)](https://www.nuget.org/packages/Equinox.MemoryStore/): In-memory store for integration testing/performance base-lining/providing out-of-the-box zero dependency storage for examples. ([depends](https://www.fuget.org/packages/Equinox.MemoryStore) on `Equinox`)
 - `Equinox.CosmosStore` [![CosmosStore NuGet](https://img.shields.io/nuget/v/Equinox.CosmosStore.svg)](https://www.nuget.org/packages/Equinox.CosmosStore/): Azure CosmosDB Adapter with integrated 'unfolds' feature, facilitating optimal read performance in terms of latency and RU costs, instrumented to meet Jet's production monitoring requirements. ([depends](https://www.fuget.org/packages/Equinox.CosmosStore) on `Equinox` v `4.0.3`, `Equinox`, `Microsoft.Azure.Cosmos` >= `3.43.1`, `System.Text.Json`, `FSharp.Control.TaskSeq`)
 - `Equinox.CosmosStore.Prometheus` [![CosmosStore.Prometheus NuGet](https://img.shields.io/nuget/v/Equinox.CosmosStore.Prometheus.svg)](https://www.nuget.org/packages/Equinox.CosmosStore.Prometheus/): Integration package providing a `Serilog.Core.ILogEventSink` that extracts detailed metrics information attached to the `LogEvent`s and feeds them to the `prometheus-net`'s `Prometheus.Metrics` static instance. ([depends](https://www.fuget.org/packages/Equinox.CosmosStore.Prometheus) on `Equinox.CosmosStore`, `prometheus-net >= 3.6.0`)
-- `Equinox.DynamoStore` [![DynamoStore NuGet](https://img.shields.io/nuget/v/Equinox.DynamoStore.svg)](https://www.nuget.org/packages/Equinox.DynamoStore/): Amazon DynamoDB Adapter with integrated 'unfolds' feature, facilitating optimal read performance in terms of latency and RC costs, patterned after `Equinox.CosmosStore`. ([depends](https://www.fuget.org/packages/Equinox.DynamoStore) on `Equinox`, `FSharp.AWS.DynamoDB` >= `0.12.0-beta`, `FSharp.Control.TaskSeq`)
+- `Equinox.DynamoStore` [![DynamoStore NuGet](https://img.shields.io/nuget/v/Equinox.DynamoStore.svg)](https://www.nuget.org/packages/Equinox.DynamoStore/): Amazon DynamoDB Adapter with integrated 'unfolds' feature, facilitating optimal read performance in terms of latency and RC costs, patterned after `Equinox.CosmosStore`. ([depends](https://www.fuget.org/packages/Equinox.DynamoStore) on `Equinox`, `FSharp.AWS.DynamoDB` >= `0.12.3-beta`, `FSharp.Control.TaskSeq`)
 - `Equinox.DynamoStore.Prometheus` [![DynamoStore.Prometheus NuGet](https://img.shields.io/nuget/v/Equinox.DynamoStore.Prometheus.svg)](https://www.nuget.org/packages/Equinox.DynamoStore.Prometheus/): Integration package providing a `Serilog.Core.ILogEventSink` that extracts detailed metrics information attached to the `LogEvent`s and feeds them to the `prometheus-net`'s `Prometheus.Metrics` static instance. ([depends](https://www.fuget.org/packages/Equinox.CosmosStore.Prometheus) on `Equinox.DynamoStore`, `prometheus-net >= 3.6.0`)
 - `Equinox.EventStore` [![EventStore NuGet](https://img.shields.io/nuget/v/Equinox.EventStore.svg)](https://www.nuget.org/packages/Equinox.EventStore/): [EventStoreDB](https://eventstore.org/) Adapter designed to meet Jet's production monitoring requirements. ([depends](https://www.fuget.org/packages/Equinox.EventStore) on `Equinox`, `EventStore.Client >= 22.0.0`, `FSharp.Control.TaskSeq`), EventStore Server version `21.10` or later). **NO NOT use for new projects - the TCP interface to EventStoreDB has long been deprecated, this package is only provided to ease migration scenarios and will be removed in due course**
 - `Equinox.EventStoreDb` [![EventStoreDb NuGet](https://img.shields.io/nuget/v/Equinox.EventStoreDb.svg)](https://www.nuget.org/packages/Equinox.EventStoreDb/): Production-strength [EventStoreDB](https://eventstore.org/) Adapter. ([depends](https://www.fuget.org/packages/Equinox.EventStoreDb) on `Equinox`, `EventStore.Client.Grpc.Streams` >= `22.0.0`, `FSharp.Control.TaskSeq`, EventStore Server version `21.10` or later)

src/Equinox.DynamoStore/DynamoStore.fs

@@ -99,7 +99,7 @@ module Unfold =
 /// NOTE names are intended to generally align with CosmosStore naming. Key Diffs:
 /// - no mandatory `id` and/or requirement for it to be a `string` -> replaced with `i` as an int64
 ///   (also Tip magic value is tipMagicI: Int32.MaxValue, not "-1")
-/// - etag is managed explicitly (on Cosmos DB, its managed by the service and named "_etag")
+/// - etag is managed explicitly (on Cosmos DB, it's managed by the service and named "_etag")
 [<NoEquality; NoComparison>]
 type Batch =
     {   p: string // "{streamName}"
@@ -180,7 +180,7 @@ module Batch =
         // If we're loading from a nominated position, we need to discard items in the batch before/after the start on the start page
         x.e |> Seq.filter (fun e -> let i = int64 e.i in i >= indexMin && int64 i < indexMax)
 
-    /// Computes base Index for the Item (`i` can bear the the magic value TipI when the Item is the Tip)
+    /// Computes base Index for the Item (`i` can bear the magic value TipI when the Item is the Tip)
     let baseIndex (x: Batch) = x.n - x.e.LongLength
     let bytesUnfolds (x: Batch) = Unfold.arrayBytes x.u
     let bytesBase (x: Batch) = 80 + x.p.Length + String.length x.etag + Event.arrayBytes x.e
@@ -493,7 +493,7 @@ type Position =
     override x.ToString() = sprintf "{ n=%d; etag=%s; e=%d; b=%d+%d }" x.index x.etag x.events.Length x.baseBytes x.unfoldsBytes
 module internal Position =
 
-    // NOTE a write of Some 0 to x.b round-trips as None
+    // NOTE writing Some 0 to x.b round-trips as None
     let fromTip (x: Batch) = { index = x.n; etag = x.etag; events = x.e; calvedBytes = defaultArg x.b 0; baseBytes = Batch.bytesBase x; unfoldsBytes = Batch.bytesUnfolds x }
     let fromElements (p, b, n, e, u, etag) = fromTip { p = p; b = Some b; i = Unchecked.defaultof<_>; n = n; e = e; u = u; etag = etag }
     let tryFromBatch (x: Batch) = if Batch.isTip x.i then fromTip x |> Some else None
@@ -508,9 +508,11 @@ module internal Sync =
 
     let private (|DynamoDbConflict|_|): exn -> _ = function
         | Precondition.CheckFailed
-        | TransactWriteItemsRequest.TransactionCanceledConditionalCheckFailed -> Some ()
+        | Transaction.TransactionCanceledConditionalCheckFailed -> Some ()
         | _ -> None
-
+    type [<NoEquality; NoComparison>] TransactWrite<'T> =
+        | Put of item: Batch.Schema * ConditionExpression<'T> option
+        | Update of TableKey * ConditionExpression<'T> option * UpdateExpression<Batch.Schema>
     let private cce: Quotations.Expr<Batch.Schema -> bool> -> ConditionExpression<Batch.Schema> = template.PrecomputeConditionalExpr
     let private cue (ue: Quotations.Expr<Batch.Schema -> Batch.Schema>) = template.PrecomputeUpdateExpr ue
 
@@ -567,11 +569,17 @@ module internal Sync =
         let etag' = let g = Guid.NewGuid() in g.ToString "N"
         let actions = generateRequests stream requestArgs etag'
         let rm = Metrics()
-        try do! let context = table.CreateContext(rm.Add)
-                match actions with
+        try let context = table.CreateContext(rm.Add)
+            do! match actions with
                 | [ TransactWrite.Put (item, Some cond) ] -> context.PutItemAsync(item, cond) |> Async.Ignore
                 | [ TransactWrite.Update (key, Some cond, updateExpr) ] -> context.UpdateItemAsync(key, updateExpr, cond) |> Async.Ignore
-                | actions -> context.TransactWriteItems actions |> Async.Ignore
+                | actions ->
+                    let req = context.CreateTransaction()
+                    for x in actions do
+                        match x with
+                        | TransactWrite.Put (item, cond) -> req.Put(context, item, ?precondition = cond)
+                        | TransactWrite.Update (key, cond, updateExpr) -> req.Update(context, key, updateExpr, ?precondition = cond)
+                    req.TransactWriteItems()
                 |> Async.executeAsTask ct
             return struct (rm.Consumed, Res.Written etag')
         with DynamoDbConflict ->
@@ -867,15 +875,15 @@ module internal Query =
 
         match primary, fallback with
         | Some { found = true }, _ -> return toPosition res, events // origin found in primary, no need to look in fallback
-        | Some { minIndex = i }, _ when i <= minI -> return toPosition res, events // primary had required earliest event Index, no need to look at fallback
+        | Some { minIndex = i }, _ when i <= minI -> return toPosition res, events // primary had required min event Index, no need to look at fallback
         | None, _ when Option.isNone tip -> return toPosition res, events // initial load where no documents present in stream
         | _, Choice1Of2 allowMissing ->
             logMissing (minIndex, i) "Origin event not found; no Archive Table supplied"
             if allowMissing then return toPosition res, events
             else return failwith "Origin event not found; no Archive Table supplied"
         | _, Choice2Of2 fallback ->
 
-        let maxIndex = match primary with Some p -> Some p.minIndex | None -> maxIndex // if no batches in primary, high water mark from tip is max
+        let maxIndex = match primary with Some p -> Some p.minIndex | None -> maxIndex // if no batches in primary, high-water mark from tip is max
         let! fallback = fallback (minIndex, maxIndex, ct)
         let events =
             match fallback with
@@ -935,7 +943,7 @@ module internal Prune =
                 for x in batches |> Seq.takeWhile (fun x -> isRelevant x || Option.isNone lwm) do
                     let batchSize = x.n - x.index |> int
                     let eligibleEvents = max 0 (min batchSize (int (indexInclusive + 1L - x.index)))
-                    if x.isTip then // Even if we remove the last event from the Tip, we need to retain a) unfolds b) position (n)
+                    if x.isTip then // Even if we remove the last event from the Tip, we need to retain a. unfolds b. position (n)
                         if eligibleEvents <> 0 then
                             let! charge = trimTip x.n eligibleEvents
                             trimCharges <- trimCharges + charge.total
@@ -1017,7 +1025,7 @@ module Internal =
     //   - TransactWriteItems is more than twice the cost in Write RU vs a normal UpdateItem, with associated latency impacts
     //   - various other considerations, e.g. we need to re-read the Tip next time around see https://stackoverflow.com/a/71706015/11635
     let defaultTipMaxBytes = 32 * 1024
-    // In general we want to minimize round-trips, but we'll get better diagnostic feedback under load if we constrain our
+    // In general, we want to minimize round-trips, but we'll get better diagnostic feedback under load if we constrain our
     // queries to shorter pages. The effect of this is of course highly dependent on the max Item size, which is
     // dictated by the TipOptions - in the default configuration that's controlled by defaultTipMaxBytes
     let defaultMaxItems = 32
@@ -1178,7 +1186,7 @@ type DynamoStoreConnector(clientConfig: Amazon.DynamoDBv2.AmazonDynamoDBConfig,
         let clientConfig = Amazon.DynamoDBv2.AmazonDynamoDBConfig(ServiceURL = serviceUrl, RetryMode = m, MaxErrorRetry = retries, Timeout = timeout)
         DynamoStoreConnector(clientConfig, Amazon.Runtime.BasicAWSCredentials(accessKey, secretKey))
 
-    /// Connect to a nominated SystemName with endpoints and credentials gathered implicitly from well-known environment variables and/or configuration etc
+    /// Connect to a nominated SystemName with endpoints and credentials gathered implicitly from well-known environment variables and/or configuration etc.
     /// systemName: Amazon SystemName, e.g. "us-west-1"
     /// timeout: Required; AWS SDK Default: 100s
     /// maxRetries: Required; AWS SDK Default: 10
@@ -1270,7 +1278,7 @@ type AccessStrategy<'event, 'state> =
     /// enable efficient reading without having to query the Event documents.
     | Snapshot of isOrigin: ('event -> bool) * toSnapshot: ('state -> 'event)
     /// Allow any events that pass the `isOrigin` test to be used in lieu of folding all the events from the start of the stream
-    /// When writing, uses `toSnapshots` to 'unfold' the <c>'state</c>, representing it as one or more Event records to be stored in
+    /// When writing, uses `toSnapshots` to 'unfold' the <c>&apos;state</c>, representing it as one or more Event records to be stored in
     /// the Tip with efficient read cost.
     | MultiSnapshot of isOrigin: ('event -> bool) * toSnapshots: ('state -> 'event[])
     /// Instead of actually storing the events representing the decisions, only ever update a snapshot stored in the Tip document

src/Equinox.DynamoStore/Equinox.DynamoStore.fsproj

@@ -15,15 +15,15 @@
 
   <ItemGroup>
     <ProjectReference Condition=" '$(Configuration)' == 'Debug' " Include="..\Equinox\Equinox.fsproj" />
-    <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.0, 5.0.0)" />
+    <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.3, 5.0.0)" />
   </ItemGroup>
 
   <ItemGroup>
     <PackageReference Include="MinVer" Version="5.0.0" PrivateAssets="All" />
 
     <PackageReference Include="FSharp.Core" Version="6.0.7" ExcludeAssets="contentfiles" />
 
-    <PackageReference Include="FSharp.AWS.DynamoDB" Version="0.12.0-beta" />
+    <PackageReference Include="FSharp.AWS.DynamoDB" Version="0.12.3-beta" />
     <PackageReference Include="FSharp.Control.TaskSeq" Version="0.4.0" />
   </ItemGroup>
 

src/Equinox.EventStore/Equinox.EventStore.fsproj

@@ -15,7 +15,7 @@
 
   <ItemGroup>
     <ProjectReference Condition=" '$(Configuration)' == 'Debug' " Include="..\Equinox\Equinox.fsproj" />
-    <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.0, 5.0.0)" />
+    <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.3, 5.0.0)" />
   </ItemGroup>
 
   <ItemGroup>

src/Equinox.EventStoreDb/Equinox.EventStoreDb.fsproj

@@ -15,7 +15,7 @@
 
   <ItemGroup>
     <ProjectReference Condition=" '$(Configuration)' == 'Debug' " Include="..\Equinox\Equinox.fsproj" />
-    <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.0, 5.0.0)" />
+    <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.3, 5.0.0)" />
   </ItemGroup>
 
   <ItemGroup>

src/Equinox.MemoryStore/Equinox.MemoryStore.fsproj

@@ -14,7 +14,7 @@
 
   <ItemGroup>
     <ProjectReference Condition=" '$(Configuration)' == 'Debug' " Include="..\Equinox\Equinox.fsproj" />
-    <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.0, 5.0.0)" />
+    <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.3, 5.0.0)" />
   </ItemGroup>
 
   <ItemGroup>

src/Equinox.MessageDb/Equinox.MessageDb.fsproj

@@ -17,7 +17,7 @@
 
     <ItemGroup>
         <ProjectReference Condition=" '$(Configuration)' == 'Debug' " Include="..\Equinox\Equinox.fsproj" />
-        <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.0, 5.0.0)" />
+        <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.3, 5.0.0)" />
     </ItemGroup>
 
     <ItemGroup>

src/Equinox.SqlStreamStore/Equinox.SqlStreamStore.fsproj

@@ -15,7 +15,7 @@
 
   <ItemGroup>
     <ProjectReference Condition=" '$(Configuration)' == 'Debug' " Include="..\Equinox\Equinox.fsproj" />
-    <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.0, 5.0.0)" />
+    <PackageReference Condition=" '$(Configuration)' == 'Release' " Include="Equinox" Version="[4.0.3, 5.0.0)" />
   </ItemGroup>
 
   <ItemGroup>