Working - Before removal of divergences reporting

Author: mjameswh

arch_docs/workflow_task_chunking.md

@@ -4,20 +4,20 @@ One source of complexity in Core is the chunking of history into "logical" Workf
 
 Workflow tasks (WFTs) always take the following form in event history:
 
-* \[Preceding Events\] (optional)
-* WFT Scheduled
-* WFT Started
-* WFT Completed
-* \[Commands\] (optional)
+- \[Preceding Events\] (optional)
+- WFT Scheduled
+- WFT Started
+- WFT Completed
+- \[Commands\] (optional)
 
 In the typical case, the "logical" WFT consists of all the commands from the last workflow task,
 any events generated in the interrim, and the scheduled/started preamble. So:
 
-* WFT Completed
-* \[Commands\] (optional)
-* \[Events\] (optional)
-* WFT Scheduled
-* WFT Started
+- WFT Completed
+- \[Commands\] (optional)
+- \[Events\] (optional)
+- WFT Scheduled
+- WFT Started
 
 Commands and events are both "optional" in the sense that:
 
@@ -32,8 +32,22 @@ There may be no events for more nuanced reasons:
    rather as a "protocol message" attached to the task.
 3. Server can forcibly generate a new WFT with some obscure APIs
 
-Core does not consider such empty WFT sequences as worthy of waking lang (on replay - as a new
-task, they always will), since nothing meaningful has happened. Thus, they are grouped together
+## Handling of empty WFTs (March 2026)
+
+Until now, Core would try to avoid waking lang when replaying empty WFT sequences (i.e. a WFT
+Completed immediately followed by a WFT Scheduled and WFT Started, with no commands and no events
+in-between), since they would presumably be no-ops and therefore waste execution resources. Instead,
+the second WFT (and potentially subsequent WFTs) would be collapsed into the first, resulting in a
+single WFT that has the inbound events and start time of the first WFT, but the resulting commands
+of the last WFT in the collapsed sequence.
+
+It was found that there are some particular edge cases where WFT collapsing might result in
+incorrect replay behavior. One particular example of this is the case where an update request would
+be sent after an empty WFT. This led to the development of some heuristics to avoid incorrect
+chunking in presence of known problematic patterns. Unfortunately, it has been found that these
+heuristics may
+
+Thus, they are grouped together task, they always will), since nothing meaningful has happened. Thus, they are grouped together
 as part of a "logical" WFT with the last WFT that had any real work in it.
 
 ## Possible issues as of this writing (5/25)
@@ -44,8 +58,8 @@ NDE.
 
 ### Possible solutions
 
-* Core can attach a flag on WFT completes in order to be explicit that that WFT may be skipped on
+- Core can attach a flag on WFT completes in order to be explicit that that WFT may be skipped on
   replay. IE: During WFT heartbeating for LAs.
-* We could legislate that server should never send empty WFTs. Seemingly the only case of this
+- We could legislate that server should never send empty WFTs. Seemingly the only case of this
   is
   the [obscure api](https://github.com/temporalio/temporal/blob/d189737aa2ed1b07c221abb9fbdd28ecf68f0492/proto/internal/temporal/server/api/adminservice/v1/service.proto#L151)

crates/common/Cargo.toml

@@ -14,7 +14,6 @@ exclude = ["protos/*/.github/*"]
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 
 [features]
-history_builders = ["rand"]
 otel = ["dep:opentelemetry", "dep:opentelemetry_sdk", "dep:opentelemetry-otlp"]
 prometheus = [
   "dep:prometheus",
@@ -25,7 +24,7 @@ prometheus = [
 ]
 envconfig = ["dep:toml", "dep:dirs"]
 serde_serialize = []
-test-utilities = ["history_builders"]
+test-utilities = []
 core-based-sdk = [
   "prometheus",
   "envconfig",
@@ -41,6 +40,7 @@ bon = { workspace = true }
 crc32fast = "1"
 dirs = { version = "6.0", optional = true }
 derive_more = { workspace = true }
+enum-iterator = "2"
 erased-serde = "0.4"
 futures = { version = "0.3", default-features = false, features = ["alloc"] }
 futures-channel = { version = "0.3", default-features = false, features = [
@@ -74,7 +74,6 @@ prometheus = { version = "0.14", optional = true, default-features = false }
 prost = { workspace = true }
 prost-wkt = "0.7"
 prost-types = { workspace = true }
-rand = { version = "0.10", optional = true }
 ringbuf = { version = "0.4", optional = true }
 serde = { version = "1.0", features = ["derive"] }
 serde_json = { workspace = true }

crates/common/src/protos/mod.rs

@@ -6,26 +6,12 @@ pub mod constants;
 /// Utility functions for working with protobuf types.
 pub mod utilities;
 
-#[cfg(feature = "test-utilities")]
-/// Pre-built test histories for common workflow patterns.
-pub mod canned_histories;
-#[cfg(feature = "history_builders")]
-mod history_builder;
-#[cfg(feature = "history_builders")]
-mod history_info;
 mod task_token;
 #[cfg(feature = "test-utilities")]
 pub mod test_utils;
 
 use std::time::Duration;
 
-#[cfg(feature = "history_builders")]
-pub use history_builder::{
-    DEFAULT_ACTIVITY_TYPE, DEFAULT_WORKFLOW_TYPE, TestHistoryBuilder, default_act_sched,
-    default_wes_attribs,
-};
-#[cfg(feature = "history_builders")]
-pub use history_info::HistoryInfo;
 pub use task_token::TaskToken;
 
 /// Payload metadata key that identifies the encoding format.

crates/common/src/protos/test_utils.rs

@@ -2,7 +2,6 @@
 //! Only available when the test-utilities feature is enabled
 
 use crate::protos::{
-    DEFAULT_ACTIVITY_TYPE,
     coresdk::workflow_commands::{
         ActivityCancellationType, QueryResult, QuerySuccess, ScheduleActivity,
         ScheduleLocalActivity, StartTimer, workflow_command,
@@ -11,6 +10,10 @@ use crate::protos::{
 };
 use std::time::Duration;
 
+/// Default activity type used by these test command builders. Kept in sync by convention with
+/// the constant of the same name in `temporalio_sdk_core::replay::DEFAULT_ACTIVITY_TYPE`.
+const DEFAULT_ACTIVITY_TYPE: &str = "default_act_type";
+
 /// Convenience macro for creating prost Duration from std::time::Duration
 #[macro_export]
 macro_rules! prost_dur {

crates/sdk-core/Cargo.toml

@@ -110,7 +110,7 @@ zip = { version = "8.4", optional = true, default-features = false, features = [
 [dependencies.temporalio-common]
 path = "../common"
 version = "0.2"
-features = ["core-based-sdk", "history_builders", "test-utilities"]
+features = ["core-based-sdk", "test-utilities"]
 
 [dependencies.temporalio-client]
 path = "../client"

crates/sdk-core/src/core_tests/activity_tasks.rs

@@ -1,5 +1,6 @@
 use crate::{
     ActivityHeartbeat, CompleteActivityError, Worker, advance_fut, job_assert, prost_dur,
+    replay::{TestHistoryBuilder, canned_histories},
     test_help::{
         FakeWfResponses, MockPollCfg, MockWorkerInputs, MocksHolder, QueueResponse, WorkerExt,
         WorkflowCachingPolicy, build_fake_worker, build_mock_pollers, build_multihist_mock_sg,
@@ -26,7 +27,6 @@ use std::{
 };
 use temporalio_common::{
     protos::{
-        TestHistoryBuilder, canned_histories,
         coresdk::{
             ActivityTaskCompletion,
             activity_result::{

crates/sdk-core/src/core_tests/mod.rs

@@ -8,6 +8,7 @@ mod workflow_tasks;
 
 use crate::{
     PollError, Worker,
+    replay::{TestHistoryBuilder, canned_histories},
     test_help::{
         MockPollCfg, build_mock_pollers, mock_worker, single_hist_mock_sg, test_worker_cfg,
     },
@@ -19,7 +20,6 @@ use crate::{
 use futures_util::FutureExt;
 use std::{sync::LazyLock, time::Duration};
 use temporalio_common::protos::{
-    TestHistoryBuilder, canned_histories,
     coresdk::{
         workflow_activation::{WorkflowActivationJob, workflow_activation_job},
         workflow_completion::WorkflowActivationCompletion,

crates/sdk-core/src/core_tests/queries.rs

@@ -1,4 +1,5 @@
 use crate::{
+    replay::{TestHistoryBuilder, canned_histories},
     test_help::{
         MockPollCfg, MocksHolder, ResponseType, WorkerExt, WorkerTestHelpers, build_mock_pollers,
         hist_to_poll_resp, mock_worker, single_hist_mock_sg,
@@ -19,7 +20,6 @@ use std::{
 };
 use temporalio_client::MESSAGE_TOO_LARGE_KEY;
 use temporalio_common::protos::{
-    TestHistoryBuilder, canned_histories,
     coresdk::{
         workflow_activation::{
             WorkflowActivationJob, remove_from_cache::EvictionReason, workflow_activation_job,

crates/sdk-core/src/core_tests/replay_flag.rs

@@ -1,10 +1,10 @@
 use crate::{
+    replay::{TestHistoryBuilder, canned_histories},
     test_help::{MockPollCfg, ResponseType, build_mock_pollers, hist_to_poll_resp, mock_worker},
     worker::{LEGACY_QUERY_ID, client::mocks::mock_worker_client},
 };
 use std::{collections::VecDeque, time::Duration};
 use temporalio_common::protos::{
-    TestHistoryBuilder, canned_histories,
     coresdk::{
         workflow_activation::{WorkflowActivationJob, workflow_activation_job},
         workflow_completion::WorkflowActivationCompletion,

crates/sdk-core/src/core_tests/updates.rs

@@ -1,13 +1,13 @@
 use crate::{
     prost_dur,
+    replay::{DEFAULT_ACTIVITY_TYPE, TestHistoryBuilder},
     test_help::{
         MockPollCfg, PollWFTRespExt, ResponseType, WorkerTestHelpers, build_mock_pollers,
         hist_to_poll_resp, mock_worker,
     },
     worker::client::mocks::mock_worker_client,
 };
 use temporalio_common::protos::{
-    DEFAULT_ACTIVITY_TYPE, TestHistoryBuilder,
     coresdk::{
         workflow_activation::{WorkflowActivationJob, workflow_activation_job},
         workflow_commands::{