Skip to content

Commit d14823a

Browse files
committed
Reconcile concurrency docs with handoff immutability plan
Distinguish preemptive concurrency hazards (removed by single-threading) from data-mutation hazards across handoffs (not removed); reframe the Threads/Ractors note as a scoping decision that is Ractor-compatible; and scope the thread-safety caveat to worker/pipeline objects and closure state. Forward-reference docs/planning/handoff-immutability-policies.md.
1 parent 43d5948 commit d14823a

3 files changed

Lines changed: 37 additions & 10 deletions

File tree

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ _"How many Ruby fibers does it take to screw in a lightbulb?"_
1111

1212
Shifty utilizes Ruby Fibers for cooperative multitasking. This means that all tasks (or "workers") run within a single operating system thread and explicitly yield control to one another. This model is intentionally chosen for its simplicity, which makes it easier to reason about and build sequential data processing pipelines.
1313

14+
Single-threading frees you from *preemptive* concurrency hazards (races, mutexes) within a pipeline. It does not, by itself, make the data passing between workers safe — a worker that mutates a value it was handed can still corrupt what later workers see. Governing those handoffs is a separate concern; see the planned [handoff immutability policies](docs/planning/handoff-immutability-policies.md). The Fiber model is also deliberately compatible with a future Ractor-backed worker type, rather than a rejection of parallelism.
15+
1416
For a more detailed explanation of Shifty's design and typical use cases, please see the [Use Cases Document](docs/use_cases.md).
1517

1618
## Quick Start

docs/use_cases.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,19 @@ Here are a few conceptual examples of how Shifty could be applied:
3939
2. A `BatchWorker` (Shifty provides a `batch_worker` for this purpose) accumulates these items. It passes the accumulated batch to the next worker once a certain number of items are collected or a timeout occurs.
4040
3. A `BatchProcessorWorker` then processes the entire batch of items at once (e.g., bulk database insert, writing to a file).
4141

42+
## A Note on Values Passed Between Workers
43+
44+
The examples above pass data from one worker to the next. Because Shifty runs
45+
each value through every worker before starting the next value, workers should
46+
treat a handed-off value as read-only and express changes as new values
47+
(`arr + [x]`, `hash.merge(...)`, `value.with(...)`) rather than mutating in
48+
place (`arr <<`, `hash[k] =`, `map!`). This keeps failures local and is the
49+
direction Shifty is moving: a planned release makes deeply frozen handoffs the
50+
default, with opt-in policies for workers that genuinely need a private scratch
51+
copy or a shared mutable reference. See
52+
[handoff immutability policies](planning/handoff-immutability-policies.md) for
53+
the full design and migration guidance.
54+
4255
## Conclusion
4356

4457
Shifty aims to provide an intuitive and straightforward way to construct data processing systems within Ruby applications. By breaking down complex tasks into manageable, cooperatively multitasking workers, it helps in building modular, maintainable, and easy-to-understand data pipelines.

lib/shifty/worker.rb

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -92,18 +92,27 @@ def task_method_accepts_a_value?
9292
# ### Concurrency Model
9393
# Shifty utilizes Ruby Fibers to achieve cooperative multitasking. This means
9494
# that workers voluntarily yield control, allowing other workers to execute.
95-
# The entire processing pipeline runs within a single thread, simplifying
96-
# state management and avoiding many common concurrency issues.
95+
# The entire processing pipeline runs within a single thread, which removes
96+
# an entire class of *preemptive* concurrency hazards (races on shared
97+
# objects, the need for mutexes around Shifty's own state).
98+
#
99+
# Note that single-threading does NOT make the data flowing between workers
100+
# safe on its own: because each value passes through every worker before the
101+
# next value begins, a worker that mutates a handed-off value can silently
102+
# corrupt what downstream workers observe. That hazard is orthogonal to
103+
# threading and is addressed separately by handoff immutability policies
104+
# (see docs/planning/handoff-immutability-policies.md).
97105
#
98106
# ### Alternatives (Threads/Ractors)
99-
# While Ruby's Threads could be used for parallelism, especially for I/O-bound
100-
# tasks, and Ractors (in Ruby 3.0+) for CPU-bound tasks, they would
101-
# introduce significant complexity. Managing thread safety with Threads
102-
# (e.g., using mutexes, avoiding race conditions) or adhering to Ractor's
103-
# message passing and object sharing restrictions would make the framework
104-
# harder to use and reason about. The current Fiber-based model aligns
105-
# well with Shifty's primary goal of providing an easy-to-use framework
106-
# for building sequential data processing pipelines.
107+
# Threads and Ractors are not used for parallelism *yet*. Shifty's current
108+
# goal is an easy-to-use framework for sequential data pipelines, and a
109+
# single-threaded Fiber model serves that directly without the overhead of
110+
# mutexes (Threads) or the sharing restrictions of Ractors. This is a
111+
# scoping decision, not a rejection: the planned move to deeply frozen,
112+
# shareable handoff values is deliberately Ractor-compatible and lays the
113+
# groundwork for a future Ractor-backed worker type. (Fibers cannot cross
114+
# Ractor boundaries, so such a worker would be a distinct type, not a
115+
# retrofit of this one.)
107116
#
108117
# ### Thread Safety
109118
# `Shifty::Worker` instances, and by extension `Shifty::Gang` or
@@ -114,6 +123,9 @@ def task_method_accepts_a_value?
114123
# Shifty pipelines in parallel using separate Threads), they are responsible
115124
# for implementing appropriate synchronization mechanisms (like mutexes)
116125
# to protect shared Shifty objects from concurrent access and modification.
126+
# (Handoff immutability policies govern only the *values* passed between
127+
# workers, not the worker/pipeline objects themselves or their closure
128+
# state — those remain the user's responsibility across threads.)
117129
# For typical use cases where a Shifty pipeline is built and run, no
118130
# external threading is usually involved by the user.
119131
end

0 commit comments

Comments
 (0)