Merge pull request #26 from joelhelbling/feat/concurrency-analysis-docs

Analyze concurrency model and add use case documentation

Author: joelhelbling

README.md

@@ -7,6 +7,14 @@
 
 _"How many Ruby fibers does it take to screw in a lightbulb?"_
 
+## Concurrency Model
+
+Shifty utilizes Ruby Fibers for cooperative multitasking. This means that all tasks (or "workers") run within a single operating system thread and explicitly yield control to one another. This model is intentionally chosen for its simplicity, which makes it easier to reason about and build sequential data processing pipelines.
+
+Single-threading frees you from *preemptive* concurrency hazards (races, mutexes) within a pipeline. It does not, by itself, make the data passing between workers safe — a worker that mutates a value it was handed can still corrupt what later workers see. Governing those handoffs is a separate concern; see the planned [handoff immutability policies](docs/planning/handoff-immutability-policies.md). The Fiber model is also deliberately compatible with a future Ractor-backed worker type, rather than a rejection of parallelism.
+
+For a more detailed explanation of Shifty's design and typical use cases, please see the [Use Cases Document](docs/use_cases.md).
+
 ## Quick Start
 
 Add this line to your application's Gemfile:

docs/use_cases.md

@@ -0,0 +1,57 @@
+# Shifty Framework: Use Cases
+
+## Introduction
+
+Shifty is a Ruby framework designed for building data processing pipelines. It utilizes a system of cooperatively multitasking "workers" that operate in a single thread. Each worker performs a specific task and can pass data to the next worker in a chain, allowing for the creation of complex data flows in a manageable and sequential manner.
+
+## When to Use Shifty?
+
+Shifty is well-suited for a variety of scenarios where data needs to be processed in a step-by-step fashion. Consider using Shifty for:
+
+*   **Building Data Processing Pipelines:** When your task can be broken down into a series of sequential steps, where each step transforms or enriches the data. Shifty allows you to encapsulate each step within a dedicated worker.
+*   **ETL-like Workflows:** For scenarios that involve extracting data from a source, transforming it according to certain rules, and then loading it elsewhere. While Shifty is primarily focused on the in-application transformation aspects, it can be a valuable part of a larger ETL process within a Ruby application.
+*   **Cooperative Multitasking Scenarios:** When you have multiple tasks that can effectively "take turns" executing. This is particularly useful if tasks involve waiting for non-blocking operations (though the current version of Shifty operates synchronously) or involve complex stateful interactions that are simpler to manage cooperatively rather than with preemptive threading.
+*   **Simplifying Complex Sequential Logic:** If you have a long, monolithic process with many stages, Shifty can help by breaking it down into a chain of smaller, focused, and more understandable workers. This improves modularity and maintainability.
+*   **Creating Reusable Components:** Shifty encourages the design of workers that perform specific, well-defined tasks. These workers can then be reused and recombined in different pipelines for various purposes, promoting code reuse.
+
+## Example Scenarios
+
+Here are a few conceptual examples of how Shifty could be applied:
+
+*   **Log Processing:**
+    Imagine a pipeline for processing application logs:
+    1.  A `FileReaderWorker` reads log lines from a file.
+    2.  A `LogParserWorker` parses each line into a structured format (e.g., timestamp, level, message).
+    3.  A `ErrorFilterWorker` checks the parsed log data and only passes on entries marked as "ERROR" or "CRITICAL".
+    4.  A `ReportFormatterWorker` formats these error entries into a human-readable report.
+    5.  A `FileWriterWorker` or `EmailNotifierWorker` outputs the report.
+
+*   **Data Transformation Chain:**
+    A simple pipeline demonstrating data manipulation:
+    1.  A `NumberSourceWorker` generates a sequence of numbers (e.g., 1, 2, 3, ...).
+    2.  A `MultiplierWorker` receives each number and multiplies it by 2.
+    3.  A `StringFormatterWorker` converts the multiplied number into a string, perhaps adding a prefix (e.g., "RESULT: 4").
+    4.  A `ConsoleOutputWorker` prints the final formatted string to the console.
+
+*   **Batch Processing:**
+    Accumulating data into batches before further processing:
+    1.  An `ItemStreamWorker` produces individual items (e.g., from a database query or an incoming data stream).
+    2.  A `BatchWorker` (Shifty provides a `batch_worker` for this purpose) accumulates these items. It passes the accumulated batch to the next worker once a certain number of items are collected or a timeout occurs.
+    3.  A `BatchProcessorWorker` then processes the entire batch of items at once (e.g., bulk database insert, writing to a file).
+
+## A Note on Values Passed Between Workers
+
+The examples above pass data from one worker to the next. Because Shifty runs
+each value through every worker before starting the next value, workers should
+treat a handed-off value as read-only and express changes as new values
+(`arr + [x]`, `hash.merge(...)`, `value.with(...)`) rather than mutating in
+place (`arr <<`, `hash[k] =`, `map!`). This keeps failures local and is the
+direction Shifty is moving: a planned release makes deeply frozen handoffs the
+default, with opt-in policies for workers that genuinely need a private scratch
+copy or a shared mutable reference. See
+[handoff immutability policies](planning/handoff-immutability-policies.md) for
+the full design and migration guidance.
+
+## Conclusion
+
+Shifty aims to provide an intuitive and straightforward way to construct data processing systems within Ruby applications. By breaking down complex tasks into manageable, cooperatively multitasking workers, it helps in building modular, maintainable, and easy-to-understand data pipelines.

lib/shifty/worker.rb

@@ -2,6 +2,12 @@
 require "shifty/taggable"
 
 module Shifty
+  # Shifty::Worker uses Ruby Fibers for cooperative multitasking.
+  # This results in a single-threaded execution model where workers
+  # explicitly yield control to one another. This model is chosen for its
+  # simplicity and suitability for creating chainable data processing
+  # pipelines, where each worker performs a specific task and passes
+  # its output to the next worker in the chain.
   class Worker
     attr_reader :supply, :tags
 
@@ -50,6 +56,9 @@ def ensure_ready_to_work!
     end
 
     def workflow
+      # This is the core of the worker's execution, managed by a Fiber.
+      # The Fiber allows the worker to pause its execution (yield) and
+      # be resumed later, enabling cooperative multitasking.
       @my_little_machine ||= Fiber.new {
         loop do
           value = supply&.shift
@@ -77,6 +86,48 @@ def task_method_exists?
     def task_method_accepts_a_value?
       method(:task).arity > 0
     end
+
+    # ## Concurrency and Thread Safety
+    #
+    # ### Concurrency Model
+    # Shifty utilizes Ruby Fibers to achieve cooperative multitasking. This means
+    # that workers voluntarily yield control, allowing other workers to execute.
+    # The entire processing pipeline runs within a single thread, which removes
+    # an entire class of *preemptive* concurrency hazards (races on shared
+    # objects, the need for mutexes around Shifty's own state).
+    #
+    # Note that single-threading does NOT make the data flowing between workers
+    # safe on its own: because each value passes through every worker before the
+    # next value begins, a worker that mutates a handed-off value can silently
+    # corrupt what downstream workers observe. That hazard is orthogonal to
+    # threading and is addressed separately by handoff immutability policies
+    # (see docs/planning/handoff-immutability-policies.md).
+    #
+    # ### Alternatives (Threads/Ractors)
+    # Threads and Ractors are not used for parallelism *yet*. Shifty's current
+    # goal is an easy-to-use framework for sequential data pipelines, and a
+    # single-threaded Fiber model serves that directly without the overhead of
+    # mutexes (Threads) or the sharing restrictions of Ractors. This is a
+    # scoping decision, not a rejection: the planned move to deeply frozen,
+    # shareable handoff values is deliberately Ractor-compatible and lays the
+    # groundwork for a future Ractor-backed worker type. (Fibers cannot cross
+    # Ractor boundaries, so such a worker would be a distinct type, not a
+    # retrofit of this one.)
+    #
+    # ### Thread Safety
+    # `Shifty::Worker` instances, and by extension `Shifty::Gang` or
+    # `Shifty::Roster` instances that manage these workers, are **not**
+    # inherently thread-safe if they are shared and modified across multiple
+    # user-created native threads. If users choose to integrate Shifty components
+    # into a multi-threaded application (e.g., processing multiple independent
+    # Shifty pipelines in parallel using separate Threads), they are responsible
+    # for implementing appropriate synchronization mechanisms (like mutexes)
+    # to protect shared Shifty objects from concurrent access and modification.
+    # (Handoff immutability policies govern only the *values* passed between
+    # workers, not the worker/pipeline objects themselves or their closure
+    # state — those remain the user's responsibility across threads.)
+    # For typical use cases where a Shifty pipeline is built and run, no
+    # external threading is usually involved by the user.
   end
 
   class WorkerError < StandardError; end