docs: Add how it works documentation (#88)

* docs: start working on how it works section in docs

* typo: fix typo in README

* format: format the docs readme files

* Finish worker docs

* Update docs index

Author: CCXLV

README.md

@@ -81,7 +81,7 @@ Running the async function in an async context will also enqueue the task.
 
 ## Installing the worker
 
-In order the tasks to be executed you need to run a FluxQueue worker. You need to install the worker on your system, recommended way of doing that using `fluxqueue-worker`:
+In order the tasks to be executed you need to run a FluxQueue worker. You need to install the worker on your system, recommended way of doing that is using `fluxqueue-worker`:
 
 ```bash
 fluxqueue worker install

docs/how-it-works/client.md

@@ -0,0 +1,31 @@
+# Client Library
+
+In this section, we are going to talk about the client library. For installation, please visit the [Installation Tutorial](../tutorial/installation.md).
+
+## FluxQueue class
+
+When you import the `FluxQueue` class from the library and initialize it:
+
+```py
+from fluxqueue import FluxQueue
+
+fluxqueue = FluxQueue(redis_url="redis://localhost:6379")
+```
+
+You might think that this establishes a TCP connection to the Redis server, but it does not. Internally, this only saves the `redis_url` you pass to the class, which is then used by the `enqueue` methods on the Rust side of the library.
+
+## `task` decorator
+
+This decorator is used to mark a function as a task function, which lets you enqueue the task by just running that function:
+
+```py
+@fluxqueue.task()
+def send_email_task(email: str):
+    send_email(email)
+
+send_email_task()
+```
+
+When the function is executed, it internally calls a Rust method that establishes a Redis connection and pushes the task into the queue. This entire process is handled on the Rust side of the library.
+
+On the Python side, the only responsibility is dispatching based on the function type. If the function is asynchronous, the async Redis connection method is used, and the resulting Rust Future is converted into a Python awaitable using `pyo3_async_runtimes::tokio::future_into_py`, allowing it to be awaited in Python. If the function is synchronous, a standard (blocking) Redis connection is used instead.

docs/how-it-works/index.md

@@ -0,0 +1,8 @@
+# How it Works
+
+As you may already know, FluxQueue is composed of two main components: a Python client library and a system-level worker. These components work together to enqueue and execute tasks. In this documentation we are going to talk about how they actually work.
+
+## Sections
+
+- **[Client Library](client.md)** - How Client library works
+- **[FluxQueue Worker](worker.md)** - How FluxQueue worker works

docs/how-it-works/worker.md

@@ -0,0 +1,58 @@
+# FluxQueue Worker
+
+The FluxQueue worker is a Rust binary which internally runs a `tokio` multi-threaded runtime. The `--concurrency` argument it takes actually determines the number of `tokio` tasks it is going to spawn. That's why even with a single worker and higher `concurrency` it performs well with high concurrency while keeping memory usage low.
+
+## Executor Loop
+
+An executor loop is a `tokio` task that the runtime spawns for a given worker. For each unit of `concurrency`, the worker creates one executor loop that:
+
+- Dequeues the next task from Redis and marks it as "processing".
+- Looks up the corresponding Python function in the task registry.
+- Executes the function and waits for it to finish.
+- On success, removes the task from the processing list.
+- On failure, marks the task as failed so it can be retried by the janitor loop.
+
+The task registry itself is a shared, thread-safe registry of task functions, so all executors can look up Python callables by name without copying them. This lets multiple executor loops perform fast, concurrent lookups from the same set of registered tasks.
+
+## Janitor Loop
+
+In addition to the executor loops, the worker also runs a single janitor loop as another `tokio` task. The janitor loop is responsible for:
+
+- Periodically checking Redis for tasks that have failed.
+- Deciding whether a failed task should be retried or considered "dead" based on its `retries` and `max_retries` fields.
+- Requeuing failed tasks back into the main queue if they still have retries left.
+- Optionally pushing tasks that have used all retries into a "dead tasks" list when `--save-dead-tasks` is enabled. These dead tasks are kept only for inspection and debugging and are not reprocessed by the worker.
+- Sending heartbeat updates for all executors so they can be tracked as healthy by the system.
+
+This separation lets executor loops focus purely on running user code, while the janitor loop handles retries, cleanup, and health tracking in the background.
+
+## Graceful Shutdown
+
+When you stop the worker with `Ctrl+C`:
+
+- The worker broadcasts a shutdown signal to all executor loops and the janitor loop.
+- Each loop finishes the task it is currently processing but does not take new work from Redis.
+- After all loops exit, the worker removes executor registrations from Redis and then terminates.
+- The main task queue is not removed, so a restarted worker continues processing any remaining tasks in the queue.
+
+## Handling Async Tasks
+
+When a task is dispatched, the worker first calls the corresponding Python function with its deserialized `args` and `kwargs`. If the function returns a regular value, it is treated as a **synchronous task** and completes on the executor loop itself.
+
+If the function returns a coroutine (i.e. an `async def` in Python), it is treated as an **asynchronous task**. In that case:
+
+- The coroutine object is sent to a dedicated Python dispatcher.
+- The dispatcher converts the Python coroutine into a Rust `Future` using `pyo3_async_runtimes::tokio::into_future`.
+- That future is then polled to completion on the Tokio runtime, so the async Python code runs concurrently with other tasks.
+
+This design lets you freely mix sync and async Python task functions while keeping the Rust worker fully async and non-blocking.
+
+## Python Dispatcher
+
+The Python dispatcher is a small component that owns a dedicated Python event loop and hides the complexity of driving async Python code from Rust. Internally, it:
+
+- Runs a background thread that initializes Python and starts a Tokio runtime bound to the Python interpreter.
+- Listens on an internal channel for incoming Python coroutine objects sent by executor loops.
+- For each coroutine, wraps it into a Rust `Future` via `pyo3_async_runtimes::tokio::into_future` and awaits it on the dispatcher’s Tokio runtime.
+
+Because all async Python work is funneled through this dispatcher, executor loops still acquire the Python GIL when calling into Python code, but they do not need to manage a Python async event loop themselves. They send the coroutine to the dispatcher over an internal `tokio::sync::mpsc` channel and then await the result like any other async Rust function.

docs/index.md

@@ -13,6 +13,7 @@ Welcome to FluxQueue documentation. FluxQueue is lightweight, high-throughput ta
 - [Quick Start](tutorial/quickstart.md) - Your first task queue
 - [Defining and Exposing Tasks](tutorial/defininig_and_exposing_tasks.md) - Organize and expose tasks for the worker
 - [Worker Setup](tutorial/worker.md) - Deploy and run workers
+- [How it Works](how-it-works/index.md) - Learn more about how FluxQueue actually works
 - [API Reference](api/index.md) - Complete API documentation
 
 ## What is FluxQueue?

docs/tutorial/installation.md

@@ -1,6 +1,6 @@
 # Installation
 
-FluxQueue has two parts: the Python client library for enqueueing tasks, and the Rust worker that executes them. Both use Rust under the hood for speed. The worker needs to be installed separately on your system. Currently Linux is supported, with Windows and macOS support coming soon. The Python version range will be expanded in future releases as well.
+FluxQueue has two parts: the Python client library for enqueueing tasks, and the Rust worker that executes them. Both use Rust under the hood for it's low memory usage. The worker needs to be installed separately on your system. Currently Linux is supported, with Windows and macOS support coming soon. The Python version range will be expanded in future releases as well.
 
 ## Prerequisites
 

mkdocs.yml

@@ -65,6 +65,10 @@ nav:
       - tutorial/quickstart.md
       - tutorial/defininig_and_exposing_tasks.md
       - tutorial/worker.md
+  - How it Works:
+      - how-it-works/index.md
+      - how-it-works/client.md
+      - how-it-works/worker.md
   - API Reference:
       - api/index.md
       - api/fluxqueue.md