|
1 | 1 | # streamfleet |
2 | 2 |
|
3 | | -Customizable Go work queue implementation backed by Redis Streams (or Valkey Streams). No Lua or fancy tricks required. |
| 3 | +Customizable Go work queue implementation backed by Redis Streams (or Valkey |
| 4 | +Streams). No Lua or fancy tricks required. |
4 | 5 |
|
5 | | -Tasks placed on the queue are processed once by the worker that can pick up the task quickest. |
6 | | -Completion can be optionally tracked. |
| 6 | +Tasks placed on the queue are processed once by the worker that can pick up the |
| 7 | +task quickest. Completion can be optionally tracked. |
7 | 8 |
|
8 | 9 | ## Features |
9 | 10 |
|
10 | | - - Optional task completion notifications |
11 | | - - Support for Redis Sentinel and Cluster |
12 | | - - Automatic retry and worker crash recovery |
13 | | - - Tolerance for Redis downtime without data loss |
| 11 | +- Optional task completion notifications |
| 12 | +- Support for Redis Sentinel and Cluster |
| 13 | +- Automatic retry and worker crash recovery |
| 14 | +- Tolerance for Redis downtime without data loss |
14 | 15 |
|
15 | 16 | ## Architecture |
16 | 17 |
|
17 | 18 | The library is broken up into two components, coordinated by Redis: |
18 | 19 |
|
19 | | - - The client, which submits (enqueues) tasks to the queue. |
20 | | - |
21 | | - Clients can optionally track completion or failure of tasks by listening for notifications about them. |
22 | | - Whether to send completion notifications is sent along with the task. |
| 20 | +- The client, which submits (enqueues) tasks to the queue. |
23 | 21 |
|
24 | | - - The server (worker), which accepts work from the queue and processes it. |
| 22 | + Clients can optionally track completion or failure of tasks by listening for |
| 23 | + notifications about them. Whether to send completion notifications is sent |
| 24 | + along with the task. |
25 | 25 |
|
26 | | - In addition to receiving new tasks, workers can claim tasks that have sat idle too long in other workers' pending lists. |
27 | | - Long-running tasks are kept fresh and not in an idle state as long as the task is pending in the functional server. |
28 | | - Only servers that have crashed or are unresponsive would have their tasks reclaimed. |
| 26 | +- The server (worker), which accepts work from the queue and processes it. |
29 | 27 |
|
30 | | - If a task fails while the worker is running, the server will put the task back on the queue and increment its failure count. |
| 28 | + In addition to receiving new tasks, workers can claim tasks that have sat idle |
| 29 | + too long in other workers' pending lists. Long-running tasks are kept fresh |
| 30 | + and not in an idle state as long as the task is pending in the functional |
| 31 | + server. Only servers that have crashed or are unresponsive would have their |
| 32 | + tasks reclaimed. |
31 | 33 |
|
32 | | -Note that servers and clients may exist on the same process; they do not need to be in separate microservices. |
| 34 | + If a task fails while the worker is running, the server will put the task back |
| 35 | + on the queue and increment its failure count. |
33 | 36 |
|
34 | | -Task notifications are sent over a client-specific stream for reliable delivery, even if there is temporary disconnection from Redis. |
35 | | -Clients clean up these streams when their Close method is called to avoid resource leaks. |
36 | | -In the event that they are not closed cleanly, other clients and servers use a simple garbage collector based on client heartbeats to find orphaned client streams and delete them. |
| 37 | +Note that servers and clients may exist on the same process; they do not need to |
| 38 | +be in separate microservices. |
37 | 39 |
|
38 | | -If the underlying Redis server is unavailable, the clients will wait until it comes back online. |
39 | | -Tasks submitted to the client stay queued in-memory until Redis is available. |
| 40 | +Task notifications are sent over a client-specific stream for reliable delivery, |
| 41 | +even if there is temporary disconnection from Redis. Clients clean up these |
| 42 | +streams when their Close method is called to avoid resource leaks. In the event |
| 43 | +that they are not closed cleanly, other clients and servers use a simple garbage |
| 44 | +collector based on client heartbeats to find orphaned client streams and delete |
| 45 | +them. |
40 | 46 |
|
41 | | -The overall design philosophy is to support queues with high error tolerance without losing anything, and to have as few moving parts as possible. |
42 | | -Many systems already have Redis, why not use its stream feature for reliable message delivery? No need to introduce a complex system like Kafka. |
43 | | -As well as being fault-tolerance, the implementation aims to be as autonomous as possible. There are no features that rely on centralized coordination, other than Redis itself (which can be clustered for high-availability). |
| 47 | +If the underlying Redis server is unavailable, the clients will wait until it |
| 48 | +comes back online. Tasks submitted to the client stay queued in-memory until |
| 49 | +Redis is available. |
| 50 | + |
| 51 | +The overall design philosophy is to support queues with high error tolerance |
| 52 | +without losing anything, and to have as few moving parts as possible. Many |
| 53 | +systems already have Redis, why not use its stream feature for reliable message |
| 54 | +delivery? No need to introduce a complex system like Kafka. As well as being |
| 55 | +fault-tolerance, the implementation aims to be as autonomous as possible. There |
| 56 | +are no features that rely on centralized coordination, other than Redis itself |
| 57 | +(which can be clustered for high-availability). |
44 | 58 |
|
45 | 59 | ## Use It |
46 | 60 |
|
|
0 commit comments