Skip to content

Commit beac8bc

Browse files
committed
Formatting
1 parent 6b27aa2 commit beac8bc

1 file changed

Lines changed: 39 additions & 25 deletions

File tree

README.md

Lines changed: 39 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,46 +1,60 @@
11
# streamfleet
22

3-
Customizable Go work queue implementation backed by Redis Streams (or Valkey Streams). No Lua or fancy tricks required.
3+
Customizable Go work queue implementation backed by Redis Streams (or Valkey
4+
Streams). No Lua or fancy tricks required.
45

5-
Tasks placed on the queue are processed once by the worker that can pick up the task quickest.
6-
Completion can be optionally tracked.
6+
Tasks placed on the queue are processed once by the worker that can pick up the
7+
task quickest. Completion can be optionally tracked.
78

89
## Features
910

10-
- Optional task completion notifications
11-
- Support for Redis Sentinel and Cluster
12-
- Automatic retry and worker crash recovery
13-
- Tolerance for Redis downtime without data loss
11+
- Optional task completion notifications
12+
- Support for Redis Sentinel and Cluster
13+
- Automatic retry and worker crash recovery
14+
- Tolerance for Redis downtime without data loss
1415

1516
## Architecture
1617

1718
The library is broken up into two components, coordinated by Redis:
1819

19-
- The client, which submits (enqueues) tasks to the queue.
20-
21-
Clients can optionally track completion or failure of tasks by listening for notifications about them.
22-
Whether to send completion notifications is sent along with the task.
20+
- The client, which submits (enqueues) tasks to the queue.
2321

24-
- The server (worker), which accepts work from the queue and processes it.
22+
Clients can optionally track completion or failure of tasks by listening for
23+
notifications about them. Whether to send completion notifications is sent
24+
along with the task.
2525

26-
In addition to receiving new tasks, workers can claim tasks that have sat idle too long in other workers' pending lists.
27-
Long-running tasks are kept fresh and not in an idle state as long as the task is pending in the functional server.
28-
Only servers that have crashed or are unresponsive would have their tasks reclaimed.
26+
- The server (worker), which accepts work from the queue and processes it.
2927

30-
If a task fails while the worker is running, the server will put the task back on the queue and increment its failure count.
28+
In addition to receiving new tasks, workers can claim tasks that have sat idle
29+
too long in other workers' pending lists. Long-running tasks are kept fresh
30+
and not in an idle state as long as the task is pending in the functional
31+
server. Only servers that have crashed or are unresponsive would have their
32+
tasks reclaimed.
3133

32-
Note that servers and clients may exist on the same process; they do not need to be in separate microservices.
34+
If a task fails while the worker is running, the server will put the task back
35+
on the queue and increment its failure count.
3336

34-
Task notifications are sent over a client-specific stream for reliable delivery, even if there is temporary disconnection from Redis.
35-
Clients clean up these streams when their Close method is called to avoid resource leaks.
36-
In the event that they are not closed cleanly, other clients and servers use a simple garbage collector based on client heartbeats to find orphaned client streams and delete them.
37+
Note that servers and clients may exist on the same process; they do not need to
38+
be in separate microservices.
3739

38-
If the underlying Redis server is unavailable, the clients will wait until it comes back online.
39-
Tasks submitted to the client stay queued in-memory until Redis is available.
40+
Task notifications are sent over a client-specific stream for reliable delivery,
41+
even if there is temporary disconnection from Redis. Clients clean up these
42+
streams when their Close method is called to avoid resource leaks. In the event
43+
that they are not closed cleanly, other clients and servers use a simple garbage
44+
collector based on client heartbeats to find orphaned client streams and delete
45+
them.
4046

41-
The overall design philosophy is to support queues with high error tolerance without losing anything, and to have as few moving parts as possible.
42-
Many systems already have Redis, why not use its stream feature for reliable message delivery? No need to introduce a complex system like Kafka.
43-
As well as being fault-tolerance, the implementation aims to be as autonomous as possible. There are no features that rely on centralized coordination, other than Redis itself (which can be clustered for high-availability).
47+
If the underlying Redis server is unavailable, the clients will wait until it
48+
comes back online. Tasks submitted to the client stay queued in-memory until
49+
Redis is available.
50+
51+
The overall design philosophy is to support queues with high error tolerance
52+
without losing anything, and to have as few moving parts as possible. Many
53+
systems already have Redis, why not use its stream feature for reliable message
54+
delivery? No need to introduce a complex system like Kafka. As well as being
55+
fault-tolerance, the implementation aims to be as autonomous as possible. There
56+
are no features that rely on centralized coordination, other than Redis itself
57+
(which can be clustered for high-availability).
4458

4559
## Use It
4660

0 commit comments

Comments
 (0)