Skip to content

Commit 986aea1

Browse files
sudoRicheeksudoRicheek
committed
docs added
1 parent 1fd549f commit 986aea1

64 files changed

Lines changed: 50416 additions & 261 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/components/discovery.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# Discovery
2+
3+
> **Source:** [`cortex.discovery.daemon`](../reference/discovery/daemon.md),
4+
> [`cortex.discovery.client`](../reference/discovery/client.md),
5+
> [`cortex.discovery.protocol`](../reference/discovery/protocol.md)
6+
7+
Discovery is Cortex's control plane: a single long-lived process that maps
8+
topic names to ZMQ endpoints. It sits off the data path — once a subscriber
9+
has an endpoint, messages flow publisher → subscriber directly without the
10+
daemon's involvement.
11+
12+
## Moving parts
13+
14+
```mermaid
15+
flowchart LR
16+
subgraph DP[discovery package]
17+
PR[protocol.py<br/>DiscoveryRequest /<br/>DiscoveryResponse /<br/>TopicInfo]
18+
DM[daemon.py<br/>DiscoveryDaemon<br/>ZMQ REP loop]
19+
CL[client.py<br/>DiscoveryClient<br/>ZMQ REQ wrapper]
20+
end
21+
22+
CL -- msgpack REQ --> DM
23+
DM -- msgpack REP --> CL
24+
PR -.-> DM
25+
PR -.-> CL
26+
```
27+
28+
Everyone agrees on the wire format via `protocol.py`. The daemon runs a
29+
single-threaded REP loop. The client speaks REQ from every publisher and
30+
subscriber in the graph.
31+
32+
## Daemon
33+
34+
Implemented in [`DiscoveryDaemon`][cortex.discovery.daemon.DiscoveryDaemon].
35+
36+
Key behaviors:
37+
38+
- Binds `zmq.REP` at `ipc:///tmp/cortex/discovery.sock` by default.
39+
- Maintains `_topics: dict[str, TopicInfo]` — **one publisher per topic**.
40+
- `RCVTIMEO=1000` on the socket so the loop can check `_running` for clean
41+
Ctrl-C. This also means the daemon is naturally single-request-at-a-time —
42+
a slow client blocks all others.
43+
44+
### State transitions
45+
46+
```mermaid
47+
stateDiagram-v2
48+
[*] --> Starting
49+
Starting --> Running: bind OK
50+
Running --> Running: REGISTER → insert
51+
Running --> Running: LOOKUP → read
52+
Running --> Running: UNREGISTER → delete
53+
Running --> Running: LIST → snapshot
54+
Running --> Stopping: SIGINT / SHUTDOWN
55+
Stopping --> [*]: close socket, unlink .sock
56+
```
57+
58+
### Registry semantics
59+
60+
| Case | Result |
61+
| -------------------------------------- | ------------------ |
62+
| New topic | Insert → OK |
63+
| Same topic, same `publisher_node` | Overwrite → OK (re-registration) |
64+
| Same topic, different `publisher_node` | Reject → ALREADY_EXISTS |
65+
| UNREGISTER missing topic | NOT_FOUND |
66+
67+
## Client
68+
69+
Implemented in [`DiscoveryClient`][cortex.discovery.client.DiscoveryClient].
70+
71+
Thin REQ wrapper around the protocol. Important operational detail: **REQ
72+
sockets stick after a timeout** — they block subsequent sends waiting for a
73+
reply that never came. The client handles this by closing and recreating the
74+
socket on every timeout (`_reconnect`). Callers don't see it.
75+
76+
### REQ timeout recovery
77+
78+
```mermaid
79+
flowchart TD
80+
S[send request] --> W[wait RCVTIMEO]
81+
W -->|reply| OK[return DiscoveryResponse]
82+
W -->|timeout| T[zmq.Again]
83+
T --> C[close REQ socket]
84+
C --> N[create fresh REQ<br/>same endpoint]
85+
N -->|attempts < retries| S
86+
N -->|exhausted| F[raise TimeoutError]
87+
```
88+
89+
### Polling helpers
90+
91+
- [`lookup_topic(name)`][cortex.discovery.client.DiscoveryClient.lookup_topic] —
92+
one-shot, returns `None` on miss.
93+
- [`wait_for_topic(name, timeout, poll_interval)`][cortex.discovery.client.DiscoveryClient.wait_for_topic] —
94+
blocking poll loop (time.sleep).
95+
- [`wait_for_topic_async(name, timeout, poll_interval)`][cortex.discovery.client.DiscoveryClient.wait_for_topic_async] —
96+
async poll loop (asyncio.sleep). This is what [`Subscriber`][cortex.core.subscriber.Subscriber]
97+
uses when `wait_for_topic=True`.
98+
99+
## Protocol
100+
101+
Implemented in [`cortex.discovery.protocol`](../reference/discovery/protocol.md).
102+
103+
| Type | Purpose |
104+
| -------------------------------------------------------------------- | ----------------------------------------- |
105+
| [`DiscoveryCommand`][cortex.discovery.protocol.DiscoveryCommand] | `REGISTER_TOPIC` / `UNREGISTER_TOPIC` / `LOOKUP_TOPIC` / `LIST_TOPICS` / `SHUTDOWN` |
106+
| [`DiscoveryStatus`][cortex.discovery.protocol.DiscoveryStatus] | `OK` / `NOT_FOUND` / `ALREADY_EXISTS` / `ERROR` |
107+
| [`TopicInfo`][cortex.discovery.protocol.TopicInfo] | name, address, message_type, fingerprint, publisher_node |
108+
| [`DiscoveryRequest`][cortex.discovery.protocol.DiscoveryRequest] | command + optional topic_info / topic_name |
109+
| [`DiscoveryResponse`][cortex.discovery.protocol.DiscoveryResponse] | status, message, topic_info, topics |
110+
111+
All payloads are msgpack. `TopicInfo` is nested as a packed sub-blob so
112+
discovery responses stay flat.
113+
114+
## Known limitations
115+
116+
Summarized here, detailed in [critique.md](../critique.md):
117+
118+
- One-publisher-per-topic.
119+
- No heartbeats or leases — crashed publishers leave stale entries.
120+
- Single-threaded REP — slow client starves others.
121+
- `retries=1` in the client is a fencepost; effective retries today is zero.
122+
- Daemon state lost on restart; publishers do not auto-re-register.
123+
124+
## See also
125+
126+
- [Concepts → Discovery protocol](../concepts/discovery-protocol.md)
127+
- [Getting started → Running the discovery daemon](../getting-started/discovery-daemon.md)
128+
- [Critique](../critique.md)

‎docs/components/messages.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Messages
2+
3+
> **Source:** [`cortex.messages.base`](../reference/messages/base.md),
4+
> [`cortex.messages.standard`](../reference/messages/standard.md)
5+
6+
Messages are just `@dataclass`es that inherit from
7+
[`Message`][cortex.messages.base.Message]. Registering with the type system,
8+
computing a fingerprint, and (de)serialization all happen automatically.
9+
10+
## Anatomy of a message
11+
12+
```mermaid
13+
classDiagram
14+
class Message {
15+
+fingerprint() int
16+
+to_bytes() bytes
17+
+to_frames() list
18+
+from_bytes(data) tuple
19+
+from_frames(frames) tuple
20+
+decode(bytes) tuple [static]
21+
-_build_header()
22+
-_field_names() tuple
23+
-_field_values() list
24+
-_next_sequence() int
25+
}
26+
class MessageHeader {
27+
+fingerprint: int
28+
+timestamp_ns: int
29+
+sequence: int
30+
+to_bytes() bytes
31+
+from_bytes(data) MessageHeader
32+
+size() int
33+
}
34+
class MessageType {
35+
+register(cls)
36+
+get(fingerprint) type
37+
+get_all() dict
38+
}
39+
Message ..> MessageHeader : emits
40+
Message ..> MessageType : auto-registers on subclass
41+
```
42+
43+
## Defining a custom message
44+
45+
```python
46+
from dataclasses import dataclass
47+
import numpy as np
48+
from cortex.messages.base import Message
49+
50+
@dataclass
51+
class JointTrajectory(Message):
52+
timestamp: float
53+
positions: np.ndarray # shape (N,)
54+
velocities: np.ndarray # shape (N,)
55+
frame_id: str = ""
56+
```
57+
58+
That is the entire contract. The class is registered into
59+
[`MessageType._registry`][cortex.messages.base.MessageType] by fingerprint at
60+
import time, and gains:
61+
62+
- `JointTrajectory.fingerprint()` — 64-bit ID.
63+
- `msg.to_frames()` / `JointTrajectory.from_frames(frames)` — the transport path.
64+
- `msg.to_bytes()` / `JointTrajectory.from_bytes(data)` — the legacy blob path.
65+
- `Message.decode(blob)` — class dispatch via fingerprint registry.
66+
67+
## Sequence numbering
68+
69+
!!! warning "Class-level counter"
70+
`Message._sequence_counter` is shared across **all publisher instances** of
71+
the same message class in the process. Two `ArrayMessage` publishers
72+
interleave sequence numbers. Per-topic gap detection therefore needs a
73+
per-publisher counter today; see [critique.md § 12](../critique.md).
74+
75+
## Built-in messages
76+
77+
| Class | Use for |
78+
| ------------------------------------------------------------------------- | --------------------------------------------- |
79+
| [`StringMessage`][cortex.messages.standard.StringMessage] | Plain strings |
80+
| [`IntMessage`][cortex.messages.standard.IntMessage] / [`FloatMessage`][cortex.messages.standard.FloatMessage] | Single scalars |
81+
| [`BytesMessage`][cortex.messages.standard.BytesMessage] | Opaque binary |
82+
| [`DictMessage`][cortex.messages.standard.DictMessage] | Nested dicts with arrays/tensors |
83+
| [`ListMessage`][cortex.messages.standard.ListMessage] | Mixed-type lists |
84+
| [`ArrayMessage`][cortex.messages.standard.ArrayMessage] | Single NumPy array + name / frame_id |
85+
| [`MultiArrayMessage`][cortex.messages.standard.MultiArrayMessage] | `dict[str, np.ndarray]` (e.g. points+colors) |
86+
| [`TensorMessage`][cortex.messages.standard.TensorMessage] | PyTorch tensor (preserves device/grad) |
87+
| [`MultiTensorMessage`][cortex.messages.standard.MultiTensorMessage] | Named tensor bundle (model I/O) |
88+
| [`ImageMessage`][cortex.messages.standard.ImageMessage] | Image + encoding + width/height |
89+
| [`PointCloudMessage`][cortex.messages.standard.PointCloudMessage] | XYZ + optional RGB / intensity / normals |
90+
| [`PoseMessage`][cortex.messages.standard.PoseMessage] | 6-DoF pose (position + quaternion) |
91+
| [`TransformMessage`][cortex.messages.standard.TransformMessage] | 4×4 homogeneous transform |
92+
| [`TimestampMessage`][cortex.messages.standard.TimestampMessage] / [`HeaderMessage`][cortex.messages.standard.HeaderMessage] | ROS-style stamps |
93+
94+
## Encode / decode lifecycle
95+
96+
```mermaid
97+
flowchart LR
98+
A[User builds dataclass] --> B[Publisher.publish]
99+
B --> C[message.to_frames]
100+
C --> D[[ZMQ multipart send]]
101+
D --> E[[ZMQ multipart recv]]
102+
E --> F[Message.from_frames]
103+
F --> G[user callback msg, header]
104+
```
105+
106+
## See also
107+
108+
- [Concept: message wire format](../concepts/message-wire-format.md)
109+
- [Concept: fingerprinting](../concepts/fingerprinting.md)
110+
- [Tutorial: custom messages](../tutorials/custom-messages.md)

0 commit comments

Comments
 (0)