README.md

Small Agent

A lightweight async log parser. It reads raw log lines, parses them into structured records, buffers them in memory, then ships them asynchronously over gRPC once its internal bus reaches 5 MiB.

The lines it is built for are the ones industrial machines write: a stream of telemetry readings with occasional text mixed in, in files large enough that reading one into memory is not an option.

Time, timestep, iteration, arc status:  21.319 ms, 8.252E-6, 1542600, 1
***** WARNING: Pressure limit @ 0.1 bar has been detected:
Post-processing results...DONE

The shape of it

flowchart LR
    F[("machine.log<br/>any size")]

    subgraph agent ["small-agent — one process, two tasks"]
        direction LR
        subgraph reader ["reader task"]
            direction TB
            L["file::lines<br/><i>one line at a time,<br/>never collected</i>"]
            P["parsing::parse_line"]
            B["Bus<br/><i>accumulate until<br/>5 MiB of payload</i>"]
            L --> P --> B
        end
        Q(["queue<br/>depth 2"])
        S["shipper task<br/><i>one batch in flight</i>"]
        B -- "full batch" --> Q --> S
        Q -. "full queue<br/>stops the reader" .-> B
    end

    C[["collector<br/>gRPC"]]
    S -- "Ship(Batch)" --> C
    C -- "Ack" --> S
    F --> L

Loading

Memory is bounded by the box, not by the file: one bus plus at most two queued batches. When the collector slows down the queue fills, the reader blocks and the heap stays flat. That is the dotted arrow.

How one line is classified:

flowchart TB
    IN["a line"] --> BLANK{"blank?"}
    BLANK -- yes --> EMPTY["PEmpty<br/><i>costs nothing</i>"]
    BLANK -- no --> SEV{"subject is one field<br/>and says WARN / ERROR / FATAL?"}
    SEV -- yes --> WARN["PLog(Warn)"]
    SEV -- no --> SHAPE{"equal arity across the colon<br/>and every value a number?"}
    SHAPE -- yes --> EV["PEv(Event)<br/><i>keys read off the line</i>"]
    SHAPE -- no --> INFO["PLog(Info)"]

Loading

Wording is consulted before shape, and that order is the whole design. ERROR: 500 passes the shape test perfectly (one key, one value, 500 is a number), so shape alone would file the alert as a metric named ERROR and nobody would see it again.

Run it

Two terminals: the agent ships, and something has to be listening. Start the collector first; it blocks, which is what a server does.

cargo run --release --bin collector -- --dump

Then, in the second terminal:

cargo run --release -- examples/logs/01-telemetry.log

examples/logs/ holds one small file per behaviour worth seeing: telemetry, severity, plain text, the edge cases, and CRLF. --dump makes the collector print every record rather than a batch tally. Useful on those files, a liability on a real one where it becomes the bottleneck.

The endpoint defaults to http://127.0.0.1:50051 and can be given as a second argument. cargo build is enough on a clean machine: protoc is vendored, not a prerequisite.

The file and the collector are both checked before a single line is parsed, in that order, so a typo in a path says so instead of reporting that some collector is unreachable. Failures print one line to stderr and exit non-zero:

small-agent: ship: transport error: tcp connect error: Connection refused (os error 111)

Why it is shaped that way

Telemetry lines name their own fields, so nothing needs a catalogue. Keys sit left of the colon, values right of it, in the same order. One parser then covers every format the machine emits, including formats nobody has seen yet; a table of per-format patterns needs a new entry, and a code change, for each one.

Two conditions decide it is telemetry: equal arity across the colon and every value a number. Together they make a test that prose reliably fails. ***** WARNING: Pressure limit @ 0.1 bar splits cleanly on its colon and is still rejected, because Pressure is not a number.

A severity must stand alone to count. The marker is looked for in the subject, and only when that subject is a single field. A telemetry header is several comma-separated fields; a severity is one word on its own. That keeps position error, tracking error, count: 0.5, 0.2, 10 a measurement and ERROR: 500 an alert, without either being listed anywhere.

The bus owns the task that ships for it. The only way to learn what the collector accepted is finish, which consumes the bus, sends the tail and waits. No order of operations to remember, and no way to wait on a queue still held open by its own sender.

Every batch carries a run id and a sequence number, so a collector can tell one pass over one file from another, and a gap from a duplicate.

Layers

file	hands out lines, one at a time, never collecting them
parsing	turns a line into a telemetry Event, a text Log, or nothing
server	accumulates records and ships a batch at the threshold
errors	the single channel every unrecoverable failure funnels through

The record shapes are defined once, in proto/client.proto, and generated from there. A hand-written twin of every message, plus the conversion between them, is a second definition to keep in step and the first thing to drift.

Measured

x86_64, release build, agent and collector on the same host, synthetic logs of the shape above. parsed is what the agent counted; received is what the collector summed independently.

file	max RSS	wall	parsed	received
51 MB	85.4 MB	0.76 s	758 035	758 035
501 MB	94.7 MB	7.18 s	7 488 598	7 488 598
2001 MB	95.1 MB	28.64 s	29 424 514	29 424 514
401 MB, no newline	8.5 MB	0.08 s	1	1

Forty times the file, and resident memory stays in an 85-95 MB band: the bound is the bus, not the file. Nothing is dropped at any size, and the two counts are reconciled rather than rounded. Throughput is roughly 74 MB/s, 1.09 M lines/s.

The last row had to be measured rather than assumed. A file with no newline in it is a single line, and a reader that grows a buffer to the next newline will hold the whole file: before lines were capped, that same 400 MB file took 823 MB resident and then died on the transport. A truncated log, a failed rotation or a binary file handed over by mistake all look exactly like this.

Limits worth knowing

• gRPC's default message ceiling is 4 MiB, below the flush threshold. Left alone, every batch the agent sends is refused on arrival, and no unit test catches it because none crosses a transport. Both ends are told the real number; see MAX_MESSAGE_BYTES, set to twice the threshold.
• A line past 1 MiB keeps its first megabyte and the rest is discarded. This is the cap that makes the memory bound true; see MAX_LINE. The truncation is silent: nothing marks the record as clipped.
• The connection is plaintext. No TLS, so log contents cross the network in the clear. Fine over loopback or inside a trusted segment, not fine anywhere else. Adding it means a ClientTlsConfig and a certificate the collector presents. Out of scope here, and named rather than left to be discovered.
• A batch in flight is not yet durable. If the agent dies between a flush and its acknowledgement, those records are gone. Surviving that means a write-ahead log on disk, which is a different program.
• NaN and inf are refused as metric values and the line goes to the text path. They parse as f64 perfectly well, and one of them poisons every average computed downstream from that point on.
• Backpressure is evidenced by measurement, not by a unit test. The flat RSS across a forty-fold range of file sizes is what shows a full queue stopping the reader. Asserting it in a test would mean asserting on timing.

Three classification calls are deliberate rather than accidental: errno: 13 is a measurement, not an alert, because adding errno to the severity list would start the catalogue of magic markers this design avoids; position error: 0.5 is read as an alert, erring toward keeping the alert; and a telemetry line with one unparseable value ships as text rather than failing a run on hour three of a large file.

Tests

cargo test

Seventeen, in three files, each pinning a claim this README makes.

bus_test pushes enough records for three crossings of the threshold and asserts that no batch leaves early, none overshoots by more than a single record, the sequence has no gaps, and every record pushed reaches the collector. Plus that a tail below the threshold still ships, which is how every real file ends.

file_test holds the memory bound. Capping a line is easy; picking the file back up at the right place afterwards is where it goes wrong, so the test that matters feeds an over-long line followed by an ordinary one and checks the ordinary one still arrives intact.

parsing_test pins the classification rules against the lines that motivated them, including the ones that get it wrong if severity is checked after shape.

License

MIT.