bpsh: add SSH-like remote shell over Bundle Protocol - #100
Conversation
Add bpsh (client) and bpshd (daemon): a remote shell that executes commands on a remote node over BPv7, in the spirit of ssh but built for store-and-forward DTN links. Implements nasa-jpl#83. Protocol (bpsh_proto): - CBOR-framed messages carried in bundle payloads: INIT / INIT_ACK, REQ, STDIN_CHUNK / STDIN_EOF, STDOUT / STDERR, CWD, EXIT, ERROR, EXIT_SESSION. - Per-session id and monotonic sequence numbers; the client delivers frames in order and treats ERROR as out-of-band. - Shared encode/decode, send, receive and attach helpers. Daemon (bpshd, bpshd_session): - One persistent /bin/sh per client, keyed by source EID, so cd, environment and shell functions persist across commands. - Split into a per-session object (the shell, its pipes and command execution) and a manager (session table, deferred-bundle queue, receive/dispatch loop). - The exit status and the command boundary travel out-of-band on a control fd: each command runs as "{ cmd ; } <&3 4>&-; echo $? >&4" while stdout/stderr stream verbatim, and the daemon finishes the command when it reads the rc. No in-band sentinels, no holdback. - stdin forwarding to the remote command; ReqAttendant-based flow control so large output (e.g. du over a big tree) blocks for ZCO space instead of being dropped; a per-command wall-clock timeout (BPSHD_CMD_TIMEOUT, default 300s) and output cap with process-group teardown; and recovery: a fresh INIT aborts a stalled command and starts a new session. Client (bpsh): - Interactive REPL and one-shot mode (-c, exits with the remote rc and forwards non-TTY stdin). - Raw-mode line editing (left/right/home/end/delete), command history on the up/down arrows plus a "history" builtin, and a prompt that shows the remote shell's working directory. - "exit" closes the session and leaves the REPL cleanly; SIGSEGV / SIGBUS are reported as a clean "lost the local ION node" exit rather than a core dump. Adds a loopback regression test (tests/bpsh-loopback) and Makefile wiring.
|
see also demo video: https://youtu.be/RYLdBJMoW4g |
|
the demo video missed some things, here is what i forgot to mention:
open / TODO:
things compared to ssh explicitly missing
|
This adds the missing documentation for new bpsh/bpshd binaries. - bpv7/doc/pod1/bpsh.pod: both invocation forms (-h/-l/-c and the positional shorthand), REPL vs. one-shot -c mode, stdin forwarding, exit-status semantics, NO_COLOR/TERM, and diagnostics. - bpv7/doc/pod1/bpshd.pod: the listen EID, per-client persistent shells, flow-controlled output, the output cap and command timeout (BPSHD_CMD_TIMEOUT, default 300s), and diagnostics.
|
pod file documentation now added |
ltpdeliv guarded the heap-buffer delivery block on sessionBuf.heapBufferObj alone, which is always non-zero after openImportSession allocates the buffer. heapBufferBytes is only updated when a red-part segment lands in the [0, heapBufferSize) range (libltpP.c gates the update on bytesForHeap > 0). If no such segment is non-redundantly inserted before the block is queued for delivery, heapBufferBytes stays 0 and the existing sdr_insert(sdr, buffer, sessionBuf.heapBufferBytes) call hits XNCHKZERO(!(nbytes == 0 || nbytes > LARGE_BLK_LIMIT)) inside _sdrmalloc. The assertion cancels the transaction after non- reversible modifications, so the lock-owner-died recovery cannot roll back, and every subsequent sdr_begin_xn in every daemon asserts -- the whole node wedges until the test harness shuts it down. Guard the block on (heapBufferObj && heapBufferBytes > 0). When there are no heap bytes to deliver, skip the read / insert / zco_append_extent triplet entirely; the file-buffer branch below still runs and delivers the actual data. The heap buffer itself is still freed at session close in closeImportSession, so this does not leak. Observed in atomic-tiers run nasa-jpl#100 (__sync, ol9, adjacent-contacts) as the cascade tracked in #1013. The exact precondition that leaves heapBufferBytes at 0 on this test is still under investigation -- nominal segment ordering for the test's bundle/ segment size puts segment 1 in the heap range and populates heapBufferBytes=560. Possible triggers include a torn read or clobber of the field under __sync memory ordering, or an insertDataSegment redundancy edge case rejecting the first arrival of segment 1. This fix addresses the symptom unconditionally; diagnostic logging of the actual nbytes value at the call site can be added as a follow-up to nail down the trigger.
The Check 8 grep failed on arc-runner-set-u22 __sync (atomic-tiers run nasa-jpl#100, batch 6) because six ion.log lines slipped past the exclude filter: - "Can't find (DTPC|BSSP|CFDP) database" -- benign shutdown chatter when those subsystems were never started - Three security-vdb init messages that appear on some ARC pods when the keystore is not pre-populated Same test passed on the other five batch-6 pods in the same run, so the failure was always a test-side gap, not an ION regression. Consolidate the two parallel grep -v chains into one LTP_ERR_EXCLUDE variable for the new entries.
The stdin forwarder thread blocked indefinitely in read(STDIN_FILENO), so when the remote command exited bpsh kept running until its own stdin was closed. Poll stdin with a short (200 ms) timeout so the forwarder periodically re-checks `running`, and have the main thread clear `running` before joining the forwarder once the remote command has exited.
…ases Exercise bpsh as a full-duplex transport for directory synchronization. Case 6 runs real rsync with bpsh as its remote shell (-e), proving bpsh can carry rsync's interactive protocol end-to-end, and reports the bundle cost. It is a feasibility/correctness check only: rsync is O(files) chatty (~36+ round-trips here even with -W --no-i-r), which is impractical over a real delay-tolerant link. Case 7 reimplements the same sync in exactly two request/response exchanges regardless of file count: roundtrip 1 pulls a per-file SHA-256 manifest from the remote, roundtrip 2 ships the differing/missing files as one gzipped tar stream that the server extracts in bulk. This is the DTN-suitable approach. Both cases assert the remote dir becomes identical and that the unchanged file is skipped. Add endpoints ipn:1.513-515 to host1.rc for the new cases.
|
@iondev33 @scburleigh In general: bpsh lets ordinary app code drive cross-DTN operations with no DTN-specific logic and no task-specific code on the remote — it needs only bpshd running, then allows to run any shell accessable programs over bpsh's full-duplex channel. |
|
There was some discussion about this and the concept of a remote shell in general a while ago that I forgot to share here. We think this is really great, but since it isn't a core issue having to do with potential mission deployment needs (yet at least), we don't have the resources to properly maintain it, so we'll have to pass for now. In the long run, apart from simple utilities used in tests, it'll be more maintainable if proper applications exist outside of ION's codebase (though this is made difficult by the messy public API that is due for a cleanup). Could you extract bpsh and make it build as its own project? It's very good to see applications like this though, and we should have a list of references of community utilities to make people aware of them (perhaps in the wiki?). Applications like your bpsh and the simulator you've used in your videos can be the first ones listed. |
|
sure @250MHz: i will make it a standalone project and let you know the reference here in the ticket once done. |
|
Hi @zebastian and all, Wanted to share that we independently implemented a very similar concept, which proved invaluable during our DTN flight tests with ION on OPS-SAT (March 2023): https://nebula.esa.int/content/smallsat-dtn (see exec. summary & presentation). We gave it a similar name: bpshell. It also proved invaluable to our joint DTN flight demo with ESA's DTN team on OPS-SAT (January 2024), where we provided the onboard DTN implementation and ESA provided the ground implementation (dtna) (see section III OPS-SAT DTN Demonstrator): https://www.techrxiv.org/doi/full/10.36227/techrxiv.170831453.30339079/v1 In the above flight campaigns, this type of tool allowed us to leverage the full Linux ecosystem available to experimenters on the OPS-SAT SEPP OBC over the Bundle Protocol, without reinventing the wheel for remote payload operations, and it provided a natural foundation for automating onboard ION DTN operations. In my opinion it would be beneficial to ION DTN advancement if this functionality finds its way either into ION upstream or as a well-maintained standalone project, especially as Linux-based OBCs are found more often in new missions. Best regards, |
|
i now moved the bpsh, together with the mqtt CLA: see subproject: build tooling / testing is close to ION-DTN, given a linux machine with ION-DTN installed this should be installable with: @aggelis thank you for your insights! Glad to see such a feature can be put to use in "the wild". Please let me know what you think. |
|
@zebastian, i will try to set up and test bpsh locally and share my feedback. |
|
@aggelis great, let me know if you into any trouble. |
FYI there's this wiki page now which you should be able to edit https://github.com/nasa-jpl/ION-DTN/wiki/Ecosystem |
|
@250MHz i changed the wiki page and extended it to the other present APPs and CLAs in ION-DTN-CONTRIB. |
3 participants
Add bpsh (client) and bpshd (daemon): a remote shell that executes commands on a remote node over BPv7, in the spirit of ssh but built for store-and-forward DTN links. Implements #83.
Protocol (bpsh_proto):
Daemon (bpshd, bpshd_session):
Client (bpsh):
Adds a loopback regression test (tests/bpsh-loopback) and Makefile wiring.