Support streaming for big messages

[robin-thoni]

Hi,

Would you consider adding support for streaming messages by chunks, when the message total size goes over the rx buffer size?

My use case here would be to do a firmware update over CAN for ESPHome. I don't need/want/can't store the full firmware in RAM, but instead, it has to be written directly to the chip flash, chunk by chunk (preferably, bigger chunks than 7 bytes).

Would that be an option?

Thanks

[SimonCahill]

Unless I misunderstand what you mean, ISOTP already supports this. The protocol supports messages up to 4095 bytes, which are sent in chunks. The protocol handles FF (first frame), CF (consecutive frame), FC (flow control frame) and SF (single frame) messages automatically.

If you can't store the entire file in memory, you could in theory map the file into memory and stream it that way.

A really simple (yet untested) example would be:

int main(int argc, char** argv) {
    if (argc != 3) {
        fprintf(stderr, "Usage: %s <input_file> <device>\n", argv[0]);
        return EXIT_FAILURE;
    }

    const char* filename = argv[1];
    const char* device   = argv[2];

    int file_fd = open(filename, O_RDONLY);
    if (file_fd < 0) {
        perror("open input file");
        return EXIT_FAILURE;
    }

    struct stat st;
    if (fstat(file_fd, &st) < 0) {
        perror("fstat");
        close(file_fd);
        return EXIT_FAILURE;
    }

    if (st.st_size == 0) {
        fprintf(stderr, "File is empty\n");
        close(file_fd);
        return EXIT_FAILURE;
    }

    void* map = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, file_fd, 0);
    if (map == MAP_FAILED) {
        perror("mmap");
        close(file_fd);
        return EXIT_FAILURE;
    }

    int dev_fd = open(device, O_WRONLY);
    if (dev_fd < 0) {
        perror("open device");
        munmap(map, st.st_size);
        close(file_fd);
        return EXIT_FAILURE;
    }

    ssize_t written = 0;
    // init isotp and let it handle the rest; only pass the memory pointer and the size.

    munmap(map, st.st_size);
    close(dev_fd);
    close(file_fd);

    return (written == st.st_size) ? EXIT_SUCCESS : EXIT_FAILURE;
}

When updating ECUs, we also compressed the BLOBs and updated the parser at the ECU side to parse incoming chunks and added message indicating when it was okay to continue streaming in the 4095 byte chunk size.

[robin-thoni]

Sorry, I forgot to mention I'm talking about the 2016 revision, which, AFAIU, is implemented in your lib, right? There are several references to it in the code.

In that case, since it can carry messages up to 4GB, it quickly becomes a problem for limited uC. Or again, am I missing something?

[SimonCahill]

No, you're entirely correct. It was a feature I had completely forgotten about!

In this case, mapping the file into memory would still be a viable option. You do not use more memory, instead the physical file is just mapped into memory.

[robin-thoni]

mmap() creates a few issues in my case:

• AFAIK, it does not exist as-is in esp-idf
• There's esp_partition_mmap() which fits this exact use case, but requires special handling for this specific case
• Only compatible w/ esp-idf. For example, I'm not sure there's an equivalent for RP2040

One WAR for now would be, as you said to run a TCP-like protocol on top of ISO-TP, w/ a minimal MTU. It'd work, but it's a shame to implement that when ISO-TP already has some control flow.

From an API PoV, I would see it like having:

• Another callback for streaming parts of the message, close or same as isotp_rx_done_cb
• A polling function isotp_receive_streaming(), close or same as isotp_receive()

OR

• A new bool is_complete argument to isotp_rx_done_cb()
• A new bool* is_complete argument to isotp_receive()

Each time an incoming FF indicates the message would not fit in the receive buffer, it would use that alternate streaming method. Once the receive buffer is full, it would return the partial data (through polling or callback), then clear it and start over.

Everything would be guarded by a build flag, so it won't impact existing usages (binary size, function signature changes, etc).

I understand that might be not a trivial request. Still, do you think it'd fit in here?

[SimonCahill]

I'm not familiar with ESP software development, so sadly I can't provide any insights on that front.

However, I am not the idea of expanding the functionality of the library, as long as the core library isn't touched.

One possible way would be to add a subproject to this, which provides the streaming functionality you're looking for. Could be named isotp-c-streamer and be enabled, as you mentioned, with a CMake option 🤔

I'm considering whether it's worth adding a new state structure for streaming, which expands the existing ISOTP state.

Although an interesting idea, I don't know when I'd be up to implementing it. As you've probably seen from my open PR, I don't have much time to invest in my personal projects unless I'm actively using them in a project, as I'm self-employed and have employees to pay/feed.

I'll see what I can do!