Usage of context for array index information: is this possible? (Proposal for new deku-attribute))

[goto40]

Thank you for this great library! I use version 0.20.2.

I would like to have access to the index of an array while serializing it. I would like to know the index for each B within the array items in A. I tried with a context to provide the information, but I failed in setting or increasing the index.

I would greatly appreciate help or hints! --> See my next comment on a possible solution, involving a new deku-attribute.

The following example does not work (test does not pass).

#[test]
fn test_use_implicit_index_of_array() {
    #[derive(Debug, Clone, Copy)]
    struct IndexContext {
        idx: usize,
    }
    #[deku_derive(DekuRead, DekuWrite)]
    #[derive(PartialEq, Debug)]
    struct A {
        #[deku(temp, temp_value = "items.len().try_into().unwrap()")]
        n: u8,
        #[deku(count = "n", ctx = "IndexContext { idx: 0 }")] // TODO: idx shall be the array index
        items: Vec<B>,
    }

    #[deku_derive(DekuRead, DekuWrite)]
    #[derive(PartialEq, Debug)]
    #[deku(ctx = "ctx: IndexContext")]
    struct B {
        x: u8,
        y: u8,
        //#[deku(temp, temp_value = "ctx.idx as u8")]
        #[deku(temp, temp_value = "ctx.idx as u8")]
        idx_automatically_filled: u8,
    }
    let test_data = A {
        items: vec![B { x: 8, y: 9 }, B { x: 7, y: 9 }, B { x: 6, y: 9 }],
    };

    let ret_write: Vec<u8> = test_data.try_into().unwrap();
    assert_eq!(vec![3, 8, 9, 0, 7, 9, 1, 6, 9, 2], ret_write);
    //                       ^        ^        ^
    //                      idx=0    idx=1    idx=2
}

[goto40]

I found a acceptable solution for me: I implemented the write function manually. However, I am not happy with the read-part: I used the default_ctx in order be able to omit the context in the read-part because I would like to make full use of the other #deku-features, like until (not shown).

If I would implement a read-function as well (to have the index available), I would have to manually code all those additional features (like until) manually (am I right?).

I think the possibility to make the index available would be great. What do you think? With some hints I could try it on my own (to extend the deku code). I would greatly appreciate some feedback (1) is my suggestion clear (or should I explain it more details?) (2) and if you think it is doable (I think so).

A possible solution could be to generate an index variable with a new deku-attribute, like #deku(loop_index="index") or similar... Then I could write:

...
#[deku(count = "n", loop_index="my_index", ctx = "IndexContext { idx: my_index }")]
        items: Vec<B>,
...

Use cases also include ASTERIX variable length fields, where a bit in the data contained in an array defines how long the array is (see #556). For serialization on has to know it an element is the last in the array or not (which my proposal solves).

The following example works (test passes).

#[test]
fn test_use_implicit_index_of_array() {
    #[derive(Debug, Clone, Copy)]
    struct IndexContext {
        idx: usize,
    }
    #[deku_derive(DekuRead, DekuWrite)]
    #[derive(PartialEq, Debug)]
    struct A {
        #[deku(temp, temp_value = "items.len().try_into().unwrap()")]
        n: u8,
        #[deku(count = "n", writer = "write_items(deku::writer, &self.items)")]
        items: Vec<B>,
    }

    fn write_items<W: std::io::Write + std::io::Seek>(
        writer: &mut Writer<W>,
        items: &[B],
    ) -> Result<(), DekuError> {
        for (idx, item) in items.iter().enumerate() {
            item.to_writer(writer, IndexContext { idx })?;
        }
        Ok(())
    }

    #[deku_derive(DekuRead, DekuWrite)]
    #[derive(PartialEq, Debug)]
    #[deku(ctx = "ctx: IndexContext", ctx_default = "IndexContext{idx: 0}")] // this struct uses a context for serialization. For deserialization it also works with the default context. 
    struct B {
        x: u8,
        y: u8,
        #[deku(temp, temp_value = "ctx.idx as u8")]
        idx_automatically_filled: u8,
    }

    let test_data = A {
        items: vec![B { x: 8, y: 9 }, B { x: 7, y: 9 }, B { x: 6, y: 9 }],
    };

    let ret_write: Vec<u8> = test_data.try_into().unwrap();
    assert_eq!(vec![3, 8, 9, 0, 7, 9, 1, 6, 9, 2], ret_write);
    //                       ^        ^        ^
    //                      idx=0    idx=1    idx=2
}

[wcampbell0x2a]

Sounds useful at first glance!

[goto40]

After playing around with the original idea of passing the index and the array length I stumbled over many problems. And at the end I was limited by the fact that I could not "return" a result from a temp-field in an array to the until-attribute-code of that array...

I made some (non-breaking) changes which enabled me to implement my wishes - this includes also a nice implementation of "Extended Length Data Items" of ASTERIX EUROCONTOL data structures.

Motivation: "Extended Length Data Items" consist of Arrays of data element like {x: u32, y: u32, fx: bool}. The fx is the control element which indicates the end of the array (fx==false). However, user code shall not be aware of the fx field - it should only be used for I/O and be always correct: for writing this means to set fx=falseonly for the last element. For reading it means to read until fx==false. Importantly, for the programmer/user the data element look like {x: u32, y: u32} and fx is completely temporary (temp and temp_value).

My changes are quite large (my feeling) - but non-breaking. I would like to discuss them in the PR #646.

An overview of the changes:

• Changed the requirement that the context (Ctx) must be Copy in some calls.
  It is changed to Clone, this allows to add interior mutability in the context
  (like Rc<Cell<bool>>). This is no breaking change, since Copy implies Clone.
• Added an attribute until_with_ctx: like until, but it also gets the context.
• Added writer_ctx for fields: overrides ctx for the writer case - this allows
  a different context value for reader and writer (the reader gets the original
  ctx).
• Added an attribute read_post_processing: code which is inserted after a field is
  read (including a temp field). You have full context access (and you can modify
  the context there).

And a working demo/unittest (all other tests passed with minor changes)):

#[test]
fn test_use_implicit_index_of_array() {
    #[derive(Debug, Clone)]
    struct IndexContext {
        idx: std::rc::Rc<std::cell::Cell<usize>>,
        n: usize,
        fx: std::rc::Rc<std::cell::Cell<bool>>,
    }
    #[deku_derive(DekuRead, DekuWrite)]
    #[derive(PartialEq, Debug, Clone)]
    struct A {
        #[deku(
            until_with_ctx = "|_:&B,ctx:IndexContext| !ctx.fx.get()",
            ctx = "IndexContext { idx: std::rc::Rc::new(std::cell::Cell::new(0)), n: 0, fx: std::rc::Rc::new(std::cell::Cell::new(false))}",
            writer_ctx = "IndexContext { idx: std::rc::Rc::new(std::cell::Cell::new(0)), n: items.len(), fx: std::rc::Rc::new(std::cell::Cell::new(false)) }"
        )]
        items: Vec<B>,
    }

    #[deku_derive(DekuRead, DekuWrite)]
    #[derive(PartialEq, Debug, Clone)]
    #[deku(
        ctx = "ctx: IndexContext",
        ctx_default = "IndexContext{idx: std::rc::Rc::new(std::cell::Cell::new(0)), n: 0, fx: std::rc::Rc::new(std::cell::Cell::new(false))}"
    )] // this struct uses a context for serialization. For deserialization it also works with the default context.
    struct B {
        x: u8,
        y: u8,
        #[deku(
            temp,
            temp_value = "{let ret = ctx.idx.get() as u8; ctx.idx.set(ctx.idx.get()+1); ret}"
        )]
        idx_automatically_filled: u8,
        #[deku(
            read_post_processing = "ctx.fx.set(*auto_fx!=0);",
            temp,
            temp_value = "if ctx.idx.get() < ctx.n {1} else {0}"
        )]
        auto_fx: u8,
    }

    let test_data = A {
        items: vec![B { x: 8, y: 9 }, B { x: 7, y: 9 }, B { x: 6, y: 9 }],
    };

    let ret_write: Vec<u8> = test_data.clone().try_into().unwrap();
    assert_eq!(vec![8, 9, 0, 1, 7, 9, 1, 1, 6, 9, 2, 0], ret_write);
    //                    ^  ^        ^  ^        ^  ^
    //                    |  fx=1     |  fx=1     |  fx=0 (last)
    //                   idx=0      idx=1       idx=2

    // read the data back
    let check_data = A::from_bytes((&ret_write, 0)).unwrap().1;
    assert_eq!(check_data, test_data);

    // check with fx=0 after the second element:
    let check_data = A::from_bytes((&[8, 9, 0, 1, 7, 9, 1, 0], 0)).unwrap().1;
    assert_eq!(check_data.items.len(), 2);
}

I would be happy to start a discussion or a suggestion how to proceed... See PR #646.