Add fragmented encoding + redesign fragmented decoding

MIGRATION_GUIDE.md

@@ -13,6 +13,17 @@
 * [Deprecation] In the function `zcbor_process_backup()` and the new `zcbor_process_backup_num()`, it is deprecated to use the argument `max_elem_count` with values other than `ZCBOR_MAX_ELEM_COUNT`.
   Using `ZCBOR_MAX_ELEM_COUNT` means the internal check on the `max_elem_count` has no effect, and the argument (and check) may be removed altogether in later releases.
 
+* The fragmented payload API has been completely redesigned to accomodate adding the encoding counterpart.
+  The docs have been updated and there's a new section in the README to explain the functionality.
+
+  * You must now define ZCBOR_FRAGMENTS to access the API
+  * `zcbor_*str_decode_fragment()` has been renamed to `zcbor_*str_fragments_start_decode()`
+  * After calling `zcbor_*str_fragments_start_decode()`, you must now retrieve the first fragment manually with `zcbor_str_fragment_decode()`, instead of via an argument.
+  * `zcbor_next_fragment()` and `zcbor_bstr_next_fragment()` have merged and is now called `zcbor_str_fragment_decode()`.
+    It does not take a `prev_fragment` argument, instead, this state is kept internally in the state struct.
+  * `zcbor_bstr_start_decode_fragment()` has been renamed to `zcbor_cbor_bstr_fragments_start_decode()` and does not return a fragment.
+    To retrieve fragments when decoding a CBOR-encoded bstr, use `zcbor_str_fragment_decode()`
+
 * Code generation:
 
   * Integers whose values are known to be within 8 or 16 bytes now use the corresponding integer types (`uint8_t`/`int8_t`/`uint16_t`/`int16_t`) instead of larger types.

README.md

@@ -88,6 +88,31 @@ ZCBOR_STATE_D(decode_state, n, payload, payload_len, elem_count, n_flags);
 ZCBOR_STATE_E(encode_state, n, payload, payload_len, 0);
 ```
 
+Fragmented payloads
+-------------------
+
+zcbor can encode and decode payloads in sections, i.e. the payload can be split into separate buffers/arrays.
+This can be useful e.g. if you send or receive your payload in multiple packets.
+When the current payload section is done, call `zcbor_update_state()` to introduce the next section.
+Note that zcbor does not allow section boundaries to split a zcbor header/value pair.
+This means that the following elements cannot be split between sections:
+
+- Numbers and simple values (integers, floats, bools, undefined, nil)
+- Tags
+- Headers of lists, maps, tstrs, and bstrs
+
+If your payload is split in an unsupported way, you can get around it by making a small section out of the remaining bytes of one section spliced with the start of the next.
+Another option is to leave a little room at the start of each section buffer, and copy the remaining end of one section into the start of the next buffer.
+8 bytes should be enough for this.
+
+Lists and maps can span multiple sections, as long as the individual elements are not split as to break the above rule.
+
+String payloads can be split across multiple payload sections, if `ZCBOR_FRAGMENTS` is enabled, and the `*str_fragments_*()` APIs are used. Note that in the zcbor docs, the term "string fragment" is used for fragmented strings, while the term "payload section" is used for fragmented CBOR payloads, as passed to `zcbor_update_state()`. These do not always line up perfectly, particularly at the start and end of fragmented strings.
+
+CBOR-encoded bstrs can be nested, and there can also be a non-CBOR-encoded innermost string.
+The current innermost string (CBOR-encoded or otherwise) is called the "current string".
+`zcbor_update_state()` modifies all backups so that outer nested CBOR-encoded strings have updated information about the new section.
+
 Configuration
 -------------
 
@@ -103,6 +128,7 @@ Name                      | Description
 `ZCBOR_STOP_ON_ERROR`     | Enable the `stop_on_error` functionality. Note that it also has to be enabled in the state variable (`state->constant_state->stop_on_error`). This makes all zcbor functions abort their execution if called when an error has already happened.
 `ZCBOR_BIG_ENDIAN`        | All decoded values are returned as big-endian. The default is little-endian.
 `ZCBOR_MAP_SMART_SEARCH`  | Applies to decoding of unordered maps. When enabled, a flag is kept for each element in an array, ensuring it is not processed twice. If disabled, a count is kept for map as a whole. Enabling increases code size and memory usage, and requires the state variable to possess the memory necessary for the flags.
+`ZCBOR_FRAGMENTS`         | Enable functions for decoding and encoding byte and text strings in fragments.
 
 Canonical encoding
 ------------------

include/zcbor_common.h

@@ -107,6 +107,24 @@ union {
 	bool payload_moved; /**< Is set to true while the state is stored as a backup
 	                         if @ref zcbor_update_state is called, since that function
 	                         updates the payload_end of all backed-up states. */
+	bool inside_cbor_bstr; /**< True if we are currently inside a CBOR-encoded bstr,
+	                            i.e. that as been started with zcbor_bstr_start_*(), or
+	                            `zcbor_cbor_bstr_fragments_start_*()`. */
+#ifdef ZCBOR_FRAGMENTS
+	bool inside_frag_str; /**< True if we are currently inside a fragmented (non-CBOR-encoded)
+	                           string. This is mutually exclusive with `inside_cbor_bstr`,
+	                           i.e. not set when using `zcbor_cbor_bstr_fragments_start_*()` */
+	ptrdiff_t frag_offset; /**< The offset in the current string at which this payload section starts.
+	                            Used for non-CBOR-encoded strings. Can be negative if the current string
+	                            started in this payload section. */
+	size_t str_total_len; /**< The total length of the string this fragment is a part of.
+	                           Used for non-CBOR-encoded strings. */
+	ptrdiff_t frag_offset_cbor; /**< The offset in the current string at which this payload section starts.
+	                                 Used for CBOR-encoded strings. Can be negative if the current string
+	                                 started in this payload section. */
+	size_t str_total_len_cbor; /**< The total length of the string this fragment is a part of.
+	                                Used for CBOR-encoded strings. */
+#endif
 
 /* This is the "decode state", the part of zcbor_state_t that is only used by zcbor_decode.c. */
 struct {
@@ -131,7 +149,7 @@ struct {
 #endif
 	size_t map_start_backup_num; /** The index of the backup made at the start of the current map.
 	                                 This is used to move to the start of the map when searching
-					 unordered maps. */
+	                                 unordered maps. */
 	size_t map_elem_count; /**< Number of elements in the current unordered map.
 	                            This also serves as the number of bits (not bytes)
 	                            in the map_search_elem_state array (when applicable). */
@@ -155,6 +173,10 @@ struct zcbor_state_constant {
 #ifdef ZCBOR_MAP_SMART_SEARCH
 	uint8_t *map_search_elem_state_end; /**< The end of the @ref map_search_elem_state buffer. */
 #endif
+	const uint8_t *curr_payload_section; /**< The currently encoded/decoded payload section.
+	                                          I.e. the payload pointer this state was created with,
+	                                          or the payload pointer of the most recent call to
+	                                          zcbor_update_state. */
 };
 
 #ifdef ZCBOR_CANONICAL
@@ -297,6 +319,10 @@ do { \
 #define ZCBOR_ERR_CONSTANT_STATE_MISSING 22
 #define ZCBOR_ERR_BAD_ARG 23
 #define ZCBOR_ERR_NO_FLAG_MEM 24
+#define ZCBOR_ERR_BAD_STATE 25 ///! The zcbor_state_t struct is in a bad state.
+#define ZCBOR_ERR_TOO_LARGE_FOR_STRING 26 ///! Trying to start a nested string that is too large to fit in the container string.
+#define ZCBOR_ERR_NOT_IN_FRAGMENT 27 ///! The action requires being inside a fragmented string, but we are currently not inside one.
+#define ZCBOR_ERR_INSIDE_STRING 28 ///! Currently encoding/decoding a non-CBOR-encoded string, so cannot use most zcbor encoding/decoding functions
 #define ZCBOR_ERR_UNKNOWN 31
 
 /** The largest possible elem_count. */
@@ -433,17 +459,23 @@ static inline void zcbor_error(zcbor_state_t *state, int err)
 	}
 }
 
-/** Whether the current payload is exhausted. */
+/** Whether the current payload (section) is exhausted. */
 static inline bool zcbor_payload_at_end(const zcbor_state_t *state)
 {
 	return (state->payload == state->payload_end);
 }
 
-/** Update the current payload pointer (and payload_end).
+/** Introduce a new payload section.
  *
+ *  Updates the current payload pointer (and payload_end and frag_offset(_cbor)).
  *  For use when the payload is divided into multiple chunks.
  *
- *  This function also updates all backups to the new payload_end.
+ *  This function also updates all backups to the new payload_end,
+ *  and also updates the frag_offset/frag_offset_cbor of all backups.
+ *
+ *  Note that if this is called before the current payload is exhausted, the
+ *  remaining payload will be abandoned.
+ *
  *  This sets a flag so that @ref zcbor_process_backup fails if a backup is
  *  processed with the flag @ref ZCBOR_FLAG_RESTORE, but without the flag
  *  @ref ZCBOR_FLAG_KEEP_PAYLOAD since this would cause an invalid state.
@@ -456,6 +488,17 @@ static inline bool zcbor_payload_at_end(const zcbor_state_t *state)
 void zcbor_update_state(zcbor_state_t *state,
 		const uint8_t *payload, size_t payload_len);
 
+/** Get the the offset into the current string to which the current payload
+ *  pointer (state->payload) points. */
+bool zcbor_current_string_offset(zcbor_state_t *state, size_t *offset);
+
+/** Get the remaining number of bytes in the current string, calculated from
+ *  the current payload pointer (state->payload). */
+bool zcbor_current_string_remainder(zcbor_state_t *state, size_t *remainder);
+
+/** Can be used on any fragment to tell if it is the final fragment of its string. */
+bool zcbor_is_last_fragment(const struct zcbor_string_fragment *fragment);
+
 /** Check that the provided fragments are complete and in the right order.
  *
  *  If the total length is not known, the total_len can have the value

include/zcbor_decode.h

@@ -398,7 +398,9 @@ bool zcbor_tstr_expect_term(zcbor_state_t *state, char const *str, size_t maxlen
 /** Decode and consume a bstr header.
  *
  * The rest of the string can be decoded as CBOR.
- * A state backup is created to keep track of the element count.
+ * A state backup is created to keep track of the element count and original payload_end.
+ * payload_end is set to the end of the string, so when the payload is exhausted,
+ * the string is considered fully decoded.
  * Call @ref zcbor_bstr_end_decode when done decoding the contents of the bstr.
  *
  * @param[inout] state   The current state of the decoding.
@@ -420,49 +422,65 @@ bool zcbor_bstr_start_decode(zcbor_state_t *state, struct zcbor_string *result);
 bool zcbor_bstr_end_decode(zcbor_state_t *state);
 
 
-/** Supplementary string (bstr/tstr) decoding functions for fragmented payloads: */
+#ifdef ZCBOR_FRAGMENTS
 
-/** Start decoding a bstr/tstr, even if the payload contains only part of it.
+/** Start decoding a fragmented string.
  *
- * This must be followed by a call to @ref zcbor_update_state, which can be
- * followed by a call to @ref zcbor_next_fragment. Do not call this function
- * again on subsequent fragments of the same string.
+ * I.e. a string spread over multiple non-contiguous payload sections.
  *
- * This consumes the remaining payload as long as it belongs to the string.
- */
-bool zcbor_bstr_decode_fragment(zcbor_state_t *state, struct zcbor_string_fragment *result);
-bool zcbor_tstr_decode_fragment(zcbor_state_t *state, struct zcbor_string_fragment *result);
-
-/** Extract the next fragment of a string.
+ * NOTE: The fragmented string API is experimental.
  *
- * Use this function to extract all but the first fragment.
+ * After calling this, you can retrieve a fragment with @ref zcbor_str_fragment_decode,
+ * then update the payload with @ref zcbor_update_state.
+ * Repeat until the string is fully decoded, then call @ref zcbor_bstr_fragments_end_decode.
  */
-void zcbor_next_fragment(zcbor_state_t *state,
-	struct zcbor_string_fragment *prev_fragment,
-	struct zcbor_string_fragment *result);
+bool zcbor_bstr_fragments_start_decode(zcbor_state_t *state);
+bool zcbor_tstr_fragments_start_decode(zcbor_state_t *state);
 
-/** Decode and consume a bstr header, assuming the payload does not contain the whole bstr.
+/** Start decoding a fragmented CBOR-encoded bytestring.
  *
- * The rest of the string can be decoded as CBOR.
- * A state backup is created to keep track of the element count.
- * Call @ref zcbor_update_state followed by @ref zcbor_bstr_next_fragment when
- * the current payload has been exhausted.
- * Call @ref zcbor_bstr_end_decode when done decoding the contents of the bstr.
+ * I.e. a string spread over multiple non-contiguous payload sections.
+ *
+ * NOTE: The fragmented string API is experimental.
+ *
+ * This is an alternative to zcbor_*str_fragments_start_decode() to be used if the payload
+ * contains CBOR data that will be decoded directly with other zcbor_*() functions.
+ *
+ * A state backup is created to keep track of the element count and original payload_end.
+ * After calling this, you can decode elements using other zcbor functions,
+ * then update the payload with @ref zcbor_update_state.
+ * Repeat until the string is fully decoded, then call @ref zcbor_bstr_fragments_end_decode.
+ * When the current payload section contains the end of the string,
+ * payload_end is set to the end of the string, so there is no risk of decoding past the end.
+ * Then, when @ref zcbor_bstr_fragments_end_decode has been called, payload_end is restored to the
+ * end of the payload section.
+ * (This is analogous to how @ref zcbor_bstr_start_decode works)
  */
-bool zcbor_bstr_start_decode_fragment(zcbor_state_t *state,
-	struct zcbor_string_fragment *result);
+bool zcbor_cbor_bstr_fragments_start_decode(zcbor_state_t *state);
 
-/** Start decoding the next fragment of a string.
+/** Retrieve a string fragment.
+ *
+ * NOTE: The fragmented string API is experimental.
+ *
+ * For non-cbor-encoded strings (started with @ref zcbor_bstr_fragments_start_decode
+ * or @ref zcbor_tstr_fragments_start_decode): Consumes bytes from the payload until
+ * either the end of the payload or the end of the string.
  *
- * Use this function to extract all but the first fragment of a CBOR-encoded
- * bstr.
+ * For cbor-encoded strings (started with @ref zcbor_cbor_bstr_fragments_start_decode):
+ * No bytes are consumed from the payload. Instead, this function reports the part of
+ * the current payload section that has already been decoded via other zcbor functions.
+ * Any such call to this function will have no memory of previous calls to this function,
+ * so the returned fragment always starts at the same point until the payload is updated.
+ *
+ * @param[inout] state     The current state of the decoding.
+ * @param[out]   fragment  The resulting string fragment.
  */
-void zcbor_bstr_next_fragment(zcbor_state_t *state,
-	struct zcbor_string_fragment *prev_fragment,
-	struct zcbor_string_fragment *result);
+bool zcbor_str_fragment_decode(zcbor_state_t *state, struct zcbor_string_fragment *fragment);
+
+/** Finish decoding a fragmented string. */
+bool zcbor_str_fragments_end_decode(zcbor_state_t *state);
 
-/** Can be used on any fragment to tell if it is the final fragment of the string. */
-bool zcbor_is_last_fragment(const struct zcbor_string_fragment *fragment);
+#endif /* ZCBOR_FRAGMENTS */
 
 #ifdef __cplusplus
 }

include/zcbor_encode.h

@@ -243,6 +243,53 @@ bool zcbor_bstr_start_encode(zcbor_state_t *state);
  */
 bool zcbor_bstr_end_encode(zcbor_state_t *state, struct zcbor_string *result);
 
+
+#ifdef ZCBOR_FRAGMENTS
+
+/** Start encoding a fragmented string. I.e. a string spread over non-consecutive payload sections.
+ *
+ * NOTE: The fragmented string API is experimental.
+ *
+ * After calling this, you can write a fragment with @ref zcbor_str_fragment_encode,
+ * then update the payload with @ref zcbor_update_state.
+ * Repeat until the string is fully decoded, then call @ref zcbor_bstr_fragments_end_encode.
+ */
+bool zcbor_bstr_fragments_start_encode(zcbor_state_t *state, size_t total_len);
+bool zcbor_tstr_fragments_start_encode(zcbor_state_t *state, size_t total_len);
+
+/** Start encoding a fragmented CBOR-encoded bytestring.
+ *
+ * I.e. a string spread over non-consecutive payload sections.
+ *
+ * NOTE: The fragmented string API is experimental.
+ *
+ * This is an alternative to zcbor_*str_fragments_start_encode() to be used if the payload
+ * contains CBOR data that will be encoded directly with other zcbor_*() functions.
+ *
+ * A state backup is created to keep track of the element count and original payload_end.
+ * After calling this, you can encode elements using other zcbor functions,
+ * then update the payload with @ref zcbor_update_state.
+ * Repeat until the string is fully decoded, then call @ref zcbor_bstr_fragments_end_encode.
+ * When the current payload section contains the end of the string,
+ * payload_end is set to the end of the string, so there is no risk of encoding past the end.
+ */
+bool zcbor_cbor_bstr_fragments_start_encode(zcbor_state_t *state, size_t total_len);
+
+/** Encode a string fragment.
+ *
+ * NOTE: The fragmented string API is experimental.
+ *
+ * Write bytes to the payload until either the end of the payload or the end of the fragment.
+ * The number of bytes written is returned in @p enc_len.
+ * Do not use this function with @ref zcbor_cbor_bstr_fragments_start_encode.
+ */
+bool zcbor_str_fragment_encode(zcbor_state_t *state, struct zcbor_string *fragment, size_t *enc_len);
+
+/** Finish encoding a fragmented string. */
+bool zcbor_str_fragments_end_encode(zcbor_state_t *state);
+
+#endif /* ZCBOR_FRAGMENTS */
+
 #ifdef __cplusplus
 }
 #endif