build: A high-level mechanism for embedding raw data from binary files

[hIAhmad]

Adding CMake functions generate_binary_embed_inc_* to introduce a portable
high-level build mechanism for embedding raw binary data.

These functions enable including raw data from one or more binary files,
along with desired symbol names for corresponding data being embedded.
Controlling aspects such as address alignment, linker section, inclusion
as read-only (default) or as writable, data compression and including only
a subset of data from the specified binary files, is supported through
named optional parameters.

These functions provide a higher-level embedding mechanism compared to
generate_inc_* functions, enabling advanced use cases like:

• Embedding of data from more than one files, where the list and number
  of files being included is determined at build time, e.g. driven by
  some device-tree fields or config options. generate_inc_* functions
  cannot handle such use-cases without a fair amount of additional custom
  support code added to corresponding CMake build files.

• generate_inc_* functions embed binary data by first converting it
  into C-source compilable form; which is fine for small binary files
  but for larger files this can increase build time and resources usage
  significantly. Whereas these new functions can transparently make use
  of GNU-style "incbin" directive, when available (and usable), for much
  faster inclusion of binary files, automatically falling back to using
  generate_inc_* functions internally, for other cases.

Testing:

• Tested on following platforms:

  native_sim
  qemu_x86
  qemu_cortex_a9
  qemu_cortex_a53
  qemu_cortex_m3
  qemu_cortex_r5
  qemu_riscv32
  qemu_riscv64
  qemu_xtensa/dc233c
  qemu_arc/qemu_arc_hs
  

• Tested both with and without Zephyr CMake linker generator feature (on qemu_cortex_m3).

Related PRs:

• This new feature is inspired by discussion under related drivers: ramdisk: Add support for embedded disk images as backing store of RAM disks #82603 (which will be rebased to use this generic feature, once approved).
• Depends on toolchain: Add assembly-side counterpart of __aligned #85758 and scripts: build: Add optimizations in file2hex.py to gzip path as well #85900 to be merged first.

Note:
Changes to include/zephyr/toolchain/common.h and scripts/build/file2hex.py are not part of this PR - they are coming from the separate PRs that this PR depends on. I've added them here to get early feedback on rest of the feature changeset. These changes will be removed from this PR (through rebase) once the respective PRs get approved and merged to main.

[nordicjm]

fix spacing to look as it looks above (remove empty newlines)

[nordicjm]

not a toolchain maintainer but this looks like the wrong thing and looks to be doing something for a specific build tool by looking at the arch, when instead it should be looking at the build tool @LoveKarlsson can you check?

[hIAhmad]

These changes are from #85758 that needs to get merged first and are being discussed and reviewed there (see: #85758 (comment)). I have added these here to get early feedback on the rest of the changeset. It will no longer be part of this PR, after that PR gets merged.

[LoveKarlsson]

I think for future proofing toolchains not based on GCC it should be a macro. IAR for example does not use .align

[nordicjm]

2 space indent for cmake

[hIAhmad]

Updated.

[nordicjm]

gzip != zip, and don't think that has a place here, if you need a zip then the zip file should be present and provided to the function (or compressed manually with a user build step then included)

[hIAhmad]

Yes, ZIP as the name of the parameter is a bit arguable. While the word zip can sometimes be used as a generic word for compression it is also a name for a specific compression method. COMPRESS can be an alternative name here, that does not suffer from this ambiguity.

Now, regarding your other point - whether this parameter should even be provided in first place. It is true that a file can be compressed in a separate build step, prior to passing it on to generate_binary_embed_inc_file but its lower-level counterpart generate_inc_file already provides this capability so I think it is better to preserve the existing capability of that function in this higher-level wrapper (and it is certainly more convenient than having to do it in a separate build step).

[hIAhmad]

I have renamed the ZIP parameter to COMPRESS.

[nordicjm]

this should be a flag and should be CONST not READONLY

[hIAhmad]

this should be a flag and should be CONST not READONLY

I am a bit divided on the name. I had thought about naming it CONST but then felt that the parameter name should better reflect the eventual logical effect that normal/intended use of the parameter should achieve, rather than an implementation detail so chose READONLY instead. Let's wait for some more feedback on the naming and then consider renaming it accordingly.

I'll make it a flag right away, though. (see below: #86204 (comment)).

[hIAhmad]

@nordicjm There is a downside to making this parameter a flag. By default, you'd typically include/embed binary data as read-only, and not as writable. So if we make this a flag, it will always have to be specified for the common and typical case. By making it take a value, you need to specify it only if you want to make the included data writable (which is relatively a rare case, currently, #82603 would be likely the only such practical use in the source tree).

I think it is better to have it take a value.