feat(flash): introduce override-partition-conf.sh

[gagath]

In the flash recipe is a section that involves replacing some lines in the partition.conf file and uses stdin/stdout, which makes it a good candidate for extraction in order to reduce the complexity of the flash recipe.

Since the for loop in the recipe makes the code duplicated into a huge script, this simplifies debugging by having the possibility to see what variables are passed to the script each time instead of trying to debug with a big line number that is almost impossible to interpret because of template for loops.

This also allows to manually test the overrides against an arbitrary partition.conf file.

The behavior around cdt_download and cdt_filename is preserved, using a shell condition instead of a template condition. A new message is printed in the case of an existing cdt partition but missing cdt_filename variable.

[lool]

Thanks for starting that: that's exactly the direction I think we need to take the flash recipe towards!

From the top of my head, we mainly need these features:

1. generate skeleton flat images with tools such as ptool and replacing filenames of key assets (rootfs, ESP, CDT)
2. manage a database of supported boards (YAML is fine, but should be a standalone file)
3. download various assets (boot binaries and CDTs) as an input for the above

My gut feeling is that while one or multiple shell scripts can do this, it will reach the ceiling of readability and testability, so I'd recommend pivoting to python.

Perhaps 1) is a standalone somewhat generic script, and 2) and 3) are another script for debos flash integration calling the first one with target boards and specific inputs?

[gagath]

manage a database of supported boards (YAML is fine, but should be a standalone file)

I have started to do this on a personal branch using JSON, but the flash recipe was too complex to introduce the standalone database in a single PR. I would prefer multiple, focused PRs that allow us to arrive there.

My gut feeling is that while one or multiple shell scripts can do this, it will reach the ceiling of readability and testability, so I'd recommend pivoting to python.

Yes. I want unit tests around the code and I like Python. However to keep this PR simple I keep bash. The script can be migrated to Python later, with the introduction of unit tests, etc.

Since I see no major objections I will mark this as ready for review. Once merged we can continue to iterate towards our common goal.

[lool]

TBH I don't see the point of moving the shell code around and I would prefer we start a pull request with incremental commits switching to an external python script.

[lool]

Here's a sample PR with the commits converting from shell to python which Robie found easy to review :) #220

[gagath]

TBH I don't see the point of moving the shell code around and I would prefer we start a pull request with incremental commits switching to an external python script.

The immediate benefit is easier debug experience. I spent time figuring out I forgot to specify cdt_filename in the Glymur PR, and by having this I would have immediately seen that the argument passed to this new script was empty instead of debugging a shell error at line 12345 which is hard to debug in the current recipe due to how templating works (duplicating code). At least errors happening is this script have proper line numbers instead of "imaginary" ones.

This shell error was also found and debugged by Adam in this commit: abickett@73dede4

• Add missing cdt_filename for glymur-crd board (caused shell syntax error)

While I do not know how much it took them to understand the error, the "shell error syntax" mentioned could have been prevented by this PR which would replace it by a simpler "empty cdt_filename argument" investigation.

If this benefit is not seen as enough as-is, I can put Python rewrite commits on top. But this will prevent other people to be in this situation again while rewritten, hence my dual-PR proposal which I still maintain for contributors’ sake.

[gagath]

Rebased on top of main to include the recent changes for nvme and spinor support.