Add docs about execution order of Cloud-init files, steps, and substeps

docs/content/en/docs/Customizing/stages.md

@@ -14,19 +14,35 @@ We have a custom augmented cloud-init syntax that allows to hook into various st
 - Network availability
 - During upgrades, installation, deployments  , and resets
 
-Cloud-init files in `/system/oem`, `/oem` and `/usr/local/oem` are applied in 5 different phases: `boot`, `network`, `fs`, `initramfs` and `reconcile`. All the available cloud-init keywords can be used in each stage. Additionally, it's possible also to hook before or after a stage has run, each one has a specific stage which is possible to run steps: `boot.after`, `network.before`, `fs.after` etc.
+Cloud-init files in `/system/oem`, `/oem`, `/usr/local/oem`, and kernel boot args are applied in 5 different stages: `boot`, `network`, `fs`, `initramfs` and `reconcile`. All the available cloud-init keywords can be used in each stage. Additionally, it's possible also to hook before or after a stage has run, each one has a specific stage which is possible to run steps: `boot.after`, `network.before`, `fs.after` etc.
 
 Multiple stages can be specified in a single cloud-init file.
 
+File extension name must be *.yaml or *.yml.
+
 {{% alert title="Note" %}}
-When a Elemental derivative boots it creates sentinel files in order to allow to execute cloud-init steps programmaticaly.
+When an Elemental derivative boots it creates sentinel files in order to allow executing cloud-init steps programmatically.
 
 - `/run/cos/recovery_mode` is being created when booting from the recovery partition
 - `/run/cos/live_mode` is created when booting from the LiveCD
 
 To execute a block using the sentinel files you can specify: `if: '[ -f "/run/cos/..." ]'`, see the examples below.
 {{% /alert %}}
 
+At every stage, Elemental derivative parse and execute Cloud-init files in the following order:
+- `/system/oem`  (Cannot be modified by users)
+- `/oem`  (Can be modified by users)
+- `/usr/local/oem` (Can be modified by users)
+- Cloud-init config URL specified in kernel boot args, e.g., elemental.setup=http://example.com/cloudinit.cfg
+- Cloud-init config encoded in kernel boot args, e.g., stages.network[0].authorized_keys.user=github:user
+
+In each above directory, files are processed in the lexical order of file name. 
+
+Unlike the standard Cloud-init, which merges all config files before execution, Elemental derivative executes
+Cloud-init files one by one.
+
+Therefore, definitions in later Cloud-init files override what previous files defined, partially or completely.
+
 ## Stages
 
 Below there is a detailed list of the stages available that can be used in the cloud-init configuration files

docs/content/en/docs/Reference/cloud_init.md

@@ -85,7 +85,84 @@ stages:
 The default cloud-config format is split into *stages* (*initramfs*, *boot*, *network*, *initramfs*, *reconcile*, called generically **STAGE_ID** below) [see also stages](../../customizing/stages) that are emitted internally during the various phases by calling `cos-setup STAGE_ID`. 
 *steps* (**STEP_NAME** below) defined for each stage are executed in order.
 
-Each cloud-config file is loaded and executed only at the apprioriate stage, this allows further components to emit their own stages at the desired time.
+Each cloud-config file is loaded and executed only at the appropriate stage, this allows further components to emit their own stages at the desired time.
+
+_Note_:
+- The execution order of multiple `substeps` within a single `step` might be not what you think, especially whether the `commands` are executed after other substeps. For example:
+  ```
+  stages:
+    fs:
+      - name: test
+        environment_file: /tmp/env_file1
+        environment:
+          ENV1: this is ENV1
+        files:
+          - path: /tmp/file1
+            content: "this is file1\n"
+        directories:
+          - path: /tmp/dir1
+        users:
+          user1:
+              passwd: password
+        commands:
+          - |
+            bash -x >> /tmp/log 2>&1 <<'EOF'
+            ls -lFd /tmp/env_file1
+            printenv ENV1
+            ls -lFd /tmp/file1
+            ls -lFd /tmp/dir1
+            id user1
+            EOF
+  ```
+
+  You might expect that when the `commands` get executed, all things defined above it has been executed, such as the user1 has been created.
+  But it is not.  As of now, the output (/tmp/log) is:
+  ```
+  + ls -lFd /tmp/env_file1
+  ls: cannot access '/tmp/env_file1': No such file or directory
+  + printenv ENV1
+  this is ENV1
+  + ls -lFd /tmp/file1
+  ---------- 1 root root 14 Aug  6 07:13 /tmp/file1
+  + ls -lFd /tmp/dir1
+  d--------- 2 root root 40 Aug  6 07:13 /tmp/dir1/
+  + id user1
+  id: ‘user1’: no such user
+  ```
+
+  The internal order is depending on the implementation, see [cloudinit.go](https://github.com/rancher/elemental-toolkit/blob/main/pkg/cloudinit/cloudinit.go#L52).
+  You can see that the `plugins.User` is behind `plugins.Commands`.
+
+  Therefore, to make sure the order is what you want, you may split the `substeps` into multiple `steps`, for example:
+  ```
+  stages:
+    fs:
+      - name: step1
+        environment_file: /tmp/env_file1
+        environment:
+          ENV1: this is ENV1
+        files:
+          - path: /tmp/file1
+            content: "this is file1\n"
+        directories:
+          - path: /tmp/dir1
+        users:
+          user1:
+              passwd: password
+      - name: step2
+        commands:
+          - |
+            bash -x >> /tmp/log 2>&1 <<'EOF'
+            ls -lFd /tmp/env_file1
+            printenv ENV1
+            ls -lFd /tmp/file1
+            ls -lFd /tmp/dir1
+            id user1
+            EOF
+  ```
+
+- The name of `steps` and output of `substeps` will be output to system journal log which can be viewed by the command `journalctl -u 'elemental*'`.
+- It is highly recommended to declare the `name` property of *steps*, so that it will be easier to investigate the output of `journalctl -u 'elemental*'` by the name.
 
 {{% pageinfo %}}
 The [cloud-init tool](https://github.com/mudler/yip#readme) can be also run standalone, this helps debugging locally and also during development, you can find separate [releases here](https://github.com/mudler/yip/releases).