Finalise standardisations remaining from the `_mapdata` implementation

[myii]

#203 has been merged but there are outstanding issues that need to be finalised, particularly when standardising across all of the formulas in our organisation (using the ssf-formula). Quoting the relevant comment here:

• [1] Use text-based instead of YAML-based comparison? including test(map): verify map.jinja dump using _mapdata state #203 (comment)
  • No change required (at least for the time being): Just use text-based comparison and update the PRs in other formulas to do the same.
• [2] Set up mapdata in _mapdata/init.sls
  • No change required (at least for the time being): Merge as-is and resolve this when centralising the solution in the ssf-formula.
• [3] Remove replacement from _mapdata_spec.rb? including test(map): verify map.jinja dump using _mapdata state #203 (comment)
  • No change required (at least for the time being): I don't believe there will be many formulas that will need this so no need to add the extra layer of complexity; however, can be resolved when centralising the solution in the ssf-formula.
• [4] Use control scope for variables in InSpec?
  • No change required (at least for the time being): Merge as-is and open up for further discussion; @n-rodriguez, do you know anything about this?
• [5] Change commands => command
  • @baby-gnu, it would be better to get this fixed before merging; perhaps I can add a tiny commit to do this.
• [6] Use InSpec shared resources across all formulas?
  • No change required (at least for the time being): Merge as-is and resolve this when centralising the solution in the ssf-formula.
• [7] Replace old platform_finger concatenation with shared library method instead
  • No change required (at least for the time being): Either resolve in an extra commit or leave for a subsequent PR.

[baby-gnu]

Hello.

For [1], we could add an extra state in _mapdata/init.sls to strip CR from the generated file on windows only?

Looking at file.managed documentation and issues, I found that there is no way to force line ending but I found that opened issue saltstack/salt#14135.

[myii]

Hello.

For [1], we could add an extra state in _mapdata/init.sls to strip CR from the generated file on windows only?

Looking at file.managed documentation and issues, I found that there is no way to force line ending but I found that opened issue saltstack/salt#14135.

@baby-gnu The whole discussion started when I was testing this _mapdata implementation with the salt-formula, since it has so many platforms and is tested on Windows as well. Using gsub was the workaround that did it:

• myii/salt-formula@3c05bde
• https://github.com/myii/salt-formula/runs/1032273456?check_suite_focus=true#step:8:1532

Of course, would just need to apply that for Windows only.

[baby-gnu]

@baby-gnu The whole discussion started when I was testing this _mapdata implementation with the salt-formula, since it has so many platforms and is tested on Windows as well. Using gsub was the workaround that did it:

yes, we have some internal formulas at work only for windows machines and found an issue this way.

I was wondering what's better between manage the difference in the state or the check. any preference?

[myii]

I was wondering what's better between manage the difference in the state or the check. any preference?

@baby-gnu Do you have a working state available? It would be useful to compare both methods.

[baby-gnu]

@baby-gnu Do you have a working state available? It would be useful to compare both methods.

You can try the following:

diff --git a/TEMPLATE/_mapdata/init.sls b/TEMPLATE/_mapdata/init.sls
index cfa4837..d253a3c 100644
--- a/TEMPLATE/_mapdata/init.sls
+++ b/TEMPLATE/_mapdata/init.sls
@@ -16,3 +16,11 @@
     - template: jinja
     - context:
         map: {{ mapdata | yaml }}
+
+{%- if grains['os_family'] == 'Windows' %}
+{{ tplroot }}-mapdata-strip-newline:
+  file.replace:
+    - name: {{ output_file }}
+    - pattern: '\r'
+    - repl: ''
+{%- endif %}

Here is the result when applying the TEMPLATE._mapdata a second time:

PC-386675.example.net:
----------
          ID: TEMPLATE-mapdata-dump
    Function: file.managed
        Name: /tmp/salt_mapdata_dump.yaml
      Result: True
     Comment: File /tmp/salt_mapdata_dump.yaml updated
     Started: 11:53:06.906250
    Duration: 109.375 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1,13 +1,13 @@
                  -# yamllint disable rule:indentation rule:line-length
                  -# Windows-7
                  ----
                  -added_in_defaults: defaults_value
                  -config: /etc/TEMPLATE
                  -pkg:
                  -  name: TEMPLATE
                  -rootgroup: root
                  -service:
                  -  name: TEMPLATE
                  -subcomponent:
                  -  config: /etc/TEMPLATE-subcomponent-formula.conf
                  -winner: defaults
                  +# yamllint disable rule:indentation rule:line-length
                  +# Windows-7
                  +---
                  +added_in_defaults: defaults_value
                  +config: /etc/TEMPLATE
                  +pkg:
                  +  name: TEMPLATE
                  +rootgroup: root
                  +service:
                  +  name: TEMPLATE
                  +subcomponent:
                  +  config: /etc/TEMPLATE-subcomponent-formula.conf
                  +winner: defaults
----------
          ID: TEMPLATE-mapdata-strip-newline
    Function: file.replace
        Name: /tmp/salt_mapdata_dump.yaml
      Result: True
     Comment: Changes were made
     Started: 11:53:07.015625
    Duration: 0.0 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1,13 +1,13 @@
                  -# yamllint disable rule:indentation rule:line-length
                  -# Windows-7
                  ----
                  -added_in_defaults: defaults_value
                  -config: /etc/TEMPLATE
                  -pkg:
                  -  name: TEMPLATE
                  -rootgroup: root
                  -service:
                  -  name: TEMPLATE
                  -subcomponent:
                  -  config: /etc/TEMPLATE-subcomponent-formula.conf
                  -winner: defaults
                  +# yamllint disable rule:indentation rule:line-length
                  +# Windows-7
                  +---
                  +added_in_defaults: defaults_value
                  +config: /etc/TEMPLATE
                  +pkg:
                  +  name: TEMPLATE
                  +rootgroup: root
                  +service:
                  +  name: TEMPLATE
                  +subcomponent:
                  +  config: /etc/TEMPLATE-subcomponent-formula.conf
                  +winner: defaults

The salt-run state.event pretty=True show the following:

salt/job/20200907095318212067/ret/PC-386675.example.net	{
    "_stamp": "2020-09-07T09:53:21.087071",
    "cmd": "_return",
    "fun": "state.apply",
    "fun_args": [
        "TEMPLATE._mapdata"
    ],
    "id": "PC-386675.example.net",
    "jid": "20200907095318212067",
    "out": "highstate",
    "retcode": 0,
    "return": {
        "file_|-TEMPLATE-mapdata-dump_|-/tmp/salt_mapdata_dump.yaml_|-managed": {
            "__id__": "TEMPLATE-mapdata-dump",
            "__run_num__": 0,
            "__sls__": "TEMPLATE._mapdata",
            "changes": {
                "diff": "--- \n+++ \n@@ -1,13 +1,13 @@\n-# yamllint disable rule:indentation rule:line-length\n-# Windows-7\n----\n-added_in_defaults: defaults_value\n-config: /etc/TEMPLATE\n-pkg:\n-  name: TEMPLATE\n-rootgroup: root\n-service:\n-  name: TEMPLATE\n-subcomponent:\n-  config: /etc/TEMPLATE-subcomponent-formula.conf\n-winner: defaults\n+# yamllint disable rule:indentation rule:line-length\r\n+# Windows-7\r\n+---\r\n+added_in_defaults: defaults_value\r\n+config: /etc/TEMPLATE\r\n+pkg:\r\n+  name: TEMPLATE\r\n+rootgroup: root\r\n+service:\r\n+  name: TEMPLATE\r\n+subcomponent:\r\n+  config: /etc/TEMPLATE-subcomponent-formula.conf\r\n+winner: defaults\r\n"
            },
            "comment": "File /tmp/salt_mapdata_dump.yaml updated",
            "duration": 109.375,
            "name": "/tmp/salt_mapdata_dump.yaml",
            "result": true,
            "start_time": "11:53:06.906250"
        },
        "file_|-TEMPLATE-mapdata-strip-newline_|-/tmp/salt_mapdata_dump.yaml_|-replace": {
            "__id__": "TEMPLATE-mapdata-strip-newline",
            "__run_num__": 1,
            "__sls__": "TEMPLATE._mapdata",
            "changes": {
                "diff": "--- \n+++ \n@@ -1,13 +1,13 @@\n-# yamllint disable rule:indentation rule:line-length\r\n-# Windows-7\r\n----\r\n-added_in_defaults: defaults_value\r\n-config: /etc/TEMPLATE\r\n-pkg:\r\n-  name: TEMPLATE\r\n-rootgroup: root\r\n-service:\r\n-  name: TEMPLATE\r\n-subcomponent:\r\n-  config: /etc/TEMPLATE-subcomponent-formula.conf\r\n-winner: defaults\r\n+# yamllint disable rule:indentation rule:line-length\n+# Windows-7\n+---\n+added_in_defaults: defaults_value\n+config: /etc/TEMPLATE\n+pkg:\n+  name: TEMPLATE\n+rootgroup: root\n+service:\n+  name: TEMPLATE\n+subcomponent:\n+  config: /etc/TEMPLATE-subcomponent-formula.conf\n+winner: defaults\n"
            },
            "comment": "Changes were made",
            "duration": 0.0,
            "name": "/tmp/salt_mapdata_dump.yaml",
            "result": true,
            "start_time": "11:53:07.015625"
        }
    },
    "success": true
}

[baby-gnu]

NB: for windows I had to create the /tmp directory manually ;-)

[baby-gnu]

I finished with the following code to handle windows minions:

diff --git a/TEMPLATE/_mapdata/init.sls b/TEMPLATE/_mapdata/init.sls
index cfa4837..18e96fc 100644
--- a/TEMPLATE/_mapdata/init.sls
+++ b/TEMPLATE/_mapdata/init.sls
@@ -16,3 +16,23 @@
     - template: jinja
     - context:
         map: {{ mapdata | yaml }}
+
+{%- if grains['os_family'] == 'Windows' %}
+{#- file.directory requires an absolute path with drive letter #}
+{% set system_drive = salt['environ.get']('SYSTEMDRIVE', 'C:') %}
+{% set tmp_dir = system_drive ~ '/tmp' %}
+
+{{ tplroot }}-mapdata-tmp-file-directory:
+  file.directory:
+    - name: {{ tmp_dir }}
+    - require_in:
+      - file: {{ tplroot }}-mapdata-dump
+
+{{ tplroot }}-mapdata-strip-newline:
+  file.replace:
+    - name: {{ output_file }}
+    - pattern: '\r'
+    - repl: ''
+    - require:
+      - file: {{ tplroot }}-mapdata-dump
+{%- endif %}

Unfortunately, I don't know how to run InSpec with my windows machine, so I don't know how to check if it's working correctly.

[dafyddj]

You can run InSpec against the Windows Vagrant image used by Kitchen in some other formulas, but it hasn't been implemented in the template-formula yet.

[baby-gnu]

I closed [3] and [4] since the latest _mapdata_spec.rb implement them.

[myii]

Actually, the rest are also resolved now (so I've ticked them):

• [1] Use text-based instead of YAML-based comparison? including test(map): verify map.jinja dump using _mapdata state #203 (comment)

  • template-formula/test/integration/default/controls/_mapdata_spec.rb

    Lines 8 to 16 in d0d9cce

    	### Method
    	# The steps below for each file appear convoluted but they are both required
    	# and similar in nature:
    	# 1. The earliest method was to simply compare the files textually but this often
    	# led to false positives due to inconsistencies (e.g. spacing, ordering)
    	# 2. The next method was to load the files back into YAML structures and then
    	# compare but InSpec provided block diffs this way, unusable by end users
    	# 3. The final step was to dump the YAML structures back into a string to use
    	# for the comparison; this both worked and provided human-friendly diffs

• [2] Set up mapdata in _mapdata/init.sls
  • https://github.com/myii/ssf-formula/blob/5a5c507434d1687d4437845346ee7a76a240f6fd/ssf/files/default/formula/_mapdata/init.sls#L8-L10
• [6] Use InSpec shared resources across all formulas?
  • https://github.com/myii/ssf-formula/blob/51160d18db22988a3dfbb952d1556a5e0fdb8acb/ssf/formulas.yaml#L120

---

Leaving this issue open for now since there may be another issue encountered when finalising situations with the v5 map.jinja conversion in these initial formulas. Let's close this issue once it has been successfully established in this formula.