[BUG] Jinja2 TemplateNotFound error with using salt-ssh

[MemoryLeak55]

Your setup

Formula commit hash / release tag

Commit hash: 90500f9 (master branch, at 2020/06/29)

Versions reports (master & minion)

salt-ssh 3001 (From the ubuntu focal repos)
salt-minion 3000.3 (From pip)

salt-ssh is run from ubuntu 20.04 desktop
salt-minion is run on a minimal 20.04 focal image for LXC containers.

Pillar / config used

Nothing relevant here. Bug relates to formula itself. When server.sls tries to render redis-3.2-ng.conf.jinja a file not found error appears relating to map.jinja

---

Bug details

Describe the bug

If you try use the redis formula with salt-ssh it fails with a file not found error (the file is there). The command

salt-ssh -i <minion> state.apply redis.server

fails with

          ID: redis_config
    Function: file.managed
        Name: /etc/redis/redis.conf
      Result: False
     Comment: Unable to manage file: Jinja error: redis/map.jinja
              Traceback (most recent call last):
                File "/var/tmp/.salt_e05380_salt/pyall/salt/utils/templates.py", line 400, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/var/tmp/.salt_e05380_salt/pyall/jinja2/environment.py", line 1090, in render
                  self.environment.handle_exception()
                File "/var/tmp/.salt_e05380_salt/pyall/jinja2/environment.py", line 832, in handle_exception
                  reraise(*rewrite_traceback_stack(source=source))
                File "/var/tmp/.salt_e05380_salt/pyall/jinja2/_compat.py", line 28, in reraise
                  raise value.with_traceback(tb)
                File "<template>", line 4, in top-level template code
                File "/var/tmp/.salt_e05380_salt/pyall/salt/utils/jinja.py", line 204, in get_source
                  raise TemplateNotFound(template)
              jinja2.exceptions.TemplateNotFound: redis/map.jinja

              ; line 4

              ---
              #
              # This file is managed by salt. Do not edit by hand.
              #
              {% from "redis/map.jinja" import redis_settings with context %}    <======================
              ################################## INCLUDES ###################################
              {% if redis_settings.include is defined %}
              {%- for include in redis_settings.include %}
              include {{ include }}
              {%- endfor %}
              [...]
              ---
     Started: 10:17:55.643913
    Duration: 98.964 ms
     Changes:

The "redis/map.jinja" file is there.

Steps to reproduce the bug

Just try using the redis formula with salt-ssh in any way.

Expected behaviour

Installed redis, with appropriately rendered config

Attempts to fix the bug

Some other formulas have the same issue. I cannot recall which ones, but quite a few mainstream ones do fail in the same way with the map.jinja error. The formula works fine with regular salt command

Additional context

N/A, ask and I will do my best to answer

[myii]

Thanks for the detailed report, @thomas-thorburn-connect. This is a known problem encountered with salt-ssh. The solution is to avoid importing directly in templates and to pass data via. context instead (from the file.managed state). A couple of recent formulas that were fixed are shown under the subtasks of this user story:

• https://tree.taiga.io/project/myii-saltstack-formulas/us/210

So follow through to either of the PRs that are mentioned will show how this needs to be tackled here. This is probably the easier of the two to adapt for this formula:

• https://github.com/saltstack-formulas/postfix-formula/pull/104/files

The problem in this formula is that there are quite a few templates that will need to be adapted. @thomas-thorburn-connect May I suggest trying to adapt the template you're using locally, to confirm this fix works for you? We can then discuss how to proceed with the rest of the templates.

[infoveinx]

Hi. So with up to at least 2019.2 I was able to get around this with the --extra-filerefs arg, but I guess this doesn't work anymore with 3002.1 (version I am on now)? I get the TemplateNotFound errors now. This has halted my ability to manage some remote systems with salt-ssh. The nginx and sudoers formulas are having the same problem. It seems that any affected formula will need to be updated to work similar to what had to be done with the postfix one?

[MemoryLeak55]

I checked and it seems that I have fixed it locally. I apologise for not making a pull request, it's not something I've ever actually done on github. But I will paste my changes here in the interest of speed.

This is for redis/server.sls. You may or may not need the gid_from_name -> usergroup fix. Otherwise, you need to add two lines to the redis_config block, pushing the redis_settings as context to the settings files it's going to render.

diff --git a/redis/server.sls b/redis/server.sls
index 34e67b9..1de77c3 100644
--- a/redis/server.sls
+++ b/redis/server.sls
@@ -30,7 +30,7 @@ redis_group:
 redis_user:
   user.present:
     - name: {{ user }}
-    - gid_from_name: True
+    - usergroup: True
     - home: {{ home }}
     - require:
       - group: redis_group
@@ -84,6 +84,9 @@ redis_config:
 {% else %}
     - source: {{ redis_settings.source_path }}
 {% endif %}
+    - context:
+      redis_settings: {{ redis_settings|tojson }}
+

 {% if install_from == 'source' %}
 redis-initd:

I'm not going to paste the diff for the following part, because it there are many files it touches and it's all the same change. In redis/files/*.jinja you will find lines such as

{% from "redis/map.jinja" import redis_settings with context %}

Remove all instances of that line in all the files you find it in redis/files/, as it's the one causing the trouble.

I needed to make no more changes for it to work

[infoveinx]

No worries. After posting I realized this isn't the appropriate place since it's the redis formula (same problems though). It's late, I should sleep. I looked at some of the other affected formulas but they are quite large and complex for me to work on fixing.

[MemoryLeak55]

Sorry, it seems my reading comprehension went out the door, you did mention you had the issue with postfix and not redis. The postfix fix is nearly identical in terms of adding context in the .sls file that has the block that renders the config files themselves and removing some imports from files it renders, but as you've said this isn't the appropriate place

[tacerus]

Remove all instances of that line in all the files you find it in redis/files/, as it's the one causing the trouble.

The solution is to not use map.jinja anymore?