[BUG] Inline Pillars are still getting cached in a particular scenario when pillar_cache is set to True

[sanket925]

Description
We have found that despite #66292 is fixed , inline pillars are still getting cached in a particular scenario.

Setup
pillar cache is enabled

#/etc/salt/master.d/master.conf

user: root
pillar_cache: True

orchestration file test_pillar_cache_orch.sls:

{%- set event_data = salt.pillar.get('event_data') %}
{# this is the master minion #}
{%- set master_minion = salt.pillar.get('master_minion') %}

{%- do salt.cmd.run('salt ' ~ master_minion ~ ' saltutil.refresh_pillar') %}
{%- do salt.cmd.run('sleep 60') %}  # Time to reflect pillar data change after pillar refresh is done.

{%- do salt.state.apply(mods='test_pillar_cache_state',
                       concurrent=True,
                       failhard=True) %}

State file test_pillar_cache_state:

{% set event_data = salt.pillar.get('event_data', None) %}

debug_msg:
  cmd.run:
    - name: "echo {{event_data}}"

Driver code driver_code.sh

rm -rf nohup.out
nohup salt-run state.orch test_pillar_cache_orch pillar="{'event_data': {'pillar1': 'value1'}, 'master_minion':'<master_minion_name>'}" &

nohup salt-run state.orch test_pillar_cache_orch pillar="{'event_data': {'pillar1': 'value2', 'pillar2': 'value2'}, 'master_minion':'<master_minion_name>'}" &

test script to print the pillar after the problem is reproduced test.sls

{% set event_data = salt.pillar.get('event_data', None) %}

debug_msg:
  cmd.run:
    - name: "echo {{event_data}}"

Please be as specific as possible and give set-up details.

My setup is on a docker container running with redhat 9 as base image.

Steps to Reproduce the behavior
1 replace the master minion name in driver_code.sh
2 run driver_code.sh
3 salt-run state.orch test

If the problem is reproduced , the pillars passed to the orchs in driver code are printed in test.sls execution.

Additionally what we found is when the problem gets reproduced , pillarenv is None which should be base as default. We have also made an attempt to fix the issue which resolves the problem .
here is the PR : #67899
Though the root cause could be different , but adding a null check similar to saltenv in the above code actually solves the problem . Until root cause is found , will this be a safe fix ?

We also attempted to throw an exception when pillarenv is none and this is what it prints

NoneType: None
2025-03-20 18:19:13,148 [salt.master      :1935][ERROR   ][6892] Error in function _pillar:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py", line 1929, in run_func
    ret = getattr(self, func)(load)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py", line 1608, in _pillar
    pillar = salt.pillar.get_pillar(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/pillar/__init__.py", line 75, in get_pillar
    return PillarCache(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/pillar/__init__.py", line 454, in __init__

Expected behavior
Inline pillars passed to an orchestration should not be cached.

Screenshots

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.8

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.44
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.16.0
         smmap: 5.0.2
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rhel 9.5 Plow
        locale: utf-8
       machine: x86_64
       release: 3.10.0-1160.59.1.el7.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 9.5 Plow

Additional context
Add any other context about the problem here.

[whytewolf]

is that second line of driver_code.sh supposed to be there? cause i don't have a plm.azure_plm orchestration. i have tried with a random orchestration a broken orchestration and just leaving that line out.

anyway. I can not seem to replicate the issue. even though i have no pillarenv set. and show None as my pillarenv when checking config. and using as close to what is shown here as possible.

I even added a sleep to the test and put the output test directly in the driver_code.sh just in case there was either to much or not enough delay.

root@salt00:/srv/salt# sh driver_code.sh
nohup: appending output to 'nohup.out'
root@salt00:/srv/salt# salt-run state.orch test
saltmaster:
----------
          ID: debug_msg
    Function: cmd.run
        Name: echo None
      Result: True
     Comment: Command "echo None" run
     Started: 18:00:41.121587
    Duration: 5.868 ms
     Changes:
              ----------
              pid:
                  3348340
              retcode:
                  0
              stderr:
              stdout:
                  None

Summary for saltmaster
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   5.868 ms
root@salt00:/srv/salt# salt-call config.get pillarenv
local:
    None
root@salt00:/srv/salt# salt-run config.get pillarenv
None

can you try on 3008.10 with a clean install and see if the problem persists?
[a clean install being backing up configs and pki info, removing the software completely, running autoremove to cover any unknown extras, nuking /opt/saltstack and /var/cache/salt then reinstalling and putting pki and configs back.]

[sanket925]

Hi @whytewolf , sorry there was a typo in the driver_code. I have corrected it. It is basically the same orchestration (test_pillar_cache_orch.sls) called twice one after the other. Could you please check now ?
Meanwhile I will try to reproduce it with a clean install on 3006.10. I hope you mean 3006.10

[whytewolf]

alright, updated the driver with that info. and still getting the same result which is the expected result. that the inline pillar data is not showing up in the test.sls when run.

root@salt00:/srv/salt# salt-run state.orch test
saltmaster:
----------
          ID: debug_msg
    Function: cmd.run
        Name: echo None
      Result: True
     Comment: Command "echo None" run
     Started: 18:46:54.659180
    Duration: 5.683 ms
     Changes:
              ----------
              pid:
                  3397104
              retcode:
                  0
              stderr:
              stdout:
                  None

Summary for saltmaster
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   5.683 ms

btw, about your wip bug fix. unfortunatly setting the config to base by default instead of None is a non starter. there is a huge functionality difference between None and Base. setting the pillarenv and saltenv directly to an Env will cause the system to ignore other environments. this will cause all sorts of problems when it comes to how top files work within a setup where the pillarenv and saltenv are not set. currently None can allow the top file to choose the environment.

[sanket925]

@whytewolf Thank you so much for the insights.
I have tried 2 scenarios now :
1 -> Docker image ubuntu with salt 3006.8 (version same as my corporate setup)
2 -> Docker image ubuntu with salt 3006.10 (latest. version)

On 1 , I was able to reproduce the issue atleast twice (tried about 10 times) but on 2 I was not able to reproduce the issue.
Since I am not consistently able to reproduce it , I would not rule out the possibility that the issue could still persist on 3006.10

When I was reproduce it once in a simple setup , I had toggled the pillarenv from none to base in the minion config file.

I can try to install 3006.10 in our corporate setup but it might take time.
Meanwhile, are there any logs or configuration which I can share to you ?
When the issue gets reproduced , I have evidence of the inline pillar getting cached in /var/cache/salt/pillar_cache/master_minion file.

[sanket925]

@whytewolf I was able to reproduce the issue on my corporate setup with 3006.10. Would some trace level logs helpful for you?

[sanket925]

Any update on this ? We are impacted due to this issue onto our production systems

[whytewolf]

Any update on this ? We are impacted due to this issue onto our production systems

sorry. as a volunteer i don't actually always have the time to work on this. unfortunately i have still not recreated the issue, and i have run the test case a hundred times. I'm unsure how it is happening in your enviroment. unless somehow the overstate pillar info is somehow crossing the streams into the full pillar data. but i don't see how that would be happening. the cache is set to throw away the overstate info so unless the data is coming in in some other method it shouldn't be saved.