elasticache : Security groups not being removed on update

[mcandre]

Summary

The Ansible documentation declares the field value data type as a list, however that is not accurate. In fact, the data type is a string. Some places neglect to clarify that the string contents are comma delimited, such as for ElastiCache security group ID's.

When I try to specify security_group_ids for ElastiCache, either as a YAML list or as a comma-delimited list, I get the same error either way. Ansible complains KeyError: 'SecurityGroups'.

There are zero examples of how to actually specify security_group_ids in the Ansible ElastiCache documentation.

Issue Type

Bug Report

Component Name

rds, elasticache, ec2_elb_lb, etc.

Ansible Version

$ ansible --version
ansible 2.10.7
  config file = /Users/andrew/go/src/github.com/kudohq/kudo-devops/ansible.cfg
  configured module search path = ['/Users/andrew/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/andrew/.pyenv/versions/3.9.2/lib/python3.9/site-packages/ansible
  executable location = /Users/andrew/.pyenv/versions/3.9.2/bin/ansible
  python version = 3.9.2 (default, Mar 23 2021, 09:59:47) [Clang 13.0.0 (https://github.com/llvm/llvm-project.git 72557476d459969dbee9525

Collection Versions

$ ansible-galaxy collection list

# /Users/andrew/.pyenv/versions/3.9.2/lib/python3.9/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    1.4.1  
ansible.netcommon             1.5.0  
ansible.posix                 1.2.0  
ansible.utils                 2.0.1  
ansible.windows               1.4.0  
arista.eos                    1.3.0  
awx.awx                       17.1.0 
azure.azcollection            1.4.0  
check_point.mgmt              2.0.0  
chocolatey.chocolatey         1.0.2  
cisco.aci                     2.0.0  
cisco.asa                     1.0.4  
cisco.intersight              1.0.11 
cisco.ios                     1.3.0  
cisco.iosxr                   1.2.1  
cisco.meraki                  2.2.1  
cisco.mso                     1.1.0  
cisco.nso                     1.0.3  
cisco.nxos                    1.4.0  
cisco.ucs                     1.6.0  
cloudscale_ch.cloud           2.1.0  
community.aws                 1.4.0  
community.azure               1.0.0  
community.crypto              1.5.0  
community.digitalocean        1.0.0  
community.docker              1.3.0  
community.fortios             1.0.0  
community.general             2.2.0  
community.google              1.0.0  
community.grafana             1.2.0  
community.hashi_vault         1.1.2  
community.hrobot              1.1.1  
community.kubernetes          1.2.0  
community.kubevirt            1.0.0  
community.libvirt             1.0.1  
community.mongodb             1.2.1  
community.mysql               1.3.0  
community.network             2.0.1  
community.okd                 1.1.0  
community.postgresql          1.1.1  
community.proxysql            1.0.0  
community.rabbitmq            1.0.3  
community.routeros            1.1.0  
community.skydive             1.0.0  
community.sops                1.0.5  
community.vmware              1.8.0  
community.windows             1.3.0  
community.zabbix              1.2.0  
containers.podman             1.4.4  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.5  
dellemc.openmanage            3.1.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.8.1  
fortinet.fortimanager         2.0.1  
fortinet.fortios              1.1.9  
frr.frr                       1.0.3  
gluster.gluster               1.0.1  
google.cloud                  1.0.2  
hetzner.hcloud                1.2.1  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.2.4  
inspur.sm                     1.1.2  
junipernetworks.junos         1.3.0  
kubernetes.core               1.2.0  
mellanox.onyx                 1.0.0  
netapp.aws                    20.9.0 
netapp.elementsw              20.11.0
netapp.ontap                  21.3.1 
netapp_eseries.santricity     1.1.0  
netbox.netbox                 2.1.0  
ngine_io.cloudstack           2.0.0  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.0  
openstack.cloud               1.3.0  
openvswitch.openvswitch       1.2.0  
ovirt.ovirt                   1.3.1  
purestorage.flasharray        1.6.2  
purestorage.flashblade        1.4.0  
sensu.sensu_go                1.9.1  
servicenow.servicenow         1.0.4  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.15.0 
theforeman.foreman            1.5.1  
vyos.vyos                     1.1.1  
wti.remote                    1.0.1  

# /Users/andrew/.ansible/collections/ansible_collections
Collection           Version
-------------------- -------
amazon.aws           1.4.1  
community.aws        1.4.0  
community.kubernetes 1.2.1  
kubernetes.core      1.2.1

AWS SDK versions

$ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: /Users/andrew/.pyenv/versions/3.9.2/lib/python3.9/site-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.17.34
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /Users/andrew/.pyenv/versions/3.9.2/lib/python3.9/site-packages
Requires: botocore, s3transfer, jmespath
Required-by: 
---
Name: botocore
Version: 1.20.34
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /Users/andrew/.pyenv/versions/3.9.2/lib/python3.9/site-packages
Requires: urllib3, python-dateutil, jmespath
Required-by: s3transfer, boto3

Configuration

$ ansible-config dump --only-changed
DEFAULT_STDOUT_CALLBACK(/Users/andrew/go/src/github.com/kudohq/kudo-devops/ansible.cfg) = skippy
DEPRECATION_WARNINGS(/Users/andrew/go/src/github.com/kudohq/kudo-devops/ansible.cfg) = False
DISPLAY_SKIPPED_HOSTS(/Users/andrew/go/src/github.com/kudohq/kudo-devops/ansible.cfg) = False
INTERPRETER_PYTHON(env: ANSIBLE_PYTHON_INTERPRETER) = /Users/andrew/.pyenv/shims/python

OS / Environment

macOS Big Sur

Steps to Reproduce

$ ansible-playbook -i hosts.ini playbooks/provision.yml

Expected Results

The specified security group is added to the ElastiCache cluster.

Actual Results

TASK [Provision KCP NG ElastiCache (Staging)] **********************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'SecurityGroups'

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[mcandre]

Updates:

• Dry run with ansible-lint does not report the error.
• Updating the community.aws and amazon.aws dependencies to v1.5.0 resolves the runtime playbook error.
• The documentation is not consistent about specifying the minimum module version for new fields.

[ansibullbot]

cc @alachaum @jillr @jsdalton @markuman @mjschultz @nand0p @s-hertel @tremble @wimnat
click here for bot help

[mcandre]

As with RDS, the ElastiCache module appears to silently ignore configurations that attempt to change security group ID's on existing clusters.

Please indicate this in the documentation.

If at all possible, please update the validation checks to indicate that it will not be possible to apply the configured security groups on existing clusters, so that users can catch this edge case earlier, without getting confused by the Ansible behavior.

[alinabuzachis]

@mcandre Thank you for reporting this. Would you be willing to open a pull request with a patch and some tests?