[AAP-34936] Manage Django Settings with Dynaconf

[rochacbruno]

AAP-34936

New or Enhanced Feature

Settings now managed by Dynaconf.

export DJANGO_SETTINGS_MODULE=awx.settings
dynaconf list
dynaconf inspect

NOTE: Currently depends on DAB ansible/django-ansible-base#689 to be merged.

DOCS https://github.com/rochacbruno/django-ansible-base/blob/dynaconf_settings/docs/lib/dynamic_config.md

[sonarqubecloud]

Quality Gate passed

Issues
9 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[AlanCoding]

So how would the awx/conf (dynamic database backed settings) be handled with this?

[codecov]

Codecov Report

Attention: Patch coverage is 67.92453% with 34 lines in your changes missing coverage. Please review.

Project coverage is 75.26%. Comparing base (70ea0a7) to head (1efb08d).

✅ All tests successful. No failed tests found.

[rochacbruno]

So how would the awx/conf (dynamic database backed settings) be handled with this?

@AlanCoding will not, this is going to be on phase 2 only.

[rooftopcellist]

FYI, it looks like the awx-operator CI check failed. The AWX deployment by the operator itself was successful, but the job template failed. This may be related to the settings changes in this PR. I would next check that job templates run successfully in the AWX dev environment. If they do, then there are likely awx-operator changes needed around how we mount-in the settings.py file into the task and web pods.

[AlanCoding]

I did an updated run and got the following traceback when awx-manage version is ran.

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/dynaconf/loaders/py_loader.py", line 121, in get_module
    mod = importlib.import_module(filename)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named '/etc/tower/conf'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
             ^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/__init__.py", line 77, in manage
    prepare_env()
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/__init__.py", line 71, in prepare_env
    if not settings.DEBUG:  # pragma: no cover
           ^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/settings/__init__.py", line 50, in <module>
    DYNACONF.load_file(settings_files_path)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/dynaconf/base.py", line 1305, in load_file
    settings_loader(
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/dynaconf/loaders/__init__.py", line 352, in settings_loader
    py_loader.load(
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/dynaconf/loaders/py_loader.py", line 35, in load
    mod, loaded_from = get_module(obj, settings_module, silent)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/dynaconf/loaders/py_loader.py", line 125, in get_module
    mod = import_from_filename(obj, filename, silent=silent)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/dynaconf/loaders/py_loader.py", line 161, in import_from_filename
    exec(compile(config_file.read(), filename, "exec"), mod.__dict__)
  File "/etc/tower/conf.d/gateway.py", line 9, in <module>
    LOGGING['loggers']['aap'] = {'handlers': ['console', 'file', 'tower_warnings', 'external_logger'], 'level': 'WARNING'}
    ^^^^^^^
NameError: name 'LOGGING' is not defined

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[AlanCoding]

With the latest fixes, this gets to the following traceback on install

Traceback (most recent call last):
  File "/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
             ^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/__init__.py", line 77, in manage
    prepare_env()
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/__init__.py", line 71, in prepare_env
    if not settings.DEBUG:  # pragma: no cover
           ^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/settings/__init__.py", line 86, in <module>
    raise ImproperlyConfigured(msg)
django.core.exceptions.ImproperlyConfigured: No AWX configuration found at ['/etc/tower', '/etc/ansible-automation-platform/'].
Define the AWX_SETTINGS_FILE environment variable to specify an alternate path.

[rochacbruno]

@AlanCoding the error above is a validation that I explicitly added to replicate the following lines

awx/awx/settings/production.py

Lines 70 to 99 in 7d30dff

	# Attempt to load settings from /etc/tower/settings.py first, followed by
	# /etc/tower/conf.d/*.py.
	try:
	include(settings_file, optional(settings_files), scope=locals())
	except ImportError:
	traceback.print_exc()
	sys.exit(1)
	except IOError:
	from django.core.exceptions import ImproperlyConfigured
	included_file = locals().get('__included_file__', '')
	if not included_file or included_file == settings_file:
	# The import doesn't always give permission denied, so try to open the
	# settings file directly.
	try:
	e = None
	open(settings_file)
	except IOError:
	pass
	if e and e.errno == errno.EACCES:
	SECRET_KEY = 'permission-denied'
	LOGGING = {}
	else:
	msg = 'No AWX configuration found at %s.' % settings_file
	msg += '\nDefine the AWX_SETTINGS_FILE environment variable to '
	msg += 'specify an alternate path.'
	raise ImproperlyConfigured(msg)
	else:
	raise

My logic was: If neither AWX_SETTINGS_FILE or any file from /etc/tower or /etc/ansible-* is found, it raises an error, because at least one config file is required.

So it look like that is not the case right?

[AlanCoding]

The expectation is that /etc/tower/settings.py exists in the context I'm pasting that from.

My logic was: If neither AWX_SETTINGS_FILE or any file from /etc/tower or /etc/ansible-* is found, it raises an error, because at least one config file is required.

Help me talk that out a little better. Again, I can wave away corner cases here and presume that settings_file is /etc/tower/settings.py. So what is this doing?

    if not included_file or included_file == settings_file:

Specifically, included_file == settings_file. In the happy path of the really default installation, I expect this is always true. Maybe I'm barking up the wrong tree here, but I'm not sure.

[rochacbruno]

@AlanCoding this is logic from split_settings

included_file = locals().get('__included_file__', '') 
     if not included_file or included_file == settings_file: 

From what I learned, split_settings keeps state of which files already loaded in the __included_file__ variable.

As I am removing split_settings, this logic is not applying anymore, as dynaconf tracks loaded files in a different way.