Same settings not available within tests

[pevogam]

Hi all, since I don't know if this is an actual problem or something we have explicitly opted for, I am reporting it as a discussion thread and not as an issue. It could easily be converted of course.

The problem I am encountering for quite a while is that lots of our tests make use of the settings module to access some universally configured avocado parameters. These however are not really propagated to the test side as the settings object there has different (perhaps default) parameter values:

# on the job/runner level
2021-09-15 13:54:01,051 runner           L0394 CRITI| <class 'avocado.core.settings.Settings'>
2021-09-15 13:54:01,051 runner           L0395 CRITI| 140005721097312
2021-09-15 13:54:01,051 runner           L0396 CRITI| {'assets.fetch.ignore_errors': False, 'assets.fetch.references': [], 'assets.fetch.timeout': 300, 'assets.list.days': None, 'assets.list.overall_limit': None, 'assets.list.size_filter': None, 'assets.purge.days': None, 'assets.purge.overall_limit': None, 'assets.purge.size_filter': None, 'assets.register.name': None, 'assets.register.sha1_hash': None, 'assets.register.url': None, 'config.datadir': False, 'core.input_encoding': 'utf-8', 'core.paginator': False, 'core.show': {'app'}, 'core.verbose': False, 'datadir.paths.base_dir': '/root/avocado', 'datadir.paths.cache_dirs': ['/root/avocado/data/cache'], 'datadir.paths.data_dir': '/root/avocado/data', 'datadir.paths.logs_dir': '/mnt/local/results', 'datadir.paths.test_dir': '/usr/share/doc/avocado/tests', 'diff.create_reports': False, 'diff.filter': ['cmdline', 'time', 'variants', 'results', 'config', 'sysinfo'], 'diff.html': None, 'diff.jobids': [], 'diff.open_browser': False, 'diff.strip_id': False, 'distro.distro_def_arch': '', 'distro.distro_def_create': False, 'distro.distro_def_name': '', 'distro.distro_def_path': '', 'distro.distro_def_release': '', 'distro.distro_def_type': '', 'distro.distro_def_version': '', 'filter.by_tags.include_empty': False, 'filter.by_tags.include_empty_key': False, 'filter.by_tags.tags': [], 'i2n.common.suite_path': '/mnt/local/intra2net-qa/tp_i2n', 'i2n.manu.params': ['only=rpmlist', 'use_pool=no'], 'job.output.loglevel': 'DEBUG', 'job.output.testlogs.logfiles': ['debug.log'], 'job.output.testlogs.statuses': [], 'job.replay.source_job_id': 'latest', 'job.run.result.html.enabled': True, 'job.run.result.html.open_browser': False, 'job.run.result.html.output': None, 'job.run.result.json.enabled': True, 'job.run.result.json.output': None, 'job.run.result.tap.enabled': True, 'job.run.result.tap.include_logs': False, 'job.run.result.tap.output': None, 'job.run.result.xunit.enabled': True, 'job.run.result.xunit.job_name': None, 'job.run.result.xunit.max_test_log_chars': 100000, 'job.run.result.xunit.output': None, 'job.run.store_logging_stream': ['avocado.core:DEBUG'], 'job.run.timeout': 0, 'jobs.get.output_files.destination': None, 'jobs.get.output_files.job_id': None, 'jobs.show.job_id': 'latest', 'json.variants.load': None, 'list.auto': False, 'list.compatiblity_with_resolver_noop': True, 'list.external_runner': None, 'list.external_runner_chdir': None, 'list.external_runner_testdir': None, 'list.loaders': ['file', '@DEFAULT'], 'list.recipes.write_to_directory': None, 'list.references': [], 'list.resolver': True, 'list.write_to_json_file': None, 'nrunner.max_parallel_tasks': 1, 'nrunner.shuffle': False, 'nrunner.spawner': 'process', 'nrunner.status_server_auto': True, 'nrunner.status_server_buffer_size': 33554432, 'nrunner.status_server_listen': '127.0.0.1:8888', 'nrunner.status_server_uri': '127.0.0.1:8888', 'plugins.cli.cmd.order': [], 'plugins.cli.order': [], 'plugins.disable': [], 'plugins.init.order': [], 'plugins.job.prepost.order': [], 'plugins.jobscripts.post': '/etc/avocado/scripts/job/post.d/', 'plugins.jobscripts.pre': '/etc/avocado/scripts/job/pre.d/', 'plugins.jobscripts.warn_non_existing_dir': False, 'plugins.jobscripts.warn_non_zero_status': True, 'plugins.resolver.order': [], 'plugins.result.order': [], 'plugins.result_events.order': [], 'plugins.result_upload.cmd': "/usr/bin/rsync -arz -e 'ssh -oLogLevel=error -o stricthostkeychecking=no -o userknownhostsfile=/dev/null -o batchmode=yes -o passwordauthentication=no '", 'plugins.result_upload.url': None, 'plugins.resultsdb.api_url': None, 'plugins.resultsdb.logs_url': None, 'plugins.resultsdb.note_size_limit': 0, 'plugins.runnable.runner.order': [], 'plugins.runner.order': [], 'plugins.skip_broken_plugin_notification': [], 'plugins.spawner.order': [], 'plugins.varianter.order': [], 'plugins.vtjoblock.dir': '/tmp', 'run.auto': False, 'run.dict_variants': [], 'run.dry_run.enabled': False, 'run.dry_run.no_cleanup': False, 'run.execution_order': 'variants-per-test', 'run.external_runner': None, 'run.external_runner_chdir': 
None, 'run.external_runner_testdir': None, 'run.failfast': False, 'run.ignore_missing_references': False, 'run.job_category': None, 'run.journal.enabled': True, 'run.keep_tmp': False, 'run.loaders': ['file', '@DEFAULT'], 'run.log_test_data_directories': False, 'run.output_check': True, 'run.output_check_record': None, 'run.references': [], 'run.replay.ignore': [], 'run.replay.job_id': None, 'run.replay.resume': False, 'run.replay.test_status': [], 'run.results.archive': False, 'run.results_dir': None, 'run.test_parameters': [], 'run.test_runner': 'nrunner', 'run.unique_job_id': None, 'run.wrapper.wrappers': [], 'runner.exectest.exitcodes.skip': [], 'runner.output.color': 'always', 'runner.output.colored': True, 'runner.output.utf8': True, 'runner.timeout.after_interrupted': 60, 'runner.timeout.process_alive': 60, 'runner.timeout.process_died': 10, 'simpletests.status.failure_fields': ['status', 'stdout', 'stderr'], 'simpletests.status.skip_location': 'all', 'simpletests.status.skip_regex': '^SKIP$', 'simpletests.status.warn_location': 'all', 'simpletests.status.warn_regex': '^WARN$', 'spawner.lxc.arch': 'i386', 'spawner.lxc.distro': 'fedora', 'spawner.lxc.release': '32', 'spawner.podman.bin': '/usr/bin/podman', 'spawner.podman.image': 'fedora:33', 'sysinfo.collect.commands_timeout': -1, 'sysinfo.collect.enabled': True, 'sysinfo.collect.installed_packages': False, 'sysinfo.collect.locale': 'C', 'sysinfo.collect.optimize': False, 'sysinfo.collect.per_test': False, 'sysinfo.collect.profiler': False, 'sysinfo.collect.sysinfodir': None, 'sysinfo.collectibles.commands': '/etc/avocado/sysinfo/commands', 'sysinfo.collectibles.fail_commands': '/usr/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/fail_commands', 'sysinfo.collectibles.fail_files': '/usr/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/fail_files', 'sysinfo.collectibles.files': '/etc/avocado/sysinfo/files', 'sysinfo.collectibles.profilers': '/etc/avocado/sysinfo/profilers', 'task.timeout.running': None, 'variants.contents': False, 'variants.debug': False, 'variants.inherit': False, 'variants.json_variants_dump': None, 'variants.summary': 0, 'variants.tree': False, 'variants.variants': 1, 'vmimage.get.arch': None, 'vmimage.get.distro': None, 'vmimage.get.version': None, 'vt.common.arch': None, 'vt.common.data_dir': '', 'vt.common.machine_type': 'i440fx', 'vt.common.mem': None, 'vt.common.netdst': 'virbr0', 'vt.common.nettype': '', 'vt.common.tmp_dir': '/tmp', 'vt.common.type_specific_only': False, 'vt.config': None, 'vt.debug.no_cleanup': False, 'vt.extra_params': [], 'vt.filter.default_filters': None, 'vt.guest_os': 'JeOS.27', 'vt.libvirt.connect_uri': None, 'vt.list_archs': False, 'vt.list_guests': False, 'vt.list_guests.guest_os': None, 'vt.no_filter': '', 'vt.only_filter': '', 'vt.qemu.accel': 'kvm', 'vt.qemu.defconfig': 'yes', 'vt.qemu.disk_bus': 'virtio_blk', 'vt.qemu.image_type': 'qcow2', 'vt.qemu.malloc_perturb': 'yes', 'vt.qemu.monitor': '', 'vt.qemu.nic_model': 'virtio_net', 'vt.qemu.qemu_bin': None, 'vt.qemu.qemu_dst_bin': '/usr/bin/qemu-kvm', 'vt.qemu.sandbox': 'on', 'vt.qemu.smp': '2', 'vt.qemu.vhost': 'off', 'vt.save_config': None, 'vt.setup.backup_image_before_test': True, 'vt.setup.keep_guest_running': False, 'vt.setup.restore_image_after_test': True, 'vt.short_names_when_config': False, 'vt.type': 'qemu'}

# on a test level
[debug] 2021-09-15 14:03:05,245 shared_customize L0042 CRITI| <class 'avocado.core.settings.Settings'>
[debug] 2021-09-15 14:03:05,245 shared_customize L0043 CRITI| 139697070876032
[debug] 2021-09-15 14:03:05,245 shared_customize L0044 CRITI| {'core.input_encoding': 'utf-8', 'core.show': {'app'}, 'datadir.paths.base_dir': '/root/avocado', 'datadir.paths.cache_dirs': ['/root/avocado/data/cache'], 'datadir.paths.data_dir': '/root/avocado/data', 'datadir.paths.logs_dir': '/mnt/local/results', 'datadir.paths.test_dir': '/usr/share/doc/avocado/tests', 'i2n.common.suite_path': '/usr/lib/python3.9/site-packages/tp_folder', 'job.output.loglevel': 'DEBUG', 'job.output.testlogs.logfiles': ['debug.log'], 'job.output.testlogs.statuses': [], 'job.run.result.html.enabled': True, 'job.run.result.html.open_browser': False, 'job.run.result.html.output': None, 'job.run.result.json.enabled': True, 'job.run.result.json.output': None, 'job.run.result.tap.enabled': True, 'job.run.result.tap.include_logs': False, 'job.run.result.tap.output': None, 'job.run.result.xunit.enabled': True, 'job.run.result.xunit.job_name': None, 'job.run.result.xunit.max_test_log_chars': 100000, 'job.run.result.xunit.output': None, 'job.run.store_logging_stream': ['avocado.core:DEBUG'], 'job.run.timeout': 0, 'json.variants.load': None, 'nrunner.max_parallel_tasks': 1, 'nrunner.shuffle': False, 'nrunner.spawner': 'process', 'nrunner.status_server_auto': True, 'nrunner.status_server_buffer_size': 33554432, 'nrunner.status_server_listen': '127.0.0.1:8888', 'nrunner.status_server_uri': '127.0.0.1:8888', 'plugins.cli.cmd.order': [], 'plugins.cli.order': [], 'plugins.disable': [], 'plugins.init.order': [], 'plugins.job.prepost.order': [], 'plugins.jobscripts.post': '/etc/avocado/scripts/job/post.d/', 'plugins.jobscripts.pre': '/etc/avocado/scripts/job/pre.d/', 'plugins.jobscripts.warn_non_existing_dir': False, 'plugins.jobscripts.warn_non_zero_status': True, 'plugins.resolver.order': [], 'plugins.result.order': [], 'plugins.result_events.order': [], 'plugins.runnable.runner.order': [], 'plugins.runner.order': [], 'plugins.skip_broken_plugin_notification': [], 'plugins.spawner.order': [], 'plugins.varianter.order': [], 'plugins.vtjoblock.dir': '/tmp', 'run.dict_variants': [], 'run.execution_order': 'variants-per-test', 'run.references': [], 'runner.exectest.exitcodes.skip': [], 'runner.output.color': 'always', 'runner.output.colored': True, 'runner.output.utf8': True, 'runner.timeout.after_interrupted': 60, 'runner.timeout.process_alive': 60, 'runner.timeout.process_died': 10, 'simpletests.status.failure_fields': ['status', 'stdout', 'stderr'], 'simpletests.status.skip_location': 'all', 'simpletests.status.skip_regex': '^SKIP$', 'simpletests.status.warn_location': 'all', 'simpletests.status.warn_regex': '^WARN$', 'spawner.lxc.arch': 'i386', 'spawner.lxc.distro': 'fedora', 'spawner.lxc.release': '32', 'spawner.podman.bin': '/usr/bin/podman', 'spawner.podman.image': 'fedora:33', 'sysinfo.collect.commands_timeout': -1, 'sysinfo.collect.enabled': True, 'sysinfo.collect.installed_packages': False, 'sysinfo.collect.locale': 'C', 'sysinfo.collect.optimize': False, 'sysinfo.collect.per_test': False, 'sysinfo.collect.profiler': False, 'sysinfo.collectibles.commands': '/etc/avocado/sysinfo/commands', 'sysinfo.collectibles.fail_commands': '/usr/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/fail_commands', 'sysinfo.collectibles.fail_files': '/usr/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/fail_files', 'sysinfo.collectibles.files': '/etc/avocado/sysinfo/files', 'sysinfo.collectibles.profilers': '/etc/avocado/sysinfo/profilers', 'task.timeout.running': None, 'vt.common.arch': None, 'vt.common.data_dir': '', 'vt.common.machine_type': 'i440fx', 'vt.common.mem': None, 'vt.common.netdst': 'virbr0', 'vt.common.nettype': '', 'vt.common.tmp_dir': '/tmp', 'vt.common.type_specific_only': False, 'vt.config': None, 'vt.debug.no_cleanup': False, 'vt.extra_params': [], 'vt.filter.default_filters': None, 'vt.guest_os': 'JeOS.27', 'vt.libvirt.connect_uri': None, 'vt.list_archs': False, 'vt.list_guests': False, 'vt.no_filter': '', 'vt.only_filter': '', 'vt.qemu.accel': 'kvm', 'vt.qemu.defconfig': 'yes', 'vt.qemu.disk_bus': 'virtio_blk', 'vt.qemu.image_type'
: 'qcow2', 'vt.qemu.malloc_perturb': 'yes', 'vt.qemu.monitor': '', 'vt.qemu.nic_model': 'virtio_net', 'vt.qemu.qemu_bin': None, 'vt.qemu.qemu_dst_bin': '/bin/qemu-kvm', 'vt.qemu.sandbox': 'on', 'vt.qemu.smp': '2', 'vt.qemu.vhost': 'off', 'vt.save_config': None, 'vt.setup.backup_image_before_test': True, 'vt.setup.keep_guest_running': False, 'vt.setup.restore_image_after_test': True, 'vt.short_names_when_config': False, 'vt.type': 'qemu'}

So my question is rather is this anticipated? Should we avoid using a parameter from avocado config in tests? Or is this an actual issue to report?

[beraldoleal]

@pevogam before I give a shot here, can you please share how you are getting the settings in your test side? By reading your comments, looks like you are importing from settings and using the .as_dict() method. Is that right?

[pevogam]

Yes, that's exactly right.

from avocado.core.settings import settings
...
testsuite_top_path = settings.as_dict().get('i2n.common.suite_path')

[beraldoleal]

So, AFAICT, settings was not designed to be delivered all way to the tests. Specially when importing, because we have to assume that tests can be executed in a remote/isolated environment where all the Avocado logic it was not started, and consequently the settings that were overwritten during the Avocado bootstrap will be not there. But I might be forgetting about something here.

Everything will be ending at a very simple runner command, like:

$ avocado-runner-exec-test task-run -i foo -j 42 -u /bin/sleep -a 10

We do have a -c (config) option for the runners, and in theory we could be passing the entire dict here too. Requiring a serialization/de serialization here. But again, AFAICT, we are not doing this and I think this config it was designed for something else.

I guess the best way to do this is passing as test parameters (-p), where we do have a logic to send those to the different types of runners (python or simple tests via environment variables).

[clebergnu]

TL;DR: as it stands, configuration is for runners, parameters is for tests. Maybe we're missing something here, but this clear separation is on purpose and it helps, for instance, with reproducible tests.

[pevogam]

Thanks, this clears things up and even provides some clearer separation between the Avocado config and the Cartesian config in Avocado VT.