Puppet Class Importer fails to update parameters: undefined method `override' for nil:NilClass

[SebbiUltimate]

When I try to import the puppet environments, I get the error "undefined method `override' for nil:NilClass".
Beginning of Backtrace:

2024-03-02T00:21:03 [I|app|92f6450f] Backtrace for 'Failed to calculate obsolete and new' error (NoMethodError): undefined method override' for nil:NilClass
92f6450f | /usr/share/gems/gems/foreman_puppet-5.0.0/app/services/foreman_puppet/puppet_class_importer.rb:277:in block in update_parameter'
92f6450f | /usr/share/gems/gems/foreman_puppet-5.0.0/app/services/foreman_puppet/puppet_class_importer.rb:275:in `each'
92f6450f | /usr/share/gems/gems/foreman_puppet-5.0.0/app/services/foreman_puppet/puppet_class_importer.rb:275:in update_parameter'
92f6450f | /usr/share/gems/gems/foreman_puppet-5.0.0/app/services/foreman_puppet/puppet_class_importer.rb:270:in block in update_classes_in_foreman'

It seems that the parameters cannot be found in the lookup_keys table even though they exist.

When I replace the line
key = db_class.class_params.find_by key: param_name
in puppet_class_importer.rb L275 with
key = db_class.class_params.where("key like ?", param_name)
it works.

[nadjaheitmann]

Hi @SebbiUltimate, I cannot reproduce it, can you please describe more detailed the configuration of your system and how you tried to import the environment?

[SebbiUltimate]

I think it happens when the default values of puppet class parameters change. So the parameter already exists in the lookup_keys table but has another default value. Then this function update_parameter gets called but doesn't find the record in the table. After changing this line, it works.

[stejskalleos]

@SebbiUltimate Can you please share the exact steps to reproduce with example classes you import?
Without it, it's hard to debug the issue.

[SebbiUltimate]

I couldn't reproduce it using a new puppet sample class.

But when I update classes like keystone::ldap, it get the error.

I added more logging:

 def update_parameter(db_class, changed_params)
      changed_params['updated'].each do |param_name, value|
        key = db_class.class_params.find_by key: param_name
        db_class.class_params.map do |dbcp|
                Foreman::Logging.logger('app').info "dbcp.key=#{dbcp.key}!"
        end
        Foreman::Logging.logger('app').info "update_parameter: key not found! class=#{db_class.name}! param_name=#{param_name}!" if key.nil?
        next unless key.override == false
        key.default_value = value
        key.key_type = nil
        key.validator_type = nil
        key.save!(context: :importer)
      end
    end

As you can see in the output, the parameter group_filter ist existing, but find_by cannot find it.

2024-03-09T18:29:06 [I|app|6bc7d633] Processing by ForemanPuppet::PuppetclassesController#obsolete_and_new as HTML
2024-03-09T18:29:06 [I|app|6bc7d633] Parameters: {"authenticity_token"=>"...", "changed"=>{"updated"=>{"production"=>"{"apache::mod::passenger":{"obsolete":["rails_autodetect","rails_auto_detect","rack_auto_detect"]},"keystone::ldap":{"updated":{"group_filter":"${::os_service_default}","user_additional_attribute_mapping":"${::os_service_default}","user_enabled_emulation_dn":"${::os_service_default}","user_enabled_emulation":"${::os_service_default}","user_pass_attribute":"${::os_service_default}","user_default_project_id_attribute":"${::os_service_default}","user_attribute_ignore":"${::os_service_default}","user_enabled_invert":"${::os_service_default}","user_enabled_default":"${::os_service_default}","user_enabled_mask":"${::os_service_default}","user_enabled_attribute":"${::os_service_default}","user_mail_attribute":"${::os_service_default}","user_description_attribute":"${::os_service_default}","user_id_attribute":"${::os_service_default}","user_filter":"${::os_service_default}","page_size":"${::os_service_default}","query_scope":"${::os_service_default}"}},"keystone::policy":{"updated":{"policy_path":"/etc/keystone/policy.yaml"}},"keystone::wsgi::apache":{"updated":{"ssl":"undef"}},"magnum::clients::glance":{"updated":{"api_version":"${::os_service_default}"}},"magnum::clients::heat":{"updated":{"api_version":"${::os_service_default}"}},"magnum::clients::nova":{"updated":{"api_version":"${::os_service_default}"}},"magnum::policy":{"updated":{"policy_path":"/etc/magnum/policy.yaml"}},"murano":{"updated":{"identity_uri":"undef","memcached_servers":"undef","project_domain_name":"undef","user_domain_name":"undef","www_authenticate_uri":"undef","admin_tenant_name":"undef","admin_user":"undef"}},"murano::db_cfapi":{"updated":{"database_idle_timeout":"undef"}},"murano::policy":{"updated":{"policy_path":"/etc/murano/policy.yaml"}},"neutron":{"updated":{"control_exchange":"${::os_service_default}","allow_overlapping_ips":"undef"}},"neutron::agents::l3":{"updated":{"gateway_external_network_id":"undef"}},"neutron::agents::ml2::networking_baremetal":{"updated":{"max_retries":"undef","retry_interval":"undef","ironic_url":"undef","auth_strategy":"undef"}},"neutron::agents::ovn_metadata":{"updated":{"ovn_remote_probe_interval":"undef"}},"neutron::designate":{"updated":{"project_id":"undef"}},"neutron::keystone::auth":{"updated":{"service_description":"OpenStack Networking Service"}},"neutron::plugins::ml2::mellanox::mlnx_sdn_assist":{"updated":{"sdn_password":"undef","sdn_username":"undef"}},"neutron::plugins::ml2::ovn":{"updated":{"vif_type":"undef","ovn_l3_mode":"undef"}},"neutron::policy":{"updated":{"policy_path":"/etc/neutron/policy.yaml"}},"neutron::quota":{"updated":{"quota_vip":"undef","quota_packet_filter":"undef","quota_network_gateway":"undef"}},"neutron::server":{"updated":{"validate":"undef"}},"neutron::server::placement":{"updated":{"username":"placement"}},"neutron::wsgi::apache":{"updated":{"ssl":"undef"}},"nova":{"updated":{"cross_az_attach":"undef","upgrade_level_console":"undef","cinder_catalog_info":"undef","os_region_name":"undef","block_device_allocate_retries_interval":"undef","block_device_allocate_retries":"undef"}},"nova::api":{"updated":{"install_cinder_client":"undef","instance_name_template":"${::os_service_default}","validation_options":"undef","validate":"undef","ratelimits_factory":"undef"}},"nova::cinder":{"updated":{"user_domain_name":"undef","username":"undef","project_domain_name":"undef","project_name":"undef","region_name":"undef","auth_url":"undef","auth_type":"undef"}},"nova::compute":{"updated":{"keymgr_backend":"undef","barbican_api_version":"undef","barbican_endpoint":"undef","barbican_auth_endpoint":"undef","heal_instance_info_cache_interval":"${::os_service_default}","reserved_host_memory":"${::os_service_default}","force_raw_images":"${::os_service_default}"}},"nova::compute::libvirt":{"updated":{"tls_priority":"${::os_service_default}","log_filters":"${::os_service_default}","log_outputs":"${::os_service_default}"}},"nova::compute::libvirt::virtlogd":{"updated":{"max_backups":"${::os_service_default}","max_size":"${::os_service_default}","admin_max_clients":"${::os_service_default}","max_clients":"${::os_service_default}","log_outputs":"${::os_service_default}","log_filters":"${::os_service_default}","log_level":"${::os_service_default}"}},"nova::compute::vmware":{"updated":{"use_linked_clone":"${::os_service_default}","task_poll_interval":"${::os_service_default}","maximum_objects":"${::os_service_default}","api_retry_count":"${::os_service_default}"}},"nova::cron::archive_deleted_rows":{"updated":{"user":"${::nova::params::user}"}},"nova::cron::purge_shadow_tables":{"updated":{"user":"${::nova::params::user}"}},"nova::keystone::auth":{"updated":{"service_description":"OpenStack Compute Service"}},"nova::metadata::novajoin::policy":{"updated":{"policy_path":"/etc/novajoin/policy.yaml"}},"nova::migration::libvirt":{"updated":{"crl_file":"${::os_service_default}","listen_address":"${::os_service_default}"}},"nova::policy":{"updated":{"policy_path":"/etc/nova/policy.yaml"}},"nova::quota":{"updated":{"max_age":"undef","until_refresh":"undef","reservation_expire":"undef","server_group_members":"${::os_service_default}","server_groups":"${::os_service_default}","key_pairs":"${::os_service_default}","security_group_rules":"undef","security_groups":"undef","injected_file_path_length":"${::os_service_default}","injected_file_content_bytes":"${::os_service_default}","injected_files":"${::os_service_default}","metadata_items":"${::os_service_default}","fixed_ips":"undef","floating_ips":"undef","ram":"${::os_service_default}","cores":"${::os_service_default}","instances":"${::os_service_default}"}},"nova::scheduler::filter":{"updated":{"max_instances_per_host":"${::os_service_default}","max_io_ops_per_host":"${::os_service_default}","scheduler_host_subset_size":"${::os_service_default}"}},"nova::wsgi::apache_api":{"updated":{"ssl":"undef"}},"nova::wsgi::apache_metadata":{"updated":{"ssl":"undef"}},"openstack_extras::repo::redhat::redhat":{"updated":{"manage_priorities":"undef","release":"${openstack_extras::repo::redhat::params::release}"}},"placement::api":{"updated":{"port":"undef","host":"undef"}},"placement::keystone::auth":{"updated":{"internal_url":"http://127.0.0.1:8778\",\"admin_url\":\"http://127.0.0.1:8778\",\"public_url\":\"http://127.0.0.1:8778\",\"service_description\":\"OpenStack Placement Service"}},"placement::policy":{"updated":{"policy_path":"/etc/placement/policy.yaml"}},"placement::wsgi::apache":{"updated":{"ensure_package":"undef","threads":1,"workers":"${::os_workers}","ssl":"undef","path":"/","api_port":8778}},"sssd":{"obsolete":["config_file"]},"swift::keystone::auth":{"updated":{"service_description_s3":"OpenStack S3 Service","service_description":"OpenStack Object-Store Service","password":"undef"}},"swift::objectexpirer":{"updated":{"pipeline":["catch_errors","proxy-logging","cache","proxy-server"]}},"swift::proxy":{"updated":{"pipeline":["catch_errors","gatekeeper","healthcheck","proxy-logging","cache","listing_formats","tempauth","copy","proxy-logging","proxy-server"]}},"swift::proxy::authtoken":{"updated":{"password":"undef","auth_plugin":"undef","signing_dir":"undef"}},"swift::proxy::bulk":{"updated":{"yield_frequency":"10"}},"swift::proxy::ceilometer":{"updated":{"password":"undef"}},"swift::proxy::dlo":{"updated":{"max_get_time":"${::os_service_default}","rate_limit_segments_per_sec":"${::os_service_default}","rate_limit_after_segment":"${::os_service_default}"}},"swift::proxy::s3api":{"updated":{"max_upload_part_num":"${::os_service_default}","ensure":"undef"}},"swift::proxy::slo":{"updated":{"max_get_time":"${::os_service_default}","rate_limit_segments_per_sec":"${::os_service_default}","rate_limit_after_segment":"${::os_service_default}","min_segment_size":"undef","max_manifest_size":"${::os_service_default}","max_manifest_segments":"${::os_service_default}"}},"swift::proxy::tempurl":{"updated":{"outgoing_allow_headers":"${::os_service_default}","outgoing_remove_headers":"${::os_service_default}","incoming_allow_headers":"${::os_service_default}","incoming_remove_headers":"${::os_service_default}","methods":"${::os_service_default}"}},"swift::storage::all":{"updated":{"allow_versions":"undef","account_port":6002,"container_port":6001,"object_port":6000}},"vswitch::ovs":{"updated":{"dkms_ensure":"undef"}}}"}}, "commit"=>"Update"}
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=assignment_driver!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=auth_pool_connection_lifetime!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=auth_pool_size!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=chase_referrals!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=credential_driver!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_additional_attribute_mapping!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_attribute_ignore!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_desc_attribute!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_filter!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_id_attribute!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_member_attribute!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_members_are_ids!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_name_attribute!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_objectclass!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=group_tree_dn!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=identity_driver!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=manage_packages!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=package_ensure!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=page_size!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=password!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=pool_connection_lifetime!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=pool_connection_timeout!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=pool_retry_delay!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=pool_retry_max!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=pool_size!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=query_scope!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=suffix!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=tls_cacertdir!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=tls_cacertfile!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=tls_req_cert!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=url!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=use_auth_pool!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=use_pool!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_additional_attribute_mapping!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_attribute_ignore!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_default_project_id_attribute!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_description_attribute!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_enabled_attribute!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_enabled_default!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_enabled_emulation!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_enabled_emulation_dn!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_enabled_invert!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_enabled_mask!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_filter!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_id_attribute!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_mail_attribute!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_name_attribute!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_objectclass!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_pass_attribute!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=user_tree_dn!
2024-03-09T18:29:06 [I|app|6bc7d633] dbcp.key=use_tls!
2024-03-09T18:29:06 [I|app|6bc7d633] update_parameter: key not found! class=keystone::ldap! param_name=group_filter!
2024-03-09T18:29:06 [W|app|6bc7d633] Failed to calculate obsolete and new
2024-03-09T18:29:06 [I|app|6bc7d633] Backtrace for 'Failed to calculate obsolete and new' error (NoMethodError): undefined method override' for nil:NilClass 6bc7d633 | /usr/share/gems/gems/foreman_puppet-6.2.0/app/services/foreman_puppet/puppet_class_importer.rb:280:in block in update_parameter'
6bc7d633 | /usr/share/gems/gems/foreman_puppet-6.2.0/app/services/foreman_puppet/puppet_class_importer.rb:274:in each' 6bc7d633 | /usr/share/gems/gems/foreman_puppet-6.2.0/app/services/foreman_puppet/puppet_class_importer.rb:274:in update_parameter'
.....

It is also in the database:

foreman=> SELECT p.name,l.key,e.id,p.id,l.id FROM environment_classes AS e JOIN puppetclasses AS p ON e.puppetclass_id=p.id JOIN lookup_keys AS l ON e.puppetclass_lookup_key_id=l.id WHERE p.name LIKE 'keystone::ldap' AND l.key LIKE 'group_filter';
name | key | id | id | id
----------------+--------------+------+-----+------
keystone::ldap | group_filter | 4714 | 539 | 4123
(1 row)

[SebbiUltimate]

Finally I found the root cause.
My database was corrupted, and this ruby code in the foreman rake console made it visible:

User.current = User.unscoped.where(admin: true).first

changes = ForemanPuppet::PuppetClassImporter.new.changes
affected = []

(changes["updated"] || {}).each do |environment_name, classes|
  environment =
    ForemanPuppet::Environment.unscoped.find_by(name: environment_name)

  next unless environment

  classes.each do |class_name, class_changes|
    updated_parameters = class_changes["updated"] || {}
    next if updated_parameters.empty?

    puppetclass =
      ForemanPuppet::Puppetclass.unscoped.find_by(name: class_name)

    next unless puppetclass

    updated_parameters.each_key do |parameter_name|
      puppetclass.association(:class_params).reset

      found_parameter =
        puppetclass.class_params.find_by(key: parameter_name)

      next if found_parameter

      sql = ActiveRecord::Base.send(
        :sanitize_sql_array,
        [
          <<~SQL,
            SELECT
              ec.id AS environment_class_id,
              e.name AS environment,
              ec.puppetclass_id,
              pc.name AS puppetclass,
              ec.puppetclass_lookup_key_id,
              lk.id AS lookup_key_id,
              lk.type AS lookup_key_type,
              lk.key,
              lk.override
            FROM environment_classes ec
            LEFT JOIN environments e
              ON e.id = ec.environment_id
            LEFT JOIN puppetclasses pc
              ON pc.id = ec.puppetclass_id
            LEFT JOIN lookup_keys lk
              ON lk.id = ec.puppetclass_lookup_key_id
            WHERE ec.puppetclass_id = ?
              AND lk.key = ?
            ORDER BY e.name, ec.id
          SQL
          puppetclass.id,
          parameter_name
        ]
      )

      raw_links =
        ActiveRecord::Base.connection.select_all(sql).to_a

      affected << {
        import_environment: environment_name,
        puppetclass_id: puppetclass.id,
        puppetclass: class_name,
        missing_parameter: parameter_name,
        incoming_default: updated_parameters[parameter_name],
        raw_database_links: raw_links
      }
    end
  end
end

pp affected

r = affected.first

klass = ForemanPuppet::Puppetclass.unscoped.find(r[:puppetclass_id])
parameter_name = r[:missing_parameter].to_s

klass.association(:class_params).reset
normal = klass.class_params.where(key: parameter_name).pluck(:id, :key)

sequential = ActiveRecord::Base.connection.transaction do
  conn = ActiveRecord::Base.connection

  conn.execute("SET LOCAL enable_indexscan = off")
  conn.execute("SET LOCAL enable_indexonlyscan = off")
  conn.execute("SET LOCAL enable_bitmapscan = off")

  klass.association(:class_params).reset
  klass.class_params.where(key: parameter_name).pluck(:id, :key)
end

pp parameter: parameter_name,
   normal_indexed_query: normal,
   forced_sequential_query: sequential

The output was:
{:parameter=>"keyfile",
:normal_indexed_query=>[],
:forced_sequential_query=>[[2585, "keyfile"]]}

This meant that the DB index was corrupt.
I fixed it by running this on the pgsql commandline:
REINDEX INDEX public.index_lookup_keys_on_key;