Unexpected and divergent behavior about secrets when using Sync Device or Sync Network data (from network) jobs that come with Device Onboarding

[itheodoridis]

Environment

• Nautobot version (Docker tag too if applicable): 3.0.3 (ghcr.io/nautobot/nautobot:$3.0.3-py3.12)
• Python version:3.12
• Database platform, version: Postres, 14.5 (docker)
• Middleware(s): (Not sure what this is).

When running the Sync Device from Network job (part of Device Onboarding) the job expects that the secrets group used is of Generic Access Type or the job will fail.
At the same time when running the Sync Network data from Network job (again part of Device Onboarding) against the same device, unless the secrets group defines that the secrets are of SSH Access Type, the job fails. Even the secrets themselves are the same, just different secret groups so we can define the Access Type separately per use case.
The target devices are cisco access switches and routers (like Catalyst 2960 series and ISR 4k series).

Steps to Reproduce

1. Create the secrets groups (by any means necessary - for me secrets come from environment variables defined for the docker containers and imported in nautobot via config - assign access type as generic for one group and ssh for the other.
2. Run the two sync jobs with both kinds of secrets groups (Sync Devices allows for choosing the secrets group before running, so defining the other as default to use with Sync Network Data somehow can do the trick, or just change between them and define what you want for each run). Use debug option for running the jobs.
3. Observe results and failure logs.

Expected Behavior

Both jobs should have the same behavior with the same secrets group (access type) with logging and accessing devices successfully instead of failing.

Observed Behavior

They don't have the same behavior as observed by the logs and debug output, plus obviously the jobs fail as they don't succeed in accessing the devices so no onboarding/updating occurs.

[glennmatthews]

Forwarding to device-onboarding repo

[nkallergis]

@itheodoridis could you please share the failing job's traceback? Just used the "Sync Network Data" job this morning and it was with Generic type 🤷 I'll try to replicate your issue in the meantime but a traceback would help understand what exactly you're facing.

Thanks!

[itheodoridis]

Forwarding to device-onboarding repo

Hey Glenn, not sure this is a device on boarding issue but I am certainly not the one to know. I will follow your lead.

[itheodoridis]

@itheodoridis could you please share the failing job's traceback? Just used the "Sync Network Data" job this morning and it was with Generic type 🤷 I'll try to replicate your issue in the meantime but a traceback would help understand what exactly you're facing.

Thanks!

Hey Niko, thank you for picking this up. I will do that in a few minutes and get back to you.

[nkallergis]

In the meantime, is there any chance you've "hardcoded" the access_type to SSH in a Config Context?

Screenshot taken from here

[itheodoridis]

Well I would have said no but maybe you are right depending on the job used and the underlying platform used for those. I remember troubleshooting Golden Config and we got into this with Jeff (Kala) I think so creating a config context was part of the solution. So there is this, applied to most device roles:
{ "nautobot_plugin_nornir": { "secret_access_type": "SSH" } }
And then there's another intervention in the admin part for the configuration where for the golden configuration I have this for default framework:
{"all": "napalm", "cisco_wlc": "netmiko"}
and this for get config framework:
{"napalm": {"cisco_nxos": "nxos_ssh"}, "netmiko": {"cisco_wlc": "cisco_wlc"}}
Those were necessary especially for NXOS devices to work with Golden Config but the last part is a result of our conversations for Cisco WLC.
Do you believe this is relevant? I know it seems relevant up to a point (the device context mostly) but not sure it applies if napalm is used for one of those jobs instead of Nornir/Netmiko.

[nkallergis]

Yes, the nautobot_plugin_nornir part is the common dispatcher used for both Golden Config (GC) and Device Onboarding (DO); the other parts are irrelevant. Not 100% sure how it relates to NXOS but pretty confident that's causing your issue. Keep in mind that if you change it there, you also have to change your SecretsGroup definitions to match, both the ones used for GC and for DO.

[itheodoridis]

Not sure I understand that fully yet. Let me run some more test and I will get back to you.

[itheodoridis]

I failed to mention before that I have setup the secrets group for the device as well (on all devices) as the ones based on generic access (you know how that goes, 'tick all boxes and let's try again').
ok, with current setup I am trying to run the sync Device from Network on a 2960 switch on a branch, using the ssh secrets group. That fails at the start since it can't parse the secrets correctly for the username and password fields. Here is the relevant messages, redacted for the ip address.

2026-04-16 19:12:04.075792 | main | Debug | — | Missing credentials for ['username', 'password']
2026-04-16 19:12:04.091089 | main | Error | — | Unable to onboard x.y.z.w failed to parse credentials

Subsequent commands fail to produce relevant output of course but the job doesn't stop.

If I define the secret group as the one based on generic access, the parsing works:

2026-04-16 19:14:20.258903 | main | Info | — | Parsing credentials from Secrets (redacted) device secrets
2026-04-16 19:14:20.322958 | main | Debug | — | Commands to run: ['show version', 'show interfaces']
2026-04-16 19:14:20.366773 | main | Info | — | Subtask starting: show version, x.y.z.w.

Subsequent commands work.

Now if I try the Sync Network Data job, there is no option for defining a secrets group for the job. The job works and commands produce output.

2026-04-16 19:21:25.913894 | run | Info | — | 1 devices will be synced
2026-04-16 19:21:25.922330 | run | Info | — | Devices: ['CreteBranchSwitch']
2026-04-16 19:21:25.955930 | sync_data | Info | — | Loading current data from source adapter...
2026-04-16 19:21:28.027156 | main | Debug | — | Commands to run: ['show version', 'show interfaces', 'show interfaces switchport', 'show etherchannel summary']
2026-04-16 19:21:28.057164 | main | Info | — | Subtask starting: show version, CreteBranchSwitch
2026-04-16 19:21:30.318821 | main | Info | — | Subtask succeeded: show version, CreteBranchSwitch.

Subsequent commands produce output.

Now if I go to the device and choose the secret group that is based on ssh access, the job fails but the problem comes from a python related crash, I believe related to password parsing again but for the nornir inventory.

2026-04-16 19:24:57.881038 | run | Debug | — | Checking for last_network_data_sync custom field
2026-04-16 19:24:57.905331 | run | Info | — | 1 devices will be synced
2026-04-16 19:24:57.913674 | run | Info | — | Devices: ['CreteBranchSwitch']
2026-04-16 19:24:57.937663 | sync_data | Info | — | Loading current data from source adapter...
2026-04-16 19:24:58.583521 | main | Info | — | Error During  Sync Network Data Command Getter: Traceback (most recent call last):   File  "/usr/local/lib/python3.12/site-packages/nautobot_device_onboarding/nornir_plays/command_getter.py",  line 347, in sync_network_data_command_getter     with InitNornir(          ^^^^^^^^^^^   File "/usr/local/lib/python3.12/site-packages/nornir/init_nornir.py",  line 71, in InitNornir     inventory=load_inventory(config),               ^^^^^^^^^^^^^^^^^^^^^^   File "/usr/local/lib/python3.12/site-packages/nornir/init_nornir.py",  line 20, in load_inventory     inv = inventory_plugin(**config.inventory.options).load()           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File  "/usr/local/lib/python3.12/site-packages/nautobot_plugin_nornir/plugins/inventory/nautobot_orm.py",  line 204, in load     host = self.create_host(device=device, cred=cred,  params=self.params)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File  "/usr/local/lib/python3.12/site-packages/nautobot_plugin_nornir/plugins/inventory/nautobot_orm.py",  line 262, in create_host     username, password, secret (redacted)  cred.get_device_creds(device=device)  # pylint:disable=unused-variable                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File  "/usr/local/lib/python3.12/site-packages/nautobot_plugin_nornir/plugins/credentials/nautobot_secrets.py",  line 165, in get_device_creds     self.secret (redacted) self.password                   (redacted) AttributeError: 'CredentialsNautobotSecrets' (redacted) has no attribute  'password'

If I remove the secrets group entirely from the device properties, then the Sync Device job takes a long time but works eventually when choosing the generic access type secrets group and fails the same way as before for the ssh access type secrets group (failure at parsing).
I will now remove the added config context from the DR instance and try those tests again and then remove also the admin added part and do the tests again and get back to you.
But that will take a while as I need to do something in between.

[nkallergis]

Took the liberty to reformat your message a bit for better readability. Sorry for not asking! 🙂

OK, let me clarify what is the expectation and what your final setup should look like (from a DO standpoint).

• Sync Device from Network: Before you execute this Job, the Devices generally don't exist yet. As such, you have to provide a SecretsGroup to be used for connecting to them. The access_type has to be "Generic" for this to work. Once this Job completes, you have Devices in your Nautobot.
• Sync Network Data from Network: Since the Devices already exist in Nautobot prior to executing this Job, the expectation is that they also have their own SecretsGroup already defined. By default (i.e. if you don't override using a Config Context), the access_type should still be "Generic".

Based on your last message, it looks like everything worked as expected when:

• you removed the Config Context override
• you used "Generic" access_type for the first Job
• your Devices had SecretsGroup defined with access_type "Generic" before you ran the second Job

Does that help?

[itheodoridis]

Thanks for the edit, I was wondering how it got corrected, you are right it was an ugly mess before (you never need to ask).
I understand how it is expected for the Sync Device job that the device doesn't exist (it didn't exist the first time, I promise , but after that I didn't want to keep deleting it on the prod instance to test things, I will do that in the DR). Thank you for the explanation of that expectation that clears it up as an intention in my mind (where to use things and for what purpose).
So removing the config context override didn't happen yet. I just removed the setting inside the device properties (by editing the device object) where I had also chosen a specific secrets group (I guess that was also an override and I removed it but I am starting to get confused with the terminology).
I have not yet removed the config context created for all those device roles. I intend to do that in the DR instance in a while (once I get home, first thing).
The rest of your assumptions/conclusions seem to be correct.
It certainly does a bit. I will continue with tests and get back to you again (no need for you to keep at it for now, we are on the same timezone, nothing urgent or life critical thing going on here).
Thanks for the help so far.

[itheodoridis]

ok so trying things in the DR instance after a sync. For each set of test I am doing a resync with prod so settings are reset to what I described earlier.

1. Deleting the config context results in the following:
   I deleted the device itself as well and tried to run the Sync Device from Network job in order to reintroduce it.

• with generic based access, the task fails at parsing credentials and that is followed by an exception for ssh with paramiko
• with ssh based access it succeeds but takes a long time and at start seems to be timing out.
  Here is the exception:

2026-04-17 00:59:57.798876 | main | Info | — | Parsing credentials from Secrets (redacted) napalm secrets
2026-04-17 00:59:57.802020 | main | Debug | — | Missing credentials for ['username', 'password']
2026-04-17 00:59:57.803515 | main | Error | — | Unable to onboard x.y.z.w, failed to parse credentials
2026-04-17 00:59:58.525937 | main | Error | — | Unable to onboard x.y.z.w, failed with exception  A paramiko SSHException occurred during connection creation: No authentication methods available

If I uncheck the dry run, let the device be added to Nautobot (it's added with ssh based secrets as a parameter in the device object), this is what happens with the Sync Network Data from Network job:

• with the ssh based secrets, it fails like before (the same long python exception for the Nornir inventory because of the password).
• if I choose to edit the device and replace the secrets group with generic based secrets, then surprise, I get the same exact failure!! So perhaps the override being removed ended up forcing the two choices down the same path.

2. for the second set, I restore the config with a db sync and then removed the admin extra config for

• device connectivity, network drivers:

{"ntc_templates": {"cisco_xe": "cisco_ios"}, "napalm": {"cisco_nxos": "nxos_ssh"}}

• Golden config default framework:

{"all": "napalm", "cisco_wlc": "netmiko"}

• Golden config config framework:

{"napalm": {"cisco_nxos": "nxos_ssh"}, "netmiko": {"cisco_wlc": "cisco_wlc"}}

Also deleted device to be discovered and config context:

{
    "nautobot_plugin_nornir": {
        "secret_access_type": "SSH"
    }
}

Here are the results for the jobs.
for the Sync Device job:

• for generic based access, paramiko ssh exception
• for ssh based access, it takes a long time and seems to timeout but finally succeeds.
  If I let the device be added with the ssh based secrets, then the sync network data job
• fails if the secrets group remains unchanged with the nornir inventory exception for the secrets
• if edited to generic based secrets fails again with same exception.
  So it seems to me that either those interventions (all or some) were made to make this work as it doesn't seem to work with the default settings, or that perhaps I am carrying a db for much too long (since the netbox days) and perhaps I need to recreate the db and sync over any information. But somehow that seems too drastic and perhaps unnecessary.
  Or something else still eludes me.