[Inband ZTP] When reboot takes place during ZTP process, interface connected to provision server is not getting IP address from DHCP on init

[liorghub]

Description

When reboot takes place during ZTP process, interface connected to provision server is not getting IP address from DHCP server on init.

Issue is in the flow of continue processing addition configuration tasks after reboot that took place as part of handling former configuration task.

ZTP engine starts DHCP on the inband interfaces but the interface that is connected to provision server is still down (link to snippet).

Discussed solutions with @rajendra-dendukuri:

1. Download the data required for all the configuration tasks before processing the first task.
2. Solution 1 will not be sufficient if there is a configuration task that requires connectivity, therefore we can add to connectivity check an option to restart DHCP if connectivity check is failing and user will add it to his ZTP JSON.

Steps to reproduce the issue:

1. Single switch connected to a server using an inband port.
2. Run inband ZTP session, use the following ZTP JSON:

{
  "ztp": {
    "01-conf-task": {
      "plugin": {
        "url":"http://30.0.0.100:8000/data/return_success.sh"
      },
      "reboot-on-success": true
    },
     "02-conf-task_fail": {
      "plugin": {
        "url":"http://30.0.0.100:8000/data/return_success.sh"
      },
      "reboot-on-failure": false,
      "ignore-result": true
    },
    "restart-ztp-no-config": false
  }
}

return_success.sh:

#!/bin/sh
echo "return success"
exit 0

Describe the results you received:

Interface connected to provision server not getting ip from DHCP server.

Describe the results you expected:

Interface connected to provision server gets ip from DHCP server.

Output of show version:

SONiC Software Version: SONiC.inband_ztp_2.3-9f32f66ea_Internal
Distribution: Debian 11.4
Kernel: 5.10.0-12-2-amd64
Build commit: 9f32f66ea
Build date: Tue Aug  9 13:16:01 UTC 2022
Built by: sw-r2d2-bot@r-build-sonic-ci03-241

Platform: x86_64-mlnx_msn3800-r0
HwSKU: ACS-MSN3800
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1937X00537
Model Number: MSN3800-CS2FO
Hardware Revision: A1
Uptime: 12:06:38 up  5:17,  1 user,  load average: 1.27, 1.11, 0.87
Date: Thu 01 Sep 2022 12:06:38

Docker images:
REPOSITORY                    TAG                                 IMAGE ID       SIZE
docker-platform-monitor       inband_ztp_2.3-9f32f66ea_Internal   e8e002790e14   988MB
docker-platform-monitor       latest                              e8e002790e14   988MB
docker-syncd-mlnx             inband_ztp_2.3-9f32f66ea_Internal   5d9fbbf79cb1   985MB
docker-syncd-mlnx             latest                              5d9fbbf79cb1   985MB
docker-orchagent              inband_ztp_2.3-9f32f66ea_Internal   b6a69f6746a7   479MB
docker-orchagent              latest                              b6a69f6746a7   479MB
docker-macsec                 latest                              230f9a8c1ace   462MB
docker-database               inband_ztp_2.3-9f32f66ea_Internal   3b5ce1c0c24e   444MB
docker-database               latest                              3b5ce1c0c24e   444MB
docker-mux                    inband_ztp_2.3-9f32f66ea_Internal   c5a1ed2eeacd   493MB
docker-mux                    latest                              c5a1ed2eeacd   493MB
docker-sonic-telemetry        inband_ztp_2.3-9f32f66ea_Internal   7caaa66c7a98   525MB
docker-sonic-telemetry        latest                              7caaa66c7a98   525MB
docker-dhcp-relay             latest                              578e055f6b73   454MB
docker-router-advertiser      inband_ztp_2.3-9f32f66ea_Internal   3a810ffe0ae8   444MB
docker-router-advertiser      latest                              3a810ffe0ae8   444MB
docker-snmp                   inband_ztp_2.3-9f32f66ea_Internal   5c18285beb0f   457MB
docker-snmp                   latest                              5c18285beb0f   457MB
docker-fpm-frr                inband_ztp_2.3-9f32f66ea_Internal   5d607a4017af   458MB
docker-fpm-frr                latest                              5d607a4017af   458MB
docker-sonic-mgmt-framework   inband_ztp_2.3-9f32f66ea_Internal   a03108ef0b40   558MB
docker-sonic-mgmt-framework   latest                              a03108ef0b40   558MB
docker-nat                    inband_ztp_2.3-9f32f66ea_Internal   1193d860e11c   431MB
docker-nat                    latest                              1193d860e11c   431MB
docker-teamd                  inband_ztp_2.3-9f32f66ea_Internal   ff4b85d7c525   429MB
docker-teamd                  latest                              ff4b85d7c525   429MB
docker-sflow                  inband_ztp_2.3-9f32f66ea_Internal   a2e39341e681   429MB
docker-sflow                  latest                              a2e39341e681   429MB
docker-lldp                   inband_ztp_2.3-9f32f66ea_Internal   9b5793de8e76   454MB
docker-lldp                   latest                              9b5793de8e76   454MB

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[rajendra-dendukuri]

Proposed fixes in

sonic-net/sonic-ztp#39
sonic-net/sonic-ztp#38