Wait for monit monitor <service> operation to complete during config reload

[tirupatihemanth]

Fixes sonic-net/sonic-buildimage#25599

What I did

Wait for monit monitor operation to complete before monit reload operation

Why I did

There is a race condition in the implementation of config reload in sonic code. 

During config reload -y -f we ask monit to unmonitor container_checker to avoid errors as containers go down. And during restarting as part of the reload we ask monit to monitor container_checker again. For this we using "monit monitor container_checker" command. This is an async operation. Please see below 

https://github.com/sonic-net/sonic-utilities/blob/cbb31f0d65c6768107f2089f6c75a617d8b519b4/config/main.py#L1058C1-L1068C55

    try:
        subprocess.check_call(['sudo', 'monit', 'status'], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
        click.echo("Enabling container and routeCheck monitoring ...")
        clicommon.run_command(['sudo', 'monit', 'monitor', 'routeCheck'])
        clicommon.run_command(['sudo', 'monit', 'monitor', 'container_checker'])
        time.sleep(1)
    except subprocess.CalledProcessError as err:
        pass
    # Reload Monit configuration to pick up new hostname in case it changed
    click.echo("Reloading Monit configuration ...")
    clicommon.run_command(['sudo', 'monit', 'reload'])

During monit reload, monit saves and restores monitoring state and since container_checker was not enabled so it will forever remained unmonitored from this point onwards.

How I did it

Wait till monitor action is complete before monit reload

How to verify it

monit checkers will not be left in not monitored state after config reload

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR updates the config reload/load_minigraph service restart flow in sonic-utilities to avoid leaving monit checkers in a Not monitored state by waiting for monit monitor <service> to actually take effect before reloading monit.

Changes:

• Added a polling helper to wait until a monit service is no longer reported as “Not monitored”.
• Replaced the fixed sleep(1) after monit monitor ... with explicit waits for routeCheck and container_checker.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).