host-select: fix compatibility with force-condemned hosts by oliver-sanders · Pull Request #6623 · cylc/cylc-flow

oliver-sanders · 2025-02-19T15:42:15Z

The "force mode" option in the auto-restart feature hasn't been used for a while.

It still works at Cylc 8, but force-condemning a host causes workflow startup to fail.

This PR:

Fixes the issue with host-select.
Adds a note about the "force-condemn" mode in the cfgspec docs.
A cylc-doc PR will follow in due course. ... Issue document automigration caveats and "host!" option cylc-doc#809

For info on the "force mode", see the Cylc 7 docs: https://cylc.github.io/cylc-doc/7.9.3/html/running-suites.html#auto-stop-restart

Check List

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
Tests are included (or explain why tests are not needed).
Changelog entry included if this is a change that can affect users
Cylc-Doc pull request opened - soon, soon ...
If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

oliver-sanders · 2025-02-20T10:17:32Z

cylc/flow/host_select.py

+    blacklist = []
+    for host in global_config.get(['scheduler', 'run hosts', 'condemned'], []):
+        if host.endswith('!'):
+            host = host[:-1]


Note, this line is covered by the test amended in this PR (confirm by running it on master), however, that test is not run in CI due to it's shared filesystem setup.

hjoliver

Looks fine, just some minor verbosity-reduction suggestions.

A couple of big-picture questions though:

It's weird that the documentation had been deliberately deleted - does that suggest there was an intention to remove this option??
Maybe it's bedded in for historical reasons as this point but I'm not sure "force-condemn" is a very good name for this feature. Is it because there are some workflows that auto-migrate won't touch? (long-running background tasks?). But even then, shouldn't force-condemn try to auto-migrate first, then kill any remaining workflows that failed to migrate? Given that it simply kills all the workflows, something like "kill and condemn" would be a better name, to indicate that auto-migration is not even attempted.

cylc/flow/cfgspec/globalcfg.py

changes.d/6623.fix.md

cylc/flow/cfgspec/globalcfg.py

oliver-sanders · 2025-02-24T09:33:53Z

It's weird that the documentation had been deliberately deleted

I think this more slipped the net than was deliberately deleted.

Maybe it's bedded in for historical reasons as this point but I'm not sure "force-condemn" is a very good name

Happy to change, suggest an alternative.

shouldn't force-condemn try to auto-migrate first, then kill any remaining workflows that failed to migrate

Some workflows cannot auto-migrate (e.g. workflows started in --no-detach mode) or might fail to migrate (e.g. user has scrambled their SSH config since starting the workflow).

The idea was to condemn the host first (i.e. ask workflows to migrate nicely), then mop up any stragglers with the "force" mode.

Co-authored-by: Hilary James Oliver <hilary.j.oliver@gmail.com>

hjoliver · 2025-02-24T21:17:23Z

The idea was to condemn the host first (i.e. ask workflows to migrate nicely), then mop up any stragglers with the "force" mode.

OK that's fine. Leave the decision on when to force it up to the humans.

We should document that though, as some will assume the options are migrate-all or kill-all, and not think of migrate-if-possible before changing the global config again to kill the rest.

We should also document the circumstances under which workflows will not successfully auto-migrate.

I think the best place for that is the general "pool of hosts" doc in the user guide, would you agree?

oliver-sanders · 2025-02-25T11:03:46Z

Yes, a cylc-doc PR will follow (see OP), it shouldn't go here in the config docs.

hjoliver · 2025-02-26T00:18:09Z

I put up a cylc-doc issue, just to record that we need to document why force-mode is needed in the first place (some workflows will not auto-migrate): cylc/cylc-doc#809

hjoliver · 2025-02-26T00:28:23Z

Regards the name "force condemn" ...

A host is "condemned" as soon as it is listed as such, whether or not it still has workflows running on it. The word implies a host shutdown at some future time.

(C.f. a condemned man hasn't been hung by the neck just yet; and a condemned building hasn't been demolished yet.)

I think we can just say that workflows running on condemned hosts will be told to auto-migrate, unless the hostname is followed by "!" in which case they will be shut down. There's probably no need to call that forced condemnation, right?

I'll re-review and provide new code suggestions along those lines ...

hjoliver · 2025-02-26T00:32:26Z

changes.d/6623.fix.md

+Auto restart: The "force condemn" option (that tells workflows running on a
+server to shutdown as opposed to migrate) hasn't worked with the host-selection
+mechanism since Cylc 8.0.0. This has now been fixed and the "force condemn"
+option has been restored in the documentation.


Suggested change

Auto restart: The "force condemn" option (that tells workflows running on a

server to shutdown as opposed to migrate) hasn't worked with the host-selection

mechanism since Cylc 8.0.0. This has now been fixed and the "force condemn"

option has been restored in the documentation.

Restored the option to shut down rather than auto-migrate workflows on a condemned

host. This option hasn't worked with the host-selection mechanism since Cylc 8.0.0.

This isn't actually correct. The mode has not been restored, a bug in the host-selection mechanism has been fixed.

hjoliver · 2025-02-26T00:33:40Z

cylc/flow/cfgspec/globalcfg.py

+                * Workflows that are running on condemned hosts will attempt
+                  to migrate to an available host (providing the


Suggested change

* Workflows that are running on condemned hosts will attempt

to migrate to an available host (providing the

* By default, workflows running on condemned hosts will attempt

to migrate to an available host (providing the

Superfluous in combo with next paragraph.

hjoliver · 2025-02-26T00:35:35Z

cylc/flow/cfgspec/globalcfg.py

+                If a hostname listed here is followed by a ``!`` character
+                ("force mode"), workflows running on it
+                will shutdown rather than attempting to
+                migrate (providing the


Suggested change

If a hostname listed here is followed by a ``!`` character

("force mode"), workflows running on it

will shutdown rather than attempting to

migrate (providing the

If a hostname listed here is followed by a ``!`` character

workflows running on it will be told to shut down

rather than migrate to another host (providing the

hjoliver · 2025-02-26T00:36:12Z

cylc/flow/cfgspec/globalcfg.py

+                           # however two have been taken out:
+                           # * workflows running on "host1" will attempt to
+                           #   restart on "host3"
+                           # * workflows running on "host2" will shutdown


Suggested change

# * workflows running on "host2" will shutdown

# * workflows running on "host2" will shut down

hjoliver · 2025-02-26T00:40:39Z

cylc/flow/cfgspec/globalcfg.py

+                   The force-condemn ("!") option caused issues at workflow
+                   startup for Cylc versions between 8.0.0 and 8.4.1
+                   inclusive.


Suggested change

The force-condemn ("!") option caused issues at workflow

startup for Cylc versions between 8.0.0 and 8.4.1

inclusive.

The option to shut down ("hostname!") rather than migrate

("hostname") workflows on condemned hosts caused problems

at workflow startup for Cylc versions 8.0.0 through 8.4.1.

hjoliver · 2025-02-26T00:42:53Z

tests/functional/restart/43-auto-restart-force-override-normal.t

+        # ensure the workflow can start if a host is force-condemned
+        # see #6623


Suggested change

# ensure the workflow can start if a host is force-condemned

# see #6623

# ensure the workflow can start if previously shut down on a

# condemned host. See #6623

oliver-sanders · 2025-02-26T10:20:59Z

I think we can just say that workflows running on condemned hosts will be told to auto-migrate, unless the hostname is followed by "!" in which case they will be shut down. There's probably no need to call that forced condemnation, right?

I think we need a name for this. It would be clunky to cross-reference this from the docs otherwise.

Please suggest an alternative to "force". I will keep using "force" as a placeholder, happy to change, but need something to change it to.

hjoliver · 2025-02-27T06:00:21Z

I have suggested an alternative. It's not simply a replacement for the word "force" in "force condemned", because - given the meaning of the word "condemned" - that just doesn't make sense.

See my response on the cylc-doc issue: cylc/cylc-doc#809 (comment)

oliver-sanders · 2025-02-28T14:07:25Z

Turns out this was already well documented, I just didn't look very hard.

https://cylc.github.io/cylc-doc/stable/html/plugins/main-loop/built-in/cylc.flow.main_loop.auto_restart.html#module-cylc.flow.main_loop.auto_restart

I'm going to close this PR and re-raise it with simpler docs pointing at the canonical documentation.

host-select: fix compatibility with force-condemned hosts

a7829b5

oliver-sanders added this to the 8.4.2 milestone Feb 19, 2025

oliver-sanders self-assigned this Feb 19, 2025

oliver-sanders added the bug Something is wrong :( label Feb 19, 2025

oliver-sanders mentioned this pull request Feb 19, 2025

doc: document auto-restart "force mode" #6622

Closed

8 tasks

doc: document auto-restart "force mode"

c5d56f8

oliver-sanders commented Feb 20, 2025

View reviewed changes

oliver-sanders requested review from MetRonnie and hjoliver February 20, 2025 16:05

hjoliver reviewed Feb 23, 2025

View reviewed changes

Apply suggestions from code review

96b9429

Co-authored-by: Hilary James Oliver <hilary.j.oliver@gmail.com>

oliver-sanders requested a review from hjoliver February 24, 2025 13:43

hjoliver mentioned this pull request Feb 26, 2025

document automigration caveats and "host!" option cylc/cylc-doc#809

Open

hjoliver requested changes Feb 26, 2025

View reviewed changes

oliver-sanders closed this Feb 28, 2025

oliver-sanders removed this from the 8.4.2 milestone Feb 28, 2025

oliver-sanders mentioned this pull request Feb 28, 2025

host-select: fix compatibility with force-condemned hosts #6643

Merged

8 tasks

oliver-sanders added the superseded label Feb 28, 2025

hjoliver mentioned this pull request Mar 20, 2025

Fix terminology used to describe "auto-restart" functionality. #6677

Open

		* Workflows that are running on condemned hosts will attempt
		to migrate to an available host (providing the

	# * workflows running on "host2" will shutdown
	# * workflows running on "host2" will shut down

		# ensure the workflow can start if a host is force-condemned
		# see #6623

Conversation

oliver-sanders commented Feb 19, 2025 • edited by hjoliver Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oliver-sanders Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hjoliver left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oliver-sanders commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hjoliver commented Feb 24, 2025

Uh oh!

oliver-sanders commented Feb 25, 2025

Uh oh!

hjoliver commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hjoliver commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hjoliver Feb 26, 2025

Choose a reason for hiding this comment

Uh oh!

oliver-sanders Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hjoliver Feb 26, 2025

Choose a reason for hiding this comment

Uh oh!

oliver-sanders Feb 28, 2025

Choose a reason for hiding this comment

Uh oh!

hjoliver Feb 26, 2025

Choose a reason for hiding this comment

Uh oh!

hjoliver Feb 26, 2025

Choose a reason for hiding this comment

Uh oh!

hjoliver Feb 26, 2025

Choose a reason for hiding this comment

Uh oh!

hjoliver Feb 26, 2025

Choose a reason for hiding this comment

Uh oh!

oliver-sanders commented Feb 26, 2025

Uh oh!

hjoliver commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oliver-sanders commented Feb 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

oliver-sanders commented Feb 19, 2025 •

edited by hjoliver

Loading

oliver-sanders Feb 20, 2025 •

edited

Loading

hjoliver left a comment •

edited

Loading

oliver-sanders commented Feb 24, 2025 •

edited

Loading

hjoliver commented Feb 26, 2025 •

edited

Loading

hjoliver commented Feb 26, 2025 •

edited

Loading

oliver-sanders Feb 28, 2025 •

edited

Loading

hjoliver commented Feb 27, 2025 •

edited

Loading