Add info if task is safe with topology changes when listing tasks

[Michal-Leszczynski]

SM tasks in XCloud page

For the purposes of handling manager locks in siren (especially XCloud clusters), it's useful to know which SM tasks can run during topology changes (planned tablet repair), and which cannot (vnode repair, backup, ...). We want to have this logic implemented on SM side and the easiest way to expose it is to extend TaskListItem with TopologyChangeResilient or SafeWithTopologyChange field.

[Michal-Leszczynski]

@karol-kokoszka @VAveryanov8 @carlosms after some more thinking, I'm not sure if it's worth to implement this on SM side.

Initially, I thought that we would have a task type agnostic api for disabling, aborting (changed to cleanup), enabling tasks incompatible with topology changes, but I decided to go with per task type abort api (#4665). This means that we still need to perform simple checks per task type, but this should be fine.

I also thought that this api could be easily extended (if needed) to handle more subtle cases like:

• backup after snapshot stage is safe with adding nodes
• backup after snapshot is safe with node decommission
• backup is not safe with removing node VMs

But since for now we don't really need it (and we don't have formalized requirements), it would be difficult to design it in a clean way so that it could be easily extended in the future.

Finally, checking if task is safe with topology changes on siren side will look like: if task is a tablet repair task, then yes, else no, so it shouldn't be too much of a hassle.

Does it seem fine to you?

[carlosms]

Finally, checking if task is safe with topology changes on siren side will look like: if task is a tablet repair task, then yes, else no, so it shouldn't be too much of a hassle.

Does it seem fine to you?

Probably it would be nicer to have this handled on the manager side, having an API to query about it.
For example if a new manager version makes a new task safe with topology changes, it would work automatically. If it's on siren side we need to add logic about which manager version is installed, what tasks are safe for each version, etc.

We are under pressure to deliver with the minimum scope, I don't consider this a big issue. So it should be ok considering the circumstances.

[karol-kokoszka]

We are under pressure to deliver with the minimum scope, I don't consider this a big issue. So it should be ok considering the circumstances.

I think this is the most important part.

---

@Michal-Leszczynski Please correct me if I'm wrong, but what you are going to implement in SM to support XCloud is:

• pause ongoing repair - resume it after the topology is change. It will either fail or continue, if it fails, it should start from scratch; doesn't matter if it's tablet repair or vnode repair. Tablet repair will not fail most likely.
  VNodes repair should start from scratch if it fails to not call non existing VM API.
• abort ongoing backup - start from scratch after the resize operation. Pause backup task - so that it's not executed during resize operation, but start it explicitly just after when you see that it missed it's schedule.

Backup task needs to be aborted and not just paused, because we want to assure that hardlinks are removed.

What's the benefit of letting tablet's repair to continue during resize operation ? It complicates the implementation.

[Michal-Leszczynski]

So we will have tablet_repair, repair and backup tasks. We don't touch tablet_repair at all. We treat repair and backup tasks in the same* way - we disable them for the time of running the resize and later on enable, soft start them from scratch. The only difference is that we also cleanup the backup after we disable it.

What's the benefit of letting tablet's repair to continue during resize operation ? It complicates the implementation.

The thing is that skipping/delaying backup is not desired, but it does not put cluster at risk. Skipping/delaying repairs can result in either data resurrection, or with the default tombstone_gc mode repair, it results in tombstone accumulation which could impact cluster performance.

In general, not touching tablet repair on siren side should be an easy thing. It required some work on SM side to implement such task, but since this work is already in review, we should use it.