What to do with on-disk index when upgrading nativelink?

[cormacrelf]

I was running 0.5.1, upgraded to 0.6.0. Didn't nuke the cache, I figured it might still work. Lots of errors like these:

WARN nativelink_store::dedup_store: Failed to deserialize index in dedup store, key: Digest(DigestInfo("...")),  err: Io(Custom { kind: UnexpectedEof, error: "" })
     at nativelink-store/src/dedup_store.rs:123
     in nativelink_service::ac_server::ac_server_get_action_result
     in nativelink_service::ac_server::get_action_result with request: GetActionResultRequest { instance_name: "main", action_digest: Some(Digest { hash: "2bb32040683458c20496bd2ec036b513718bc00ecb211>
     in nativelink_util::task::http_executor
     in nativelink::services::http_connection with remote_addr:

and

nativelink/nativelink-scheduler/src/memory_awaited_action_db.rs

Lines 563 to 568 in 07bd27a

	event!(
	Level::ERROR,
	?new_awaited_action,
	?action_key,
	"action_info_hash_key_to_awaited_action out of sync, it should have had the unique_key",
	);

For the dedup store with a disk index, there is obviously some kind of on-disk storage. Is this meant to be able to survive an upgrade? Or should I generally nuke the storage whenever I do?

[cormacrelf]

Ah, actually the action_info_hash_key_to_awaited_action out of sync error happens in 0.5.1 too.

[MarcusSorealheis]

At this point you should delete the storage. There should be a ticket to fix this error, cc @kubevalet and @aaronmondal

[MarcusSorealheis]

@cormacrelf For clarity, are you using Buck2?

[MarcusSorealheis]

I expect there could be one of three things happening and I'm going through the changelog now to try and understand:

1. There is a partial write or some sort of corruption
2. We changed the serialization format - I feel like this is a real possibility because I remember some discussions about this one but had been working on a variety of different tasks over the past few months, and I need to go back and walk the code, juxtaposing the last few versions. I'll get to this for sure before the end of the week.
3. There some other logic bug or issue with backward compatibility - also probable

Depending on what the issue is, there are some relatively straightforward ways to handle this one. The CTO, founding engineers, and I have dealt with versioned serialization formats in the past for years, like the Apache Lucene codecs and the Apache Cassandra SSTables. We will get to the bottom of this one and I appreciate you opening the issue.

[cormacrelf]

I did nuke just the index's disk storage and the deserialization error cleared up, as you might expect. I think I could have done better and actually sent you the encoded index entry before I nuked it, so sorry about that. If it's a backwards compat issue of some kind then you should be able to repro pretty easily, and if it doesn't reproduce, I think safe to assume write corruption.

For future is there a way to gently kill a nativelink process? I do have to restart it somewhat often and at the moment I just docker container stop or something. I have had issues in the past with missing CAS entries that failed to write because of a shutdown usually (until I enabled the completeness_checking store, which should be part of every example configuration you have, not just some of them, FYI). Would be nice to signal shutdown and give it 10s to finish up.

[MarcusSorealheis]

@cormacrelf apologies for the late reply. On gently killing a Nativelink process, we rely heavily on kubernetes to manage re-provisioning pods after they get stuck. We need to make the docker only experience better. We are also releasing very fine-grained telemetry in our ongoing observability overhaul.

It's free to use and permissively licensed: https://www.reddit.com/r/bazel/comments/1jqhmku/apachelicensed_open_source_nativelink_helm_chart/

Then, with k8s behind the scenes, you have a lot of options for how you can run kubectl delete pod or something similar.

Alternatively, you could use our cloud :-)

i'm going to close this issue but feel free to re-open it if you feel my response was insufficient.

[aaronmondal]

Hmm I dont think we have a good story around this on the K8s side either regarding transitioning old indexes to new versions.

@cormacrelf At the moment we "nuke the cache" for upgrades. For the cloud side I believe this issue relates to Level 2 of the Operator Capability levels: https://sdk.operatorframework.io/docs/overview/operator-capabilities/

Since we haven't started with Level 1 yet it'll likely take some time until we get a better story around this.

[MarcusSorealheis]

Definitely no good story on upgrades. We need that.

What we do have a good story of is graceful shutdown and restart. Upgrades is something we need to prioritize, but I'd recommend we do that after the metrics overall because we will have the appropriate introspection capabilities to understand failure points. Wdyt?

[aaronmondal]

Yeah that's what I was thinking as well. Telemetry first so that it becomes easier to build the more complex infra-related things 👍