CDS Updates with many clusters often fail

[Starblade42]

envoy/source/common/upstream/cds_api_impl.cc

Lines 52 to 101 in 2966597

	void CdsApiImpl::onConfigUpdate(const std::vector<Config::DecodedResourceRef>& added_resources,
	const Protobuf::RepeatedPtrField<std::string>& removed_resources,
	const std::string& system_version_info) {
	Config::ScopedResume maybe_resume_eds;
	if (cm_.adsMux()) {
	const auto type_urls =
	Config::getAllVersionTypeUrls<envoy::config::endpoint::v3::ClusterLoadAssignment>();
	maybe_resume_eds = cm_.adsMux()->pause(type_urls);
	}
	ENVOY_LOG(info, "cds: add {} cluster(s), remove {} cluster(s)", added_resources.size(),
	removed_resources.size());
	std::vector<std::string> exception_msgs;
	std::unordered_set<std::string> cluster_names;
	bool any_applied = false;
	for (const auto& resource : added_resources) {
	envoy::config::cluster::v3::Cluster cluster;
	try {
	cluster = dynamic_cast<const envoy::config::cluster::v3::Cluster&>(resource.get().resource());
	if (!cluster_names.insert(cluster.name()).second) {
	// NOTE: at this point, the first of these duplicates has already been successfully applied.
	throw EnvoyException(fmt::format("duplicate cluster {} found", cluster.name()));
	}
	if (cm_.addOrUpdateCluster(cluster, resource.get().version())) {
	any_applied = true;
	ENVOY_LOG(info, "cds: add/update cluster '{}'", cluster.name());
	} else {
	ENVOY_LOG(debug, "cds: add/update cluster '{}' skipped", cluster.name());
	}
	} catch (const EnvoyException& e) {
	exception_msgs.push_back(fmt::format("{}: {}", cluster.name(), e.what()));
	}
	}
	for (const auto& resource_name : removed_resources) {
	if (cm_.removeCluster(resource_name)) {
	any_applied = true;
	ENVOY_LOG(info, "cds: remove cluster '{}'", resource_name);
	}
	}
	if (any_applied) {
	system_version_info_ = system_version_info;
	}
	runInitializeCallbackIfAny();
	if (!exception_msgs.empty()) {
	throw EnvoyException(
	fmt::format("Error adding/updating cluster(s) {}", absl::StrJoin(exception_msgs, ", ")));
	}
	}

This is a performance issue, not a bug per se.

When doing CDS updates with many clusters, Envoy will often get "stuck" evaluating the CDS update. This manifests as EDS failing, and in more extreme cases, the envoy ceases to receive any XDS updates. When this happens, Envoy needs to be restarted to get it updating again.

In our case we're seeing issues with the current implementation of void CdsApiImpl::onConfigUpdate with a number of clusters in the 3000-7000 range. If envoy could speedily evaluate a CDS update with 10000 clusters this would represent a HUGE improvement in Envoy's behavior for us. Right now, only around 2500 clusters in a CDS update seems to evaluate in reasonable amount of time.

Because the function void CdsApiImpl::onConfigUpdate pauses EDS while doing CDS evaluation, envoy's config will drift. With many clusters in CDS, this can mean envoy is hundreds of seconds behind what is current, and results in 503's.

Some context:

• Envoy in my test environment is being run without constraints on 8 core VMs that are running at a max of 30% CPU utilization and max 40% memory (of 32 GB):

        resources:
          limits:
            memory: "32212254720"
          requests:
            cpu: 100m
            memory: 256M

• We're using Project Contour as our Ingress Controller in K8s.
  • https://projectcontour.io/
  • https://github.com/projectcontour/contour

Contour currently doesn't use the incremental APIs of Envoy, so when K8s services change in the cluster it sends ALL of the current config again to Envoy, which means a small change like adding or removing a K8s SVC which maps to an envoy cluster results in Envoy having to re-evaluate ALL clusters.

With enough K8s Services behind an Ingress (7000+) envoy can spontaneously cease to receive any new updates indefinitely, and will fail to do EDS because it gets stuck in the CDS evaluation.

Given a high enough number of clusters this would be a Very Hard Problem, but given that we're in the low thousands, I'm hoping there are some things that could be done to improve performance without resorting to exotic methods.

Please let me know if there's any information I can provide that could help!

[snowp]

Are these clusters changing on a regular basis? CDS updates are known to be somewhat expensive (though I wouldn't expect them to block the main thread for as long as you're describing). If most of the clusters are the same then each update should just be a hash of the proto + compare with existing, which still isn't free but >100s update latency sounds like a lot.

I'm not sure if we've been looking much into optimizing CDS cluster load time (maybe @jmarantz knows?), but the biggest improvement you'll get is most likely from using Delta CDS, where only the updated clusters are sent instead of the entire state of the world.

[Starblade42]

Yeah, my understanding is the Project Contour doesn’t use this API yet and that there would be significant work required to make use of the delta API. I’m also a bit unclear on the underlying details, but however the XDS apis are being used in Contour, it seems like some kinds of typical/normal changes in the state of the kubernetes cluster objects prompt a CDS evaluation, if not any real update, because a cluster that’s in steady state as far as objects goes (but not necessarily pods/endpoints) can cause envoy to get deadlocked like I described with 7000+ clusters. I believe that the expensiveness of the CDS and possibly something in the gRPC sessions are combining to cause the issue.

On Fri, Jul 17, 2020 at 6:13 PM Snow Pettersen ***@***.***> wrote: Are these clusters changing on a regular basis? CDS updates are known to be somewhat expensive (though I wouldn't expect them to block the main thread for as long as you're describing). If most of the clusters are the same then each update should just be a hash of the proto + compare with existing, which still isn't free but >100s update latency sounds like a lot. I'm not sure if we've been looking much into optimizing CDS cluster load time (maybe @jmarantz <https://github.com/jmarantz> knows?), but the biggest improvement you'll get is most likely from using Delta CDS, where only the updated clusters are sent instead of the entire state of the world. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#12138 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOT7S7TAYWLWYHLDS6UU3DR4DSK7ANCNFSM4O46FJUQ> .

-- -Jonathan Huff

[ramaraochavali]

CDS cluster initialization is definitely taking time. For 800 clusters, we have seen 20s to initialize - while Delta CDS helps to update new clusters, this slowness is actually affecting the envoy readiness during startup.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

[alandiegosantos]

A similar behaviour happened with some of our nodes. The node I investigated received 280 clusters and removed 1, as shown in this log:

[2020-10-12 18:00:14.160][54554][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:77] cds: add 280 cluster(s), remove 1 cluster(s)

And, as you can see, the last CDS update time is approximately to that time:

# TYPE envoy_cluster_manager_cds_update_time gauge
envoy_cluster_manager_cds_update_time{envoy_dc="someplace"} 1602518414207

After that, the pending EDS responses were processed and Envoy got stuck. This issue started since we started to using the GRPC to communicate with our Controlplane. We used REST before that.

The issue only solves when we restart Envoy. We are using version 1.14.5.

EDIT: This behaviour was detected around 13h on October 13th.

EDIT 2: The issue was caused by closing the connection between Envoy and Controlplane. In this context, we haven't enabled TCP keepalive on connections to Controlplane. So, when the firewall drops the packets from this TCP connection, Envoy was not able to detect that and waited indefinitely for the xDS response from Controlplane.

Thanks

[ramaraochavali]

@htuch @mattklein123 can we please reopen this? This is still an issue