feat(sentry-kube): support combining files with keyword

[victoria-yining-huang]

Context:
streaming team needs github CODEOWNERS to consumer values so they can merge changes without depending on SRE aproval

PR explained:
for every level of overriding logic, find all files that start with the keyword (usually it will be the default value of _values) and combine them together, before proceeding with the original overriding logic

Impacts:
for services that only has one _values.yaml file, this changes nothing

Rollout plan:
after this PR is merged, cut a new release, update sentrykubelib version in ops, then make a new _consumer_values.yaml file in getsentry service, in all overriding levels

[fpacifici]

Please see my comment inline. There are quite a few more cases to adjust.

[mwarkentin]

Not sure where, but it would be nice to have some documentation about how our values merging works, maybe in https://github.com/getsentry/sentry-infra-tools/tree/main/libsentrykube/README.md?

Do we provide any sort of ordering logic? Is it purely alphabetical? What happens in case of a conflict?

[rgibert]

Not sure where, but it would be nice to have some documentation about how our values merging works, maybe in https://github.com/getsentry/sentry-infra-tools/tree/main/libsentrykube/README.md?

Do we provide any sort of ordering logic? Is it purely alphabetical? What happens in case of a conflict?

merge_values_files_no_conflict() should enforce no conflicts.

[victoria-yining-huang]

@mwarkentin in terms of documentation, I was pointed to this block of comments

sentry-infra-tools/libsentrykube/kube.py

Line 107 in 57a7eb1

We have multiple levels of overrides for our value files.

Not sure if there exists another doc like this

[mwarkentin]

Ok, at a minimum we should be updating that block with the new functionality, however it does feel like we should have something a little more discoverable / usable for introducing people to how sentry-kube works (values generation being a big part).

I don't have a strong opinion but either in a sentry-infra-tools readme or docs/ dir, or a notion page.

[rgibert]

typos

Suggested change

	4. `sk8s/services/seer/region_overrides/de/default.yaml` content will be combined with ``sk8s/services/seer/region_overrides/de/default_2.yaml`
	4. `k8s/services/seer/region_overrides/de/default.yaml` content will be combined with `k8s/services/seer/region_overrides/de/default_2.yaml`

Is there a reason we use a default_2 example instead of a default_consumers?

[victoria-yining-huang]

just because i don't think the seer service would have a consumers file, i was trying to give real examples. Not sure what example to give here.

[fpacifici]

This is not enough.
Let's say the issues tame takes over the ingest consumers and the subscription results consumers, it would not be possible to give them access to some specific consumers as we would have to break the value file according to a subkey of consumer_groups.

So you will need to allow some level of overlap. You could allow the same key to be defined in multiple places if it is a collection (like a map) as long as there is no conflict between the subkeys. The result of the merge would be a merge of the collections rather than a substitution.

[victoria-yining-huang]

Let's say the issues tame takes over the ingest consumers

is that really happening soon that we should build the support now? I think it's a good idea but I was only under the assumption that streaming team would own all consumers for the next little while

[fpacifici]

is that really happening soon

Yes it is happening as part of the bearhug for celery workers and hopefully for consumers.
Actually this is already supposed to happen now. The streaming platform is not going to take over the metrics indexer (sns) and it is a getsentry consumer.