Description
Please select the type of request
Enhancement
Tell us more
Describe the request
Currently the readiness probe used in a Splunk search head cluster tests if port 8089 is running, if it is running the instance is "ready", if not it is not ready. However I'd like to have this further customized to ignore nodes that are in manual (or automatic detention).
Expected behavior
The probe should check the status of the member, for example it could hit the endpoint https://localhost:8089/services/shcluster/member/ready and a response without errors would be considered successful.
A response such as:
<?xml version="1.0" encoding="UTF-8"?>
<!--This is to override browser formatting; see server.conf[httpServer] to disable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .-->
<?xml-stylesheet type="text/xml" href="/static/atom.xsl"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
<title>shclusterready</title>
<id>https://localhost:8089/services/shcluster/member/ready</id>
<updated>2024-04-13T15:50:05+10:00</updated>
<generator build="d95b3299fa65" version="9.1.3"/>
<author>
<name>Splunk</name>
</author>
<opensearch:totalResults>0</opensearch:totalResults>
<opensearch:itemsPerPage>30</opensearch:itemsPerPage>
<opensearch:startIndex>0</opensearch:startIndex>
<s:messages/>
</feed>
Would be successful/search head is ready for traffic, a response such as:
<?xml version="1.0" encoding="UTF-8"?>
<response>
<messages>
<msg type="ERROR">Search Head is in detention</msg>
</messages>
</response
Would result in that search head not receiving new traffic
Ideally this would be a switch/parameter in case someone wants to send traffic to members in detention.
Splunk setup on K8S
Splunk search head clusters will have this feature, and only search head clusters...
Reproduction/Testing steps
Any search head cluster has this feature, you can manually put a node in detention as per Put a search head cluster member into detention
K8s environment
N/A
Proposed changes(optional)
Provide either a flag or a new default that for the SHC CRD the readiness probe checks the search head status and members in manual detention as considered "not ready"
K8s collector data(optional)
N/A
Additional context(optional)
I've raised the related issue #1321