Expose CircuitBreaker current condition (or state) to the metrics recorder interface and Prometheus. by mickdwyer · Pull Request #25 · slok/goresilience

mickdwyer · 2019-05-05T10:20:38Z

No description provided.

… so the metrics record closure

slok

Hi @mickdwyer thanks for the contribution.

I think that this metric is valuable, adding to this I've written you some comments on the implementation, but the most important thing is how the metric is designed, having the user to "interpret" what state is the gauge value is difficult, a better approach that I've seen in some exporters and apps (e.g. kube-metrics with kube_pod_status_phase) would be to have the state on a label and setting the active state with a 1 and the others to 0.

What do you think?

slok · 2019-05-07T04:12:18Z

circuitbreaker/circuitbreaker.go

+	return stateStrings[st]
+}
+
+func (st state) condition () int {


I think that we could remove the integer usage for state and use only the type and the string representation of it. It would simplify the usage and we remove code.

I agree, will refactor accordingly. (Am I right in that you're happy with the use of iota?

The iota is perfect

slok · 2019-05-07T04:15:51Z

circuitbreaker/circuitbreaker.go

+}
+
+func (st state) label () string {
+	return stateStrings[st]


2 things:

Instead of label what do you think about implementing the Stringer interface?

This could panic if a state does not have an associated string in the array, although we control this it could lead to a runtime panic in a change. A better approach could be using a switch with a fallback string, would be safer.

I'll have to read up on the Stringer interface but it feels a more natural fit.
I agree using a switch offers a more defensive implementation.

Will consider both when refactoring.

slok · 2019-05-07T04:21:32Z

circuitbreaker/circuitbreaker.go

-	stateOpen     state = "open"
-	stateHalfOpen state = "halfopen"
-	stateClosed   state = "closed"
+	stateNew   state = iota


New state is the same as a closed state because a new circuitbreaker starts in a closed state. Do you have a use case where you need to know the difference between new and closed?

My choice to implement stateNew and the transition to stateClosed on first traversal provided a mechanism to ensure that the state of the breaker was recorded on first use.

Otherwise, I found that there are no state metrics recorded until the first state change occurs.

I might take a few moments to review how I'm using the circuitbreaker instance by my application.

slok · 2019-05-07T04:24:40Z

metrics/metrics.go

 	// IncCircuitbreakerState increments the number of state change.
 	IncCircuitbreakerState(state string)
+	// SetCircuitbreakerCurrentCondition sets the condition of ciruit breaker.
+	SetCircuitbreakerCurrentCondition(condition int)


Would be better to have uniformity between method signatures:

On name: SetCircuitbreakerCurrentCondition -> SetCircuitbreakerCurrentState

On parameter: condition int -> state string

Will make the name change.
I suspect the parameter change will fall out of the refactoring of the metric approach. Stay tuned!

slok · 2019-05-07T04:26:55Z

circuitbreaker/circuitbreaker.go

 	switch state {
+	case stateNew:
+		// Close the breaker as this is the first time through and generate a statistic.
+		c.moveState(stateClosed, metricsRec)


As stated before, the initial state of a circuit breaker is closed, so the above change of state: stateNew, and this moveState would not be required.

True, although it's coupled to my comment above.

slok · 2019-05-07T04:28:31Z

metrics/prometheus.go

 	bulkProcessed                  *prometheus.CounterVec
 	bulkTimeouts                   *prometheus.CounterVec
 	cbStateChanges                 *prometheus.CounterVec
+	cbCurrentCondition             *prometheus.GaugeVec


Uniformity naming: cbCurrentCondition -> cbCurrentState

Will make this change

slok · 2019-05-07T04:29:00Z

metrics/prometheus.go

 		Help:      "Total number of state changes made by the circuit breaker runner.",
 	}, []string{"id", "state"})

+	p.cbCurrentCondition = prometheus.NewGaugeVec(prometheus.GaugeOpts{


Same as above Condition -> State

Will make this change.

slok · 2019-05-07T04:34:26Z

metrics/prometheus.go

 	p.cbStateChanges.WithLabelValues(p.id, state).Inc()
 }

+func (p prometheusRec) SetCircuitbreakerCurrentCondition(condition int) {


The problem with this metric approach is that the user needs to be aware of what is the number when it queries, this is not very natural and could lead to easy misinterpretation of the metrics.

To have the state of a resource in a metric what I have seen as a standardized way is that the state is in a label and when one of the states is set to "active" is set to 1 and the others to 0. Is the same approach as the state counter but with a gauge of (0,1).

What do you think?

Yes, I think that your proposal to follow the conventions you have seen elsewhere is worth adopting. I'll make the necessary refactoring to implement this design.

mickdwyer · 2019-05-07T11:13:13Z

Hi @slok, thank you for the very constructive feed back. I found it particularly helpful as this is my first use of Golang.

I'll comment again once I've recoded to the feedback above.

…tate

mickdwyer added 7 commits March 20, 2019 21:57

WIP: expose a Recorder built from Prometheus' DefaultRegisterer

b5e2dd3

WIP: create an unused circuit breaker state and close it on first use…

4d29958

… so the metrics record closure

WIP: fix typo

fcfae81

WIP: fix typo

57a6f52

WIP: rename the unused state to new

8978c16

WIP: use an itoa for the breaker state

db7bf6a

WIP: record the current circuit breaker state in the metrics recorder

2bf49fb

mickdwyer requested a review from slok as a code owner May 5, 2019 10:20

slok reviewed May 7, 2019

View reviewed changes

rename SetCircuitbreakerCurrentCondition to SetCircuitbreakerCurrentS…

c89af62

…tate

Conversation

mickdwyer commented May 5, 2019

Uh oh!

slok left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mickdwyer May 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mickdwyer May 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mickdwyer commented May 7, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mickdwyer May 7, 2019 •

edited

Loading

mickdwyer May 7, 2019 •

edited

Loading