Emit metrics / events for monitoring service brokers

[henrytk]

Note: this has been re-opened following this comment: #1165 (comment)

A short explanation of the proposed change

Emit extra http status code metrics for certain API endpoints, namely service broker-related endpoints. HTTP requests with the following prefixes will send extra metrics:

/[version]/service_instances
/[version]/service_bindings
/[version]/service_keys

The emitted metric name is:

cc.requests.#{route}.#{method.downcase}.http_status.#{status.to_s[0]}XX

E.g. cc.requests.service_bindings.put.http_status.5XX

An explanation of the use cases our change solves

Monitoring of service brokers to ensure operations are not failing, such as binding for example.

The current metrics only cover things internal to Cloud Foundry. These metrics provide the opportunity to monitor external components the platform relies on.

Links to any other associated PRs

Associated issue: #929
Resurrected from: #1165

---

• I have reviewed the contributing guide

• I have viewed, signed, and submitted the Contributor License Agreement

• I have made this pull request to the master branch

• I have run all the unit tests using bundle exec rake

• I have run CF Acceptance Tests

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/161509538

The labels on this github issue will be updated when the story is started.

[tcdowney]

Hi, I left a comment in the PR since there was some more discussion there:
#1165 (comment)

Seeking some feedback from the CAPI/SAPI PMs on this issue since there does seem like there could be observability improvements here.

[mattmcneeney]

Thanks @tcdowney. I can see that we are waiting for some further feedback after you've raised some technical concerns.

The SAPI team haven't touched much of the metrics stuff before, so we are relying on you to help us understand the feasibility of this request! Any thoughts @zrob?

[georgi-lozev]

Any updates/thoughts on this one ?
CC @zrob @tcdowney

[williammartin]

For anyone following this thread, I think feedback was provided on the associated PR #1165 and we created a story to track this https://www.pivotaltracker.com/n/projects/2105761/stories/161509538 though it has not yet been prioritised.