Add log setting to dump failing HTTP request/response values using a ring buffer

[hodgesrm]

Diagnosing HTTP call failures often requires detailed logging of the request/response. The problem is that such logging is either feast or famine:

• HTTP logging is turned off so there's no additional information beyond the error type itself
• HTTP logging is turned on so the log is flooded with information that is useless if there are no errors. Even when there's an error you need to trek through irrelevant information to get to the error.

As a result users tend to keep logging off and then have to repeat operations with logging on once an error appears. This is inefficient and does not help with errors that appear intermittently.

We can address this problem using an in-memory ring buffer to hold the last N request/responses. If an error occurs we dump the ring buffer as if logging had been turned on. This gives clients diagnostic information without requiring HTTP logging to be turned on.

There are some subtleties in the implementation that must be considered.

• If HTTP logging is turned on the ring buffer is disabled.
• The ring buffer keeps entries in memory, so they cannot be large. Binary data like OVA uploads should not be stored.
• Clients should be able to control the number of entries to keep and dump in the event of an error. Just keeping the last request is a good default.
• Each Client struct should have its own ring buffer to prevent confusion in concurrent applications.

Golang has a ring implementation that may be helpful for implementation. See https://golang.org/pkg/container/ring/ for more information.

[dataclouder]

@hodgesrm,
I had a look at container/ring, but it does not seem to do what we want. However, I was able to create a fixed sized stack using container/list as storage.

type FixedSizeStack struct{
	list     *list.List
	maxItems int
}

func NewFixedSizeStack(wanted_size int) (fss FixedSizeStack) {
	fss.maxItems = wanted_size
	fss.list = list.New()
	return
}

func (fss *FixedSizeStack) Len() int {
	return fss.list.Len()
}

func (fss *FixedSizeStack) Push(item interface{}) {
	if fss.list.Len() >= fss.maxItems {
		fss.list.Remove(fss.list.Back())
	}
	fss.list.PushFront(item)
}

func (fss *FixedSizeStack) Pop() interface{} {
	latest := fss.list.Front()
	fss.list.Remove(fss.list.Front())
	return latest
}

And it does what we want, allowing us to store arbitrary objects.

	type Operation struct {
		Req Request
		Resp Response
	}
	max_elements := 2
	var fss FixedSizeStack =  NewFixedSizeStack(max_elements)

	for {
		err, request, response := get_request_and_response_from_somewhere()
		fss.Push(Operation{request, response})
		if err != nil {
			break
		}
	}

	for j := 0; j < max_elements; j++ {
		fmt.Printf("%#v\n", fss.Pop())
		fmt.Println(fss.Len())
	}

What I don't have clear is how do we want to implement it in such a way that each client gets its own ring of stored operations. Shall we create a client identifier and pass it to the two logging functions?