Revisit docappender config autosizing values

[kruskall]

We use some heuristic to adjust the max buffer size, max decoder size and max requests.

apm-server/internal/beater/beater.go

Lines 543 to 552 in 6309df6

	func maxConcurrentDecoders(memLimitGB float64) uint {
	// Allow 128 concurrent decoders for each 1GB memory, limited to at most 2048.
	const max = 2048
	// Use 80% of the total memory limit to calculate decoders
	decoders := uint(128 * memLimitGB * 0.8)
	if decoders > max {
	return max
	}
	return decoders
	}

apm-server/internal/beater/beater.go

Lines 769 to 799 in 6309df6

	func docappenderConfig(
	opts docappender.Config, memLimit float64, logger *logp.Logger,
	) docappender.Config {
	const logMessage = "%s set to %d based on %0.1fgb of memory"
	// Use 80% of the total memory limit to calculate buffer size
	opts.DocumentBufferSize = int(1024 * memLimit * 0.8)
	if opts.DocumentBufferSize >= 61440 {
	opts.DocumentBufferSize = 61440
	}
	logger.Infof(logMessage,
	"docappender.DocumentBufferSize", opts.DocumentBufferSize, memLimit,
	)
	if opts.MaxRequests > 0 {
	logger.Infof("docappender.MaxRequests set to %d based on config value",
	opts.MaxRequests,
	)
	return opts
	}
	// This formula yields the following max requests for APM Server sized:
	// 1 2 4 8 15 30
	// 10 12 14 19 28 46
	maxRequests := int(float64(10) + memLimit*1.5)
	if maxRequests > 60 {
	maxRequests = 60
	}
	opts.MaxRequests = maxRequests
	logger.Infof(logMessage,
	"docappender.MaxRequests", opts.MaxRequests, memLimit,
	)
	return opts
	}

This formulas are really old and we had a lot of improvements since then, we should revisit them to ensure we are not underutilizing apm-server

[marclop]

We could easily update the max_requests formula to memLimit * 2 instead of memLimit * 1.5.

 	// 1	2 	4	8	15	30
 	// 12	14	18	26	40	70
 	maxRequests := int(float64(10) + memLimit*2) 
 	if maxRequests > 80 { 
 		maxRequests = 80 
 	} 

[rubvs]

I've updated the benchmark report in https://github.com/elastic/apm-dev/pull/1088.

Please review and do not hesitate to ask questions.

@simitt @kruskall @marclop @1pkg