prometheus scraper does not batch metrics when emitting EMF

[svix-jbrown]

Describe the bug
If you configure cloudwatch-agent to scrape Prometheus metrics and you scrape more than 100 metrics, they will not actually appear in CloudWatch. This is because the agent just dumps all the metrics into a single log line, but as per the spec, a log line may not contain more than 100 metrics.

Steps to reproduce

1. Create an endpoint that exposes more than 100 Prometheus metrics (you can just have a static file on a web server at /metrics)
2. Configure cwagent with a prometheus.yaml that simply ingests all metrics and a cwagentconfig.json that contains a .logs.metrics_collected.prometheus.emf_processor.metric_declaration that ingests them all
3. Launch cwagent

What did you expect to see?
The metrics should all appear in the desired log stream, and also as Metrics in Cloudwatch

What did you see instead?
The metrics are all in the log stream, but are not in CloudWatch Metrics. The EMFValidationErrors metric under AWS/Logs increments to 1.

What version did you use?
1.300057.1b1167

What config did you use?

Environment
OS: N/A (deploying via Docker on Kubernetes)

Additional context
The agent should simply group into 100 metrics at a time and emit them as separate log lines.

[dricross]

Hi James, thank you for taking the time to report this detailed observation. We appreciate the description and reproduction steps you've provided. We have documented this feedback internally and will take it into consideration.

For now, as a workaround, you may want to consider being more selective in the metrics you collect, focusing on the most critical ones for your use case.

Please let us know if you have any additional questions or observations.