Performance bottleneck in kubernetes_logs source caused by excessive VRL BTree insert operations

[huanghuangzym]

Describe the performance issue
When using the kubernetes_logs source, we observed significantly lower log ingestion throughput compared to the file source.

• Same environment (HDD storage):

  • file source: ~120k events/sec
  • kubernetes_logs source: ~30k events/sec

During profiling, we found that a large amount of CPU time is spent in VRL BTree insert operations inside the metadata enrichment process.

---

To Reproduce

1. Deploy Vector as a DaemonSet using kubernetes_logs source.

2. Generate high log throughput from multiple busy-looping pods.

3. Monitor ingestion performance.

4. Compare with a configuration using the file source directly reading /var/log/containers/....

5. Run perf or similar profiler; observe hotspots in:

   vrl::value::value::crud::insert::insert
   

---

Expected behavior
The kubernetes_logs source should have ingestion throughput closer to the file source.
Metadata enrichment should not become the dominant CPU bottleneck at high event rates.

---

Proposed optimization ideas

• Evaluate alternative map implementations (HashMap or specialized small-map data structures) for VRL values.

[huanghuangzym]

vectordotdev/vector#23546

[pront]

Hi @huanghuangzym, thank you for creating this issue and I appreciate that you added a lot of information. For this specific issue, do you have any optimizations in mind specific to the crud implementation? Or this is a matter of how Vector is using this module? If it's the latter, we should close this issue.

[huanghuangzym]

Hi @huanghuangzym, thank you for creating this issue and I appreciate that you added a lot of information. For this specific issue, do you have any optimizations in mind specific to the crud implementation? Or this is a matter of how Vector is using this module? If it's the latter, we should close this issue.

Thanks for following
​​In VRL, is it required to use BTreeMap? Is there a specific reason for this choice? Since it significantly impacts insert performance, could we consider switching to HashMap instead?

[pront]

Hi @huanghuangzym, thank you for creating this issue and I appreciate that you added a lot of information. For this specific issue, do you have any optimizations in mind specific to the crud implementation? Or this is a matter of how Vector is using this module? If it's the latter, we should close this issue.

Thanks for following​​In VRL, is it required to use BTreeMap? Is there a specific reason for this choice? Since it significantly impacts insert performance, could we consider switching to HashMap instead?

I would need to take some time and evaluate the implications of this. I wouldn't be surprised if something depends on the sorting the BTreeMap provides. But overall, it's worth evaluating.