Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: LRU cache for flows in packetparser #590

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat: LRU cache for flows in packetparser #590

wants to merge 3 commits into from

Conversation

nddq
Copy link
Contributor

@nddq nddq commented Aug 8, 2024

Description

This PR optimizes the packetparser plugin by introducing an LRU cache to reduce memory and CPU usage under heavy traffic. Instead of generating a new Flow object for each packet event, the cache stores Flow objects based on source/destination IPs, ports, protocol, and direction. Packet events with matching values will reuse the existing Flow object, preventing excessive ToFlow invocations and potential OOM kills for the agent.

Related Issue

If this pull request is related to any issue, please mention it here. Additionally, make sure that the issue is assigned to you before submitting this pull request.

Checklist

  • I have read the contributing documentation.
  • I signed and signed-off the commits (git commit -S -s ...). See this documentation on signing commits.
  • I have correctly attributed the author(s) of the code.
  • I have tested the changes locally.
  • I have followed the project's style guidelines.
  • I have updated the documentation, if necessary.
  • I have added tests, if applicable.

Screenshots (if applicable) or Testing Completed

Over a similar period of time under the same workload, the total memory allocation for ToFlow without cache
image
vs with cache
image

Please add any relevant screenshots or GIFs to showcase the changes made.

Additional Notes

Add any additional notes or context about the pull request here.


Please refer to the CONTRIBUTING.md file for more information on how to contribute to this project.

@nddq nddq added type/enhancement New feature or request area/plugins scope/S Change is Small area/flows labels Aug 8, 2024
@nddq nddq self-assigned this Aug 8, 2024
@nddq nddq marked this pull request as ready for review August 9, 2024 22:24
@nddq nddq requested a review from a team as a code owner August 9, 2024 22:24
@nddq nddq mentioned this pull request Aug 15, 2024
11 tasks
Copy link

This PR will be closed in 7 days due to inactivity.

@github-actions github-actions bot added the meta/waiting-for-author Blocked and waiting on the author label Sep 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/flows area/plugins meta/waiting-for-author Blocked and waiting on the author scope/S Change is Small type/enhancement New feature or request
Projects
Status: In review
Development

Successfully merging this pull request may close these issues.

1 participant