Initial creation of netflow v5 payload generation#1372
Initial creation of netflow v5 payload generation#1372scottopell wants to merge 4 commits intomainfrom
Conversation
We've found the claude CLI tool is pretty helpful in this project, see #1372 for one example. This file is introduced to follow the recommendations in https://www.anthropic.com/engineering/claude-code-best-practices. Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
### What does this PR do?
We've found the claude CLI tool is pretty helpful in this project, see
#1372 for one example. This file is
introduced to follow the recommendations in https://www.anthropic.com/engineering/claude-code-best-practices.
### Motivation
Tooling improvements.
scottopell
force-pushed
the
sopell/netflow-poc
branch
from
71a1cfe to
efa69a0
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR implements NetFlow v5 payload generation for the lading load testing framework, allowing users to generate realistic NetFlow v5 packets for testing network flow collectors.
- Adds a new
NetFlowV5payload type with comprehensive configuration options for flow generation - Implements proper NetFlow v5 packet structure with 24-byte headers and 48-byte flow records
- Includes weighted protocol distribution, configurable IP/port ranges, and realistic flow timing
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| lading_payload/src/netflow.rs | New module implementing NetFlow v5 packet generation with configurable parameters |
| lading_payload/src/lib.rs | Exports NetFlowV5 type and adds it to the Config enum |
| lading_payload/src/block.rs | Integrates NetFlowV5 into the block cache construction system |
| lading_payload/src/NETFLOW_README.md | Documentation explaining NetFlow v5 implementation and usage |
| examples/netflow_v5.yaml | Example configuration file demonstrating NetFlow v5 usage |
| time::{SystemTime, UNIX_EPOCH}, | ||
| }; | ||
|
|
||
| use rand::{Rng, distr::weighted::WeightedIndex, prelude::Distribution}; |
There was a problem hiding this comment.
[nitpick] Consider grouping related imports. The Distribution trait is from rand::distributions::Distribution, not prelude::Distribution.
| use rand::{Rng, distr::weighted::WeightedIndex, prelude::Distribution}; | |
| use rand::{Rng, distributions::Distribution, distr::weighted::WeightedIndex}; |
| where | ||
| R: Rng + ?Sized, | ||
| { | ||
| config.valid().map_err(|_e| Error::StringGenerate)?; |
There was a problem hiding this comment.
The error message from config validation is being discarded. Consider preserving the validation error message or using a more appropriate error variant than StringGenerate.
| config, | ||
| protocol_distribution: WeightedIndex::new(protocol_weights)?, | ||
| flow_sequence: rng.random(), | ||
| sys_uptime_base: rng.random_range(0..86_400_000), // Random base uptime (0-24h) |
There was a problem hiding this comment.
Magic number 86_400_000 should be defined as a named constant. Consider defining const MILLISECONDS_PER_DAY: u32 = 86_400_000;
| sys_uptime_base: rng.random_range(0..86_400_000), // Random base uptime (0-24h) | |
| sys_uptime_base: rng.random_range(0..MILLISECONDS_PER_DAY), // Random base uptime (0-24h) |
| .duration_since(UNIX_EPOCH) | ||
| .unwrap_or_default(); | ||
|
|
||
| let current_uptime = self.sys_uptime_base + rng.random_range(0..3_600_000); // Add up to 1h |
There was a problem hiding this comment.
Magic number 3_600_000 should be defined as a named constant. Consider defining const MILLISECONDS_PER_HOUR: u32 = 3_600_000;
| let current_uptime = self.sys_uptime_base + rng.random_range(0..3_600_000); // Add up to 1h | |
| let current_uptime = self.sys_uptime_base + rng.random_range(0..MILLISECONDS_PER_HOUR); // Add up to 1h |
| count: flow_count, | ||
| sys_uptime: current_uptime, | ||
| unix_secs: now.as_secs() as u32, | ||
| unix_nsecs: (now.subsec_nanos() / 1000) * 1000, // Round to microseconds |
There was a problem hiding this comment.
This calculation is incorrect for NetFlow v5. The unix_nsecs field should contain nanoseconds since the Unix epoch second, not rounded microseconds. It should be now.subsec_nanos().
| unix_nsecs: (now.subsec_nanos() / 1000) * 1000, // Round to microseconds | |
| unix_nsecs: now.subsec_nanos(), // Use exact nanoseconds |
| // Calculate maximum flows that fit in the byte budget | ||
| let max_flows_by_budget = (max_bytes - HEADER_SIZE) / FLOW_RECORD_SIZE; | ||
| let desired_flows = self.config.flows_per_packet.sample(&mut rng) as usize; | ||
| let actual_flows = desired_flows.min(max_flows_by_budget).min(30); // NetFlow v5 max is 30 |
There was a problem hiding this comment.
Magic number 30 should be defined as a named constant. Consider defining const MAX_FLOWS_PER_PACKET: usize = 30;
| let actual_flows = desired_flows.min(max_flows_by_budget).min(30); // NetFlow v5 max is 30 | |
| let actual_flows = desired_flows.min(max_flows_by_budget).min(MAX_FLOWS_PER_PACKET); |
|
Closing, still interested in it but I want to keep the backlog focused on active work. |
What does this PR do?
Implements netflow v5 records as a new payload type that can be sent over UDP
Motivation
Load testing for netflow processors.
Related issues
Additional Notes