Skip to content

Chunk encoder has multiple issues #7526

Open
@sspaink

Description

@sspaink

The chunk encoder used by the decision log plugin has these issues:

1. Checking if the incoming event is too large compared the uncompressed size to the compressed configurable limit which could cause an incoming event to lose the ND cache unnecessarily or be dropped entirely. The event could potentially fit if it had been compressed. Comparing it to the "uncompressed limit" (aka the soft limit) isn't possible because it could mean the limit hasn't grown big enough. You can see this happen in TestChunkMaxUploadSizeLimitNDBCacheDropping, when compressed the event could fit within the upload limit of 400 bytes if it was compressed.
2. The size and event buffer reset the chunk encoder on each upload, discarding the soft limit which requires multiple decision events to stabilize. event buffer: https://github.com/open-policy-agent/opa/blob/main/v1/plugins/logs/eventBuffer.go#L131 and size buffer: https://github.com/open-policy-agent/opa/blob/main/v1/plugins/logs/plugin.go#L906-L915
3. The term "soft limit" was confusing, see my previous PR where I went on a journey to find the meaning of soft limit haha #7517
4. The configurable decision_logs.reporting.upload_size_limit_bytes is used to calculate the uncompressed size by exponentially increasing it, this could cause an overflow if it was set too high. https://github.com/open-policy-agent/opa/blob/main/v1/plugins/logs/encoder.go#L144-L153

Metadata

Metadata

Assignees

Labels

bugmonitoringIssues related to decision log and status pluginsruntime

Type

No type

Projects

Status

In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions