Add a `nginxprocess` package for working with NGINX processes by ryepup · Pull Request #956 · nginx/agent

ryepup · 2024-12-27T19:50:19Z

Proposed changes

Add a new nginxprocess package to export useful utilities around working with NGINX processes, and update internals to use that package. NGINXaaS for Azure has some proprietary code that also wants to use these utilities.

This comes with a significant speed improvement by filtering for NGINX processes earlier and fetching less data from the OS. See individual commits for more details.

Checklist

Before creating a PR, run through this checklist and mark each as complete.

I have read the CONTRIBUTING document
I have run make install-tools and have attached any dependency changes to this pull request
If applicable, I have added tests that prove my fix is effective or that my feature works
If applicable, I have checked that any relevant tests pass after adding my changes
If applicable, I have updated any relevant documentation (README.md)
If applicable, I have tested my cross-platform changes on Ubuntu 22, Redhat 8, SUSE 15 and FreeBSD 13

github-actions · 2024-12-27T19:50:35Z

✅ All required contributors have signed the F5 CLA for this PR. Thank you!
_{Posted by the CLA Assistant Lite bot.}

ryepup · 2024-12-27T19:50:59Z

I have hereby read the F5 CLA and agree to its terms

ryepup · 2024-12-27T20:18:10Z

I'm not sure why the pipeline is failing; make unit-test and make performance-test work if I run them locally, and I didn't change the otel collector section.

internal/watcher/health/nginx_health_watcher_operator.go

pkg/nginxprocess/process.go

ryepup · 2024-12-30T19:44:35Z

I'm not sure why the pipeline is failing; make unit-test and make performance-test work if I run them locally, and I didn't change the otel collector section.

unit-test fixed itself, and the performance-test was my fault; I added a benchmark that failed if there were no nginx processes. I switched that up so hopefully it'll be happy now.

pkg/nginxprocess/process.go

internal/watcher/health/nginx_health_watcher_operator.go

pkg/nginxprocess/process_test.go

ryepup · 2025-01-17T17:59:25Z

Did a rebase, looks like there's some test flake around the otel collector? panic: close of closed channel

ryepup · 2025-01-27T22:00:32Z

Did another rebase, now getting a different seemingly unrelated test failure with TestConvertToStructs. I think that one will fail occasionally because range over a map is non-deterministic, and the test is expecting to get keys in a certain order.

ryepup · 2025-01-27T23:07:26Z

opened #968 to fix the test thats broken my latest pipeline

This package is intended for use by any software that wants to identify NGINX processes. It's mostly a refactor of the existing `process.ProcessOperator` with a few differences and some features wanted by NGINXaaS. - minimizes `gopsutil` calls, many of which end up shelling out to `ps`. `nginxprocess.List` filters out non-NGINX processes early. `process.ProcessOperator` gathered details for all processes, and then non-NGINX processes were filtered out later by `instance.NginxProcessParser`. `nginxprocess.List` also fetches less process information, keeping only what the agent codebase actually used. A quick benchmark on my laptop (~750 processes, 12 of which are NGINX) showed a drastic speed improvement: | Benchmark | Iterations | Time (ns/op) | Memory (B/op) | Allocations (allocs/op) | | ----------- | ---------- | ------------ | ------------- | ----------------------- | | old | 1 | 26832273195 | 18282768 | 133775 | | new | 9 | 114558263 | 3439672 | 12836 | - uses a functional options pattern to optionally fetch process status, which is only needed in a few places and saves some more shelling out to `ps`. This could be expanded in the future to support other options while retaining backwards compatibility. - includes some error testing funcs to let callers handle different errors without needing to know how those errors are implemented. This helps trap `gopsutil` implementation details in `package nginxprocess` and makes future `gopsutil` upgrades easier. - respects context cancellation a little faster

Swap internal packages to use the shared `nginxprocess` package for details about NGINX processes. This is mostly switching `model.Process` for `nginxprocess.Process`, updating tests, and using features of `nginxprocess. There is one non-trivial change: `model.Process.Running` has been dropped, there is no equivalent in `nginxprocess`. `Running` was almost always true, because it was calculated immediately after a successful call to `NewProcessWithContext`. In order for it to be false, a process would have to end after `NewProcessWithContext`, but before `IsRunningWithContext`, which is an incredibly tiny time window. There is also no guarantee that the process is _still_ running by the time callers check `Running`, so the field was a little misleading. The system now relies solely on errors from `NewProcessWithContext` to identify terminated processes. Worst case we catch a dead process on the next tick of `HealthWatcherService`

Existing names were really similar to imported package names or other variables.

@dhurley

Based on @dhurley feedback in nginx#956 (comment)

ryepup requested a review from a team as a code owner December 27, 2024 19:50

github-actions bot added the chore Pull requests for routine tasks label Dec 27, 2024

ryepup force-pushed the export-pid-features branch from 22a996f to 39ba7e4 Compare December 27, 2024 20:01

ryepup force-pushed the export-pid-features branch from 39ba7e4 to eed2042 Compare December 27, 2024 20:18

ryepup commented Dec 27, 2024

View reviewed changes

internal/watcher/health/nginx_health_watcher_operator.go Outdated Show resolved Hide resolved

ryepup force-pushed the export-pid-features branch from eed2042 to f27ef37 Compare December 27, 2024 20:34

nginx-nickc reviewed Dec 30, 2024

View reviewed changes

pkg/nginxprocess/process.go Outdated Show resolved Hide resolved

pkg/nginxprocess/process.go Outdated Show resolved Hide resolved

pkg/nginxprocess/process.go Outdated Show resolved Hide resolved

pkg/nginxprocess/process.go Outdated Show resolved Hide resolved

ryepup force-pushed the export-pid-features branch 4 times, most recently from 2a306bf to fe5c421 Compare December 30, 2024 19:43

ryepup commented Jan 2, 2025

View reviewed changes

pkg/nginxprocess/process.go Outdated Show resolved Hide resolved

ryepup force-pushed the export-pid-features branch 2 times, most recently from 690b7a5 to 04c36a0 Compare January 2, 2025 20:46

dhurley reviewed Jan 9, 2025

View reviewed changes

internal/watcher/health/nginx_health_watcher_operator.go Outdated Show resolved Hide resolved

pkg/nginxprocess/process_test.go Outdated Show resolved Hide resolved

pkg/nginxprocess/process_test.go Outdated Show resolved Hide resolved

ryepup force-pushed the export-pid-features branch from 04c36a0 to 8344487 Compare January 9, 2025 15:33

github-actions bot added the documentation Improvements or additions to documentation label Jan 9, 2025

ryepup force-pushed the export-pid-features branch from 8344487 to 988212a Compare January 9, 2025 16:40

ryepup requested a review from dhurley January 15, 2025 18:48

ryepup force-pushed the export-pid-features branch from 988212a to 9bc5a98 Compare January 17, 2025 16:21

ryepup force-pushed the export-pid-features branch from 9bc5a98 to 735b647 Compare January 27, 2025 20:50

ryepup force-pushed the export-pid-features branch from 735b647 to 86cd976 Compare January 28, 2025 14:28

ryepup force-pushed the export-pid-features branch from 86cd976 to 540aefc Compare February 3, 2025 15:06

dhurley approved these changes Feb 10, 2025

View reviewed changes

aphralG approved these changes Feb 10, 2025

View reviewed changes

RRashmit approved these changes Feb 10, 2025

View reviewed changes

ryepup added 4 commits February 10, 2025 09:48

chore: improve some variable names

8449039

Existing names were really similar to imported package names or other variables.

docs: add note about test case prefixing

8f7f5b1

Based on @dhurley feedback in nginx#956 (comment)

ryepup force-pushed the export-pid-features branch from 540aefc to 8f7f5b1 Compare February 10, 2025 14:50

dhurley merged commit 96b0501 into nginx:v3 Feb 10, 2025
14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a `nginxprocess` package for working with NGINX processes#956

Add a `nginxprocess` package for working with NGINX processes#956
dhurley merged 4 commits intonginx:v3from
ryepup:export-pid-features

ryepup commented Dec 27, 2024

Uh oh!

github-actions bot commented Dec 27, 2024 •

edited

Loading

Uh oh!

ryepup commented Dec 27, 2024

Uh oh!

ryepup commented Dec 27, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryepup commented Dec 30, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryepup commented Jan 17, 2025

Uh oh!

ryepup commented Jan 27, 2025 •

edited

Loading

Uh oh!

ryepup commented Jan 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

ryepup commented Dec 27, 2024

Proposed changes

Checklist

Uh oh!

github-actions bot commented Dec 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ryepup commented Dec 27, 2024

Uh oh!

ryepup commented Dec 27, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryepup commented Dec 30, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryepup commented Jan 17, 2025

Uh oh!

ryepup commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ryepup commented Jan 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

github-actions bot commented Dec 27, 2024 •

edited

Loading

ryepup commented Jan 27, 2025 •

edited

Loading