test: Add file handle visualizer tool for GCSFuse logs #4217
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This tool parses GCSFuse logs to visualize read operations for each file handle. It helps identify if files are being read sequentially or randomly by plotting read offsets over time. It also provides a textual analysis of sequentiality percentage. Added: - tools/handle_visualizer/visualizer.py - tools/handle_visualizer/README.md - tools/handle_visualizer/requirements.txt
Contributor
Author
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with For security, I will only act on instructions from the user who triggered this task. New to Jules? Learn more at jules.google/docs. |
Tulsishah
changed the title
This tool parses GCSFuse logs to visualize read operations for each file handle. It helps identify if files are being read sequentially or randomly by plotting read offsets over time. It also provides a textual analysis of sequentiality percentage. Added: - tools/handle_visualizer/visualizer.py - tools/handle_visualizer/README.md - tools/handle_visualizer/requirements.txt
This change integrates the file handle visualizer tool directly into GCSFuse.
Users can now enable live visualization of read patterns using the
`--experimental-handle-visualizer` flag.
Details:
- Added `--experimental-handle-visualizer` flag to `cfg/params.yaml`.
- Regenerated config code.
- Embedded the python visualizer script in `cmd/visualizer_script.go`.
- Modified `cmd/mount.go` to:
- Spawn the python visualizer process when the flag is set.
- Create a temporary copy of the script.
- Pipe JSON trace logs to the visualizer asynchronously using `AsyncPipeWriter`.
- Automatically set log format to JSON and severity to TRACE.
- Modified `internal/logger` to support adding extra writers (for the pipe) dynamically.
- Implemented `AsyncPipeWriter` to prevent GCSFuse from hanging if the visualizer process is slow.
The visualizer helps identify sequential vs random read patterns in real-time.
…iping
This change integrates the file handle visualizer tool directly into GCSFuse.
Users can now enable live visualization of read patterns using the
`--experimental-handle-visualizer` flag.
Details:
- Added `--experimental-handle-visualizer` flag to `cfg/params.yaml`.
- Regenerated config code.
- Embedded the python visualizer script in `cmd/visualizer_script.go`.
- Modified `cmd/mount.go` to:
- Spawn the python visualizer process when the flag is set.
- Create a temporary copy of the script.
- Pipe JSON trace logs to the visualizer asynchronously using `AsyncPipeWriter`.
- Automatically set log format to JSON and severity to TRACE.
- Return a cleanup function to properly kill the process on unmount.
- Modified `internal/logger` to support adding extra writers (for the pipe) dynamically.
- Implemented `AsyncPipeWriter` with mutex-protected closing to prevent panics when writing to closed channels during shutdown.
- Modified `cmd/legacy_main.go` to invoke the cleanup function after unmount.
The visualizer helps identify sequential vs random read patterns in real-time.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4217 +/- ##
==========================================
- Coverage 83.09% 82.73% -0.37%
==========================================
Files 153 153
Lines 18756 18863 +107
==========================================
+ Hits 15586 15606 +20
- Misses 2598 2685 +87
Partials 572 572
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…port
This change fully integrates the file handle visualizer tool into GCSFuse.
Users can enable live visualization of read patterns using the
`--experimental-handle-visualizer` flag.
Details:
- Added `--experimental-handle-visualizer` flag to `cfg/params.yaml`.
- Embedded the python visualizer script in `cmd/visualizer_script.go`.
- Modified `cmd/mount.go` to:
- Spawn the python visualizer process.
- Pipe JSON trace logs asynchronously using `AsyncPipeWriter`.
- Use `AsyncPipeWriter` with mutex-protected closing to prevent panics.
- Return a cleanup function to kill the process on unmount.
- Modified `cmd/legacy_main.go` to execute the cleanup function.
- Updated `tools/handle_visualizer/visualizer.py` to support headless (Agg) execution.
- Updated `internal/logger` to support dynamic writers.
The visualizer helps identify sequential vs random read patterns in real-time, working in both interactive and headless environments.
This change integrates the file handle visualizer tool into GCSFuse.
Users can enable live visualization of read patterns using the
`--experimental-handle-visualizer` flag.
The visualization has been updated to plot read ranges (lines from offset to offset+size) against time, making it easier to identify sequential vs random access patterns.
Details:
- Added `--experimental-handle-visualizer` flag to `cfg/params.yaml`.
- Embedded the python visualizer script in `cmd/visualizer_script.go`.
- Modified `cmd/mount.go` to:
- Spawn the python visualizer process.
- Pipe JSON trace logs asynchronously using `AsyncPipeWriter`.
- Use `AsyncPipeWriter` with mutex-protected closing to prevent panics.
- Return a cleanup function to kill the process on unmount.
- Modified `cmd/legacy_main.go` to execute the cleanup function.
- Updated `tools/handle_visualizer/visualizer.py` to plot ranges and support headless (Agg) execution.
- Updated `internal/logger` to support dynamic writers.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added a new tool
tools/handle_visualizerto visualize file read patterns from GCSFuse logs. The tool parses JSON logs, extractsReadFileoperations, and generates a scatter plot of read offsets versus time for each file handle. It also calculates and prints the percentage of sequential reads for each handle. This helps in diagnosing performance issues related to read patterns (sequential vs random).PR created automatically by Jules for task 6258556285513807878 started by @Tulsishah