fix: (cli) make export more stable and introduce stream in sketch.search() as optional

[jaegeral]

Summary

This pull request significantly improves the export functionality within Timesketch by addressing offering more granular control over export content. It introduces a streaming mechanism for search results, allowing users to export extensive data without encountering memory issues, and provides options to perform either a focused event export or a complete sketch export directly from the command-line interface.

Highlights

• Streaming Export for Large Results: Introduced an optional "stream" parameter in sketch.search().to_file() to enable streaming of large search results directly to a file.
• Enhanced CLI Export Command: The timesketch sketch export command now supports a --stream flag to leverage the new streaming capability and a --use_sketch_export flag to choose between event-only export (default) and a comprehensive sketch export (including stories, aggregations, views, and metadata).
• Updated Documentation: Both the API client and CLI client documentation have been updated to reflect the new stream parameter and the enhanced sketch export command options.
• New End-to-End Tests: Added comprehensive end-to-end tests for the sketch export command, covering default, streaming, and full sketch export scenarios to ensure stability and correctness.
  Also:

For large results, you can use the optional stream parameter to avoid loading the whole result into memory:

     search_obj.to_file('/tmp/myresults.zip', stream=True)                                           

Also introducing test for it and updating docs

[jaegeral]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a valuable streaming feature for sketch exports to improve stability with large datasets. The implementation across the API client and CLI is well-executed, with corresponding updates to documentation and the addition of end-to-end tests. I've included one suggestion to refactor the new test case for better maintainability by reducing code duplication. Overall, this is a great enhancement.

[jaegeral]

/gemini description

[gemini-code-assist]

It looks like you might have intended to use /gemini summary to get a summary of the pull request. The command /gemini description is not a recognized command. Please use /gemini help for a list of available commands.

[jaegeral]

/gemini summary

[gemini-code-assist]

Summary of Changes

This pull request significantly improves the export functionality within Timesketch by addressing memory efficiency for large datasets and offering more granular control over export content. It introduces a streaming mechanism for search results, allowing users to export extensive data without encountering memory issues, and provides options to perform either a focused event export or a complete sketch export directly from the command-line interface.

Highlights

• Streaming Export for Large Results: Introduced an optional "stream" parameter in sketch.search().to_file() to enable streaming of large search results directly to a file, preventing memory exhaustion.
• Enhanced CLI Export Command: The timesketch sketch export command now supports a --stream flag to leverage the new streaming capability and a --use_sketch_export flag to choose between event-only export (default) and a comprehensive sketch export (including stories, aggregations, views, and metadata).
• Updated Documentation: Both the API client and CLI client documentation have been updated to reflect the new stream parameter and the enhanced sketch export command options.
• New End-to-End Tests: Added comprehensive end-to-end tests for the sketch export command, covering default, streaming, and full sketch export scenarios to ensure stability and correctness.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• api_client/python/timesketch_api_client/search.py
  • Added a "stream" boolean parameter to _execute_query and to_file methods to enable streaming of HTTP responses.
  • Implemented chunked writing to file when the "stream" parameter is set to True in _execute_query.
  • Updated docstrings for _execute_query and to_file to document the new "stream" parameter.
• cli_client/python/timesketch_cli_client/commands/sketch.py
  • Introduced --stream and --use_sketch_export options for the sketch export command.
  • Modified the export_sketch function to conditionally call sketch.export() or search.Search().to_file() based on the --use_sketch_export flag.
  • Updated the docstring for export_sketch to describe the new export options and their behavior.
  • Broadened exception handling in export_sketch from ValueError to a general Exception.
• docs/developers/api-client.md
  • Added a Python code example demonstrating the use of the stream=True parameter with search_obj.to_file().
• docs/guides/user/cli-client.md
  • Expanded the 'Export a sketch' section to detail the --stream and --use_sketch_export options, including their purpose and example usage.
• end_to_end_tests/cli_client_e2e_test.py
  • Added import os and import uuid.
  • Added a new test method test_cli_sketch_export to verify the functionality of the sketch export command with default, streaming, and full export options.

Activity

• jaegeral requested a Gemini review.
• jaegeral attempted to use an unrecognized command /gemini description.
• gemini-code-assist[bot] informed jaegeral that /gemini description is not a recognized command and suggested /gemini summary.
• jaegeral requested a Gemini summary.
• gemini-code-assist[bot] provided a review comment suggesting refactoring the test_cli_sketch_export method in end_to_end_tests/cli_client_e2e_test.py to use a loop for better maintainability.
• jaegeral responded to the review comment, indicating that the current implementation is acceptable.

[jkppr]

What happens if I run the CLI with both flags --stream and use_sketch_export at the same time? If I understand this correctly, my stream flag would be ignored?

Maybe we can change the export method in the API client to also support a stream flag?

[jaegeral]

sgtm