JSON Compression for Machine Events #100
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduce configuration options for machine event data compression, including enablement, compression level, targeted fields, and size threshold. Supports efficient storage by reducing payload size.
Replace `json` fields with `longText` and `binary` charset for `payload`, `context`, and `meta` columns to support compressed binary data. Improves storage efficiency for machine event data.
Introduce `CompressionManager` class to support configurable data compression and decompression. Includes global enablement, compression level, field targeting, size threshold, and compression statistics functionalities. Supports gzip compression for optimized data storage and transmission.
…tion Introduce unit tests for `CompressionManager` to validate the detection of compressed data. Ensure proper handling of compressed, uncompressed, null, and empty inputs.
Introduce unit tests to verify `CompressionManager`'s ability to compress and decompress data accurately. Ensure decompressed data matches the original.
Introduce a unit test to ensure `CompressionManager` can handle uncompressed JSON inputs for backward compatibility. Validate that decompressed results match the original data.
Introduce tests to verify `CompressionManager` respects the configured compression threshold. Ensure small data remains uncompressed while large data is compressed as expected.
Introduce tests to ensure `CompressionManager` respects the configured fields for compression. Validate that only specified fields are compressed while others remain uncompressed.
Introduce a unit test to verify `CompressionManager` respects the global compression disablement configuration. Ensure data remains uncompressed when compression is disabled globally.
Introduce a unit test to verify compression statistics calculation in `CompressionManager`. Ensure keys like `original_size`, `compressed_size`, `compression_ratio`, and `savings_percent` are accurate and valid.
Add tests to verify `CompressionManager` handles null values correctly during compression and decompression. Ensure `null` is converted and restored as expected.
Introduce a unit test to verify `CompressionManager` handles compression failures gracefully. Ensure JSON data is returned if compression fails.
Introduce a unit test to ensure `CompressionManager` throws an exception when attempting to decompress invalid compressed data. Validate proper error handling with `InvalidArgumentException`.
Introduce tests to verify `CompressionManager` handles edge compression levels correctly. Validate behavior for fastest (0) and best (9) compression, and ensure invalid levels throw `InvalidArgumentException`.
Introduce a unit test to verify `CompressionManager` handles very large data (1MB+) correctly. Ensure data is compressed and decompressed accurately without loss or error.
Introduce a unit test to verify `CompressionManager` handles special JSON characters, encoding, and formatting correctly. Ensure data is compressed and decompressed accurately without loss or alteration.
Introduce a unit test to verify `CompressionManager` correctly detects malformed zlib headers. Ensure valid compressed data is identified while invalid headers and partial data return false.
Ensure zlib extension is explicitly required in composer.json for compression-related functionality.
Introduce `CompressedJsonCast` to handle attribute compression and decompression using `CompressionManager`. Enables automated storage of compressed JSON data in models.
Switch MachineEvent attributes (payload, context, meta) to use CompressedJsonCast. Enables automatic compression and decompression of JSON data for optimized storage.
Introduce `CompressEventsCommand` to compress MachineEvent data for storage optimization. Includes options for chunk size, dry run, and force confirmation. Implements analysis and compression operations for payload, context, and meta fields.
Introduce `CompressMachineEventsJob` to enable batch compression of `MachineEvent` data. Includes configurable chunk size, retry logic, and logging for processed, compressed, and skipped records. Automatically dispatches subsequent jobs for remaining data. Handles failures with detailed logging.
Introduce integration tests for verifying automatic compression and decompression of MachineEvent data. Validate compression behavior for payload, context, and meta fields, ensuring data integrity and storage optimization. Confirm that machine_value remains uncompressed for querying.
Add integration test to validate compression threshold and field-specific configuration. Verify large payloads are compressed while small contexts remain uncompressed. Ensure data integrity after compression.
…sed data Verify handling of legacy uncompressed MachineEvent data. Ensure payload, context, and meta fields are correctly read and identified as uncompressed using `CompressionManager`. Validate data integrity and compatibility with existing attributes.
Introduce integration test to evaluate compression effectiveness on real-world MachineEvent contexts. Verify significant storage savings and maintain data integrity. Validate compression ratio and minimum savings percentage.
Introduce schema migration for upgrading MachineEvent storage to support compressed binary data. Adds new LONGBLOB columns, migrates JSON data, and maintains backward compatibility. Includes rollback support for decompression.
Add service provider registration for the new `CompressEventsCommand` and associated migration `upgrade_machine_events_for_compression_v2`. Enhances MachineEvent compression capabilities and ensures required resources are properly loaded.
Introduce unit test to verify automatic compression and decompression of MachineEvent payloads. Validate raw database storage as compressed and ensure accessors return decompressed data.
Introduce unit test to verify automatic compression of `context` on save. Ensure raw database storage is compressed and accessors return decompressed data.
Add `MigrateEventsCommand` and `CheckUpgradeCommand` to the service provider, enabling their registration for use as artisan commands.
Provide detailed migration paths for datasets of different sizes, including new commands and strategies for small, medium, and large datasets. Add troubleshooting steps, performance tips, and queue-based migration examples.
Introduce `machine:compression-benchmark` artisan command to evaluate compression and decompression performance across different levels. Includes benchmarking for speed, ratio, and recommendations on optimal levels.
Add `CompressionBenchmarkCommand` to MachineServiceProvider, ensuring its availability as an artisan command for compression performance testing.
Expand v3.0 upgrade documentation with detailed compression benchmarks, recommended levels, and key insights on performance trade-offs. Update data reduction expectations from 70% to 82-87%.
Remove `CheckUpgradeCommand`, `CompressEventsCommand`, and `CompressionBenchmarkCommand` along with related compression logic. These commands are no longer needed as the compression process has been integrated into the migration workflows.
Replace `longText` columns for `payload`, `context`, and `meta` with `json` type to improve readability and enforce structured data storage. This change removes unnecessary binary data handling for these fields.
Replace `CompressedJsonCast` with native `array` casts for `payload`, `context`, and `meta` attributes in `MachineEvent`. Simplifies code by removing dependency on custom casting logic.
Introduce `create_machine_events_archive_table` migration to store archived machine events with compression metadata. Includes fields for event data, compression details, event timestamps, and restore tracking.
Introduce `MachineEventArchive` to handle archived machine events with compression. Includes attributes for event data, timestamps, compression stats, and restore tracking. Provides methods to archive and restore events, along with querying scopes for machines and date ranges.
Introduce comprehensive tests for MachineEventArchive, covering event archival, restoration, compression statistics, custom compression levels, edge cases like empty collections, and querying archives by machine ID and date range.
Include new tests for JSON compression, compression thresholds, level validation, failure handling, and archival settings. Refactor test structure for clarity and add comprehensive scenarios to ensure reliability.
Introduce integration tests for ArchiveTransparency, covering state restoration from archived events, preference for active events over archived, and exception handling for missing machine states.
Introduce `machine:archive-status` command to show archive status with metrics, manage archived machine events, and perform restore/cleanup actions.
Introduce `ArchiveService` to handle machine event archival and restoration with compression. Includes batch archival, restoration, cooldown management, eligibility checks, and archive statistics. Add comprehensive unit tests for functionality.
Introduce `ArchiveMachineEventsJob` to handle automated archival of machine events. Implements triggers based on inactivity, event count, and size. Ensures efficient batch processing with error handling. Includes unit tests to verify job functionality under various configurations.
Introduce `machine:archive-events` command to facilitate machine event archival. Supports batch processing, dry-run analysis, queued execution, and direct archival with detailed statistics and compression support.
Introduce detailed configuration for machine event archival in `config/machine.php`. Includes options for triggers, compression, retention, batch processing, and advanced overrides for enterprise environments.
Introduce `restoreFromArchive` method in `Machine` to retrieve events from the archive table when not found in the active table. Ensure seamless state restoration by leveraging `ArchiveService`.
Add `source`, `meta`, and `root_event_id` fields to MachineEventFactory. Integrate `SourceType` enum for `source` field generation, ensuring consistency and improved test data creation.
Replace deprecated compression commands with new archival commands (`ArchiveEventsCommand`, `ArchiveStatusCommand`). Add migration for machine events archive table.
Add archive table migration in `TestCase` to ensure integration tests properly validate archive-related functionality.
Provide detailed instructions for upgrading to EventMachine v3.0. Include new features such as machine event archival, archive restoration, cooldown management, and enterprise settings. Explain configuration options, CLI commands, and testing archival transparency.
Remove unused advanced archival settings and redundant triggers from `machine.php`. Streamline configuration for better clarity and maintenance.
Remove advanced triggers (max events, multiple conditions) and streamline logic to use inactivity days only. Enhance readability and maintainability with fewer configurations.
Streamline `ArchiveMachineEventsJob` by removing support for advanced triggers such as max events and size. Refactor to use inactivity days as the sole trigger. Simplify configuration and improve maintainability.
Reflect updates to the archival system by removing advanced triggers and adding new settings like `restore_cooldown_hours`. Improve clarity and align with the refactored configuration in v3.0.
Member
Author
|
Consolidated into v3 release branch. All changes from this PR are included in the upcoming v3 release. |
Open
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.