NC | Lifecycle | Gaps / TODOs from PRs

[romayalon]

Environment info

• NooBaa Version: 5.19
• Platform: NC

Gaps/TODOs from PRs list -

• Add automatic unit tests to nc lifecycle lock mechanism - 8858.
• Add automatic unit tests to events, timout, and status - 8860.
• Add implementation of the processing of the 4 rules in POSIX. 8839.
• Add implementation of the processing of expiration GPFS
• Add implementation of the processing of nonCurrentDays GPFS.
• Add notifications usage. 8839
• Decide on batch size. 8839
• Endpoints Stats should be supported? currently, we will not collect stats on the lifecycle worker process. 8839
• Add CLI unit tests 8839
• Add CLI documentation. 8839
  • specific example - health changes see comment.
• currently object-sdk is request-account uses _simple_load_requesting_account. need to check what account to use 8853
• add validation that policy doesn't have both AbortIncompleteMultipartUpload and tags / sizes in filter 8853 - fixed in Added required validations in lifecycle rules #8914
• modify list-parts for better error handling and add pagination. after use list-parts for lifecycle instead of current implementation 8853 - think maybe we should use here ILM policies as well.
• enable working in batches. 8862
• Decide when we want to write timestamp file and when we don't.
• Take account from cache in process_bucket()
• Check if we need to add DIRECTIRY_PLUS to our ILM policies or not.
• delete reports, ilm policies, ilm candidates after 30 days (should be configurable)
• check for progress on each run and make sure that the gpfs ilm processing didn't take the whole time of the run
• go over PRs and make sure all gaps are written in here
• complex tests + GPFS tests from the design doc.
• move while (!this.lifecycle_run_status.state.is_finished) and while (!bucket_state.is_finished && bucket_state.num_processed_objects < config.NC_LIFECYCLE_BUCKET_BATCH_SIZE) { loops to the upper level, same for updating the lifecycle/bucket status loops.

 bucket_state.is_finished = Object.values(this.lifecycle_run_status.buckets_statuses[bucket_name].rules_statuses)
                .reduce(
                (acc, rule) => acc && (_.isEmpty(rule.state) || rule.state.is_finished),
                true
            );
this.lifecycle_run_status.state.is_finished = Object.values(this.lifecycle_run_status.buckets_statuses).reduce(
    (acc, bucket) => acc && (bucket.state?.is_finished),
    true
);

• Add get/set bucket state like we have for rule state and use them in the needed places.
• GPFS optimization - use fileset as the starting point of the mmapplypolicy command instead of the mount point.
• backslashes check on GPFS - we use backslash as an escape char, might cause issues when using backslash on the file system paths such as the bucket path.

Gaps/TODOs from testing -

• The key_marker is written even if the state is_finished is true (see test lifecycle batching - with lifecycle rule, multiple list batches, one bucket batch). - NC | lifecycle | remove key_marker and version_marker from state when finished #8991
• The error message of "Lifecycle worker reached timeout - configured timeout is 28800000 ms" - when changing the timeout doesn't change even when the config of timeout was changed (see test lifecycle batching - with lifecycle rule, multiple list batches, one bucket batch with change to NC_LIFECYCLE_TIMEOUT_MS).
• Mismatch error - on s3api put-bucket-lifecycle-configuration with NoncurrentVersionExpiration without NoncurrentDays (only NewerNoncurrentVersions) - in NooBaa it is InvalidArgument, in AWS it is MalformedXML. - Add more validation for lifecycle #8970

[github-actions]

This issue had no activity for too long - it will now be labeled stale. Update it to prevent it from getting closed.

[github-actions]

This issue is stale and had no activity for too long - it will now be closed.