Implement in-memory caching and checkpointing

[DanielRyanSmith]

Overview

This PR implements in-memory caching for recent statuses and checkpointing to minimize GCS and Datastore I/O.

Root Cause / Motivation

Even with parallelization, downloading and uploading large JSON status files from GCS for every single revision creates significant network overhead. Caching them in memory and only writing checkpoints to GCS/Datastore periodically (every 20 revisions) dramatically speeds up the catch-up process.

Detailed Changelog

• process_test_history.py:
  • Implemented global in-memory cache _prev_test_statuses_cache for recent test statuses.
  • Implemented gzip compression/decompression for GCS status files.
  • Removed GCS status upload from process_single_run.
  • Implemented flush_previous_statuses_to_gcs to upload cached statuses.
  • Implemented commit_checkpoint to update date in Datastore and flush GCS cache.
  • Refactored main loop to use checkpointing (commit every 20 revisions).
  • Refactored get_aligned_run_info and process_runs to return dates instead of directly updating Datastore.

[jcscottiii]

One bug and one question

[jcscottiii]

• Critical Bug: generate_new_statuses_json is broken
  • Issue: When running with --generate-new-statuses-json (to create baseline statuses), _populate_previous_statuses returns an empty dict {} directly:

    if parsed_args.generate_new_statuses_json:
        verboseprint('Generating new statuses, so returning empty dict.')
        return {}

    This returns a new dict that is not stored in _prev_test_statuses_cache.
    Later, commit_checkpoint calls flush_previous_statuses_to_gcs, which checks the cache:

    if browser_name not in _prev_test_statuses_cache:
        verboseprint(f'No cached statuses to flush for {browser_name}')
        return

    Since the browser is not in the cache, it exits early, and the baseline statuses are never uploaded to GCS.
  • Fix: Initialize the cache when generating new statuses:

    if parsed_args.generate_new_statuses_json:
        verboseprint('Generating new statuses, so returning empty dict.')
        _prev_test_statuses_cache[browser_name] = {}
        return _prev_test_statuses_cache[browser_name]

[jcscottiii]

• Question: Checkpoint Commit Order

  • Thought: In commit_checkpoint, it updates the Datastore date before flushing GCS caches:

    # 1. Update date in Datastore
    update_recent_processed_date(date_entity, new_date)
    # 2. Flush GCS caches
    for browser in ...: flush_previous_statuses_to_gcs(browser)

    If GCS flush fails (e.g., due to transient network error), the Datastore date has already been advanced. On the next run, the script will start from new_date but GCS will still have old statuses, potentially leading to incorrect diffs (duplicate or missed deltas) in subsequent runs because we skipped updating GCS for the current window.

    Should we reverse the order?

    # 1. Flush GCS caches
    for browser in ...: flush_previous_statuses_to_gcs(browser)
    # 2. Update date in Datastore (only if flush succeeded)
    update_recent_processed_date(date_entity, new_date)

    If GCS flush fails with this reversed order, we crash and the Datastore date is not advanced. On the next run, we start from the last successful checkpoint and re-process the runs. Because of deterministic keys (PR 4949), re-processing is safe (idempotent).

    But are there other implications of reversing this order that we should consider?

[jcscottiii]

For the record, Gemini thought, we should reverse it initially. But I'm wondering if there are some implications it may have missed so I had it phrase the feedback here as a question instead.

[DanielRyanSmith]

I should have waited to merge the previous PR, because this one still needs to be reviewed, and the previous PR mentioned that it should be merged together with this one.

It should be okay, since these changes need to be added to the cloud function anyway to be finalized. 🙂