[Feature][history server] Add Google Cloud Storage (GCS) support to history server

[chiayi]

Why are these changes needed?

Provide Google Cloud Storage support for the history server

Related issue number

Part of #4453

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[chiayi]

I was able to utilize the github.com/fsouza/fake-gcs-server/fakestorage library in the test suite but it also looks like there is also an emulator that we could use in our CI. I think that can be the next step.

[chiayi]

To manually test, I upload collector and historyserver to Google Artifact Registry after building it using make -C historyserver localimage-build. Setup GKE cluster and created GCS Bucket. Both the AR and GCS should have correct permissions for access.

My GKE cluster was previously created so I updated it to use workload identity using

gcloud container clusters update <clustername> \
    --location=<region> \
    --workload-pool=<project_id>.svc.id.goog

Followed this guide to setup workload identity and connect Kubernetes SA with Google SA.

Deploy RayCluster with this collector:

- name: collector
          image: us-west1-docker.pkg.dev/aaronliang-dev/ray-ar/ray-collector:v0.1.0
          imagePullPolicy: Always
          env:
          - name: GCS_BUCKET
            value: "hs-ray-bucket"
          command:
          - collector
          - --role=Head
          - --runtime-class-name=gcs
          - --ray-cluster-name=raycluster-historyserver
          - --ray-root-dir=log
          - --events-port=8084
          volumeMounts:
          - name: historyserver
            mountPath: /tmp/ray
        tolerations:
        - key: ray
          operator: Equal
          value: cpu
        volumes:
        - name: historyserver
          emptyDir: {}

And historyserver:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: historyserver-demo
  labels:
    app: historyserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: historyserver
  template:
    metadata:
      labels:
        app: historyserver
    spec:
      serviceAccountName: historyserver
      containers:
      - name: historyserver
        env:
          - name: GCS_BUCKET
            value: "hs-ray-bucket"
        image: us-west1-docker.pkg.dev/aaronliang-dev/ray-ar/ray-historyserver:v0.1.0
        imagePullPolicy: Always
        command:
        - historyserver
        - --runtime-class-name=gcs
        - --ray-root-dir=log
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: "500m"

Viewing GCS should then start populating and every current endpoint should work as expected.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

[cursor]

Equality check instead of errors.Is for sentinel error

Medium Severity

The check err == gstorage.ErrBucketNotExist uses direct equality instead of errors.Is(). If the error is wrapped, this comparison will fail, causing the code to fall through to the generic error branch instead of attempting bucket creation. The same file correctly uses errors.Is(err, gstorage.ErrObjectNotExist) on line 48, making this inconsistent.