Add filestore as a storage support option for TPU 7x (#5059)

Author: agrawalkhushi18

examples/gke-tpu-7x/README.md

@@ -114,6 +114,7 @@ This repository also includes an advanced blueprint, `gke-tpu-7x-advanced.yaml`,
 - Performance-tuned GCS FUSE mounts pre-configured in the cluster as Persistent Volumes.
 - **Optional** High-Performance Storage: [Hyperdisk Balanced](https://docs.cloud.google.com/compute/docs/disks/hyperdisks) support for highly available and consistent performance across GKE nodes. For details of configuring Hyperdisk Balanced, please refer to the [appendix](#understanding-hyperdisk-balanced-integration).
 - **Optional** High-Performance Storage: [Managed Lustre](https://cloud.google.com/managed-lustre/docs/overview) for high-performance, fully managed parallel file system optimized for heavy AI and HPC workloads. For details of configuring Managed Lustre, please refer to the [appendix](#understanding-managed-lustre-integration).
+- **Optional** Shared File Storage: [Filestore](https://docs.cloud.google.com/filestore/docs/overview) for managed NFS capabilities allowing multiple TPU hosts to share logs, code, or datasets. For details, refer to the [appendix](#understanding-filestore-integration).
 
 ### Deploying the Advanced Blueprint
 
@@ -295,7 +296,7 @@ Once deployed, the `Lustre` filesystem is available to the cluster as a `Persist
 1. Connect to your cluster:
 
     ```sh
-    gcloud container clusters get-credentials DEPLOYMENT_NAME --region=REGION --project_id=PROJECT_ID
+    gcloud container clusters get-credentials DEPLOYMENT_NAME --region=REGION --project=PROJECT_ID
     ```
 
    Replace the `DEPLOYMENT_NAME`,`REGION` and `PROJECT_ID` with the ones used in the blueprint.
@@ -366,7 +367,7 @@ After making these changes, run the `gcluster deploy` command as usual.
 1. Connect to your cluster:
 
     ```sh
-    gcloud container clusters get-credentials DEPLOYMENT_NAME --region=REGION --project_id=PROJECT_ID
+    gcloud container clusters get-credentials DEPLOYMENT_NAME --region=REGION --project=PROJECT_ID
     ```
 
    Replace the `DEPLOYMENT_NAME`,`REGION` and `PROJECT_ID` with the ones used in the blueprint.
@@ -391,3 +392,42 @@ After making these changes, run the `gcluster deploy` command as usual.
     ```
 
    The logs of the pod verifies the disk is mounted successfully and performs a mixed I/O test to validate the disk's provisioned performance.
+
+### Understanding Filestore integration
+
+To enable Filestore integration, perform the following steps before deploying:
+
+1. In the `gke-tpu-7x-cluster` module settings, ensure `enable_filestore_csi: true` is set.
+2. Find the section commented `--- FILESTORE ADDITIONS ---`. Uncomment the following modules:
+   - `filestore`: Provisions the Filestore instance and specifies the `local_mount` point.
+   - `shared-filestore-pv`: Creates the Kubernetes Persistent Volume and Claim.
+   - `shared-fs-job`: (Optional) A test job template to verify multi-node shared writing.
+
+#### Testing the Shared Filestore Mount
+The blueprint includes a sample job (`shared-fs-job`) that demonstrates how two different pods can write to and read from the same file simultaneously.
+
+1. Connect to your cluster:
+
+    ```sh
+    gcloud container clusters get-credentials DEPLOYMENT_NAME --region=REGION --project=PROJECT_ID
+    ```
+
+    Replace the `DEPLOYMENT_NAME`,`REGION` and `PROJECT_ID` with the ones used in the blueprint.
+
+2. Apply the Filestore test manifest,whose path is provided in the final deployment instructions:
+
+    ```sh
+    kubectl apply -f <path/to/shared-fs-job.yaml>
+    ```
+
+3. Verify the Shared Output: Once the pods are running, check the logs of the first pod to see it reading data written by the second pod:
+
+    ```sh
+    # Get pod names
+    kubectl get pods
+    
+    # Check logs for the first pod
+    kubectl logs <pod-name-0>
+    ```
+
+The logs will display content from `shared_output.txt`, showing timestamps and hostnames from both pods, confirming that the filesystem is truly shared.

examples/gke-tpu-7x/gke-tpu-7x-advanced.yaml

@@ -167,6 +167,7 @@ deployment_groups:
       enable_gcsfuse_csi: true # enable GCS Fuse for the cluster
       enable_persistent_disk_csi: true # enable Hyperdisk for the cluster
       enable_managed_lustre_csi: true  # enable Managed Lustre for the cluster
+      enable_filestore_csi: true # enable Filestore for the cluster
       configure_workload_identity_sa: true
       master_authorized_networks:
       - cidr_block: $(vars.authorized_cidr) # Allows your machine to run the kubectl command. Required for multi network setup.
@@ -454,3 +455,45 @@ deployment_groups:
   #     node_count: 1
 
   #   outputs: [instructions]
+
+  # # --- FILESTORE ADDITIONS ---
+  # - id: filestore
+  #   source: modules/file-system/filestore
+  #   use: [gke-tpu-7x-net-0]
+  #   settings: {local_mount: /mnt/7x-filestore}
+
+  # - id: shared-filestore-pv
+  #   source: modules/file-system/gke-persistent-volume
+  #   use: [gke-tpu-7x-cluster, filestore]
+
+  # # Shared Filestore Job
+  # - id: shared-fs-job
+  #   source: modules/compute/gke-job-template
+  #   use:
+  #   - gke-tpu-7x-cluster
+  #   - gke-tpu-7x-pool
+  #   - shared-filestore-pv
+  #   settings:
+  #     image: alpine/git:latest
+  #     command:
+  #     - sh
+  #     - -c
+  #     - |
+  #       echo "Pod ${HOSTNAME} is starting..."
+  #       SHARED_DIR=/mnt/7x-filestore
+  #       mkdir -p $SHARED_DIR
+  #       cd $SHARED_DIR
+
+  #       # Simulate some work and write to a shared file
+  #       cmd="date +%s"
+  #       TIMESTAMP=`$cmd`
+  #       echo "Pod ${HOSTNAME} writing data at $$TIMESTAMP" >> shared_output.txt
+  #       sleep 5
+  #       echo "Displaying content of shared_output.txt:"
+  #       echo "---"
+  #       cat shared_output.txt # Read the content to show it's shared
+  #       echo "---"
+  #       sleep 20
+  #       echo "Pod ${HOSTNAME} finished."
+  #     node_count: 2
+  #   outputs: [instructions]