Mounting GCS bucket as volume on GKE PODs using FUSE CSI Driver

Google Kubernetes Engine (GKE) provides a robust platform for deploying containerised applications, and one common requirement is to mount a Google Cloud Storage (GCS) bucket as a volume in your GKE pods. This allows seamless access to GCS data from within your containers. In this blog post, we’ll explore how to achieve this using the Cloud Storage FUSE CSI driver and discuss a practical use case.

As we know that PODs are ephemeral under deployment. If we run out application as a work load on Kubernetes under deployment then the data or application logs will be lost. So In my scenario, I have mounted GCS bucket as a volume and routed the application logs on that GCS. So that logs will be persistent and the application team can seamlessly get access to the logs files

Benefits of Using the Cloud Storage FUSE CSI Driver

1. Simplified Configuration: The Cloud Storage FUSE CSI driver eliminates the need for PersistentVolumeClaim (PVC) and PersistentVolume (PV) objects. You can directly configure your GCS-backed volumes using the Kubernetes API.
2. Security: Unlike traditional FUSE clients, the Cloud Storage FUSE CSI driver does not require privileged access. This enhances security by reducing the attack surface.
3. Access Modes: The driver supports ReadWriteMany, ReadOnlyMany, and ReadWriteOnce access modes, allowing flexibility in volume usage.
4. ML Workloads: If you’re running ML training or serving workloads with frameworks like Ray, PyTorch, Spark, or TensorFlow, the Cloud Storage FUSE CSI driver simplifies data access without code changes.

How to Set Up the Cloud Storage FUSE CSI Driver

1. Enable the Google Kubernetes Engine API.
2. Install and Initialise the Google Cloud CLI (gcloud).
3. PV and PVC creation
4. Workload deployment

Create a GKE cluster with enabling FUSE Driver addon. You can use command below

gcloud container clusters create CLUSTER_NAME \
    --addons GcsFuseCsiDriver \
    --cluster-version=VERSION \
    --region=COMPUTE_REGION \
    --workload-pool=PROJECT_ID.svc.id.goog

To enable addon on the existing cluster use the below command

gcloud container clusters update CLUSTER_NAME \
    --update-addons GcsFuseCsiDriver=ENABLED \
    --region=COMPUTE_REGION

Once the GKE cluster is created, authenticate and connect to the cluster. Now we can proceed with setting up fUSE CSI Driver.

gcloud container clusters get-credentials CLUSTER_NAME \
    --region=COMPUTE_REGION

Creation of GCS bucket, Google cloud service account(GSA),Kubernets service account in the Namespace(KSA), binding storage object admin permission to the GSA on the created GCS and Binding the KSA with GSA.

export BUCKET_NAME=gcs-fuse-demo-bucket
export GCP_SA_NAME=cs-fuse-demo
export K8S_NAMESPACE=default
export K8S_SA_NAME=cs-fuse-demo-app
# Create a GCP service account in the Cloud Storage bucket project.
gcloud iam service-accounts create $GCP_SA_NAME - project=$GCP_PROJECT
#Apply permissions to a specific bucket.
gcloud storage buckets add-iam-policy-binding gs://$BUCKET_NAME \
 - member "serviceAccount:$GCP_SA_NAME@$GCP_PROJECT.iam.gserviceaccount.com" \
 - role "roles/storage.objectAdmin" \
 - project $GCP_PROJECT
#Bind the the Kubernetes Service Account with the GCP Service Account. The kubernetes resources will be created in the next step.
gcloud iam service-accounts add-iam-policy-binding $GCP_SA_NAME@$GCP_PROJECT.iam.gserviceaccount.com \
 - role roles/iam.workloadIdentityUser \
 - member "serviceAccount:$GCP_PROJECT.svc.id.goog[$K8S_NAMESPACE/$K8S_SA_NAME]" \
 - project $GCP_PROJECT

Creation of PV and PVC:

cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gcs-fuse-prd
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 30Gi
  storageClassName: #does not need to refer to an existing StorageClass object
  mountOptions:
    - implicit-dirs
    - uid=1001
    - gid=3003
  csi:
    driver: gcsfuse.csi.storage.gke.io
    volumeHandle: gcs-fuse-demo-bucket
    readOnly: true #optional
EOF

cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gcs-fuse-prd
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 30Gi
  volumeName: cs-fuse-pv
  storageClassName: cs-fuse-demo
EOF

Persistent volume

Persistent volume claim

Creation of deployment file:

This file below will do two things. First one is , it will create a KSA annotation and second is, it will create a deployment which contains volume and volume mount where the GCS will be mounted on that volume mount

The CSI driver relies on Pod annotations to identify if your Pod uses Cloud Storage-backed volumes. If the driver detects the necessary annotations, it injects a sidecar container called gke-gcsfuse-sidecar into your workload Pod. The Cloud Storage FUSE instances run inside the sidecar container and mount the Cloud Storage buckets for your workload.To enable the CSI driver to mount the Cloud Storage buckets, make sure you specify the annotation gke-gcsfuse/volumes: "true" in your Pod specification

apiVersion: v1
kind: ServiceAccount
metadata:
 name: cs-fuse-demo-app
 namespace: ${K8S_NAMESPACE}
 annotations:
 iam.gke.io/gcp-service-account: ${GCP_SA_NAME}@${GCP_PROJECT}.iam.gserviceaccount.com
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  template:
    metadata:
      annotations:
        gke-gcsfuse/volumes: "true"
    spec:
      containers:
        image: gcr.io/test-project/busybox:latest
        imagePullPolicy: Always
        volumeMounts:
        - mountPath: /opt/pega/logs
          name: gcs-fuse-prd
      volumes:
      - name: gcs-fuse-prd
        persistentVolumeClaim:
          claimName: gcs-fuse-prd

Apply this file and log into the POD and check the mount options, we can find the GCS added to the mount points.

You can see two containers running in the created POD, one is the main container and other will be a side car which will mount the GCS bucket using the CSI Fuse driver addon

GCS bukcet mounted as a volume on the defined volume mount path

Conclusion

Mounting a GCS bucket as a volume in GKE pods simplifies data access and enables seamless sharing between containers and GCS. Whether you’re serving static files, reading configuration data, or handling backups, the Cloud Storage FUSE CSI driver streamlines the process. Happy containerizing! 🚀