CI/CD for Private GKE Clusters via DNS Endpoints

This guide documents the setup for a fully private GKE cluster that remains accessible to internal CI/CD pipelines without needing proxy VMs or complex VPN configurations.

This is solving: Transitive Peering Issue

1. Provisioning the Private GKE Cluster

The following command creates an Autopilot cluster that is completely isolated from the public internet. It uses the --enable-dns-access flag to create a local DNS endpoint within the VPC.

gcloud beta container clusters create-auto "autopilot-cluster" \
    --project "sidekick-1024" \
    --region "us-central1" \
    --release-channel "regular" \
    --enable-private-nodes \
    --enable-dns-access \
    --no-enable-ip-access \
    --no-enable-google-cloud-access \
    --network "default" \
    --subnetwork "default" \
    --cluster-ipv4-cidr "/17" \
    --binauthz-evaluation-mode=DISABLED

Key Networking Flags:

--enable-dns-access: Enables the DNS-based control plane (PSC), allowing the cluster to be reached via a local VPC IP.
--no-enable-ip-access: Disables the public IP endpoint for the control plane.
--no-enable-google-cloud-access: Ensures only traffic originating from within your specific VPC (like a Private Worker Pool) can reach the master.

2. Application Source Code

A basic Python Flask application used for the Proof of Concept (POC).

`app.py`

1
from flask import Flask
2
import os
3

4
app = Flask(__name__)
5

6
@app.route('/')
7
def hello():
8
    version = os.environ.get('VERSION', 'v1.0')
9
    return f"Hello from Private GKE! (Version: {version})\nDeployed via Cloud Build DNS Endpoint."
10

11
if __name__ == '__main__':
12
    app.run(host='0.0.0.0', port=8080)

`Dockerfile`

1
FROM python:3.9-slim
2
WORKDIR /app
3
COPY requirements.txt .
4
RUN pip install --no-cache-dir -r requirements.txt
5
COPY . .
6
EXPOSE 8080
7
CMD ["python", "app.py"]

3. Kubernetes Manifests

Stored in k8s/app.yaml. Note the PYTHON_IMAGE_PLACEHOLDER which is dynamically replaced by the pipeline.

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: python-poc-deployment
5
spec:
6
  replicas: 2
7
  selector:
8
    matchLabels:
9
      app: python-poc
10
  template:
11
    metadata:
12
      labels:
13
        app: python-poc
14
    spec:
15
      containers:
16
      - name: python-poc-container
17
        image: PYTHON_IMAGE_PLACEHOLDER
18
        ports:
19
        - containerPort: 8080
20
---
21
apiVersion: v1
22
kind: Service
23
metadata:
24
  name: python-poc-service
25
spec:
26
  type: LoadBalancer
27
  selector:
28
    app: python-poc
29
  ports:
30
  - port: 80
31
    targetPort: 8080

4. CI/CD Pipeline Configuration

The cloudbuild.yaml file is configured to run on a Private Worker Pool. It combines authentication and deployment into a single step to maintain the container’s credential context.

1
steps:
2
  # 1. Build the Docker image
3
  - name: 'gcr.io/cloud-builders/docker'
4
    id: build
5
    args:
6
      - build
7
      - -t
8
      - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPO_NAME/python-poc:$SHORT_SHA'
9
      - -t
10
      - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPO_NAME/python-poc:latest'
11
      - .
12

13
  # 2. Push to Artifact Registry
14
  - name: 'gcr.io/cloud-builders/docker'
15
    id: push
16
    args:
17
      - push
18
      - --all-tags
19
      - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPO_NAME/python-poc'
20
    waitFor: [build]
21

22
  # 3. Auth & Deploy (Combined Step to share ~/.kube/config)
23
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
24
    id: deploy
25
    entrypoint: 'bash'
26
    args:
27
      - '-c'
28
      - |
29
        # Fetch credentials using the DNS endpoint (PSC)
30
        gcloud container clusters get-credentials $_CLUSTER_NAME \
31
          --region $_REGION \
32
          --dns-endpoint
33

34
        # Inject the unique image tag into the manifest
35
        sed -i "s|PYTHON_IMAGE_PLACEHOLDER|$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPO_NAME/python-poc:$SHORT_SHA|g" k8s/app.yaml
36

37
        # Apply the update
38
        kubectl apply -f k8s/app.yaml
39
    waitFor: [push]
40

41
images:
42
  - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPO_NAME/python-poc:$SHORT_SHA'
43
  - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPO_NAME/python-poc:latest'
44

45
options:
46
  pool:
47
    name: 'projects/$PROJECT_ID/locations/$_REGION/privatePools/$_PRIVATE_POOL_NAME'
48

49
substitutions:
50
  _REGION: us-central1
51
  _REPO_NAME: poc-repo
52
  _CLUSTER_NAME: autopilot-cluster
53
  _PRIVATE_POOL_NAME: worker-pool

5. Deployment Pre-requisites

IAM Roles for Cloud Build Service Account:

Grant these to [PROJECT_NUMBER]@cloudbuild.gserviceaccount.com:

Artifact Registry Writer: To push images.
Kubernetes Engine Developer: To fetch credentials and modify cluster resources.

Networking Requirements:

Cloud Build Private Pool: Must be peered with the cluster’s VPC.
DNS Endpoint: The GKE cluster must have dns-access enabled to bypass the transitive peering restriction.