CI/CD for Private GKE Clusters via DNS Endpoints

This guide documents the setup for a fully private GKE cluster that remains accessible to internal CI/CD pipelines without needing proxy VMs or complex VPN configurations.

The Problem: Transitive VPC Peering

By default, VPC network peering in GCP is not transitive. If the VPC hosting your Cloud Build Private Worker Pool (VPC A) is peered with your customer VPC (VPC B), and your customer VPC is peered with the GKE Control Plane VPC (VPC C), Cloud Build cannot route traffic directly to the GKE Control Plane.

graph TD
    subgraph VPCA ["VPC A (Cloud Build Pool)"]
        CB["Cloud Build Worker"]
    end
    subgraph VPCB ["VPC B (Your VPC)"]
        InternalIP["Internal Route Gateway"]
    end
    subgraph VPCC ["VPC C (GKE Master VPC)"]
        GKE["GKE Control Plane"]
    end
    
    VPCA <-->|"VPC Peering"| VPCB
    VPCB <-->|"VPC Peering"| VPCC
    CB -.->|"❌ Blocked (No Transitive Routing)"| GKE
    
    style VPCA fill:#F8F9FA,stroke:#DADCE0,color:#1a1a1a
    style VPCB fill:#E8F0FE,stroke:#4285F4,color:#1a1a1a
    style VPCC fill:#FCE8E6,stroke:#EA4335,color:#1a1a1a
    style CB fill:#34A853,stroke:#1E7E34,color:#fff
    style GKE fill:#1A73E8,stroke:#0D47A1,color:#fff

The Solution: Private Service Connect (PSC) with GKE DNS Access

When enabling DNS access, GCP provisions a Private Service Connect (PSC) endpoint inside VPC B that exposes the GKE master API. The control plane generates a local DNS zone in your network so that any peered VPC (like VPC A) can resolve and communicate directly to the master via this endpoint.

graph TD
    subgraph VPCA ["VPC A (Cloud Build Pool)"]
        CB["Cloud Build Worker"]
    end
    subgraph VPCB ["VPC B (Your VPC)"]
        PSC["PSC Endpoint<br/>(VPC Internal IP)"]
        DNS["Private DNS Zone<br/>(*.gke.private)"]
    end
    subgraph VPCC ["VPC C (GKE Master VPC)"]
        GKE["GKE Control Plane"]
    end
    
    VPCA <-->|"VPC Peering"| VPCB
    VPCB <-->|"VPC Peering"| VPCC
    CB -->|"1. Resolves DNS"| DNS
    CB -->|"2. Sends API request"| PSC
    PSC -->|"3. Bridges tunnel"| GKE
    
    style VPCA fill:#F8F9FA,stroke:#DADCE0,color:#1a1a1a
    style VPCB fill:#E8F0FE,stroke:#4285F4,color:#1a1a1a
    style VPCC fill:#FCE8E6,stroke:#EA4335,color:#1a1a1a
    style CB fill:#34A853,stroke:#1E7E34,color:#fff
    style GKE fill:#1A73E8,stroke:#0D47A1,color:#fff
    style PSC fill:#F9AB00,stroke:#E37400,color:#1a1a1a

1. Provisioning the Private GKE Cluster

The following command creates an Autopilot cluster that is completely isolated from the public internet. It uses the --enable-dns-access flag to create a local DNS endpoint within the VPC.

gcloud beta container clusters create-auto "autopilot-cluster" \
    --project "sidekick-1024" \
    --region "us-central1" \
    --release-channel "regular" \
    --enable-private-nodes \
    --enable-dns-access \
    --no-enable-ip-access \
    --no-enable-google-cloud-access \
    --network "default" \
    --subnetwork "default" \
    --cluster-ipv4-cidr "/17" \
    --binauthz-evaluation-mode=DISABLED

Key Networking Flags:

• --enable-dns-access: Enables the DNS-based control plane (PSC), allowing the cluster to be reached via a local VPC IP.
• --no-enable-ip-access: Disables the public IP endpoint for the control plane.
• --no-enable-google-cloud-access: Ensures only traffic originating from within your specific VPC (like a Private Worker Pool) can reach the master.

2. Application Source Code

A basic Python Flask application used for the Proof of Concept (POC).

app.py

1
from flask import Flask
2
import os
3

4
app = Flask(__name__)
5

6
@app.route('/')
7
def hello():
8
    version = os.environ.get('VERSION', 'v1.0')
9
    return f"Hello from Private GKE! (Version: {version})\nDeployed via Cloud Build DNS Endpoint."
10

11
if __name__ == '__main__':
12
    app.run(host='0.0.0.0', port=8080)

Dockerfile

1
FROM python:3.9-slim
2
WORKDIR /app
3
COPY requirements.txt .
4
RUN pip install --no-cache-dir -r requirements.txt
5
COPY . .
6
EXPOSE 8080
7
CMD ["python", "app.py"]

3. Kubernetes Manifests

Stored in k8s/app.yaml. Note the PYTHON_IMAGE_PLACEHOLDER which is dynamically replaced by the pipeline.

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: python-poc-deployment
5
spec:
6
  replicas: 2
7
  selector:
8
    matchLabels:
9
      app: python-poc
10
  template:
11
    metadata:
12
      labels:
13
        app: python-poc
14
    spec:
15
      containers:
16
      - name: python-poc-container
17
        image: PYTHON_IMAGE_PLACEHOLDER
18
        ports:
19
        - containerPort: 8080
20
---
21
apiVersion: v1
22
kind: Service
23
metadata:
24
  name: python-poc-service
25
spec:
26
  type: LoadBalancer
27
  selector:
28
    app: python-poc
29
  ports:
30
  - port: 80
31
    targetPort: 8080

4. CI/CD Pipeline Configuration

The cloudbuild.yaml file is configured to run on a Private Worker Pool. It combines authentication and deployment into a single step to maintain the container’s credential context.

1
steps:
2
  # 1. Build the Docker image
3
  - name: 'gcr.io/cloud-builders/docker'
4
    id: build
5
    args:
6
      - build
7
      - -t
8
      - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPO_NAME/python-poc:$SHORT_SHA'
9
      - -t
10
      - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPO_NAME/python-poc:latest'
11
      - .
12

13
  # 2. Push to Artifact Registry
14
  - name: 'gcr.io/cloud-builders/docker'
15
    id: push
16
    args:
17
      - push
18
      - --all-tags
19
      - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPO_NAME/python-poc'
20
    waitFor: [build]
21

22
  # 3. Auth & Deploy (Combined Step to share ~/.kube/config)
23
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
24
    id: deploy
25
    entrypoint: 'bash'
26
    args:
27
      - '-c'
28
      - |
29
        # Fetch credentials using the DNS endpoint (PSC)
30
        gcloud container clusters get-credentials $_CLUSTER_NAME \
31
          --region $_REGION \
32
          --dns-endpoint
33

34
        # Inject the unique image tag into the manifest
35
        sed -i "s|PYTHON_IMAGE_PLACEHOLDER|$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPO_NAME/python-poc:$SHORT_SHA|g" k8s/app.yaml
36

37
        # Apply the update
38
        kubectl apply -f k8s/app.yaml
39
    waitFor: [push]
40

41
images:
42
  - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPO_NAME/python-poc:$SHORT_SHA'
43
  - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_REPO_NAME/python-poc:latest'
44

45
options:
46
  pool:
47
    name: 'projects/$PROJECT_ID/locations/$_REGION/privatePools/$_PRIVATE_POOL_NAME'
48

49
substitutions:
50
  _REGION: us-central1
51
  _REPO_NAME: poc-repo
52
  _CLUSTER_NAME: autopilot-cluster
53
  _PRIVATE_POOL_NAME: worker-pool

5. Deployment Pre-requisites

IAM Roles for Cloud Build Service Account

Grant these roles to the Cloud Build Service Account ([PROJECT_NUMBER]@cloudbuild.gserviceaccount.com):

• Artifact Registry Writer (roles/artifactregistry.writer): Required to push Docker images.
• Kubernetes Engine Developer (roles/container.developer): Required to fetch cluster credentials and apply configurations.

Networking Configuration

1. Cloud Build Private Pool: Must be peered with the customer VPC network.
2. GKE Cluster Configuration: The cluster must be launched with --enable-dns-access enabled to bypass transitive peering blocks.