← Back to Blog

Proxmox + Kubernetes: End-to-End CI/CD Automation

In modern software development, CI/CD (Continuous Integration/Continuous Deployment) is an indispensable process. In this article, I'll show you how to create a fully automated deployment pipeline by setting up a Kubernetes cluster on Proxmox virtual server infrastructure and integrating it with GitLab CI/CD.

Architecture Overview

Main components of our system:

  • Proxmox VE: Hypervisor and VM management
  • Kubernetes: Container orchestration
  • GitLab: CI/CD and source control
  • Docker Registry: Container image storage
  • Cloudflare: DNS and CDN

Proxmox Setup and Configuration

Proxmox is a powerful KVM-based virtual server platform. First, we'll create our VMs.

Creating VM Template

# Download Ubuntu cloud image
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img

# Create VM
qm create 9000 --memory 2048 --cores 2 --name ubuntu-cloud --net0 virtio,bridge=vmbr0
qm importdisk 9000 jammy-server-cloudimg-amd64.img local-lvm
qm set 9000 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-9000-disk-0
qm set 9000 --boot c --bootdisk scsi0
qm set 9000 --ide2 local-lvm:cloudinit
qm set 9000 --serial0 socket --vga serial0
qm set 9000 --agent enabled=1

# Convert to template
qm template 9000

Kubernetes Cluster Setup

Let's create our Kubernetes nodes using the template.

Master Node Setup

# Install kubeadm, kubelet, kubectl
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

# Initialize cluster
sudo kubeadm init --pod-network-cidr=10.244.0.0/16

# Configure kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

CNI Plugin (Calico) Installation

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/custom-resources.yaml

GitLab Runner Installation

We'll run GitLab Runner on Kubernetes.

apiVersion: v1
kind: Namespace
metadata:
  name: gitlab-runner
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab-runner
  namespace: gitlab-runner
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gitlab-runner
  template:
    metadata:
      labels:
        app: gitlab-runner
    spec:
      containers:
      - name: gitlab-runner
        image: gitlab/gitlab-runner:latest
        env:
        - name: RUNNER_EXECUTOR
          value: "kubernetes"
        volumeMounts:
        - name: config
          mountPath: /etc/gitlab-runner
      volumes:
      - name: config
        configMap:
          name: gitlab-runner-config

CI/CD Pipeline Configuration

Our .gitlab-ci.yml file:

stages:
  - build
  - test
  - deploy

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

deploy:
  stage: deploy
  image: bitnami/kubectl:latest
  script:
    - kubectl set image deployment/myapp myapp=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA -n production
    - kubectl rollout status deployment/myapp -n production

Docker Registry Integration

Let's run our private Docker registry on Kubernetes:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: docker-registry-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: docker-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
      - name: registry
        image: registry:2
        ports:
        - containerPort: 5000
        volumeMounts:
        - name: storage
          mountPath: /var/lib/registry
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: docker-registry-pvc

Cloudflare DNS and CDN

Domain management and CDN setup via Cloudflare:

  1. Add domain to Cloudflare
  2. Update DNS records
  3. Activate SSL/TLS certificate
  4. Set cache strategy with Page Rules
# Add Cloudflare API token to Kubernetes
kubectl create secret generic cloudflare-api-token \
  --from-literal=api-token=YOUR_TOKEN \
  -n kube-system

Monitoring and Logging

Monitoring with Prometheus and Grafana:

# Install Prometheus Operator
kubectl create -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/bundle.yaml

# Install Grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana -n monitoring --create-namespace

Auto Scaling

Horizontal Pod Autoscaler (HPA):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Backup Strategy

Kubernetes backup with Velero:

# Install Velero
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.8.0 \
  --bucket velero-backups \
  --backup-location-config region=eu-west-1 \
  --snapshot-location-config region=eu-west-1

# Schedule automatic backup
velero schedule create daily-backup --schedule="0 2 * * *"

Security Measures

  1. Network Policies: Restrict inter-pod communication
  2. RBAC: Apply role-based access control
  3. Pod Security Standards: Enforce security contexts
  4. Secret Encryption: Encrypt secrets
  5. Image Scanning: Scan container images

Conclusion

With this architecture:

  • ✅ Fully automated CI/CD pipeline
  • ✅ Scalable Kubernetes cluster
  • ✅ Monitoring and alerting
  • ✅ Automatic backup
  • ✅ CDN and DNS management

The Proxmox + Kubernetes combination provides a powerful and flexible infrastructure. With GitLab CI/CD integration, the entire process from development to production is automated.

"Everything that can be automated should be automated. Time is our most valuable resource."

← Back to Blog