本文详细记录了在已有的 Kubernetes 集群上，使用自定义 Docker 镜像和 Helm 之外的方式，搭建一个高可用、可扩展的 Jenkins 主从（Master-Worker）CI/CD 系统的完整过程。本方案采用 StatefulSet 保证 Master 的稳定性和数据持久化，并通过 Kubernetes 插件动态创建 Pod 作为构建 Agent。

---

📋 架构概览

项目	配置
Jenkins 版本	2.516.2 (war 包)
操作系统	Ubuntu 22.04 LTS
容器运行时	containerd
部署方式	Kubernetes (YAML Manifests)
集群角色	Master (StatefulSet), Agent (Dynamic Pods)
存储	PersistentVolume + PersistentVolumeClaim
网络	Ingress (Nginx) 对外暴露

🖥️ 节点信息

IP 地址	Hostname	Role
192.168.122.96	k8s-master	Control-plane
192.168.122.190	k8s-node1	Worker
192.168.122.79	k8s-node2	Worker

💡 说明：Jenkins Master 将固定调度到 k8s-node1 节点以确保稳定性。

---

✅ 第一步：基础镜像制作

本方案的核心是构建两个自定义的 Docker 镜像：jenkins-master 和 jenkins-agent。

1. Jenkins Master 镜像

文件结构

.
├── Dockerfile.master
├── entrypoint.sh
├── jdk-17.0.12    // JDK 17 安装目录
└── jenkins.war    // Jenkins 2.516.2 WAR 包

Dockerfile (Dockerfile.master)

# Dockerfile.master
FROM ubuntu:22.04

LABEL maintainer="devops@example.com"

# 设置环境变量
ENV JAVA_HOME=/data/jdk-17.0.12 \
    JENKINS_HOME=/var/jenkins_home \
    JENKINS_VERSION=2.516.2

# 使用阿里云镜像源加速 apt
RUN sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list && \
    sed -i 's/security.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list

# 创建目录并复制 JDK
RUN mkdir -p /data
COPY jdk-17.0.12 /data/jdk-17.0.12

# 安装基础工具
RUN apt-get update && \
    apt-get install -y wget net-tools curl libfreetype6 fonts-dejavu-core fontconfig  && \
    rm -rf /var/lib/apt/lists/*

# 创建 jenkins 用户 (UID=1000)
RUN groupadd --gid 1000 jenkins && \
    useradd -m -u 1000 -g jenkins -d /home/jenkins jenkins && \
    mkdir -p /var/jenkins_home && \
    chown -R jenkins:jenkins /var/jenkins_home

# 添加 Java 到 PATH
ENV PATH="${JAVA_HOME}/bin:${PATH}"

WORKDIR /home/jenkins

# 复制 Jenkins WAR 包
COPY jenkins.war /home/jenkins/jenkins.war
RUN chown jenkins:jenkins jenkins.war

# 使用 root 用户运行
USER root

# 复制启动脚本
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

EXPOSE 8080

ENTRYPOINT ["/entrypoint.sh"]

启动脚本 (entrypoint.sh)

#!/bin/bash
set -e

# 指向正确的 WAR 路径
JAR="/home/jenkins/jenkins.war"

# 确保 Jenkins home 目录权限正确
chown -R jenkins:jenkins $JENKINS_HOME

exec java \
  -Djenkins.install.runSetupWizard=false \
  -Djenkins.CLI.disabled=true \
  -Djenkins.model.Jenkins.pluginsUrl=https://mirrors.huaweicloud.com/jenkins/latest/plugins/%s.hpi \
  -Djenkins.model.Jenkins.updateCenterUrl=https://mirrors.huaweicloud.com/jenkins/updates/update-center.json \
  -Dhudson.model.DownloadService.noSignatureCheck=true \
  -jar ${JAR} \
  --httpPort=8080 \
  --webroot=$JENKINS_HOME/war \
  --argumentsRealm.passwd.jenkins=jenkins \
  --argumentsRealm.roles.jenkins=admin

🔐 关键配置说明：

• -Djenkins.install.runSetupWizard=false: 跳过初始化向导。
• --argumentsRealm: 内置用户 jenkins，密码为 jenkins。
• pluginsUrl 和 updateCenterUrl: 使用华为云镜像，解决插件下载慢的问题。

构建命令

docker build -f Dockerfile.master -t swr.cn-east-3.myhuaweicloud.com/bocheng-test/jenkins-master:v1 .

---

2. Jenkins Agent 镜像

文件结构

.
├── Dockerfile.agent
├── apache-maven-3.8.8
├── jdk-17.0.12
└── jenkins-agent.sh

Dockerfile (Dockerfile.agent)

# Dockerfile.agent
FROM ubuntu:22.04

LABEL maintainer="devops@example.com"

# 设置环境变量
ENV AGENT_WORKDIR=/home/jenkins/agent \
    JAVA_HOME=/data/jdk-17.0.12 \
    MAVEN_HOME=/opt/apache-maven-3.8.8 \
    NVM_DIR=/home/jenkins/.nvm \
    USER=jenkins

# 添加 Java 和 Maven 到 PATH
ENV PATH="${JAVA_HOME}/bin:${MAVEN_HOME}/bin:${PATH}"

# 使用阿里云镜像源
RUN sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list && \
    sed -i 's/security.ubuntu.com/mirrors.aliyun.com/g' /etc/apt/sources.list

# 安装基础工具
RUN apt-get update && \
    apt-get install -y curl git vim python3-pip && \
    rm -rf /var/lib/apt/lists/*

# 创建 jenkins 用户
RUN groupadd --gid 1000 jenkins && \
    useradd -m -u 1000 -g jenkins -d /home/jenkins jenkins

# 创建 agent 工作目录
RUN mkdir -p $AGENT_WORKDIR && \
    chown -R jenkins:jenkins $AGENT_WORKDIR

# 拷贝 JDK 和 Maven
COPY jdk-17.0.12 /data/jdk-17.0.12
COPY apache-maven-3.8.8 /opt/apache-maven-3.8.8

# 设置 Maven 权限
RUN chmod +x /opt/apache-maven-3.8.8/bin/mvn && \
    chown -R jenkins:jenkins /opt/apache-maven-3.8.8

# 使用 heredoc 安装 NVM, Node.js v18.20.6 和 pnpm
RUN su - jenkins <<'EOF'
    export NVM_DIR="$HOME/.nvm"
    mkdir -p "$NVM_DIR"

    # 安装 NVM (使用 Gitee 镜像)
    curl -fsSL https://gitee.com/cubxxw/nvm/raw/master/install.sh | \
        NVM_DIR='$NVM_DIR' \
        PROFILE=/dev/null \
        bash

    # 加载 NVM
    [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"

    # 设置国内镜像源
    nvm node_mirror https://npmmirror.com/mirrors/node/
    nvm npm_mirror https://npmmirror.com/mirrors/npm/

    # 安装 Node.js v18.20.6
    nvm install v18.20.6
    nvm use v18.20.6

    # 设置 npm 国内源并安装 pnpm
    npm config set registry https://registry.npmmirror.com
    npm install -g pnpm

    # 创建自动加载脚本
    mkdir -p "$HOME/.profile.d"
    cat > "$HOME/.profile.d/nvm.sh" <<'INNER'
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"
INNER
EOF

# 拷贝启动脚本
COPY jenkins-agent.sh /usr/local/bin/jenkins-agent.sh
RUN chmod +x /usr/local/bin/jenkins-agent.sh && \
    chown jenkins:jenkins /usr/local/bin/jenkins-agent.sh

# 确保所有文件归属 jenkins 用户
RUN chown -R jenkins:jenkins /home/jenkins

# 关键设置：切换用户并定义入口点
USER jenkins
WORKDIR $AGENT_WORKDIR
ENTRYPOINT ["/usr/local/bin/jenkins-agent.sh"]

✅ 功能亮点：

• 预装 JDK 17、Maven 3.8.8、Node.js 18.20.6、pnpm。
• 自动配置国内镜像源（npmmirror.com）。
• 使用 su - jenkins 在非登录 shell 中执行复杂的安装逻辑。

启动脚本 (jenkins-agent.sh)

#!/bin/bash
# jenkins-agent.sh

set -e

if [ -z "$1" ] || [ -z "$2" ]; then
  echo "❌ 错误：缺少 JNLP secret 或 agent name"
  echo "💡 用法: $0 <secret> <agent-name>"
  exit 1
fi

export AGENT_WORKDIR="/home/jenkins/agent"
export JAVA_HOME="/data/jdk-17.0.12"
export PATH="$JAVA_HOME/bin:$PATH"

cd "$AGENT_WORKDIR"

# 从 Jenkins Master 下载 agent.jar
curl -fsSL -o agent.jar http://jenkins-master.jenkins.svc.cluster.local:8080/jnlpJars/agent.jar

exec java \
  -Duser.home=/home/jenkins \
  -Djava.awt.headless=true \
  -jar ./agent.jar \
  -url http://jenkins-master.jenkins.svc.cluster.local:8080 \
  -webSocket \
  -workDir "$AGENT_WORKDIR" \
  -headless \
  "$1" "$2"

🔗 服务发现：使用 Kubernetes 内部 DNS jenkins-master.jenkins.svc.cluster.local 连接 Master。

构建命令

docker build -f Dockerfile.agent -t swr.cn-east-3.myhuaweicloud.com/bocheng-test/jenkins-agent:v1 .

---

✅ 第二步：准备 Kubernetes 存储

为 Jenkins Master 和 Maven 缓存创建持久化存储。

1. Jenkins Master 存储

PersistentVolume (jenkins-pv.yaml)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/jenkins-pv
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - k8s-node1  # 替换为实际节点名

PersistentVolumeClaim (jenkins-pvc.yaml)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: local-storage
  volumeName: jenkins-pv

---

2. Maven 缓存存储

PersistentVolume (local-pv-maven.yaml)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-maven-local
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/jenkins/maven-repo
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - k8s-node1  # 替换为实际节点名

PersistentVolumeClaim (local-pvc-maven.yaml)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: maven-cache-pvc
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: local-storage
  volumeName: pv-maven-local

📌 操作步骤：

1. 在 k8s-node1 上创建目录：sudo mkdir -p /data/jenkins-pv /data/jenkins/maven-repo
2. 授权：sudo chmod 755 /data/jenkins-pv /data/jenkins/maven-repo
3. 应用 YAML 文件。

---

✅ 第三步：配置 RBAC 权限

为 Jenkins ServiceAccount 授予必要的 Kubernetes API 访问权限。

方案一：最小权限原则（推荐）

# rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: jenkins
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "secrets", "configmaps", "events"]
    verbs: ["get", "list", "watch", "create", "delete", "update"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: jenkins

方案二：管理员权限（测试环境）

# jenkins-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: jenkins

⚠️ 生产建议：使用方案一，遵循最小权限原则。

---

✅ 第四步：部署 Jenkins Master

使用 StatefulSet 部署 Jenkins Master。

Jenkins Master YAML (jenkins-master.yaml)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: jenkins
  namespace: jenkins
spec:
  serviceName: jenkins-headless
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-master
  template:
    metadata:
      labels:
        app: jenkins-master
    spec:
      serviceAccountName: jenkins
      # 固定调度到 k8s-node1
      nodeSelector:
        kubernetes.io/hostname: k8s-node1
      # 容忍 control-plane/master 节点污点
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      # 挂载 PVC
      volumes:
        - name: jenkins-home
          persistentVolumeClaim:
            claimName: jenkins-pvc   # 复用已有 PVC
      containers:
        - name: jenkins
          image: swr.cn-east-3.myhuaweicloud.com/bocheng-test/jenkins-master:v4
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
              name: http
            - containerPort: 50000
              name: jnlp
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
          resources:
            requests:
              memory: "2Gi"
              cpu: "500m"
            limits:
              memory: "4Gi"
              cpu: "1000m"
---
# Headless Service
apiVersion: v1
kind: Service
metadata:
  name: jenkins-headless
  namespace: jenkins
spec:
  clusterIP: None
  selector:
    app: jenkins-master
  ports:
    - port: 8080
      targetPort: 8080
      name: http
    - port: 50000
      targetPort: 50000
      name: jnlp
---
# 外部访问 Service
apiVersion: v1
kind: Service
metadata:
  name: jenkins-master
  namespace: jenkins
spec:
  type: LoadBalancer
  selector:
    app: jenkins-master
  ports:
    - name: http
      port: 8080
      targetPort: 8080
    - name: jnlp
      port: 50000
      targetPort: 50000

🔧 关键配置：

• nodeSelector: 将 Master 固定在 k8s-node1。
• tolerations: 允许调度到 master 节点。
• volumeMounts: 挂载 jenkins-pvc 实现数据持久化。

---

✅ 第五步：部署 Ingress 控制器与路由

1. 部署 Ingress-Nginx 控制器

使用提供的 ingress-nginx-controller.yaml 文件进行部署。该文件包含完整的 Deployment、Service、RBAC 等配置。

🔗 注意：此控制器将用于暴露 Jenkins 和其他应用。

---

2. 创建 Jenkins Ingress 规则 (jenkins-ingress.yaml)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins-ingress
  namespace: jenkins
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
    nginx.ingress.kubernetes.io/websocket-services: jenkins-master
spec:
  ingressClassName: nginx
  rules:
    - host: jenkins.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: jenkins-master
                port:
                  number: 8080

⚙️ 注解说明：

• proxy-* timeout: 解决长时间构建任务超时问题。
• websocket-services: 启用 WebSocket 支持，确保 JNLP 连接稳定。

---

✅ 第六步：Jenkins UI 核心配置

通过 Jenkins UI 或 REST API 配置 Kubernetes 云集成。

1. Kubernetes Cloud 配置

{
  "cloud": {
    "name": "Kubernetes",
    "type": "kubernetes",
    "kubernetesUrl": "https://kubernetes.default.svc.cluster.local",
    "disableHttpsCertificateCheck": true,
    "namespace": "jenkins",
    "credentials": {
      "type": "Secret text"
    },
    "connection": {
      "useWebSocket": true,
      "useDirectConnection": false
    },
    "jenkinsUrl": "http://jenkins-master.jenkins.svc.cluster.local:8080",
    "jenkinsTunnel": "",
    "timeouts": {
      "connectionTimeout": 5,
      "readTimeout": 15,
      "podStartupTimeoutSeconds": 600,
      "containerCleanupTimeoutSeconds": 5
    },
    "limits": {
      "containerCap": 10,
      "maxRequestsPerSecond": -1,
      "maxConnectionsToKubernetesApi": 32
    },
    "podLabels": [
      {
        "key": "jenkins",
        "value": "jnlp"
      }
    ],
    "podRetention": "Never",
    "agentDockerRegistry": "",
    "injectPSSRestrictedSecurityContext": false,
    "passControllerEnvironmentVariablesToAgents": false
  }
}

🔑 核心参数：

• kubernetesUrl: 使用 Kubernetes 服务账户的默认 API 服务器地址。
• disableHttpsCertificateCheck: 测试环境可开启，生产环境应禁用。
• jenkinsUrl: Agent 连接 Master 的内部 URL。

---

2. Pod Template 配置

{
  "podTemplate": {
    "name": "jnlp-slave",
    "namespace": "jenkins",
    "label": "jnlp-slave",
    "usage": "Only run jobs tied to this node",
    "containerTemplates": [
      {
        "name": "jnlp",
        "image": "swr.cn-east-3.myhuaweicloud.com/bocheng-test/jenkins-agentv2",
        "alwaysPullImage": false,
        "workingDir": "/home/jenkins/agent",
        "command": "",
        "args": "${computer.jnlpMac} ${computer.name}",
        "ttyEnabled": true,
        "envVars": [
          {
            "key": "CONTAINERD_ADDRESS",
            "value": "/run/containerd/containerd.sock"
          },
          {
            "key": "CONTAINERD_NAMESPACE",
            "value": "default"
          }
        ]
      }
    ],
    "envVars": [],
    "volumes": [
      {
        "type": "HostPathVolume",
        "hostPath": "/run/containerd/containerd.sock",
        "mountPath": "/run/containerd/containerd.sock",
        "readOnly": false
      },
      {
        "type": "HostPathVolume",
        "hostPath": "/etc/localtime",
        "mountPath": "/etc/localtime",
        "readOnly": false
      },
      {
        "type": "PersistentVolumeClaim",
        "claimName": "maven-cache-pvc",
        "mountPath": "/root/.m2/repository",
        "readOnly": false
      }
    ],
    "rawYaml": "spec:\n  containers:\n  - name: jnlp\n    securityContext:\n      privileged: true\n      capabilities:\n        add:\n        - SYS_ADMIN\n        - SETFCAP\n      allowPrivilegeEscalation: true\n    volumeMounts:\n    - name: containers-storage\n      mountPath: /var/lib/containers\n    - name: dev-mapper\n      mountPath: /dev/mapper\n    - name: containerd-socket\n      mountPath: /run/containerd/containerd.sock\n    - name: local-time\n      mountPath: /etc/localtime\n    - name: maven-cache\n      mountPath: /root/.m2/repository\n  volumes:\n  - name: containers-storage\n    emptyDir: {}\n  - name: dev-mapper\n    hostPath:\n      path: /dev/mapper\n  - name: containerd-socket\n    hostPath:\n      path: /run/containerd/containerd.sock\n  - name: local-time\n    hostPath:\n      path: /etc/localtime\n  - name: maven-cache\n    persistentVolumeClaim:\n      claimName: maven-cache-pvc\n  tolerations:\n  - key: \"node-role.kubernetes.io/master\"\n    operator: \"Exists\"\n    effect: \"NoSchedule\"",
    "serviceAccount": "jenkins",
    "runAsUser": 0,
    "runAsGroup": 0,
    "imagePullSecrets": [
      {
        "name": "swr-secret"
      }
    ]
  }
}

🔧 高级特性：

• rawYaml: 定义更精细的安全上下文（privileged）、挂载路径和容忍度。
• volumes: 挂载宿主机的 containerd.sock，实现容器内构建镜像。
• maven-cache-pvc: 挂载 PVC 以缓存 Maven 依赖，加速构建。

---

✅ 第七步：核心插件列表

插件名称	作用说明
Kubernetes Plugin	核心插件！提供与 Kubernetes 集群集成的能力，支持动态创建 Pod 作为 Jenkins Agent。
Kubernetes Credentials Plugin	提供在 Jenkins 中管理 Kubernetes 凭据（如 Token、CA 证书）的功能。
Kubernetes Client API Plugin	提供对 Kubernetes API 的访问能力，是 Kubernetes Plugin 的依赖。
Kubernetes CLI Plugin	允许 Jenkins 使用 kubectl 命令行工具操作 Kubernetes 集群。

💡 提示：这些插件可通过华为云镜像快速下载安装。

---

✅ 总结

你已完成以下目标：

✅ 成功构建了包含 JDK、Maven、Node.js 的 Jenkins Master 和 Agent 镜像
✅ 在 Kubernetes 上部署了高可用的 Jenkins Master 并实现数据持久化
✅ 配置了 RBAC 权限，确保安全访问 Kubernetes API
✅ 通过 Ingress 实现了 Jenkins 的外部访问
✅ 在 Jenkins UI 中配置了 Kubernetes 云，实现了动态 Agent 扩缩容
✅ Agent 可访问宿主机的 containerd，支持 CI/CD 中的镜像构建

此方案为构建企业级 CI/CD 流水线提供了坚实的基础。