This page looks best with JavaScript enabled

Set Up a Jenkins CI/CD Pipeline with Kubernetes

使用Jenkins进行持续构建与发布应用到kubernetes集群中

 ·  ☕ 5 min read

本文将介绍 jenkins 在kubernetes下的CI/CD发布流程。

imagepng

以下内容仅是个人测试所用，如有不妥，可在下方评论留言，评论可能需要科学上网😂。

jenkins在k8s下，可以实现节点自动调度，无需关注节点，且可以直接使用内部的api-server

使用helm安装jenkins

  1. 创建一个pv，因为容器是无状态的，需要创建一个volume，用于存放jenkins的一些配置信息。

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

    kind: PersistentVolume
apiVersion: v1
metadata:
  name: jenkins-pv
spec:
#  selector:
#      jenkins: master
#  storageClassName: jenkins-pv
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: "/data/jenkins"

  2. 使用helm安装jenkins

    1

    helm install --name jenkins stable/jenkins

    如果开启了rbac可以使用helm upgrade jenkins stable/jenkins --set rbac.install=true进行更新，无需重新安装。

  3. 配置ingress（可选）

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

    apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins
  labels:
    app: jenkins
  annotations:
    kubernetes.io/ingress.class: nginx
  namespace: default
spec:
  rules:
    - host: jenkins.192.168.2.20.xip.io
      http:
        paths:
          - backend:
              serviceName: jenkins
              servicePort: 80
            path: /

    使用kubectl create -f jenkins-ingress.yaml创建。

  4. 到此，jenkins基本安装完成，如果出现问题，可以使用kubectl describe 和kubectl logs进行排查。

配置jenkins

  1. 安装完成过后，默认jenkins的系统配置下会自动添加好kubernetes cloud，如下图

以及配置好的kubernetes pod template

  1. 因为待会儿会将程序部署到k8s中，所以还需要安装kubernetes-cd 👉官方介绍，直接在插件管理搜索下载即可。

  2. 代码及配置地址，👉github

  3. 配置好待会儿会用到的docker hub的账号密码及k8s的kubeconfig

    开始一个新项目

    创建一个pipeline任务，👉官方介绍

    这里使用一个golang写的demo web，在docker容器中进行构建。

    jenkins会自动在k8s集群中创建自己的slave及docker容器，然后把挂载工作目录，完成后自动销毁刚刚创建的容器。

    在pipeline处填写下面的代码，

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

    def label = "mypod-${UUID.randomUUID().toString()}"
   
podTemplate(label: label, 
containers: [
    containerTemplate(name: 'docker', image: 'docker:18.06.2', command: 'cat', ttyEnabled: true)],
volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock')]
)  {
   
    node(label) {
        def ImageName = 'zouhl/demo-app'
   
        stage('Get a Golang project'){
            git url: 'https://github.com/zou2699/demoweb.git'
            container('docker') {
                stage('Build a Go project and docker image')
                sh """
                docker build -t ${ImageName}:${BUILD_NUMBER} .
                """
   
                stage('push image') {
                withDockerRegistry(credentialsId: 'dockerHub') {
                    sh """
                    docker push ${ImageName}:${BUILD_NUMBER}
                    """
                    }
                }
   
                stage('Remove the local docker image') {
                    sh "docker rmi ${ImageName}:${BUILD_NUMBER}"
                }
            }
        }
        stage('deploy to k8s') {
           kubernetesDeploy(
               kubeconfigId: 'kubeconfig',
               configs: 'deploy.yaml',
               enableConfigSubstitution: true,
               )
        }
        stage('create or update ingress') {
            input 'Ready to go?'
            kubernetesDeploy(
               kubeconfigId: 'kubeconfig',
               configs: 'app-ingress.yaml',
               enableConfigSubstitution: true,
               )
        }
}
}

    其中deploy.yaml的镜像名字填写为自己的hub地址。

进行构建

下面是构建和部署的过程，仅作参考。

Started by user admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
‘jenkins-slave-5qg6x-q2mst’ is offline
Agent jenkins-slave-5qg6x-q2mst is provisioned from template Kubernetes Pod Template
Agent specification [Kubernetes Pod Template] (mypod-c92aa982-03f8-42f1-857e-66cca7e96e21): 
* [docker] docker:18.06.2

Running on jenkins-slave-5qg6x-q2mst in /home/jenkins/workspace/demo-app
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Get a Golang project)
[Pipeline] git
Cloning the remote Git repository
Cloning repository https://github.com/zou2699/demoweb.git
 > git init /home/jenkins/workspace/demo-app # timeout=10
Fetching upstream changes from https://github.com/zou2699/demoweb.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/zou2699/demoweb.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/zou2699/demoweb.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/zou2699/demoweb.git # timeout=10
Fetching upstream changes from https://github.com/zou2699/demoweb.git
 > git fetch --tags --progress https://github.com/zou2699/demoweb.git +refs/heads/*:refs/remotes/origin/*
Checking out Revision fbe3e5c71305b8ec166e84f47b3e824235119570 (refs/remotes/origin/master)
Commit message: "add ingress file"
[Pipeline] container
[Pipeline] {
[Pipeline] stage (Build a Go project and docker image)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Build a Go project and docker image
Proceeding
[Pipeline] sh
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fbe3e5c71305b8ec166e84f47b3e824235119570
 > git branch -a -v --no-abbrev # timeout=10
 > git checkout -b master fbe3e5c71305b8ec166e84f47b3e824235119570
 > git rev-list --no-walk 561c05039df2a1a5effd018139b244d2f23fa427 # timeout=10
+ docker build -t zouhl/demo-app:20 .
Sending build context to Docker daemon  11.29MB

Step 1/9 : FROM golang:1.11
1.11: Pulling from library/golang
741437d97401: Already exists
34d8874714d7: Already exists
0a108aa26679: Already exists
7f0334c36886: Already exists
d35724ed4672: Already exists
e75ac00960df: Already exists
08686fc4280e: Already exists
Digest: sha256:baee78b46407117bfa6777b1e0361018b503936f9f878a4e18068207a0a1ff0c
Status: Downloaded newer image for golang:1.11
 ---> 901414995ecd
Step 2/9 : WORKDIR /go/src/app
 ---> Running in bd8a5da92ec0
Removing intermediate container bd8a5da92ec0
 ---> 664736b487fb
Step 3/9 : COPY . .
 ---> b1e7198a6d62
Step 4/9 : RUN go get -d -v ./...
 ---> Running in 1258cae3f35b
Removing intermediate container 1258cae3f35b
 ---> e6c54702bfe2
Step 5/9 : RUN go install -v ./...
 ---> Running in 0e5dfd0fa037
app/vendor/github.com/gin-gonic/gin/json
app/vendor/github.com/golang/protobuf/proto
app/vendor/github.com/gin-contrib/sse
app/vendor/github.com/ugorji/go/codec
app/vendor/gopkg.in/go-playground/validator.v8
app/vendor/gopkg.in/yaml.v2
app/vendor/github.com/mattn/go-isatty
app/vendor/github.com/gin-gonic/gin/render
app/vendor/github.com/gin-gonic/gin/binding
app/vendor/github.com/gin-gonic/gin
app
Removing intermediate container 0e5dfd0fa037
 ---> fc1ea9025ea4
Step 6/9 : RUN rm -rf /go/src/*
 ---> Running in db72fd36ca53
Removing intermediate container db72fd36ca53
 ---> 717e4dca6895
Step 7/9 : WORKDIR /go/bin/
 ---> Running in 037344a8cdb6
Removing intermediate container 037344a8cdb6
 ---> 9acf3bd33867
Step 8/9 : EXPOSE 8080
 ---> Running in cdf47489216b
Removing intermediate container cdf47489216b
 ---> 14a9757f4895
Step 9/9 : CMD ["./app"]
 ---> Running in 072100ec5df1
Removing intermediate container 072100ec5df1
 ---> a9587242060c
Successfully built a9587242060c
Successfully tagged zouhl/demo-app:20
[Pipeline] stage
[Pipeline] { (push image)
[Pipeline] withDockerRegistry
Executing shell script inside container [docker] of pod [jenkins-slave-5qg6x-q2mst]
Executing command: "docker" "login" "-u" "zouhl" "-p" ******** "https://index.docker.io/v1/" 
exit
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
WARNING! Your password will be stored unencrypted in /home/jenkins/workspace/demo-app@tmp/924b09bb-ba00-4377-8a9b-42e0fe5173e1/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

[Pipeline] {
[Pipeline] sh
+ docker push zouhl/demo-app:20
The push refers to repository [docker.io/zouhl/demo-app]
fb0291753bf7: Preparing
c1ff945be567: Preparing
a5e3559c27d7: Preparing
06a8ef03aeff: Preparing
c5472925cd03: Preparing
77afbe0f4fed: Preparing
c3b2e3d53ad1: Preparing
77b4b6493272: Preparing
6257fa9f9597: Preparing
578414b395b9: Preparing
abc3250a6c7f: Preparing
13d5529fd232: Preparing
77b4b6493272: Waiting
6257fa9f9597: Waiting
578414b395b9: Waiting
abc3250a6c7f: Waiting
13d5529fd232: Waiting
77afbe0f4fed: Waiting
c3b2e3d53ad1: Waiting
c5472925cd03: Pushed
fb0291753bf7: Pushed
a5e3559c27d7: Pushed
c3b2e3d53ad1: Layer already exists
77afbe0f4fed: Layer already exists
06a8ef03aeff: Pushed
77b4b6493272: Layer already exists
578414b395b9: Layer already exists
6257fa9f9597: Layer already exists
abc3250a6c7f: Layer already exists
13d5529fd232: Layer already exists
c1ff945be567: Pushed
20: digest: sha256:40a10e22a19508962ef00e469db4091e29633460e56e9e8e4b958e9abf1beff4 size: 2838
[Pipeline] }
[Pipeline] // withDockerRegistry
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Remove the local docker image)
[Pipeline] sh
+ docker rmi zouhl/demo-app:20
Untagged: zouhl/demo-app:20
Untagged: zouhl/demo-app@sha256:40a10e22a19508962ef00e469db4091e29633460e56e9e8e4b958e9abf1beff4
Deleted: sha256:a9587242060c6de1a06293900c04b94e1937b3748f238d377f80a147635ea42c
Deleted: sha256:14a9757f4895ad23bded56d5df14c9b7c5d1a4f2b882ae9da0c4e9d27986a5ea
Deleted: sha256:9acf3bd338676f2772f72a55bebd8a30c27156a88ab0c4115292d24a527b33aa
Deleted: sha256:717e4dca6895a79899982af59313840ddfe321d7217ca00400864aa2bfd2ef19
Deleted: sha256:aeaba5e7b1eeda47794f51e53b5c8435d87aec22dfa04e221fb029ac8840263c
Deleted: sha256:fc1ea9025ea4cab736ae85d3c23de6294dcff7cb9509f19b57265b9c4d5121bc
Deleted: sha256:94209db674e4f9dc6bd2f561cfc9c2c5dac2796a1d82e3e662abc24e11177efb
Deleted: sha256:e6c54702bfe235a25c3b94383dc93e2ff61e67d2aeb9eb870710ae9767d0f990
Deleted: sha256:70c2e14e69370aae7df8921073263454e4945174223b6b2bb427d4474b0a8e4b
Deleted: sha256:b1e7198a6d629bacd799a4fc0c6df060da05f8d953c6ecb6ab44b73df78db959
Deleted: sha256:ded0ab7056905293366361f3ccef519fdf97e9660c80a8fcbe74cbab5fa5779c
Deleted: sha256:664736b487fbbe1d83938e23061060ee8fe863c4a64bc49273e4609cb91cca11
Deleted: sha256:52a27f01339de7babc1c83e843cb76b7ee68e5f438f562e1ee6c61cd0822d1ae
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy to k8s)
[Pipeline] kubernetesDeploy
Starting Kubernetes deployment
Loading configuration: /home/jenkins/workspace/demo-app/deploy.yaml
Finished Kubernetes deployment
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (create or update ingress)
[Pipeline] input
Ready to go?
Proceed or Abort
Applied Deployment: Deployment(apiVersion=extensions/v1beta1, kind=Deployment, metadata=ObjectMeta(annotations=null, clusterName=null, creationTimestamp=2019-03-03T07:40:41Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=4, initializers=null, labels={run=demo-web}, name=demo-web, namespace=default, ownerReferences=[], resourceVersion=399848, selfLink=/apis/extensions/v1beta1/namespaces/default/deployments/demo-web, uid=a54a482c-3d87-11e9-b446-000c29c5c464, additionalProperties={}), spec=DeploymentSpec(minReadySeconds=null, paused=null, progressDeadlineSeconds=2147483647, replicas=2, revisionHistoryLimit=2147483647, rollbackTo=null, selector=LabelSelector(matchExpressions=[], matchLabels={run=demo-web}, additionalProperties={}), strategy=DeploymentStrategy(rollingUpdate=RollingUpdateDeployment(maxSurge=IntOrString(IntVal=1, Kind=null, StrVal=null, additionalProperties={}), maxUnavailable=IntOrString(IntVal=1, Kind=null, StrVal=null, additionalProperties={}), additionalProperties={}), type=RollingUpdate, additionalProperties={}), template=PodTemplateSpec(metadata=ObjectMeta(annotations=null, clusterName=null, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels={run=demo-web}, name=null, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=null, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[], command=[], env=[], envFrom=[], image=zouhl/demo-app:20, imagePullPolicy=Always, lifecycle=null, livenessProbe=null, name=demo-web, ports=[ContainerPort(containerPort=80, hostIP=null, hostPort=null, name=null, protocol=TCP, additionalProperties={})], readinessProbe=null, resources=ResourceRequirements(limits=null, requests=null, additionalProperties={}), securityContext=null, stdin=null, stdinOnce=null, terminationMessagePath=/dev/termination-log, terminationMessagePolicy=File, tty=null, volumeMounts=[], workingDir=null, additionalProperties={})], dnsPolicy=ClusterFirst, hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=null, nodeSelector=null, restartPolicy=Always, schedulerName=default-scheduler, securityContext=PodSecurityContext(fsGroup=null, runAsNonRoot=null, runAsUser=null, seLinuxOptions=null, supplementalGroups=[], additionalProperties={}), serviceAccount=null, serviceAccountName=null, subdomain=null, terminationGracePeriodSeconds=30, tolerations=[], volumes=[], additionalProperties={}), additionalProperties={}), additionalProperties={}), status=DeploymentStatus(availableReplicas=2, collisionCount=null, conditions=[DeploymentCondition(lastTransitionTime=2019-03-03T07:53:53Z, lastUpdateTime=2019-03-03T07:53:53Z, message=Deployment has minimum availability., reason=MinimumReplicasAvailable, status=True, type=Available, additionalProperties={})], observedGeneration=3, readyReplicas=2, replicas=2, unavailableReplicas=null, updatedReplicas=2, additionalProperties={}), additionalProperties={})
Applied Service: Service(apiVersion=v1, kind=Service, metadata=ObjectMeta(annotations=null, clusterName=null, creationTimestamp=2019-03-03T07:40:41Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels=null, name=demo-web, namespace=default, ownerReferences=[], resourceVersion=396922, selfLink=/api/v1/namespaces/default/services/demo-web, uid=a5673bb9-3d87-11e9-b446-000c29c5c464, additionalProperties={}), spec=ServiceSpec(clusterIP=10.1.115.186, externalIPs=[], externalName=null, externalTrafficPolicy=Cluster, healthCheckNodePort=null, loadBalancerIP=null, loadBalancerSourceRanges=[], ports=[ServicePort(name=null, nodePort=31907, port=80, protocol=TCP, targetPort=IntOrString(IntVal=80, Kind=null, StrVal=null, additionalProperties={}), additionalProperties={})], selector={run=demo-web}, sessionAffinity=None, type=NodePort, additionalProperties={}), status=ServiceStatus(loadBalancer=LoadBalancerStatus(ingress=[], additionalProperties={}), additionalProperties={}), additionalProperties={})
Approved by admin
[Pipeline] kubernetesDeploy
Starting Kubernetes deployment
Finished Kubernetes deployment
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS
Page

总结

流程说明

应用构建和发布流程说明。

  1. 用户向Gitlab提交代码，代码中必须包含Dockerfile
  2. 将代码提交到远程仓库
  3. 用户在发布应用时需要填写git仓库地址和分支、服务类型、服务名称、资源数量、实例个数，确定后触发Jenkins自动构建
  4. Jenkins的CI流水线自动编译代码并打包成docker镜像推送到Harbor镜像仓库
  5. Jenkins的CI流水线中包括了自定义脚本，根据我们已准备好的kubernetes的YAML模板，将其中的变量替换成用户输入的选项
  6. 生成应用的kubernetes YAML配置文件
  7. 更新Ingress的配置，根据新部署的应用的名称，在ingress的配置文件中增加一条路由信息
  8. Jenkins调用kubernetes的API，部署应用

这里只是一个简单的例子，可以对ingress进行优化，实现灰度发布。

代码的发布和构建只要按照步骤来，即可。

既然jenkins在k8s中，就要遵循其设计的理念，使用容器进行构建，而不是传统的ssh。

Share on
tux
WRITTEN BY
tux
devops

See Also

What's on this Page