kubernetes 基于jenkins spinnaker的ci/cd实践一增加制品镜像扫描

kubernetes 基于jenkins spinnaker的ci/cd实践一增加制品镜像扫描

前言：

早期jenkins承担了kubernetes中的ci/cd全部功能Jenkins Pipeline演进,这里准备将cd持续集成拆分出来到spinnaker!当然了 正常的思路应该是将jenkins spinnaker的用户账号先打通集成ldap.spinnaker账号系统已经集成ldap.jenkins之前也做过相关的试验。这里关于jenkins集成ldap的步骤就先省略了。毕竟目标是拆分pipeline流水线实践。账号系统 互通还没有那么有紧迫性！。当然了第一步我觉得还是少了镜像的扫描的步骤，先搞一波镜像的扫描！毕竟安全才是首位的

关于jenkins流水线pipeline的镜像扫描

注：image 镜像仓库使用了harbor

Trivy

查看harbor的api (不能与流水线集成提供扫描报告)

import groovy.json.JsonSlurper //Docker 镜像仓库信息 registryServer = "harbor.layame.com" projectName = "${JOB_NAME}".split('-')[0] repoName = "${JOB_NAME}" imageName = "${registryServer}/${projectName}/${repoName}" harborAPI = "" //pipeline pipeline{ agent { node { label "build01"}} //设置构建触发器 triggers { GenericTrigger( causeString: 'Generic Cause', genericVariables: [[defaultValue: '', key: 'branchName', regexpFilter: '', value: '$.ref']], printContributedVariables: true, printPostContent: true, regexpFilterExpression: '', regexpFilterText: '', silentResponse: true, token: 'spinnaker-nginx-demo') } stages{ stage("CheckOut"){ steps{ script{ srcUrl = "https://gitlab.xxxx.com/zhangpeng/spinnaker-nginx-demo.git" branchName = branchName - "refs/heads/" currentBuild.description = "Trigger by ${branchName}" println("${branchName}") checkout([$class: 'GitSCM', branches: [[name: "${branchName}"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'gitlab-admin-user', url: "${srcUrl}"]]]) } } } stage("Push Image "){ steps{ script{ withCredentials([usernamePassword(credentialsId: 'harbor-admin-user', passwordVariable: 'password', usernameVariable: 'username')]) { sh """ sed -i -- "s/VER/${branchName}/g" app/index.html docker login -u ${username} -p ${password} ${registryServer} docker build -t ${imageName}:${data} . docker push ${imageName}:${data} docker rmi ${imageName}:${data} """ } } } } stage("scan Image "){ steps{ script{ withCredentials([usernamePassword(credentialsId: 'harbor-admin-user', passwordVariable: 'password', usernameVariable: 'username')]) { sh """ sed -i -- "s/VER/${branchName}/g" app/index.html docker login -u ${username} -p ${password} ${registryServer} docker build -t ${imageName}:${data} . docker push ${imageName}:${data} docker rmi ${imageName}:${data} """ } } } } stage("Trigger File"){ steps { script{ sh """ echo IMAGE=${imageName}:${data} >trigger.properties echo ACTION=DEPLOY >> trigger.properties cat trigger.properties """ archiveArtifacts allowEmptyArchive: true, artifacts: 'trigger.properties', followSymlinks: false } } } } }

改造spinnaker-nginx-demo pipeline

依旧拿我spinnaker-nginx-demo的实例去验证，参见：关于jenkins的配置-spinnaker-nginx-demo，修改pipeline如下：

//Docker 镜像仓库信息 registryServer = "harbor.xxxx.com" projectName = "${JOB_NAME}".split('-')[0] repoName = "${JOB_NAME}" imageName = "${registryServer}/${projectName}/${repoName}" //pipeline pipeline{ agent { node { label "build01"}} //设置构建触发器 triggers { GenericTrigger( causeString: 'Generic Cause', genericVariables: [[defaultValue: '', key: 'branchName', regexpFilter: '', value: '$.ref']], printContributedVariables: true, printPostContent: true, regexpFilterExpression: '', regexpFilterText: '', silentResponse: true, token: 'spinnaker-nginx-demo') } stages{ stage("CheckOut"){ steps{ script{ srcUrl = "https://gitlab.xxxx.com/zhangpeng/spinnaker-nginx-demo.git" branchName = branchName - "refs/heads/" currentBuild.description = "Trigger by ${branchName}" println("${branchName}") checkout([$class: 'GitSCM', branches: [[name: "${branchName}"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'gitlab-admin-user', url: "${srcUrl}"]]]) } } } stage("Push Image "){ steps{ script{ withCredentials([usernamePassword(credentialsId: 'harbor-admin-user', passwordVariable: 'password', usernameVariable: 'username')]) { sh """ sed -i -- "s/VER/${branchName}/g" app/index.html docker login -u ${username} -p ${password} ${registryServer} docker build -t ${imageName}:${data} . docker push ${imageName}:${data} docker rmi ${imageName}:${data} """ } } } } stage("scan Image "){ steps{ script{ withCredentials([usernamePassword(credentialsId: 'harbor-admin-user', passwordVariable: 'password', usernameVariable: 'username')]) { harborAPI = "https://harbor.xxxx.com/api/v2.0/projects/${projectName}/repositories/${repoName}" apiURL = "artifacts/${data}/scan" sh """ curl -X POST "${harborAPI}/${apiURL}" -H "accept: application/json" -u ${username}:${password} """ } } } } stage("Trigger File"){ steps { script{ sh """ echo IMAGE=${imageName}:${data} >trigger.properties echo ACTION=DEPLOY >> trigger.properties cat trigger.properties """ archiveArtifacts allowEmptyArchive: true, artifacts: 'trigger.properties', followSymlinks: false } } } } }

参考阳明大佬清理镜像的pipeline脚本进行了修改，增加了scan Image的stage! 其实都是参照harbor的api文档来的，更为详细的可以参考harbor的官方api。

触发jenkins构建

登陆harbor仓库验证：

anchore-engine

anchore-engine helm的安装

anchore-engine，是无意间搜jenkins scan image这些关键词的时候网上看到的:anchore-engine]# helm repo add anchore https://charts.anchore.io [root@k8s-master-01 anchore-engine]# helm repo list

[root@k8s-master-01 anchore-engine]# helm search repo anchore/anchore-engine [root@k8s-master-01 anchore-engine]# helm repo update [root@k8s-master-01 anchore-engine]# helm search repo anchore/anchore-engine [root@k8s-master-01 anchore-engine]# helm fetch anchore/anchore-engine

​

[root@k8s-master-01 anchore-engine]# ls [root@k8s-master-01 anchore-engine]# tar zxvf anchore-engine-1.15.1.tgz [root@k8s-master-01 anchore-engine]# cd anchore-engine

vim values.yaml

helm install anchore-engine -f values.yaml . -n anchore-engine

jenkins的配置

构建流水线：

由于这里是测试就先搞了一下使用Anchore Enine来完善DevSecOps工具链里面的demo（修改了一下构建节点，github仓库还有dockerhub仓库秘钥）：

jenkins 新建pipeline任务anchore-enchore

pipeline { agent { node { label "build01"}} environment { registry = "duiniwukenaihe/spinnaker-cd" //仓库地址，用于把镜像push到镜像仓库。按照实际情况修改 registryCredential = 'duiniwukenaihe' //用于登陆镜像仓库的凭证，按照实际情况修改 } stages { //jenkins从代码仓库里下载代码 stage('Cloning Git') { steps { git 'https://github.com.cnpmjs.org/duiniwukenaihe/docker-dvwa.git' } } //构建镜像 stage('Build Image') { steps { script { app = docker.build(registry+ ":$BUILD_NUMBER") } } } //把镜像推送到仓库 stage('Push Image') { steps { script { docker.withRegistry('', registryCredential ) { app.push() } } } } //镜像扫描 stage('Container Security Scan') { steps { sh 'echo "'+registry+':$BUILD_NUMBER `pwd`/Dockerfile" > anchore_images' anchore engineRetries: "240", name: 'anchore_images' } } stage('Cleanup') { steps { sh script: "docker rmi " + registry+ ":$BUILD_NUMBER" } } } }

注：github.com修改为github.com.cnpmjs.org就是为了加速....毕竟墙裂无法pull的动代码

运行pipeline任务

docker-compose 安装anchore-engine

按照教程使用Anchore Enine来完善DevSecOps工具链搞了一个docker-compose的部署方式：注：我的集群默认cri 是containerd，k8s-node-06节点是docker做运行时，且不参与调度，anchore-engine就准备在这台服务器上面安装了！内网ip：10.0.4.18。

前提安装docker-compose:

docker-compose up -d

直接使用了默认的yaml文件并没有进行额外修改，比较前期只是测试。

# curl > docker-compose.yaml # docker-compose up -d

# This is a docker-compose file for development purposes. It refereneces unstable developer builds from the HEAD of master branch in https://github.com/anchore/anchore-engine # For a compose file intended for use with a released version, see https://engine.anchore.io/docs/quickstart/ # --- version: '2.1' volumes: anchore-db-volume: # Set this to 'true' to use an external volume. In which case, it must be created manually with "docker volume create anchore-db-volume" external: false services: # The primary API endpoint service api: image: anchore/anchore-engine:v1.0.0 depends_on: - db - catalog ports: - "8228:8228" logging: driver: "json-file" options: max-size: 100m environment: - ANCHORE_ENDPOINT_HOSTNAME=api - ANCHORE_ADMIN_PASSWORD=foobar - ANCHORE_DB_HOST=db - ANCHORE_DB_PASSWORD=mysecretpassword command: ["anchore-manager", "service", "start", "apiext"] # Catalog is the primary persistence and state manager of the system catalog: image: anchore/anchore-engine:v1.0.0 depends_on: - db logging: driver: "json-file" options: max-size: 100m expose: - 8228 environment: - ANCHORE_ENDPOINT_HOSTNAME=catalog - ANCHORE_ADMIN_PASSWORD=foobar - ANCHORE_DB_HOST=db - ANCHORE_DB_PASSWORD=mysecretpassword command: ["anchore-manager", "service", "start", "catalog"] queue: image: anchore/anchore-engine:v1.0.0 depends_on: - db - catalog expose: - 8228 logging: driver: "json-file" options: max-size: 100m environment: - ANCHORE_ENDPOINT_HOSTNAME=queue - ANCHORE_ADMIN_PASSWORD=foobar - ANCHORE_DB_HOST=db - ANCHORE_DB_PASSWORD=mysecretpassword command: ["anchore-manager", "service", "start", "simplequeue"] policy-engine: image: anchore/anchore-engine:v1.0.0 depends_on: - db - catalog expose: - 8228 logging: driver: "json-file" options: max-size: 100m environment: - ANCHORE_ENDPOINT_HOSTNAME=policy-engine - ANCHORE_ADMIN_PASSWORD=foobar - ANCHORE_DB_HOST=db - ANCHORE_DB_PASSWORD=mysecretpassword - ANCHORE_VULNERABILITIES_PROVIDER=grype command: ["anchore-manager", "service", "start", "policy_engine"] analyzer: image: anchore/anchore-engine:v1.0.0 depends_on: - db - catalog expose: - 8228 logging: driver: "json-file" options: max-size: 100m environment: - ANCHORE_ENDPOINT_HOSTNAME=analyzer - ANCHORE_ADMIN_PASSWORD=foobar - ANCHORE_DB_HOST=db - ANCHORE_DB_PASSWORD=mysecretpassword volumes: - /analysis_scratch command: ["anchore-manager", "service", "start", "analyzer"] db: image: "postgres:9" volumes: - anchore-db-volume:/var/lib/postgresql/data environment: - POSTGRES_PASSWORD=mysecretpassword expose: - 5432 logging: driver: "json-file" options: max-size: 100m healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] # # Uncomment this section to add a prometheus instance to gather metrics. This is mostly for quickstart to demonstrate prometheus metrics exported # prometheus: # image: docker.io/prom/prometheus:latest # depends_on: # - api # volumes: # - ./anchore-prometheus.yml:/etc/prometheus/prometheus.yml:z # logging: # driver: "json-file" # options: # max-size: 100m # ports: # - "9090:9090" # # # Uncomment this section to run a swagger UI service, for inspecting and interacting with the anchore engine API via a browser (by default, change if needed in both sections below) # swagger-ui-nginx: # image: docker.io/nginx:latest # depends_on: # - api # - swagger-ui # ports: # - "8080:8080" # volumes: # - ./anchore-swaggerui-nginx.conf:/etc/nginx/nginx.conf:z # logging: # driver: "json-file" # options: # max-size: 100m # swagger-ui: # image: docker.io/swaggerapi/swagger-ui # environment: # - URL=http://localhost:8080/v1/swagger.json # logging: # driver: "json-file" # options: # max-size: 100m #

[root@k8s-node-06 anchore]# docker-compose ps Name Command State Ports ------------------------------------------------------------------------------------------------ anchore_analyzer_1 /docker-entrypoint.sh anch ... Up (healthy) 8228/tcp anchore_api_1 /docker-entrypoint.sh anch ... Up (healthy) 0.0.0.0:8228->8228/tcp anchore_catalog_1 /docker-entrypoint.sh anch ... Up (healthy) 8228/tcp anchore_db_1 docker-entrypoint.sh postgres Up (healthy) 5432/tcp anchore_policy-engine_1 /docker-entrypoint.sh anch ... Up (healthy) 8228/tcp anchore_queue_1 /docker-entrypoint.sh anch ... Up (healthy) 8228/tcp

修改jenkins配置

Pipeline Test：

错误的猜测：

helm部署不可以初步估计是我的containerd不是docker的原因?又或者是服务端版本太高？

随之而来的问题：

如何扫描私有仓库镜像？

但是随之问题又来了：anchore-enchorepipeline中镜像仓库默认的是dockerhub,我的仓库是私有harbor仓库，spinnaker-nginx-demo的应用pipeline增加扫描都跑不起来.......

//Docker 镜像仓库信息 registryServer = "harbor.xxxx.com" projectName = "${JOB_NAME}".split('-')[0] repoName = "${JOB_NAME}" imageName = "${registryServer}/${projectName}/${repoName}" //pipeline pipeline{ agent { node { label "build01"}} //设置构建触发器 triggers { GenericTrigger( causeString: 'Generic Cause', genericVariables: [[defaultValue: '', key: 'branchName', regexpFilter: '', value: '$.ref']], printContributedVariables: true, printPostContent: true, regexpFilterExpression: '', regexpFilterText: '', silentResponse: true, token: 'spinnaker-nginx-demo') } stages{ stage("CheckOut"){ steps{ script{ srcUrl = "https://gitlab.xxxx.com/zhangpeng/spinnaker-nginx-demo.git" branchName = branchName - "refs/heads/" currentBuild.description = "Trigger by ${branchName}" println("${branchName}") checkout([$class: 'GitSCM', branches: [[name: "${branchName}"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'gitlab-admin-user', url: "${srcUrl}"]]]) } } } stage("Push Image "){ steps{ script{ withCredentials([usernamePassword(credentialsId: 'harbor-admin-user', passwordVariable: 'password', usernameVariable: 'username')]) { sh """ sed -i -- "s/VER/${branchName}/g" app/index.html docker login -u ${username} -p ${password} ${registryServer} docker build -t ${imageName}:${data} . docker push ${imageName}:${data} docker rmi ${imageName}:${data} """ } } } } stage('Container Security Scan') { steps { script{ sh """ echo "开始扫描" echo "${imageName}:${data} ${WORKSPACE}/Dockerfile" > anchore_images """ anchore engineRetries: "360",forceAnalyze: true, name: 'anchore_images' } } } stage("Trigger File"){ steps { script{ sh """ echo IMAGE=${imageName}:${data} >trigger.properties echo ACTION=DEPLOY >> trigger.properties cat trigger.properties """ archiveArtifacts allowEmptyArchive: true, artifacts: 'trigger.properties', followSymlinks: false } } } } }

github issuse找到的灵感：

怎么回事？功夫不负有心人看了一遍github anchore仓库的issue：anchore]# docker exec -it d21c8ed1064d bash [anchore@d21c8ed1064d anchore-engine]$ anchore-cli registry add harbor.xxxx.com zhangpeng xxxxxx [anchore@d21c8ed1064d anchore-engine]$ anchore-cli --url --u admin --p foobar --debug image add harbor.layame.com/spinnaker/spinnaker-nginx-demo:202111192008

[anchore@d21c8ed1064d anchore-engine]$ anchore-cli image list

顺便修改一下helm搭建的anchore-enchore

同理 我现在怀疑我的helm部署的harbor也是这错误...开始怀疑错了，修改一下试试！

[root@k8s-master-01 anchore-engine]# kubectl get pods -n anchore-engine NAME READY STATUS RESTARTS AGE anchore-engine-anchore-engine-analyzer-fcf9ffcc8-dv955 1/1 Running 0 10h anchore-engine-anchore-engine-api-7f98dc568-j6tsz 1/1 Running 0 10h anchore-engine-anchore-engine-catalog-754b996b75-q5hqg 1/1 Running 0 10h anchore-engine-anchore-engine-policy-745b6778f7-hbsvx 1/1 Running 0 10h anchore-engine-anchore-engine-simplequeue-695df4498-wgss4 1/1 Running 0 10h anchore-engine-postgresql-9cdbb5f7f-4dcnk 1/1 Running 0 10h [root@k8s-master-01 anchore-engine]# kubectl get svc -n anchore-engine NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE anchore-engine-anchore-engine-api ClusterIP 172.19.255.231 8228/TCP 10h anchore-engine-anchore-engine-catalog ClusterIP 172.19.254.163 8082/TCP 10h anchore-engine-anchore-engine-policy ClusterIP 172.19.254.91 8087/TCP 10h anchore-engine-anchore-engine-simplequeue ClusterIP 172.19.253.141 8083/TCP 10h anchore-engine-postgresql ClusterIP 172.19.252.126 5432/TCP 10h [root@k8s-master-01 anchore-engine]# kubectl run -i --tty anchore-cli --restart=Always --image anchore/engine-cli --env ANCHORE_CLI_USER=admin --env ANCHORE_CLI_PASS=xxxxxx --env ANCHORE_CLI_URL=http://172.19.255.231:8228/v1 [anchore@anchore-cli anchore-cli]$ anchore-cli registry add harbor.xxxx.com zhangpeng xxxxxxxx [anchore@anchore-cli anchore-cli]$ anchore-cli --url --u admin --p xxxx --debug image add harbor.xxxx.com/spinnaker/spinnaker-nginx-demo:202111192008

运行jenkins 任务 spinnaker-nginx-demo pipeline

比较一下Trivy与anchore-engine

总结一下：

harbor自定镜像扫描插件tivy，嗯也可以选择clair 貌似也可以与anchore-engine打通 anchore-engine要add参加私有仓库，heml安装拼装地址记得修改cluster.local，如果是自定义集群 anchore-engine比trivy的扫描更为严格 要善于使用--help命令：anchore-cli --help

​

​

​

​

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。