Azure Kubernetes Service 和 Argo CD

Azure Kubernetes Service 和 Argo CD

概述

最近一个Azure的项目需要先做一系列Proof of Concept （PoC）可行性的演示。下面简单的记录一下做过的准备。

首先是通过Gitlab和Terraform来搭建一个Azure的服务器的环境 https://blog./beanxyz/5547469 通过Gitlab来创建容器，并能上传到Azure Container Registry里面 https://blog./beanxyz/5551081https://blog./beanxyz/5565612 打算创建Azure Kubernetes Service，并通过Argo CD来部署第二步创建的容器镜像。

下面看看第三步是如何实现的。

Terraform 创建AKS

关于service principal和相关的环境变量我在第一步已经搭建好了，因此这里直接就不记录了。下面是相关的tf 代码。微软的官方网站提供的演示代码使用的是azurerm 2.x的版本，我这里改成了最新的3.x的版本，因此addon_profile的语法有所不同

main.tf

# Generate random resource group name resource "random_pet" "rg-name" { prefix = var.resource_group_name_prefix } resource "azurerm_resource_group" "k8s" { name = random_pet.rg-name.id location = var.resource_group_location } resource "random_id" "log_analytics_workspace_name_suffix" { byte_length = 8 } resource "azurerm_log_analytics_workspace" "test" { # The WorkSpace name has to be unique across the whole of azure, not just the current subscription/tenant. name = "${var.log_analytics_workspace_name}-${random_id.log_analytics_workspace_name_suffix.dec}" location = var.log_analytics_workspace_location resource_group_name = azurerm_resource_group.k8s.name sku = var.log_analytics_workspace_sku } resource "azurerm_log_analytics_solution" "test" { solution_name = "ContainerInsights" location = azurerm_log_analytics_workspace.test.location resource_group_name = azurerm_resource_group.k8s.name workspace_resource_id = azurerm_log_analytics_workspace.test.id workspace_name = azurerm_log_analytics_workspace.test.name plan { publisher = "Microsoft" product = "OMSGallery/ContainerInsights" } } resource "azurerm_kubernetes_cluster" "k8s" { name = var.cluster_name location = azurerm_resource_group.k8s.location resource_group_name = azurerm_resource_group.k8s.name dns_prefix = var.dns_prefix # linux_profile { # admin_username = "ubuntu" # # # ssh_key { # key_data = file(var.ssh_public_key) # } # } default_node_pool { name = "agentpool" node_count = var.agent_count vm_size = "Standard_D2_v2" } identity { type = "SystemAssigned" } # service_principal { # client_id = var.aks_service_principal_app_id # client_secret = var.aks_service_principal_client_secret # } # addon_profile { # oms_agent { # enabled = true # log_analytics_workspace_id = azurerm_log_analytics_workspace.test.id # } # { # enabled = true # } # } = true oms_agent { log_analytics_workspace_id = azurerm_log_analytics_workspace.test.id } network_profile { load_balancer_sku = "standard" network_plugin = "kubenet" } tags = { Environment = "Development" } } data "azurerm_container_registry" "acr" { name = "gitlabazuredemo" resource_group_name = "rg1" } # add the role to the identity the kubernetes cluster was assigned resource "azurerm_role_assignment" "kubweb_to_acr" { scope = data.azurerm_container_registry.acr.id role_definition_name = "AcrPull" principal_id = azurerm_kubernetes_cluster.k8s.kubelet_identity[0].object_id }

provider.tf

terraform { required_version = ">=0.12" required_providers { azurerm = { source = "hashicorp/azurerm" version = "~>3.0" } } backend "azurerm" { resource_group_name = "rg1" storage_account_name = "gitlabstoragetest" container_name = "terraform-state-file" key = "test.terraform.tfstate" } } provider "azurerm" { features {} }

variables.tf

variable "resource_group_name_prefix" { default = "rg" description = "Prefix of the resource group name that's combined with a random ID so name is unique in your Azure subscription." } variable "resource_group_location" { default = "eastasia" description = "Location of the resource group." } variable "agent_count" { default = 1 } #variable "ssh_public_key" { # default = "~/.ssh/id_rsa.pub" #} variable "dns_prefix" { default = "k8stest" } variable cluster_name { default = "k8stest" } variable resource_group_name { default = "azure-k8stest" } variable location { default = "East Asia" } variable log_analytics_workspace_name { default = "testLogAnalyticsWorkspaceName" } # refer for log analytics available regions variable log_analytics_workspace_location { default = "eastasia" } # refer for log analytics pricing variable log_analytics_workspace_sku { default = "PerGB2018" }

output.tf

output "resource_group_name" { value = azurerm_resource_group.k8s.name }

在gitlab里面我的CI的架构仍然是使用子目录的形式，如下所示

.gitlab-ci.yml .sub-gitlab-ci.yml angularjs_project - .gitlab-ci.yml - Dockerfile - other files.. dotnet_project - .gitlab-ci.yml - Dockerfile - other files.. aks_project - .gitlab-ci.yml - terraform files - test folder

最外面的 .gitlab-ci.yml

stages: - child_pipeline aks: stage: child_pipeline variables: CHILD_PIPELINE_EXECUTION_CONTEXT: "aks_project" trigger: include: .sub-gitlab-ci.yml strategy: depend dotnet: stage: child_pipeline when: manual variables: CHILD_PIPELINE_EXECUTION_CONTEXT: "dotnet_project" trigger: include: .sub-gitlab-ci.yml strategy: depend angularjs: stage: child_pipeline when: manual variables: CHILD_PIPELINE_EXECUTION_CONTEXT: "angularjs_project" trigger: include: .sub-gitlab-ci.yml strategy: depend

最外面的.sub-gitlab-ci.yml,这里我使用了before_script,他会在每个stage里面的job的script执行之前执行下面的代码

before_script: - "echo Running child pipeline in subdirectory: $CHILD_PIPELINE_EXECUTION_CONTEXT" - cd $CHILD_PIPELINE_EXECUTION_CONTEXT - apk add terraform --repository=https://dl-cdn.alpinelinux.org/alpine/edge/community - rm -rf .terraform - terraform --version - terraform init include: $CHILD_PIPELINE_EXECUTION_CONTEXT/.gitlab-ci.yml

最后是aks_project目录里面的.gitlab-ci.yml.gitlab-ci.yml CI的文件

image: name: mcr.microsoft.com/azure-cli entrypoint: - '/usr/bin/env' - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' stages: - validate - plan - apply validate: stage: validate script: - terraform validate plan: stage: plan script: - terraform plan dependencies: - validate apply: stage: apply script: - terraform apply -auto-approve dependencies: - plan

执行相关的代码 会成功创建AKS

user@VirtualBox:~/devops/gitlab-2-azure/aks_project$ kubectl get node NAME STATUS ROLES AGE VERSION aks-agentpool-11728208-vmss000000 Ready agent 21h v1.22.11

配置使用Argo CD

安装 argo cd

执行下面的命令，即可安装argo cd

kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml kubectl get all -n argocd

为了访问argo cd的web界面，最简单的方式是可以直接通过service来暴露端口，然后访问对应的外网IP即可

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

如果不想给外界暴露端口，也可以从本地端口转发的形式来访问 port-forward svc/argocd-server -n argocd 8080:443

默认的用户名是 admin, 密码可以通过下面的命令获取

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

测试manifest文件

简单的说，他会从acr上面pull我之前上传的镜像，创建deployment，service和ingress。 我在之前创建aks的时候，已经自动创建了ingress controller的addon，和绑定了acr，因此这里可以直接使用。如果我的docker registry是保存在第三方的，那么我还需要创建secret来保存docker的用户密码，然后挂载在pod下面。但是因为secret本身保存的是base4格式的明文，出于安全原因所以不可以直接存放在git repo中，那有需要额外的解决方案来保存相关的key和value。从演示的角度，这种azure的全家桶解决方案是最简单的。

deployment.yml

apiVersion: apps/v1 kind: Deployment metadata: name: hello-dotnet spec: selector: matchLabels: name: hello-dotnet template: metadata: labels: name: hello-dotnet spec: containers: - name: test image: gitlabazuredemo.azurecr.io/dotnet_demo:latest imagePullPolicy: IfNotPresent ports: - containerPort: 80

ingress.yml

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: hello-dotnet annotations: kubernetes.io/ingress.class: addon-http-application-routing spec: rules: - http: paths: - path: / pathType: Prefix backend: service: name: hello-dotnet port: number: 80

svc.yml

apiVersion: v1 kind: Service metadata: name: hello-dotnet spec: type: ClusterIP ports: - port: 80 targetPort: 80 selector: name: hello-dotnet

Argo cd配置 manifest文件

然后添加app

刷新一下

最后验证一下

user@VirtualBox:~/devops/gitlab-2-azure/aks_project$ kubectl get all NAME READY STATUS RESTARTS AGE pod/hello-dotnet-cfd9bd58c-8gqpc 1/1 Running 0 20h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/hello-dotnet ClusterIP 10.0.217.138 80/TCP 20h service/kubernetes ClusterIP 10.0.0.1 443/TCP 22h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/hello-dotnet 1/1 1 1 20h NAME DESIRED CURRENT READY AGE replicaset.apps/hello-dotnet-cfd9bd58c 1 1 1 20h

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。