【aws】ECS上构筑基于Concourse的CI/CD方案

Amazon Elastic Container Service （Amazon ECS）是一项高度可扩展的快速容器管理服务，它可轻松运行、停止和管理群集上的容器。您的容器在任务定义中定义，用于运行服务中的单个任务或服务。在此上下文中，服务是一种配置，使您能够同时在集群中运行和维护指定数量的任务。您可以在由 AWS Fargate 管理的无服务器基础设施上运行您的任务和服务。或者，要更好地控制您的基础设施，您可以在管理的 Amazon EC2 实例集群上运行您的任务和服务。

Concourse CI 是一款 CI/CD 工具，它的魅力在于极简设计，被广泛应用于 Cloud Foundry 各个模块的 CI/CD。Concourse CI 官方提供了标准的 Docker 镜像，可以通过AWS ECS容器服务部署一套 Concourse CI 应用。

本次构筑Concourse CI的后端存储，选用的postgres数据库，利用AWS Fargate部署容器版的postgres。另外需要部署Concourse CI的管理Web和worker组件。其中Web也是利用AWS Fargate部署Serverless的容器服务，Worker需要开启特权privileged，而AWS Fargate并不支持privileged，所以选用EC2集群执行Worker容器。

前置条件：

需要创建一个名称为concourse-data的efs，作为postgres, web以及worker容器的存储卷
2. 配置ecsTaskExecutionRole，开放给容器执行必要的权限，比如SystemManager的访问权限等。

构筑步骤：

定义postgres任务

{"ipcMode": null,"executionRoleArn": "xxxx/ecsTaskExecutionRole","containerDefinitions": [{"dnsSearchDomains": null,"environmentFiles": null,"logConfiguration": {"logDriver": "awslogs","secretOptions": null,"options": {"awslogs-group": "/ecs/concourse-db","awslogs-region": "cn-north-1","awslogs-stream-prefix": "ecs"}},"entryPoint": null,"portMappings": [],"command": null,"linuxParameters": null,"cpu": 0,"environment": [{"name": "PGDATA","value": "/database"},{"name": "POSTGRES_PASSWORD","value": ""},{"name": "POSTGRES_DB","value": "concourse"}],"resourceRequirements": null,"ulimits": null,"dnsServers": null,"mountPoints": [{"readOnly": null,"containerPath": "/database","sourceVolume": "concourse-db"}],"workingDirectory": null,"secrets": null,"dockerSecurityOptions": null,"memory": null,"memoryReservation": null,"volumesFrom": [],"stopTimeout": null,"image": "postgres:12.2","startTimeout": null,"firelensConfiguration": null,"dependsOn": null,"disableNetworking": null,"interactive": null,"healthCheck": null,"essential": true,"links": null,"hostname": null,"extraHosts": null,"pseudoTerminal": null,"user": null,"readonlyRootFilesystem": null,"dockerLabels": null,"systemControls": null,"privileged": null,"name": "concourse-db"}],"memory": "512","taskRoleArn": "xxxx/ecsTaskExecutionRole","family": "concourse-db","pidMode": null,"requiresCompatibilities": ["FARGATE"],"networkMode": "awsvpc","cpu": "256","inferenceAccelerators": null,"proxyConfiguration": null,"volumes": [{"efsVolumeConfiguration": {"transitEncryptionPort": null,"fileSystemId": "fs-12835e8f","authorizationConfig": {"iam": "DISABLED","accessPointId": null},"transitEncryption": "DISABLED","rootDirectory": "/data/concourese/database"},"name": "concourse-db","host": null,"dockerVolumeConfiguration": null}],"tags": []
}

定义Web任务

{"ipcMode": null,"executionRoleArn": "xxxx/ecsTaskExecutionRole","containerDefinitions": [{"dnsSearchDomains": null,"environmentFiles": null,"logConfiguration": {"logDriver": "awslogs","secretOptions": null,"options": {"awslogs-group": "/ecs/concourse-web","awslogs-region": "cn-north-1","awslogs-stream-prefix": "ecs"}},"entryPoint": null,"portMappings": [{"hostPort": 443,"protocol": "tcp","containerPort": 443}],"command": ["web"],"linuxParameters": null,"cpu": 0,"environment": [{"name": "CONCOURSE_AWS_SSM_PIPELINE_SECRET_TEMPLATE","value": "/concourse/{{.Team}}/{{.Pipeline}}/{{.Secret}}"},{"name": "CONCOURSE_ADD_LOCAL_USER","value": "admin:xxxx,platform:xxxx"},{"name": "CONCOURSE_POSTGRES_HOST","value": "concourse-db.local"},{"name": "CONCOURSE_LOG_LEVEL","value": "debug"},{"name": "CONCOURSE_TLS_CERT","value": "/concourse-keys/server.crt"},{"name": "CONCOURSE_AWS_SSM_TEAM_SECRET_TEMPLATE","value": "/concourse/{{.Team}}/{{.Secret}}"},{"name": "CONCOURSE_TLS_KEY","value": "/concourse-keys/server.key"},{"name": "CONCOURSE_AWS_SSM_SECRET_KEY","value": "Y/"},{"name": "CONCOURSE_POSTGRES_PASSWORD","value": ""},{"name": "CONCOURSE_POSTGRES_DATABASE","value": "concourse"},{"name": "CONCOURSE_AWS_SSM_REGION","value": "cn-north-1"},{"name": "CONCOURSE_TLS_BIND_PORT","value": "443"},{"name": "CONCOURSE_MAIN_TEAM_LOCAL_USER","value": "admin"},{"name": "CONCOURSE_AWS_SSM_ACCESS_KEY","value": ""},{"name": "CONCOURSE_EXTERNAL_URL","value": "https://xxxx:8443"},{"name": "CONCOURSE_POSTGRES_USER","value": "concourse@postgres"}],"resourceRequirements": null,"ulimits": null,"dnsServers": null,"mountPoints": [{"readOnly": null,"containerPath": "/concourse-keys","sourceVolume": "concourse-keys"}],"workingDirectory": null,"secrets": null,"dockerSecurityOptions": null,"memory": null,"memoryReservation": null,"volumesFrom": [],"stopTimeout": null,"image": "voss2018/concourse:6.5.1.1","startTimeout": null,"firelensConfiguration": null,"dependsOn": null,"disableNetworking": null,"interactive": null,"healthCheck": null,"essential": true,"links": null,"hostname": null,"extraHosts": null,"pseudoTerminal": null,"user": null,"readonlyRootFilesystem": null,"dockerLabels": null,"systemControls": null,"privileged": null,"name": "concourse-web"}],"memory": "2048","taskRoleArn": "arn:aws-cn:iam::348769610664:role/ecsTaskExecutionRole","family": "concourse-web","pidMode": null,"requiresCompatibilities": ["FARGATE"],"networkMode": "awsvpc","cpu": "1024","inferenceAccelerators": null,"proxyConfiguration": null,"volumes": [{"efsVolumeConfiguration": {"transitEncryptionPort": null,"fileSystemId": "fs-12835e8f","authorizationConfig": {"iam": "DISABLED","accessPointId": null},"transitEncryption": "DISABLED","rootDirectory": "/data/concourese/web"},"name": "concourse-keys","host": null,"dockerVolumeConfiguration": null}],"tags": []
}

定义worker任务

{"ipcMode": null,"executionRoleArn": "xxxx/ecsTaskExecutionRole","containerDefinitions": [{"dnsSearchDomains": null,"environmentFiles": null,"logConfiguration": {"logDriver": "awslogs","secretOptions": null,"options": {"awslogs-group": "/ecs/concourse-worker","awslogs-region": "cn-north-1","awslogs-stream-prefix": "ecs"}},"entryPoint": null,"portMappings": [],"command": ["worker"],"linuxParameters": null,"cpu": 0,"environment": [{"name": "CONCOURSE_TSA_HOST","value": "concourse-web.local:2222"}],"resourceRequirements": null,"ulimits": null,"dnsServers": null,"mountPoints": [{"readOnly": null,"containerPath": "/concourse-keys","sourceVolume": "concourse-keys"}],"workingDirectory": null,"secrets": null,"dockerSecurityOptions": null,"memory": null,"memoryReservation": null,"volumesFrom": [],"stopTimeout": null,"image": "voss2018/concourse:6.5.1.1","startTimeout": null,"firelensConfiguration": null,"dependsOn": null,"disableNetworking": null,"interactive": null,"healthCheck": null,"essential": true,"links": null,"hostname": null,"extraHosts": null,"pseudoTerminal": null,"user": null,"readonlyRootFilesystem": null,"dockerLabels": null,"systemControls": null,"privileged": true,"name": "concourse-worker"}],"memory": "4096","taskRoleArn": "xxxx/ecsTaskExecutionRole","family": "concourse-worker","pidMode": null,"requiresCompatibilities": ["EC2"],"networkMode": "awsvpc","cpu": "1024","inferenceAccelerators": null,"proxyConfiguration": null,"volumes": [{"efsVolumeConfiguration": {"transitEncryptionPort": null,"fileSystemId": "fs-12835e8f","authorizationConfig": {"iam": "DISABLED","accessPointId": null},"transitEncryption": "DISABLED","rootDirectory": "/data/concourese/worker"},"name": "concourse-keys","host": null,"dockerVolumeConfiguration": null}],"placementConstraints": [],"tags": []
}

创建cluster platform，利用下面的cluster模板创建，并且添加一台EC2实例，以及配置必要的VPC，安全组

      EC2 Linux + 联网要创建的资源:集群VPC子网带 Linux AMI 的 Auto Scaling 组

在Platform cluster中创建service：concourse-db
在platform cluster中创建concourse-web service
在platform cluster中创建concourse-worker service
确认三个service是否正常Running
配置负载均衡，登录concourse web

【aws】ECS上构筑基于Concourse的CI/CD方案相关推荐

Jenkins X：基于Kubernetes的CI/CD平台
背景 Jenkins自动化服务器一直都是DevOps工具链的重要组成部分,并且已发展成为开源持续集成和持续部署(CI&CD)软件的领导者.据不完全统计,截至2019年6月,已有将近26万的Je ...
基于 Kubernetes 实现 CI/CD 配置
基于 Kubernetes 实现 CI/CD 配置一.基本介绍二.基于 Kubernetes 实现 CI/CD 配置 1.配置 GitLab 2.配置 Jenkins 3.实现 CI/CD 配置 ...
通过Amazon SageMaker与Amazon Step Functions实现机器学习的CI/CD 方案
在传统的机器学习工作流程当中,经常会面临两个问题: (1)数据迭代迅速,需要定期对模型进行重新训练,每次训练完成后,都需要重新部署模型,如何实现训练与部署过程的的自动化,从而提升工作效率: (2)算法 ...
基于K8s的CI/CD自动化持续集成部署框架
文章目录什么是CI/CD? 1 环境准备 2 组件部署 2.1 jenkins快速部署 2.2 gitlab快速部署 2.3 harbor快速部署 3 CI/CD交付什么是CI/CD? CI/CD ...
基于docker的CI/CD
准备条件使用docker的机器,添加用户时需要指定用户的uid : sudo groupadd -g 500 work && useradd -g 500 -u 500 work ...
基于docker-compose的Gitlab CI/CD实践排坑指南
长话短说经过长时间实操验证,终于完成基于Gitlab的CI/CD实践,本次实践的坑位很多, 实操过程尽量接近最佳实践(不做hack, 不做骚操作),记录下来加深理解. 看过博客园<docker ...
git原理详解与实操指南_基于dockercompose的Gitlab CI/CD实践amp;排坑指南
长话短说经过长时间实操验证,终于完成基于Gitlab的CI/CD实践,本次实践的坑位很多, 实操过程尽量接近最佳实践(不做hack, 不做骚操作),记录下来加深理解. 看过博客园<docker ...
基于OpenStack+Docker设计与实现CI/CD
本文所述内容的背景是,基于Docker容器技术的OpenStack研发.测试.运维及其相关的CI/CD.DevOps等活动.思想是相通的,读者可以取其可用部分用于自己的业务需求中. IaaS云和容器云 ...
OpenShift 4 Hands-on Lab (8) 基于Gogs+Nexus+Sonarqube的Jenkins CI/CD Pipeline
<OpenShift 4.x HOL教程汇总> 说明:本文已经在 OpenShift 4.8 环境中验证(OpenShift 4.9 环境的 Jenkins 版本较新,编译 Java 报错 ...

【aws】ECS上构筑基于Concourse的CI/CD方案

【aws】ECS上构筑基于Concourse的CI/CD方案相关推荐

最新文章

热门文章