aws ecs 理解元数据和mock本地测试环境

资料

Under the Hood: Task Networking for Amazon ECS
amazon-ecs-local-container-endpoints
A Guide to Locally Testing Containers with Amazon ECS Local Endpoints and Docker Compose
Amazon ECS 容器代理自检
Amazon ECS 任务元数据端点

本文讨论了ecs 任务获取元数据和凭证的逻辑和测试方法

任务元数据和凭证的端点是由ecs agent提供的。ecs agent使用host模式启动，能够阻止ecs agent启动的容器访问http://169.254.169.254获取实例元数据，并避免docker0上的链接和流量争用。

ecs agent提供了一个 API （自检），用于收集有关正在运行该代理的容器实例以及在该实例上正在运行的相关任务的详细信息

$ curl -s http://localhost:51678/v1/metadata | jq .
{"Cluster": "xxxxxx","ContainerInstanceArn": "arn:aws-cn:ecs:cn-north-1:xxxxxx:container-instance/xxxxx/8293f9feb64d409fa069d329e63ba984","Version": "Amazon ECS Agent - v1.62.2 (*a1a5ecbc)"
}

ecs 元数据

注意：为任务指定 IAM 角色时，该任务的容器中的 Amazon CLI 或其他开发工具包只使用该任务角色提供的 Amazon 凭证，它们不再从 Amazon EC2 或它们运行所在的外部实例继承任何 IAM 权限

ecs元数据端点有多个版本，目前支持的有2，3和4

版本4（>1.39.0，> 1.54.0 for windows using awsvpc），容器元数据，docker统计信息

任务元数据和联网速率统计数据将发送到 cw container insight
版本3（>1.21.0，> 1.54.0 for windows using awsvpc），容器元数据，docker统计信息

从代理版本 1.21.0 开始，代理将称为 ECS_CONTAINER_METADATA_URI 的环境变量注入任务中的每个容器
版本2（fargate没有）（>1.17.0，> 1.54.0 for windows using awsvpc），容器元数据，docker统计信息

查看ecs 实例上容器注入的环境变量如下

"ECS_CONTAINER_METADATA_URI_V4=http://169.254.170.2/v4/ef186256-092c-49b5-ace1-7c426b4aaaf2",
"AWS_EXECUTION_ENV=AWS_ECS_EC2",
"AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/af07beab-d714-4b4a-aa27-75168bbe0c66",
"ECS_CONTAINER_METADATA_URI=http://169.254.170.2/v3/ef186256-092c-49b5-ace1-7c426b4aaaf2",

获取元数据路径

$ curl $ECS_CONTAINER_METADATA_URI_V4 #元数据
$ curl $ECS_CONTAINER_METADATA_URI_V4/task #任务元数据
$ curl $ECS_CONTAINER_METADATA_URI_V4/taskWithTags #带标签的任务元数据
$ curl $ECS_CONTAINER_METADATA_URI_V4/stats #docker统计信息，cpu，内存，网络数据收发
$ curl $ECS_CONTAINER_METADATA_URI_V4/task/stats #任务中容器的docker统计数据

获取凭证

# curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
{"RoleArn":"arn:aws-cn:iam::xxxxxx:role/ecsTaskRole","AccessKeyId":"ASIAQRIBWRJKBUNH7I67","SecretAccessKey":"Gm0dkB3zpoumsj/IVVcEKXXg8M8mC6JmGQuScFD0","Token":"xxxxxxxxxxx","Expiration":"2022-11-21T17:48:24Z"
}

ecs agent在启动时为了使容器使用task role设置了iptable规则，因此访问127.0.0.1同样能获到元数据

$ sysctl -w net.ipv4.conf.all.route_localnet=1
$ iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 -j DNAT --to-destination 127.0.0.1:51679
$ iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679

访问169.254.170.2实际是访问agent获取信息

#ecs agent 监听51679端口
$ sudo netstat -ntulp | grep 51679
tcp        0      0 127.0.0.1:51679         0.0.0.0:*               LISTEN      4162/agent

ecs容器代理通过任务元数据端点提供了检索任务元数据和docker统计指标的方法，可以在官方文档找到完整输出

bash-4.2# curl $ECS_CONTAINER_METADATA_URI_V4
{"DockerId":"4bedfcb6e1df59992d7e579b11edfaed6b987c2ce67bf9c937d9b287a5f937b3","Name":"amazonlinux","DockerName":"ecs-amazonlinux-4-amazonlinux-fc9ecebdc0d7c9ac7300","Image":"amazonlinux:latest","ImageID":"sha256:6e809582795f51280dda491769531ca101af7ce73ff67ec039597b1f000fef8c","Labels":{"com.amazonaws.ecs.cluster":"xxxxxx","com.amazonaws.ecs.container-name":"amazonlinux","com.amazonaws.ecs.task-arn":"arn:aws-cn:ecs:cn-north-1:xxxxxx:task/xxxxxx/10b941d27ed9472681f226ae63b34771","com.amazonaws.ecs.task-definition-family":"amazonlinux","com.amazonaws.ecs.task-definition-version":"4"},"DesiredStatus":"RUNNING","KnownStatus":"RUNNING","Limits":{"CPU":2,"Memory":0},"Type":"NORMAL","ContainerARN":"arn:aws-cn:ecs:cn-north-1:xxxxxx:container/xxxxxx/10b941d27ed9472681f226ae63b34771/e832efc3-3529-472d-b145-92a754f79ab3","Networks":[{"NetworkMode":"bridge","IPv4Addresses":["172.17.0.5"]}]
}

获取的统计信息如下

{"read": "2022-12-10T04:05:36.877932601Z","preread": "2022-12-10T04:05:35.857775774Z","pids_stats": {"current": 2},"blkio_stats": {"io_service_bytes_recursive": [],"io_serviced_recursive": [],"io_queue_recursive": [],"io_service_time_recursive": [],"io_wait_time_recursive": [],"io_merged_recursive": [],"io_time_recursive": [],"sectors_recursive": []},"num_procs": 0,"storage_stats": {},"cpu_stats": {"cpu_usage": {"total_usage": 471454000,"percpu_usage": [236453619,235000381],"usage_in_kernelmode": 50000000,"usage_in_usermode": 370000000},"system_cpu_usage": 183090590000000,"online_cpus": 2,"throttling_data": {"periods": 0,"throttled_periods": 0,"throttled_time": 0}},"precpu_stats": {"cpu_usage": {"total_usage": 471325043,"percpu_usage": [236417821,234907222],"usage_in_kernelmode": 50000000,"usage_in_usermode": 370000000},"system_cpu_usage": 183088550000000,"online_cpus": 2,"throttling_data": {"periods": 0,"throttled_periods": 0,"throttled_time": 0}},"memory_stats": {"usage": 1527808,"max_usage": 5378048,"stats": {"active_anon": 458752,"active_file": 0,"cache": 0,"dirty": 0,"hierarchical_memory_limit": 268435456,"hierarchical_memsw_limit": 9223372036854771712,"inactive_anon": 0,"inactive_file": 0,"mapped_file": 0,"pgfault": 8118,"pgmajfault": 0,"pgpgin": 4785,"pgpgout": 4662,"rss": 401408,"rss_huge": 0,"total_active_anon": 458752,"total_active_file": 0,"total_cache": 0,"total_dirty": 0,"total_inactive_anon": 0,"total_inactive_file": 0,"total_mapped_file": 0,"total_pgfault": 8118,"total_pgmajfault": 0,"total_pgpgin": 4785,"total_pgpgout": 4662,"total_rss": 401408,"total_rss_huge": 0,"total_unevictable": 0,"total_writeback": 0,"unevictable": 0,"writeback": 0},"limit": 4072448000},"name": "/ecs-amazonlinux-4-amazonlinux-86ced49c948ed8aed001","id": "e61b98b2996af65daa69f422256ce25e66d809e9de5bf553dc16760cb874032a","networks": {"eth0": {"rx_bytes": 10902,"rx_packets": 55,"rx_errors": 0,"rx_dropped": 0,"tx_bytes": 4854,"tx_packets": 59,"tx_errors": 0,"tx_dropped": 0}}
}

在本地mock ecs容器环境

该工具的使用场景如下，当没有aws账号或者跨系统开发时需要依赖容器数据端点，可以使用该工具mock。但是这个工具貌似不支持v4版本的端点

A Guide to Locally Testing Containers with Amazon ECS Local Endpoints and Docker Compose

Testing a container that needs credentials to interact with AWS Services

Testing a container which uses Task Metadata

Testing a multi-container app which uses the awsvpc or host network mode on Docker For Mac and Docker For Windows (in linux mode)

Testing multiple containerized applications using local service discovery

在开始mock之前需要对网路进行配置，有以下两种方式

使用docker自定义网络（更推荐）
像ecs agent一样设置iptables规则将流量导入local container endpoint容器中

这里使用自定义网络配置，看一下docker-compose.yaml文件内容

version: "2"
networks:#定义自定义网络，网段为169.254.170.0/24，桥接模式credentials_network:driver: bridgeipam:config:- subnet: "169.254.170.0/24"gateway: 169.254.170.1
services:# 为容器提供凭证的服务ecs-local-endpoints:image: amazon/amazon-ecs-local-container-endpointsvolumes:# 挂载docker.sock文件- /var/run:/var/run# 挂载凭证配置- $HOME/.aws/:/home/.aws/environment:# 使用宿主机的默认凭证AWS_PROFILE: "default"networks:credentials_network:# 这个ip在aws cli和aws sdk中写死了ipv4_address: "169.254.170.2"# 我们的应用程序的配置app:image: amazonlinux:latestcommand: sleep infinitydepends_on:- ecs-local-endpointsnetworks:credentials_network:ipv4_address: "169.254.170.3"environment:AWS_DEFAULT_REGION: "cn-north-1"AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds"ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3"

在一台普通的ec2实例上，启动环境进入amazonlinux容器查看，环境变量已经注入

# env
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/creds
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
AWS_DEFAULT_REGION=cn-north-1
ECS_CONTAINER_METADATA_URI=http://169.254.170.2/v3

mock 凭证

在容器内安装aws cli工具查看当前的凭证，确实获取到了宿主机的凭证

# aws sts get-caller-identity
{"UserId": "xxxxxxxxx:i-xxxxxxx","Account": "xxxxxx","Arn": "arn:aws-cn:sts::xxxxxx:assumed-role/MyEc2xxxxxAccess/i-xxxxxx"
}

mock 元数据

访问任务元数据端点，mock的数据可以拿到

$ curl $ECS_CONTAINER_METADATA_URI
{"DockerId": "d8bc2359f93bf89a4a25560c76965d3ff91f4e82c1aeee8b0b8414f5dd28ae86","Name": "mock-app-1","DockerName": "mock-app-1","Image": "amazonlinux:latest","ImageID": "sha256:6e809582795f51280dda491769531ca101af7ce73ff67ec039597b1f000fef8c","Labels": {"com.docker.compose.config-hash": "7f1cb58c41f4f568d23c307f248ec102fcfac153932a6a26c726013b79aace70","com.docker.compose.container-number": "1","com.docker.compose.depends_on": "ecs-local-endpoints:service_started","com.docker.compose.image": "sha256:6e809582795f51280dda491769531ca101af7ce73ff67ec039597b1f000fef8c","com.docker.compose.oneoff": "False","com.docker.compose.project": "mock","com.docker.compose.project.config_files": "/home/ec2-user/amazon-ecs-local-container-endpoints/mock/docker-compose.yaml","com.docker.compose.project.working_dir": "/home/ec2-user/amazon-ecs-local-container-endpoints/mock","com.docker.compose.service": "app","com.docker.compose.version": "2.4.1"},"DesiredStatus": "RUNNING","KnownStatus": "RUNNING","Limits": {},"Type": "NORMAL","Networks": [{"NetworkMode": "mock_credentials_network","IPv4Addresses": ["169.254.170.3"]}]
}

mock awsvpc网咯模式

使用awsvpc模式启动的任务独占一张网卡，因此相同任务中的容器可以通过lcoalhost进行通信。将容器启动在同一个网络空间下，需要pause容器进行连接（类似于kubernetes的pod）

构建pause容器

$ cat dockerfile
FROM amazonlinux:latest
RUN yum install -y iptablesCMD iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 -j DNAT --to-destination 127.0.0.1:51679 \&& iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679 \&& iptables-save \&& /bin/bash -c 'while true; do sleep 30; done;'$ docker build -t local-pause:latest .

之后我们创建的所有容器的网络空间都应该在pause容器中

version: "2"
services:ecs-local-endpoints:image: amazon/amazon-ecs-local-container-endpointsvolumes:- /var/run:/var/run- $HOME/.aws/:/home/.aws/environment:ECS_LOCAL_METADATA_PORT: "51679"HOME: "/home"# 修改网络模式network_mode: container:local-pauseapp:image: amazonlinux:latestcommand: sleep infinitydepends_on:- ecs-local-endpointsnetwork_mode: container:local-pauseenvironment:ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3/containers/app"AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds"

在运行之前首先需要启动pause容器将基本的网络空间建立，使用-p设置在任务定义中容器需要expose的端口

$ docker run -d -p 8080:8080 --name local-pause --cap-add=NET_ADMIN local-pause

进入app查看网络信息，可以看到ecs-local-endpoints监听的端口，此时容器可以通过localhost相互通信

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft forever
24: eth0@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group defaultlink/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0valid_lft forever preferred_lft forever# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::51679                :::*                    LISTEN      -