aws ecs 通过efs挂载实现动态更新firelens日志配置

参考：

Amazon ECS FireLens Examples
用于 Amazon ECS 任务的 FireLens
自定义日志路由
AWS Container Logging Deep Dive: FireLens, Fluentd, and Fluent Bit
使用 AWS FireLens 轻松实现 AWS Fargate 容器日志处理

aws firelens

firelens是ecs平台上容器独有的log driver，firelens可以和 fluentbit 和 fluentd 集成（实际上还是使用fluentbit容器对日志进行处理），将fluentbit托管在firelens下能够更好地和aws service集成，并且更灵活地进行配置。

值得一提的是，firelens官方提供的example比较全面，降低了配置难度

FireLens for Fluent Bit 支持将日志发送到：

Amazon CloudWatch Logs
Amazon Kinesis Data Firehose
Amazon Kinesis Data Streams
Fluent Bit 原生支持的所有目标

FireLens 在端口 24224 上侦听，因此为了确保从任务外部无法访问 FireLens 日志路由器，不得允许任务使用的安全组中端口 24224 上的入站流量。对于使用 awsvpc 网络模式的任务，这是与任务关联的安全组。对于使用 host 网络模式的任务，它是与托管任务的 Amazon EC2 实例关联的安全组。对于使用 bridge 网络模式的任务，请勿创建任何使用端口 24224 的端口映射。

注意事项：

windows不支持firelens
FireLens 在端口 24224 上侦听，需要确保任务外部无法访问此端口（安全组）

默认情况下，FireLens 将 集群、任务定义名称、集群的ARN 作为元数据键添加到 stdout/stderr 容器日志中，可以不配置

"firelensConfiguration":{"type":"fluentbit","options":{"enable-ecs-log-metadata":"false","config-file-type":"file","config-file-value":"/extra.conf"
}

如果需要获取ecr或者secret manager需要配置taskexecrole
如果配置从s3获取配置文件，需要s3的相关权限

配置ecs任务

使用python的bootle创建简单的web server

from bottle import route, error, get
import loggingdef logger_config(log_path,logging_name):logger = logging.getLogger(logging_name)logger.setLevel(level=logging.DEBUG)handler = logging.FileHandler(log_path, encoding='UTF-8')handler.setLevel(logging.INFO)formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')handler.setFormatter(formatter)console = logging.StreamHandler()console.setLevel(logging.DEBUG)logger.addHandler(handler)logger.addHandler(console)return loggerlogger = logger_config("infolog.txt","test-log-info")@route('/log')
def mylog():logger.info("info")logger.error("error")logger.debug("debug")logger.warning("warning")@get('/')
def health():return 'success'@error(404)
def error404(error):return 'Nothing here, sorry'run(host='0.0.0.0', port=8090)

将脚本打包上传到ecr中

FROM python:latest
WORKDIR /opt/web
COPY . .
EXPOSE 80
ENTRYPOINT python ./web.py

创建任务定义引用如下，创建服务启动即可（这里使用ec2平台，便于查看容器）

{"name": "myweb","image": "xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/xxxxxx","executionRoleArn": "arn:aws-cn:iam::xxxxxxxxxx:role/ecsTaskExecutionRole","containerDefinitions": [{// 指定业务容器地日志驱动为awsfirelens"logConfiguration": {"logDriver": "awsfirelens","options": {"log_group_name": "/aws/ecs/containerinsights/$(ecs_cluster)/application","log_stream_name": "$(ecs_task_id)","auto_create_group": "true","log_key": "log","region": "cn-north-1","Name": "cloudwatch","retry_limit": "2"}},// 容器暴露8090端口"portMappings": [{"hostPort": 0,"protocol": "tcp","containerPort": 80}],// 挂载日志卷"mountPoints": [{"readOnly": null,"containerPath": "/opt/web/","sourceVolume": "var-log"}]},{"name": "log_router","image": "128054284489.dkr.ecr.cn-north-1.amazonaws.com.cn/aws-for-fluent-bit:latest",// 配置firelens类型为fluentbit，配置文件位置为/configs/extra-myweb.conf"firelensConfiguration": {"type": "fluentbit","options": {"config-file-type": "file","config-file-value": "/configs/extra-myweb.conf"}},// 将fluentbit本身的日志发送到/ecs/myweb日志组"logConfiguration": {"logDriver": "awslogs","secretOptions": null,"options": {"awslogs-group": "/ecs/myweb","awslogs-region": "cn-north-1","awslogs-stream-prefix": "ecs"}},"mountPoints": [// 挂载firelens-conf卷到/configs下{"readOnly": null,"containerPath": "/configs","sourceVolume": "firelens-conf"}，// 挂载日志卷{"readOnly": null,"containerPath": "/opt/web/","sourceVolume": "var-log"}]}],"volumes": [// 创建efs卷存放配置文件{"fsxWindowsFileServerVolumeConfiguration": null,"efsVolumeConfiguration": {"transitEncryptionPort": null,"fileSystemId": "fs-xxxxxxx","authorizationConfig": {"iam": "ENABLED","accessPointId": null},"transitEncryption": "ENABLED","rootDirectory": "/test-ecs/firelens-config"},"name": "firelens-conf","host": null,"dockerVolumeConfiguration": null}，// 创建日志卷{"name": "var-log","host": {"sourcePath": null}}],"memory": "512","cpu": "1024","taskRoleArn": "arn:aws-cn:iam::xxxxxxxxxx:role/ecsTaskRole","family": "myweb","requiresCompatibilities": ["EC2"],"networkMode": "bridge"
}

创建fluentbit配置文件，抓取

[INPUT]Name              tailTag               myweb-firelensPath              /opt/web/infolog.txtDB                /var/log/myweb_service.dbDB.locking        trueSkip_Long_Lines   OnRefresh_Interval  10Rotate_Wait       30[OUTPUT]Name  stdoutMatch myweb-firelens

进入ecs 的ec2实例查看fluentbit容器的配置文件，可以看到在其中引用了@INCLUDE /configs/extra-myweb.conf配置

bash-4.2# cat /fluent-bit/etc/fluent-bit.conf[INPUT]Name forwardunix_path /var/run/fluent.sock[INPUT]Name forwardListen 0.0.0.0Port 24224[INPUT]Name tcpTag firelens-healthcheckListen 127.0.0.1Port 8877[FILTER]Name record_modifierMatch *Record ec2_instance_id i-xxxxxxRecord ecs_cluster worktestRecord ecs_task_arn arn:aws-cn:ecs:cn-north-1:xxxxxxxx:task/worktest/0a5ecaa32f784edab3a723b8c8390a06Record ecs_task_definition myweb:9@INCLUDE /configs/extra-myweb.conf[OUTPUT]Name nullMatch firelens-healthcheck[OUTPUT]Name cloudwatchMatch myweb-firelens*auto_create_group truelog_group_name /aws/ecs/containerinsights/$(ecs_cluster)/applicationlog_key loglog_stream_name $(ecs_task_id)region cn-north-1retry_limit 2

在efs卷中配置的fluentbit extra-config，为input打上tag，抓取/opt/web/infolog.txt路径下日志

[INPUT]Name              tailTag               myweb-firelensPath              /opt/web/infolog.txtDB                /var/log/myweb_service.dbDB.locking        trueSkip_Long_Lines   OnRefresh_Interval  10Rotate_Wait       30[OUTPUT]Name  stdoutMatch myweb-firelens

查看业务容器访问日志

$ cat infolog.txt
2022-09-27 13:23:47,244 - test-log-info - INFO - info
2022-09-27 13:23:47,244 - test-log-info - ERROR - error
2022-09-27 13:23:47,248 - test-log-info - WARNING - warning

我们将容器访问日志打到fluentbit的stdout中

$ docker logs d8963f1052a4
[0] myweb-firelens: [1669883566.583834793, {"log"=>"2022-09-27 13:23:47,244 - test-log-info - INFO - info", "ec2_instance_id"=>"i-xxxxxxxxxxxx", "ecs_cluster"=>"worktest", "ecs_task_arn"=>"arn:aws-cn:ecs:cn-north-1:xxxxxxxx:task/worktest/0a5ecaa32f784edab3a723b8c8390a06", "ecs_task_definition"=>"myweb:9"}]
[1] myweb-firelens: [1669883566.583836988, {"log"=>"2022-09-27 13:23:47,244 - test-log-info - ERROR - error", "ec2_instance_id"=>"i-xxxxxxxxxxxxxx", "ecs_cluster"=>"worktest", "ecs_task_arn"=>"arn:aws-cn:ecs:cn-north-1:xxxxxxxx:task/worktest/0a5ecaa32f784edab3a723b8c8390a06", "ecs_task_definition"=>"myweb:9"}]
[2] myweb-firelens: [1669883566.583837240, {"log"=>"2022-09-27 13:23:47,248 - test-log-info - WARNING - warning", "ec2_instance_id"=>"i-xxxxxxxxxxxxxx", "ecs_cluster"=>"worktest", "ecs_task_arn"=>"arn:aws-cn:ecs:cn-north-1:xxxxxxxx:task/worktest/0a5ecaa32f784edab3a723b8c8390a06", "ecs_task_definition"=>"myweb:9"}]

此时ecs容器已经处于运行状态，我们只需要将efs下的/test-ecs/firelens-config路径挂载到任意ec2上对fluentbit配置进行编辑，此时查看容器中的配置会动态修改，实现动态配置的目的