「Harbor」- HTTP status 504 Gateway Time-out @20210318

# 07/20/2019 # 我刚想处理这个问题，好了个屁的。这种最可怕了。

问题背景

使用docker push命令推送镜像。

在某个镜像层推送将要结束的时候，收到如下错误信息：

# docker push docker-registry.example.com/project/image-foo:v2.5.0
The push refers to repository [docker-registry.example.com/project/image-foo:v2.5.0]
3d10bce3ce31: Layer already exists
1b70b131f01d: Layer already exists
2c5ef295a217: Pushing [==================================================>]  375.4MB
2ce4f2676a16: Layer already exists
35bbe47ffc77: Layer already exists
05b61dc1031f: Layer already exists
d6d37cde236f: Layer already exists
ecbc53aebc27: Layer already exists
1585039add0a: Layer already exists
692d855fb28e: Layer already exists
717b092b8c86: Layer already exists
received unexpected HTTP status: 504 Gateway Time-out

查看/var/log/registry.log文件，日志中有如下信息（经过了简单的处理）：

# Jul 22 18:26:55 172.18.0.1 proxy[1114]: 2019/07/22 10:26:55 [error] 8#0: *11392 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.50.63, server: , request: "PATCH /v2/veryeast/ve-m-ssr/blobs/uploads/a8008eea-5412-40d2-a566-4efc71b72cb3?_state=gvZ4sODnz4enCabFx1TGEpNdkLaLAjkitundRI3az197Ik5hbWUiOiJ2ZXJ5ZWFzdC92ZS1tLXNzciIsIlVVSUQiOiJhODAwOGVlYS01NDEyLTQwZDItYTU2Ni00ZWZjNzFiNzJjYjMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDctMjJUMTA6MjU6MTkuMzU4NzkwNzk1WiJ9 HTTP/1.1", upstream: "http://172.18.0.6:8080/registryproxy/v2/veryeast/ve-m-ssr/blobs/uploads/a8008eea-5412-40d2-a566-4efc71b72cb3?_state=gvZ4sODnz4enCabFx1TGEpNdkLaLAjkitundRI3az197Ik5hbWUiOiJ2ZXJ5ZWFzdC92ZS1tLXNzciIsIlVVSUQiOiJhODAwOGVlYS01NDEyLTQwZDItYTU2Ni00ZWZjNzFiNzJjYjMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDctMjJUMTA6MjU6MTkuMzU4NzkwNzk1WiJ9", host: "docker-registry.cluster.dfwsgroup.cn"
# Jul 22 18:26:55 172.18.0.1 proxy[1114]: 192.168.50.63 - "PATCH /v2/veryeast/ve-m-ssr/blobs/uploads/a8008eea-5412-40d2-a566-4efc71b72cb3?_state=gvZ4sODnz4enCabFx1TGEpNdkLaLAjkitundRI3az197Ik5hbWUiOiJ2ZXJ5ZWFzdC92ZS1tLXNzciIsIlVVSUQiOiJhODAwOGVlYS01NDEyLTQwZDItYTU2Ni00ZWZjNzFiNzJjYjMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDctMjJUMTA6MjU6MTkuMzU4NzkwNzk1WiJ9 HTTP/1.1" 504 176 "-" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.18.12-1.el7.elrepo.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \x5C(linux\x5C))" 95.187 95.186 .

环境信息

操作系统：CentOS Linux release 7.4.1708 (Core) 这种问题和操作的关系不大。

软件版本：Harbor v1.4.0

所需资料

仓库地址：https://github.com/goharbor/harbor/tree/v1.4.0
安装配置：Installation & Configuration Guide
用户手册：User Guide
软件架构：Architecture Overview of Harbor

问题排查

暂缓问题

「replication for big images failed with Gateway Time-out intermittently #3446」

修改common/templates/nginx/nginx.http(s).conf文件中的location /v2/部分，添加proxy_send_timeout 900;与proxy_read_timeout 900;配置行。因为从Nginx的日志看是访问/v2/时产生的超时。

可能原因

系统磁盘IO存在问题：

   #1 在进行镜像传输时候，系统的「平均负载」比较高，四核，处于3.5以上。而''iostat''命令显示系统IO并不是很高。

因此推测是IO较低，导致镜像写入磁盘的时间比较长，由于超过了Nginx的等待时间，而返回了504错误。

另外，系统中并没有相关Harbor错误日志。在时间点附近，大部分日志都是正常的。

参考文献

docker registry keep showing manifest unknown error for image pull after somedays of push image #17031
Delete tag failed #6515