k8s(十三)、企业级docker仓库Harbor在kubernetes上搭建使用

前言

Harbor是Docker镜像仓库管理系统，用于企业级管理容器镜像，支持对接ldap进行权限管理,由VMVare中国团队开发。
项目地址:
https://github.com/goharbor/harbor/
安装方式:
1.Docker Compose方式部署:
https://github.com/goharbor/harbor/blob/master/docs/installation_guide.md
2.通过kubernetes直接部署:
新版本已废弃
3.通过helm/kubernetes方式部署:
https://github.com/goharbor/harbor-helm/blob/master/README.md
推荐使用,本文将采用此方式

Helm安装

helm简介

Helm是一个用于Kubernetes的包管理工具,每一个安装包构成一个chart基本单位.想象一下,一个完整的部署,可能包含前端\后端\中间件\DB的部署实例,以及k8s管理对应的secret/configmap/svc/ing/pv/pvc等众多资源类型,在部署管理或者卸载之前,如果逐一编辑管理这些资源的yaml文件,工作量比较庞大且容易疏漏.helm的出现很好的解决了这个问题,使用模板以及变量替换的方式,按需生成部署yaml文件包,并且向kubernetes api发送调度请求,虽然额外地引入了一些学习成本,但对于部署包的管理无疑是带来了很大的便利性的.

helm组件

Helm
Helm是一个cli客户端，可以完成如下内容：

1.创建/打包/调试Chart
2.创建本地Chart仓库,管理本地和远程Chart仓库
3.与Tiller交互并完成Chart的安装，升级，删除，回滚，查看等操作

Tiller
1.监听helm客户端的请求,根据chart生成相应的release
2.将release的资源信息发送给kubernetes api,实现资源增删改查.
3.追踪release的状态,实现资源的更新/回滚操作

安装:
安装较为简单,参考官方文档,不做复述:
https://docs.helm.sh/using_helm/#installing-helm

Harbor安装

这里采用次新版本的v1.5.3版本

root@h1:# wget https://github.com/goharbor/harbor/archive/v1.5.3.tar.gz
root@h1:# tar -xf https://github.com/goharbor/harbor/archive/v1.5.3.tar.gz
root@h1:# cd harbor-1.5.3/contrib/helm/harbor/

helm方式安装的harbor默认是https的,因此需要k8s集群ingress网关开启https协议,traefik开启https参考此前文章:
k8s（二）、对外服务

更新helm dependency:

harbor的helm部署依赖了postgresql的helm，在官方的安装文档没有明确说明，直接按照官方文档说明安装，就会缺失postgresql的部署，导致helm install failed

helm dependency update

修改value.yaml文件,添加pvc持久存储并将相应组件上下文中的volume部分注释打开:

mysql:volumes:data:storageClass: "cephrbd"accessMode: ReadWriteOnceadminserver:volumes:config:storageClass: "cephrbd"accessMode: ReadWriteOncesize: 1Gregistry:volumes:data:storageClass: "cephrbd"accessMode: ReadWriteOncesize: 5G

完整value.yaml文件如下:

# Configure persisten Volumes per application
## Applications that require storage have a `volumes` definition which will be used
## when `persistence.enabled` is set to true.
## example
# mysql:
#   volumes:
#     data:
## Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
##   set, choosing the default provisioner.  (gp2 on AWS, standard on
##   GKE, AWS & OpenStack)
##
#     storageClass: "-"
#     accessMode: ReadWriteOnce
#     size: 1Gimysql:volumes:data:storageClass: "cephrbd"accessMode: ReadWriteOnce
adminserver:volumes:config:storageClass: "cephrbd"accessMode: ReadWriteOncesize: 1Gregistry:volumes:data:storageClass: "cephrbd"accessMode: ReadWriteOncesize: 5G## Configure resource requests and limits per application
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
# mysql:
#   resources:
#     requests:
#       memory: 256Mi
#       cpu: 100mpersistence:enabled: true# The tag for Harbor docker images.
harborImageTag: &harbor_image_tag v1.4.0# The FQDN for Harbor service.
externalDomain: harbor.my.domain
# If set to true, you don't need to set tlsCrt/tlsKey/caCrt, but must add
# Harbor FQDN as insecure-registries for your docker client.
insecureRegistry: false
#insecureRegistry: true
# The TLS certificate for Harbor. The common name of tlsCrt must match the externalDomain above.
tlsCrt:
#tlsCrt: root/traefik/ssl/kokoerp.crt
tlsKey:
#tlsKey: root/traefik/ssl/kokoerp.key
caCrt:# The secret key used for encryption. Must be a string of 16 chars.
secretKey: not-a-secure-key# These annotations allow the registry to work behind the nginx
# ingress controller.
ingress:annotations:ingress.kubernetes.io/ssl-redirect: "true"ingress.kubernetes.io/body-size: "0"ingress.kubernetes.io/proxy-body-size: "0"adminserver:image:repository: vmware/harbor-adminservertag: *harbor_image_tagpullPolicy: IfNotPresentemailHost: "smtp.mydomain.com"emailPort: "25"emailUser: "sample_admin@mydomain.com"emailSsl: "false"emailFrom: "admin <sample_admin@mydomain.com>"emailIdentity: ""emailInsecure: "False"emailPwd: not-a-secure-passwordadminPassword: Harbor12345authenticationMode: "db_auth"selfRegistration: "on"ldap:url: "ldaps://ldapserver"searchDN: ""baseDN: ""filter: "(objectClass=person)"uid: "uid"scope: "2"timeout: "5"verifyCert: "True"## Persist data to a persistent volumevolumes:config:storageClass: "cephrbd"accessMode: ReadWriteOncesize: 1Gi# resources:#  requests:#    memory: 256Mi#    cpu: 100m## jobservice
#
jobservice:image:repository: vmware/harbor-jobservicetag: *harbor_image_tagpullPolicy: IfNotPresentsecret: not-a-secure-secret
# resources:
#   requests:
#     memory: 256Mi
#     cpu: 100m## UI
#
ui:image:repository: vmware/harbor-uitag: *harbor_image_tagpullPolicy: IfNotPresentsecret: not-a-secure-secretprivateKeyPem: |-----BEGIN RSA PRIVATE KEY-----MIIJKAIBAAKCAgEA4WYbxdrFGG6RnfyYKlHYML3lEqtA9cYWWOynE9BeaEr/cMnMbBr1dd91/Nm6RiYhQvTDU2Kc6NejqjdliW5B9xUoVKayri8OU81a8ViXeNgKwCPRAiTTla1zoX5DnvoxpO9G3lxyNvTKXc0cw8NjQDAXpaDbzJYLkshCeuyD9bco8R96/zrpBEX8tADN3+3yA3fMcZzVXsBm4BTpHJRk/qBpHYEPSHzxyH3iGMNKk3vMUBZze0EYkK8NCA2CuEKMnC3acx9IdRwkx10abGvHQCLRCVY7rGoak+b0oZ99RJIRQ9IqYXsn8fsMBQly6xxvSeY5XuSP7Xb6JKDt3y8Spi4gR1M/5aEzhuOyu201rMna7Rs/GPfaKjBlbX0jiLDa7v4zjsBPsPaf/c4uooz3ICLsdukaom+E538R0EiOkXt/wyw22YmaWNCsYlEpke7cVC33e/0dPBq4IHsVflawSF9OWS23ikVAs/n+76KjuucEDmbTaKUYAJjvAmZL14j+EKc/CoplhCe6pKhavjmNIOfCSdlreIPBhOVbf1f817wKoSIZqVyCA1AYNkI9RYS00axtJGBGMlKbdQqCNpLL58c6To2awmckIZCEcATKOp++NoGmIb0bhdSasdGB5VCtwZVluN8bLl13zBKoxTGjNlEatUGDRnDAnLdZbXXffjsCAwEAAQKCAgBEUigO8/4UJse6xKr3APHv7E94NjKtjMqPT8RhDCLhqAH/lRuClTVb8k0YRILi6oHggsKGDvkS1vJEESCU5LfYBjDAX/r/M0I7gp6TU1AukAXKMdETvkfoMbg/9j7W/G152hF4KztvjwmcHyUd7aay+SDh0n1taPm/FzaXfgONwmQFmo40uQ2SfwhXI3tD6iMWjASLV4eRfe5w88WpJQ3r5IGYMNuKFF1RcV7MNL3xMHBAwl1kudmRWY4wp6+83Gc0m+2AQbY70TkQuRbeUFkIBsWn99yEqXC+7h2us+JLm57iGN1ByQvVnEwLZs7Pl0Hge4leSxeZWhv+aE1R/jm/VdG4dglInuhED0ug8WAJg58IkDYfMKOOALHx+0CNHE02XqqUIFwboZJSYTjMYvFL1i14L30FWnqH/0kDs4whXHbnGWhVustsMSK9iyIGepuGhMnvtUF1wa/SrBd12qfDj68QHDXsKKbs6eTNYHfn3QL9uisrfMIa5HAtnX2YOsAVxg+yvxkWD6n1DU+a/+pAu6iAgiwyxSZiyn6vJUE2zO6pJNbk1kJW6jU3A69srtbO4jQn4EM859XYSqdqwXgJL+XJEYNbBcHalmiIOvRg9CCvDSKS7M5rJ0M1L7oCzl6EW+zUb4JHkSO7V5uxIZu2sEduw5gofQ3OT9L/qDhDIQKCAQEA8T/8okF2Q7SOj3su6KKX6H/ab31SvHECf/oeJtH8ZfLBYL55Yof0pZwq8iXQ26d8cH7FPKBohz0RZ9i2S3bYkzEVCPv9ISFg1NACxL3dU0PMBnmbmg2vPhMzEuQI2JOUu6ILOXENmImvfjZXps/b8OjQgzicH0skBBcbUlXT3a4fF52ktC8FiXgBG9JYg5CsXmfPRxciITa4w4ZLEuECmtJieS0MdKXPLwUVv3e2BlNS6c1JzXyp6EyX/euJ8cCe3n/GHbTY2j1OO+xTQfQJVf6S9f2mSzjdHe9KZwWKgyxQ9dZ9Qtho2z/gUN9/UkL52fdljjlw++b/z9Ppcl9K0QKCAQEA7y4Fv8dPFLLnr0R/S7eoAKa0S95xVe97EJHVUhWyOI09K9VdZHp6be8W0Yd9h/Ks8Zi4EPRiTTaF3yA3iADwdKFeZt49jGzeM+Gl7Q2Ll98WI5gOdJkHSVAP2uK7qSjZ8lPCu4iUYRsae+Psam7Yd6X17RP0M966PlUFj1nnrJjQEN4zeh/m01q9vqebB9C1W/ZiJ6rpt6VVHAcOQQ69F/lKdTif4XCvbMIhIXTYNifk1oIv2qTDnfzzv+bgrlvpBJPpPYR0Oc7WoEpyd1Y9IzienLZi8RnujV//FXEmJ45EF9GE1HOmoERdEWA1bMYhOO5OfRY1HSMuFMA4+5ojSwKCAQEAmwubio/1uMemw3HQkPRGGsdolDR/4tniWGtfy2UzCDY+r7Vaf8eOpIy8UQmatEBsykO+8RrKcvf9Yrc1WUSVJevqb+67HPq9p6fTz6uSPXwZ+KNZLGXVFVjzfxWM1dvrP7eB7TXKHhmG7t9v76Yw3SBTObI9LCN3jyVmisDcO+E23E+VVbPOpC260K2b81ocXUPsQ+0LIztu/UImp4hyyxug6+3WznTttXNYKch+9IvCgr5Ly0NuUvw+xpMFAZjgwXBu3BKpN4Ek8YANdhqnkVveCTguErQF78IlGBbIkUr+8TAbKsW4hggEWxV4V17yAnJsEz65bTtldqTjqHyzsQKCAQBGhv6g/2d9Rgf1cbBLpns+vel6Wbx3x6c1SptpmgY0kMlR7JeeclM5qX/EBzzn4pJGp27XaQi3lfVBxyE41HYTHiZVFQF3L/8Rs18XGKBqBxljI4pXrWwtnRMfyy3lAqvJvhM082A1hiV4FMx40fi4x1JON00SIoIusSlzjOI4zdLEtpDdWRzag+5hktCvLEbeODfXVJmYUoNXQWldm7f8osDm8eyLMIw5+MCGOgsrZPYgnsD3qxAXvSgvFSh5oZaDiA4F2tHe3fQBzhIUyHQ8t4xlz447ZBcozv7L1tKWZWgE0f5mGzguGBqNbh4y1fWj8Plp/ytoTSBgdBIZdukjAoIBAELJPSVFnlf/gv6OWRCHyKxquGjvfEn/E8bw5WSqMcj/7wiSJozr0Y8oyWjtWXObliLRQXcEhC8w3lLMjNqnFzQOAI7sOa6BQPigqyXZPXG5GK+V0TlUYvZQn9sfCq4YCxUBNtQ4GHbKKl3FGQL3rJiuFr6GfVcetuDFNCiIGYbUF+giJ2cEN3a/Q+7fR6V4xC7VDdL+BqM09wZ6R98G48XzCKKpekNpEfmvJiuk9tFFQwDPWcQ6uyHqesK/Wiweo5nh5y2ZPipwcb0uBoYOQH60NqEL6MXRVNdtKujjl1XZkG053Nvcz/YfF6lFjDekwgfd9m49b/s0EGTrl7z9z8Y=-----END RSA PRIVATE KEY-----
# resources:
#  requests:
#    memory: 256Mi
#    cpu: 100m## MySQL Settings. Currently Harbor does not support an external
## MySQL server, only their own image. Until this is fixed, do not
## Change the settings below.
#
mysql:image:repository: vmware/harbor-dbtag: *harbor_image_tagpullPolicy: IfNotPresent# If left blank will use the included mysql service name.host: ~port: 3306user: "root"pass: "registry"database: "registry"volumes:data:storageClass: "cephrbd"accessMode: ReadWriteOncesize: 1Gi# resources:#  requests:#    memory: 256Mi#    cpu: 100mregistry:image:repository: vmware/registry-photontag: v2.6.2-v1.4.0pullPolicy: IfNotPresenthttpSecret: not-a-secure-secretlogLevel:
# comment out one of the below to use your cloud's object storage.
#  objectStorage:
#    gcs:
#      keyfile: ""
#      bucket: ""
#      chunksize: "5242880"
#    s3:
#      region: ""
#      accesskey: ""
#      secretkey: ""
#      bucket: ""
#      encrypt: "true"
#    azure:
#      accountname: ""
#      accountkey: ""
#      container: ""rootCrt: |-----BEGIN CERTIFICATE-----MIIE0zCCArugAwIBAgIJAIgs3S+hsjhmMA0GCSqGSIb3DQEBCwUAMAAwHhcNMTcxMTA5MTcyNzQ5WhcNMjcxMTA3MTcyNzQ5WjAAMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA4WYbxdrFGG6RnfyYKlHYML3lEqtA9cYWWOynE9BeaEr/cMnMbBr1dd91/Nm6RiYhQvTDU2Kc6NejqjdliW5B9xUoVKayri8OU81a8ViXeNgKwCPRAiTTla1zoX5DnvoxpO9G3lxyNvTKXc0cw8NjQDAXpaDbzJYLkshCeuyD9bco8R96/zrpBEX8tADN3+3yA3fMcZzVXsBm4BTpHJRk/qBpHYEPSHzxyH3iGMNKk3vMUBZze0EYkK8NCA2CuEKMnC3acx9IdRwkx10abGvHQCLRCVY7rGoak+b0oZ99RJIRQ9IqYXsn8fsMBQly6xxvSeY5XuSP7Xb6JKDt3y8Spi4gR1M/5aEzhuOyu201rMna7Rs/GPfaKjBlbX0jiLDa7v4zjsBPsPaf/c4uooz3ICLsdukaom+E538R0EiOkXt/wyw22YmaWNCsYlEpke7cVC33e/0dPBq4IHsVflawSF9OWS23ikVAs/n+76KjuucEDmbTaKUYAJjvAmZL14j+EKc/CoplhCe6pKhavjmNIOfCSdlreIPBhOVbf1f817wKoSIZqVyCA1AYNkI9RYS00axtJGBGMlKbdQqCNpLL58c6To2awmckIZCEcATKOp++NoGmIb0bhdSasdGB5VCtwZVluN8bLl13zBKoxTGjNlEatUGDRnDAnLdZbXXffjsCAwEAAaNQME4wHQYDVR0OBBYEFCMYYMOL0E/Uyj5wseDfIl7o4ELsMB8GA1UdIwQYMBaAFCMYYMOL0E/Uyj5wseDfIl7o4ELsMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQELBQADggIBABG8fPvrrR+erpwQFuB/56j2i6sO+qoOJPpAMYwkzICrT0eerWAavwoyf0UAKN7cUeEJXjIR7s7CogGFijWdaWaQsXUD0zJq5aotLYZLimEc1O0uAmJEsfYCv7mG07eU6ge22sSo5hxhVplGt52hnXnT0DdgSRZpq2mvgd9lcopAidM+KHlaasXkIecHKM99KX9D8smr0AcQ6M/Ygbf2qjO9YRmpBIjyQWEake4y/4LWm+3+v08ecg4Bg+iMC0Rw1QcPqgwaGaWu71RtYhyTg7SnAknb5nBcHIbLb0hdLgQTa3ZdtXgqchIiGuFlEBmHFZP6bLJORRUQ0ari5wpXIsYfrB4T8PybTzva3OCMlEsMjuysFr9ewhzM9UGLiSQNDyKA10J8WwlzbeD0AAW944hW4Dbg6SWv4gAo51T+6AukRdup5y6lfQ5ah4Lbo6pzaA369IsJBntvKvia6hUf/SghnbG7pCHX/AEilcgTb13HndF/G+7aZgKRmi9qvNRSDsE/BrgZawovp81+j6aL4y6UtXYspHr+SuWsKYsaH7pl5HspNCyJ5vV6dpJAwosFBqSEnI333wAunpMYmi/jKHH/j4WqjLnCInp0/wouzYu42l8Pmz591BSpJag500bEBxqI2RLELgMt/bUdjp4N2M7mrxdrN+2579HTzb6Hviu9-----END CERTIFICATE-----## Persist data to a persistent volumevolumes:data:storageClass: "cephrbd"accessMode: ReadWriteOncesize: 5Gi# resources:#  requests:#    memory: 256Mi#    cpu: 100mclair:enabled: trueimage:repository: vmware/clair-photontag: v2.0.1-v1.4.0pullPolicy: IfNotPresent
## The following needs to match the credentials
## in the `postgresql` configuration under the
## `postgresql` namespace below.postgresPassword: not-a-secure-passwordpostgresUser: clairpostgresDatabase: clair
# resources:
#  requests:
#    memory: 256Mi
#    cpu: 100m
# pgResources:
#  requests:
#    memory: 256Mi
#    cpu: 100m
#  volumes:
#    pgData:
#      storageClass: "cephrbd"
#      accessMode: ReadWriteOnce
#      size: 1Gi# resources:#  requests:#    memory: 256Mi#    cpu: 100m## Notary support is not yet fully implemented in the Helm Charts
## Enabling it will just break things.
#
notary:enabled: false## Settings for postgresql dependency.
## see https://github.com/kubernetes/charts/tree/master/stable/postgresql
## for further configurables.
postgresql:postgresUser: clairpostgresPassword: not-a-secure-passwordpostgresDatabase: clairpersistence:# enabled: falseenabled: truestorageClass: "cephrbd"accessMode: ReadWriteOncesize: 8Gi

部署:

helm install . --debug --name hub --set externalDomain=hub.test.com

等待一段时间的镜像拉取后,查看pod:

hub-harbor-adminserver-0                 1/1       Running            1          3h
hub-harbor-clair-6c7d9dcdb7-q4lv4        1/1       Running            2          3h
hub-harbor-jobservice-75f7fbcc9c-ggwp4   1/1       Running            2          3h
hub-harbor-mysql-0                       1/1       Running            0          3h
hub-harbor-registry-0                    1/1       Running            1          3h
hub-harbor-ui-57b4674ff9-kcfq6           1/1       Running            0          3h
hub-postgresql-ccf8d56d5-jg4wq           1/1       Running            1          3h

添加dns,指向ingress gateway(即traefik的node ip):

echo "192.168.1.238  hub.test.com"  >> /etc/hosts

浏览器打开测试,默认登录口令为admin/Harbor12345

可以看到默认有一个公开项目library,尝试往这里推送镜像

Docker推送测试

找一台安装了docker的机器,修改docker服务脚本:

~# cat /lib/systemd/system/docker.service#ExecStart=/usr/bin/dockerd -H fd://
ExecStart=/usr/bin/dockerd  --insecure-registry hub.kokoerp.com

重启docker并登录测试:

~# systemctl daemon-reload
~# systemctl restart dockerroot@h2:~# docker login hub.test.com
Username (admin): admin
Password:
Login Succeeded

推送镜像测试:

root@h2:~# docker tag busybox hub.test.com/library/busybox:test1
root@h2:~# docker push hub.test.com/library/busybox:test1
The push refers to a repository [hub.test.com/library/busybox]
8a788232037e: Pushed
test1: digest: sha256:915f390a8912e16d4beb8689720a17348f3f6d1a7b659697df850ab625ea29d5 size: 527

可以看到推送成功

ldap配置:

harbor可以基于企业内部的ldap来获取人员信息,可以在ui上创建非公开项目,将ldap中获取的人员加入项目并赋予相应pull push权限,ui操作界面比较友好,这里就不展示了.