Kubernetes存储之Rook的ceph搭建

环境准备：

主机名	IP	角色
master01	192.168.200.182	k8s-master,rook release-0.9
node	192.168.200.183	k8s-node01
node	192.68.200.184	k8s-node02

ceph简介：

Ceph 是一个开源的分布式存储系统，包括对象存储、块设备、文件系统。它具有高可靠性、安装方便、管理简便、能够轻松管理海量数据。Ceph 存储集群具备了企业级存储的能力，它通过组织大量节点，节点之间靠相互通讯来复制数据、并动态地重分布数据，从而达到高可用分布式存储功能

rook简介：

Rook 是专用于 Cloud-Native 环境的文件、块、对象存储服务。它实现了一个自动管理的、自动扩容的、自动修复的分布式存储服务。Rook 支持自动部署、启动、配置、分配、扩容/缩容、升级、迁移、灾难恢复、监控以及资源管理。为了实现所有这些功能，Rook 需要依赖底层的容器编排平台，例如 kubernetes、CoreOS 等。Rook 目前支持 Ceph、NFS、Minio Object Store、Edegefs、Cassandra、CockroachDB 存储的搭建，后期会支持更多存储方案。

Rook的主要组件有两个，功能如下：
Rook Operator:
Rook与Kubernetes交互的组件
整个Rook集群只有一个

Rook Agent:
与Rook Operator交互，执行命令
每个Kubernetes的Node上都会启动一个
不同的存储系统，启动的Agent是不同的

获取rook源，切换为release1.1

[root@master01 ~]# git clone https://github.com/rook/rook.git
[root@master01 ~]# cd rook/
[root@master01 rook]# [root@master01 rook]# git checkout -b release-0.9 remotes/origin/release-0.9
分支 release-0.9 设置为跟踪来自 origin 的远程分支 release-0.9。
切换到一个新分支 'release-0.9'
[root@master01 rook]# git branch -amaster
* release-0.9release-1.1remotes/origin/HEAD -> origin/masterremotes/origin/masterremotes/origin/release-0.4remotes/origin/release-0.5remotes/origin/release-0.6remotes/origin/release-0.7remotes/origin/release-0.8remotes/origin/release-0.9

为什么要切换？这里是个深坑。

[root@master01 ~]# git clone https://github.com/rook/rook.git
[root@master01 ~]# cd rook//cluster/examples/kubernetes/ceph/
[root@master01 ceph]# kubectl create -f operator.yaml
[root@master01 ceph]# kubectl create -f cluster.yaml

如果正常的话，Rook 会创建好所有需要的资源，但是很遗憾，你会发现当 cluster.yaml 创建完毕后，不会创建 rook-ceph-mgr、rook-ceph-mon、rook-ceph-osd 等资源。参考https://github.com/rook/rook/issues/2338这里，我们可以通过查看 rook-ceph-operator Pod 的日志来分析下：

[root@master01 ~]# kubectl logs -n rook-ceph-system rook-ceph-operator-68576ff976-m9m6l
......
E0107 12:06:23.272607       6 reflector.go:205] github.com/rook/rook/vendor/github.com/rook/operator-kit/watcher.go:76: Failed to list *v1beta1.Cluster: the server could not find the requested resource (get clusters.ceph.rook.io)
E0107 12:06:24.274364       6 reflector.go:205] github.com/rook/rook/vendor/github.com/rook/operator-kit/watcher.go:76: Failed to list *v1beta1.Cluster: the server could not find the requested resource (get clusters.ceph.rook.io)
E0107 12:06:25.288800       6 reflector.go:205] github.com/rook/rook/vendor/github.com/rook/operator-kit/watcher.go:76: Failed to list *v1beta1.Cluster: the server could not find the requested resource (get clusters.ceph.rook.io)

类似以上日志输出，这是因为创建的 CRDs资源版本不匹配导致的。正确的方法就是切换到最新固定版本

部署Rook Operator

[root@master01 ~]# cd rook/cluster/examples/kubernetes/ceph/
[root@master01 ceph]# kubectl create -f operator.yaml
namespace/rook-ceph-system created
customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created
clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
role.rbac.authorization.k8s.io/rook-ceph-system created
clusterrole.rbac.authorization.k8s.io/rook-ceph-global created
clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
serviceaccount/rook-ceph-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-system created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created
deployment.apps/rook-ceph-operator created
[root@master01 ceph]# kubectl get pods -n rook-ceph-system
NAME                                 READY     STATUS              RESTARTS   AGE
rook-ceph-operator-9f9c5ffd7-zzpzn   0/1       ContainerCreating   0          22s
[root@master01 ceph]# kubectl get pods -n rook-ceph-system
NAME                                 READY     STATUS    RESTARTS   AGE
rook-ceph-agent-765kj                1/1       Running   0          50s
rook-ceph-agent-9bk72                1/1       Running   0          50s
rook-ceph-operator-9f9c5ffd7-85btz   1/1       Running   0          1m
rook-discover-4b5qh                  1/1       Running   0          50s
rook-discover-gkt67                  1/1       Running   0          50s

说明一下，这里先创建了 rook-ceph-operator，然后在由它在每个节点创建 rook-ceph-agent 和 rook-discover。接下来，就可以部署 CephCluster 了。

[root@master01 ceph]# kubectl create -f cluster.yaml
namespace/rook-ceph created
serviceaccount/rook-ceph-osd created
serviceaccount/rook-ceph-mgr created
role.rbac.authorization.k8s.io/rook-ceph-osd created
role.rbac.authorization.k8s.io/rook-ceph-mgr-system created
role.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
cephcluster.ceph.rook.io/rook-ceph created
[root@master01 ceph]# kubectl get cephcluster -n rook-ceph
NAME        DATADIRHOSTPATH   MONCOUNT   AGE       STATE
rook-ceph   /var/lib/rook     3          34s
[root@master01 ceph]# kubectl get pod -n rook-ceph
NAME                               READY     STATUS     RESTARTS   AGE
rook-ceph-mon-a-ddbc448d6-5q466    0/1       Init:1/3   0          2m
rook-ceph-mon-d-5d696f6fbb-t2pd2   1/1       Running    0          1m
[root@master01 ceph]# kubectl get pod -n rook-ceph -o wide
NAME                                 READY     STATUS      RESTARTS   AGE       IP            NODE
rook-ceph-mgr-a-56d5cbc754-p9gsn     1/1       Running     0          1m        10.244.2.12   node02
rook-ceph-mon-a-ddbc448d6-5q466      1/1       Running     0          12m       10.244.1.10   node01
rook-ceph-mon-d-5d696f6fbb-t2pd2     1/1       Running     0          10m       10.244.2.11   node02
rook-ceph-mon-f-576847df6c-ntwjf     1/1       Running     0          8m        10.244.1.11   node01
rook-ceph-osd-0-66469b499d-ls9pg     1/1       Running     0          19s       10.244.1.13   node01
rook-ceph-osd-1-757c69cfcf-th667     1/1       Running     0          17s       10.244.2.14   node02
rook-ceph-osd-prepare-node01-bgk4j   0/2       Completed   1          34s       10.244.1.12   node01
rook-ceph-osd-prepare-node02-d5q54   0/2       Completed   1          34s       10.244.2.13   node02

cephcluster 是一个 CRD 自定义资源类型，通过它来创建一些列 ceph 的 mgr、osd 等。我们可以直接使用默认配置，默认开启 3 个 mon 资源，dataDirHostPath 存储路径在 /var/lib/rook，当然也可以自定义配置，例如 DATADIRHOSTPATH、MONCOUNT 等，可以参考https://rook.github.io/docs/rook/v0.9/ceph-cluster-crd.html

配置 Rook Dashboard

[root@master01 ceph]# kubectl get svc -n rook-ceph
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
rook-ceph-mgr             ClusterIP   10.102.154.66    <none>        9283/TCP   4m
rook-ceph-mgr-dashboard   ClusterIP   10.105.132.253   <none>        8443/TCP   4m
rook-ceph-mon-a           ClusterIP   10.109.149.87    <none>        6790/TCP   15m
rook-ceph-mon-d           ClusterIP   10.99.79.25      <none>        6790/TCP   13m
rook-ceph-mon-f           ClusterIP   10.96.202.164    <none>        6790/TCP   12m
#修改service类型为NodePort：
[root@master01 ceph]# vim dashboard-external-https.yaml
[root@master01 ceph]# cat dashboard-external-https.yaml
apiVersion: v1
kind: Service
metadata:name: rook-ceph-mgr-dashboard-external-httpsnamespace: rook-cephlabels:app: rook-ceph-mgrrook_cluster: rook-ceph
spec:ports:- name: dashboardport: 8443protocol: TCPtargetPort: 8443nodePort: 30007 # 固定端口访问selector:app: rook-ceph-mgrrook_cluster: rook-cephsessionAffinity: Nonetype: NodePort # 修改类型
#部署
[root@master01 ceph]# kubectl create -f dashboard-external-https.yaml
service/rook-ceph-mgr-dashboard-external-https created
[root@master01 ceph]# kubectl get svc -n rook-ceph
NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
rook-ceph-mgr                            ClusterIP   10.102.154.66    <none>        9283/TCP         6m
rook-ceph-mgr-dashboard                  ClusterIP   10.105.132.253   <none>        8443/TCP         6m
rook-ceph-mgr-dashboard-external-https   NodePort    10.99.174.253    <none>        8443:30007/TCP   7s
rook-ceph-mon-a                          ClusterIP   10.109.149.87    <none>        6790/TCP         18m
rook-ceph-mon-d                          ClusterIP   10.99.79.25      <none>        6790/TCP         16m
rook-ceph-mon-f                          ClusterIP   10.96.202.164    <none>        6790/TCP         14m

测试访问https://:30007

获取两种密码方式(默认用户admin)：

方式一：rook-ceph 默认创建了一个 rook-ceph-dashboard-password 的 secret，可以用这种方式获取 password。
[root@master01 ceph]# kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}'  |  base64 --decode
QIOiG58xcR方式二:从 rook-ceph-mgr Pod 的日志中获取，日志会打印出来用户名和密码。
[root@master01 ceph]# kubectl get pod -n rook-ceph | grep mgr
rook-ceph-mgr-a-56d5cbc754-p9gsn     1/1       Running     0          10m
[root@master01 ceph]# kubectl -n rook-ceph logs rook-ceph-mgr-a-56d5cbc754-p9gsn | grep password
2019-11-29 08:26:11.446 7f25e6654700  0 log_channel(audit) log [DBG] : from='client.4139 10.244.1.8:0/1021643349' entity='client.admin' cmd=[{"username": "admin", "prefix": "dashboard set-login-credentials", "password": "QIOiG58xcR", "target": ["mgr", ""], "format": "json"}]: dispatch

部署Rook toolbox测试

[root@master01 ceph]# kubectl create -f toolbox.yaml
deployment.apps/rook-ceph-tools created
[root@master01 ceph]# kubectl -n rook-ceph get pod -l "app=rook-ceph-tools"
NAME                               READY     STATUS    RESTARTS   AGE
rook-ceph-tools-76bf8448f6-lhk26   1/1       Running   0          35s
#验证是否成功（错误提示不用管）
[root@master01 ceph]# kubectl -n rook-ceph exec -it rook-ceph-tools-76bf8448f6-lhk26 bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
#测试
[root@node02 /]# ceph statuscluster:id:     8104f7e4-26fc-4b45-b67b-6fa7a128dce6health: HEALTH_OKservices:mon: 3 daemons, quorum f,d,amgr: a(active)osd: 2 osds: 2 up, 2 indata:pools:   0 pools, 0 pgsobjects: 0  objects, 0 Busage:   12 GiB used, 22 GiB / 34 GiB availpgs:     [root@node02 /]# ceph osd status
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| id |  host  |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| 0  | node01 | 6049M | 11.0G |    0   |     0   |    0   |     0   | exists,up |
| 1  | node02 | 6007M | 11.1G |    0   |     0   |    0   |     0   | exists,up |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
[root@node02 /]# ceph df
GLOBAL:SIZE       AVAIL      RAW USED     %RAW USED 34 GiB     22 GiB       12 GiB         34.66
POOLS:NAME     ID     USED     %USED     MAX AVAIL     OBJECTS
[root@node02 /]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR total_objects    0
total_used       12 GiB
total_avail      22 GiB
total_space      34 GiB创建pool：
[root@node02 /]# ceph osd pool create pool1 5
pool 'pool1' created
[root@node02 /]# ceph df
GLOBAL:SIZE       AVAIL      RAW USED     %RAW USED 34 GiB     22 GiB       12 GiB         34.65
POOLS:NAME      ID     USED     %USED     MAX AVAIL     OBJECTS pool1     1       0 B         0        20 GiB           0

dashbool查看pool：

Kubernetes存储之Rook的ceph搭建相关推荐

kubernetes存储：local，openEBS，rook ceph
文章目录 Local 存储(PV) 概念 hostPath Local PV storageClassName指定延迟绑定动作 pv的删除流程 OpenEBS存储控制平面 OpenEBS PV Pr ...
Kubernetes 集群基于 Rook 的 Ceph 存储之块设备、文件系统、对象存储
文章目录 1.Rook & Ceph 介绍 2.环境.软件准备 3.Block 块存储 3.1.创建 CephBlockPool.StorageClass 3.2.验证并测试 4.File S ...
k8s——kubernetes使用rook部署ceph集群
kubernetes使用rook部署ceph集群一:环境准备 1.安装时间服务器进行时间同步所有的kubernetes的集群节点 [root@master ~]# yum -y install n ...
K8S集群rook部署ceph集群
前言: 之前自己用rook部署过几次ceph集群,每次部署或多或少都会遇到一些问题.有些网上还能找到解决方法,有的只能靠自己去解决,毕竟每个人部署遇到的问题不一定都相同.因为每次部署完自己也没做记录, ...
云原生存储项目ROOK
本周一,原生云计算基金会(CNCF)技术监管委员会(TOC)投票接受了新的托管项目Rook,这是CNCF的第15个项目也是目前为止唯一一个存储类的项目,其他项目分别是Kubernetes,Promet ...
使用Rook安装Ceph
layout: post title: 使用Rook安装Ceph catalog: true tag: [K8S, Ceph] 1. Rook简介 2. Rook架构 3. Rook部署 3.1. 前 ...
从零开始入门 K8s | Kubernetes 存储架构及插件使用
作者 | 阚俊宝阿里巴巴高级技术专家本文整理自<CNCF x Alibaba 云原生技术公开课>第 21 讲. 关注"阿里巴巴云原生"公众号,回复关键词**&quo ...
KUBERNETES存储之PERSISTENT VOLUMES简介
KUBERNETES存储之PERSISTENT VOLUMES简介简介管理存储和管理计算有着明显的不同.PersistentVolume子系统给用户和管理员提供了一套API,从而抽象出存储是如何提 ...
ceph搭建及使用详解
第1章 ceph介绍 1.1 Ceph的主要特点统一存储无任何单点故障数据多份冗余存储容量可扩展自动容错及故障自愈 1.2 Ceph三大角色组件及其作用在Ceph存储集群中,包含了三大角色 ...

Kubernetes存储之Rook的ceph搭建

Kubernetes存储之Rook的ceph搭建相关推荐

最新文章

热门文章