Ceph Ansible部署ceph集群

收录于 存储
  约 1700 字 预计阅读 4 分钟  - 次阅读  

记录一下ceph集群的安装过程，因为服务器是centos7 最高就能安装N版 所以ceph版本为N版。

基础环境

ip 主机名 系统版本 组件 数据盘
172.16.88.26 K8s-88-26 centos7.6 mons,osds,mgrs,mdss,clients,rgws,grafana-server /dev/sdb
172.16.88.27 K8s-88-27 centos7.6 mons,osds,mgrs,mdss,clients,rgws /dev/sdb
172.16.88.28 K8s-88-28 centos7.6 mons,osds,mgrs,mdss,clients,rgws /dev/sdb
172.16.88.29 k8s-88-29 centos7.6 部署机、ansible

准备工作

关闭防火墙

1
2
3

systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld

关闭SELinux

1
2

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config #此操作重启后生效

配置yum源

  • CentOS-Base.repo

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

    # CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#
​
[base]
name=CentOS-$releasever - Base - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
​
#released updates
[updates]
name=CentOS-$releasever - Updates - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
​
#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
​
#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
​
#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
​

  • epel.repo

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

    [epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
​
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
​
[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0

  • 清理缓存，并生成新的缓存

    1

    yum clean all && yum makecache

以上操作需要在所有的机器上操作，以下的操作只需要在部署机器上部署就可以

下载安装包、安装ansible等

  • 下载安装包

    1
2
3
4
5

    wget https://github.com/ceph/ceph-ansible/archive/refs/tags/v4.0.60.tar.gz 
tar -zxvf v4.0.60.tar.gz
#验证系统版本是否满足
cd ceph-ansible-4.0.60/group_vars/
grep "This version is supported on RHEL"  all.yml.sample

  • 安装pip、ansible等

    1
2
3
4
5
6
7
8

    # 安装pip、git
yum install epel-release -y
yum install -y python-pip git
#将pip升级到指定版本
python -m pip install --upgrade pip==20.3.4
#下面命令会安装ansible，如果有报错就手动安装ansible。
cd ceph-ansible-4.0.60/
pip install -r requirements.txt

配置免密登录

1
2
3
4
5

ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub root@172.16.88.26
ssh-copy-id -i ~/.ssh/id_rsa.pub root@172.16.88.26
ssh-copy-id -i ~/.ssh/id_rsa.pub root@172.16.88.26
# 注意一一登录进行验证

部署ceph集群

修改相关文件

  • 修改group_vars文件夹内容

    1
2

    cd ceph-ansible-4.0.62/group_vars
for file in *;do cp $file ${file%.*};done

  • all.yml内容

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

    ---
dummy:
mon_group_name: mons
osd_group_name: osds
rgw_group_name: rgws
mds_group_name: mdss
client_group_name: clients
mgr_group_name: mgrs
grafana_server_group_name: grafana-server
configure_firewall: False
ceph_repository_type: repository # <== repository表示使用新的库，不使用官方仓库
ceph_origin: repository #<== 安装方式，repository值表示指定使用仓库安装，
ceph_repository: community # <== 选择使用库的来源类型,community为免费社区版
ceph_mirror: http://mirrors.aliyun.com/ceph # <== ceph仓库的路径
ceph_stable_key: http://mirrors.aliyun.com/ceph/keys/release.asc # <== ceph key
ceph_stable_release: nautilus # <== 安装的ceph版本 nautilus为14，最新的stable版本 
ceph_stable_repo: "{{ ceph_mirror }}/rpm-{{ ceph_stable_release }}" # <== ceph nautilus仓库地址
public_network: "172.16.0.0/16" # <== 公共网段 
cluster_network: "172.16.0.0/16" # <== 集群网段
monitor_interface: eth0  # <== 网卡名 
#monitor_address: "172.16.0.0/16"

#monitor_interface: eth0
osd_auto_discovery: true
#osd_objectstore: filestore
osd_objectstore: bluestore  # <== ceph存储引擎
radosgw_interface: eth0
#radosgw_interface: eth0
dashboard_enabled: True  #是否开启dashboard
dashboard_admin_user: admin
dashboard_admin_password: admin000
dashboard_protocol: http
dashboard_port: 9443
pg_autoscale_mode: True
grafana_admin_user: admin
grafana_admin_password: admin000

# .........
### 文件修改的内容都能在ceph-ansible-4.0.62/roles/ceph-defaults/defaults/main.yml下找到对应的参数

  • osd.yml内容

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

    ........
# Declare devices to be used as OSDs
# All scenario(except 3rd) inherit from the following device declaration
# Note: This scenario uses the ceph-volume lvm batch method to provision OSDs

devices:
  - /dev/sdb   #将这行及上一行的注释去掉就可以了
#  - /dev/sdc
#  - /dev/sdd
#  - /dev/sde
........

  • site.yml修改

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

    cd ceph-ansible-4.0.62
cp site.yml.sample  site.yml

vim site.yml
#注释一些组件
---
# Defines deployment design and assigns role to server groups

- hosts:
  - mons
  - osds
  - mdss
  - rgws
#  - nfss
#  - rbdmirrors
  - clients
  - mgrs
#  - iscsigws
#  - iscsi-gws # for backward compatibility only!
  - grafana-server
#  - rgwloadbalancers

  • 创建hosts文件，定义相关主机角色

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

    cd ceph-ansible-4.0.62
cat > hosts  << EOF
[mons]
172.16.88.26
172.16.88.27
172.16.88.28

[osds]
172.16.88.26
172.16.88.27
172.16.88.28


[mgrs]
172.16.88.26
172.16.88.27
172.16.88.28
[mdss]
172.16.88.26
172.16.88.27
172.16.88.28

[clients]
172.16.88.26
172.16.88.27
172.16.88.28

[rgws]
172.16.88.26
172.16.88.27
172.16.88.28

[grafana-server]
172.16.88.26
EOF

# 测试连接是否正常
ansible all -m ping -i hosts

部署ceph

  • 部署

    1
2
3
4
5
6
7

    cd ceph-ansible-4.0.62
#安装
ansible-playbook -vv -i hosts site.yml

#清理集群  如果有什么问题 可以用下面命令清理集群
cp infrastructure-playbooks/purge-cluster.yml purge-cluster.yml # 必须copy到项目根目录下
ansible-playbook -vv -i hosts purge-cluster.yml

  • 安装完成后，执行ceph -s可能会有一些警告，可以通过以下命令清理

    1
2
3
4
5
6
7
8

    # 1 pools have too few placement groups处理
#使用ceph health detail查询一下具体是哪个pool 然后对应执行以下命令
ceph osd pool set cephfs_data pg_autoscale_mode on  #其中cephfs_data就是具体的pool


# mons are allowing insecure global_id reclaim警告处理
ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false
ceph config set mon auth_allow_insecure_global_id_reclaim false

参考链接

官方文档

https://codeantenna.com/a/nTMBz7T23d

https://blog.csdn.net/Hearz/article/details/120739376

Buy me a coffee
支付宝
微信
更新于 2023-01-27 
CC BY-NC 4.0
阅读原始文档
cephansiblelinux存储
返回 | 主页
0%