ceph集群用户管理

• 1.用户格式及权限说明

ceph的用户格式"TYPEID.USERID"
	- TYPEID:
		指定的是用户类型。
		包括内置组件用户(mon,mds,rgw,osd,mgr)和普通用户(client)。
		
	- USERID:
		就是用户名，可以是数字，比如表示ods的第0块磁盘，对应的是"ods.0",
		也可以是字符串，比如管理员用户，对应的是"client.admin"。
		当然，用户可以自定义USERID，比如"client.jason","client.violet"。

每个用户都可以授权，使用caps字段关联。授权的格式"allow 权限"
	
	常用的权限有:
		- r：
			读权限
		- w: 
			写权限
		- x: 
			执行权限，可以调用方法(这些方法可能存在读写等操作)，还可以执行mon的auth等相关命令。
		- *: 
			拥有rwx等权限。
        - class-read:
            拥有x能力的子集，授予用户调用类读取方法的能力。
        - class-write:
            拥有x能力的子集，授予用户调用类写入方法的能力。
		- profile osd:
			授予用户一某个OSD身份连接到其他OSD或监视器的权限，可以获取OSD的状态信息。
		- profile mds:
			授予用户以某个MDS身份连接到其他MDS或监视器的权限，可以获取mds的状态信息。
        - profile bootstrap-osd:
           授予用于引导OSD的权限，在部署时候产生。
	    - profile bootstrap-mds:
           授予用于引导元数据服务器的权限，在部署时候产生。
			
举例说明：(通过上面的介绍，那就可以看懂下面的配置啦)
[root@ceph141 ~]#  cat /etc/ceph/ceph.client.admin.keyring 
[client.admin]
	key = AQAkRepnl8QHDhAAajK/aMH1KaCoVJWt5H2NOQ==
	caps mds = "allow *"
	caps mgr = "allow *"
	caps mon = "allow *"
	caps osd = "allow *"
[root@ceph141 ~]# 



关于更多权限信息请参考官网:
	https://docs.ceph.com/en/latest/rados/operations/user-management/#authorization-capabilities
	https://docs.ceph.com/en/nautilus/rados/operations/user-management/ 

• 2.查看内置的用户

参考链接:
	https://docs.ceph.com/en/nautilus/rados/operations/user-management/#get-a-user
	
		2.1 查看指定用户
[root@ceph141 ~]# ceph auth get client.admin
[client.admin]
	key = AQAkRepnl8QHDhAAajK/aMH1KaCoVJWt5H2NOQ==
	caps mds = "allow *"
	caps mgr = "allow *"
	caps mon = "allow *"
	caps osd = "allow *"
[root@ceph141 ~]# 

               2.2 查看所有用户
[root@ceph141 ~]# ceph auth list  # 和"ceph auth ls"等效
osd.0
	key: AQBGG7pllktDHxAAt1KWf87MZAgaaP67aCeSiA==
	caps: [mgr] allow profile osd
	caps: [mon] allow profile osd
	caps: [osd] allow *
...
osd.8
	key: AQAiZ+pnH1bKHxAAXV/Syqud2m31XeLSyCF7ew==
	caps: [mgr] allow profile osd
	caps: [mon] allow profile osd
	caps: [osd] allow *
client.admin
	key: AQAkRepnl8QHDhAAajK/aMH1KaCoVJWt5H2NOQ==
	caps: [mds] allow *
	caps: [mgr] allow *
	caps: [mon] allow *
	caps: [osd] allow *
...
mgr.ceph141.mbakds
	key: AQAkRepnel2jFBAAG6PFgfB9GN/ret87V+WMhQ==
	caps: [mds] allow *
	caps: [mon] profile mgr
	caps: [osd] allow *
mgr.ceph142.qgifwo
	key: AQB9Wupnxt4tGxAA4W5hHnlxx0UhzDnjwaIbvQ==
	caps: [mds] allow *
	caps: [mon] profile mgr
	caps: [osd] allow *

[root@ceph141 ~]# 
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph -s
  cluster:
    id:     11e66474-0e02-11f0-82d6-4dcae3d59070
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 40h)
    mgr: ceph141.mbakds(active, since 40h), standbys: ceph142.qgifwo
    osd: 9 osds: 9 up (since 40h), 9 in (since 40h)
 
  data:
    pools:   2 pools, 9 pgs
    objects: 298 objects, 640 MiB
    usage:   2.8 GiB used, 5.3 TiB / 5.3 TiB avail
    pgs:     9 active+clean
 
[root@ceph141 ~]# 

• 3.三种方式自定义普通用户

参考链接:
	https://docs.ceph.com/en/nautilus/rados/operations/user-management/#add-a-user
	
		3.1 "ceph auth add" 创建用户
[root@ceph141 ~]# ceph auth add client.linux mon 'allow r' osd 'allow rwx pool=violet'
added key for client.linux
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.linux
[client.linux96]
	key = AQCgo+xn7ZV/NRAAq+64uGDJ2SFsarfFwyVFhw==
	caps mon = "allow r"
	caps osd = "allow rwx pool=violet"
[root@ceph141 ~]# 
 
                3.2 "ceph auth get-or-create"创建用户
[root@ceph141 ~]# ceph auth get client.lax  # 查看用户不存在
Error ENOENT: failed to find client.lax in keyring
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get-or-create client.lax mon 'allow r' osd 'allow rwx'  # 如果用户不存在则直接创建并返回认证信息
	key = AQDdo+xn9Z45NRAAPWt+OW/ad2Sn3B9bM+hJIQ==
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.lax 
[client.lax]
	key = AQDdo+xn9Z45NRAAPWt+OW/ad2Sn3B9bM+hJIQ==
	caps mon = "allow r"
	caps osd = "allow rwx"
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get-or-create client.lax mon 'allow rwx' osd 'allow r' 
Error EINVAL: key for client.lax exists but cap mon does not match
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get-or-create client.lax mon 'allow r' osd 'allow rwx'   # 如果用户已存在，直接获取用户不会创建，早期版本会报错，但19.2.1不会报错
[client.lax]
	key = AQDdo+xn9Z45NRAAPWt+OW/ad2Sn3B9bM+hJIQ==
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.lax  # 很明显，上一条命令没有执行成功
[client.yinzhengjie]
	key = AQDdo+xn9Z45NRAAPWt+OW/ad2Sn3B9bM+hJIQ==
	caps mon = "allow r"
	caps osd = "allow rwx"
[root@ceph141 ~]# 


		3.3 "ceph auth get-or-create-key"创建用户
[root@ceph141 ~]# ceph auth get client.k8s  # 注意，用户是不存在的
Error ENOENT: failed to find client.k8s in keyring
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get-or-create-key client.k8s mon 'allow r' osd 'allow rwx'  # 创建用户并返回KEY
AQBkQrxlR6aVGBAAerMOjQ5Nah/HYafJu+aTsg==
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.k8s  # 再次查看用户信息
[client.k8s]
	key = AQBkQrxlR6aVGBAAerMOjQ5Nah/HYafJu+aTsg==
	caps mon = "allow r"
	caps osd = "allow rwx"
exported keyring for client.k8s
[root@ceph141 ~]# 

• 4 “ceph auth print-key”打印已经存在用户的KEY

[root@ceph141 ~]# ceph auth get client.violet007
Error ENOENT: failed to find client.violet007 in keyring
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.lax
[client.lax]
	key = AQDdo+xn9Z45NRAAPWt+OW/ad2Sn3B9bM+hJIQ==
	caps mon = "allow r"
	caps osd = "allow rwx"
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth print-key client.lax;echo  # 如果用户存在则打印该用户对应的KEY信息。
AQDdo+xn9Z45NRAAPWt+OW/ad2Sn3B9bM+hJIQ==
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth print-key client.violet007  # 如果用户不存在则报错
Error ENOENT: don't have client.violet007 
[root@ceph141 ~]# 

• 5.修改用户权限，直接覆盖权限

参考链接:
	https://docs.ceph.com/en/nautilus/rados/operations/user-management/#modify-user-capabilities
	

[root@ceph141 ~]# ceph auth get client.lax
[client.lax]
	key = AQDdo+xn9Z45NRAAPWt+OW/ad2Sn3B9bM+hJIQ==
	caps mon = "allow r"
	caps osd = "allow rwx"
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth caps client.lax  mon 'allow rx' osd 'allow r pool=lax'
[client.lax]
	key = AQDdo+xn9Z45NRAAPWt+OW/ad2Sn3B9bM+hJIQ==
	caps mon = "allow rx"
	caps osd = "allow r pool=lax"
updated caps for client.lax
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.lax
[client.lax]
	key = AQDdo+xn9Z45NRAAPWt+OW/ad2Sn3B9bM+hJIQ==
	caps mon = "allow rx"
	caps osd = "allow r pool=lax"
[root@ceph141 ~]# 

• 6.删除用户

参考链接:
	https://docs.ceph.com/en/nautilus/rados/operations/user-management/#delete-a-user
	
[root@ceph141 ~]# ceph auth get client.lax
[client.yinzhengjie]
	key = AQDdo+xn9Z45NRAAPWt+OW/ad2Sn3B9bM+hJIQ==
	caps mon = "allow rx"
	caps osd = "allow r pool=lax"
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth del client.lax  # 删除名为"lax"的普通用户(client)。
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.lax
Error ENOENT: failed to find client.lax in keyring
[root@ceph141 ~]# 

ceph用户的备份和恢复

参考链接:
https://docs.ceph.com/en/nautilus/rados/operations/user-management/#get-a-user
https://docs.ceph.com/en/nautilus/rados/operations/user-management/#import-a-user-s

测试数据创建

• 1.创建测试用户

[root@ceph141 ~]# ceph auth add client.violet007 mon 'allow rwx' osd 'allow r pool=lax-rbd'
added key for client.violet007
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.violet007
[client.jasonyin2020]
	key = AQBpq+xneZxCLhAAoPsxA/063t2Iy/qcw2zdcw==
	caps mon = "allow rwx"
	caps osd = "allow r pool=lax-rbd"
[root@ceph141 ~]# 

• 2.三种导出用户到文件，用于模拟备份

                 2.1 先创建一个600的权限文件，然后再导入内容
[root@ceph141 ~]# ceph-authtool --create-keyring ceph.client.violet007.keyring  # 说白了，只是创建了一个普通文件。
creating ceph.client.violet007.keyring
[root@ceph141 ~]# 
[root@ceph141 ~]# ll ceph.client.violet007.keyring 
-rw------- 1 root root 0 Feb  2 09:28 ceph.client.violet007.keyring
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.violet007 -o ceph.client.violet007.keyring  # 将内容导出到指定文件
exported keyring for client.violet007
[root@ceph141 ~]# 
[root@ceph141 ~]# cat ceph.client.violet007.keyring
[client.violet007]
	key = AQBpq+xneZxCLhAAoPsxA/063t2Iy/qcw2zdcw==
	caps mon = "allow rwx"
	caps osd = "allow r pool=lax-rbd"
[root@ceph141 ~]# 

		2.2 直接导出到文件，但是文件的权限是644
[root@ceph141 ~]# ceph auth export client.violet007 -o violet007.keyring  # 也可以使用这种方式导入用户信息到文件
export auth(key=AQDtRLxl0V3wFRAA8Cz4Vaeey+k049B761iRZA==)
[root@ceph141 ~]# 
[root@ceph141 ~]# ll violet007.keyring 
-rw-r--r-- 1 root root 137 Apr  2 11:15 violet007.keyring
[root@ceph141 ~]# 
[root@ceph141 ~]# cat violet007.keyring
[client.violet007]
	key = AQBpq+xneZxCLhAAoPsxA/063t2Iy/qcw2zdcw==
	caps mon = "allow rwx"
	caps osd = "allow r pool=lax-rbd"
[root@ceph141 ~]# 

		2.3 直接重定向到文件，权限默认为644
[root@ceph141 ~]# ceph auth get client.violet007 > myuser.keyring
[root@ceph141 ~]# 
[root@ceph141 ~]# ll myuser.keyring
-rw-r--r-- 1 root root 137 Apr  2 11:16 myuser.keyring
[root@ceph141 ~]# 
[root@ceph141 ~]# cat myuser.keyring
[client.violet007]
	key = AQBpq+xneZxCLhAAoPsxA/063t2Iy/qcw2zdcw==
	caps mon = "allow rwx"
	caps osd = "allow r pool=lax-rbd"
[root@ceph141 ~]# 

• 3.删除用户

[root@ceph141 ~]# ceph auth get client.violet007
[client.violet007]
	key = AQDtRLxl0V3wFRAA8Cz4Vaeey+k049B761iRZA==
	caps mon = "allow rwx"
	caps osd = "allow r pool=lax-rbd"
exported keyring for client.violet007
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth del client.violet007
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.violet007 
Error ENOENT: failed to find client.violet007 in keyring
[root@ceph141 ~]# 

• 4.导入用户，用于模拟恢复

[root@ceph141 ~]# cat ceph.client.violet007.keyring 
[client.violet007]
	key = AQDtRLxl0V3wFRAA8Cz4Vaeey+k049B761iRZA==
	caps mon = "allow rwx"
	caps osd = "allow r pool=lax-rbd"
[root@ceph141 ~]# 
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.violet007 
Error ENOENT: failed to find client.violet007 in keyring
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth import -i ceph.client.violet007.keyring 
imported keyring
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.violet007 
[client.violet007]
	key = AQDtRLxl0V3wFRAA8Cz4Vaeey+k049B761iRZA==
	caps mon = "allow rwx"
	caps osd = "allow r pool=lax-rbd"
exported keyring for client.violet007
[root@ceph141 ~]# 

导出授权文件并验证用户权限

• 1.创建用户

[root@ceph141 ~]# ceph auth get-or-create client.k3s mon 'allow r'  osd 'allow * pool=violet-rbd'
[client.k3s]
	key = AQDvrOxn2rd5EBAATTk4WdDFbFw3ecT/RRfiTQ==
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph auth get client.k3s
[client.k3s]
	key = AQDvrOxn2rd5EBAATTk4WdDFbFw3ecT/RRfiTQ==
	caps mon = "allow r"
	caps osd = "allow * pool=violet-rbd"
[root@ceph141 ~]# 

• 2.导出用户授权文件，钥匙环(keyring)

[root@ceph141 ~]# ceph auth export client.k3s -o ceph.client.k3s.keyring
[root@ceph141 ~]# 
[root@ceph141 ~]# cat ceph.client.k3s.keyring
[client.k3s]
	key = AQDvrOxn2rd5EBAATTk4WdDFbFw3ecT/RRfiTQ==
	caps mon = "allow r"
	caps osd = "allow * pool=violet-rbd"
[root@ceph141 ~]# 

• 3.拷贝授权文件前，观察客户端是否有查看集群的权限

[root@prometheus-server31 ~]# rm -f /etc/ceph/ceph.c*
[root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]# ll /etc/ceph/
total 20
drwxr-xr-x   2 root root  4096 Apr  2 11:21 ./
drwxr-xr-x 132 root root 12288 Apr  2 06:39 ../
-rw-r--r--   1 root root    92 Dec 18 22:48 rbdmap
[root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]#  ceph -s
Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
[root@prometheus-server31 ~]# 

• 4.将授权文件拷贝到客户端

[root@ceph141 ~]# scp ceph.client.k3s.keyring /etc/ceph/ceph.conf  10.0.0.31:/etc/ceph

• 5.验证权限

[root@prometheus-server31 ~]# ceph -s --user k3s
  cluster:
    id:     11e66474-0e02-11f0-82d6-4dcae3d59070
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 41h)
    mgr: ceph141.mbakds(active, since 41h), standbys: ceph142.qgifwo
    osd: 9 osds: 9 up (since 41h), 9 in (since 41h)
 
  data:
    pools:   2 pools, 9 pgs
    objects: 298 objects, 643 MiB
    usage:   2.8 GiB used, 5.3 TiB / 5.3 TiB avail
    pgs:     9 active+clean
 
[root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]# rbd -p violet-rbd ls  -l --user k3s
NAME               SIZE     PARENT  FMT  PROT  LOCK
child-xixi-001      20 GiB            2            
harbor               1 TiB            2        excl
mysql80            500 GiB            2        excl
node-exporter       20 GiB            2            
prometheus         500 MiB            2        excl
prometheus-server   40 GiB            2        excl
ubuntu-2204          2 MiB            2            
rbd: --user is deprecated, use --id
[root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]# rbd -p violet-rbd ls  -l --id k3s
NAME               SIZE     PARENT  FMT  PROT  LOCK
child-xixi-001      20 GiB            2            
harbor               1 TiB            2        excl
mysql80            500 GiB            2        excl
node-exporter       20 GiB            2            
prometheus         500 MiB            2        excl
prometheus-server   40 GiB            2        excl
ubuntu-2204          2 MiB            2            
[root@prometheus-server31 ~]# 

• 6.服务端创建rbd块设备

[root@ceph141 ~]# ceph osd pool ls
.mgr
violet-rbd
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd pool create linux 16 16 --autoscale_mode off --size 3
pool 'linux' created
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd pool application enable linux rbd
enabled application 'rbd' on pool 'linux'
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd create -s 2G linux/xixi
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd create -s 4G linux/haha
[root@ceph141 ~]# 
[root@ceph141 ~]# rbd ls -l linux
NAME  SIZE   PARENT  FMT  PROT  LOCK
haha  4 GiB            2            
xixi  2 GiB            2            
[root@ceph141 ~]# 

• 7.客户端验证

[root@prometheus-server31 ~]# cat /etc/ceph/ceph.client.k3s.keyring 
[client.k3s]
	key = AQDvrOxn2rd5EBAATTk4WdDFbFw3ecT/RRfiTQ==
	caps mon = "allow r"
	caps osd = "allow * pool=violet-rbd"
[root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]# rbd -p linux ls  -l --id k3s
2025-04-02T11:28:22.100+0800 7fe53571f4c0 -1 librbd::api::Image: list_images: error listing v1 images: (1) Operation not permitted

rbd: listing images failed: (1) Operation not permitted
[root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]# rbd -p violet-rbd ls  -l --id k3s
NAME               SIZE     PARENT  FMT  PROT  LOCK
child-xixi-001      20 GiB            2            
harbor               1 TiB            2        excl
mysql80            500 GiB            2        excl
node-exporter       20 GiB            2            
prometheus         500 MiB            2        excl
prometheus-server   40 GiB            2        excl
ubuntu-2204          2 MiB            2            
[root@prometheus-server31 ~]# 

授权文件加载顺序总结：

        1 如果使用"--user k3s"指定用户，则默认去找以下文件，找不到就报错:
- /etc/ceph/ceph.client.k3s.keyring
- /etc/ceph/ceph.keyring
- /etc/ceph/keyring
- /etc/ceph/keyring.bin


	2 如果不使用"--user"选项，咱们可以立即为默认为"--user amdin"
- /etc/ceph/ceph.client.admin.keyring
- /etc/ceph/ceph.keyring
- /etc/ceph/keyring
- /etc/ceph/keyring.bin
		

	3 对于认证文件不能随便起名字。
而是需要遵循上述2条的规范文件命名，否则ceph不识别用户的配置文件
	

	4 客户端在连接ceph集群时，仅需要读取keyring文件中的KEY值。
其他caps字段会被忽视。也就是说，对于文件中只要保留key值依旧是有效的。