一、 统计PG
1.1 统计pg分布(shell)
$ ceph pg dump
dumped all
version 45845
stamp 2018-03-08 15:33:38.325088
last_osdmap_epoch 0
last_pg_scan 0
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP
1.13 21 0 0 0 0 84561920 131 131 active+clean 2018-03-07 11:39:45.529003 40'131 40:56 [1,2,0] 1 [1,2,0] 1 32'115 2018-03-07 11:39:45.528965 0'0 2018-03-05 10:50:19.509987
1.12 22 0 0 0 0 83931136 1365 1365 active+clean 2018-03-08 12:02:04.227542 40'1365 40:110779 [0,1,2] 0 [0,1,2] 0 40'1322 2018-03-08 12:02:04.227502 39'1017 2018-03-07 11:39:20.769705
1.11 13 0 0 0 0 54525952 88 88 active+clean 2018-03-07 11:39:20.526098 40'88 40:46 [1,2,0] 1 [1,2,0] 1 32'78 2018-03-07 11:39:20.526064 0'0 2018-03-05 10:50:19.509987
1.10 19 0 0 0 0 79691776 100 100 active+clean 2018-03-07 11:39:30.633761 40'100 40:90 [0,1,2] 0 [0,1,2] 0 32'84 2018-03-07 11:39:30.633719 0'0 2018-03-05 10:50:19.509987
1.f 41 0 0 0 0 158822400 215 215 active+clean+remapped 2018-03-07 11:39:38.528202 40'215 40:84 [1,2] 1 [1,2,0] 1 32'184 2018-03-07 11:39:38.528171 0'0 2018-03-05 10:50:19.509987
1.e 43 0 0 0 0 174785536 227 227 active+clean+remapped 2018-03-07 11:39:55.530052 40'227 40:72 [1,2] 1 [1,2,0] 1 32'192 2018-03-07 11:39:55.529999 0'0 2018-03-05 10:50:19.509987
1.d 34 0 0 0 0 132616192 796 796 active+clean+remapped 2018-03-07 11:39:22.183113 40'796 40:881 [2,1] 2 [2,1,0] 2 35'771 2018-03-07 11:39:22.183073 0'0 2018-03-05 10:50:19.509987
1.c 45 0 0 0 0 169186304 211 211 active+clean 2018-03-08 14:38:05.445574 40'211 40:90 [1,2,0] 1 [1,2,0] 1 40'211 2018-03-08 14:38:05.445532 0'0 2018-03-05 10:50:19.509987
1.b 46 0 0 0 0 173675536 260 260 active+clean+remapped 2018-03-07 11:39:40.529252 40'260 40:86 [1,2] 1 [1,2,0] 1 32'233 2018-03-07 11:39:40.529212 0'0 2018-03-05 10:50:19.509987
1.a 37 0 0 0 0 129726483 503 503 active+clean 2018-03-07 11:39:27.183893 40'503 40:598 [2,0,1] 2 [2,0,1] 2 32'183 2018-03-07 11:39:27.183851 0'0 2018-03-05 10:50:19.509987
1.0 35 0 0 0 0 126824448 1511 1511 active+clean 2018-03-08 14:59:22.562887 40'2011 40:758 [1,2,0] 1 [1,2,0] 1 40'1996 2018-03-08 14:59:22.562845 0'0 2018-03-05 10:50:19.509987
1.1 20 0 0 0 0 82513920 88 88 active+clean 2018-03-08 14:25:50.254050 40'88 40:162 [2,1,0] 2 [2,1,0] 2 40'88 2018-03-08 14:25:50.254013 0'0 2018-03-05 10:50:19.509987
1.2 21 0 0 0 0 84226048 108 108 active+clean 2018-03-07 11:39:54.530063 40'108 40:56 [1,0,2] 1 [1,0,2] 1 32'92 2018-03-07 11:39:54.529983 0'0 2018-03-05 10:50:19.509987
1.3 24 0 0 0 0 93069312 138 138 active+clean 2018-03-08 14:50:28.899028 40'138 40:67 [1,0,2] 1 [1,0,2] 1 40'138 2018-03-08 14:50:28.898988 40'138 2018-03-08 14:50:28.898988
1.4 39 0 0 0 0 161689600 214 214 active+clean+remapped 2018-03-08 11:45:44.496188 40'214 40:76 [1,2] 1 [1,2,0] 1 40'214 2018-03-08 11:45:44.496148 0'0 2018-03-05 10:50:19.509987
1.5 38 0 0 0 0 148865024 1422 1422 active+clean 2018-03-08 14:36:27.308800 40'1422 40:1513 [2,1,0] 2 [2,1,0] 2 40'1410 2018-03-08 14:36:27.308765 0'0 2018-03-05 10:50:19.509987
1.6 28 0 0 0 0 98939904 289 289 active+clean+remapped 2018-03-08 13:25:11.947475 40'289 40:409 [2,1] 2 [2,1,0] 2 40'235 2018-03-08 13:25:11.947434 0'0 2018-03-05 10:50:19.509987
1.7 40 0 0 0 0 153969664 228 228 active+clean+remapped 2018-03-07 11:39:36.528359 40'228 40:76 [1,2] 1 [1,2,0] 1 32'201 2018-03-07 11:39:36.528318 0'0 2018-03-05 10:50:19.509987
1.8 45 0 0 0 0 178663424 221 221 active+clean+remapped 2018-03-07 11:40:01.531165 40'221 40:89 [1,2] 1 [1,2,0] 1 32'187 2018-03-07 11:40:01.531113 0'0 2018-03-05 10:50:19.509987
1.9 32 0 0 0 0 122352640 540 540 active+clean+remapped 2018-03-07 11:39:22.717506 40'540 40:361 [1,2] 1 [1,2,0] 1 32'226 2018-03-07 11:39:22.717456 32'226 2018-03-07 11:39:22.717456
1 643 0 0 0 0 2492637219 8655 8655
sum 643 0 0 0 0 2492637219 8655 8655
OSD_STAT USED AVAIL TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM
2 3.39G 6.68G 10.1G [0,1] 20 5
1 3.39G 6.68G 10.1G [0,2] 20 13
0 3.39G 6.68G 10.1G [1,2] 20 2
sum 10.2G 20.0G 30.2G
$ ceph pg dump pgs|grep ^1|awk 'BEGIN {print "PG \t\t OBJECTS \t\t BYTES \t\tosd"} {print $1 "\t\t" $2 "\t\t" $7 "\t\t" $15}'
PG OBJECTS BYTES osd
1.13 21 84561920 [1,2,0]
1.12 22 83931136 [0,1,2]
1.11 13 54525952 [1,2,0]
1.10 19 79691776 [0,1,2]
1.f 41 158822400 [1,2]
1.e 43 174785536 [1,2]
1.d 34 132616192 [2,1]
1.c 45 169186304 [1,2,0]
1.b 46 173675536 [1,2]
1.a 37 129726483 [2,0,1]
1.0 35 126824448 [1,2,0]
1.1 20 82513920 [2,1,0]
1.2 21 84226048 [1,0,2]
1.3 24 93069312 [1,0,2]
1.4 39 161689600 [1,2]
1.5 38 148865024 [2,1,0]
1.6 28 98939904 [2,1]
1.7 40 153969664 [1,2]
1.8 45 178663424 [1,2]
1.9 32 122352640 [1,2]
参数详解：
- 第一列：PG名称
- 第二列：这个PG当中的OBJECTS对象的数目
- 第三列：这个PG当中的BYTES对象的占用空间
- 第四列：PG所在的OSD（第一个是主，后面两个是副本)
1.2 统计pg分布(火焰图)
1.2.1 脚本内容
cat stackcollapse-crush.py
#! /bin/python
# -*- coding: UTF-8 -*-
import os
import commands
import json
def main():
global list_all_host
list_all_host = commands.getoutput('ceph osd tree -f json-pretty 2>/dev/null')
getpgmap()
def getosd(osd):
mylist=[]
crushid={}
json_str = json.loads(list_all_host)
for item in json_str['nodes']:
if item.has_key('children'):
crushid[str(item['id'])]=str(item['name'])
for child in item['children']:
tmplist=[item['id'],child]
mylist.append(tmplist)
if item['type'] == "osd":
crushid[str(item['id'])]=str(item['name'])
listnum=len(mylist)
compareindex=0
###从数组开始跟后面的数组进行比较，如果有就改变后面的数组，然后删除当前比较的list(index),进行list更新
###如果没有改变，就把索引往后推即可
while compareindex < len(mylist):
change = False
for index,num in enumerate(mylist):
if compareindex != index and compareindex < index:
if str(mylist[compareindex][-1]) == str(num[0]):
del mylist[index][0]
mylist[index]=mylist[compareindex]+mylist[index]
change=True
if str(mylist[compareindex][0]) == str(num[-1]):
del mylist[index][-1]
mylist[index]=mylist[index]+mylist[compareindex]
change=True
if change == True:
del mylist[compareindex]
if change == False:
compareindex = compareindex + 1
for index,crushlist in enumerate(mylist):
osdcrushlist=[]
for osdlocaltion in crushlist:
local=str(crushid['%s' %osdlocaltion])
osdcrushlist.append(local)
if osdcrushlist[-1] == osd:
return osdcrushlist
def getpgmap():
list_all_host = commands.getoutput('ceph pg ls --format json-pretty 2>/dev/null')
json_str = json.loads(list_all_host)
for item in json_str:
for osdid in item['up']:
osd="osd."+str(osdid)
b=""
for a in getosd(osd):
b=b+str(a)+";"
print b+item['pgid']+" "+str(item['stat_sum']['num_objects']+1)
if __name__ == '__main__':
main()
1.2.2 获取数据
python stackcollapse-crush > /tmp/mydata
cat /tmp/mydata
default;ceph-ystest2-osd000;osd.1;1.0 36
default;ceph-ystest2-osd004;osd.2;1.0 36
default;ceph-ystest2-osd000;osd.0;1.0 36
default;ceph-ystest2-osd004;osd.2;1.1 21
内容详解：
- 第一列：机架信息
- 第二列：主机名称
- 第三列：OSD信息
- 第四列：对象数(默认每个数值都进行了加一操作)
1.2.3 解析数据
#获取解析脚本
wget -O flamegraph https://raw.githubusercontent.com/brendangregg/FlameGraph/master/flamegraph.pl
#解析数据
flamegraph --title "Ceph crush flame graph" --width "1800" --countname "num" /tmp/mydata > /tmp/mycrush.svg
1.2.4 查看解析效果
- 通过颜色来区分比例占用的区别
- 支持搜索
- tree方式，可以清楚看到分布
- 可以查看pg对象数目
- 可以查看osd上面有哪些pg，主机上有哪些osd
