将设为首页浏览此站
开启辅助访问 天气与日历 收藏本站联系我们切换到窄版

易陆发现论坛

 找回密码
 开始注册
查看: 349|回复: 6
收起左侧

ceph osd down message日志

[复制链接]
发表于 2021-7-20 12:55:06 | 显示全部楼层 |阅读模式

马上注册,结交更多好友,享用更多功能,让你轻松玩转社区。

您需要 登录 才可以下载或查看,没有帐号?开始注册

x
Jul 20 12:47:56 compute03 ceph-osd: 2021-07-20 12:47:56.933 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)+ E4 C$ C& J5 Z$ L6 v$ a/ F% r' J
Jul 20 12:47:57 compute03 ceph-osd: 2021-07-20 12:47:57.950 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)- J0 |: a+ h- w. J* h7 l
Jul 20 12:47:58 compute03 ceph-osd: 2021-07-20 12:47:58.986 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
9 s3 F! f) `' |7 l  g2 l9 s# j2 aJul 20 12:47:59 compute03 ceph-osd: 2021-07-20 12:47:59.978 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
6 e# ], ~6 W* QJul 20 12:48:00 compute03 ceph-osd: 2021-07-20 12:48:00.955 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
5 T% N. Z' F3 c' n9 mJul 20 12:48:01 compute03 ceph-osd: 2021-07-20 12:48:01.925 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)8 J/ y% @: U* K3 |/ h0 k
Jul 20 12:48:02 compute03 ceph-osd: 2021-07-20 12:48:02.949 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)! G% f4 H! k9 E' E
Jul 20 12:48:03 compute03 ceph-osd: 2021-07-20 12:48:03.971 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
8 |; q$ h1 m. W
6 k( _/ h5 _6 }. q; e, k; J% T
7 h7 I' [  [2 G- W7 T/ f通过stop NetworkMange服务,和firewalld服务:
7 B- \5 B" ?$ [2 G/ o[root@compute03 ceph]# systemctl status firewalld.service ! T6 Z, r4 V) I5 i0 q6 O- u5 v, c* k
● firewalld.service - firewalld - dynamic firewall daemon
+ A, }8 f( R% g' I1 M8 K   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)1 l1 H1 @" T* L2 K
   Active: inactive (dead)8 ~+ g' D! e  D% D5 X2 a
     Docs: man:firewalld(1)1 W( v6 t5 c8 |3 ^
[root@compute03 ceph]# systemctl disable firewalld.service
. L! }& y/ p: o5 K$ l) w% N[root@compute03 ceph]# systemctl status firewalld.service
: v) L3 J/ S/ q! X/ L● firewalld.service - firewalld - dynamic firewall daemon
, k0 U" R$ J5 K- M5 {" B% K, ^   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)' S: i) q# I! c
   Active: inactive (dead)
% e! ]/ h# s, X/ `     Docs: man:firewalld(1)% u1 ^  T' `. _; }! M/ s9 m& I
[root@compute03 ceph]#
( b) p  Q6 d6 r
8 M, b0 J. o2 C4 [4 B6 y[root@compute02 ceph]# systemctl disable firewalld.service) S+ o1 f/ X: v$ m( i* z7 E, b9 |- E
[root@compute02 ceph]# systemctl stop firewalld.service9 X9 t! m/ X, r6 z% _4 H
[root@compute02 ceph]# ceph osd tree8 ?9 U& P( j/ H/ B! S
ID CLASS WEIGHT TYPE NAME    STATUS REWEIGHT PRI-AFF
) s# {8 X- E" v, ]/ l4 Q-1            0 root default                        
3 ]2 R4 L7 K6 b. y' r: U1 L 0   hdd      0 osd.0            up  1.00000 1.00000
' w; \1 f4 z1 W4 ~) t4 l  m4 m 1   hdd      0 osd.1            up  1.00000 1.00000
$ U3 c) ~. x+ |* g1 D/ D. U+ \ 2   hdd      0 osd.2            up  1.00000 1.00000
. K, B* j! m0 k4 p, j9 G9 @. ]
 楼主| 发表于 2021-7-20 12:56:55 | 显示全部楼层
Jul 20 12:50:11 compute03 ceph-osd: 2021-07-20 12:50:11.806 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)" M  }6 X* }3 k6 h
Jul 20 12:50:12 compute03 ceph-osd: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804), G( o6 c9 J. N( C# v9 m: V
Jul 20 12:50:12 compute03 ceph-osd: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
5 N5 ?' }% X: D6 ?- c- P2 p( l4 YJul 20 12:50:13 compute03 ceph-osd: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)
. a( q: p) w0 B! GJul 20 12:50:13 compute03 ceph-osd: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
/ F/ L) s( {9 h; ~! ^Jul 20 12:50:14 compute03 ceph-osd: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)
0 A4 Z$ h& ^9 I& r0 eJul 20 12:50:14 compute03 ceph-osd: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
8 u% ]! i& }: u  n% ZJul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0& _4 v5 x, U2 e
Jul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Got signal Interrupt ***/ }9 K' B$ m. e9 X5 A- _
Jul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) ***
 楼主| 发表于 2021-7-20 12:59:55 | 显示全部楼层
ceph-osd@2.service - Ceph object storage daemon osd.24 {  x$ U& V. I9 {/ R
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: disabled)3 y1 r1 u- ^, u  r
   Active: inactive (dead) since Tue 2021-07-20 12:50:15 CST; 6min ago8 O8 E  n# ~6 l. C% b( O, @
  Process: 4680 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS), W2 n8 }2 n4 u! O/ G/ V  P
Main PID: 4680 (code=exited, status=0/SUCCESS)/ N, H2 s" I' K: S+ N% F8 ?# r

. W& b* d$ j- {Jul 20 12:50:11 compute03 ceph-osd[4680]: 2021-07-20 12:50:11.806 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)# e0 d% q- |! X2 J( M  b
Jul 20 12:50:12 compute03 ceph-osd[4680]: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)
) y5 W+ i( E' L& v# RJul 20 12:50:12 compute03 ceph-osd[4680]: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)2 m6 o) u8 r; Z% V: I
Jul 20 12:50:13 compute03 ceph-osd[4680]: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)9 S2 [0 _4 h  M1 E' x
Jul 20 12:50:13 compute03 ceph-osd[4680]: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
3 V3 z5 D, A/ B/ ~, u2 |Jul 20 12:50:14 compute03 ceph-osd[4680]: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)! b! V. n% F6 @0 U# k$ G
Jul 20 12:50:14 compute03 ceph-osd[4680]: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
' t' g2 x" Z8 f* \' f9 ]1 \Jul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthr...) ) UID: 01 C* A/ d8 c: h- o: ^
Jul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Got signal Interrupt ***0 X7 Q% r8 @6 o8 e
Jul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) ***
) N5 @8 G: m' v1 bHint: Some lines were ellipsized, use -l to show in full.
 楼主| 发表于 2021-7-20 13:09:18 | 显示全部楼层
Jul 20 13:03:22 compute03 ceph-osd: 2021-07-20 13:03:22.222 7f308763f700 -1 osd.2 79 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 13:02:43.327130 (oldest deadline 2021-07-20 13:03:03.327130)- }1 I+ \& _, X- V; D
Jul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 04 b, ^7 d+ c0 p; q& \
Jul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 osd.2 80 *** Got signal Interrupt ***
) C. r; H4 b! M- z4 A* h4 ?Jul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 osd.2 80 *** Immediate shutdown (osd_fast_shutdown=true) ***
 楼主| 发表于 2021-7-20 13:43:56 | 显示全部楼层
back, first ping sent 2021-07-20 13:34:07.693512 (oldest deadline 2021-07-20 13:34:27.693512)
4 n7 p5 [3 \8 A3 i$ \Jul 20 13:35:13 compute03 ceph-osd: 2021-07-20 13:35:13.237 7f3703776700 -1 osd.2 87 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 13:34:07.693512 (oldest deadline 2021-07-20 13:34:27.693512)" c+ B' c8 e! E  `1 T  f
^C
# g0 W' O) p) E, \3 G0 ~[root@compute03 ~]# service iptables stop
, k: B" E  B/ A, X+ S2 t2 bRedirecting to /bin/systemctl stop iptables.service% v0 a2 B* C. y9 v2 k  }8 X% v
Failed to stop iptables.service: Unit iptables.service not loaded.
) {+ V  K! v) ^[root@compute03 ~]# getenforce
# A1 y8 M( F/ _5 dPermissive
 楼主| 发表于 2021-7-20 13:44:38 | 显示全部楼层
关闭selinux5 {3 N4 v6 D) H- s  Y
永久性关闭(这样需要重启服务器后生效)
$ x$ ]/ l( N1 S7 U8 p # sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config  ]% z# b: Y( {  q( Q7 l

" L1 I0 P2 |: d/ W# O  C6 y: H, D( S# d2 o  n- Y+ m
sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
  o' D9 c$ J+ m/ ^
 楼主| 发表于 2021-7-20 14:23:29 | 显示全部楼层
最后发现是有张网卡的ip地址配置成一模一样的,造成ip地址冲突。排查网络ip问题,也是问题原因。
您需要登录后才可以回帖 登录 | 开始注册

本版积分规则

关闭

站长推荐上一条 /4 下一条

如有购买积分卡请联系497906712

QQ|返回首页|Archiver|手机版|小黑屋|易陆发现 点击这里给我发消息

GMT+8, 2021-9-27 03:22 , Processed in 0.043765 second(s), 22 queries .

Powered by LR.LINUX.cloud bbs168x X3.2 Licensed

© 2012-2022 Comsenz Inc.

快速回复 返回顶部 返回列表