我有一台在 Debian Wheezy 上运行的服务器3.2.68-1+deb7u1 x86_64
。最近/var/log/messages
出现了奇怪的记录。该日志的第一条消息在服务器重新启动后立即开始。
6.222.18 LEN=44 TOS=0x00 PREC=0x00 TTL=58 ID=61546 PROTO=TCP SPT=80 DPT=42918 WINDOW=16384 RES=0x00 ACK SYN URGP=0
Jun 6 08:02:49 s02 kernel: [ 29.615405] iptables-input-drop: IN=eth0 OUT= MAC=d4:3d:7e:ed:0a:9a:3c:94:d5:4a:e5:03:08:00 SRC=104.31.176.10 DST=144.
76.222.18 LEN=44 TOS=0x00 PREC=0x00 TTL=58 ID=0 DF PROTO=TCP SPT=80 DPT=24586 WINDOW=29200 RES=0x00 ACK SYN URGP=0
Jun 6 08:03:02 s02 rsyslogd-2177: imuxsock begins to drop messages from pid 6754 due to rate-limiting
Jun 6 08:03:06 s02 kernel: [ 46.752749] iptables-input-drop: IN=lxc OUT= MAC=01:00:5e:00:00:01:fe:12:86:6b:0f:b8:08:00 SRC=0.0.0.0 DST=224.0.0.1 L
EN=32 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2
Jun 6 08:03:11 s02 kernel: 472]pbsntr:Nt TM=::::::::::::: C01.58D=4621L= Sx E00T5I0FROCS= T46IO20R=0A NR=
Jun 6 08:03:43 s02 kernel: 4 89 cnolf=o,doe
Jun 6 08:03:43 s02 kernel: 8apamneo<82:, 2a3-#a6b>.]T<84ff6da7d 6[f4?_3
Jun 6 08:03:43 s02 kernel: 1<f8 ngx9 6[f1?slqex 7 fb kox7 7 f3 lh/430f8feuor/433f88eul0x>.r+f 6[f6?__a3d 7 fb ree7f 6[f6?ghoxc 6[f1?ea1b 7 fc ok03 7
f5 al0
Jun 6 08:03:43 s02 kernel: 9<f6 a/436f85al/639 c.dtdrotxhoe 7m 1ki5f 634r: i04kl<82-<87e u 7C0 t1 [9 b >.] : :438 i, d<84 h0:s
Jun 6 08:03:43 s02 kernel: 6U hu0 7P: c 7C7 t1 [9 p: 7P:1c 6 7C1 t1 [9 b3 >.] : :437 i, d<88 h6:s
Jun 6 08:03:43 s02 kernel: 1U 8hu9 8P:1c 7NNp: 8P:1c 2 7C1 t1 [9 b3 >.] : :433 i, d<86 h6:s
Jun 6 08:03:43 s02 kernel: 6U 8hu5 8P:1c 5 7aa7cn3tn[9 e3n_5s_4 8ua4t6e5a<81e4bi1leb7 8m3h3t5u<80e 5mB0haaBinci v:elid:ofke8okyrkakmlleleb _0e:s0n a
:gnaeb
Jun 6 08:04:01 s02 rsyslogd-2177: imuxsock lost 1237 messages from pid 6754 due to rate-limiting
Jun 6 08:04:02 s02 rsyslogd-2177: imuxsock begins to drop messages from pid 6754 due to rate-limiting
Jun 6 08:04:11 s02 rsyslogd-2177: imuxsock lost 1322 messages from pid 6754 due to rate-limiting
Jun 6 08:05:02 s02 rsyslogd-2177: imuxsock begins to drop messages from pid 6754 due to rate-limiting
Jun 6 08:05:11 s02 rsyslogd-2177: imuxsock lost 1446 messages from pid 6754 due to rate-limiting
Jun 6 08:08:11 s02 kernel: 2ws:2 4> 1l_e0 <89eme3i4:B9 _5 v: _4 v:2vekao e) t6l4 1ke4a4 7scl0blekekBa2ueokempc0nmn 8lr[ >.]00*1361*0 k44=k 8ND18075
9 Bk22864B 2oo2*1361:ea:95:30C.27=72L 0R0==PTTP2W 0SP:0 0N=ETDD==PN< .1anr=U=7a:43 46T684xCTI T==W2E N06 et OSFA:f3:8=20LOP0 5TP LhA:0:10C.42=0CT=FIE
=6 = v9e6:76 0S06x=L3 D9=4C2=.L xE062DT 3PL DU=DL
Jun 6 08:10:40 s02 kernel: :0939da5:0121T784x=L0TPTW9=KGv e6f2098R0D. 30R0== OP3D 340]tst:c =Of:83:0.S6 T 049TTO=Qe35:051D..==R015OP93 -=UYe7=8f:27
.D..==R049PC=08
Jun 6 08:11:40 s02 kernel: 0M3d:d5:12S6 T 08FT85D00Y0cT IexM:::::::C.T.NSP069 U=DL
Jun 6 08:14:11 s02 kernel: hAe:bE066 D8=4<4>[ 620.699091] iptables-input-drop: IN=xc OUT HSNezO C:8::0:0.S6 T 048O CDEUNx=6b:900D.=0CT= D5=4>60ilnd
xTYvxM:::::::C.S.2ET0C 44FO 8=3EM3e99400=48=61=S 0TIFOS 6I2R0 R
Jun 6 08:22:40 s02 kernel: 01. 1.L3= = 6=8P=P45= et He7f6:b1:S0S7180E 6RM==61>056plppxPeL:b5:7S.=2ESR I OYD5=4 867pe-IUIrC8:ea:1 0N=ETDD==PNrO IhM
e:8:700 0N=ETDD==PNaiudpIl =HNz e:be:8=2.ESR I OTDE07]tsu:cPvO=8fb10C.0LOP0 9TP L7:68:a8C0=. 0=TD T 3 UNrC8:ea:1 0N=ETDD==PNPtLe:be:0R.T684xCTI0OP
8001.016=PxLD OP5D 3pO=e9:5071=2ESR IPP94OES=eTde:9a:S4D78 0x58R 03 71:22:R01 T 043OS8
Jun 6 08:23:40 s02 kernel: 0 7. 42ESR IPP 4WR
Jun 6 08:25:13 s02 kernel: a .D0N=ETDD==PNE044OTDE::a8:R01 T 047OS6
Jun 6 08:28:11 s02 kernel: 0L3= =T43DT 5TN430C23D4.86=P053==PN cUPIerLC268e:0R.T68400=9RP I =6f2098R0D. 30R0== OP4D 3r=UYeOC2b8e:0C.42=0CT=FIE=11N=NrA2:53a0S0D0EO x=5F=T
Jun 6 08:37:40 s02 kernel: pr===Of:83:0.S06x=L5 D2=44]aidNTNx=6b:900D.=0CT=FU4T=66tir=TIzM1b:39:=.12LOP0 2TTO=E8TTO=EerO Ce26bf::31:00C..S462 =T0 C0T44 RIT8ED01:a:e8C. .1400= OT=I20 P hM20:10C.42=0CT=FIE=01TeL:b5:7S.=1 0x60R 33 :27 .D..==R049PCEE1EE=PxLD1RD= 34Uv7e6:05 0T.LOP0 OYD1Q 8:ea:1 0N=ETDD==PNcSzA:fb:8=2.ESR I OTDE::0a:0=.D0.NTxRxT 4DODT9TL3Of8fb:8=24.8x=L1 C=03 bnrNTIrC8:ea:1 .1400=3PM8 5
Jun 6 08:38:14 s02 kernel: 99bnpcHt 1:22:R012N=ETDFT3TI60
Jun 6 08:45:35 s02 kernel: [ 2178.791699] ipabei-:lUH=wM:::6:8=30LOP0 0TP LD6RUT85=t- x=IrC8:ea:1 0N=ETDD==PN<4>[ 2299.711247] iptables-input-drop: IN=lxcO=Y=zL=26:23:70=0T7.NO0=T 2PCEE1Eo=OPNh7C160:e25:C.S0L xE062DT 9PL 71R0SPsd===Hf:8c:0.D.=0CT=FU4T=SL:b5:7S.=2ESR I OYD1Q4d3:3S1742=0CT=OSP 2SCRz=:fb:8=2.ESR I OTDERL5 D4=4===of:8e:0.S06x=L4 D7=46:::5 0T.300=7PP55327ap: YhM20:10C.0LOP0 1TP LN7f6:b1:R.=.60E 7RP9TN456rl v7e6:05 0T.300=7PP353OHNtx =16:b2:29:0R00D1. =T0 C0L 7 O 5 5N
我有一个丢弃数据包的规则iptables
:
-A INPUT -m limit --limit 2/min -j LOG --log-prefix "iptables-input-drop: " --log-level 4
这个规则导致前几行,之后日志就变得非常混乱。是否是由某些格式错误的数据包引起的?是某种攻击还是硬件问题(内存损坏等)?
更新:
我关闭了 rsyslog 限制$SystemLogRateLimitInterval 0
以避免丢失消息。但可读的消息仍然不多:
Jun 8 04:32:15 s02 kernel: 05[f7?h01[.]f1>c+x[.]f1>seu/404ff0_ice0
Jun 8 04:32:15 s02 kernel: 05[f7?ic4166 f3 lh/401ff1mo_o4<14<ff]___1816 f0]_ue0>13f87rnksx<15<fa ctr2266 ff u_+x[.]f0>aek+xt[.]f1>mp_
+x[.]f1>dged0>10f8f_c+x[.]f1>dgex666 fb p1
Jun 8 04:32:15 s02 kernel: 07[f5?_g2
Jun 8 04:32:15 s02 kernel: 07[ff?obr1
Jun 8 04:32:15 s02 kernel: 07[f5?i_ee/<12<fd ahax1[.]f1>_+x[.]f1>ni_x5[.]f0>t_0xx>13f84orb
Jun 8 04:32:15 s02 kernel: 07[fb?ef<13<f6 a/407ff7scs0116 nr..l li cmn[.]ye2i4,n5[.]y 0m77Bc408-<15dAp[.] : :403 h0:s
Jun 8 04:32:15 s02 kernel: 07P: c 16 b >19 i, d<16U hu066C6 t1 [.] : :407e3c>17 i, d<17U 8hu366C2 t11[.] : :407 h6:s
Jun 8 04:32:15 s02 kernel: 07P:1c 816 b3 >17 i, d<18drr<18U 8hu166C1 t11[.] : :403 h6:s
Jun 8 04:32:15 s02 kernel: 07P:1c 516 b3 >15 i, d<10U 8hu466aa7n_0ln403tl9ci3of
Jun 8 04:32:15 s02 kernel: 08ua6y1b6a<10r4_m1leb216 d 2ae6e[.]0r0nlB4toie0ien_ki:oakal tkk 0t0p 0bi0bleetBakaBerkkselle<12ws:2 <13dAe6: 0g4in4cnkv:ktl3nb1oakalke4m:B:Bb2p2ek_m4sra6rakt2ueokempc0nmn16 _e0 <15dre4:B5h6cn5ie2 _9Bie6nb0lnBteBn7m: 1ke9m2B:Bra4sra5et2ae0tkcwc0enlcl<16ws: 16 *862422*1B80 B66ND2 B6*23Bk213B91B66NN B8*3 k855* k97
Jun 8 04:32:15 s02 kernel: 095tae
Jun 8 04:32:15 s02 kernel: 09 i <10ae:,en<10e
Jun 8 04:32:15 s02 kernel: 09oaB<4>[160101.733911] 8383984 pages RAM
Jun 8 04:32:15 s02 kernel: <>[160101.733929] 164294 ae eee<67]2ga41313ss
Jun 8 04:32:15 s02 kernel: 67]dugt ruamen614[]082 3 0 i[190 1 9 0
Jun 8 04:32:15 s02 kernel: 67]1 9 25- -s614[]053 5 0 t>021 s<03 6 4 u614[]06 0 0 l614[]349 9 0 s>031 4 g[138 1 8 0i1.03 8 38
Jun 8 04:32:15 s02 kernel: 67]7 3 86 u<03 9376 4 w614[]309 6 0 s>061 8 g[158 1 4 0i1.63 8 06
Jun 8 04:32:15 s02 kernel: 67]8 3 06 u<03 5386 9 w614[]360 4 0 s>081 8 g[168 1 4 0i1.23 8 40
Jun 8 04:32:15 s02 kernel: 67]9 3 73 u<03 1397 6 w615[]321 7 0 s>001 5 g[188 1 01 0i1.03 8 29
Jun 8 04:32:15 s02 kernel: 67]9 3 44 u<03 7398 6 w615[]382 4 0 s>021 5 g[108 1 11 0i1.64 8 72
Jun 8 04:32:15 s02 kernel: 67]0 4 65 u<03 9309 2 w615[]326 5 0 s>041 2 g[118 1 01 0i1.34 8 14
Jun 8 04:32:15 s02 kernel: 67]1 4 95 u<03 2348 2 w615[]356 9 0 s>012 3 g[127 2 31 0i1.84 7 33
Jun 8 04:32:24 s02 kernel: 67]7 0 13 u<03 9 5 8 r>04 0 <03 5 1 4 ut[185 4 0o<03 7 7 0 ot[165 1 2 0i1.0orum: s1sc9acl[15lr ia3ko:kl:B062]oc=pgt_e413P8oot e.m1a.d
Jun 8 04:32:24 s02 kernel: 62] :1.3ff7 pe8d1.5ff4 s+0413 f1ctnc+0413 f17_plroe<08 f84 io0x413 f18s_e/<08 f81 go_y/<08 f8f gh_xx414 f1apas+0414 f17_plroe<08 f81 _pc+/
我注意到其他两台服务器(相同的硬件)上也有类似的行为。有可能所有 3 台服务器都损坏了内存,但所有应用程序仍然运行良好,并且重复memtester
检查没有显示任何问题(在其他服务器上,我总是能够在测试期间重现内存问题)。
更新2:我忘了提及,所有 3 台机器上都运行着 LXC 容器。我在没有 LXC 容器的机器上没有看到类似的问题。
答案1
读完这篇文章后:Linux 内核错误导致 TCP/IP 损坏,我想这些问题可能是有联系的。原来的veth驱动程序出现在内核中2.6.24-rc1,该错误于 2010 年引入这个补丁(v2.6.37-rc8)。现在,此问题的修复应该向后移植到内核 3.14+ 中,因此升级到其中一个内核可能会解决此问题。