Ubuntu 12.04 服务器上的异常 sendmail 活动耗尽了服务器内存

Ubuntu 12.04 服务器上的异常 sendmail 活动耗尽了服务器内存

环境

  • Rackspace
  • Ubuntu 12.04
  • WordPress 的
  • MySQL的

问题

在过去的几天里,我遇到了相当严重的内存不足问题。

尽管我解决了一个可能导致该问题的原因我仍然发现非常可疑的活动sendmail

有什么建议可以解决这个问题吗?我认为这肯定是某种恶意软件,但我没有解决此类攻击的经验。

顶部

  1  [|||||||||||||||||||||||||                                                          27.0%]     Tasks: 101, 50 thr; 1 running
  2  [|||||||||||||||||||||||||||||||||||||||||                                          45.7%]     Load average: 12.96 12.55 11.95 
  Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||1183/1995MB]     Uptime: 09:53:28
  Swp[||||                                                                           93/2047MB]

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
19704 root       20   0  120M 25328  2896 S  2.0  1.2  0:46.16 sendmail: MTA: ./s6HH4rLv009027 gmail.co.: user open
 3298 root       20   0   99M  5612  1684 S  2.0  0.3  2:46.31 sendmail: MTA: s6OABpf4003298 localhost [127.0.0.1]: DATA
 3301 root       20   0   99M  5544  1684 S  2.0  0.3  2:40.89 sendmail: MTA: s6OAGAAh003301 localhost [127.0.0.1]: DATA
19510 root       20   0 26488  2568  1212 R  2.0  0.1  0:23.73 htop
  771 syslog     20   0  244M  3892   516 S  1.0  0.2  2:22.43 rsyslogd -c5
 1226 smmsp      20   0  133M 56328  1396 S  0.0  2.8  1:56.85 sendmail: MSP: ./s6K1OdvJ030780 [127.0.0.1]: client DATA status
32488 root       20   0  102M  7168  2748 S  0.0  0.4  0:00.02 sendmail: MTA: ./s6OAcr6I032488 aspmx.l.google.com.: client EHLO
31723 www-data   39  19  448M 72676 47276 S  0.0  3.6  0:01.14 /usr/sbin/apache2 -k start
29624 root       20   0  120M 25916  2884 S  0.0  1.3  0:29.65 sendmail: MTA: ./s6NHPdHs002287 todito.com.: user open
  898 mysql      20   0 1315M  105M  3296 S  0.0  5.3 23:25.23 /usr/sbin/mysqld
30966 root       20   0  101M  5092   460 D  0.0  0.2  0:01.52 sendmail: MTA: running queue: /var/spool/mqueue
 5013 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:25.58 /usr/sbin/mysqld
25504 root       20   0  120M 25904  2900 S  0.0  1.3  0:24.57 sendmail: MTA: ./s6JHcEdS028616 hotamil.com.: user open
 1033 root       20   0  630M  6228  2356 S  0.0  0.3  1:17.85 /usr/local/bin/driveclient --daemon
 1062 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.50 /usr/local/bin/driveclient --daemon
 1082 newrelic   20   0  107M  1576  1072 S  0.0  0.1  0:46.81 /usr/sbin/nrsysmond -c /etc/newrelic/nrsysmond.cfg -p /var/run/nrsysmond.pid
 1089 newrelic   20   0  107M  1576  1072 S  0.0  0.1  0:46.80 /usr/sbin/nrsysmond -c /etc/newrelic/nrsysmond.cfg -p /var/run/nrsysmond.pid
  822 syslog     20   0  244M  3892   516 S  0.0  0.2  1:35.12 rsyslogd -c5
 1061 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.80 /usr/local/bin/driveclient --daemon
 8532 root       20   0  105M  9444   460 D  0.0  0.5  0:06.40 sendmail: MTA: running queue: /var/spool/mqueue
31711 www-data   39  19  445M 75316 52764 S  0.0  3.7  0:01.50 /usr/sbin/apache2 -k start
27927 root       20   0  120M 25904  2900 S  0.0  1.3  0:32.35 sendmail: MTA: ./s6NKLEhE005721 yahoo.co.: user open
13821 mysql      20   0 1315M  105M  3296 S  0.0  5.3  2:25.39 /usr/sbin/mysqld
31924 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:49.12 /usr/sbin/mysqld
31713 www-data   39  19  446M 68484 45496 S  0.0  3.4  0:00.79 /usr/sbin/apache2 -k start
 4195 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:29.08 /usr/sbin/mysqld
 9799 mysql      20   0 1315M  105M  3296 S  0.0  5.3  2:29.95 /usr/sbin/mysqld
 2664 smmsp      20   0  133M 56424  1476 D  0.0  2.8  1:52.68 sendmail: MSP: ./s6K3MC7s027126 [127.0.0.1]: client DATA status
  853 syslog     20   0  244M  3892   516 S  0.0  0.2  0:47.23 rsyslogd -c5
31714 www-data   39  19  446M 68404 45420 S  0.0  3.3  0:00.73 /usr/sbin/apache2 -k start
31903 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:47.96 /usr/sbin/mysqld
 1063 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.40 /usr/local/bin/driveclient --daemon
31600 www-data   39  19  448M 71340 46228 S  0.0  3.5  0:00.92 /usr/sbin/apache2 -k start
 4308 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:28.28 /usr/sbin/mysqld
 1064 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.41 /usr/local/bin/driveclient --daemon
31727 www-data   39  19  447M 70324 45756 S  0.0  3.4  0:00.84 /usr/sbin/apache2 -k start
31725 www-data   39  19  447M 70340 45756 S  0.0  3.4  0:00.86 /usr/sbin/apache2 -k start
31724 www-data   39  19  447M 70548 45932 S  0.0  3.5  0:00.84 /usr/sbin/apache2 -k start
 1715 mysql      20   0 1315M  105M  3296 S  0.0  5.3  3:05.00 /usr/sbin/mysqld
23774 root       39  19  425M  6636  4676 S  0.0  0.3  0:06.00 /usr/sbin/apache2 -k start
 1065 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.35 /usr/local/bin/driveclient --daemon
 1060 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.43 /usr/local/bin/driveclient --daemon
F1Help  F2Setup F3SearchF4FilterF5Tre

巨大的 /var/mail

root@web:/var/mail# ls -alh
total 1.2G
drwxrwsrwt  2 root     mail 4.0K Jul 24 10:51 .
drwxr-xr-x 15 root     root 4.0K Jul 24 00:45 ..
-rw-rw----  1 munin    mail  83K Jul 19 18:48 munin
-rw-------  1 root     mail 1.1G Jul 24 10:51 root
-rw-rw----  1 www-data mail  98M Jul 23 22:34 www-data

我的根邮件账户不断发送电子邮件

注:我已将我的域名替换为filtered.com

Return-Path: <MAILER-DAEMON>
Received: from localhost (localhost)
    by web.filtered.com (8.14.4/8.14.4/Debian-2ubuntu2) id s6OAqpuv010033;
    Thu, 24 Jul 2014 10:52:51 GMT
Date: Thu, 24 Jul 2014 10:52:51 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON>
Message-Id: <[email protected]>
To: <[email protected]>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
    boundary="s6OAqpuv010033.1406199171/web.filtered.com"
Subject: Returned mail: see transcript for details
Auto-Submitted: auto-generated (failure)

--s6OAqpuw010033.1406199172/web.filtered.com--

From MAILER-DAEMON  Thu Jul 24 10:52:53 2014
Return-Path: <MAILER-DAEMON>
Received: from localhost (localhost)
    by web.filtered.com (8.14.4/8.14.4/Debian-2ubuntu2) id s6OAqq6J010047;
    Thu, 24 Jul 2014 10:52:53 GMT
Date: Thu, 24 Jul 2014 10:52:53 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON>
Message-Id: <[email protected]>
To: postmaster
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
    boundary="s6OAqq6J010047.1406199173/web.filtered.com"
Subject: Postmaster notify: see transcript for details
Auto-Submitted: auto-generated (postmaster-notification)

This is a MIME-encapsulated message

--s6OAqq6J010047.1406199173/web.filtered.com

The original message was received at Thu, 24 Jul 2014 10:52:52 GMT
from localhost
with id s6OAqq6I010047

   ----- The following addresses had permanent fatal errors -----
<[email protected]>
    (reason: 550-5.1.1 The email account that you tried to reach does not exist. Please try)

   ----- Transcript of session follows -----
... while talking to aspmx.l.google.com.:
>>> RCPT To:<[email protected]>
<<< 550-5.1.1 The email account that you tried to reach does not exist. Please try
<<< 550-5.1.1 double-checking the recipient's email address for typos or
<<< 550-5.1.1 unnecessary spaces. Learn more at
<<< 550 5.1.1 http://support.google.com/mail/bin/answer.py?answer=6596 sq8si14059110obc.83 - gsmtp
550 5.1.1 <[email protected]>... User unknown
>>> DATA
<<< 503 5.5.1 RCPT first. sq8si14059110obc.83 - gsmtp

--s6OAqq6J010047.1406199173/web.filtered.com
Content-Type: message/delivery-status

Reporting-MTA: dns; web.filtered.com
Received-From-MTA: DNS; localhost
Arrival-Date: Thu, 24 Jul 2014 10:52:52 GMT

Final-Recipient: RFC822; [email protected]
Action: failed
Status: 5.1.1
Remote-MTA: DNS; aspmx.l.google.com
Diagnostic-Code: SMTP; 550-5.1.1 The email account that you tried to reach does not exist. Please try
Last-Attempt-Date: Thu, 24 Jul 2014 10:52:53 GMT

--s6OAqq6J010047.1406199173/web.filtered.com
Content-Type: text/rfc822-headers

Return-Path: <MAILER-DAEMON>
Received: from localhost (localhost)
    by web.filtered.com (8.14.4/8.14.4/Debian-2ubuntu2) id s6OAqq6I010047;
    Thu, 24 Jul 2014 10:52:52 GMT
Date: Thu, 24 Jul 2014 10:52:52 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON>
Message-Id: <[email protected]>
To: <[email protected]>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
    boundary="s6OAqq6I010047.1406199172/web.filtered.com"
Subject: Returned mail: see transcript for details
Auto-Submitted: auto-generated (failure)

--s6OAqq6J010047.1406199173/web.filtered.com--

ps -ef | grep sendmail

root@web:/var/mail# ps -ef | grep sendmail
smmsp     1226     1  0 00:45 ?        00:02:04 sendmail: MSP: ./s6KKDDVU014035 [127.0.0.1]: client DATA status
smmsp     2644  2641  0 01:00 ?        00:00:00 /bin/sh -c test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp
smmsp     2647  2644  0 01:00 ?        00:00:00 /bin/sh /usr/share/sendmail/sendmail cron-msp
smmsp     2664  2647  0 01:00 ?        00:01:58 sendmail: MSP: [127.0.0.1]: idle              
root      3298     1  1 07:57 ?        00:03:16 sendmail: MTA: s6OB1dam003298 localhost [127.0.0.1]: DATA
root      3301     1  1 07:57 ?        00:03:05 sendmail: MTA: server localhost [127.0.0.1] cmd read
root     19675     1  0 11:20 ?        00:00:00 sendmail: MTA: ./s6OBKJuv019675 aspmx.l.google.com.: client DATA 354
root     19689     1  0 11:20 ?        00:00:00 sendmail: MTA: ./s6OBKLuv019689 aspmx.l.google.com.: client DATA 354
root     19800     1  0 11:20 ?        00:00:00 sendmail: MTA: ./s6OBKbuv019800 aspmx.l.google.com.: client DATA 354
root     20178     1  0 11:21 ?        00:00:00 sendmail: MTA: ./s6OBLSuv020178 aspmx.l.google.com.: client DATA 354
root     20270     1  0 11:21 ?        00:00:00 sendmail: MTA: ./s6OBLZuv020270 aspmx.l.google.com.: client DATA 354
root     20537     1  0 11:21 ?        00:00:00 sendmail: MTA: ./s6OBM0uv020537 aspmx.l.google.com.: client DATA 354
root     20646     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBM5uv020646 aspmx.l.google.com.: client DATA 354
root     21006     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMZ6I021006 aspmx.l.google.com.: client DATA 354
root     21015     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMZ6I021015 aspmx.l.google.com.: client DATA 354
root     21027     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMauv021027 aspmx.l.google.com.: client DATA 354
root     21036     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMb6I021036 aspmx.l.google.com.: client DATA 354
root     21063     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMeuv021063 aspmx.l.google.com.: client DATA 354
root     21065     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021065 aspmx.l.google.com.: client DATA 354
root     21086     1  1 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021086 aspmx.l.google.com.: client DATA 354
root     21094     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021094 aspmx.l.google.com.: client DATA 354
root     21098     1  2 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021098 aspmx.l.google.com.: client DATA 354
root     21103     1  1 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021103 aspmx.l.google.com.: client DATA 354
root     21105     1  1 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMguv021105 aspmx.l.google.com.: client DATA 354
root     21108     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OB1dag003298 mx-eu.mail.am0.yahoodns.net.: client MAIL
root     21111     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021111 aspmx.l.google.com.: client RCPT
root     21113     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OAsOi1003301 mx-ha03.web.de.: client greeting
root     21117     1  1 11:22 ?        00:00:00 sendmail: MTA: ./s6OAsOi3003301 gmail-smtp-in.l.google.com.: client DATA status
root     21123     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OAsOi5003301 gmail-smtp-in.l.google.com.: client EHLO
root     21127 18604  0 11:22 pts/0    00:00:00 grep --color=auto sendmail

Sendmail 状态

root@web:/var/mail# /etc/init.d/sendmail status
MSP: is run via cron (20m)
MTA: is not running
QUE: Same as MTA

/var/spool/mqueue

root@web:/var/spool# ls -alh
total 48M
drwxr-xr-x  7 root  root  4.0K Mar 29  2013 .
drwxr-xr-x 15 root  root  4.0K Jul 24 00:45 ..
drwxr-xr-x  5 root  root  4.0K May  1  2012 cron
lrwxrwxrwx  1 root  root     7 May  1  2012 mail -> ../mail
drwxr-s---  2 smmta smmsp  14M Jul 24 11:44 mqueue
drwxrws---  2 smmsp smmsp  34M Jul 24 12:25 mqueue-client
drwxr-xr-x  2 root  root  4.0K Apr 13  2012 plymouth
drwxr-xr-x  2 root  root  4.0K Mar 30  2012 rsyslog


root@web:/var/spool# du -h -d 1
4.0K    ./plymouth
1.6G    ./mqueue    <=====
4.0K    ./rsyslog
来自 /var/spool/mqueue 的一条消息
root@web:/var/spool/mqueue# more qfs6OBTUZY003298 
V8
T1406201622
K1406201622
N1
P120781
I202/1/476577
MDeferred: 421 4.7.1 : (DYN:T1) http://postmaster.info.aol.com/errors/421dynt1.html
Fbs
$_localhost [127.0.0.1]
$rESMTP
$sweb.anybots.com
${daemon_flags}
${if_addr}127.0.0.1
S<[email protected]>
MDeferred: 421 4.7.1 : (DYN:T1) http://postmaster.info.aol.com/errors/421dynt1.html
rRFC822; [email protected]
RPFD:<[email protected]>
H?P?Return-Path: <?g>
H??Received: from web.anybots.com (localhost [127.0.0.1])
    by web.anybots.com (8.14.4/8.14.4/Debian-2ubuntu2) with ESMTP id s6OBTUZY003298
    for <[email protected]>; Thu, 24 Jul 2014 11:33:42 GMT
H??Received: (from www-data@localhost)
    by web.anybots.com (8.14.4/8.14.4/Submit) id s6JHVJId026134;
    Sat, 19 Jul 2014 17:31:19 GMT
H??Date: Sat, 19 Jul 2014 17:31:19 GMT
H??Message-Id: <[email protected]>
H??X-Authentication-Warning: web.anybots.com: www-data set sender to [email protected] using -f
H??To: [email protected]
H??Subject: Fw:  Hi Generic Drugs Online Products
H??X-PHP-Originating-Script: 33:dirs.php
H??From: "Patty Jennings" <[email protected]>
H??Reply-To:"Patty Jennings" <[email protected]>
H??X-Priority: 3 (Normal)
H??MIME-Version: 1.0
H??Content-Type: text/html; charset="iso-8859-1"
H??Content-Transfer-Encoding: 8bit
.

答案1

您的问题可能是由巨大的两个 sendmail 队列中的垃圾邮件数量。
(请参阅https://serverfault.com/a/490890/163277

检查两个 sendmail 队列中的消息数量

sendmail -O QueueSortOrder=none -Am -bp
sendmail -O QueueSortOrder=none -Ac -bp 

最耗费内存的 sendmail 进程是 MTA 队列处理 (-Am)。剩余的进程是从 MSA 到 MTA 队列的传输,以及传输后首次尝试向外部服务器投递。


您可以使用qtool.pl脚本将 www-data(Web 服务器)发送的消息移动到另一个队列/目录。该脚本在 sendmail.org 发行版的 contrib 目录中以及 Debian-Linux 的 sendmail-base 包中提供。

相关内容