昨晚午夜左右,我们的应用程序崩溃了,我正在努力找出原因。我们目前在 EC2 上有一个 nginx 前端服务器和 2 个 unicorn 工作器(应用程序)服务器。
我们的独角兽工人几乎一直在超时,因此被主人重新启动。
据我所见,我们此时没有设置 crontabs 或任何要运行的东西,所以我有点困惑。
今天早上我醒来后(停机 6 小时后),通过终止 unicorn 进程并重新执行 unicorn 二进制文件,我能够启动该应用程序。(unicorn_rails -c unicorn.rb 等)
有什么想法可以查看吗?两个应用服务器都宕机了,这让我认为可能是数据库 (RDS) 的问题?
日志中填写了以下内容(持续6个小时..等等);
E, [2013-02-28T00:07:40.367981 #11097] ERROR -- : worker=2 PID:26941 timeout (31s > 30s), killing
E, [2013-02-28T00:07:40.468495 #11097] ERROR -- : reaped #<Process::Status: pid 26941 SIGKILL (signal 9)> worker=2
I, [2013-02-28T00:07:40.756724 #28319] INFO -- : worker=2 ready
E, [2013-02-28T00:07:44.519818 #11097] ERROR -- : worker=1 PID:11292 timeout (31s > 30s), killing
E, [2013-02-28T00:07:44.626362 #11097] ERROR -- : worker=0 PID:26933 timeout (31s > 30s), killing
E, [2013-02-28T00:07:44.726936 #11097] ERROR -- : reaped #<Process::Status: pid 11292 SIGKILL (signal 9)> worker=1
E, [2013-02-28T00:07:44.727254 #11097] ERROR -- : worker=0 PID:26933 timeout (31s > 30s), killing
E, [2013-02-28T00:07:44.932858 #11097] ERROR -- : reaped #<Process::Status: pid 26933 SIGKILL (signal 9)> worker=0
I, [2013-02-28T00:07:45.661356 #28329] INFO -- : worker=1 ready
I, [2013-02-28T00:07:45.828289 #28334] INFO -- : worker=0 ready
E, [2013-02-28T00:08:11.113970 #11097] ERROR -- : worker=2 PID:28319 timeout (31s > 30s), killing
E, [2013-02-28T00:08:11.214770 #11097] ERROR -- : reaped #<Process::Status: pid 28319 SIGKILL (signal 9)> worker=2
I, [2013-02-28T00:08:11.518723 #28368] INFO -- : worker=2 ready
E, [2013-02-28T00:08:16.270463 #11097] ERROR -- : worker=1 PID:28329 timeout (31s > 30s), killing
E, [2013-02-28T00:08:16.371067 #11097] ERROR -- : worker=0 PID:28334 timeout (31s > 30s), killing
E, [2013-02-28T00:08:16.471684 #11097] ERROR -- : reaped #<Process::Status: pid 28329 SIGKILL (signal 9)> worker=1
E, [2013-02-28T00:08:16.471983 #11097] ERROR -- : reaped #<Process::Status: pid 28334 SIGKILL (signal 9)> worker=0
I, [2013-02-28T00:08:17.038915 #28376] INFO -- : worker=0 ready
I, [2013-02-28T00:08:17.128931 #28379] INFO -- : worker=1 ready
E, [2013-02-28T00:08:42.628665 #11097] ERROR -- : worker=2 PID:28368 timeout (31s > 30s), killing
E, [2013-02-28T00:08:42.729290 #11097] ERROR -- : reaped #<Process::Status: pid 28368 SIGKILL (signal 9)> worker=2
I, [2013-02-28T00:08:43.015140 #28390] INFO -- : worker=2 ready
E, [2013-02-28T00:08:48.778221 #11097] ERROR -- : worker=0 PID:28376 timeout (31s > 30s), killing
E, [2013-02-28T00:08:48.878530 #11097] ERROR -- : worker=1 PID:28379 timeout (31s > 30s), killing
答案1
伙计们,我设法解决了这个问题。进一步调查显示,在凌晨 12 点到凌晨 4 点之间,网络流量(和 CPU 使用率!)非常大。原来,我们的 Bing 网站管理员设置在这段时间内设置为“高”,导致 Bingbot 发疯并彻底摧毁了 Unicorn。(可怜的 Unicorn)。