停电后,我们的 OpenStack 安装中断了。第一步是尝试让 MAAS 恢复运行,但我遇到了麻烦。RC 和 CC 正在运行并且可以通信(有网络)。
据我了解,MAAS Region Controller API 通过 Apache 上的 WSGI 作为 Django 应用程序运行。Apache 运行,但无法加载 MAAS WSGI 脚本(/usr/share/maas/wsgi.py)。因此服务和 API 从未启动。RabbitMQ 和其他服务似乎正在 RC 上运行。我还尝试过:
dpkg-reconfigure maas-region-controller
但没有成功。
下面的日志可以在/var/log/apache/error.log。
有什么想法我可以尝试吗?
[Wed Jan 08 11:01:06.312804 2014] [mpm_event:notice] [pid 5725:tid 139810242742144] AH00491: caught SIGTERM, shutting down
[Wed Jan 08 11:01:06.326364 2014] [:error] [pid 5728:tid 139810242742144] mod_wsgi (pid=5728): Target WSGI script '/usr/share/maas/wsgi.py' cannot be loaded as Python module.
[Wed Jan 08 11:01:06.326607 2014] [:error] [pid 5728:tid 139810242742144] mod_wsgi (pid=5728): Exception occurred processing WSGI script '/usr/share/maas/wsgi.py'.
[Wed Jan 08 11:01:06.326846 2014] [:error] [pid 5728:tid 139810242742144] Traceback (most recent call last):
[Wed Jan 08 11:01:06.327054 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/share/maas/wsgi.py", line 32, in <module>
[Wed Jan 08 11:01:06.327424 2014] [:error] [pid 5728:tid 139810242742144] start_up()
[Wed Jan 08 11:01:06.327611 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 70, in start_up
[Wed Jan 08 11:01:06.327934 2014] [:error] [pid 5728:tid 139810242742144] post_start_up()
[Wed Jan 08 11:01:06.328109 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 110, in post_start_up
[Wed Jan 08 11:01:06.328320 2014] [:error] [pid 5728:tid 139810242742144] NodeGroup.objects.refresh_workers()
[Wed Jan 08 11:01:06.328491 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/maasserver/models/nodegroup.py", line 116, in refresh_workers
[Wed Jan 08 11:01:06.328908 2014] [:error] [pid 5728:tid 139810242742144] refresh_worker(nodegroup)
[Wed Jan 08 11:01:06.329456 2014] [:error] [pid 5727:tid 139810242742144] mod_wsgi (pid=5727): Target WSGI script '/usr/share/maas/wsgi.py' cannot be loaded as Python module.
[Wed Jan 08 11:01:06.329662 2014] [:error] [pid 5727:tid 139810242742144] mod_wsgi (pid=5727): Exception occurred processing WSGI script '/usr/share/maas/wsgi.py'.
[Wed Jan 08 11:01:06.329879 2014] [:error] [pid 5727:tid 139810242742144] Traceback (most recent call last):
[Wed Jan 08 11:01:06.330074 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/share/maas/wsgi.py", line 32, in <module>
[Wed Jan 08 11:01:06.330389 2014] [:error] [pid 5727:tid 139810242742144] start_up()
[Wed Jan 08 11:01:06.330567 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 70, in start_up
[Wed Jan 08 11:01:06.330879 2014] [:error] [pid 5727:tid 139810242742144] post_start_up()
[Wed Jan 08 11:01:06.331050 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/maasserver/start_up.py", line 110, in post_start_up
[Wed Jan 08 11:01:06.331260 2014] [:error] [pid 5727:tid 139810242742144] NodeGroup.objects.refresh_workers()
[Wed Jan 08 11:01:06.331426 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/maasserver/models/nodegroup.py", line 116, in refresh_workers
[Wed Jan 08 11:01:06.331831 2014] [:error] [pid 5727:tid 139810242742144] refresh_worker(nodegroup)
[Wed Jan 08 11:01:06.332021 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/maasserver/refresh_worker.py", line 41, in refresh_worker
[Wed Jan 08 11:01:06.332207 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/maasserver/refresh_worker.py", line 41, in refresh_worker
[Wed Jan 08 11:01:06.332485 2014] [:error] [pid 5727:tid 139810242742144] refresh_secrets.apply_async(queue=nodegroup.work_queue, kwargs=items)
[Wed Jan 08 11:01:06.332758 2014] [:error] [pid 5728:tid 139810242742144] refresh_secrets.apply_async(queue=nodegroup.work_queue, kwargs=items)
[Wed Jan 08 11:01:06.332938 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 459, in apply_async
[Wed Jan 08 11:01:06.333143 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 459, in apply_async
[Wed Jan 08 11:01:06.333856 2014] [:error] [pid 5727:tid 139810242742144] **options)
[Wed Jan 08 11:01:06.334453 2014] [:error] [pid 5728:tid 139810242742144] **options)
[Wed Jan 08 11:01:06.334627 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 196, in delay_task
[Wed Jan 08 11:01:06.334805 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 196, in delay_task
[Wed Jan 08 11:01:06.335226 2014] [:error] [pid 5727:tid 139810242742144] entity = self._declare_queue(queue, retry, _retry_policy)
[Wed Jan 08 11:01:06.335642 2014] [:error] [pid 5728:tid 139810242742144] entity = self._declare_queue(queue, retry, _retry_policy)
[Wed Jan 08 11:01:06.335823 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 172, in _declare_queue
[Wed Jan 08 11:01:06.336002 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/celery/app/amqp.py", line 172, in _declare_queue
[Wed Jan 08 11:01:06.336208 2014] [:error] [pid 5728:tid 139810242742144] queue.declare()
[Wed Jan 08 11:01:06.336396 2014] [:error] [pid 5727:tid 139810242742144] queue.declare()
[Wed Jan 08 11:01:06.336560 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 492, in declare
[Wed Jan 08 11:01:06.336733 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 492, in declare
[Wed Jan 08 11:01:06.337737 2014] [:error] [pid 5727:tid 139810242742144] self.exchange.declare(nowait)
[Wed Jan 08 11:01:06.337941 2014] [:error] [pid 5728:tid 139810242742144] self.exchange.declare(nowait)
[Wed Jan 08 11:01:06.338112 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
[Wed Jan 08 11:01:06.338318 2014] [:error] [pid 5727:tid 139810242742144] nowait=nowait, passive=passive,
[Wed Jan 08 11:01:06.338492 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
[Wed Jan 08 11:01:06.338690 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 603, in exchange_declare
[Wed Jan 08 11:01:06.338897 2014] [:error] [pid 5728:tid 139810242742144] nowait=nowait, passive=passive,
[Wed Jan 08 11:01:06.340001 2014] [:error] [pid 5727:tid 139810242742144] (40, 11), # Channel.exchange_declare_ok
[Wed Jan 08 11:01:06.340193 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 603, in exchange_declare
[Wed Jan 08 11:01:06.341389 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait
[Wed Jan 08 11:01:06.341656 2014] [:error] [pid 5728:tid 139810242742144] (40, 11), # Channel.exchange_declare_ok
[Wed Jan 08 11:01:06.341972 2014] [:error] [pid 5727:tid 139810242742144] self.channel_id, allowed_methods)
[Wed Jan 08 11:01:06.342292 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait
[Wed Jan 08 11:01:06.342618 2014] [:error] [pid 5728:tid 139810242742144] self.channel_id, allowed_methods)
[Wed Jan 08 11:01:06.342805 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 204, in _wait_method
[Wed Jan 08 11:01:06.343462 2014] [:error] [pid 5727:tid 139810242742144] self.method_reader.read_method()
[Wed Jan 08 11:01:06.343659 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 204, in _wait_method
[Wed Jan 08 11:01:06.344309 2014] [:error] [pid 5728:tid 139810242742144] self.method_reader.read_method()
[Wed Jan 08 11:01:06.344499 2014] [:error] [pid 5727:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method
[Wed Jan 08 11:01:06.344883 2014] [:error] [pid 5727:tid 139810242742144] raise m
[Wed Jan 08 11:01:06.345057 2014] [:error] [pid 5728:tid 139810242742144] File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method
[Wed Jan 08 11:01:06.345561 2014] [:error] [pid 5728:tid 139810242742144] raise m
[Wed Jan 08 11:01:06.345765 2014] [:error] [pid 5727:tid 139810242742144] error: [Errno 4] Interrupted system call
[Wed Jan 08 11:01:06.349404 2014] [:error] [pid 5728:tid 139810242742144] error: [Errno 4] Interrupted system call
[Wed Jan 08 11:01:07.375856 2014] [mpm_event:notice] [pid 7064:tid 140623783114624] AH00489: Apache/2.4.6 (Ubuntu) mod_wsgi/3.4 Python/2.7.5+ configured -- resuming normal operations
[Wed Jan 08 11:01:07.389375 2014] [core:notice] [pid 7064:tid 140623783114624] AH00094: Command line: '/usr/sbin/apache2'
答案1
检查 /var/log/rabbitmq/[电子邮件保护]对于如下消息:
=INFO REPORT==== 20-Jan-2014::03:45:19 ===
Disk free limit set to 1000MB
=INFO REPORT==== 20-Jan-2014::03:45:19 ===
Disk free space insufficient. Free bytes:967696384 Limit:1000000000
=WARNING REPORT==== 20-Jan-2014::03:45:19 ===
disk resource limit alarm set on node rabbit@maas.
**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************
如果您看到此消息,请释放磁盘空间。如果它提醒内存资源限制,请释放 RAM 空间或增加机器的 RAM 大小。