间歇性灰渣问题

间歇性灰渣问题

例如,Cinder 操作……cinder list在一次不受控制的关闭之后突然变得异常不稳定。

简而言之,我可以执行 3 次,并以随机顺序成功和失败,并出现以下错误:

root@openstack1:~# cinder list
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-bb341379-ebed-459f-9bd7-e1908588276a)

在 RabbitMQ 中:

=WARNING REPORT==== 31-Oct-2019::11:00:06 ===
closing AMQP connection <0.13826.0> (10.10.10.169:39138 -> 10.10.10.167:5672 - cinder-volume-usage-audit:983624:8aabffdd-a947-4c4a-89b6-7d7f69f6910a, vhost: 'openstack', user: 'cinder'):
client unexpectedly closed TCP connection

但是从 10.10.10.169 ping 到 RabbitMQ 服务器之类的操作并没有显示任何我能看到的网络问题,但这确实意味着我目前无法启动任何实例,但运行节点似乎正常。

有任何想法吗?

== 编辑 ==

好的,我找到了调试标志:

DEBUG:keystoneauth:REQ: curl -g -i -X GET http://10.10.10.169:8776/v3/26c85db719e64f6595a109f45b2715e1/volumes/detail -H "Accept: application/json" -H "OpenStack-API-Version: volume 3.52" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA256}e4f755976c1a91b0ff0ee77adde8556b0d1c9bf201edd6bc35cc5e80578070e5"
DEBUG:keystoneauth:RESP: [500] Connection: close Content-Length: 128 Content-Type: application/json Date: Thu, 31 Oct 2019 11:56:36 GMT OpenStack-API-Version: volume 3.52 Server: Apache/2.4.29 (Ubuntu) Vary: OpenStack-API-Version x-compute-request-id: req-abfd0b01-ddd5-43d2-adcf-adb9d40f0569 x-openstack-request-id: req-abfd0b01-ddd5-43d2-adcf-adb9d40f0569
DEBUG:keystoneauth:RESP BODY: {"computeFault": {"code": 500, "message": "The server has either erred or is incapable of performing the requested operation."}}
DEBUG:keystoneauth:GET call to volumev3 for http://10.10.10.169:8776/v3/26c85db719e64f6595a109f45b2715e1/volumes/detail used request id req-abfd0b01-ddd5-43d2-adcf-adb9d40f0569
DEBUG (shell:913) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-abfd0b01-ddd5-43d2-adcf-adb9d40f0569)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cinderclient/shell.py", line 905, in main
    OpenStackCinderShell().main(sys.argv[1:])
  File "/usr/lib/python3/dist-packages/cinderclient/shell.py", line 728, in main
    args.func(self.cs, args)
  File "/usr/lib/python3/dist-packages/cinderclient/v3/shell.py", line 388, in do_list
    sort_dir=args.sort_dir, sort=args.sort)
  File "/usr/lib/python3/dist-packages/cinderclient/v2/volumes.py", line 325, in list
    return self._list(url, resource_type, limit=limit)
  File "/usr/lib/python3/dist-packages/cinderclient/base.py", line 80, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 200, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 191, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 177, in request
    raise exceptions.from_response(resp, body)
cinderclient.exceptions.ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-abfd0b01-ddd5-43d2-adcf-adb9d40f0569)
DEBUG:cinderclient.shell:The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-abfd0b01-ddd5-43d2-adcf-adb9d40f0569)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cinderclient/shell.py", line 905, in main
    OpenStackCinderShell().main(sys.argv[1:])
  File "/usr/lib/python3/dist-packages/cinderclient/shell.py", line 728, in main
    args.func(self.cs, args)
  File "/usr/lib/python3/dist-packages/cinderclient/v3/shell.py", line 388, in do_list
    sort_dir=args.sort_dir, sort=args.sort)
  File "/usr/lib/python3/dist-packages/cinderclient/v2/volumes.py", line 325, in list
    return self._list(url, resource_type, limit=limit)
  File "/usr/lib/python3/dist-packages/cinderclient/base.py", line 80, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 200, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 191, in _cs_request
    return self.request(url, method, **kwargs)
  File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 177, in request
    raise exceptions.from_response(resp, body)
cinderclient.exceptions.ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-abfd0b01-ddd5-43d2-adcf-adb9d40f0569)
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-abfd0b01-ddd5-43d2-adcf-adb9d40f0569)

所以,如果我没看错的话,cinder 会因为某些随机原因而偶尔中断吗?

== 编辑更多 ==

所以我发现一些数据库内容被吐进了 Cinder 日志中,并且匹配https://review.opendev.org/#/c/668646/这个错误,所以我将提交粘贴到 cinder py 文件中并将其踢出,这样就摆脱了日志中的垃圾,并且我能够再次删除和创建卷,但只有当它对上述错误不感到难过时。

尽管每隔 3 次左右cinder list仍然会出现一次呕吐,而且在我看来,它似乎没有在日志中留下任何东西。

相关内容