我有 2 个数据中心,每个数据中心都有一个运行 Percona 服务器 5.5.23 的主/从数据库。由于某种原因,其中一个从属数据库崩溃并出现以下错误:
03:00:40 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=1
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346809 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x80000
/usr/local/Percona-Server-5.5.23-rel25.3- 240.Linux.x86_64/bin/mysqld(my_print_stacktrace+0x35)[0x7d4c85]
/usr/local/Percona-Server-5.5.23-rel25.3-240.Linux.x86_64/bin/mysqld(handle_fatal_signal+0x3e1)[0x690cb1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fd2b94c7cb0]
/usr/local/Percona-Server-5.5.23-rel25.3-240.Linux.x86_64/bin/mysqld[0x8f2df2]
/usr/local/Percona-Server-5.5.23-rel25.3-240.Linux.x86_64/bin/mysqld[0x81d607]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fd2b94bfe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd2b86a93fd]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.
140802 03:00:41 mysqld_safe Number of processes running now: 0
140802 03:00:41 mysqld_safe mysqld restarted
/usr/local/Percona-Server-5.5.23-rel25.3-240.Linux.x86_64/bin/mysqld: error while loading shared libraries: unexpected PLT reloc type 0x00
140802 03:00:41 mysqld_safe mysqld from pid file /dbdata1/mysqld.pid ended
每次尝试重新启动该从属服务器时,都会出现上一个错误日志的最后 3 行。它没有说明它在加载时遇到问题的共享库。我搜索了“意外的 PLT 重定位类型”,但没有具体解释那是什么。我确实看到了一些关于损坏的二进制文件如何导致它的信息,所以我在所有 4 个数据库服务器上对 mysqld 运行了校验和。结果发现我所有工作的数据库都显示相同的校验和:
sha256sum mysqld
0b42e4625a87de52e5f51f2eb74fb7f2db63116e2b78f51d2897c1938a0e03d1 mysqld
正如我的损坏数据库显示的那样:
sha256sum mysqld
7bfd58d1c1948a36cf4602c697dadd60e422d61ff75eeb4a0344f8ec395b03ea mysqld
因此二进制文件似乎已损坏,但奇怪的是,所有二进制文件的修改日期和字节数都相同。我不确定正在运行的服务器的二进制文件可能发生了什么,导致其损坏。
我可以尝试重新安装 percona 来获取可运行的二进制文件,但我想知道这里发生了什么,以便我可以防止它将来再次发生。
答案1
感谢这篇博客文章:https://blogs.oracle.com/ksplice/entry/attack_of_the_cosmic_rays1
因此,二进制文件也具有与其他正常工作的服务器相同的字节数和修改时间(2012 年),因此奇怪的是磁盘上的二进制文件没有更改,但校验和仍然不同。似乎二进制文件完全缓存在 RAM 中,这就是损坏的原因。清除缓存有效!