服务器持续崩溃

服务器持续崩溃

描述:我有一个 MySQL 服务器版本:5.0.77-log

它似乎每隔几分钟就会不断崩溃,崩溃的原因可能是信号 11、信号 6、断言失败,也可能是内存损坏(例如 Malloc 失败或 Free())...

该服务器采用双四核,8 GB 内存,运行 CentOS 5.x

仅供参考,使用了很多 innodb 表。总体数据库大小为 6-7 GB

不太确定是什么导致了这些持续的崩溃。

-------------------------
Extract of mysqld.log
-------------------------
101101 12:20:01 - mysqld got signal 11 ;
....
....
101101 12:29:01 - mysqld got signal 11 ;
..... //May be this was because i set force recovery to 1
InnoDB: A new raw disk partition was initialized or
InnoDB: innodb_force_recovery is on: we do not allow
InnoDB: database modifications by the user. Shut down
InnoDB: mysqld and edit my.cnf so that newraw is replaced
InnoDB: with raw, and innodb_force_... is removed.
..................
InnoDB: database modifications by the user. Shut down
InnoDB: mysqld and edit my.cnf so that newraw is replaced
InnoDB: with raw, and innodb_force_... is removed.
101101 14:50:11 [Note] /usr/libexec/mysqld: Normal shutdown
...........
....
101101 15:24:04  InnoDB: Error: MySQL is freeing a thd
InnoDB: though trx->n_mysql_tables_in_use is 1
InnoDB: and trx->mysql_n_tables_locked is 124.
TRANSACTION 0 598822756, not started, process no 28938, OS thread id 1629404048
mysql tables in use 1, locked 124
MySQL thread id 987, query id 43042622 localhost mv_php_usr
 len 860; hex 065...   Thank you!te is to cater to the ne;101102  9:13:47 - mysqld got
signal 11 ;
......
....
101102  9:46:29 - mysqld got signal 11 ;
...
101102 10:08:01InnoDB: Assertion failure in thread 1629285264 in file row0mysql.c line
145
InnoDB: Failing assertion: lenlen == 1
.....
.....
*** glibc detected *** /usr/libexec/mysqld: free(): invalid pointer: 0x6080eb50 ***
======= Backtrace: =========
/lib/libc.so.6[0x28e0f1]
/lib/libc.so.6(cfree+0x90)[0x291bc0]
/usr/libexec/mysqld(delete_dynamic+0x18)[0x848fe08]
/usr/libexec/mysqld(_ZN3THD7cleanupEv+0x8d)[0x817e04d]
/usr/libexec/mysqld(_Z10end_threadP3THDb+0x16)[0x818bc86]
/usr/libexec/mysqld(handle_one_connection+0x403)[0x81ad1b3]
/lib/libpthread.so.0[0x39f49b]
/lib/libc.so.6(clone+0x5e)[0x2f642e]
======= Memory map: ========
00169000-0016a000 r-xp 00169000 00:00 0          [vdso]
00179000-00182000 r-xp 00000000 fd:00 53478649   /lib/libcrypt-2.5.so
00182000-00183000 r--p 00008000 fd:00 53478649   /lib/libcrypt-2.5.so
00183000-00184000 rw-p 00009000 fd:00 53478649   /lib/libcrypt-2.5.so
00184000-001ab000 rw-p 00184000 00:00 0 
...
...
..
101102 12:03:44 - mysqld got signal 11 ;
....
...
*** glibc detected *** /usr/libexec/mysqld: malloc(): memory corruption: 0x60d23ec8 ***
======= Backtrace: =========
/lib/libc.so.6[0x28e883]
/lib/libc.so.6(__libc_malloc+0x7b)[0x2903ab]
/usr/libexec/mysqld(my_malloc+0x25)[0x8489355]
/usr/libexec/mysqld(init_alloc_root+0x73)[0x8489ce3]
/usr/libexec/mysqld(_Z14init_sql_allocP11st_mem_rootjj+0x20)[0x8158520]
.....
....
*** glibc detected *** /usr/libexec/mysqld: free(): invalid next size (fast): 0x0b2eea20
***
======= Backtrace: =========
/lib/libc.so.6[0x28e0f1]
/lib/libc.so.6(cfree+0x90)[0x291bc0]
/usr/libexec/mysqld(_Z19close_thread_tablesP3THDbb+0x55)[0x81cee95]
/usr/libexec/mysqld(_ZN18Prepared_statement12cleanup_stmtEv+0x49)[0x8202c39]
/usr/libexec/mysqld(_ZN18Prepared_statement7prepareEPKcj+0x1c8)[0x8202e98]
/usr/libexec/mysqld(_Z18mysql_stmt_prepareP3THDPKcj+0xcb)[0x8204e2b]
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1069)[0x81ac439]
/usr/libexec/mysqld(handle_one_connection+0xa78)[0x81ad828]
/lib/libpthread.so.0[0x39f49b]
/lib/libc.so.6(clone+0x5e)[0x2f642e]
======= Memory map: ========
00179000-00182000 r-xp 00000000 fd:00 53478649   /lib/libcrypt-2.5.so
00182000-00183000 r--p 00008000 fd:00 53478649   /lib/libcrypt-2.5.so
00183000-00184000 rw-p 00009000 fd:00 53478649   /lib/libcrypt-2.5.so
00184000-001ab000 rw-p 00184000 00:00 0 
....
...
101102 13:17:11 - mysqld got signal 11 ;
....
....
...
101102 15:11:33  InnoDB: Error: MySQL is freeing a thd
InnoDB: though trx->n_mysql_tables_in_use is 1
InnoDB: and trx->mysql_n_tables_locked is 7670.
TRANSACTION 0 602727896, not started, process no 2261, OS thread id 1629776784
mysql tables in use 1, locked 7670
MySQL thread id 5309, query id 113616026 localhost mv_php_usr
 len 860; hex ....

*** glibc detected *** /usr/libexec/mysqld: free(): invalid pointer: 0x60d1cc18 ***
======= Backtrace: =========
/lib/libc.so.6[0x28e0f1]
/lib/libc.so.6(cfree+0x90)[0x291bc0]
/usr/libexec/mysqld[0x8212d86]
/usr/libexec/mysqld(_Z19ha_close_connectionP3THD+0x2b)[0x824b87b]
/usr/libexec/mysqld(_ZN3THDD0Ev+0x80)[0x81812f0]
/usr/libexec/mysqld(_Z10end_threadP3THDb+0x3e)[0x818bcae]
/usr/libexec/mysqld(handle_one_connection+0x403)[0x81ad1b3]
/lib/libpthread.so.0[0x39f49b]
/lib/libc.so.6(clone+0x5e)[0x2f642e]
======= Memory map: ========
00110000-00119000 r-xp 00000000 fd:00 53477416   /lib/libnss_files-2.5.so
00119000-0011a000 r--p 00008000 fd:00 53477416   /lib/libnss_files-2.5.so
0011a000-0011b000 rw-p 00009000 fd:00 53477416   /lib/libnss_files-2.5.so
00179000-00182000 r-xp 00000000 fd:00 53478649   /lib/libcrypt-2.5.so
00182000-00183000 r--p 00008000 fd:00 53478649   /lib/libcrypt-2.5.so
00183000-00184000 rw-p 00009000 fd:00 53478649   /lib/libcrypt-2.5.so
00184000-001ab000 rw-p 00184000 00:00 0 

-----------------------
my.cnf 
-----------------------
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
skip_name_resolve
#bind-address=192.168.1.3

#General Querieis
log=/var/log/mysql_log_queries.log
#Slow Queries
log_slow_queries=/var/log/mysql_slow_queries.log
long_query_time=2
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1

# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links=0

#MyIsam Variables
myisam_sort_buffer_size=64M
#MySql Variables
key_buffer_size=128M
max_allowed_packet=1M
table_cache=1024
sort_buffer_size=1M
read_buffer_size=1M
read_rnd_buffer_size=4M
thread_cache=16
flush
delay_key_write=Off
#thread_concuency=4

#InnoDB Settings
innodb_buffer_pool_size = 1G
innodb_additional_mem_pool_size = 64M
innodb_log_file_size=256M
innodb_log_buffer_size = 32M
innodb_flush_log_at_trx_commit = 1
innodb_thread_concurrency=8
innodb_file_per_table
#innodb_force_recovery=1

#Query Cache
query_cache_size=128M
query_cache_type=1
#query_cache_size=0
#query_cache_type=0

max_connections=300

#Replication
log-bin=/var/log/mysql/mysql-bin.log
binlog-do-db=mv_sys_data
server-id=1
sync_binlog=1

[mysqldump]
quick
max_allowed_packet=16M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

如何重复:可能是因为我的硬件或配置或其他原因。只需启动服务器并让其工作一段时间,它就会重新启动。

答案1

在服务器上运行内存测试(通常可以从 grub 执行)。尝试通过编译内核或实际上并行编译多个内核进行压力测试。

您的内存是 ECC 内存吗?如果不是,那您就应该有!:-)

还要检查 mysql 已知的错误是否与您的硬件不兼容。

这听起来像是硬件故障而不是软件故障,但是可能是您在使用 mysql 时遇到了一些限制,请确保您运行的是最新版本,并且安装了与您的硬件相关的所有补丁。

http://www.linuxquestions.org/questions/slackware-14/stress-testing-my-cpu-237025/

还有更多信息。

再次,如果您尝试对内核进行编译以进行压力测试,请将 /usr/src/linux.xxx 复制到另一个目录并并行运行另一个编译。

在我的四核系统上,我可以运行 32 个编译。或者,您可以增加 -j 参数来增加 make 的解析度,但我发现运行完全独立的编译会更快地发现问题。

相关内容