我的服务器死机了。我重新启动了服务器，在手动运行 fsck 后，服务正常重新启动。我在分析故障发生前后的 /var/log/messages，发现了以下条目：
Nov 29 22:24:27 cloud kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x4090000 action 0xe frozen
Nov 29 22:24:27 cloud kernel: ata1: irq_stat 0x00400040, connection status changed
Nov 29 22:24:27 cloud kernel: ata1: SError: { PHYRdyChg 10B8B DevExch }
Nov 29 22:24:27 cloud kernel: ata1: hard resetting link
Nov 29 22:24:34 cloud kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Nov 29 22:24:34 cloud kernel: ata1.00: configured for UDMA/133
Nov 29 22:24:34 cloud kernel: ata1: EH complete
这可能表明硬盘有故障，但我对硬盘进行了 SMART 长测试，输出如下：
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-573.7.1.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: Seagate Constellation ES (SATA 6Gb/s)
Device Model: ST1000NM0011
Serial Number: Z1N2D4JT
LU WWN Device Id: 5 000c50 04e008fb2
Firmware Version: PA07
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Mon Nov 30 15:29:22 2015 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 609) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 151) minutes.
Conveyance self-test routine
recommended polling time: ( 3) minutes.
SCT capabilities: (0x10bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 069 063 044 Pre-fail Always - 8118850
3 Spin_Up_Time 0x0003 095 095 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 15
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 084 060 030 Pre-fail Always - 4627909443
9 Power_On_Hours 0x0032 073 073 000 Old_age Always - 24302
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 21
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 075 064 045 Old_age Always - 25 (Min/Max 23/25)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 27
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 29
194 Temperature_Celsius 0x0022 025 040 000 Old_age Always - 25 (0 20 0 0 0)
195 Hardware_ECC_Recovered 0x001a 105 099 000 Old_age Always - 8118850
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 169642618276709
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 3860229105
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 1373595990
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 24302 -
# 2 Short offline Completed without error 00% 24299 -
# 3 Short offline Completed without error 00% 24298 -
# 4 Short offline Completed without error 00% 4 -
# 5 Extended offline Completed without error 00% 4 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
HDD 已于近期更换。内核版本为 2.6.32-573.7.1.el6.x86_64
答案1
正如 Andrey Sapegin 在上面的评论中指出的那样，这很可能是硬件。
目前我们无法确定是硬盘还是电缆出了问题 - 但很可能是物理故障。他们最近更换了硬盘，这一点可以忽略不计，因为他们没有更换新的硬盘。
他们说更换的硬盘之所以好，是因为
他们在新磁盘（已使用但经过测试）上克隆了旧磁盘，并确认新旧磁盘均 100% 正常，因为克隆过程中没有出现任何错误。
这个论点并不能说服我。过去一周，我的硬盘出现故障，无法写入，但读取性能非常好。所以“我们从旧硬盘上读取了所有数据“ 但对老人的健康状况却只字未提。”我们将所有数据写入新数据“在我看来，他们对替换硬盘的健康状况只字未提；如果这就是他们的全部测试方法，那么他们对替换硬盘的保证可能会被忽略。
谷歌的著名论文明确指出不能仅仅因为驱动器通过了smartctl健康检查就知道它是健康的（如果它失败了，你只能知道确实知道它病了）。
那么你该怎么办呢？你可以肯定你有问题，而且相当肯定问题出在硬件上。所以请做专业的事情：用散弹枪把电缆和驱动器都拆下来。硬盘和电缆很便宜，根本不值得您花费时间（或冒着数据风险）来追踪此错误的来源。购买新电缆和新硬盘，并让您的托管公司更换电缆和驱动器。
和永远不能重新部署没有RAID的服务器。