我们有一个包含 10 个节点的 Cassandra 集群，其中四个节点显示了 所报告的一些磁盘错误dmesg。这些报告或多或少每小时都会出现一次。
错误中报告的磁盘是 8TB 逻辑卷的一半 - 另一个磁盘似乎没有报告错误。
[Wed Sep 28 12:29:02 2022] sas: Enter sas_scsi_recover_host busy: 1 failed: 1
[Wed Sep 28 12:29:02 2022] sas: ata7: end_device-0:0: cmd error handler
[Wed Sep 28 12:29:02 2022] sas: ata7: end_device-0:0: dev error handler
[Wed Sep 28 12:29:02 2022] ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Wed Sep 28 12:29:02 2022] sas: ata8: end_device-0:1: dev error handler
[Wed Sep 28 12:29:02 2022] ata7.00: failed command: READ DMA EXT
[Wed Sep 28 12:29:02 2022] ata7.00: cmd 25/00:08:10:d8:5d/00:00:84:01:00/e0 tag 8 dma 4096 in
res 51/40:00:10:d8:5d/00:00:84:01:00/e0 Emask 0x9 (media error)
[Wed Sep 28 12:29:02 2022] ata7.00: status: { DRDY ERR }
[Wed Sep 28 12:29:02 2022] ata7.00: error: { UNC }
[Wed Sep 28 12:29:02 2022] ata7.00: supports DRM functions and may not be fully accessible
[Wed Sep 28 12:29:02 2022] ata7.00: supports DRM functions and may not be fully accessible
[Wed Sep 28 12:29:02 2022] ata7.00: configured for UDMA/133
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] CDB: Read(16) 88 00 00 00 00 01 84 5d d8 10 00 00 00 08 00 00
[Wed Sep 28 12:29:02 2022] blk_update_request: I/O error, dev sda, sector 6515709968
[Wed Sep 28 12:29:02 2022] ata7: EH complete
[Wed Sep 28 12:29:02 2022] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1
[Wed Sep 28 12:29:02 2022] ata7.00: Enabling discard_zeroes_data
Aniostat -xdct 1 100显示了非常合理的 R/W 速率和平均队列大小的数字：
grep "^sda" iostat-xdct | awk '{n+=1; r+=$6; w+=$7; qsz+=$9; qmx=qmx<$9?$9:qmx; qmi=qmi>$9?$9:qmi}END{printf("%.3f\t%.3f\t%.3f\t\t[%.3f , %.3f]\n",r/n,w/n,qsz/n,qmi,qmx)}'
818.239 6.939 0.038 [0.000 , 0.890]
根据我们所有常用的指标，系统运行良好。但是，我担心这可能会在最意想不到的时间导致致命的系统错误。
我不知道报告的错误意味着什么。我更不知道如何评估保持系统这样运行所涉及的风险。我应该立即更换磁盘吗？
我将在此处粘贴 smartctl -a /dev/sda 的输出：
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-3.10.0-693.21.1.el7.x86_64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: Samsung SSD 870 EVO 4TB
Serial Number: S6BBNJ0R700321V
LU WWN Device Id: 5 002538 f31706f50
Firmware Version: SVT01B6Q
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: Unknown(0x09fc), ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is: SATA >3.2 (0x1ff), 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Sep 28 12:58:46 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x53) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 320) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 068 068 010 Pre-fail Always - 1510
9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 9429
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 6
177 Wear_Leveling_Count 0x0013 099 099 000 Pre-fail Always - 11
179 Used_Rsvd_Blk_Cnt_Tot 0x0013 068 068 010 Pre-fail Always - 1510
181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always - 0
182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always - 0
183 Runtime_Bad_Block 0x0013 068 068 010 Pre-fail Always - 1510
187 Reported_Uncorrect 0x0032 099 099 000 Old_age Always - 8242
190 Airflow_Temperature_Cel 0x0032 068 056 000 Old_age Always - 32
195 Hardware_ECC_Recovered 0x001a 199 199 000 Old_age Always - 8242
199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
235 Unknown_Attribute 0x0012 099 099 000 Old_age Always - 4
241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 35415578583
SMART Error Log Version: 1
ATA Error Count: 8242 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 8242 occurred at disk power-on lifetime: 9429 hours (392 days + 21 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 10 d8 5d e0 Error:
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
Error 8241 occurred at disk power-on lifetime: 9428 hours (392 days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 e8 d7 5d e0 Error:
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
Error 8240 occurred at disk power-on lifetime: 9427 hours (392 days + 19 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 e8 d7 5d e0 Error:
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
Error 8239 occurred at disk power-on lifetime: 9427 hours (392 days + 19 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 e8 d7 5d e0 Error:
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
Error 8238 occurred at disk power-on lifetime: 9426 hours (392 days + 18 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 e8 d7 5d e0 Error:
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
256 0 65535 Read_scanning was never started
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
答案1
这是一个非常糟糕的磁盘。看这里：
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] CDB: Read(16) 88 00 00 00 00 01 84 5d d8 10 00 00 00 08 00 00
[Wed Sep 28 12:29:02 2022] blk_update_request: I/O error, dev sda, sector 6515709968
这告诉您操作系统无法从磁盘读取数据。
看这里：
SMART Attributes Data Structure revision number: 1
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 068 068 010 Pre-fail Always - 1510
9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 9429
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 6
该磁盘已经运行了大约 13 个月（假设 24x7，这在仅记录了 6 个电源周期的情况下是合理的）。不幸的是，1510 个重新分配的扇区比我满意的多了大约 1505 个扇区。
在这里我们可以看到有大量失败的磁盘读取：
183 Runtime_Bad_Block 0x0013 068 068 010 Pre-fail Always - 1510
187 Reported_Uncorrect 0x0032 099 099 000 Old_age Always - 8242
195 Hardware_ECC_Recovered 0x001a 199 199 000 Old_age Always - 8242
您已经写入了相当数量的块（可能是 131 TB），但没有什么特别的（SSD 在其生命周期内有保证的上限写入期望，而您的是 2400 TB），当然也没有不成比例：
241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 35415578583
我有点担心操作系统将磁盘视为具有 512B 扇区；它们应该是 4kB 扇区/块。可能值得使用诸如fdisk或 之类的工具检查一下，parted看看这是否是smartctl.
总之，我想说是时候换一张新磁盘了。鉴于它在 13 个月后就报废了，我本以为它有 12 个月的保修期，但 EVO 似乎有五年。
OP 在评论中问道：“尽管存在硬件问题（例如注销扇区），这种重新定位是否是磁盘继续工作的一种方式？为什么iostat还是显示一切正常？”。
如果无法读取磁盘扇区/块，您将在操作系统级别收到错误。通常，当尝试读取（编辑、复制、备份）文件时，最终会出现 IO 错误。磁盘会将扇区/块标记为不可读，并且不可纠正的计数将会增加。当操作系统（或应用程序）将数据写入磁盘上的该扇区/块时，固件将利用机会将该扇区/块重新映射到为此目的而保留的备用扇区/块之一。不可纠正的计数可能会减少，而重新分配的计数将会增加。
到目前为止，磁盘固件已成功重新分配 1510 个扇区/块，并且大多数 IO 都很好（这就是为什么iostat看起来没问题）。每隔一段时间，您就会遇到另一个不可读的扇区/块 - 如图所示dmesg，并且除非您尝试写入它，否则这种情况不会消失。磁盘迟早会耗尽其保留的扇区/块，并且您会遇到一系列写入错误，然后是文件系统损坏。这确实不是您想要的地方，所以我建议您尝试更换保修期内的磁盘。
不管，确保您拥有并继续保留良好的备份。