"Inherited" maintenance of lab Ubuntu server 16.04 system with zfs filesystem. The zfs part is new to me. Getting I/O error with ls for a specific directory but everything looks clean. "zpool status" happy "smartctl -l selftest" happy on all devs. I would normally worry. What should I do next? Yes filesystem a little full. Weirdly not getting error in system log (dmesg) from I/O error. Maybe it's a zfs thing?
(base) root@tacitus:/home/obl# ls Phil
ls: cannot open directory 'Phil': Input/output error
(base) root@tacitus:/home/obl# uname -a
Linux tacitus 4.4.0-141-generic #167-Ubuntu SMP Wed Dec 5 10:40:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
(base) root@tacitus:/home/obl# df .
Filesystem 1K-blocks Used Available Use% Mounted on
data 29970265472 28567412096 1402853376 96% /home
(base) root@tacitus:/home/obl# zpool status
pool: data
state: ONLINE
scan: scrub repaired 0 in 39h18m with 0 errors on Mon Nov 9 15:42:45 2020
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-HGST_HDN728080ALE604_R6GS4HTY-part7 ONLINE 0 0 0
ata-HGST_HDN728080ALE604_R6GS1HYY-part7 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ata-HGST_HDN728080ALE604_R6GS1NBY-part1 ONLINE 0 0 0
ata-HGST_HDN728080ALE604_R6GRTJGY-part1 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdf ONLINE 0 0 0
errors: No known data errors
(base) root@tacitus:/home/obl# sudo smartctl -l selftest /dev/sda
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-141-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 22250 -
# 2 Short offline Completed without error 00% 21247 -
# 3 Short offline Completed without error 00% 147 -
(base) root@tacitus:/home/obl# sudo smartctl -l selftest /dev/sdb
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-141-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 22250 -
# 2 Short offline Completed without error 00% 21248 -
(base) root@tacitus:/home/obl# sudo smartctl -l selftest /dev/sdc
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-141-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 8311 -
# 2 Short offline Completed without error 00% 7308 -
(base) root@tacitus:/home/obl# sudo smartctl -l selftest /dev/sdd
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-141-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 22250 -
# 2 Short offline Completed without error 00% 21247 -
(base) root@tacitus:/home/obl# sudo smartctl -l selftest /dev/sde
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-141-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 22250 -
# 2 Short offline Completed without error 00% 21247 -
(base) root@tacitus:/home/obl# sudo smartctl -l selftest /dev/sdf
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-141-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 8311 -
# 2 Short offline Completed without error 00% 7308 -
(base) root@tacitus:/home/obl# dmesg | tail
[ 27.932402] audit: type=1400 audit(1605539563.228:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/ippusbxd" pid=2575 comm="apparmor_parser"
[ 27.938559] audit: type=1400 audit(1605539563.232:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/lightdm/lightdm-guest-session" pid=2564 comm="apparmor_parser"
[ 28.242146] IPv6: ADDRCONF(NETDEV_UP): enp4s0f0: link is not ready
[ 28.326252] cgroup: new mount options do not match the existing superblock, will be ignored
[ 30.742336] IPv6: ADDRCONF(NETDEV_UP): enp4s0f1: link is not ready
[ 30.842395] e1000e: enp4s0f0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[ 30.870143] IPv6: ADDRCONF(NETDEV_UP): enp4s0f1: link is not ready
[ 30.870205] IPv6: ADDRCONF(NETDEV_CHANGE): enp4s0f0: link becomes ready
[13121.872076] perf interrupt took too long (2513 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
[40206.668058] perf interrupt took too long (5006 > 5000), lowering kernel.perf_event_max_sample_rate to 25000