☰

lvchange -ay 在 lvmcached 逻辑卷内存泄漏 Ubuntu 22.04

tag-icon • tag-icon lvm 22.04 memory-leak caching lvmcache
lvchange -ay 在 lvmcached 逻辑卷内存泄漏 Ubuntu 22.04

Ubuntu 22.04，我使用 lvmcache（类型 writecache）配置了 SSD 缓存 LV，如下所示lvmcache 手册页。

系统启动正常，但缓存的 LV 状态始终不可用。

  LV Path                /dev/vg_fast3/disk3_lv
  LV Name                disk3_lv
  VG Name                vg_fast3
  LV UUID                UdAPKL-ZQQq-1ytp-CI54-FwVS-4iz5-EAJ0l0
  LV Write Access        read/write
  LV Creation host, time minio1, 2022-05-07 18:13:22 +0200
  LV Status              NOT available
  LV Size                <9.10 TiB
  Current LE             2384383
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

然后我尝试手动激活 LV（我有 3 个）。第一次激活需要一点时间，并且不会消耗任何实际的系统内存，但第二次激活需要更长时间，并且会消耗所有主机内存（32GB）。这是一个新系统，RAM 使用率相当低。当我尝试激活第三个 LV 时，OOM 杀手开始行动。

请查看当我激活第二个 LV 时系统的内存使用情况如何变化。

root@minio1 /etc/systemd/system # free
               total        used        free      shared  buff/cache   available
Mem:        32602580     3594536    28791428        1272      216616    28629324
Swap:        4194300           0     4194300
root@minio1 /etc/systemd/system # lvchange -ay -vvvv /dev/vg_fast2/disk2_lv
root@minio1 /etc/systemd/system # free
               total        used        free      shared  buff/cache   available
Mem:        32602580    29741396     2644252        1284      216932     2482304
Swap:        4194300           0     4194300

如您所见，大部分时间（2 分钟）都花在了 RESUME 任务上。

12:08:48.119176 lvchange[1154] device_mapper/libdm-common.c:2677  Udev cookie 0xd4d1f00 (semid 1) assigned to RESUME task(5) with flags DISABLE_LIBRARY_FALLBACK         (0x20)
12:08:48.119186 lvchange[1154] device_mapper/ioctl/libdm-iface.c:1876  dm resume   (253:5) [ noopencount flush ]   [16384] (*1)
12:11:38.391470 lvchange[1154] device_mapper/libdm-common.c:1484  vg_fast2-disk2_lv: Stacking NODE_ADD (253,5) 0:6 0660 [trust_udev]
12:11:38.391498 lvchange[1154] device_mapper/libdm-common.c:1495  vg_fast2-disk2_lv: Stacking NODE_READ_AHEAD 256 (flags=1)
12:11:38.391514 lvchange[1154] activate/dev_manager.c:3686  Creating CLEAN tree for vg_fast2/disk2_lv.
12:11:38.391551 lvchange[1154] activate/dev_manager.c:817  Getting device info for vg_fast2-disk2_lv [LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqieWtt9lFS16fwbF1XWqeoZhb7zyj3vDzc].
12:11:38.391576 lvchange[1154] device_mapper/ioctl/libdm-iface.c:1876  dm info  LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqieWtt9lFS16fwbF1XWqeoZhb7zyj3vDzc [ opencount flush ]   [16384] (*1)
12:11:38.391609 lvchange[1154] device_mapper/ioctl/libdm-iface.c:1876  dm deps   (253:5) [ opencount flush ]   [16384] (*1)
12:11:38.391655 lvchange[1154] device_mapper/ioctl/libdm-iface.c:1876  dm deps   (253:3) [ opencount flush ]   [16384] (*1)
12:11:38.391675 lvchange[1154] device_mapper/ioctl/libdm-iface.c:1876  dm deps   (253:4) [ opencount flush ]   [16384] (*1)
12:11:38.391703 lvchange[1154] activate/dev_manager.c:817  Getting device info for vg_fast2-disk2_lv-real [LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqieWtt9lFS16fwbF1XWqeoZhb7zyj3vDzc-real].
12:11:38.391714 lvchange[1154] device_mapper/ioctl/libdm-iface.c:1876  dm info  LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqieWtt9lFS16fwbF1XWqeoZhb7zyj3vDzc-real [ opencount flush ]   [16384] (*1)
12:11:38.391728 lvchange[1154] activate/dev_manager.c:817  Getting device info for vg_fast2-disk2_lv-cow [LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqieWtt9lFS16fwbF1XWqeoZhb7zyj3vDzc-cow].
12:11:38.391737 lvchange[1154] device_mapper/ioctl/libdm-iface.c:1876  dm info  LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqieWtt9lFS16fwbF1XWqeoZhb7zyj3vDzc-cow [ opencount flush ]   [16384] (*1)
12:11:38.391750 lvchange[1154] activate/dev_manager.c:817  Getting device info for vg_fast2-ssd2_lv_cvol-real [LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqiZmOf0I2TWMotwHgUKQs49RxDYIbrB8oq-real].
12:11:38.391760 lvchange[1154] device_mapper/ioctl/libdm-iface.c:1876  dm info  LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqiZmOf0I2TWMotwHgUKQs49RxDYIbrB8oq-real [ opencount flush ]   [16384] (*1)
12:11:38.391775 lvchange[1154] activate/dev_manager.c:817  Getting device info for vg_fast2-disk2_lv_wcorig [LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqiq94eYXDdC8TGT1RVERIDZQNnf5iaJFx9-real].
12:11:38.391785 lvchange[1154] device_mapper/ioctl/libdm-iface.c:1876  dm info  LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqiq94eYXDdC8TGT1RVERIDZQNnf5iaJFx9-real [ opencount flush ]   [16384] (*1)
12:11:38.391798 lvchange[1154] activate/dev_manager.c:817  Getting device info for vg_fast2-disk2_lv_wcorig-real [LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqiq94eYXDdC8TGT1RVERIDZQNnf5iaJFx9-real].
12:11:38.391807 lvchange[1154] device_mapper/ioctl/libdm-iface.c:1876  dm info  LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqiq94eYXDdC8TGT1RVERIDZQNnf5iaJFx9-real [ opencount flush ]   [16384] (*1)
12:11:38.391820 lvchange[1154] activate/dev_manager.c:817  Getting device info for vg_fast2-disk2_lv_wcorig-cow [LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqiq94eYXDdC8TGT1RVERIDZQNnf5iaJFx9-cow].
12:11:38.391828 lvchange[1154] device_mapper/ioctl/libdm-iface.c:1876  dm info  LVM-Wm7lKJTqlwERGip92OXeIgBi5nEqzsqiq94eYXDdC8TGT1RVERIDZQNnf5iaJFx9-cow [ opencount flush ]   [16384] (*1)
12:11:38.391844 lvchange[1154] mm/memlock.c:641  Leaving section (activated).
12:11:38.391856 lvchange[1154] mm/memlock.c:597  Unlock: Memlock counters: prioritized:1 locked:0 critical:0 daemon:0 suspended:0
12:11:38.391867 lvchange[1154] mm/memlock.c:506  Restoring original task priority 0.
12:11:38.391874 lvchange[1154] activate/fs.c:492  Syncing device names
12:11:38.391888 lvchange[1154] device_mapper/libdm-common.c:2479  Udev cookie 0xd4d1f00 (semid 1) decremented to 1
12:11:38.391896 lvchange[1154] device_mapper/libdm-common.c:2765  Udev cookie 0xd4d1f00 (semid 1) waiting for zero
12:11:38.603440 lvchange[1154] device_mapper/libdm-common.c:2494  Udev cookie 0xd4d1f00 (semid 1) destroyed
12:11:38.603459 lvchange[1154] device_mapper/libdm-common.c:1484  vg_fast2-ssd2_lv_cvol: Skipping NODE_ADD (253,3) 0:6 0660 [trust_udev]
12:11:38.603466 lvchange[1154] device_mapper/libdm-common.c:1495  vg_fast2-ssd2_lv_cvol: Processing NODE_READ_AHEAD 256 (flags=1)
12:11:38.603498 lvchange[1154] device_mapper/libdm-common.c:1249  vg_fast2-ssd2_lv_cvol (253:3): read ahead is 256
12:11:38.603506 lvchange[1154] device_mapper/libdm-common.c:1373  vg_fast2-ssd2_lv_cvol: retaining kernel read ahead of 256 (requested 256)
12:11:38.603510 lvchange[1154] device_mapper/libdm-common.c:1484  vg_fast2-disk2_lv_wcorig: Skipping NODE_ADD (253,4) 0:6 0660 [trust_udev]
12:11:38.603515 lvchange[1154] device_mapper/libdm-common.c:1495  vg_fast2-disk2_lv_wcorig: Processing NODE_READ_AHEAD 256 (flags=1)
12:11:38.603531 lvchange[1154] device_mapper/libdm-common.c:1249  vg_fast2-disk2_lv_wcorig (253:4): read ahead is 256
12:11:38.603537 lvchange[1154] device_mapper/libdm-common.c:1373  vg_fast2-disk2_lv_wcorig: retaining kernel read ahead of 256 (requested 256)
12:11:38.603542 lvchange[1154] device_mapper/libdm-common.c:1484  vg_fast2-disk2_lv: Skipping NODE_ADD (253,5) 0:6 0660 [trust_udev]
12:11:38.603547 lvchange[1154] device_mapper/libdm-common.c:1495  vg_fast2-disk2_lv: Processing NODE_READ_AHEAD 256 (flags=1)
12:11:38.603561 lvchange[1154] device_mapper/libdm-common.c:1249  vg_fast2-disk2_lv (253:5): read ahead is 256
12:11:38.603567 lvchange[1154] device_mapper/libdm-common.c:1373  vg_fast2-disk2_lv: retaining kernel read ahead of 256 (requested 256)
12:11:38.603576 lvchange[1154] misc/lvm-flock.c:84  Unlocking /run/lock/lvm/V_vg_fast2
12:11:38.603583 lvchange[1154] misc/lvm-flock.c:47  _undo_flock /run/lock/lvm/V_vg_fast2
12:11:38.603604 lvchange[1154] metadata/vg.c:79  Freeing VG vg_fast2 at 0x55a132b4b5a0.
12:11:38.603754 lvchange[1154] notify/lvmnotify.c:54  Nofify dbus at com.redhat.lvmdbus1.
12:11:38.604168 lvchange[1154] notify/lvmnotify.c:69  D-Bus notification failed: The name com.redhat.lvmdbus1 was not provided by any .service files
12:11:38.604210 lvchange[1154] cache/lvmcache.c:2091  Destroy lvmcache content
12:11:38.649830 lvchange[1154] lvmcmdline.c:3168  Completed: lvchange -ay -vvvv /dev/vg_fast2/disk2_lv

我想创建一个 bug，但我决定先在这里交换一些想法。

提前感谢您的反馈。

答案1

我又做了几个测试，发现当缓存的 LV 被激活时，内存会被使用。使 LV 处于非活动状态会恢复丢失的内存。

  --- Logical volume ---
  LV Path                /dev/vg_fast3/disk3_lv
  LV Name                disk3_lv
  VG Name                vg_fast3
  LV UUID                DiqET1-VMau-4zuc-4vVt-4WDk-yfeJ-KORpwq
  LV Write Access        read/write
  LV Creation host, time minio2, 2022-05-08 13:36:15 +0200
  LV Status              available
  # open                 0
  LV Size                <9.10 TiB
  Current LE             2384383
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Path                /dev/vg_fast2/disk2_lv
  LV Name                disk2_lv
  VG Name                vg_fast2
  LV UUID                H0FAlm-1KOe-fLeL-EsgX-iupN-JsYl-zsThtM
  LV Write Access        read/write
  LV Creation host, time minio2, 2022-05-07 19:03:33 +0200
  LV Status              NOT available
  LV Size                <9.10 TiB
  Current LE             2384383
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

  --- Logical volume ---
  LV Path                /dev/vg_fast1/disk1_lv
  LV Name                disk1_lv
  VG Name                vg_fast1
  LV UUID                yo32U4-IfzJ-W0sj-JTKn-Qj4B-syJd-zgu9qK
  LV Write Access        read/write
  LV Creation host, time minio2, 2022-05-07 18:58:01 +0200
  LV Status              NOT available
  LV Size                <9.10 TiB
  Current LE             2384383
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

root@minio2 ~ # free
               total        used        free      shared  buff/cache   available
Mem:        32600636    26401620     5984508        1284      214508     5821340
Swap:        4194300           0     4194300
root@minio2 ~ # lvchange -an /dev/vg_fast3/disk3_lv
root@minio2 ~ # free
               total        used        free      shared  buff/cache   available
Mem:        32600636      291212    32095032        1272      214392    31931824
Swap:        4194300           0     4194300

我使用 160GB 作为写入缓存，也许这太多了。

答案2

好的，看起来内存需求会根据缓存 lv 大小而变化，几百 GB 占用了我所有的服务器内存。我不确定我是否遗漏了什么，但我决定使用 bcache，它似乎不太耗内存。

现在一切都按预期工作，没有任何明显的内存消耗。

谢谢。

相关内容