我被一个问题困扰了好几天,希望这里有人可以帮忙或提供一些提示。
问题如下:当我运行一些执行大量 I/O 操作的 docker 镜像时,容器在运行过程中变为只读。我的 docker 容器在 AWS EC2 上的 AWS ECS 中运行。我运行的镜像基于此处https://hub.docker.com/r/jenkinsci/jnlp-slave/。
我已将 docker 基本存储大小更新为 200G,当容器运行时,我可以看到它肯定有足够的空间来存储它生成的文件,因此看起来不像是磁盘空间问题。我也进行了一些谷歌搜索,似乎主机遇到了 XFS ENOSPC 错误 - 但我不知道这意味着什么以及如何解决它。
当我检查主机 EC2 实例中的 dmesg 日志时,我可以看到以下错误:
[Tue Oct 16 06:16:14 2018] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode
[Tue Oct 16 06:17:15 2018] device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode
[Tue Oct 16 06:17:15 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5588992)
[Tue Oct 16 06:17:15 2018] buffer_io_error: 585671 callbacks suppressed
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588992
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588993
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588994
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588995
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588996
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588997
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5588998
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5588999
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5589000
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5589001
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5589504)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5590016)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5590528)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5591040)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5591552)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5592064)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5592576)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 805306368 size 8388608 starting block 5593088)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 805306368 size 8388608 starting block 5593600)
[Tue Oct 16 06:17:16 2018] JBD2: Detected IO errors while flushing file data on dm-4-8
[Tue Oct 16 06:17:16 2018] JBD2: Detected IO errors while flushing file data on dm-4-8
[Tue Oct 16 06:17:16 2018] Aborting journal on device dm-4-8.
[Tue Oct 16 06:17:16 2018] EXT4-fs error (device dm-4): ext4_journal_check_start:61: Detected aborted journal
[Tue Oct 16 06:17:16 2018] EXT4-fs error (device dm-4): ext4_journal_check_start:61: Detected aborted journal
[Tue Oct 16 06:17:16 2018] EXT4-fs (dm-4): Remounting filesystem read-only
以下是 EC2 规范:
操作系统:Linux ip-10-119-5-115 4.14.59-64.43.amzn1.x86_64 #1 SMP 2018 年 8 月 2 日星期四 21:29:33 UTC x86_64 x86_64 x86_64 GNU/Linux
Docker 信息:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 18.06.1-ce
Storage Driver: devicemapper
Pool Name: docker-docker--pool
Pool Blocksize: 524.3kB
Base Device Size: 214.7GB
Backing Filesystem: ext4
Udev Sync Supported: true
Data Space Used: 802.2MB
Data Space Total: 23.33GB
Data Space Available: 22.53GB
Metadata Space Used: 606.2kB
Metadata Space Total: 25.17MB
Metadata Space Available: 24.56MB
Thin Pool Minimum Free Space: 2.333GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.59-64.43.amzn1.x86_64
Operating System: Amazon Linux AMI 2018.03
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.1GiB
Name: ip-10-119-5-115
ID: NJ4J:RBYM:3A25:QOZJ:HRSH:6BPD:3UO3:FW3X:4ZMI:5NWV:6IHV:OOHF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 58
Goroutines: 86
System Time: 2018-10-16T06:19:26.381414812Z
EventsListeners: 2
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Docker 版本:
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a215d7133c34aa18e3b72b4a21fd0c6136
Built: Tue Oct 2 18:06:45 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: e68fc7a/18.06.1-ce
Built: Tue Oct 2 18:08:26 2018
OS/Arch: linux/amd64
Experimental: false