На двух машинах возникает повторяющаяся проблема, когда корневая файловая система заполняется на 100% из-за /var/log/messages заполнение миллионами сообщений, как показано ниже.
У нас есть много других машин с идентичной конфигурацией аппаратного и программного обеспечения, но только два демонстрируют эту проблему. Конечно, когда корневая файловая система заполнена на 100%, все идет не так хорошо.
Я искал причины, но ничего не вышло. Есть идеи или предложения? Никогда не встречал этой проблемы ни на каких других машинах ...
РЕДАКТИРОВАТЬ: Добавлен вывод mount команда ниже. (сообщения ядра не говорят какое устройство вызывает проблему ...
РЕДАКТИРОВАТЬ II: Копаясь в миллионах строк /var/log/messages Я обнаружил несколько странных сообщений ядра, касающихся памяти. Не знаю, откуда они, поскольку многие не называют себя, просто "(UME)" или "160 * 4096 КБ" и т. Д.
Увидеть ниже.
Centos 6.6 32-bit
# uname -a
Linux 4.4.127-1.el6.elrepo.i686 #1 SMP Sun Apr 8 09:44:43 EDT 2018 i686 i686 i386 GNU/Linux
/ var / log / содержание сообщений
Вот некоторые из /var/log/messages:
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483134
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483166
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483198
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483230
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483262
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483294
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483326
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483358
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483390
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483422
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483454
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483486
Dec 8 14:43:12 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483518
И несколько других:
Dec 15 05:35:51 localhost kernel: vmscan: shrink_slab: super_cache_scan+0x0/0x1b0 negative objects to delete nr=-2147436183
Dec 15 05:35:51 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147482917
Dec 15 05:35:51 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147482429
Dec 15 05:35:51 localhost kernel: vmscan: shrink_slab: ext4_es_scan+0x0/0x130 [ext4] negative objects to delete nr=-2147483561
Dec 15 05:35:51 localhost kernel: vmscan: shrink_slab: super_cache_scan+0x0/0x1b0 negative objects to delete nr=-2147479632
Установленные тома:
# mount
/dev/mapper/vgTorque-lvRoot on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext3 (rw)
/dev/mapper/vgTorque-lvHome on /home type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
/dev/sdb1 on /mnt/das.b type ext4 (rw,nosuid,nodev,noatime)
/dev/sdc1 on /mnt/das.c type ext4 (rw,nosuid,nodev,noatime)
/dev/sdd1 on /mnt/das.d type ext4 (rw,nosuid,nodev,noatime)
/dev/sde1 on /mnt/das.e type ext4 (rw,nosuid,nodev,noatime)
/dev/sdf1 on /mnt/das.f type ext4 (rw,nosuid,nodev,noatime)
/dev/sdg1 on /mnt/das.g type ext4 (rw,nosuid,nodev,noatime)
/dev/sdh1 on /mnt/das.h type ext4 (rw,nosuid,nodev,noatime)
/dev/sdi1 on /mnt/das.i type ext4 (rw,nosuid,nodev,noatime)
/dev/sdj1 on /mnt/das.j type ext4 (rw,nosuid,nodev,noatime)
/dev/sdk1 on /mnt/das.k type ext4 (rw,nosuid,nodev,noatime)
/dev/sdl1 on /mnt/das.l type ext4 (rw,nosuid,nodev,noatime)
Сообщения, связанные с памятью (?)
Dec 16 20:16:47 localhost kernel: (UME)
Dec 16 20:16:47 localhost kernel: (UME)
Dec 16 20:16:47 localhost kernel: 15*64kB
Dec 16 20:16:47 localhost kernel: 2*2048kB
Dec 16 20:16:47 localhost kernel: (UE)
Dec 16 20:16:47 localhost kernel: (UM)
Dec 16 20:16:47 localhost kernel: 6*128kB
Dec 16 20:16:47 localhost kernel: 160*4096kB
Dec 16 20:16:47 localhost kernel: (UE)
Dec 16 20:16:47 localhost kernel: (UMH)
Dec 16 20:16:47 localhost kernel: 1*256kB
Dec 16 20:16:47 localhost kernel: = 668348kB
Dec 16 20:16:47 localhost kernel: (U)
Dec 16 20:16:47 localhost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 16 20:16:47 localhost kernel: 0*512kB
Dec 16 20:16:47 localhost kernel: 386803 total pagecache pages
Dec 16 20:16:47 localhost kernel: 0*1024kB
Dec 16 20:16:47 localhost kernel: 348 pages in swap cache
Dec 16 20:16:47 localhost kernel: 0*2048kB
Dec 16 20:16:47 localhost kernel: Swap cache stats: add 733208, delete 732860, find 59216733/59365389
Dec 16 20:16:47 localhost kernel: 0*4096kB
Dec 16 20:16:47 localhost kernel: Free swap = 3788784kB
Dec 16 20:16:47 localhost kernel: = 3812kB
Dec 16 20:16:47 localhost kernel: Total swap = 3833852kB
Dec 16 20:16:47 localhost kernel: Node 0
Dec 16 20:16:47 localhost kernel: 966443 pages RAM
Dec 16 20:16:47 localhost kernel: Normal:
Dec 16 20:16:47 localhost kernel: 738704 pages HighMem/MovableOnly
Dec 16 20:16:47 localhost kernel: 3200*4kB
Dec 16 20:16:47 localhost kernel: 13635 pages reserved
Dec 16 20:16:47 localhost kernel: (UME) 1414*8kB (UME) 729*16kB (UME) 486*32kB (UME) 241*64kB (UME) 117*128kB (UME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 81728kB
Dec 16 20:16:47 localhost kernel: Node 0 HighMem: 22*4kB (E) 29*8kB (UME) 59*16kB (UME) 42*32kB (UME) 12*64kB (UME) 7*128kB (ME) 2*256kB (UM) 2*512kB (ME) 3*1024kB (UME) 2*2048kB (UM) 160*4096kB (UMH) = 668336kB
Dec 16 20:16:47 localhost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 16 20:16:47 localhost kernel: 386803 total pagecache pages
Dec 16 20:16:47 localhost kernel: 348 pages in swap cache
Dec 16 20:16:47 localhost kernel: Swap cache stats: add 733208, delete 732860, find 59216733/59365389
Dec 16 20:16:47 localhost kernel: Free swap = 3788784kB
Dec 16 20:16:47 localhost kernel: Total swap = 3833852kB
Dec 16 20:16:47 localhost kernel: 966443 pages RAM
Dec 16 20:16:47 localhost kernel: 738704 pages HighMem/MovableOnly
Dec 16 20:16:47 localhost kernel: 13635 pages reserved
Dec 16 20:16:47 localhost kernel: 486*32kB (UME) 241*64kB (UME) 117*128kB (UME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 81736kB
Dec 16 20:16:47 localhost kernel: Node 0 HighMem: 23*4kB (ME) 27*8kB (UME) 68*16kB (UME) 55*32kB (UME) 15*64kB (UME) 7*128kB (ME) 2*256kB (UM) 2*512kB (ME) 3*1024kB (UME) 2*2048kB (UM) 160*4096kB (UMH) = 669076kB
Dec 16 20:16:47 localhost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 16 20:16:47 localhost kernel: 386803 total pagecache pages
Dec 16 20:16:47 localhost kernel: 348 pages in swap cache
Dec 16 20:16:47 localhost kernel: Swap cache stats: add 733208, delete 732860, find 59216733/59365389
Dec 16 20:16:47 localhost kernel: Free swap = 3788784kB
Dec 16 20:16:47 localhost kernel: Total swap = 3833852kB
Dec 16 20:16:47 localhost kernel: 966443 pages RAM
Dec 16 20:16:47 localhost kernel: 738704 pages HighMem/MovableOnly
Dec 16 20:16:47 localhost kernel: 13635 pages reserved
Dec 16 20:16:47 localhost kernel: vmap allocation for size 266240 failed: use vmalloc=<size> to increase size.