Неоднократно сталкивался с нехваткой памяти mysqld. Возникло множество процессов PHP, занимающих весь своп.
[562074.326710] mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 [562074.329646] mysqld cpuset=/ mems_allowed=0 [562074.330882] Pid: 21351, comm: mysqld Not tainted 2.6.32-642.1.1.el6.x86_64 #1 [562074.333171] Call Trace: [562074.334870] [] ? dump_header+0x90/0x1b0 [562074.336812] [] ? security_real_capable_noaudit+0x3c/0x70 [562074.338692] [] ? oom_kill_process+0x82/0x2a0 [562074.340270] [] ? select_bad_process+0xe1/0x120 [562074.341582] [] ? out_of_memory+0x220/0x3c0 [562074.343229] [] ? __alloc_pages_nodemask+0x93c/0x950 [562074.344930] [] ? ext4_get_block+0x0/0x120 [ext4] [562074.346436] [] ? alloc_pages_current+0xaa/0x110 [562074.347604] [] ? __page_cache_alloc+0x87/0x90 [562074.348889] [] ? find_get_page+0x1e/0xa0 [562074.350660] [] ? filemap_fault+0x1a7/0x500 [562074.352207] [] ? __do_fault+0x54/0x530 [562074.353872] [] ? handle_pte_fault+0xf7/0xb20 [562074.355966] [] ? rwsem_down_failed_common+0x95/0x1d0 [562074.357531] [] ? rwsem_down_read_failed+0x26/0x30 [562074.359503] [] ? handle_mm_fault+0x299/0x3d0 [562074.361105] [] ? call_rwsem_down_read_failed+0x14/0x30 [562074.362522] [] ? __do_page_fault+0x146/0x500 [562074.364590] [] ? lock_sock_nested+0xac/0xc0 [562074.387996] [] ? _spin_unlock_bh+0x1b/0x20 [562074.389187] [] ? pvclock_clocksource_read+0x58/0xd0 [562074.390457] [] ? kvm_clock_read+0x1c/0x20 [562074.391586] [] ? kvm_clock_get_cycles+0x9/0x10 [562074.392673] [] ? getnstimeofday+0x58/0xf0 [562074.394697] [] ? do_page_fault+0x3e/0xa0 [562074.395974] [] ? page_fault+0x25/0x30 [562074.397218] Mem-Info: [562074.399103] Node 0 DMA per-cpu: [562074.400605] CPU 0: hi: 0, btch: 1 usd: 0 [562074.402129] CPU 1: hi: 0, btch: 1 usd: 0 [562074.403506] Node 0 DMA32 per-cpu: [562074.405629] CPU 0: hi: 186, btch: 31 usd: 0 [562074.406749] CPU 1: hi: 186, btch: 31 usd: 30 [562074.407914] Node 0 Normal per-cpu: [562074.409325] CPU 0: hi: 186, btch: 31 usd: 0 [562074.410787] CPU 1: hi: 186, btch: 31 usd: 30 [562074.411941] active_anon:717855 inactive_anon:182000 isolated_anon:5083 [562074.411943] active_file:32 inactive_file:192 isolated_file:64 [562074.411944] unevictable:0 dirty:0 writeback:259 unstable:0 [562074.411944] free:21353 slab_reclaimable:5762 slab_unreclaimable:10906 [562074.411945] mapped:63 shmem:43 pagetables:21560 bounce:0 [562074.418218] Node 0 DMA free:15724kB min:248kB low:308kB high:372kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15312kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [562074.424382] lowmem_reserve[]: 0 3512 4017 4017 [562074.426602] Node 0 DMA32 free:61060kB min:58868kB low:73584kB high:88300kB active_anon:2667432kB inactive_anon:524160kB active_file:4kB inactive_file:396kB unevictable:0kB isolated(anon):15980kB isolated(file):384kB present:3596500kB mlocked:0kB dirty:0kB writeback:560kB mapped:232kB shmem:172kB slab_reclaimable:15184kB slab_unreclaimable:17264kB kernel_stack:6736kB pagetables:66680kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no [562074.432920] lowmem_reserve[]: 0 0 505 505 [562074.434773] Node 0 Normal free:8392kB min:8464kB low:10580kB high:12696kB active_anon:203604kB inactive_anon:203616kB active_file:124kB inactive_file:320kB unevictable:0kB isolated(anon):4992kB isolated(file):128kB present:517120kB mlocked:0kB dirty:0kB writeback:476kB mapped:20kB shmem:0kB slab_reclaimable:7864kB slab_unreclaimable:26360kB kernel_stack:6832kB pagetables:19560kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no [562074.441562] lowmem_reserve[]: 0 0 0 0 [562074.443251] Node 0 DMA: 3*4kB 0*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15724kB [562074.449972] Node 0 DMA32: 492*4kB 4530*8kB 511*16kB 140*32kB 89*64kB 3*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 60784kB [562074.455851] Node 0 Normal: 1455*4kB 68*8kB 3*16kB 2*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 8396kB [562074.461844] 22462 total pagecache pages [562074.463320] 21989 pages in swap cache [562074.464571] Swap cache stats: add 34726956, delete 34704967, find 25405257/28809084 [562074.466666] Free swap = 0kB [562074.467793] Total swap = 4194252kB [562074.508867] 1048575 pages RAM [562074.510066] 67960 pages reserved [562074.511918] 7658 pages shared [562074.513082] 947230 pages non-shared [562074.514178] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [562074.516236] [ 490] 0 490 2680 0 1 -17 -1000 udevd [562074.518561] [ 790] 0 790 2662 0 1 -17 -1000 udevd [562074.520752] [ 1342] 0 1342 62368 50 1 0 0 rsyslogd [562074.522842] [ 1357] 0 1357 4563 23 0 0 0 irqbalance [562074.525130] [ 1376] 25 1376 59678 129 0 0 0 named [562074.527174] [ 1394] 81 1394 5359 1 0 0 0 dbus-daemon [562074.529698] [ 1432] 0 1432 1020 0 0 0 0 acpid [562074.532021] [ 1451] 28 1451 336791 177 0 0 0 nscd [562074.534053] [ 1569] 0 1569 16086 24 1 -17 -1000 sshd [562074.536171] [ 1580] 38 1580 6262 27 1 0 0 ntpd [562074.539282] [ 2053] 0 2053 4662 1 0 0 0 dovecot [562074.542072] [ 2065] 401 2065 10639 1 0 0 0 pop3-login [562074.545036] [ 2066] 401 2066 10637 1 0 0 0 imap-login [562074.547647] [ 2067] 97 2067 3368 1 0 0 0 anvil [562074.550029] [ 2068] 0 2068 3401 1 1 0 0 log [562074.552190] [ 2070] 401 2070 10655 1 0 0 0 pop3-login [562074.554224] [ 2071] 401 2071 10670 1 0 0 0 imap-login [562074.556886] [ 2072] 0 2072 3659 1 0 0 0 config [562074.560112] [ 2082] 47 2082 18238 1 1 0 0 exim [562074.562160] [ 2100] 0 2100 49742 335 1 0 0 /usr/local/cpan [562074.564197] [ 2102] 0 2102 53119 223 0 0 0 spamd child [562074.566209] [ 2129] 0 2129 45352 1 0 0 0 abrtd [562074.568273] [ 2139] 0 2139 45197 1 0 0 0 abrt-dump-oops [562074.570489] [ 2150] 0 2150 19189 124 0 0 0 httpd [562074.572607] [ 2165] 0 2165 34191 0 0 0 0 pure-ftpd [562074.574719] [ 2167] 0 2167 34036 1 0 0 0 pure-authd [562074.577609] [ 2260] 0 2260 29271 27 1 0 0 crond [562074.580212] [ 2275] 0 2275 4857 5 1 0 0 atd [562074.582301] [ 2375] 0 2375 26952 125 0 0 0 cpsrvd (SSL) - [562074.584483] [ 2381] 0 2381 8172 69 0 0 0 queueprocd - wa [562074.586500] [ 2391] 0 2391 27506 53 0 0 0 dnsadmin - serv [562074.589509] [ 2419] 0 2419 20822 258 0 0 0 tailwatchd [562074.592045] [ 2433] 0 2433 19586 1 0 0 0 cPhulkd - proce [562074.594295] [ 2476] 0 2476 33028 60 0 0 0 cpdavd - accept [562074.597210] [ 2512] 0 2512 5915 102 1 0 0 cpanellogd - sl [562074.600503] [ 2540] 0 2540 1016 1 1 0 0 mingetty [562074.603689] [ 2542] 0 2542 1016 1 0 0 0 mingetty [562074.610302] [ 2545] 0 2545 1016 1 1 0 0 mingetty [562074.615154] [ 2547] 0 2547 2679 0 0 -17 -1000 udevd [562074.617989] [ 2548] 0 2548 1016 1 1 0 0 mingetty [562074.622920] [ 2550] 0 2550 1016 1 1 0 0 mingetty [562074.625853] [ 2552] 0 2552 1016 1 1 0 0 mingetty [562074.628530] [30499] 0 30499 147473 23 0 0 0 clamd [562074.631934] [ 8712] 0 8712 2394 1 1 0 0 mysqld_safe [562074.634865] [ 9408] 402 9408 564397 42943 0 0 0 mysqld [562074.637620] [24553] 0 24553 27283 1 1 0 0 sshd [562074.641062] [24562] 0 24562 13893 1 0 0 0 sftp-server [562074.643785] [25296] 0 25296 27283 1 1 0 0 sshd [562074.646513] [25312] 0 25312 26649 1 0 0 0 bash [562074.649571] [25954] 99 25954 19222 60 0 0 0 httpd [562074.652579] [25955] 0 25955 19061 1 0 0 0 leechprotect [562074.655407] [25958] 99 25958 334134 795 0 0 0 httpd [562074.657912] [25959] 99 25959 334193 806 1 0 0 httpd [562074.660551] [25960] 99 25960 334438 802 0 0 0 httpd [562074.663110] [26044] 99 26044 334439 803 0 0 0 httpd [562074.665618] [26206] 99 26206 334191 844 1 0 0 httpd [562074.668698] [26361] 99 26361 334134 886 0 0 0 httpd [562074.671241] [28324] 0 28324 27283 1 1 0 0 sshd [562074.673759] [28333] 0 28333 26649 1 0 0 0 bash [562074.676257] [28546] 99 28546 334167 821 0 0 0 httpd [562074.702428] [ 1091] 0 1091 27283 1 1 0 0 sshd [562074.704538] [ 1130] 0 1130 26649 1 0 0 0 bash [562074.706652] [ 5276] 500 5276 77474 7801 1 0 0 php [562074.708866] [ 5278] 500 5278 77582 7649 0 0 0 php [562074.711662] [ 5281] 500 5281 77580 7259 1 0 0 php [562074.714235] [ 5290] 500 5290 77601 7474 1 0 0 php [562074.716490] [ 5295] 500 5295 77538 7540 1 0 0 php [562074.718824] [ 5299] 500 5299 77414 7551 1 0 0 php [562074.721384] [ 5302] 500 5302 77518 8104 1 0 0 php [562074.724114] [ 5304] 500 5304 77414 7522 1 0 0 php [562074.726653] [ 5308] 500 5308 77582 7241 0 0 0 php [562074.729482] [ 5309] 500 5309 77542 7294 0 0 0 php [562074.731758] [ 5367] 500 5367 79215 8223 0 0 0 php [562074.734948] [ 5369] 500 5369 79230 7073 1 0 0 php [562074.738000] [ 5389] 500 5389 74670 6863 1 0 0 php [562074.740609] [ 5407] 500 5407 74996 5832 0 0 0 php [562074.742809] [ 5415] 500 5415 75850 6233 1 0 0 php [562074.745813] [ 5416] 500 5416 75061 5948 1 0 0 php [562074.748265] [ 5418] 500 5418 74992 5796 1 0 0 php [562074.750694] [ 5420] 500 5420 76190 7332 0 0 0 php [562074.753004] [ 5430] 500 5430 74932 6113 1 0 0 php [562074.755384] [ 5434] 500 5434 75460 6480 1 0 0 php [562074.757639] [ 5439] 500 5439 74928 5879 1 0 0 php [562074.761562] [ 5440] 500 5440 75125 5701 1 0 0 php [562074.764306] [ 5442] 500 5442 75914 6693 1 0 0 php [562074.766600] [ 5446] 500 5446 75214 5475 1 0 0 php [562074.768955] [ 5448] 500 5448 74705 6309 1 0 0 php [562074.771372] [ 5450] 500 5450 73976 6564 0 0 0 php [562074.773778] [ 5451] 500 5451 73831 6459 1 0 0 php [562074.776178] [ 5458] 500 5458 74611 7603 0 0 0 php [562074.778666] [ 5459] 500 5459 75781 8240 1 0 0 php [562074.781649] [ 5465] 500 5465 73772 7310 1 0 0 php [562074.784412] [ 5470] 500 5470 74414 7797 1 0 0 php [562074.788349] [ 5476] 500 5476 74864 8214 1 0 0 php [562074.790625] [ 5480] 500 5480 73510 8366 0 0 0 php [562074.793613] [ 5482] 500 5482 73708 8698 0 0 0 php [562074.796689] [ 5492] 500 5492 72444 11027 1 0 0 php [562074.799184] [ 5496] 500 5496 72784 12415 1 0 0 php [562074.802301] [ 5499] 500 5499 73768 11388 0 0 0 php [562074.804993] [ 5500] 500 5500 72582 11302 0 0 0 php [562074.808597] [ 5503] 500 5503 70752 9687 0 0 0 php [562074.810941] [ 5513] 500 5513 73447 10543 0 0 0 php [562074.813474] [ 5519] 500 5519 72728 12005 1 0 0 php [562074.815596] [ 5520] 500 5520 72710 11134 1 0 0 php [562074.818435] [ 5523] 500 5523 72246 11313 0 0 0 php [562074.821278] [ 5533] 500 5533 73576 12705 0 0 0 php [562074.824227] [ 5536] 500 5536 70305 11674 1 0 0 php [562074.826292] [ 5541] 500 5541 73378 13119 1 0 0 php [562074.828638] [ 5545] 500 5545 69769 10932 1 0 0 php [562074.831461] [ 5548] 500 5548 69382 10296 0 0 0 php [562074.833803] [ 5549] 500 5549 73458 13978 1 0 0 php [562074.836589] [ 5564] 500 5564 69708 11383 1 0 0 php [562074.839962] [ 5568] 500 5568 69380 12267 0 0 0 php [562074.842304] [ 5570] 500 5570 69580 13324 1 0 0 php [562074.844533] [ 5574] 500 5574 69057 13993 0 0 0 php [562074.847082] [ 5577] 500 5577 69450 13875 1 0 0 php [562074.849309] [ 5578] 500 5578 70048 14829 1 0 0 php [562074.851653] [ 5587] 500 5587 69514 15274 1 0 0 php [562074.854140] [ 5589] 500 5589 69382 16196 1 0 0 php [562074.856537] [ 5593] 500 5593 68012 14763 0 0 0 php [562074.858820] [ 5597] 500 5597 71343 18344 0 0 0 php [562074.861408] [ 5598] 500 5598 66115 12838 1 0 0 php [562074.864285] [ 5603] 500 5603 69319 16732 1 0 0 php [562074.866510] [ 5612] 500 5612 63573 11969 0 0 0 php [562074.868753] [ 5613] 500 5613 61574 9964 1 0 0 php [562074.871009] [ 5616] 500 5616 63637 12150 0 0 0 php [562074.874032] [ 5624] 500 5624 63726 12707 1 0 0 php [562074.876563] [ 5625] 500 5625 55276 4013 1 0 0 php [562074.879156] [ 5630] 500 5630 52951 2211 0 0 0 php [562074.881476] [ 5631] 500 5631 53736 3070 1 0 0 php [562074.883968] [ 5633] 500 5633 53245 2474 0 0 0 php [562074.886305] [ 5634] 500 5634 54533 3868 1 0 0 php [562074.888585] [ 5635] 500 5635 54650 3839 0 0 0 php [562074.890872] [ 5636] 500 5636 56802 6104 0 0 0 php [562074.893228] [ 5637] 500 5637 54189 3375 1 0 0 php [562074.895530] [ 5638] 500 5638 54563 3901 0 0 0 php [562074.897784] [ 5639] 500 5639 52667 1958 1 0 0 php [562074.900122] [ 5640] 500 5640 54533 3711 0 0 0 php [562074.902475] [ 5641] 500 5641 52922 2197 0 0 0 php [562074.904697] [ 5642] 500 5642 52667 1954 1 0 0 php [562074.907013] [ 5643] 500 5643 52958 2226 0 0 0 php [562074.909381] [ 5644] 500 5644 52667 1927 0 0 0 php [562074.911628] [ 5645] 500 5645 54015 3402 1 0 0 php [562074.913914] [ 5646] 500 5646 52951 2165 0 0 0 php [562074.916614] [ 5647] 500 5647 54635 3863 1 0 0 php [562074.919109] [ 5648] 99 5648 219408 400 1 0 0 httpd [562074.921510] [ 5649] 500 5649 53928 3331 1 0 0 php [562074.924011] [ 5650] 500 5650 56653 5958 0 0 0 php [562074.926651] [ 5651] 500 5651 52951 2223 0 0 0 php [562074.929157] [ 5652] 500 5652 52735 1997 1 0 0 php [562074.954567] [ 5653] 500 5653 52457 1707 1 0 0 php [562074.957251] [ 5654] 99 5654 235792 481 1 0 0 httpd [562074.959480] [ 5660] 500 5660 53928 3321 0 0 0 php [562074.961682] [ 5686] 500 5686 54417 3563 0 0 0 php [562074.964573] [ 5712] 500 5712 52958 2216 0 0 0 php [562074.966804] [ 5713] 500 5713 53887 3247 0 0 0 php [562074.969044] [ 5714] 500 5714 52951 2206 0 0 0 php [562074.971840] [ 5715] 500 5715 53736 3103 1 0 0 php [562074.974827] [ 5716] 500 5716 52663 1938 1 0 0 php [562074.977527] [ 5717] 500 5717 56118 5385 0 0 0 php [562074.980867] [ 5726] 500 5726 57172 6396 1 0 0 php [562074.984658] [ 5727] 500 5727 52523 1760 0 0 0 php [562074.987267] [ 5728] 500 5728 52587 1869 1 0 0 php [562074.989641] [ 5730] 500 5730 52430 1711 1 0 0 php [562074.992423] [ 5731] 500 5731 53672 3061 0 0 0 php [562074.995140] [ 5732] 500 5732 52587 1860 1 0 0 php [562074.998136] [ 5733] 500 5733 52770 2016 0 0 0 php [562075.002104] [ 5734] 500 5734 53216 2537 1 0 0 php [562075.004987] [ 5735] 500 5735 54351 3615 1 0 0 php [562075.007912] [ 5737] 500 5737 54329 3665 0 0 0 php [562075.010646] [ 5738] 500 5738 53864 3231 1 0 0 php [562075.013388] [ 5750] 500 5750 53992 3332 1 0 0 php [562075.016259] [ 5751] 500 5751 53992 3356 1 0 0 php [562075.018239] [ 5754] 500 5754 52457 1707 0 0 0 php [562075.020361] [ 5755] 500 5755 53672 3008 0 0 0 php [562075.022769] [ 5758] 500 5758 53928 3334 1 0 0 php [562075.024717] [ 5759] 500 5759 53800 3153 0 0 0 php [562075.026820] [ 5760] 500 5760 52457 1715 1 0 0 php [562075.029743] [ 5761] 500 5761 53672 2999 1 0 0 php [562075.031868] [ 5767] 500 5767 52430 1709 0 0 0 php [562075.033922] [ 5768] 500 5768 53672 3015 1 0 0 php [562075.035960] [ 5772] 500 5772 52457 1708 1 0 0 php [562075.038017] [ 5774] 500 5774 53928 3307 0 0 0 php [562075.040342] [ 5777] 500 5777 52430 1709 0 0 0 php [562075.042585] [ 5780] 500 5780 52922 2200 1 0 0 php [562075.044900] [ 5781] 500 5781 29874 419 0 0 0 php [562075.046949] [ 5782] 500 5782 52430 1710 0 0 0 php [562075.049163] [ 5787] 500 5787 53736 3111 0 0 0 php [562075.051348] [ 5788] 500 5788 52428 1681 0 0 0 php [562075.053411] [ 5789] 500 5789 16226 45 1 0 0 php [562075.055455] [ 5790] 500 5790 53992 3361 0 0 0 php [562075.057385] [ 5796] 500 5796 53800 3145 0 0 0 php [562075.059354] [ 5802] 500 5802 51933 1164 0 0 0 php [562075.061403] [ 5803] 500 5803 53736 3088 1 0 0 php [562075.063319] [ 5828] 500 5828 52457 1712 1 0 0 php [562075.065328] [ 5830] 500 5830 53992 3362 1 0 0 php [562075.067304] [ 5833] 500 5833 52457 1711 0 0 0 php [562075.069272] [ 5834] 500 5834 52457 1710 0 0 0 php [562075.071342] [ 5838] 500 5838 25576 65 0 0 0 php [562075.073811] [ 5839] 99 5839 121104 248 1 0 0 httpd [562075.075876] [ 5848] 99 5848 121104 279 1 0 0 httpd [562075.078224] [ 5849] 500 5849 29230 282 1 0 0 php [562075.080532] [ 5850] 500 5850 29118 217 1 0 0 php [562075.082861] [ 5851] 500 5851 53352 2643 0 0 0 php [562075.084933] [ 5852] 99 5852 121104 281 1 0 0 httpd [562075.087091] [ 5854] 500 5854 52430 1708 0 0 0 php [562075.089180] [ 5856] 500 5856 51703 882 0 0 0 php [562075.091154] [ 5863] 99 5863 121104 286 1 0 0 httpd [562075.093112] [ 5970] 500 5970 28797 186 1 0 0 php [562075.095140] [ 5986] 0 5986 5651 60 0 0 0 upcp-running [562075.097396] [ 5988] 500 5988 29118 223 0 0 0 php [562075.099621] [ 5991] 99 5991 104720 241 0 0 0 httpd [562075.101929] [ 5993] 500 5993 3116 43 0 0 0 suphp [562075.104055] [ 5994] 500 5994 29329 408 0 0 0 php [562075.106361] [ 6026] 500 6026 29118 223 0 0 0 php [562075.108660] [ 6027] 500 6027 29118 231 1 0 0 php [562075.111327] [ 6028] 500 6028 7365 23 1 0 0 php [562075.113563] [ 6032] 99 6032 121104 279 1 0 0 httpd [562075.115682] Out of memory: Kill process 9408 (mysqld) score 44 or sacrifice child [562075.117786] Killed process 9408, UID 402, (mysqld) total-vm:2257588kB, anon-rss:171860kB, file-rss:544kB
Это происходит один или два раза в день. В противном случае нагрузка находится в пределах (<2,00). Когда возникает эта ошибка, нагрузка достигает> 100. Судя по журналам доступа, я не получал больше трафика, когда это произошло. Что мне делать, чтобы это исправить?
Что мне делать, чтобы это исправить?
Увеличьте объем памяти или уменьшите нагрузку на память от процессов.
Установить мониторинг
Соберите данные.
Использовать Научный метод выяснить, где находятся узкие места.
Примите соответствующие меры.
При необходимости повторите.