Назад | Перейти на главную страницу

Ошибка segfault 4 в libmysqlclient.so

Вот уже несколько дней я вижу в своем системном журнале такие сообщения:

Sep 23 14:28:42 server kernel: [138926.637593] php5-fpm[6455]: segfault at 7f9ade735018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:28:44 server kernel: [138928.314016] php5-fpm[22742]: segfault at 7f9ade3db018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:32:11 server kernel: [139135.318287] php5-fpm[16887]: segfault at 7f9ade4b3018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:32:49 server kernel: [139173.050377] php5-fpm[668]: segfault at 7f9ade61a018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:33:19 server kernel: [139203.396935] php5-fpm[26277]: segfault at 7f9ade6c0018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:35:06 server kernel: [139310.048740] php5-fpm[27017]: segfault at 7f9ade46c018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:35:19 server kernel: [139323.494188] php5-fpm[31263]: segfault at 7f9ade5e2018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:36:10 server kernel: [139374.904308] php5-fpm[26422]: segfault at 7f9ade6cf018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:37:25 server kernel: [139449.360384] php5-fpm[20806]: segfault at 7f9ade644018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]

Я использую debian 8 и MariaDB. Вначале это было только раз в 2 или 3 часа, но теперь это несколько раз в час. После некоторого исследования я понял, что это должна быть проблема с памятью, но я не нашел никакого решения для ее решения.

Вот что я вижу в mysqltuner:

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +Aria +BLACKHOLE +CSV +FEDERATED +InnoDB +MRG_MyISAM 
[--] Data in InnoDB tables: 2G (Tables: 79)
[--] Data in MyISAM tables: 96M (Tables: 146)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 52)
[!!] Total fragmented tables: 34

-------- Security Recommendations  -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 1d 16h 44m 38s (73M q [502.853 qps], 196K conn, TX: 572B, RX: 14B)
[--] Reads / Writes: 97% / 3%
[--] Total buffers: 17.3G global + 56.2M per thread (500 max threads)
[!!] Maximum possible memory usage: 44.8G (142% of installed RAM)
[OK] Slow queries: 0% (2K/73M)
[OK] Highest usage of available connections: 28% (141/500)
[OK] Key buffer size / total MyISAM indexes: 1.0G/32.6M
[OK] Key buffer hit rate: 100.0% (132M cached / 53K reads)
[OK] Query cache efficiency: 44.9% (50M cached / 113M selects)
[!!] Query cache prunes per day: 260596
[OK] Sorts requiring temporary tables: 0% (2K temp sorts / 2M sorts)
[OK] Temporary tables created on disk: 21% (6K on disk / 28K total)
[OK] Thread cache hit rate: 99% (141 created / 196K connections)
[OK] Table cache hit rate: 72% (500 open / 692 opened)
[OK] Open file limit used: 17% (429/2K)
[OK] Table locks acquired immediately: 99% (25M immediate / 25M locks)
[OK] InnoDB buffer pool / data size: 16.0G/2.4G
[!!] InnoDB log waits: 30

Таким образом, максимальный объем используемой памяти слишком велик, но я скорректировал размер буферного пула innodb на 16Go, для 32Go RAM это должно быть нормально, я не знаю, что делать для оптимизации.

Дело в том, что общее использование памяти на сервере всегда ниже 89% (плюс кеширование). MySQL фактически использует 50,6% оперативной памяти. Не знаю, есть ли связь между всем этим, но я предпочитаю поместить ее здесь. В остальном вроде все в порядке со стороны MySQL ...

Наконец, я настроил основные переменные в my.cnf, которые могли повлиять на это:

max_connections = 100
max_heap_table_size = 64M
read_buffer_size = 4M
read_rnd_buffer_size = 32M
sort_buffer_size = 8M
query_cache_size = 256M
query_cache_limit = 4M
query_cache_type = 1
query_cache_strip_comments =1
thread_stack = 192K
transaction_isolation = READ-COMMITTED
tmp_table_size = 64M
nnodb_additional_mem_pool_size = 16M
innodb_buffer_pool_size = 16G
thread_cache_size   = 4M
max_connections     = 500
join_buffer_size    = 12M
interactive_timeout = 30
wait_timeout        = 30
open_files_limit    = 800
innodb_file_per_table
key_buffer_size     = 1G
table_open_cache    = 500
innodb_log_file_size    = 256M

Два дня назад сервер без причины отказал в системном журнале, за исключением segfault. Может ли segfault вывести систему из строя? Любая идея по причине segfault? Несколько способов понять причину проблемы?

Итак, segfaults соответствуют ошибкам на моем клиентском веб-сайте. По этим ошибкам:

Sep 23 21:39:07 server kernel: [164772.739464] php5-fpm[10375]: segfault at 7f9ade4be018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 22:08:47 server kernel: [166554.093566] php5-fpm[12582]: segfault at 7f9ade5d9018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 22:09:17 server kernel: [166584.252833] php5-fpm[31132]: segfault at 7f9ade669018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 22:10:30 server kernel: [166656.875041] php5-fpm[24992]: segfault at 7f9ade6e9018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 22:39:19 server kernel: [168387.935728] php5-fpm[27380]: segfault at 7f9ade654018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 22:48:55 server kernel: [168963.750727] php5-fpm[6309]: segfault at 7f9ade371018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 22:49:33 server kernel: [169001.961155] php5-fpm[15416]: segfault at 7f9ade40f018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 23:04:22 server kernel: [169892.025852] php5-fpm[5763]: segfault at 7f9ade422018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 23:04:50 server kernel: [169919.986130] php5-fpm[16016]: segfault at 7f9ade5cf018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 23:09:11 server kernel: [170181.423023] php5-fpm[5129]: segfault at 7f9ade6cc018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]

Я получил эти ошибки в моей ошибке журнала nginx, точно в то же время, поэтому не случайно:

2017/09/23 21:39:07 [error] 808#0: *7346034 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /page/a-page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com", referrer: "https://example.com/page/another-page"
2017/09/23 22:08:47 [error] 808#0: *7443193 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /engine/search?q=query", upstream: "fastcgi://127.0.0.1:9000", host: "example.com"
2017/09/23 22:09:17 [error] 808#0: *7450102 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com"
2017/09/23 22:10:30 [error] 808#0: *7450989 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /page/a-page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com", referrer: "https://example.com/page/another-page"
2017/09/23 22:39:19 [error] 808#0: *7553713 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 21.1.1.1, server: example.com, request: "GET /page/a-page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com", referrer: "https://example.com/engine/search?q=query"
2017/09/23 22:48:55 [error] 7800#0: *7578116 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /engine/search?q=query HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com"
2017/09/23 22:49:33 [error] 7800#0: *7582962 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /page/a-page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com"
2017/09/23 23:04:22 [error] 7800#0: *7639594 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /page/a-page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com"
2017/09/23 23:04:50 [error] 7800#0: *7636486 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /page/a-page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com", referrer: "https://example.com/engine/search?q=query"
2017/09/23 23:09:11 [error] 7800#0: *7654327 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /engine/search?q=query3 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com"

Итак, это мой виртуальный хост nginx для этого сайта:

server {

   listen  80;
   listen  443;
   server_name example.com;

   if ($http_x_forwarded_proto = "http") {
      return 301 https://$server_name$request_uri;
   }

   ssl  on;
   ssl_certificate /etc/nginx/cloudflare_cert/fullchain.pem;
   ssl_certificate_key /etc/nginx/cloudflare_cert/privkey.pem;

   include /etc/nginx/conf.d/cloudflare;

   root   /var/www/website;
   index  index.php;

   location / {
      # main rewrite rule
      try_files $uri $uri/ /index.php?/$request_uri;

      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forward-Proto http;
      proxy_set_header X-Nginx-Proxy true;

   }

   # php parsing
   location ~ \.php$ {
        include fastcgi_params;
        fastcgi_pass  127.0.0.1:9000;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
   }

   error_log  /var/log/nginx/example_error.log;
   access_log  /var/log/nginx/example_access.log;

}

server {
   listen  80;
   listen  443;
   server_name www.example.com;
   return 301 https://example.com$request_uri;
}

Так что это проблема на стороне клиента. Теперь я должен понять, что recv() failed (104: Connection reset by peer) while reading response header from upstream средства и как это решить.