Назад | Перейти на главную страницу

Nginx, Varnish 4.1 Backend fetch не удалось

Я могу получить доступ к сайту через https и раздел администратора, но не могу получить доступ к не https / guest или чему-либо, что должно быть кэшировано.

varnishadm -S /etc/varnish/secret -T localhost:6082 debug.health
Connection failed (localhost:6082): (null)

Varnishlog

Begin          bereq 2 fetch
Timestamp      Start: 1489151629.565881 0.000000 0.000000
BereqMethod    GET
BereqURL       /
BereqProtocol  HTTP/1.1
BereqHeader    Accept: text/html, application/xhtml+xml, image/jxr, */*
BereqHeader    AcceptLanguage: enGB
BereqHeader    UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393
BereqHeader    Host: www.website.com
BereqHeader    Cookie: __cfduid=d960b1ddcd82e149ba07d1d08b51868f01488714214; __atuvc=83%7C10; jbcookies=yes; onOffreadingmode=; nextPrevfont=Default; nextPrevfs=Medium; 2e6bfd3da2be20fdd818219a928632de=jh5ko0a723b6l4806mkj2tmjn5; 4dbbb894f976294264bd50dc4b48c008=48
BereqHeader    XForwardedFor: 23.227.207.10, 23.227.207.10
BereqHeader    AcceptEncoding: gzip
BereqHeader    XVarnish: 3
VCL_call       BACKEND_FETCH
VCL_return     fetch
FetchError     no backend connection
Timestamp      Beresp: 1489151629.566007 0.000126 0.000126
Timestamp      Error: 1489151629.566015 0.000134 0.000007
BerespProtocol HTTP/1.1
BerespStatus   503
BerespReason   Service Unavailable
BerespReason   Backend fetch failed
BerespHeader   Date: Fri, 10 Mar 2017 13:13:49 GMT
BerespHeader   Server: Varnish
VCL_call       BACKEND_ERROR
BerespHeader   ContentType: text/html; charset=utf8
BerespHeader   RetryAfter: 5
VCL_return     deliver
Storage        malloc Transient
ObjProtocol    HTTP/1.1
ObjStatus      503
ObjReason      Backend fetch failed
ObjHeader      Date: Fri, 10 Mar 2017 13:13:49 GMT
ObjHeader      Server: Varnish
ObjHeader      ContentType: text/html; charset=utf8
ObjHeader      RetryAfter: 5
Length         278
BereqAcct      0 0 0 0 0 0

Вывод Netstart

netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      3123/nginx: master
tcp        0      0 127.0.0.1:6082          0.0.0.0:*               LISTEN      27943/varnishd
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      807/php-fpm: master
tcp        0      0 127.0.0.1:9002          0.0.0.0:*               LISTEN      807/php-fpm: master
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN      858/memcached
tcp        0      0 127.0.0.1:9003          0.0.0.0:*               LISTEN      807/php-fpm: master
tcp        0      0 127.0.0.1:9004          0.0.0.0:*               LISTEN      807/php-fpm: master
tcp        0      0 127.0.0.1:9005          0.0.0.0:*               LISTEN      807/php-fpm: master
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      27943/varnishd
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      3123/nginx: master
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      28446/pure-ftpd (SE
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      570/master
tcp6       0      0 :::3306                 :::*                    LISTEN      433/mysqld
tcp6       0      0 :::80                   :::*                    LISTEN      27943/varnishd
tcp6       0      0 :::21                   :::*                    LISTEN      28446/pure-ftpd (SE
tcp6       0      0 ::1:25                  :::*                    LISTEN      570/master

Лак default.vcl

# new 4.0 format.
vcl 4.0;

# Imports
import std;

# Default backend definition. Set this to point to your content server.
backend default {
.host = "127.0.0.1"; # don't change this if the web server is on the same machine
.port = "8080"; # replace XXXX with your web server's (internal) port, e.g. 8080
}

sub vcl_recv {

/*
# If we host multiple domains on a server, here you can list the domains you DO NOT want to cache
# The first check matches both naked & "www" subdomains. Use the second for non generic subdomains.
if (
    req.http.host ~ "(www\.)?(domain1.com|domain2.org|domain3.net)" ||
    req.http.host ~ "(subdomain.domain4.tld|othersubdomain.domain5.tld)"
) {
    return (pass);
}
*/

# Forward client's IP to the backend
if (req.restarts == 0) {
    if (req.http.X-Real-IP) {
        set req.http.X-Forwarded-For = req.http.X-Real-IP;
    } else if (req.http.X-Forwarded-For) {
        set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    } else {
        set req.http.X-Forwarded-For = client.ip;
    }
}

# httpoxy
unset req.http.proxy;

# Normalize the query arguments
set req.url = std.querysort(req.url);

# Non-RFC2616 or CONNECT which is weird.
if (
    req.method != "GET" &&
    req.method != "HEAD" &&
    req.method != "PUT" &&
    req.method != "POST" &&
    req.method != "TRACE" &&
    req.method != "OPTIONS" &&
    req.method != "DELETE"
) {
    return (pipe);
}

# We only deal with GET and HEAD by default
if (req.method != "GET" && req.method != "HEAD") {
    return (pass);
}

# Don't cache HTTP authorization/authentication pages and pages with certain headers or cookies
if (
    req.http.Authorization ||
    req.http.Authenticate ||
    req.http.X-Logged-In == "True" ||
    req.http.Cookie ~ "userID" ||
    req.http.Cookie ~ "joomla_[a-zA-Z0-9_]+" ||
    req.http.Cookie ~ "(wordpress_[a-zA-Z0-9_]+|wp-postpass|comment_author_[a-zA-Z0-9_]+)"
) {
    #set req.http.Cache-Control = "private, max-age=0, no-cache, no-store";
    #set req.http.Expires = "Mon, 01 Jan 2001 00:00:00 GMT";
    #set req.http.Pragma = "no-cache";
    return (pass);
}

# Exclude the following paths (e.g. backend admins, user pages or ad URLs that require tracking)
# In Joomla specifically, you are advised to create specific entry points (URLs) for users to
# interact with the site (either common user logins or even commenting), e.g. make a menu item
# to point to a user login page (e.g. /login), including all related functionality such as
# password reset, email reminder and so on.
if(
    req.url ~ "^/administrator" ||
    req.url ~ "^/component/banners" ||
    req.url ~ "^/component/socialconnect" ||
    req.url ~ "^/component/users" ||
    req.url ~ "^/contact" ||
    req.url ~ "^/connect" ||
    req.url ~ "^/wp-admin" ||
    req.url ~ "^/wp-login.php"
) {
    #set req.http.Cache-Control = "private, max-age=0, no-cache, no-store";
    #set req.http.Expires = "Mon, 01 Jan 2001 00:00:00 GMT";
    #set req.http.Pragma = "no-cache";
    return (pass);
}

# Don't cache ajax requests
if(req.http.X-Requested-With == "XMLHttpRequest" || req.url ~ "nocache") {
    #set req.http.Cache-Control = "private, max-age=0, no-cache, no-store";
    #set req.http.Expires = "Mon, 01 Jan 2001 00:00:00 GMT";
    #set req.http.Pragma = "no-cache";
    return (pass);
}

# Check for the custom "X-Logged-In" header (used by K2 and other apps) to identify
# if the visitor is a guest, then unset any cookie (including session cookies) provided
# it's not a POST request.
if(req.http.X-Logged-In == "False" && req.method != "POST") {
    unset req.http.Cookie;
}

# Properly handle different encoding types
if (req.http.Accept-Encoding) {
  if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$") {
    # No point in compressing these
    unset req.http.Accept-Encoding;
  } elseif (req.http.Accept-Encoding ~ "gzip") {
    set req.http.Accept-Encoding = "gzip";
  } elseif (req.http.Accept-Encoding ~ "deflate") {
    set req.http.Accept-Encoding = "deflate";
  } else {
    # unknown algorithm (aka crappy browser)
    unset req.http.Accept-Encoding;
  }
}

# Cache files with these extensions
#if (req.url ~ "\.(js|css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$") {
#    return (hash);
#}

# Remove all cookies for static files & deliver directly
if (req.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|ogg|ogm|opus|otf|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
    unset req.http.Cookie;
    return (hash);
}

return (hash);

}

sub vcl_backend_response {

/*
# If we host multiple domains on a server, here you can list the domains you DO NOT want to cache
# The first check matches both naked & "www" subdomains. Use the second for non generic subdomains.
if (
    bereq.http.host ~ "(www\.)?(domain1.com|domain2.org|domain3.net)" ||
    bereq.http.host ~ "(subdomain.domain4.tld|othersubdomain.domain5.tld)"
) {
    set beresp.uncacheable = true;
    return (deliver);
}
*/

# Don't cache 50x responses
if (
    beresp.status == 500 ||
    beresp.status == 502 ||
    beresp.status == 503 ||
    beresp.status == 504
) {
    return (abandon);
}

# Exclude the following paths (e.g. backend admins, user pages or ad URLs that require tracking)
# In Joomla specifically, you are advised to create specific entry points (URLs) for users to
# interact with the site (either common user logins or even commenting), e.g. make a menu item
# to point to a user login page (e.g. /login), including all related functionality such as
# password reset, email reminder and so on.
if(
    bereq.url ~ "^/administrator" ||
    bereq.url ~ "^/component/banners" ||
    bereq.url ~ "^/component/socialconnect" ||
    bereq.url ~ "^/component/users" ||
    bereq.url ~ "^/contact" ||
    bereq.url ~ "^/connect" ||
    bereq.url ~ "^/wp-admin" ||
    bereq.url ~ "^/wp-login.php"
) {
    #set beresp.http.Cache-Control = "private, max-age=0, no-cache, no-store";
    #set beresp.http.Expires = "Mon, 01 Jan 2001 00:00:00 GMT";
    #set beresp.http.Pragma = "no-cache";
    set beresp.uncacheable = true;
    return (deliver);
}

# Don't cache HTTP authorization/authentication pages and pages with certain headers or cookies
if (
    bereq.http.Authorization ||
    bereq.http.Authenticate ||
    bereq.http.X-Logged-In == "True" ||
    bereq.http.Cookie ~ "userID" ||
    bereq.http.Cookie ~ "joomla_[a-zA-Z0-9_]+" ||
    bereq.http.Cookie ~ "(wordpress_[a-zA-Z0-9_]+|wp-postpass|comment_author_[a-zA-Z0-9_]+)"
) {
    #set beresp.http.Cache-Control = "private, max-age=0, no-cache, no-store";
    #set beresp.http.Expires = "Mon, 01 Jan 2001 00:00:00 GMT";
    #set beresp.http.Pragma = "no-cache";
    set beresp.uncacheable = true;
    return (deliver);
}

# Don't cache ajax requests
if(beresp.http.X-Requested-With == "XMLHttpRequest" || bereq.url ~ "nocache") {
    #set beresp.http.Cache-Control = "private, max-age=0, no-cache, no-store";
    #set beresp.http.Expires = "Mon, 01 Jan 2001 00:00:00 GMT";
    #set beresp.http.Pragma = "no-cache";
    set beresp.uncacheable = true;
    return (deliver);
}

# Don't cache backend response to posted requests
if (bereq.method == "POST") {
    set beresp.uncacheable = true;
    return (deliver);
}

# Ok, we're cool & ready to cache things
# so let's clean up some headers and cookies
# to maximize caching.

# Check for the custom "X-Logged-In" header to identify if the visitor is a guest,
# then unset any cookie (including session cookies) provided it's not a POST request.
if(bereq.method != "POST" && beresp.http.X-Logged-In == "False") {
    unset beresp.http.Set-Cookie;
}

# Unset the "etag" header (suggested)
unset beresp.http.etag;

# Unset the "pragma" header
unset beresp.http.Pragma;

# Allow stale content, in case the backend goes down
set beresp.grace = 6h;

# This is how long Varnish will keep cached content
set beresp.ttl = 2m;

# Modify "expires" header - https://www.varnish-cache.org/trac/wiki/VCLExampleSetExpires
#set beresp.http.Expires = "" + (now + beresp.ttl);

# If your backend server does not set the right caching headers for static assets,
# you can set them below (uncomment first and change 604800 - which 1 week - to whatever you
# want (in seconds)
#if (req.url ~ "\.(ico|jpg|jpeg|gif|png|bmp|webp|tiff|svg|svgz|pdf|mp3|flac|ogg|mid|midi|wav|mp4|webm|mkv|ogv|wmv|eot|otf|woff|ttf|rss|atom|zip|7z|tgz|gz|rar|bz2|tar|exe|doc|docx|xls|xlsx|ppt|pptx|rtf|odt|ods|odp)(\?[a-zA-Z0-9=]+)$") {
#    set beresp.http.Cache-Control = "public, max-age=604800";
#}

if (bereq.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|ogg|ogm|opus|otf|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
    unset beresp.http.set-cookie;
    set beresp.do_stream = true;
}

# We have content to cache, but it's got no-cache or other Cache-Control values sent
# So let's reset it to our main caching time (2m as used in this example configuration)
# The additional parameters specified (stale-while-revalidate & stale-if-error) are used
# by modern browsers to better control caching. Set there to twice & five times your main
# cache time respectively.
# This final setting will normalize CMSs like Joomla which set max-age=0 even when
# Joomla's cache is enabled.
if (beresp.http.Cache-Control !~ "max-age" || beresp.http.Cache-Control ~ "max-age=0") {
    set beresp.http.Cache-Control = "public, max-age=120, stale-while-revalidate=240, stale-if-error=480";
}

return (deliver);

}   

sub vcl_deliver {

/*
# Send a special header for excluded domains only
# The if statement can be identical to the ones in the vcl_recv() and vcl_fetch() functions above
if (
    req.http.host ~ "(www\.)?(domain1.com|domain2.org|domain3.net)" ||
    req.http.host ~ "(subdomain.domain4.tld|othersubdomain.domain5.tld)"
) {
    set resp.http.X-Domain-Status = "EXCLUDED";
}

# Enforce redirect to HTTPS for specified domains only
if (
    req.http.host ~ "(subdomain.domain4.tld|othersubdomain.domain5.tld)" &&
    req.http.X-Forwarded-Proto !~ "(?i)https"
) {
    set resp.http.Location = "https://" + req.http.host + req.url;
    set resp.status = 302;
}
*/
# Send special headers that indicate the cache status of each web page
if (obj.hits > 0) {
    set resp.http.X-Cache = "HIT";
    set resp.http.X-Cache-Hits = obj.hits;
} else {
    set resp.http.X-Cache = "MISS";
}

return (deliver);

}

Nginx nginxdomain.conf

# redirect from non-www to www 
# uncomment, save file and restart Nginx to enable
# if unsure use return 302 before using return 301
server {
        listen   8080;
        listen 443 ssl http2;
        server_name domain.com www.domain.com;


# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

#add_header X-Frame-Options SAMEORIGIN;
add_header X-Xss-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;  

# limit_conn limit_per_ip 16;
# ssi  on;

access_log /home/nginx/domains/domain.com/log/access.log main_ext buffer=256k flush=60m;
error_log /home/nginx/domains/domain.com/log/error.log;

include /usr/local/nginx/conf/autoprotect/domain.com/autoprotect-domain.com.conf;
root /home/nginx/domains/domain.com/public;
# uncomment cloudflare.conf include if using cloudflare for
# server and/or vhost site
include /usr/local/nginx/conf/cloudflare.conf;
include /usr/local/nginx/conf/503include-main.conf;

# prevent access to ./directories and files
# location ~ (?:^|/)\. {
# deny all;
#}

location / {
include /usr/local/nginx/conf/503include-only.conf;

# block common exploits, sql injections etc
# include /usr/local/nginx/conf/block.conf;

# Enables directory listings when index file not found
#autoindex  on;

# Shows file listing times as local time
#autoindex_localtime on;

# Enable Dynamic Proxy Cache
include /usr/local/nginx/conf/proxy.conf;

# Enable for Joomla URL SEF usage
try_files $uri $uri/ /index.php?q=$request_uri;

}

include /usr/local/nginx/conf/staticfiles.conf;
include /usr/local/nginx/conf/php.conf;
include /usr/local/nginx/conf/drop.conf;
#include /usr/local/nginx/conf/errorpage.conf;
include /usr/local/nginx/conf/vts_server.conf;
}

Лак Params

# Varnish environment configuration description. This was derived from
# the old style sysconfig/defaults settings

# Set this to 1 to make systemd reload try to switch VCL without restart.
RELOAD_VCL=1

# Set WARMUP_TIME to force a delay in reload-vcl between vcl.load and vcl.use
# This is useful when backend probe definitions need some time before declaring
# configured backends healthy, to avoid routing traffic to a non-healthy backend.
#WARMUP_TIME=0

# Main configuration file. You probably want to change it.
VARNISH_VCL_CONF=/etc/varnish/default.vcl

# Default address and port to bind to. Blank address means all IPv4
# and IPv6 interfaces, otherwise specify a host name, an IPv4 dotted
# quad, or an IPv6 address in brackets.
# VARNISH_LISTEN_ADDRESS=192.168.1.5
VARNISH_LISTEN_PORT=80

# Admin interface listen address and port
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082

# Shared secret file for admin interface
VARNISH_SECRET_FILE=/etc/varnish/secret

# Backend storage specification, see Storage Types in the varnishd(5)
# man page for details.
VARNISH_STORAGE="malloc,256M"

# User and group for the varnishd worker processes
VARNISH_USER=varnish
VARNISH_GROUP=varnish

# Other options, see the man page varnishd(1)
#DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300"

Ваш vcl довольно сложный.

Что я вижу:

  • Varnish не может добраться до серверной части nginx (в журнале FetchError no backend connection)
  • Бэкэнд-хост, на который varnish пересылает запрос, - BereqHeader Host: www.website.com
  • Varnish не пересылает запросы на порт 8080 соединение с сервером, кажется, сделано, чтобы покрыть себя лаком
  • Ваш сервер nginx не слушает www.website.com хозяин, только для server_name domain.com www.domain.com;

Я бы попробовал

  • Убедитесь, что nginx слушает хост, который вы запрашиваете
  • Проверьте свою активную конфигурацию (используя varnishadm затем backend.list или vcl.show), чтобы узнать, используется ли порт 8080

На основе вашего default.vcl и ошибку я заключаю, что вы настраиваете бэкэнд по умолчанию но не использую его.

Для этого вы можете добавить строку ниже в начало vcl_recv так это выглядит так:

sub vcl_recv {
  set req.backend_hint = default;
  ...
}