Ошибка плагина logstash output-http 500 при попытке отправить на слабый веб-перехватчик

Я запускаю докер-контейнер со стеком ELK. Все работает отлично и хорошо. Я бы хотел использовать https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html плагин.

Устанавливаю его плагин:

root@elk:/# /opt/logstash/bin/plugin install logstash-output-http
Validating logstash-output-http
Installing logstash-output-http
Installation successful

/etc/logstash/conf.d/logstash.conf основной файл конфигурации logstash:

input {
  beats {
   port => 5044
   type => "logs"
  }
}
filter {
  grok {
    match => { "message" => "\"name\":\"Error\""}
    add_tag => ["error"]
  }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
  if "error" in [tags] {
    http {
      http_method => "post"
      url => "https://hooks.slack.com/services/<MY_ID>"
      format => "json"
      content_type => "application/json"
      message => "{\"text\": \"dis is workinz\"}"
    }
  }
}

Я также пробовал изменить формат в http:

...
format => "message"
...

Конфигурация также проверена и действительна:

root@elk:/# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/logstash.conf 
Configuration OK

Оба приводят к тому, что он не работает. Я дважды проверил, чтобы убедиться, что веб-перехватчик работает с тестом (из контейнера докеров logstash):

curl -H "Content-Type: application/json" -X POST -d '{"text":"testing"}' https://hooks.slack.com/services/<MY_ID>

Я знаю, что теги применяются правильно, проверено в Кибане. Также предложение if также работает, поскольку я тестировал установку elasticsearch внутри него, и он действительно отправлял только журналы с добавленным error теги в elasticsearch.

Наконец, святое сообщение об ошибке от /var/log/logstash/logstah.log:

{: timestamp => "2016-02-25T15:55:35.926000+0000", 
: message => "[HTTP Output Failure] Encountered non-200 HTTP code 200", 
: response_code => 500, : url => "https://hooks.slack.com/services/<YES_THE_ACTUAL_WEBHOOK_ID_WAS_HERE>", 
: event => # < LogStash::Event: 0x408e202@ metadata = {
        "beat" => "filebeat", "type" => "log"
    }, @accessors = # < LogStash::Util::Accessors: 0x5dc514d7@ store = {
        "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
            "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
        }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
    }, @lut = {
        "@timestamp" => [{
            "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
                "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
            }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
        }, "@timestamp"], "beat" => [{
            "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
                "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
            }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
        }, "beat"], "count" => [{
            "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
                "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
            }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
        }, "count"], "fields" => [{
            "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
                "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
            }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
        }, "fields"], "input_type" => [{
            "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
                "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
            }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
        }, "input_type"], "offset" => [{
            "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
                "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
            }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
        }, "offset"], "source" => [{
            "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
                "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
            }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
        }, "source"], "type" => [{
            "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
                "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
            }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
        }, "type"], "[beat][hostname]" => [{
            "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
        }, "hostname"], "host" => [{
            "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
                "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
            }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
        }, "host"], "tags" => [{
            "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
                "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
            }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
        }, "tags"], "message" => [{
            "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
                "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
            }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
        }, "message"], "[tags]" => [{
            "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
                "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
            }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
        }, "tags"]
    } > , @data = {
        "message" => "{\"name\":\"myapp\",\"hostname\":\"sensu-node-dev\",\"pid\":749,\"level\":50,\"err\":{\"message\":\"Cannot find module 'lol'\",\"name\":\"Error\",\"stack\":\"Error: Cannot find module 'lol'\\n    at Function.Module._resolveFilename (module.js:339:15)\\n    at Function.Module._load (module.js:290:25)\\n    at Module.require (module.js:367:17)\\n    at require (internal/module.js:16:19)\\n    at Object.<anonymous> (/srv/www/dev.site/app.js:27:6)\\n    at next (native)\\n    at Object.<anonymous> (/srv/www/dev.site/node_modules/koa-compose/index.js:29:5)\\n    at next (native)\\n    at onFulfilled (/srv/www/dev.site/node_modules/co/index.js:65:19)\\n    at /srv/www/dev.site/node_modules/co/index.js:54:5\",\"code\":\"MODULE_NOT_FOUND\"},\"msg\":\"Cannot find module 'lol'\",\"time\":\"2016-02-25T15:55:25.509Z\",\"v\":0}", "@version" => "1", "@timestamp" => "2016-02-25T15:55:34.004Z", "beat" => {
            "hostname" => "sensu-node-dev", "name" => "sensu-node-dev"
        }, "count" => 1, "fields" => nil, "input_type" => "log", "offset" => 29352, "source" => "/var/log/bunyan/myapp-info.log", "type" => "log", "host" => "sensu-node-dev", "tags" => ["beats_input_codec_plain_applied", "error"]
    }, @metadata_accessors = # < LogStash::Util::Accessors: 0x22cd112d@ store = {
        "beat" => "filebeat", "type" => "log"
    }, @lut = {
        "[type]" => [{
            "beat" => "filebeat", "type" => "log"
        }, "type"], "[beat]" => [{
            "beat" => "filebeat", "type" => "log"
        }, "beat"]
    } > , @cancelled = false > , : level => : error
}

Как видите, в нем говорится response_code => 500 внутренняя ошибка. Я получаю ту же ошибку с обоими format => "json" и format => "message".

Я предполагаю, что это как-то связано с моей конфигурацией (очевидно).