配置Filebeat-Logstash SSL / TLS连接的简便方法

在本教程中,我们将向您展示配置Filebeat-Logstash SSL / TLS连接的简单方法。为了将加密的数据从Filebeat发送到Logstash,您需要启用它们之间的SSL / TLS相互通信。

配置Filebeat-Logstash SSL / TLS连接的简便方法

目录

  • 配置Filebeat-Logstash SSL / TLS连接的简便方法
    • 安装和设置ELK堆栈
    • 安装和设置Filebeat
    • 生成ELK堆栈CA和服务器证书
      • 将密钥转换为标准Elastic Beats PKCS#8密钥格式
    • 配置Filebeat-Logstash SSL / TLS连接
      • 测试Logstash配置
    • 为Logstash SSL / TLS通信配置Filebeat
      • 验证Logstash服务器的证书
      • 测试Filebeat配置
    • 进一步阅读
    • 相关教程

在继续之前,我们假设您已经在收集事件数据的端点上安装并设置了ELK堆栈以及Filebeat。

安装和设置ELK堆栈

您可以按照下面的任何指南来安装和设置Elastic Stack;否则,请执行以下步骤。

在Ubuntu 20.04上安装ELK Stack

在CentOS 8上安装ELK Stack

在Docker容器上部署单节点弹性堆栈集群

安装和设置Filebeat

请按照下面的链接安装和设置Filebeat;

在CentOS 8上安装和配置Filebeat

在Fedora 30 / Fedora 29 / CentOS 7上安装Filebeat

在Ubuntu 18.04 / Debian 9.8上安装和配置Filebeat 7

生成ELK堆栈CA和服务器证书

在此演示中,我们将使用 elasticsearch-certutil

elasticsearch-certutil 是Elastic Stack实用程序,可简化X.509证书和证书签名请求的生成,以与Elastic Stack中的SSL / TLS一起使用。

使用elasticsearch-certutil,可以为特定节点或多个节点生成证书。但是,在此演示中,由于我们仅运行具有所有组件的单个节点Elastic Stack,因此我们将仅为此单个节点生成证书。

要以静默方式生成节点证书,请创建一个YAML文件,以下列格式定义节点的专有名称(可以是主机名)和节点FQDN;

vim $HOME/instances.yml
instances:
  - name: 'elk'
    dns: [ 'elk.kifarunix-demo.com' ]

完成后,运行以下命令以生成ELK堆栈TLS证书。

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --keep-ca-key --pem --in $HOME/instances.yml --out $HOME/elk-cert.zip --days 365

该命令将创建CA密钥和证书,节点密钥和证书,并将其存档在有效期为一年的$ HOME / elk-cert.zip文件中。

列出存档文件的内容;

unzip -l $HOME/elk-cert.zip
Archive:  /root/elk-cert.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2020-10-16 17:48   ca/
     1200  2020-10-16 17:48   ca/ca.crt
     1675  2020-10-16 17:48   ca/ca.key
        0  2020-10-16 17:48   elk/
     1188  2020-10-16 17:48   elk/elk.crt
     1675  2020-10-16 17:48   elk/elk.key
---------                     -------
     5738                     6 files

在上阅读有关elasticsearch-certutil工具的更多信息 Elasticsearch参考页

将证书文件提取到某个目录。在下面的命令中,我们提取到我的主目录。

unzip -d $HOME $HOME/elk-cert.zip

您现在应该拥有这些文件;

ls $HOME/ca/ -1
ca.crt
ca.key
ls $HOME/elk -1
elk.crt
elk.key

确保私钥尽可能安全。

将密钥转换为标准Elastic Beats PKCS#8密钥格式

为了使Beat通过TLS连接到Logstash,您需要将生成的节点密钥转换为Elastic Beat –通过TLS进行Logstash通信所需的PKCS#8标准。

openssl pkcs8 -in $HOME/elk/elk.key -topk8 -nocrypt -out $HOME/elk/elk.pkcs8.key

配置Filebeat-Logstash SSL / TLS连接

接下来,将节点证书$ HOME / elk / elk.crt和Beats标准密钥复制到相关的配置目录中。在此设置中,我们将证书/密钥安装在 /etc/logstash 目录;

cp $HOME/elk/{elk.pkcs8.key,elk.crt} /etc/logstash/

配置Filebeat-Logstash SSL / TLS连接;

vim /etc/logstash/conf.d/test.conf
input {
  beats {
    port => 5044
    ssl => true
    ssl_key => '/etc/logstash/elk.pkcs8.key'
    ssl_certificate => '/etc/logstash/elk.crt'
  }
}
output {
 #  elasticsearch {
 #    hosts => ["https://localhost:9200"]
 #    manage_template => false
 #    index => "ssh_auth-%{+YYYY.MM}"
 #    cacert => "/etc/logstash/logstash.ca.crt"
 #}
 stdout { }
}

上面突出显示的行启用Filebeat-Logstash连接。

保存并退出配置文件。

测试Logstash配置

在运行Logstash之前,最好检查是否有任何配置错误。

/usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

如果一切顺利,您应该从命令输出中看到这样的行;

...
Configuration OK
[2020-10-16T19:03:05,994][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

现在,您可以在调试模式下运行Logstash,以查看是否根据Logstash配置文件出现任何错误。

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf --path.settings /etc/logstash/
...
[INFO ] 2020-10-16 19:07:34.788 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2020-10-16 19:07:34.899 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2020-10-16 19:07:35.212 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
...

如果您看到这条线, Successfully started Logstash API endpoint,那您就好了。

为Logstash SSL / TLS通信配置Filebeat

假设您已经在要从中收集日志的系统上安装了Filebeat,请按以下步骤为Logstash TLS通信配置它:

将上面生成的CA证书复制到远程远程系统。

scp $HOME/ca/ca.crt [email protected]:

将CA证书复制到运行filebeat的远程主机后,请继续配置Filebeat-Logstash SSL / TLS通信。

将复制的CA证书放在某个相关目录中,例如/ etc / filebeat;

cp $HOME/ca.crt /etc/filebeat

现在,通过在Logstash输出上指定CA cert的路径,将Filebeat配置为使用SSL / TLS。

output.logstash:
  hosts: ["elk.kifarunix-demo.com:5044"]
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]

请参阅我们的样本Filebeat配置文件。 确保Logstash主机名与创建证书时使用的FQDN匹配

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/auth.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:
  hosts: ["elk.kifarunix-demo.com:5044"]
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

保存配置文件。

验证Logstash服务器的证书

在运行Filebeat之前,您需要验证Logstash服务器的证书信任。

curl -v --cacert /etc/filebeat/ca.crt https://elk.kifarunix-demo.com:5044

如果可以在Logstash和Filebeat之间建立信任关系,则该命令应从服务器返回空响应。

*   Trying 192.168.57.3:5044...
* TCP_NODELAY set
* Connected to elk.kifarunix-demo.com (192.168.57.3) port 5044 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/filebeat/ca.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=elk
*  start date: Oct 17 15:06:00 2020 GMT
*  expire date: Oct 15 15:06:00 2030 GMT
*  subjectAltName: host "elk.kifarunix-demo.com" matched cert's "elk.kifarunix-demo.com"
*  issuer: CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: elk.kifarunix-demo.com:5044
> User-Agent: curl/7.68.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS alert, close notify (256):
* Empty reply from server
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (52) Empty reply from server

如果您在生成TLS证书时使用了IP地址,请按照以下步骤运行验证:

curl -v --cacert /etc/filebeat/ca.crt https://192.168.57.3:5044

测试Filebeat配置

Filebeat在调试模式下检查是否一切正常。

filebeat -e

您应该看到Filebeat开始收集日志文件并连接到Logstash主机。

...
2020-10-16T20:05:49.564Z	INFO	cfgfile/reload.go:224	Loading of config files completed.
2020-10-16T20:05:49.563Z	INFO	log/harvester.go:299	Harvester started for file: /var/log/auth.log
2020-10-16T20:05:52.543Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2020-10-16T20:05:53.544Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(async(tcp://elk.kifarunix-demo.com:5044))
2020-10-16T20:05:53.547Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2020-10-16T20:05:53.549Z	INFO	[publisher]	pipeline/retry.go:223	  done
2020-10-16T20:05:53.624Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(async(tcp://elk.kifarunix-demo.com:5044)) established

如果您在调试模式下运行Logstash,则应该能够看到日志已填充到标准输出中。

...
{
    "@timestamp" => 2020-10-16T20:05:52.544Z,
         "input" => {
        "type" => "log"
    },
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
      "@version" => "1",
         "agent" => {
             "version" => "7.9.2",
                "name" => "elk.kifarunix-demo.com",
                "type" => "filebeat",
            "hostname" => "elk.kifarunix-demo.com",
        "ephemeral_id" => "1241500c-8f5f-401b-a9f9-1526e8651878",
                  "id" => "726660dc-4b6b-464f-b19b-62f343792a18"
    },
          "host" => {
        "containerized" => false,
         "architecture" => "x86_64",
                  "mac" => [
            [0] "08:00:27:5c:05:2a",
            [1] "08:00:27:7f:84:15"
        ],
                 "name" => "elk.kifarunix-demo.com",
             "hostname" => "elk.kifarunix-demo.com",
                   "os" => {
            "codename" => "focal",
             "version" => "20.04.1 LTS (Focal Fossa)",
                "name" => "Ubuntu",
            "platform" => "ubuntu",
              "family" => "debian",
              "kernel" => "5.4.0-51-generic"
        },
                   "ip" => [
            [0] "10.0.2.15",
            [1] "fe80::a00:27ff:fe5c:52a",
            [2] "192.168.57.3",
            [3] "fe80::a00:27ff:fe7f:8415"
        ],
                   "id" => "57e55f802e0648f885bfe16101cb8d55"
    },
           "log" => {
        "offset" => 6926,
          "file" => {
            "path" => "/var/log/auth.log"
        }
    },
           "ecs" => {
        "version" => "1.5.0"
    },
       "message" => "Oct 16 20:03:50 ubuntu20 sshd[8512]: Failed password for johndoe from 192.168.57.1 port 54196 ssh2"

现在,停止Filebeat和Logstash调试模式,并启动并启用服务以在启动时启动;

systemctl enable --now logstash
systemctl enable --now filebeat

这标志着配置Filebeat-Logstash SSL / TLS连接的简便方法的终结。请享用。

进一步阅读

Filebeat参考:与Logstash的安全通信

Sidebar