配置Filebeat-Logstash SSL / TLS連接的簡便方法

在本教程中,我們將向您展示配置Filebeat-Logstash SSL / TLS連接的簡單方法。為了將加密的數據從Filebeat發送到Logstash,您需要啟用它們之間的SSL / TLS相互通信。

配置Filebeat-Logstash SSL / TLS連接的簡便方法

目錄

  • 配置Filebeat-Logstash SSL / TLS連接的簡便方法
    • 安裝和設置ELK堆棧
    • 安裝和設置Filebeat
    • 生成ELK堆棧CA和服務器證書
      • 將密鑰轉換為標準Elastic Beats PKCS#8密鑰格式
    • 配置Filebeat-Logstash SSL / TLS連接
      • 測試Logstash配置
    • 為Logstash SSL / TLS通信配置Filebeat
      • 驗證Logstash服務器的證書
      • 測試Filebeat配置
    • 進一步閱讀
    • 相關教程

在繼續之前,我們假設您已經在收集事件數據的端點上安裝並設置了ELK堆棧以及Filebeat。

安裝和設置ELK堆棧

您可以按照下面的任何指南來安裝和設置Elastic Stack;否則,請執行以下步驟。

在Ubuntu 20.04上安裝ELK Stack

在CentOS 8上安裝ELK Stack

在Docker容器上部署單節點彈性堆棧集群

安裝和設置Filebeat

請按照下面的鏈接安裝和設置Filebeat;

在CentOS 8上安裝和配置Filebeat

在Fedora 30 / Fedora 29 / CentOS 7上安裝Filebeat

在Ubuntu 18.04 / Debian 9.8上安裝和配置Filebeat 7

生成ELK堆棧CA和服務器證書

在此演示中,我們將使用 elasticsearch-certutil

elasticsearch-certutil 是Elastic Stack實用程序,可簡化X.509證書和證書籤名請求的生成,以與Elastic Stack中的SSL / TLS一起使用。

使用elasticsearch-certutil,可以為特定節點或多個節點生成證書。但是,在此演示中,由於我們僅運行具有所有組件的單個節點Elastic Stack,因此我們將僅為此單個節點生成證書。

要以靜默方式生成節點證書,請創建一個YAML文件,以下列格式定義節點的專有名稱(可以是主機名)和節點FQDN;

vim $HOME/instances.yml
instances:
  - name: 'elk'
    dns: [ 'elk.kifarunix-demo.com' ]

完成後,運行以下命令以生成ELK堆棧TLS證書。

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --keep-ca-key --pem --in $HOME/instances.yml --out $HOME/elk-cert.zip --days 365

該命令將創建CA密鑰和證書,節點密鑰和證書,並將其存檔在有效期為一年的$ HOME / elk-cert.zip文件中。

列出存檔文件的內容;

unzip -l $HOME/elk-cert.zip
Archive:  /root/elk-cert.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2020-10-16 17:48   ca/
     1200  2020-10-16 17:48   ca/ca.crt
     1675  2020-10-16 17:48   ca/ca.key
        0  2020-10-16 17:48   elk/
     1188  2020-10-16 17:48   elk/elk.crt
     1675  2020-10-16 17:48   elk/elk.key
---------                     -------
     5738                     6 files

在上閱讀有關elasticsearch-certutil工具的更多信息 Elasticsearch參考頁

將證書文件提取到某個目錄。在下面的命令中,我們提取到我的主目錄。

unzip -d $HOME $HOME/elk-cert.zip

您現在應該擁有這些文件;

ls $HOME/ca/ -1
ca.crt
ca.key
ls $HOME/elk -1
elk.crt
elk.key

確保私鑰儘可能安全。

將密鑰轉換為標準Elastic Beats PKCS#8密鑰格式

為了使Beat通過TLS連接到Logstash,您需要將生成的節點密鑰轉換為Elastic Beat –通過TLS進行Logstash通信所需的PKCS#8標準。

openssl pkcs8 -in $HOME/elk/elk.key -topk8 -nocrypt -out $HOME/elk/elk.pkcs8.key

配置Filebeat-Logstash SSL / TLS連接

接下來,將節點證書$ HOME / elk / elk.crt和Beats標準密鑰複製到相關的配置目錄中。在此設置中,我們將證書/密鑰安裝在 /etc/logstash 目錄;

cp $HOME/elk/{elk.pkcs8.key,elk.crt} /etc/logstash/

配置Filebeat-Logstash SSL / TLS連接;

vim /etc/logstash/conf.d/test.conf
input {
  beats {
    port => 5044
    ssl => true
    ssl_key => '/etc/logstash/elk.pkcs8.key'
    ssl_certificate => '/etc/logstash/elk.crt'
  }
}
output {
 #  elasticsearch {
 #    hosts => ["https://localhost:9200"]
 #    manage_template => false
 #    index => "ssh_auth-%{+YYYY.MM}"
 #    cacert => "/etc/logstash/logstash.ca.crt"
 #}
 stdout { }
}

上面突出顯示的行啟用Filebeat-Logstash連接。

保存並退出配置文件。

測試Logstash配置

在運行Logstash之前,最好檢查是否有任何配置錯誤。

/usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

如果一切順利,您應該從命令輸出中看到這樣的行;

...
Configuration OK
[2020-10-16T19:03:05,994][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

現在,您可以在調試模式下運行Logstash,以查看是否根據Logstash配置文件出現任何錯誤。

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf --path.settings /etc/logstash/
...
[INFO ] 2020-10-16 19:07:34.788 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2020-10-16 19:07:34.899 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2020-10-16 19:07:35.212 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
...

如果您看到這條線, Successfully started Logstash API endpoint,那您就好了。

為Logstash SSL / TLS通信配置Filebeat

假設您已經在要從中收集日誌的系統上安裝了Filebeat,請按以下步驟為Logstash TLS通信配置它:

將上面生成的CA證書複製到遠程遠程系統。

scp $HOME/ca/ca.crt [email protected]:

將CA證書複製到運行filebeat的遠程主機後,請繼續配置Filebeat-Logstash SSL / TLS通信。

將複製的CA證書放在某個相關目錄中,例如/ etc / filebeat;

cp $HOME/ca.crt /etc/filebeat

現在,通過在Logstash輸出上指定CA cert的路徑,將Filebeat配置為使用SSL / TLS。

output.logstash:
  hosts: ["elk.kifarunix-demo.com:5044"]
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]

請參閱我們的樣本Filebeat配置文件。 確保Logstash主機名與創建證書時使用的FQDN匹配

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/auth.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:
  hosts: ["elk.kifarunix-demo.com:5044"]
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

保存配置文件。

驗證Logstash服務器的證書

在運行Filebeat之前,您需要驗證Logstash服務器的證書信任。

curl -v --cacert /etc/filebeat/ca.crt https://elk.kifarunix-demo.com:5044

如果可以在Logstash和Filebeat之間建立信任關係,則該命令應從服務器返回空響應。

*   Trying 192.168.57.3:5044...
* TCP_NODELAY set
* Connected to elk.kifarunix-demo.com (192.168.57.3) port 5044 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/filebeat/ca.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=elk
*  start date: Oct 17 15:06:00 2020 GMT
*  expire date: Oct 15 15:06:00 2030 GMT
*  subjectAltName: host "elk.kifarunix-demo.com" matched cert's "elk.kifarunix-demo.com"
*  issuer: CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: elk.kifarunix-demo.com:5044
> User-Agent: curl/7.68.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS alert, close notify (256):
* Empty reply from server
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (52) Empty reply from server

如果您在生成TLS證書時使用了IP地址,請按照以下步驟運行驗證:

curl -v --cacert /etc/filebeat/ca.crt https://192.168.57.3:5044

測試Filebeat配置

Filebeat在調試模式下檢查是否一切正常。

filebeat -e

您應該看到Filebeat開始收集日誌文件並連接到Logstash主機。

...
2020-10-16T20:05:49.564Z	INFO	cfgfile/reload.go:224	Loading of config files completed.
2020-10-16T20:05:49.563Z	INFO	log/harvester.go:299	Harvester started for file: /var/log/auth.log
2020-10-16T20:05:52.543Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2020-10-16T20:05:53.544Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(async(tcp://elk.kifarunix-demo.com:5044))
2020-10-16T20:05:53.547Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2020-10-16T20:05:53.549Z	INFO	[publisher]	pipeline/retry.go:223	  done
2020-10-16T20:05:53.624Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(async(tcp://elk.kifarunix-demo.com:5044)) established

如果您在調試模式下運行Logstash,則應該能夠看到日誌已填充到標準輸出中。

...
{
    "@timestamp" => 2020-10-16T20:05:52.544Z,
         "input" => {
        "type" => "log"
    },
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
      "@version" => "1",
         "agent" => {
             "version" => "7.9.2",
                "name" => "elk.kifarunix-demo.com",
                "type" => "filebeat",
            "hostname" => "elk.kifarunix-demo.com",
        "ephemeral_id" => "1241500c-8f5f-401b-a9f9-1526e8651878",
                  "id" => "726660dc-4b6b-464f-b19b-62f343792a18"
    },
          "host" => {
        "containerized" => false,
         "architecture" => "x86_64",
                  "mac" => [
            [0] "08:00:27:5c:05:2a",
            [1] "08:00:27:7f:84:15"
        ],
                 "name" => "elk.kifarunix-demo.com",
             "hostname" => "elk.kifarunix-demo.com",
                   "os" => {
            "codename" => "focal",
             "version" => "20.04.1 LTS (Focal Fossa)",
                "name" => "Ubuntu",
            "platform" => "ubuntu",
              "family" => "debian",
              "kernel" => "5.4.0-51-generic"
        },
                   "ip" => [
            [0] "10.0.2.15",
            [1] "fe80::a00:27ff:fe5c:52a",
            [2] "192.168.57.3",
            [3] "fe80::a00:27ff:fe7f:8415"
        ],
                   "id" => "57e55f802e0648f885bfe16101cb8d55"
    },
           "log" => {
        "offset" => 6926,
          "file" => {
            "path" => "/var/log/auth.log"
        }
    },
           "ecs" => {
        "version" => "1.5.0"
    },
       "message" => "Oct 16 20:03:50 ubuntu20 sshd[8512]: Failed password for johndoe from 192.168.57.1 port 54196 ssh2"

現在,停止Filebeat和Logstash調試模式,並啟動並啟用服務以在啟動時啟動;

systemctl enable --now logstash
systemctl enable --now filebeat

這標誌着配置Filebeat-Logstash SSL / TLS連接的簡便方法的終結。請享用。

進一步閱讀

Filebeat參考:與Logstash的安全通信

Sidebar