使用Beats将服务器日志和指标传输到Elasticsearch

Elasticsearch是一个日志分析引擎,允许用户在其仪表板上存储,索引,分析和可视化日志和指标。 Elastic Search使用Kibana在仪表板上可视化数据。本指南描述了如何安装ElasticSearch,Kibana,以及如何使用Beats将日志发送到您的ElasticSearch实例。

Beats是用于从各个端点向Elasticsearch发送日志的载体。它们作为代理程序安装在客户端上,因此您可以将日志发送到您的Elasticsearch实例。

有不同类型的节拍,如下所述。

  1. 文件拍 –分析日志文件
  2. 包拍 –分析网络数据包
  3. Winlogbeat –用于分析Windows事件
  4. 公制节拍 –用于为云环境提供指标
  5. 审核信号 –用于发送有关系统审核数据的信息
  6. 心跳 –用于监视基础结构可用性

在Ubuntu / Debian上安装ElasticSearch

在本指南中,请按照以下步骤在Ubuntu / Debian上安装ElasticSearch。

更新系统

sudo apt update && sudo apt upgrade -y

安装Open-JDK 11(推荐)

sudo apt install default-jdk -y

导入Elasticsearch GPG密钥

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch --no-check-certificate | sudo apt-key add -

添加ElasticSearch存储库

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

安装ElasticSearch

sudo apt update
sudo apt install elasticsearch

配置Elasticsearch以允许远程连接 网络主机 IP到0.0.0.0 /etc/elasticsearch/elasticsearch.yml 文件

$ sudo nano /etc/elasticsearch/elasticsearch.yml

# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#

启动Elasticsearch并启用它

sudo /etc/init.d/elasticsearch start
sudo systemctl enable --now elasticsearch

确保Elasticsearch正在运行。

$ curl http://127.0.0.1:9200

样本输出:

$ curl http://127.0.0.1:9200
{
  "name" : "ubuntu",
  "cluster_name" : "computingforgeeks",
  "cluster_uuid" : "EVzpAqUUSV6wQhO7yiPeKw",
  "version" : {
    "number" : "7.10.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "1c34507e66d7db1211f66f3513706fdf548736aa",
    "build_date" : "2020-12-05T01:00:33.671820Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"

在Ubuntu / Debian上安装Kibana

Kibana提供了一个Web界面,可让您直观地分析所收集的数据。

使用以下步骤在同一主机上安装Kibana。

sudo apt install kibana

配置Kibana以允许外部IP连接。编辑 /etc/kibana/kibana.yaml 创建一个文件并 服务器主机 外部IP或0.0.0.0的选项。

$ sudo nano /etc/kibana/kibana.yaml
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

...
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"

我在同一台主机上安装了Kibana和Elasticsearch,因此无需更改它们。 elasticsearch.hosts 领域。

启动Kibana并启用它

sudo systemctl enable --now kibana

现在,您可以使用服务器的IP(即http://服务器IP:5601)访问端口5601上的Kibana仪表板。

您可能需要允许端口通过防火墙。

sudo ufw allow 5601/tcp

安装Metricbeat

配置Elasticsearch和Kibana之后,您需要在客户端服务器上设置Beats。

这篇文章介绍了如何在客户端服务器上安装Filebeat和Metricbeats。

安装Metricbeat

可以从APT和YUM存储库下载Metricbeat。

易于

  1. 设置Elasticsearch GPG密钥
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

2.安装Metricbeat储存库

sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

3.安装Metricbeat

sudo apt-get update && sudo apt-get install metricbeat

百胜

  1. 下载GPG密钥
sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

2.在以下位置创建一个存储库文件 /etc/yum.repos.d/ 具有以下内容:

sudo tee /etc/yum.repos.d/elastic.repo<<EOF
[elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

3.安装Metricbeat

sudo yum -y install metricbeat

使用Metricbeat将系统指标发送到Elasticsearch

  1. 启用系统模块
sudo metricbeat modules enable system

2.将Metricbeat链接到远程ElasticSearch服务器。编辑 /etc/metricbeat/metricbeat.yml 创建和编辑文件 主办 的详细内容 基巴纳Elasticsearch输出

$ sudo vim /etc/metricbeat/metricbeat.yml

将运行Elasticsearch和kibana的实例的IP添加到主机选项。在这种情况下,Elasticsearch在172.16.56.5主机上运行。

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.

setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "172.16.56.5:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:
Do the same for Elasticsearch
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["172.16.56.5:9200"]

3.初始环境设置–这将加载Kibana仪表板。如果已经设置,则可以省略。 -e 国旗。

sudo metricbeat setup -e

您将看到尝试连接到Elasticsearch主机并创建Kibana仪表板的尝试。

$ sudo metricbeat setup -e
......

2020-12-19T09:56:50.585Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'metricbeat-7.10.1' as ILM is enabled.
2020-12-19T09:56:50.585Z	INFO	eslegclient/connection.go:99	elasticsearch url: http://172.16.56.5:9200
2020-12-19T09:56:50.586Z	INFO	[publisher]	pipeline/module.go:113	Beat name: master
2020-12-19T09:56:50.612Z	INFO	add_kubernetes_metadata/kubernetes.go:71	add_kubernetes_metadata: kubernetes env detected, with version: v1.18.9+k3s1
2020-12-19T09:56:50.620Z	INFO	eslegclient/connection.go:99	elasticsearch url: http://172.16.56.5:9200
2020-12-19T09:56:50.622Z	INFO	[kubernetes]	kubernetes/util.go:138	kubernetes: Using node master discovered by machine-id matching	{"libbeat.processor": "add_kubernetes_metadata"}
2020-12-19T09:56:50.625Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.10.1
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

2020-12-19T09:56:50.681Z	INFO	[index-management]	idxmgmt/std.go:261	Auto ILM enable success.
2020-12-19T09:56:50.683Z	INFO	[index-management.ilm]	ilm/std.go:139	do not generate ilm policy: exists=true, overwrite=false
2020-12-19T09:56:50.683Z	INFO	[index-management]	idxmgmt/std.go:274	ILM policy successfully loaded.
2020-12-19T09:56:50.683Z	INFO	[index-management]	idxmgmt/std.go:407	Set setup.template.name to '{metricbeat-7.10.1 {now/d}-000001}' as ILM is enabled.
2020-12-19T09:56:50.683Z	INFO	[index-management]	idxmgmt/std.go:412	Set setup.template.pattern to 'metricbeat-7.10.1-*' as ILM is enabled.
2020-12-19T09:56:50.683Z	INFO	[index-management]	idxmgmt/std.go:446	Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.10.1 {now/d}-000001} as ILM is enabled.
2020-12-19T09:56:50.683Z	INFO	[index-management]	idxmgmt/std.go:450	Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-12-19T09:56:50.686Z	INFO	template/load.go:183	Existing template will be overwritten, as overwrite is enabled.
2020-12-19T09:56:51.231Z	INFO	template/load.go:117	Try loading template metricbeat-7.10.1 to Elasticsearch
2020-12-19T09:56:52.677Z	INFO	template/load.go:109	template with name 'metricbeat-7.10.1' loaded.
2020-12-19T09:56:52.677Z	INFO	[index-management]	idxmgmt/std.go:298	Loaded index template.
2020-12-19T09:56:52.681Z	INFO	[index-management]	idxmgmt/std.go:309	Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2020-12-19T09:56:52.681Z	INFO	kibana/client.go:119	Kibana url: http://172.16.56.5:5601
2020-12-19T09:56:53.517Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2020-12-19T09:56:53.518Z	INFO	kibana/client.go:119	Kibana url: http://172.16.56.5:5601
2020-12-19T09:58:43.294Z	INFO	instance/beat.go:815	Kibana dashboards successfully loaded.
Loaded dashboards

3.启动Metricbeat并启用它

sudo service metricbeat start
sudo systemctl enable metricbeat

您现在可以在Kibana仪表板中可视化数据 仪表板..

弹性搜索指标

Filebeat设置

您可以使用APT和YUM存储库来设置Filebeat。

易于

#Download GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

#Install apt-transport-https
sudo apt-get install apt-transport-https

#Add repository
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

#Install Filebeat
sudo apt-get update && sudo apt-get install filebeat

百胜

##Download GPG key
sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

##create repo file
sudo tee /etc/yum.repos.d/elastic.repo<<EOF
[elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
sudo tee /etc/yum.repos.d/elasticsearch.repo<<EOF


##Install filebeat
sudo yum install filebeat

将Filebeat连接到Elastic Stack

编辑 /etc/filebeat/filebeat.yml创建文件并添加Elasticsearch远程主机和端口。您还可以添加授权用户的用户名和密码。

output.elasticsearch:
  hosts: ["elasticsearch-IP:9200"]
  username: "filebeat_internal"
  password: "YOUR_PASSWORD" 

另外,在同一文件中设置Kibana详细信息,然后连接到安装Kibana的主机。

setup.kibana:
    host: "mykibanahost:5601"

交换 弹性搜索mykibanahost 服务器使用Elasticsearch服务器IP。

启用Filebeat模块

列出并标识要启用的模块。

filebeat modules list

启用所选模块

filebeat modules enable <module-name>

设置文件节拍环境

filebeat setup -e

启动Filebeat服务

systemctl start filebeat

您需要确保已成功创建仪表盘孔

2020-12-19T11:11:55.731Z	INFO	template/load.go:183	Existing template will be overwritten, as overwrite is enabled.
2020-12-19T11:11:58.580Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2020-12-19T11:11:59.711Z	INFO	template/load.go:117	Try loading template filebeat-7.10.1 to Elasticsearch
2020-12-19T11:12:00.075Z	INFO	template/load.go:109	template with name 'filebeat-7.10.1' loaded.
2020-12-19T11:12:00.075Z	INFO	[index-management]	idxmgmt/std.go:298	Loaded index template.
2020-12-19T11:12:00.077Z	INFO	[index-management]	idxmgmt/std.go:309	Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2020-12-19T11:12:00.078Z	INFO	kibana/client.go:119	Kibana url: http://172.16.56.5:5601
2020-12-19T11:12:03.995Z	INFO	kibana/client.go:119	Kibana url: http://172.16.56.5:5601
2020-12-19T11:13:13.600Z	INFO	instance/beat.go:815	Kibana dashboards successfully loaded.
Loaded dashboards

转到kibana仪表板以可视化您的数据。

elasticsearchfilebeat仪表板

结论

我安装了Elastic Stack,并将Beats配置为获取指标和日志。可以使用与上述相同的过程配置其他拍子。

查看其他文章,了解有趣的监视工具。

如何使用Helm在Kubernetes上安装Netdata

在CentOS上使用IcingaDirector自动执行Icinga2配置| RHEL 8

Sidebar