While we don’t have a log management solution (yet, but stay tuned) in our offerings, we help customers to integrate their existing monitoring platforms into Performance Analyzer. One of the most complete and popular solutions we encounter is Elasticsearch, Logstash and Kibana, also known als ELK stack. Furthermore, we notice that File beat is getting more popular to collect application logs and docker container logs.
As we were setting up our latest test lab to include ELK stack to integrate into Performance Analyzer, we decided to blog about the steps we did. You might get some nice insights or learn some new tricks.
The ELK Stack
If you don’t know the ELK stack yet, let’s start with a quick intro.
The ELK stack consists of Elasticsearch, Logstash, and Kibana. All built as separate projects by the open-source company Elastic these 3 components are a perfect fit to work together. In the meantime its a complete log management and analytics software suite.
Central log management and indexing is a functionality that is simply required to run a data center, cluster, distributed server, and much more.
While Elasticsearch is the backend that functions as a data storage, Logstash is the data processing engine (server-side data-processing pipeline) that ingests data from a variety of sources, transforms it into the required format and forwards it into the data storage (like Elasticsearch), Kibana is the dashboard front-end engine that allows to view and analyse log data (and more) using a browser.
If you want to install the complete ELK stack from scratch, there are plenty of guidelines available. If you don’t want to spend much time and go for a jump start, think about docker hub.
ELK Docker container
While there are plenty of ELK container we figured that one project is maintained very well and is very popular:
https://hub.docker.com/r/sebp/elk/ (5m+ downloads, 800+ stars)
There is also a great documentation about how to use the container as well: elk-dockerAs we want to run multiple container, docker-compose is always the best bet to go for.docker-compose.yml
version: ‘3.7’
services:
elk:
image: sebp/elk
ports:
- "5601:5601"
- "9200:9200"
- "5044:5044"
restart: "unless-stopped"
container_name: opvizor-elk
docker-compose up -d && docker-compose ps
All done, ELK stack in a minimal config up and running as a daemon.
Filebeat
Next thing we wanted to do is collecting the log data from the system the ELK stack was running on.
is a great and fast option to do so.Let’s run the filebeat container without a config to customize the existing ELK stack (create index, add dashboards and more). Make sure to change the Kibana and Elasticsearch host parameter to match your installation. If the container is running on the same host as the ELK stack (docker-compose up) you can also use the container_name within the docker-compose.yml file (if you use the correct network too).To find the network name, simply run
docker network ls
Typically the network is named like the service and _default is added.
There are so many ways to connect docker container, feel free to use alternatives you like more.
docker run
–network=elk_default docker.elastic.co/beats/filebeat:6.6.0
setup -E setup.kibana.host=opvizor-elk:5601
-E output.elasticsearch.hosts=["opvizor-elk:9200"]
If all went well, you should see the following output:
_Loaded index templateLoading dashboards (Kibana must be running and reachable)Loaded dashboards_Loaded machine learning job configurations
Run Filebeat
First thing to do is creating a configuration file for filebeat – you can also download a template:
curl -L -O https://raw.githubusercontent.com/elastic/beats/6.6/deploy/docker/filebeat.docker.yml
This config has out-of-the-box docker support and only needs Elasticsearch target:
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: falsefilebeat.autodiscover:
providers:
- type: docker
hints.enabled: trueprocessors:- add_cloud_metadata: ~output.elasticsearch:
hosts: ‘${ELASTICSEARCH_HOSTS:opvizor-elk:9200}’
- add_cloud_metadata: ~output.elasticsearch:
In our case adding filebeat to the docker-compose.yml makes sense:
filebeat:
image: docker.elastic.co/beats/filebeat:${ELASTIC_VERSION:-6.5.0}
hostname: "opvizor-elk-filebeat"
container_name: filebeat
restart: unless-stopped
user: root
volumes:
- ./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml
- filebeat:/usr/share/filebeat/data
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/containers/:/var/lib/docker/containers/:ro
- /var/log/:/var/log/:ro
environment: - ELASTICSEARCH_HOST=opvizor-elk:9200
disable strict permission checks
command: ["–strict.perms=false"]
volumes: filebeat:
Restart the container
docker-compose down && docker-compose up -d
Kibana
Elasticsearch is ready to receive data, Filebeat is sending data to Logstash, Logstash is processing data and stores it in Elasticsearch.What is missing? Of course Kibana, the UI to analyze and search through the log data.Kibana runs on Port 5601 and has no authentication by default.
As we have only filebeat data incoming right now, create a index filebeat- and use @timestamp
Syslog
Next thing to add are VMware ESXi logs via syslog. Of course, you could setup logstash to receive syslog messages, but as we have Filebeat already up and running, why not using the syslog input plugin of it.VMware ESXi syslog only support port 514 udp/tcp or port 1514 tcp for syslog.
Change the firewall to allow outgoing syslog – 1514 TCP
Restart the syslog service
more information can be found here:
https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.esxi.upgrade.doc/GUID-9F67DB52-F469-451F-B6C8-DAE8D95976E7.html
Change of the docker-compose.yml file to open port 1514
filebeat:
image: docker.elastic.co/beats/filebeat:${ELASTIC_VERSION:-6.5.0}
hostname: "opvizor-elk-filebeat"
container_name: filebeat
restart: unless-stopped
user: root
ports:
- "1514:1514"
volumes: - ./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml
- filebeat:/usr/share/filebeat/data
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/containers/:/var/lib/docker/containers/:ro
- /var/log/:/var/log/:ro
environment: - ELASTICSEARCH_HOST=opvizor-elk:9200
disable strict permission checks
command: ["–strict.perms=false"]
volumes: filebeat:
Change the Filebeat.yml to add the Ports section
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: falsefilebeat.autodiscover:
providers:
- type: docker
hints.enabled: trueprocessors:- add_cloud_metadata: ~output.elasticsearch:
hosts: ‘${ELASTICSEARCH_HOSTS:opvizor-elk:9200}’
filebeat.inputs: - type: syslog
protocol.udp:
host: ":1514"
filebeat.inputs: - type: syslog
protocol.tcp:
host: ":1514"
- add_cloud_metadata: ~output.elasticsearch:
Kibana VMware Syslog
Let’s see if data is incoming:
Looks good, now we can start customizing dashboards and input stream processing.
Performance Analyzer Integration
All data is flowing nicely into our setup, so we can start to add the most important information to a Performance Analyzer dashboard.To integrate Elasticsearch into Performance Analyzer we need to add a data source.
Make sure to set the correct index and use a working url, when adding the Elasticsearch Data Source
Create a new Dashboard for Elasticsearch
You can change the name using the gearwheel icon (1) and add a new table to view Elasticsearch Entries (2).
Select the correct data source and create your Elasticsearch query.
If all done well, you can mix Log error widgets or counts with ESXi or VM information.
Clean up job Curator
You might want to clean up the Elasticsearch content from time to time, to avoid storing unneeded data. There is a nice tool available called Curator, that runs nicely in a docker container.The following example cleans the Elasticsearch database and deletes data older than 30 days:
docker run –rm –network=elk_default -e ES_HOST=opvizor-elk -e DAYS_TO_KEEP=30 omerha/docker-elasticsearch-curator:latest
Of course you can setup a daily cron job to make sure the clean up is done on a regular basis.