DevOps - ELK

Quiz

Good log management and data visualization help us keep our systems running well and reliable. The ELK Stack is a great tool for this. It includes Elasticsearch, Logstash, and Kibana. This stack helps us collect, analyze, and show log data from many different sources.

In this chapter, we will look at the parts of the ELK Stack. We will also see how we can set it up for better log management. This setup will help us connect easily with CI/CD pipelines and follow best practices in our DevOps work.

ELK Stack in DevOps

ELK Stack has three main tools: Elasticsearch, Logstash, and Kibana. This stack is really helpful for logging and showing data in DevOps. We can use it to collect, look at, and display log data from many places. This helps us fix problems faster and keep an eye on performance.

Components of the ELK Stack

Following are the three components of the ELK Stack −

Elasticsearch − This is a tool that helps us search and analyze data. It stores and organizes log data. This way, we can search quickly and gather information easily.
Logstash − It is like a pipeline for data. It takes data from different sources, changes it, and sends it to Elasticsearch. Logstash can take many kinds of data, like logs, metrics, and events.
Kibana − This is a tool we use on the web to see our data. It helps us create dashboards, charts, and graphs. With Kibana, we can visualize our log data in a simple way.

Setting Up Elasticsearch for Log Management

We know that Elasticsearch is a distributed search and analytics engine. It is very important for the ELK Stack. To set up Elasticsearch for log management, we can follow these simple steps.

Elasticsearch Installation

For Debian or Ubuntu, use the following commands −

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.x.x-amd64.deb
sudo dpkg -i elasticsearch-7.x.x-amd64.deb

For RPM-based systems, we can run these commands −

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.x.x-x86_64.rpm
sudo rpm -ivh elasticsearch-7.x.x-x86_64.rpm

Start Elasticsearch − We can start Elasticsearch with this command −

sudo service elasticsearch start

Elasticsearch Configuration

Edit elasticsearch.yml − Find this file in /etc/elasticsearch/. Change it like this −

network.host: localhost
http.port: 9200
cluster.initial_master_nodes: ["node-1"]

Verify Installation

Check if Elasticsearch is running. We can do that with this command −

curl -X GET "localhost:9200/"

Index Creation

Now we create an index for log management. We can run this command −

curl -X PUT "localhost:9200/logs/"

This setup helps Elasticsearch to collect, store and manage logs well. It also allows us to search and analyze data in a powerful way.

Configuring Logstash for Data Ingestion

We know that Logstash is a strong tool for processing data. It takes data from different places, changes it, and sends it to a place we choose, like Elasticsearch. To set up Logstash, we need to define input, filter, and output plugins in a configuration file.

Basic Configuration Structure

A standard Logstash configuration file has three main parts −

input {
   # Define input sources
   beats {
      port => 5044
   }
}

filter {
   # Data transformation
   grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
   }
   date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
   }
}

output {
   # Define output destination
   elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "logs-%{+YYYY.MM.dd}"
   }
}

Note the following Key Points −

Use Input Plugins to confirm where the data comes from (like Beats, Kafka, or files).
Filter Plugins help us change the data (like parsing or adding information).
Use Output Plugins to send data to where we want (like Elasticsearch or files).

Example Input Plugin

For getting input from files, we can use −

input {
   file {
      path => "/var/log/myapp/*.log"
      start_position => "beginning"
   }
}

Running Logstash

To start Logstash with our configuration, we can run −

bin/logstash -f path/to/your/logstash.conf

This setup allows Logstash to take logs, process them, and send them to Elasticsearch for more checking.

Using Kibana for Data Visualization

We can use Kibana as a strong tool for visualization. It works very well with Elasticsearch. It helps us explore and show our data in a simple way. We can easily create interactive dashboards, charts, and graphs with its easy-to-use interface.

Following are the Key Features of Kibana −

Dashboards − We can combine many visualizations into one view. This gives us a complete look at our data.
Visualizations − We can create different types of visualizations. This includes line charts, pie charts, and maps.
Search and Filtering − We can use the Lucene query language. This helps us search and filter our logs in a better way.

Getting Started with Kibana

First of all, let's install Kibana −

sudo apt-get install kibana

Next, you need to configure Kibana. We need to change the kibana.yml file to connect it to our Elasticsearch −

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]

Now let's start Kibana −

sudo service kibana start

To access Kibana, open your browser and go to http://localhost:5601.

Creating Visualizations

You can use the Visualize tab to choose the type of visualization you want. Then, pick the index pattern that matches our data. Next, set up metrics and buckets. It would help you decide how to show and group the data.

By using Kibana, you can get real-time insights into how your applications perform and look at log data.

Integrating ELK Stack with CI / CD Pipelines

We can make our monitoring and troubleshooting better by adding the ELK Stack to our CI/CD pipelines. This gives us real-time views of application logs and performance data. Here is how we can add ELK components to our CI/CD workflow −

Continuous Log Collection − We can use Logstash or Filebeat to get logs from our applications during the build and deployment stages. We should format logs the same way so they are easier to read.
Automated Data Ingestion − Let's set up Logstash to take in logs from different sources automatically −

input {
   beats {
      port => 5044
   }
}
filter {
   # Example filter for parsing application logs
   grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
   }
}
output {
   elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "app-logs-%{+YYYY.MM.dd}"
   }
}

Dashboard Creation − We can use Kibana to make dashboards that show application performance and error rates. We should automate how we deploy Kibana dashboards in our CI/CD pipeline.
Alerting − We need to set up alerts in Kibana or use Elasticsearch Watcher. This will help us tell teams about serious issues when we deploy.
Feedback Loop − We can use the logs and metrics from ELK to keep improving our CI/CD process.

By doing these steps, we can have a strong integration of the ELK Stack in our CI/CD pipelines. This will help us see more clearly and respond to problems faster.

Conclusion

In this chapter, we looked at the important parts of the ELK Stack. These parts are Elasticsearch, Logstash, and Kibana. They play a big role in making DevOps better.

We set up Elasticsearch for managing logs. We also configured Logstash for taking in data. Then, we used Kibana for showing the data in a clear way. By putting the ELK Stack together with CI/CD pipelines, we showed how it can help us with monitoring and fixing problems.

Print Page