Businesses today deal with complex systems and processes, with some spread over many geographies. In turn, this necessitates comprehensive monitoring and subsequent log generation. These logs provide a way for businesses to better understand the health and performance of the different components within their network and infrastructure.
That said, reading through log files to understand the state of systems is not practical. For one, it's time-consuming to read through millions of lines of data. Plus, not all log files may be readable. A good way to overcome the limitations of log files is to send the data of these files to visualization tools that can help to make sense of them.
One such tool is Kibana, and in this article, we take a detailed look into what this tool can do for your business and how you can get started with it.
What is Kibana?
Kibana is a data visualization tool that helps you to make sense of the data stored in Elasticsearch Clusters.
Kibana is developed and run by a company called Elastic, which also has two other products – Elasticsearch and Logstash. Elasticsearch is a database that's used for storing documents, structured, and semi-structured data while Logstash stores, parses, and collects logs. All these three tools work well together to provide a comprehensive understanding of your environment, and together they are called the Elastic Stack.
With Kibana, you can visually see what's happening in your Elastic stack. Specifically, you can do the following:
- Discover documents
- Analyze logs
- Find vulnerabilities
- Search for not-so-evident patterns and insights
- Combine different data into intuitive dashboards
- Monitor the health of your clusters
- Streamline access
Who Can Use Kibana?
Kibana is a good choice for IT admins who have to ensure that all systems are working fine. If you're an IT admin, simply connect your logs to the Elastic Stack, and use Kibana to understand the root cause of problems.
It is also a good choice for data analysts who have to scour through tons of data to identify patterns and insights that can help an organization with its marketing, sales, and operations. With Kibana, you not only get to find the insights you want but more importantly, you get to share them with anyone instantly. You can even create custom dashboards to highlight your findings, so others can understand these insights and their potential impact on your organization.
Kibana can be used by small business owners, CXOs, and just about anyone else who wants to identify specific aspects to help with decision-making, strategic planning, budgeting, and more.
Data Types
One of the powerful aspects of Kibana is that it works well with all kinds of data. You can use it on structured, unstructured, and semi-structured data. It also works well on logs, metrics, time-series data, security insights, and more. Regardless of your data type or where it's located, Kibana can help you visualize them for better understanding.
With this background, let's delve a bit deeper into where Kibana fits into your infrastructure and how it can benefit your organization.
What Can You Do With Kibana?
Kibana is a versatile tool to visualize data and find insights that can help with decision-making. Specifically, Kibana can do the following things for you.
Create a Unique Search Experience
You can create a unique search experience for your end users. You can create a search on your website, domains, databases, and just about anywhere. Furthermore, the search results are filtered based on relevance, recommendations, and more. This functionality works well across any kind of content-based application.
Observe your Environment
Kibana can provide the context and insights you need to better understand all that's happening in your environment. In this sense, it's a single point to unify data from different sources such as logs, metrics, traces, and more.
Stay on Top of Security
With Kibana, you can better manage the alerts and notifications in your infrastructure. You can create alerts for specific events, and this can protect your organization from possible attacks and damage.
Analyze Data
As mentioned earlier, one of the key benefits of Kibana is to ingest large amounts of data from different sources and generate comprehensive insights from them.
Visualize the Data
You can use Kibana to visualize the data collected and analyzed across different sources. Kibana's drag-and-drop interface makes it easy to build charts, tables, graphs, and more.
Now that you have a fair idea of what Kibana can do for your organization, let's get down to using Kibana.
Setting Up Kibana
In this section, we'll talk about how you can set up Kibana. But before that, note that Kibana runs well on Windows, Linux, and Darwin operating systems. Also, you must have the Node.js binaries for these operating systems because Kibana runs only on Node.js.
Moreover, you must configure Kibana to run against an Elasticsearch node, as this is the supported configuration. What this means is that Kibana and Elasticsearch must be of the same version.
After you have all these dependencies set up, you're all set to install Kibana.
Download Kibana
If you're going for the hosted version, you can access Kibana on the cloud. All that you have to do is signup and start using it.
On the other hand, you can download Kibana on-premises. Just ensure that the operating systems are 64-bit only, especially if you want to use the latest versions.
The easiest option is to download and install the “tar” package. Download “tar.gz” for Linux and Darwin operating systems and “tar.zip” for Windows.
If you plan to run Kibana on Debian and Ubuntu, download the “deb” package. For Red Hat and OpenSUSE, go for the rpm package. You can even run Kibana as a Docker container, and to do this, download Kibana from the Elastic Docker Registry.
Install Kibana
To install Kibana on Linux or macOS, use the below command.
tar -xzf kibana-8.4.2-linux-x86_64.tar.gz
For Windows, simply unzip the tar.zip file, and this will create a folder called “Kibana-version-windows-x86_64”
Next, head to the command line and enter the below code to install Kibana.
C:\kibana-8.4.2-windows-x86_64
.\bin\kibana.bat
Before starting Kibana, make sure the Elasticsearch is running. When you run Kibana for the first time, the system generates a unique link in your terminal. Copy this unique link and paste it into your browser. In the page that opens, look for the button and click it to connect Kibana to the Elasticsearch that's already running in the background.
Finally, log in with the username and password.
Once you install Kibana, it's time to configure it, and you can do so using the config file.
Configuring Kibana
Kibana.yml is the config file that Kibana uses. It is located by default in the “Kibana-version-windows-x86_64” folder, and you can always change its location using the “KBN_PATH_CONF” environment variable. By default, Kibana runs on localhost: 5601.
Here's a look at some of the important variables in the config file.
- Console.ui.enabled The server regenerates assets on startup. The default is true.
- Csp.strict Blocks access to older browsers, as they may suffer from security issues. The default value is true.
- Elasticsearch.maxsockets This value is the maximum number of sockets that can be used to connect to Elasticsearch. The default value is infinity.
- Logging.root This is a preconfigured dedicated logger that logs everything by default at “info”. If you make any changes to your logging, you must explicitly configure this variable.
- map.emsURL If you have a self-hosted Elastic Maps Server, specify its URL in this config variable.
- migrations.batchSize The maximum number of documents that can be migrated at one-time.
- Newsfeed.enabled Enables the newsfeed for Kibana's UI notification center.
- Path.data This is where Kibana stores all persistent data that are not stored on Elasticsearch.
- Pid.file Specifies the path of Kibana's process ID file.
- Ops.interval The time interval (in milliseconds) to sample and process metrics.
The above variables are some of the more important ones, and their default values are enough to help you use Kibana. However, if you want any specific behavior or change, make changes to any of these variables.
Next, let's talk about how you can start and stop Kibana.
Starting and Stopping Kibana
The easiest way to start Kibana is through the command line, but you must've installed a tar.gz package on Linux or Darwin, or a .zip package for Windows for this method.
The code for starting Kibana on Linux or Darwin is,
./bin/kibana
For Windows, the code is
.\bin\kibana.bat
You can stop Kibana by pressing CTRL+C. Until then, Kibana will run in the foreground and print its logs to stdout.
Another option is to configure Kibana to start as soon as the system starts, and the code for this configuration is,
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kibana.service
Note that the above code is for Debian and RPM packages. To manually start and stop Kibana, the code is,
sudo systemctl start kibana.service
sudo systemctl stop kibana.service
The one disadvantage here is you'll not get any notification about whether Kibana was started successfully or not. You can only check the log files to know this.
If you have a subscription to cloud-hosted Kibana, simply head to the browser, enter the URL, and log in with your username and password to get started.
For a self-managed deployment of Kibana Cloud, head to http://YOURDOMAIN.com:5601 and log in with your credentials.
If you have any issues with starting Kibana, check the log files, as this is where you can know what's wrong.
Next, let's see how you can manage this versatile tool.
Managing Kibana
If you're an IT admin managing access for your organization, a good first step is to secure access. Make sure that only authorized employees can view Kibana's dashboards, add data, or make changes to its configuration. You can even set varying access levels based on an employee's designation and department.
Control Access with Spaces
Kibana offers a feature called Spaces, which is essentially a unique workspace for a specific group or department.
As an admin, here's how you can create spaces.
- Open Kibana and navigate to Spaces
- Click the option – “Create a Space”
- Provide a unique name to the space. Note that this must also be easy and intuitive to identify
- Click on “Create Space”
Next, select the features that can be visible to members of this space. Choose from Analytics. Observability, Management, Security, and more. Ensure that the features you enable are sufficient for the group to be productive and complete their respective tasks.
Streamline Privileges
Next, create a role to provide the required privileges. In general, provide only read privileges to data and dashboards and restrict the edit or delete privileges to the IT admins.
To create a role,
- Open Kibana and select Slack Management
- Navigate to Security and choose Roles
- Click the “Create Role” option
- Provide a unique name for this role. Remember, this is not an individual, but a role that can be assigned to any individual
- Head to the space you created earlier. Expand the feature and provide Read privilege. Map the recently-created Kibana role to this privilege
Now that your spaces and roles are ready, it's time to create a user who can access them.
Create a User
To create a user who can use the above role,
- Go to Slack Management > Security > Users
- Click the “Create User” option
- Provide a username, preferably one that's easy to remember, and create a default strong password
- Assign the role you created earlier to this user
- Finally, click “Create User”
With this, you've created users and streamlined access to them. Lastly, log out of Kibana and log in back again with the credentials of the user you just created. Look at what the user can access with Kibana and ensure that this is in line with the agreed privileges for the user.
Now, your users are all set to use Kibana.
Using Kibana
Kibana is simple to use and fairly intuitive. All that you have to do is log into Kibana with your username and password, and view the dashboards relevant to your work. There aren't a lot of things to do if you only have read privileges, so use Kibana to understand the graphs and charts it displays for you.
Adding Data
If you have the privilege to add data, there are many ways to do it. The easiest option is to add data to the Elastic Search using the available integrations, such as the website search crawler. You can even upload a data file through Integrations. However, the file must be in JSON, CSV, or TSV format with a max size of 100 MB.
You can also use programming languages like JavaScript, Python, and Ruby to add data.
Create Dashboards
Visualizing data in easily understandable formats is one of the core uses of Kibana, and dashboards are an integral part of these visualizations. Here are the steps to create them.
- Go to the Main Menu and navigate to Dashboard
- Click on “Create dashboard”, and this will automatically take you to the edit mode
- Provide a title for your dashboard. Make sure it's easy to understand
Next, add panels to your dashboard in one of the following ways.
- You can add the panels stored in your Visualize Library
- Search results directly from the Discover feature
- Create panels
Now, we'll see the steps for creating panels. The easiest way is to use the drag-and-drop editor and choose from the editor and panel type, depending on what kind of visualization you want to create. Add context to your panels with a relevant title and a brief description of what's displayed. Finally, create a query and add it to the panel.
Once you're done creating panels, save them in your Visualize Library, so they can be reused later. On your dashboard, arrange the different panels in the order you want.
Generate and Share Reports
You can quickly generate and share reports with colleagues and team members using Kibana.
Some of the kinds of reports you can generate include PDFs, PNGs, CSVs, permalinks, JSON, Canvas work pads, embedded code, and more.
You have complete control over the layout and size of the reports. Simply resize the container before the reports are generated. Note that this may require some trial and error.
Here are the steps to create and share reports.
- Open the main menu and navigate to dashboards, saved searches, or panels that you want to convert to reports
- On the toolbar, click Share and choose from PDF, CSV, and PNG formats. If you choose the PDF format, select the “Optimize for printing” option
- Once you've selected the format, generate it
- A message appears on the screen, and from it, select “Download report”
This will download the report to your device, and you can easily share it with anyone. To manage the reports, head to Stack Management > Reporting
Note that you can also create permalinks of the reports and share them directly through IMs or emails.
With these basics, you're all set to start using Kibana.
Before we end, a few final thoughts on this versatile tool.
Final Thoughts
In all, Kibana is an advanced data visualization tool that enables you to quickly create dashboards and panels for a better understanding of all that's going on in your stack. Kibana takes data from the ElasticSearch stack, though you can control what data must be displayed with appropriate queries. In this article, we saw in detail how you can configure Kibana and use it to create the visualizations you need. We hope this information acts as a good starting point to explore this tool.
For more such guides, browse through www.ittsystems.com
