Analytics is an integral part of every business operation as it provides a reality check on your current state and enables you to take the right steps to reach your goal. However, collecting pertinent data from different sources for further analysis is a big challenge. The growing complexity of networks, a remote work culture, and the ever-increasing diversity of devices further compound this data collection process.
Before we go into how you can address this challenge, let's briefly talk about the collection process itself.
What are Metrics?
Metrics are the units of measurement of a specific aspect such as the CPU utilization rate of devices, available memory, speed of execution of a given task, and more. These metrics provide information about the current state of a device and the specific functionality you're looking to measure. Also, metrics are generated at regular intervals to help you understand the progression of what you're looking to measure.
In this sense, most metrics consist of four things:
- A timestamp that provides the date and time at which the data was collected.
- Name of the attribute or functionality you're collecting. Example – CPU utilization rate.
- A measurement or the value of that attribute.
- Dimensions are the contextual information of the measurement to help you better understand what this data means. Some examples include the hostname, related attributes, etc.
When you put together all the above points, you'll get a complete metric that provides meaningful information and progression of a device's functionality, health, or performance.
These metrics are generated by a process (also known as a daemon) running on an application or server. This metric is then sent over a network to a server that gathers different metrics and their values and sends them to another tool for further analysis.
Now that you know what metrics are and how they are collected, let's talk about an important tool called collectd that handles the information gathering and transmission process.
What is collectd?
Collectd is a UNIX daemon(process) that collects information and transmits it to a server using plugins. Typically, collectd has multiple plugins configured on a server, and each plugin is responsible for gathering and transmitting a single attribute of a device. For example, you can have one plugin collecting information about a device's workload, another one for gathering data about the same device's memory utilization, another one for its I/O usage, and more. Each plugin is also responsible for sending the information to the configured destination server and can store these values until they are transmitted to the server.
Features of collectd
Here's a detailed look at some salient features of the collectd.
- Easy to Use Collectd is easy to use and doesn't require any complex configuration. All that you have to input is the modules and attributes to monitor. You can even customize the daemon if you like, but that's not important for its operations.
- Flexible and Modular Collectd uses plugins to monitor different attributes, and this is what makes it highly modular. Further, it has no dependencies on any external tool and can run on any operating system as it's written in “C” language. The community claims that collectd works on a wide range of operating systems such as Linux, UNIX, and its variations, macOS, Solaris, AIX, NetBSD, and OpenBSD.
- Leverages Cryptography Starting with version 4.7, collectd's network plugin uses cryptography to encrypt or sign traffic before it's sent to the server. This ability to leverage cryptography adds another security layer to your data. You can even configure the server to accept only encrypted data, so no malicious data can enter your network. More importantly, you can avoid data interceptions without impacting functionality.
- Highly Scalable Collectd is highly scalable, as it can handle any number of hosts or plugins. The highlight is that it uses as few resources as possible to improve efficiency. It even uses a multithreaded layout when needed for parallel data collection.
- Integrates with Many Services Collectd is not a monitoring solution per se but integrates well with a wide range of services and applications. You can use the data collected to create graphs and charts, analyze the data, and generate notifications and alerts.
- Uses a Data Push Model A data push model is where a daemon collects data and pushes it to a server immediately, and collectd follows this model. This means there are no storage or centralized instances to query or store any value. The advantage of this data push model is that you can configure collection and transmission separately.
Note that collectd doesn't analyze the data, simply gathers and transmits them. You'll need specific tools to further analyze this data.
Benefits of collectd
Given that collectd only collects and transmits data, should you use it? Also, why choose collectd when there are so many other similar services?
A look at the benefits below will answer these questions.
- Collectd is written in “C” language and hence, it's highly portable. You don't need any specific scripting language or cron daemon and can work well on embedded systems as well.
- It is highly extensible and can handle hundreds of thousands of metrics at any given time.
- Offers more than 100 plugins that cover almost all attributes of different devices.
- It is well-supported by a thriving community and is well-documented, so you can find any information about collectd. This extensive documentation makes troubleshooting easy.
- Collectd is based on a modular architecture and this allows a plugin to be as small as possible needed to complete the task. This is also why collectd is lightweight and doesn't take up too much memory and space resources.
- It is highly extensible as the data it collects and transmitted can be plugged into any analytics tool.
- You have the flexibility to decide what information you want to collect and the frequency of gathering.
Above everything, collectd is free and open-source, so you can use as many plugins as you want to capture any information for further analysis.
That said, collectd comes with some limitations too. Let's look at them next to get a clear understanding of what this tool can do for your organization.
Limitations of collectd
Understanding what a tool can't do is important for an unbiased decision, and here are the limitations of collectd.
- You're responsible for distributing the agents to collect information from different devices. Managing these agents can be complex, and may often require you to use third-party tools.
- Collectd doesn't generate charts or graphs, so you may need other tools to generate them for you.
- It can't send notifications or alerts if any of the measured values go above or below established thresholds.
- Collectd can't index any data.
The above limitations point to one conclusion – collectd is a highly efficient, free, and lightweight option to collect and transmit data only. It can't handle further processing like comparing the data against thresholds, indexing the data for search, creating graphs and charts, and analyzing the trends.
With this background, let's move on to how you can configure and use collectd.
How to Configure collectd?
The exact configuration depends on your operating system and the metrics you want to collect. In this example, we're going to see how to configure collectd on Ubuntu.
Download collectd
There are two ways to download collectd. You can find a binary package for your operating system or you can get the source code and compile it yourself. Opt for the first option unless you have specific needs and have to make changes to the source file.
Make sure to find a binary that has the latest version/source code of collectd and download it.
Install collectd
The first step is to install collectd on your device. The code is,
get install collectd collectd-utils
This command installs the collectd daemon.
Change the Configuration File
To configure collectd, open the collectd.conf file located at /etc/collectd/collectd.conf.
One of the first things to set up is the hostname of the server to which data must be sent. Open the configuration file, and change this tag.
<pre> Hostname server1 </pre>
You'll also see a parameter called “interval” which is the time that collectd waits before polling the host again. By default, it's set to 24 hours. In most cases, there's no need to change this value, but that said, change this parameter if it is relevant to you. Note that this “interval” parameter applies to all plugins, so keep this in mind while configuring your interval value.
Add the Plugins
As mentioned before, collectd uses plugins to gather information. You have to load these plugins, so collectd knows what to collect. To do this, simply add the plugin name like this,
LoadPlugin memory
LoadPlugin users
LoadPlugin CPU
LoadPlugin logfile
Note that some plugins may require additional configuration. To add this extra information, scroll down the configuration file and create a block for each plugin that needs extra information.
Configure the Plugin
Know which plugins need additional configuration, and create separate blocks for them as needed. Inside these blocks, set the values that are relevant to you.
For example, let's take the logfile plugin. Here you have to give the location for your logs file and you may want to add a timestamp to every log entry. To do this, you can have the following parameters configured.
<Plugin logfile>
File “/etc/collectd/collectd.log”
Timestamp true
</Plugin>
Once you're done with the configurations, head to your domain page. Here, you can see a tree with the collectd information.
That's pretty much the basics about configuring collectd. Every plugin will have a different set of parameters, so check the plugin's documentation if you're looking to implement specific functionality or settings through this configuration.
Next, let's talk a bit about the schema and naming conventions followed in collectd.
Naming Schema
Collectd uses a specific naming schema to uniquely identify different values, and you must understand them to avoid errors and ambiguities. Though it can be confusing at first, the schema is generic and works well across all scenarios.
As a first step towards understanding this schema, let's know what some terminologies mean.
Value
A value is a short form for a value list and it is of the data type value_list_t. This data type is the main data structure for passing the measured values to a host. It is accompanied by another structure called data_set_t to describe the stored values.
The structure of value_list_t is defined in the plugin source file.
You can have one or more data points in this list.
Identifier
An identifier is a unique name you give to a value. This identifier can be any text field with a maximum of 63 characters.
Serialization
You can define a plugin and its values in the form of a block or a serial form, with one value following another. However, make sure the serialization form you use is defined in the src/common.h file.
Data Source
A data source is defined as an entry in the data set. Every data source follows a structure called data_source_t that's defined in the src/daemon/plugin.h file.
A data source construct looks like this.
struct data_source_s
{
char name[DATA_MAX_NAME_LEN];
int type;
double min;
double max;
};
typedef struct data_source_s data_source_t;
Every data source consists of four values, namely:
- Name
- Type
- Minimum value
- Maximum value
These data sources come in handy to categorize the metrics that it collects.
Understanding the naming schemas and the finer details associated with them can help you to make the most out of collectd and its functionality. More importantly, it will help you to configure the tools that will input the data gathered by collectd. For example, if you're sending this data to Graphite or Splunk for further processing, you can ensure that the data source structure and schemas match for a seamless flow of data between these applications.
The above information may seem overwhelming at first, but it's sure to get easier and more familiar as you get down to configuring collectd and transmitting the data to other applications.
Next, let's talk a bit about how collectd fits into a cloud or hybrid environment, given the ever-increasing rate of cloud adoption.
Collectd in the Cloud
Using collectd in the cloud is both a challenge and a necessity. As more organizations move to the cloud today, having a unified view of operations spanning different servers across cloud and on-prem environments becomes important. The good news is that collectd can be extended to the cloud, but not natively. You'll have to use a cloud service like AWS and configure collectd on an EC2 server.
Installing and Configuring collectd on AWS EC2
The exact configuration varies among different Amazon Machine Images (AMIs). For context, an AMI is a virtual appliance used to create a virtual machine within the EC2 environment. The AMI depends on the operating system running on the virtual machine.
In this article, we'll talk about setting up collectd on a Linux AMI. As with any Linux installation, Yellowdog Updater Modifier (YUM) is used
Here are the steps.
- Do an SSH to the AMI. The code for that is,
ssh -i <ssh-key> ec2-user@<aws-externalip>
- Install collectd
yum -y install collectd
- Install the write_http and disk plugins. To clarify, the write_http plugin sends the values to web servers using HTTP POST requests while the disk plugin gathers performance statistics of hard disks.
yum -y install collectd collectd-write_http.x86_64
yum -y install collectd collectd-disk.x86_64
- Next, configure collectd to include the plugins you want.
- Finally, restart collectd.
With this, you can start collecting metrics on AWS. The process is similar for other cloud vendors too with minor variations in the commands. The biggest advantage of using collectd is you can collate metrics from different cloud providers and on-prem servers and analyze them all to get a complete contextual understanding of your environment.
We hope this information helps you to get started on collectd. The good thing about collectd is that the website has extensive documentation about the plugins, so you can run through it to understand parameters about specific plugins. With more than 130 plugins, you have a lot of flexibility to collect anything you want.
Before we end, a quick recap of all that we talked about in this article.
Summary
To conclude, collectd is a popular open-source tool for gathering metrics from different endpoints to assess their health and performance and take proactive steps to mitigate failures. Though collectd doesn't analyze metrics, it gathers them from almost any device at a predetermined frequency. To better understand the impact of using collectd on your business, we had a detailed look into the benefits, limitations, and features, so you can make an informed decision. Finally, we looked at how you can configure and use collectd on the cloud and on-premises to enable you to configure and effectively use collectd. We also touched on some technical aspects of configuring collectd to shorten the learning curve for you.
We're sure this information will come in handy for you to gather and analyze metrics. For more such detailed guides, browse through www.ittsystems.com.
