Intro to Server Monitoring

Whether you are a system administrator at a large tech company or a hobbyist with a raspberry pi at home, monitoring your machines is important. It allows you to find out when a server goes down, when it is running low on disk space, or even when it is time to upgrade to something more powerful.

In this article I am going to give a brief overview of how I setup monitoring on my own servers using a combination of Prometheus, Node Exporter, and Grafana.

Overview

• Prometheus is the main piece of this setup, it pulls metrics together from different services across different machines and collects them all in one place.
• Node exporter is a small application that queries the operating system for a variety of metrics and exposes them over HTTP for other services to consume. Prometheus will query one or several Node Exporter instances to aggregate the metrics.
• Grafana is the cherry on top. It takes all the metrics Prometheus has aggregated and displays them as graphs and diagrams organized into dashboards.

Getting Started

Node Exporter

In my opinion, the first step is to install Node Exporter. You can find the github project at the link below and the download links for each release. At the time of writing the latest release is 0.18.1.

https://github.com/prometheus/node_exporter

Node Exporter is a single binary file that can run from anywhere on the system. This makes installation quite easy. Once it is up and running, point your browser at <server_address>:9100/metrics to see the metrics it is exposing. It isn’t really human readable, but that is the data Prometheus will collect.

Prometheus

The second step is to install Prometheus and point it at the node exporter instance we just installed. The project is linked below along with the downloads for each release. At the time of writing the latest release is 2.13.1.

https://github.com/prometheus/prometheus

Installing Prometheus is a little more involved, but still quite simple. Just like Node Exporter it is a single binary file that can be run from anywhere, but in this case it requires a config file and a folder to store data in. At the end of the article I am including more detail on how I install and setup Prometheus.

To configure Prometheus to pull metrics from the Node Exporter instance we just installed, the config file needs to be modified. For this example all we need to do is add another target URL in the static configs section. It will look something like this:

static_configs:
  - targets: [‘localhost:9090’, ‘localhost:9100’]

This change instructs Prometheus to collect metrics from two services: itself (localhost:9090) and a local Node Exporter instance (localhost:9100).

Once Prometheus is up and running you can open <server_address>:9090 in your browser to view the Prometheus interface. On the /targets page you can see the status of the systems you are pulling metrics from. If all is well, you should see the state of both targets as “UP”.

Using Prometheus alone it is already possible to query the data it is collecting and generate graphs using the tools on the /graph page. This is a good tool to explore the metrics that are being collected and use complex queries to view the data you are interested in. But in this setup, that job will be left to Grafana.

This is as far as I go with Prometheus with this simple guide, but I encourage you to explore more of its capabilities if you are interested in trying it out.

Grafana

Grafana is the last piece of this simple setup, its job is to connect to Prometheus and pull its metrics into graphs and dashboards you can view in your browser. It only needs to interact with Prometheus as it will expose all metrics from all the metric providers it knows about, giving Grafana a single point to connect to.

Just like the other tools in this setup Grafana is open source. The github project can be found here:

https://github.com/grafana/grafana

Unlike the Prometheus and Node Exporter, however, Grafana is not a single binary file. This makes installation a little more complex. Fortunately, there are great installation guides provided by Grafana on their website. Since I am using Ubuntu the installation is as easy as downloading the deb package and installing it:

wget https://dl.grafana.com/oss/release/grafana_6.4.3_amd64.deb
dpkg -i grafana_6.4.3_amd64.deb

Once the installation is complete, you can navigate to <server_address>:3000 to open the web interface and start the setup process. By default the credentials are admin/admin but you will be prompted to change the password on first login.

Setting up Grafana

Now that we have everything installed, we can spend some time setting up Grafana.

Datasource

The first step is to setup a new datasource. This basically tells Grafana where to go to collect metrics. In this case we want to pull metrics from our Prometheus server, so we select the Prometheus datasource. Most options are filled out by default.

It is worth noting that while I am using Grafana to query Prometheus, it can query and process data from a variety of other systems.

Dashboard

Now that we have a datasource, we can configure a dashboard. A dashboard is simply a collection of panels arranged on a page. There are panels of different types, from simple text to pie charts, and each panel can be configured to display a variety of metrics.

There are a lot of options with this, and it can be a bit overwhelming at first. Fortunately, Grafana provides an easy way to install dashboards other users have put together and shared. You can then tweak them to suit what you are looking for.

In this example I am going to use a pre-built dashboard to keep things simple. The one I am using was created by “cordobatyc” and can be found at: https://grafana.com/grafana/dashboards/10795

To install a dashboard, copy the dashboard ID and paste it into the import section in Grafana.

Once installed, you should be able to navigate to the newly added dashboard and view it using the drop-down in the top left corner. When you open the dashboard for the first time it should look something like this.

Plugins

While plugins aren’t strictly necessary, the particular dashboard I used in this example contains a panel that depends on a third-part plugin that isn’t installed by default. You have the choice of either removing that panel (maybe replace it with something else), or install the missing plugin.

In this particular case, the missing plugin is to display pie-charts. And can be found at the link below.

https://grafana.com/grafana/plugins/grafana-piechart-panel

To install the plugin, we can follow the instructions provided on the plugin page. We need to go back to the command line and run the following command:

grafana-cli plugins install grafana-piechart-panel

Restart Grafana once the plugin is installed to complete the installation process:

service grafana-server restart

Once you have installed the missing plugin you can refresh the page and the pie chart panel should now display correctly.

Conclusion

After installing all three pieces and setting up Grafana you should end up with something like this:

By looking at this page you can get a great overview of the status of the server, from how long it has been running for to CPU load and memory usage.

This guide is just an introduction to monitoring, there is plenty more you can do with these projects, including setting up automated alerts when certain thresholds are reached. But you now have the basic building blocks to get some basic monitoring up and running.

If you are interested in exploring this further, I encourage you to read the documentation on Prometheus and Grafana to learn more about what they can do.

Resources

• Node Exporter Github project
• Prometheus Github project
• Grafana Github project
• Grafana Homepage
• Grafana dashboard used
• Grafana Pie-Chart plugin

Annex

Details on installing Node Exporter

For this guide I installed Node Exporter to the following folder: /usr/sbin/node_exporter

And used the example systemd files available in the github project to have it run at startup: https://github.com/prometheus/node_exporter/tree/master/examples/systemd

Details on installing Prometheus

To install Prometheus I placed the executable file in the /usr/sbin folder (just like Node Exporter), put the configuration file under /etc/prometheus and setup a data folder at /var/prometheus. The systemd service file below ensures the service starts at boot.

I also created a non-privileged user to run the Prometheus service.

[Unit]
Description=Prometheus

[Service]
User=prometheus
ExecStart=/usr/sbin/prometheus — config.file=/etc/prometheus/prometheus.yml — storage.tsdb.path=/var/prometheus/data
Environment=USER=prometheus

[Install]
WantedBy=multi-user.target