How to Deploy Prometheus + Grafana + Node Exporter to Monitor Cloud Servers

Metrics are among the most important components of observability. If you’re finding out about outages from your customers rather than your monitoring systems, it’s time to think about redesigning your monitoring strategy. 

At first, it may seem that deploying and configuring such a stack in the cloud is no different from doing it in a traditional infrastructure. But trust me, that’s only at first. 

In this article, we will walk you through the entire workflow and provide practical tips regarding what to watch out for in order to keep your operation efficient. 

Infrastructure Preparation

Choosing a Configuration

First, you need to decide on your deployment environment. For example, for test or dev environments, there’s no need to distribute stack components across different nodes. It’s enough to place them on a single cloud server and properly configure their interaction. 

Of course, if the virtual server goes down for some reason, the entire stack will go down with it. Sure, that’s far from ideal, but it’s not critical in a test environment.

‍

Need a server that matches these specs? 3HCloud offers transparent, pay-as-you-go cloud servers with fast deployment. Choose your server →

Generally speaking, 2 vCPUs should be enough to collect metrics from a small number of targets at 15–30 second intervals. This will also be sufficient for rendering basic dashboards with a low number of stored and indexed time series.

Memory requirements are largely determined by the fact that the index is kept in RAM. So, the more metrics there are, the more memory is consumed. Once it’s exhausted, the daemon will start using swap space first, which leads to a sharp drop in performance. Eventually, the process will be terminated by the OOM killer.

It is also important to choose the right retention strategy, which will define how long raw time series will be stored in the local database. Typically, one active host with Node Exporter can generate between 300 MB and 500 MB of data over 30 days, depending on the number of collected metrics and the scrape interval. 

A retention period of 7–15 days is usually more than enough for a test environment. In a production environment, however, a different approach is required. Placing the entire stack on a single virtual machine is not recommended, as the failure of that node would leave you without monitoring at the exact moment you need it most. At a minimum, the components can be split as follows:

• Prometheus VM collects metrics, executes PromQL queries, and manages rules.
• Grafana VM visualizes the collected data.
• Node Exporter is installed on every cloud machine that needs to be monitored.
• Alertmanager receives alerts from Prometheus and can be installed on the same VM.

In such a setup, Grafana doesn’t store the collected metrics. Instead, it only connects to Prometheus as a data source and queries it to build dashboards. As a result, if Grafana VM goes down for some reason, it won’t affect your metric history in any way.

A Prometheus failure is the only event that can make local historical data unavailable, which is why production infrastructure often uses several identically configured Prometheus instances that collect the same data. This enables it to continue operating even if one instance fails. Alertmanager can also operate in a cluster, increasing the availability of the notification system.

Disclaimer: Multiple Prometheus instances do not form a single distributed TSDB. Each instance stores its own copy of the data. For long-term storage and a single query endpoint, separate solutions such as Thanos, Mimir, or VictoriaMetrics are commonly used. We won’t cover these solutions in this article.

As for virtual machine specifications, for a production environment with 10–50 virtual machines running Node Exporter and a scrape interval of 15–30 seconds, a reasonable starting point would be 4 vCPU / 8 GB RAM / 100 GB NVMe SSD.

Creating a User

Cloud servers created in 3HCloud give you access as the root superuser by default. While this gives you full control, it also creates a security hole and introduces the risk of accidentally breaking something. That’s why we recommend creating a separate user right away:

useradd -m -s /bin/bash monitor

The new user doesn’t have a password by default, so you should set one:

passwd monitor

To allow the user to perform administrative tasks, add it to the sudo group:

usermod -aG sudo monitor

Switching to SSH Key Authentication

If you select password authentication when creating the server, you will quickly become a target for an entire army of bots continuously scanning IP addresses and attempting brute-force attacks against the target host. The simplest countermeasure is to completely abandon password authentication and use an SSH key instead. 

Even if you’ve never done this before, there’s nothing to worry about. It’s pretty simple.

On the local machine, generate a new key pair:

ssh-keygen -t ed25519 -C "monitor@node1" -f ~/.ssh/id_ed25519_node1 

Two files will be created as a result:

• ~/.ssh/id_ed25519_node1 - private key;
• ~/.ssh/id_ed25519_node1.pub - public key.

The private key must never leave the machine on which it was generated, while the public part can be copied to the server:

ssh-copy-id -i ~/.ssh/id_ed25519_node1.pub [email protected]

The utility will automatically add it to the  ~/.ssh/authorized_keys file of the monitor user and set the required permissions. Now you can safely log in using the key:

ssh -i ~/.ssh/id_ed25519_node1 [email protected]

Disabling Password Authentication

As the final step, let’s close the potential security loophole by completely disabling SSH password access and the ability to connect as the superuser. After that, the only way to access the server will be with the private key of the user we created earlier.

Important: Don't close your current SSH session until all the steps in this section are complete. Open a second terminal window and verify that key-based login works for the unprivileged user and that sudo commands are available. This will be the only way to retain access if you make a configuration mistake!

On a regular, non-cloud server, we would simply edit /etc/ssh/sshd_config. However, starting with Ubuntu 22.04 LTS, you’ll find the following directive near the top of this configuration file:

Include /etc/ssh/sshd_config.d/*.conf

Cloud providers use this mechanism to set custom password values and other parameters that allow the server to be managed through their own control panel. When creating a server, 3HCloud uses Cloud-Init to place a 99-root-ssh.conf configuration file there, containing the following directives:

/etc/ssh/sshd_config.d/99-root-ssh.conf:1:PermitRootLogin yes
/etc/ssh/sshd_config.d/99-root-ssh.conf:2:PasswordAuthentication yes
/etc/ssh/sshd_config.d/99-root-ssh.conf:3:KbdInteractiveAuthentication yes
/etc/ssh/sshd_config.d/99-root-ssh.conf:4:ChallengeResponseAuthentication yes
/etc/ssh/sshd_config.d/99-root-ssh.conf:5:UsePAM yes

There’s no point fighting this. It’s easier to add our own drop-in with a lower numeric prefix. The sshd daemon follows the “first value encountered wins” logic, which gives us the ability to override the cloud directives with our own:

sudo tee /etc/ssh/sshd_config.d/10-hardening.conf > /dev/null <<'EOF'
# Disallow all root logins, including key-based login
PermitRootLogin no

# Disable password authentication
PasswordAuthentication no

# Disable password login through PAM
KbdInteractiveAuthentication no

# Enable key-based authentication
PubkeyAuthentication yes
EOF

And finally, don’t forget about file permissions:

sudo chmod 600 /etc/ssh/sshd_config.d/10-hardening.conf

As a result, the 10- prefix takes precedence over 99-, which allows us to completely eliminate the influence of values set by Cloud-Init. Restart the daemon:

sudo sshd -t && sudo systemctl reload ssh

Installing the Stack

Let’s start with the simplest option: the classic installation of binaries and management via systemd. Despite the common opinion that systems like these should be deployed in containers, this approach has some very practical advantages, especially when it comes to metric collection agents.

Node Exporter runs directly on the hosts and has access to all directories without going through abstraction layers. Without containers, network and firewall configuration becomes significantly simpler. On top of that, systemd allows us to create a fine-grained kernel-level “sandbox” for each running service. 

There is, however, one significant downside. While updating a couple of dozen or even hundreds of containers can simply mean changing a tag and running docker-compose up -d in practice, here, you’ll have to both distribute new executable files and restart units. Ideally, this shouldn’t be done manually but with a configuration management system.

The initial dilemma is whether to install from the system repository or download standalone binaries. The former is convenient because everything works out of the box, but there is a catch: package versions are frozen at the time of the OS release and receive only security updates afterward.

Prometheus is actively developed, so it makes sense to get its builds directly from GitHub. Grafana, on the other hand, has its own APT repository through which all updates are delivered as usual. No need to make things more complicated here.

Node Exporter

Instead of configuring the metric collector right away, it makes sense to set up the hosts first. We will assume that all of them are running a fresh Ubuntu 26.04 installation (the current version at the time this article was written).

Note: We’re using the 10.0.0.11 IP address purely as an example. You need to replace this value with your own external IP address. This will allow you to build a system in which metrics can be securely collected from outside. If this isn’t required, the external address can be replaced with a private address at any time without changing the overall architecture.

First of all, we need to create a system user under which Node Exporter will run. It doesn’t need a home directory or the ability to log in, so:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin
node_exporter

Download the release:

VER=1.12.1

curl -fsSLO https://github.com/prometheus/node_exporter/releases/download/v${VER}/node_exporter-${VER}.linux-amd64.tar.gz 

Download the checksums together with the tarball. This is basic hygiene, since we are installing a binary with permission to read the entire system:

curl -fsSLO https://github.com/prometheus/node_exporter/releases/download/v${VER}/sha256sums.txt 

Verify the checksum:

grep "node_exporter-${VER}.linux-amd64.tar.gz" sha256sums.txt | sha256sum -c -

Now that we have made sure the executable files have not been tampered with, it’s time to extract the archive:

tar xzf node_exporter-${VER}.linux-amd64.tar.gz

Install it into the system as the superuser:

sudo install -o root -g root -m 0755 node_exporter-${VER}.linux-amd64/node_exporter /usr/local/bin/ 

sudo mkdir -p /etc/node_exporter /var/lib/node_exporter/textfile

sudo chown -R node_exporter:node_exporter /var/lib/node_exporter

Now it’s time to make sure the connection between Prometheus and Node Exporter is protected.

Important: We should note right away that we will generate individual certificates for the sake of this example. However, if you have a large number of virtual machines, it makes sense to deploy a small Private PKI, generate your own root certificate, and use it to sign the certificates you issue. This is beyond the scope of our guide.

sudo openssl req -x509 -nodes -newkey rsa:2048 -sha256 -days 825 \
  -keyout /etc/node_exporter/node1.key \
  -out /etc/node_exporter/node1.crt \
  -subj "/CN=node1" \
  -addext "subjectAltName=IP:10.0.0.11,DNS:node1" \
  -addext "keyUsage=critical,digitalSignature,keyEncipherment" \
  -addext "extendedKeyUsage=serverAuth"

Check the extensions:

sudo openssl x509 -in /etc/node_exporter/node1.crt -noout -text | grep -A1 'Subject Alternative Name'

Do not forget about permissions:

sudo chown root:node_exporter /etc/node_exporter/node1.key 

sudo chmod 640 /etc/node_exporter/node1.key 

sudo chmod 644 /etc/node_exporter/node1.crt

To avoid storing the password in plain text, we can convert it into a bcrypt hash:

sudo apt install -y apache2-utils 

htpasswd -nBC 12 "" | tr -d ':\n'

Create an additional configuration file:

sudo nano /etc/node_exporter/web-config.yml:

tls_server_config:
  cert_file: /etc/node_exporter/node1.crt
  key_file: /etc/node_exporter/node1.key
  min_version: TLS12

basic_auth_users:
  prometheus: 'your_bcrypt_hash_here'

Now create a systemd unit:

sudo nano /etc/systemd/system/node_exporter.service

[Unit]
Description=Prometheus Node Exporter
Documentation=https://github.com/prometheus/node_exporter
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=10.0.0.11:9100 \
  --web.config.file=/etc/node_exporter/web-config.yml \
  --collector.textfile.directory=/var/lib/node_exporter/textfile \
  --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run/credentials/.+)($|/)
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s

# Sandbox
NoNewPrivileges=true
CapabilityBoundingSet=
ProtectSystem=strict
ProtectHome=read-only
PrivateTmp=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallArchitectures=native
SystemCallFilter=@system-service 
SystemCallFilter=adjtimex clock_adjtime 
SystemCallErrorNumber=EPERM

[Install]
WantedBy=multi-user.target

This part warrants an explanation. We’ve added a large number of additional directives. Though they may look like mindless copy-paste from other guides, each one limits Node Exporter in a way that reduces the potential attack surface. Let’s go through them briefly:

NoNewPrivileges=true
CapabilityBoundingSet=

The first line explicitly prevents the process and its descendants from gaining new privileges. The second clears the capability set, since the exporter doesn’t need any capabilities for normal operation.

ProtectSystem=strict
ProtectHome=read-only
PrivateTmp=true

Node Exporter doesn’t write anything during normal operation, so we mount the entire filesystem as read-only, with the exception of /dev, /proc, and /sys. Technically, ProtectHome could be set to the stricter true, but this would replace /home, /root, and /run/user with empty values. As a result, the collector would begin reporting incorrect data.

ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true

The lines above enable kernel protection. Reading, loading, and unloading modules is now explicitly prohibited, while the cgroups hierarchy is protected against writes. Again, the exporter only reads these values, so these restrictions do not affect its normal operation.

RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallArchitectures=native

Here we remove all the non-essential functionalities attackers could potentially exploit. By definition, the Node Exporter daemon has no need to create namespaces and SUID/SGID files, change the process personality, or manage memory pages with write permissions. The last among these is, incidentally, particularly effective at breaking JIT compilation, but it doesn’t interfere with Go applications. 

SystemCallArchitectures=native protects against the classic technique of bypassing a seccomp filter by invoking the same syscall through a 32-bit ABI.

SystemCallFilter=@system-service 
SystemCallFilter=adjtimex clock_adjtime 
SystemCallErrorNumber=EPERM

The final three directives are among the most important ones. The first creates an “allowlist,” a set of system calls available to the daemon. For Node Exporter, @system-service is suitable, but it doesn’t contain two calls required by the timex collector, adjtimex and clock_adjtime. We add them explicitly.

And finally, there’s SystemCallErrorNumber=EPERM. The standard seccomp behavior when a prohibited syscall is encountered is to immediately “kill” the process with SIGSYS. The result can be almost comical: systemctl reports everything in green, while the process crashes exactly when Prometheus accesses it. Setting EPERM, on the other hand, merely reports an error while leaving the process running, which means we can see in the logs what was missing and take appropriate action.

All that remains is to reload the daemon configuration and start the service:

sudo systemctl daemon-reload

sudo systemctl enable --now node_exporter

Check that everything is up and running:

sudo systemctl status node_exporter

Use systemd’s built-in auditing utility to make sure we’ve done everything correctly:

sudo systemd-analyze security node_exporter.service

It assigns the unit an exposure score from 0 to 10, with lower being better. There is no point in aiming for a perfect zero: a value below 5 is considered acceptable, while anything in the 2–3 range is already a fairly seriously hardened service from a security perspective:

→ Overall exposure level for node_exporter.service: 2.6 OK 

The only thing left is to verify that the exporter is accessible externally. Copy the certificate from the application directory into the monitor user’s home directory:

sudo cp /etc/node_exporter/node1.crt /home/monitor/node1.crt

Change the owner:

sudo chown monitor:monitor /home/monitor/node1.crt

And retrieve it from the local machine via scp:

scp -i ~/.ssh/id_ed25519_node1 \ [email protected]:~/node1.crt ~/node1.crt

After the file has been successfully copied, remove the temporary file from the server:

ssh -i ~/.ssh/id_ed25519_node1 [email protected] 'rm ~/node1.crt'

Now query it with curl from the local machine:

curl -sS --cacert ~/node1.crt \
  -u prometheus:<your_password> https://10.0.0.11:9100/metrics | grep -c '^node_'

Alternatively, you can open a browser and go to:

https://10.0.0.11:9100/metrics

This procedure must be repeated for all machines that you plan to monitor. Doing this manually is difficult. It would be easier to write some kind of Ansible playbook. 

Next, we move directly to configuring the Prometheus + Grafana pair.

Prometheus

As mentioned above, we’ll take the metric collector directly from GitHub. But first, let’s take care of security and create a separate user just as before:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus

Download the current version (3.14.0 at the time of writing):

VER=3.14.0

curl -fsSLO https://github.com/prometheus/prometheus/releases/download/v${VER}/prometheus-$VER.linux-amd64.tar.gz

Don’t forget the checksums:

curl -fsSLO https://github.com/prometheus/prometheus/releases/download/v${VER}/sha256sums.txt

Verify them:

grep "prometheus-${VER}.linux-amd64.tar.gz" sha256sums.txt | sha256sum -c -

Now extract the downloaded tarball, enter the directory, and install two binaries. The first is the prometheus executable itself, while the second is the promtool configuration validation utility:

tar xzf prometheus-${VER}.linux-amd64.tar.gz

cd prometheus-${VER}.linux-amd64

sudo install -o root -g root -m 0755 prometheus promtool /usr/local/bin/

Prepare directories for the configuration and data:

sudo mkdir -p /etc/prometheus/tls /var/lib/prometheus

sudo chown -R prometheus:prometheus /var/lib/prometheus

Place the certificate from node1, which we retrieved during the node configuration, into the first directory. Since it’s self-signed, it serves both as the root and as the certificate itself.

sudo install -o root -g prometheus -m 0644 ~/node1.crt /etc/prometheus/tls/

Again, when there are only a few nodes, you can add each of them individually. However, once there are more than a dozen, it makes sense to deploy a Private PKI. In this case, it will be enough to add the self-signed root certificate to all targets. 

Move the password outside the main configuration:

printf '%s' 'your_password' | sudo tee /etc/prometheus/scrape_password > /dev/null

Unfortunately, while we can use a bcrypt hash on the exporter side, the password has to be stored in plain text here. All we can do is restrict who is allowed to read the file:

sudo chown root:prometheus /etc/prometheus/scrape_password

sudo chmod 640 /etc/prometheus/scrape_password

Now create a minimal Prometheus configuration:

sudo nano /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: node
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/tls/node1.crt
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/scrape_password
    static_configs:
      - targets: ['10.0.0.11:9100']
        labels:
          env: prod
          role: web

This example is already fully functional. It collects metrics from the first node over a protected HTTPS connection. Note that labels are almost always necessary. Without them, it becomes much harder to write meaningful queries independent of IP addresses or hostnames. 

As long as there are only a few nodes and they are static, you can edit them manually. The simplest step toward automation is to replace static_configs with file_sd_configs:

    file_sd_configs:
      - files:
          - /etc/prometheus/targets/*.json
        refresh_interval: 30s

The daemon will reread files matching the specified pattern whenever they change, without requiring a restart. JSON format is used:

[
  {
    "targets": ["10.0.0.11:9100", "10.0.0.12:9100"],
    "labels": { "env": "prod", "role": "web" }
  }
]

Creating such files manually doesn’t make much sense. It’s better to assign this job to the same Ansible or Terraform, using localfile. In some cases, you can get away with just a couple dozen lines of Python.

Now it’s time to create a systemd unit responsible for starting and running the daemon. Here, we’ll use exactly the same kind of “sandbox”, but with slightly less restrictive rules. After all, unlike Node Exporter, Prometheus needs to write data to its own database:

nano /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --storage.tsdb.retention.size=40GB \
  --web.listen-address=10.0.0.10:9090
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
TimeoutStopSec=120s
LimitNOFILE=65536

# Sandbox
NoNewPrivileges=true
CapabilityBoundingSet=
ProtectSystem=strict
ReadWritePaths=/var/lib/prometheus
ProtectHome=true
PrivateTmp=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

[Install]
WantedBy=multi-user.target

Here, we can see that we allow the daemon to write to /var/lib/prometheus, while ProtectHome is set to true. The collector itself doesn’t collect filesystem metrics, so hiding the home directories from it completely is safe. The exceptions for the timex collector (adjtimex and clock_adjtime) aren’t needed here either.

It makes sense to validate the YAML with promtool before making any changes, to avoid accidental configuration errors.

promtool check config /etc/prometheus/prometheus.yml

To avoid accidental configuration errors, it makes sense to validate the YAML with promtool before applying any changes:

sudo systemctl daemon-reload

sudo systemctl enable --now prometheus

sudo systemctl status prometheus

The standard procedure comes next. Reload the systemd configuration, enable autostart, and immediately start the daemon. Then check its status:

http://10.0.0.10:9090/query

At this stage, port 9090 is temporarily available directly, but this is purely for testing. Don’t leave the configuration like this, as it isn’t suitable for a production environment. Below, we’ll immediately close off this connection method by hiding Prometheus behind a reverse proxy.

Historically, Prometheus didn’t have separate authentication mechanisms at all. Later, it became possible to protect it with the same --web.config.file mechanism used by Node Exporter. But this is not the best solution, since basic auth would then have to be configured everywhere. Even if you wanted to enable self-scraping, Prometheus itself would also need authentication configuration.

This is exactly the point where a reverse proxy becomes necessary. Change the single --web.listen-address=10.0.0.10:9090 parameter in the systemd unit to --web.listen-address=127.0.0.1:9090. This locks it inside localhost and removes external access. 

Now install Caddy. First, check that the required system packages for working with the repository are installed: 

sudo apt install -y curl gnupg

Add the repository GPG key:

curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' \ | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg 

Add the repository to the package source list:

curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' \ | sudo tee /etc/apt/sources.list.d/caddy-stable.list 

Update the package cache and install Caddy:

sudo apt update && sudo apt install caddy

Convert the password into bcrypt:

caddy hash-password 

Enter the password twice and you will get a string like:

$2a$14$jNe61gM7GsJxKrXU3/TPv.MECBKSusdiQiAhdXFLRdwtRC123b6ru

Create the configuration file:

sudo nano /etc/caddy/Caddyfile

https://10.0.0.10 {
    tls internal
    basic_auth {
        admin $2a$14$jNe61gM7GsJxKrXU3/TPv.MECBKSusdiQiAhdXFLRdwtRC123b6ru
    }
    reverse_proxy 127.0.0.1:9090
}

Now restart the service:

sudo systemctl restart caddy

We can now access Prometheus over HTTPS with basic username/password authentication. Don’t forget to replace 10.0.0.10 with your external IP address. 

Caddy acts as its own CA here, so the browser will still display a warning. You can remove it by adding Caddy’s root CA certificate to your operating system’s or browser's trusted root certificate store.

Grafana

All that’s left is to take care of data visualization. As mentioned earlier, there’s no point in downloading standalone binaries when Grafana can be installed directly from its own repository. Updates will then be delivered along with the rest of operating system packages.

Check that we have everything required to work with the repository:

sudo apt install -y apt-transport-https software-properties-common wget gpg

Download the GPG key and add it to the system:

wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

Now add the Grafana repository and reference the downloaded key:

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

Update the package cache and install Grafana:

sudo apt update && sudo apt install -y grafana

Reload systemd daemon configuration:

sudo systemctl daemon-reload

Enable automatic startup and start the service immediately:

sudo systemctl enable --now grafana-server

Check that the server is up:

curl -s http://127.0.0.1:3000/api/health

You should get a response similar to:

{
  "database": "ok",
  "version": "13.2.0",
  "commit": "f681b1359f6a0b8ecb9f2c49a88ac72b75bde73b"
}

Unlike with the previous units, there is no point in building our own sandbox here. The daemon already comes with reasonable restrictions out of the box:

sudo systemd-analyze security grafana-server.service

→ Overall exposure level for grafana-server.service: 2.9 OK 

Grafana itself is configured through a huge configuration file located at /etc/grafana/grafana.ini. Going through the entire file would make no sense within a single article, so we will focus on a few targeted changes. Don’t forget to remove the semicolon before each line you modify:

[server] 
http_addr = 127.0.0.1 
http_port = 3000 
root_url = https://10.0.0.10:8443/

[security] 
cookie_secure = true 
disable_gravatar = true

[users] 
allow_sign_up = false

[analytics] 
reporting_enabled = false 
check_for_updates = false

The same logic as with Prometheus applies here. We won’t expose everything directly to the outside world; instead, we will protect the service using the already installed Caddy, despite the fact that Grafana has its own built-in authentication mechanism. There’s no reason to expose port 3000 directly to the public Internet. 

On the first login, the credentials admin/admin are used. You’ll then immediately be prompted to replace them with your own.

You also need to pay attention to the root_url parameter. The system uses it to build links inside the interface. If it’s configured incorrectly, you’ll end up with broken redirects and unclickable links in notification emails. Port 8443 was also chosen for a reason; namely, 443 is already occupied by Prometheus.

All that remains is to add the service information to the Caddyfile:

sudo nano /etc/caddy/Caddyfile

https://10.0.0.10:8443 {
    tls internal 
    reverse_proxy 127.0.0.1:3000 
}

Restart the reverse proxy:

sudo systemctl restart caddy

The good thing here is that there’s no need to configure WebSocket support separately. Caddy handles it automatically, unlike nginx, for instance. At this point, any typical guide would tell you to open the interface and create a data source there manually. But if you ever recreate the cloud machine, you would have to enter everything again. It is much easier to apply the IaC approach from the very beginning and define our configuration through provisioning:

sudo nano /etc/grafana/provisioning/datasources/prometheus.yml

apiVersion: 1

datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://127.0.0.1:9090
    isDefault: true
    jsonData:
      timeInterval: 15s

Every time Grafana starts, it reads YAML files from /etc/grafana/provisioning/datasources/, which means the configuration can be stored in Git alongside the rest of the infrastructure code. The file is read by the grafana group, so set the owner and permissions as follows:

sudo chown root:grafana /etc/grafana/provisioning/datasources/prometheus.yml

sudo chmod 640 /etc/grafana/provisioning/datasources/prometheus.yml

In the same way, provide the system with a dashboard configuration:

sudo nano /etc/grafana/provisioning/dashboards/default.yml

apiVersion: 1

providers:
  - name: infra
    orgId: 1
    folder: 'Infrastructure'
    type: file
    options:
      path: /var/lib/grafana/dashboards

There is no need to reinvent the wheel, as grafana.com has an excellent classic Node Exporter dashboard, number 1860. Create a directory we specified earlier in the configuration:

sudo mkdir -p /var/lib/grafana/dashboards

Download the dashboard definition:

sudo curl -fsSL -o /var/lib/grafana/dashboards/node-exporter-full.json https://grafana.com/api/dashboards/1860/revisions/latest/download

And transfer ownership of the directory to the grafana group:

sudo chown -R grafana:grafana /var/lib/grafana/dashboards

Important: Keep in mind that we specify the uid ourselves instead of leaving it to the generator. The reason is that the system expects the ${DS_PROMETHEUS} placeholder inside the dashboard. With a manual import, Grafana itself would ask which data source should be substituted. However, since everything is being done automatically here, we need to handle that ourselves. Fortunately, it takes only one command:

sudo sed -i 's/${DS_PROMETHEUS}/prometheus/g' /var/lib/grafana/dashboards/node-exporter-full.json

The only thing left to do is restart the service. Grafana will reread the configurations and prepare everything automatically:

sudo systemctl restart grafana-server

Conclusion

We’ve built a monitoring stack that, though simple, is perfectly suitable for real-world operation. Data on the hosts is collected by Node Exporter instances, Prometheus manages the time series and executes queries, while Grafana visualizes all of it in convenient dashboards.

The basic security issues are addressed during the initial configuration stage, from tuning systemd units and subsequently evaluating them to adding authentication mechanisms where they were not originally provided. All services run under separate users, and data exchange between the components is protected.

Of course, building a complete observability system doesn’t end here. The next steps may include developing and adding alerting rules, automating the connection of new nodes using Ansible or Terraform, and moving toward centralized long-term metric storage as the infrastructure grows.

Regardless of the exact set of tools used, this kind of stack follows one of the fundamental principles of monitoring: it remains available when the main infrastructure begins to fail. A full disk, a crashed server, or an abnormal spike in load can now be detected before they actually start affecting end users.

Ready to build this monitoring stack yourself?

Spin up a cloud server on 3HCloud and follow along. Deploy your server →