Automating System Health Checks: Prometheus Node Exporter on Ubuntu
This guide details the advanced process of installing and configuring the Prometheus Node Exporter on an Ubuntu server to scrape vital system metrics (CPU, Memory, Disk I/O) for centralized monitoring.
Direct Diagnostic & Remediation Summary
- Prerequisite Installation: Update packages and install necessary dependencies:
sudo apt update && sudo apt upgrade -y. - Download Exporter: Download the latest stable Node Exporter binary matching your architecture from Prometheus releases.
- Configure Service: Create a systemd service file to ensure the exporter runs persistently and restarts on failure:
sudo systemctl enable prometheus-node-exporter.
๐ Executive Overview: System Monitoring Automation
This tutorial guides you through deploying the Prometheus Node Exporter on an Ubuntu instance. The Node Exporter exposes hardware and OS metrics (CPU utilization, memory usage, filesystem stats) over a dedicated HTTP endpoint. By scraping this endpoint with a central Prometheus server, you achieve continuous, automated system health monitoring without manual intervention.
๐จ Problem Overview: Manual Monitoring Limitations
Relying on manual checks (e.g., running top or df -h) provides only a snapshot of the system's state at that moment. This is insufficient for identifying intermittent issues, tracking long-term performance degradation, or building historical trend graphs required for proactive capacity planning.
Symptoms You Might Observe:
- Unexpected downtime correlating with resource spikes (e.g., high CPU during peak load).
- Difficulty in determining the root cause of slow performance due to lack of historical data.
- Alerts triggered only after a failure has occurred, instead of predicting it.
Mechanism: The Node Exporter reads system files (like /proc) and executes specialized collectors to generate metrics in the Prometheus exposition format. These are then served via HTTP on port 9100 by default.
Initial Triage & Environment Isolation (Preparation)
Before deployment, ensure the system is updated and necessary packages for downloading tools are installed. We will download the exporter binary directly.
sudo apt update && sudo apt upgrade -yNext, create a dedicated directory for the exporter files:
sudo mkdir /opt/node_exporterDownload the appropriate Node Exporter binary (always check the official Prometheus GitHub releases page for the latest version compatible with your Ubuntu architecture):
wget https://github.com/prometheus/node_exporter/releases/download/v{LATEST_VERSION}/node_exporter-*.linux-amd64.tar.gzExtract the binary:
tar xvf node_exporter-*.linux-amd64.tar.gz && sudo mv node_exporter-*/node_exporter /opt/node_exporter/Service Wrapper Creation (Systemd Integration)
For production use, the exporter must run as a background service that survives reboots and restarts automatically upon failure. We achieve this using systemd.
Create the systemd unit file:
sudo nano /etc/systemd/system/node_exporter.servicePaste the following content into the editor, ensuring paths are correct:
[Unit]
Description=Node Exporter
StartDefault=no
After=network.target
[Service]
User=prometheus # Recommended: Create a non-root user for security
Group=prometheus # Recommended: Group matching the user
Type=simple
ExecStart=/opt/node_exporter/node_exporter --collector.textfile.directory=/etc/node_exporter/textfile_collector
Restart=on-failure
[Install]
WantedBy=multi-user.targetCreate the necessary secure user and group (if they don't exist):
sudo useradd --no-create-home prometheus && sudo systemctl daemon-reloadEnable and start the service:
sudo systemctl enable node_exporter.service
sudo systemctl start node_exporter.serviceFirewall Configuration & Verification
The metrics endpoint (default port 9100) must be accessible by your central Prometheus server, but it should not be exposed publicly if unnecessary.
Allow traffic on the necessary port through UFW:
sudo ufw allow 9100/tcp comment 'Prometheus Node Exporter Metrics'Restarting a service or rebooting the machine is not required if you used systemctl start, but we verify everything:
- Check Service Status:
sudo systemctl status node_exporter.service(Should show active/running). - Test Local Access: Open a web browser or use curl on the server itself to hit the endpoint:
curl http://localhost:9100/metrics
The output should be a large block of text containing metrics like node_cpu_seconds_total, confirming successful data exposure.
Common Pitfalls & Advanced Diagnostics
If the endpoint returns an error or incomplete data, check these common failure points:
- Permission Denied: If accessing via a web browser fails with 401/403, verify that the service user (
prometheus) has read access to system files and that UFW rules are correctly applied. - Incorrect Binary Path: The most common error is pointing
ExecStartin the unit file to the wrong binary location. Always confirm the full path under/opt/node_exporter/node_exporter. - Firewall Blocking: If Prometheus cannot scrape, check both the local firewall (
ufw status) and any external cloud security groups (AWS Security Groups, Azure NSGs) to ensure port 9100 is open bi-directionally.
# Check systemd journal for deeper errors
sudo journalctl -u node_exporter.service --since "5 minutes ago"Verification, Results & Long-Term Stability Validation
Once the exporter is running and accessible (e.g., on http://your-server-ip:9100/metrics), the final step is integrating it with Prometheus.
- Prometheus Configuration Update: Edit your main Prometheus configuration file (
prometheus.yml) and add a new scrape job pointing to this server:scrape_configs: - job_name: 'ubuntu_server' static_configs: - targets: [':9100'] - Restart Prometheus: Restart the central Prometheus service for the new target to take effect.
- Validation Check: After waiting a minute, access the Prometheus UI (usually port 9090) and navigate to 'Status -> Targets'. Your Ubuntu server should appear with a healthy state (green indicator).
For long-term stability, consider setting up automated backups of your prometheus.yml file and implementing alerting rules in Prometheus Alertmanager.
No reader questions posted yet. Have you tried this solution or experienced an edge case? Post below.