--- title: "How Do I Know If My Server is Down? Prometheus and OpenTelemetry Monitoring Guide For Beginners" description: "Learn how to monitor servers and applications using Prometheus with practical setup examples, key metrics, and PromQL queries for effective system observability." canonical_url: "https://claritytech.io/posts/prometheus-monitoring-setup/" --- > Index of every page on this site: [llms.txt](https://claritytech.io/llms.txt) If you're someone who wants to know how to monitor your server for downtime, CPU usage, memory usage etc., this blog is for you. ## What is Prometheus and Server Monitoring? First thing first. Very simply **Prometheus is a database that stores server metrics** like cpu usage or memory usage over time etc. But don't think that it only stores server metrics; it stores much more than that like application metrics e.g. no. of http requests, time duration of HTTP responses, DB query execution time etc. Let's now go to definition. Prometheus is an **open-source monitoring and alerting system/tool**. In simple words, it watches your servers and applications, collects numbers (metrics), and tells you if something goes wrong. It is widely used in DevOps, cloud systems, Docker, and Kubernetes environments. ## Why do we use Prometheus? Imagine you have: - A backend server (Laravel, Node.js, Spring Boot) - A database (MySQL) - A Docker setup - Maybe Kubernetes You need answers like: - Is CPU usage too high? - Is memory almost full? - How many users are hitting my API? - Are containers restarting? - Is my website slow? Prometheus helps you in the following ways: 1. **Monitor System Health:** CPU usage, Memory usage, Disk space, Network traffic 2. **Monitor Application Metrics:** Request count, Response time, Error rate, Active users 3. **Alert When Something Breaks:** If CPU > 90% for 5 minutes → send alert, if server is down notify on Slack/Email ## Architecure & Components of Prometheus The Prometheus ecosystem is made up of several components. Some are essential, others are optional depending on your setup. ![Prometheus Architecture](https://claritytech.io/blog-post-images/prometheus-monitoring/prometheus-architecture.png) 1. **Prometheus Server (Core Component)**
This is the main engine of the system which periodically scrapes (pulls) metrics from targets (`/metrics` endpoints) and then stores the data as time-series. Also executes queries and alert rules. 2. **Client Libraries**
You need this when you want application metrics like request count, response time or error rate. Your Node.js, Java, or PHP app exposes a `/metrics` endpoint using these libraries. They are used in your application code. 3. **Pushgateway**
It's used for short-lived jobs (like batch jobs or cron jobs). Since such jobs may finish before Prometheus scrapes them, they push metrics to the Pushgateway (Note: Prometheus is a pull based model that pulls metrics from `/metrics` endpoints). Prometheus then scrapes from Pushgateway. This is useful for temporary or scheduled tasks. 4. **Exporters**
Exporters collect metrics from third-party systems and convert them into Prometheus format. For example, - **Node Exporter** collects system metrics (CPU, RAM) - HAProxy Exporter collects load balancer metrics - cAdvisor collects container metrics
Exporters are used when the system cannot natively expose Prometheus metrics.> We'll use node-exporter in this tutorial. 5. **Alertmanager**
This component handles alerts generated by Prometheus and sends notifications via email, slack, webhooks. It supports grouping, silencing, and routing alerts. You don't necessarily use this tool; rather you can use Grafana Alerts. Other than these components, you can use other supporting tools like Grafana for visualizing the metrics. ## How to Setup Prometheus and Node-exporter Before going ahead, let's know what exactly node-exporter is. It's lightweight agent that runs on your server. It reads system metrics (CPU, RAM, Disk I/O) from the kernel and exposes them via an HTTP endpoint (usually port 9100). Prometheus will pull these metrics from Node Exporter's `/metrics` endpoint at regular intervals and stores them in a time-series database. ### Step 0: Folder Structure Let me not confuse you. Just create a simple folder structure as shown below to keep your configuration files organized. ```bash prometheus-setup/ ├── alertmanager/ │ └── alertmanager.yml ├── prometheus/ │ ├── prometheus.yml │ └── rules/ │ └── alerts.yml ├── docker-compose.yml ``` ```bash # 1. Create the main project directory mkdir prometheus-setup cd prometheus-setup # 2. Create subdirectories for configuration files mkdir prometheus mkdir alertmanager mkdir prometheus/rules ``` ### Step 1: Install Prometheus You can install Prometheus on your server using the following commands:
Method Command Description
Download Binary tar xvfz prometheus-*.tar.gz & cd prometheus-* Quick testing on Linux/ macOS.
Docker docker run -d -p 9090:9090 prom/prometheus Isolated environments, quick starts.
Docker Compose Use a docker-compose.yml file to define Prometheus, Node Exporter, and Alertmanager. Running a full, multi-service monitoring stack locally.
Other than these methods, you can also install Prometheus using package managers like `apt` for Ubuntu or `yum` for CentOS, and you can also use Helm charts for Kubernetes environments. We'll use docker compose method in this tutorial. Create a `docker-compose.yml` file in the `prometheus-setup` directory with the following content: ```yaml services: # other services like app, db etc. prometheus: image: prom/prometheus:latest command: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' - '--web.console.libraries=/etc/prometheus/console_libraries' - '--web.console.templates=/etc/prometheus/consoles' - '--storage.tsdb.retention.time=30d' - '--web.enable-lifecycle' volumes: - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml - ./prometheus/rules:/etc/prometheus/rules - monitoring_prometheus:/prometheus ports: - '${FORWARD_PROMETHEUS_PORT:-9090}:9090' networks: - monitoring # this is the same network as your app and db services so that prometheus can scrape metrics from them # we will add node exporter service in the next step volumes: monitoring_prometheus: driver: local ``` ### Step 2: Install Node Exporter & Alertmanager Add the following service definition to your `docker-compose.yml` file to set up Node Exporter: ```yaml services: # prometheus ... node-exporter: image: prom/node-exporter:latest container_name: node-exporter restart: unless-stopped volumes: - /proc:/host/proc:ro - /sys:/host/sys:ro - /:/rootfs:ro command: - '--path.procfs=/host/proc' - '--path.rootfs=/rootfs' - '--path.sysfs=/host/sys' - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)' ports: - '9100:9100' networks: - monitoring alertmanager: image: prom/alertmanager:latest container_name: alertmanager restart: unless-stopped volumes: - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml command: - '--config.file=/etc/alertmanager/alertmanager.yml' - '--storage.path=/alertmanager' ports: - '9093:9093' networks: - monitoring ``` ### Step 3: Configure Prometheus to Scrape Node Exporter Prometheus is configured via a YAML file, typically `prometheus.yml`. The core of the configuration is the `scrape_configs` section, where you define what to monitor. Here’s a basic configuration that scrapes the Prometheus server itself and a Node Exporter (system metrics). ```yaml global: scrape_interval: 15s # How often to scrape targets by default evaluation_interval: 15s # How often to evaluate alerting rules rule_files: - "/etc/prometheus/rules/*.yml" alerting: alertmanagers: - static_configs: - targets: - alertmanager:9093 scrape_configs: # monitor Prometheus server itself - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] # monitor system metrics via node exporter - job_name: 'node-exporter' static_configs: - targets: ['node-exporter:9100'] ``` ### Step 4: Configure Alertmanager Alertmanager is configured via a YAML file, typically `alertmanager.yml`. This file defines how to handle alerts generated by Prometheus, including routing and notification settings. You can send alerts to email, Slack, or other services. ```yaml # alertmanager/alertmanager.yml global: smtp_smarthost: 'localhost:25' smtp_from: 'alertmanager@localhost' route: group_by: ['alertname'] group_wait: 10s group_interval: 10s repeat_interval: 1h receiver: 'email-notifications' receivers: - name: 'email-notifications' email_configs: - to: 'admin@example.com' headers: subject: '{{ .GroupLabels.alertname }}' # Uncomment this section to use Slack instead of email: # - name: 'slack-notifications' # slack_configs: # - api_url: 'YOUR_SLACK_WEBHOOK_URL' # channel: '#alerts' # send_resolved: true inhibit_rules: - source_match: severity: 'critical' target_match: severity: 'warning' equal: ['alertname', 'instance'] ``` Create alert rules in the `prometheus/rules` directory. For example, create a file named `alerts.yml` with the following content to define some basic alerts for instance down, high memory usage, high CPU usage, and low disk space: ```yaml # prometheus/rules/alerts.yml groups: - name: instance_alerts rules: - alert: InstanceDown expr: up == 0 for: 1m labels: severity: critical annotations: summary: "Instance {{ $labels.instance }} down" description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute." - name: node_alerts rules: - alert: HighMemoryUsage expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 80 for: 5m labels: severity: warning annotations: summary: "High memory usage on {{ $labels.instance }}" description: "Memory usage is above 80% for 5 minutes. Current value: {{ $value }}%" - alert: HighCPUUsage expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80 for: 5m labels: severity: warning annotations: summary: "High CPU usage on {{ $labels.instance }}" description: "CPU usage is above 80% for 5 minutes. Current value: {{ $value }}%" - alert: DiskSpaceLow expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 20 for: 10m labels: severity: warning annotations: summary: "Low disk space on {{ $labels.instance }}" description: "Disk space on {{ $labels.mountpoint }} is below 20%. Current value: {{ $value }}%" ``` ### Step 5: Start the Services Run the following command in the terminal to start Prometheus and Node Exporter: ```bash # Start all services docker-compose up -d # Check that all containers are running docker-compose ps # View the logs docker-compose logs -f ``` The prometheus web interface will be available at `http://localhost:9090`. The node exporter metrics will be available at `http://localhost:9100/metrics`. The alertmanager web interface will be available at `http://localhost:9093`. You can check if Prometheus is successfully scraping metrics from Node Exporter by going to `http://localhost:9090/targets`. You should see both the `prometheus` and `node-exporter` targets listed as "UP". ![Prometheus Targets Page](https://claritytech.io/blog-post-images/prometheus-monitoring/prometheus-targets-page.png) ### Step 6: Query Metrics in Prometheus With PromQL Now that you have Prometheus and Node Exporter running, you can start querying metrics. Go to the Prometheus web interface at `http://localhost:9090` and click on the "Graph" tab. - **List all metrics:** Enter a metric name like `node_cpu_seconds_total`. - **Filter with labels:** Use curly braces: `http_requests_total{status="500"}`. It's not available currently though because we haven't set up application metrics. - **Calculate rates:** Use the `rate()` function for counters: `rate(node_cpu_seconds_total[5m])`. - **Check if targets are up:** The `up` job is automatically generated. Query `up` to see which targets are healthy. ![Prometheus Graph Tab](https://claritytech.io/blog-post-images/prometheus-monitoring/prometheus-graph-tab.png) > You can also use Grafana to visualize these metrics in a more user-friendly way. Grafana can connect to Prometheus as a data source and allows you to create dashboards with graphs, tables, and alerts based on your Prometheus metrics. ## A Better Architecture: Using OpenTelemetry Collector For Monitoring In a real-life production application, you might want to use the OpenTelemetry Collector as an middle layer between your applications/server metrics and Prometheus. The OpenTelemetry Collector can receive metrics from various sources (like your application, Node Exporter, etc.), process them (e.g., add labels, filter), and then export them to Prometheus. ![OpenTelemetry Architecture](https://claritytech.io/blog-post-images/prometheus-monitoring/otel-architecture.png) This architecture allows for more flexibility and scalability. You can have multiple applications and services sending metrics to the OpenTelemetry Collector, which then forwards them to Prometheus. It also allows you to use a single agent (the OpenTelemetry Collector) instead of multiple exporters for different systems. ### What is OpenTelemetry? Observability is the ability to understand the internal state of a system by looking at its outputs. OpenTelemetry is the standardized way to produce those outputs. OpenTelemetry (OTel) is **an open-source, vendor-agnostic framework** for generating, collecting, and exporting telemetry data (traces, metrics, logs) from your applications and infrastructure. It provides a single set of APIs, SDKs, and tools that let you instrument your code once and then send the data to any observability backend (like Jaeger, Prometheus, Datadog, New Relic, etc.) without being locked into a specific vendor. ### Why Use OpenTelemetry? Before OpenTelemetry, every observability vendor had its own agent and proprietary format. If you wanted to use multiple backends (e.g., Prometheus for metrics + Jaeger for traces), you often had to run different, incompatible instrumentation libraries. This created vendor lock‑in and operational complexity. In other words: before OTel, every vendor had its own proprietary protocol and agent. OpenTelemetry solves this by separating instrumentation (how you collect data) from exporting (where you send it). You instrument once, and the data can be routed to any compatible backend. ### Types of Data/Signals OpenTelemetry Handles OpenTelemetry focuses on three core data types, often called **signals**. Think of them as different lenses for viewing your system. Until now, we have mostly talked about metrics, but OpenTelemetry also handles traces and logs. **1. Traces (The "What happened during a request" lens)** A trace follows a single request (e.g., a user clicking "login") as it travels through your entire system. - **Span:** The smallest unit of work. For the "login" request, you might have a span in the API gateway, a span in the authentication service, and a span in the database. - **Trace ID:** A unique ID that connects all those spans together. If the request is slow, you can look at the trace and see exactly which span took the longest (e.g., the database query took 2 seconds). **2. Metrics (The "What is happening over time" lens)** Metrics are numerical data aggregated over time. They answer questions like "How many requests per second are we getting?" or "What is the average memory usage?" Examples: Request count, error rate, CPU usage, queue length. Unlike traces, you don’t look at a single metric event; you look at the pattern over the last hour, for example. **3. Logs (The "What happened at this specific moment" lens)** We all know logs. Logs are timestamped text records with structured data (usually in JSON). These are the traditional `console.log` or `print` statements. OpenTelemetry aims to correlate logs with traces. When you view a trace (a request), you should be able to see exactly which logs were generated during that specific span. ### OpenTelemetry Architecture and Components When you first approach OpenTelemetry, it’s easy to think of it as just another library. But in reality, it’s a carefully designed ecosystem of components that work together to give you vendor‑neutral, high‑quality observability. Understanding these pieces will help you see how OpenTelemetry fits into your application stack—and why it’s so flexible. Let’s walk through the main components. ![OpenTelemetry Collector Architecture](https://claritytech.io/blog-post-images/prometheus-monitoring/otel-collector-architecture.png) OpenTelemetry introduces a few key conceptual components that live inside your application. - **API (The "What do I call" layer)** This is the specification. If you use the OpenTelemetry API, you are writing code against a standard. If you later want to change the vendor you send data to (e.g., from Datadog to New Relic), you don't have to rewrite your instrumentation code. - **SDK (The "How does it work" layer)**
This is the concrete implementation of the API. It handles: - **Sampling:** In a high-traffic system, capturing every trace would be too expensive. The SDK decides which requests to record (e.g., record only 1% of healthy requests, but record 100% of error requests). - **Batching:** Instead of sending every span to the backend immediately (which would slow down your app), the SDK collects them and sends them in batches.
The spec is great, but you need something you can actually run. That’s where the language‑specific SDKs come in. - **Instrumentation (The "How do I collect data" layer)**
This is the code you add (or automatically inject) into your application to capture telemetry. - **Auto-instrumentation:** Libraries that automatically detect your web framework (e.g., Django, Spring Boot) or database client and generate spans and metrics without you changing your business code. - **Manual instrumentation:** Writing specific code to create custom spans (e.g., to measure a critical business logic function).
There exists instrumentation libraries for popular languages (Java, Python, Node.js, Go, etc.) that implement the OpenTelemetry API and SDK. You can use these to instrument your application code. - **OpenTelemetry Collector**
Once your application generates telemetry, it needs to leave your application and go to a backend where you can visualize it. This is handled by the OpenTelemetry Collector.
If the SDKs are the “sensors” inside your app, the Collector is the central pipeline or a smart router. It’s a standalone service that: - Receives telemetry in many formats (OTLP, Jaeger, Prometheus, etc.). (Receivers) - Processes it – batching, filtering, adding metadata, or redacting sensitive fields. (Processors) - Exports it to one or more backends (Jaeger, Prometheus, Datadog, etc.). (Exporters) ### The Standard: OTLP (OpenTelemetry Protocol) This is the critical piece that makes OpenTelemetry a "standard.". OTLP (OpenTelemetry Protocol) is the universal protocol for telemetry data. - Your application speaks OTLP. - The Collector speaks OTLP. - Any backend that accepts OTLP can visualize your data. ### OpenTelemetry vs Prometheus OpenTelemetry and Prometheus are often mentioned together, but they serve different—and complementary—roles in observability. OpenTelemetry is a vendor‑neutral framework for generating telemetry—traces, metrics, and logs—using consistent APIs, SDKs, and the OpenTelemetry Collector. Prometheus, on the other hand, is a dedicated time‑series database and alerting system purpose‑built for storing and acting on metrics. Rather than choosing one over the other, it's better to use both: instrument applications with OpenTelemetry, then route metrics to Prometheus for powerful querying, alerting, and long‑term storage. This combination gives you the flexibility of vendor‑neutral instrumentation paired with a battle‑tested monitoring backend. ## An Example Otel Collector Configuration You can configure the OpenTelemetry Collector using a YAML file. ```yaml receivers: otlp: protocols: # This defines how the Collector receives telemetry data. OTLP is the standard protocol for OpenTelemetry, and it supports both gRPC and HTTP. grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: # This section defines how the Collector processes telemetry data before exporting it. The batch processor is commonly used to improve performance by sending data in batches rather than one at a time. batch: timeout: 1s send_batch_size: 1024 exporters: # This section defines where the Collector sends the processed telemetry data. In this example, we have two exporters: a debug exporter that prints telemetry to the console (useful for development and debugging), and a Prometheus exporter that exposes metrics on an HTTP endpoint for Prometheus to scrape. debug: verbosity: detailed # prints telemetry to console (for development) prometheus: endpoint: "0.0.0.0:8889" # exposes metrics for Prometheus to scrape namespace: otel service: pipelines: # This section defines the pipelines for different types of telemetry data (traces, metrics, logs). Each pipeline specifies which receivers to use, which processors to apply, and which exporters to send the data to. In this example, all three pipelines (traces, metrics, logs) receive data from the OTLP receiver, process it with the batch processor, and then export it to the debug exporter. The metrics pipeline also exports to Prometheus. traces: receivers: [otlp] processors: [batch] exporters: [debug] metrics: receivers: [otlp] processors: [batch] exporters: [debug, prometheus] logs: receivers: [otlp] processors: [batch] exporters: [debug] ``` Make sure, you have the OpenTelemetry Collector running and configured to receive telemetry from your application. Then, you can set up Prometheus to scrape metrics from the Collector's Prometheus exporter endpoint (e.g., `http://localhost:8889/metrics`). This writing doesn't cover how to work with the OpenTelemetry Collector in detail, but you can find more information in the official OpenTelemetry documentation. ## Conclusion In this blog, we covered how to set up Prometheus for monitoring your server and applications, including installing Prometheus and Node Exporter, configuring them to work together, and defining alerting rules. We also introduced OpenTelemetry as a powerful framework for generating vendor‑neutral telemetry data and how it can complement Prometheus in a modern observability stack. I suggest you to explore OpenTelemetry and Prometheus further along with Tempo or Jaeger for tracing and Loki for logging, Grafana for visualization, and Alertmanager for alerting to build a comprehensive monitoring solution for your applications and infrastructure. Thanks for reading! ## Sitemap See the full [sitemap](https://claritytech.io/sitemap.md) for all pages.