OpenTelemetry, Loki, Prometheus, and Grafana: Complete Guide

🔥Advanced · 3+ yrsAdvanced3+ yrs experience Deep technical material — distributed-systems theory, low-level implementation details, and case studies of real production systems.Prerequisites: Production experience with at least one distributed system. Comfort reading research papers and RFCs.

1. Executive summary

These tools solve different parts of observability:

Tool	Primary role	Signal	Stores data?	Query language / interface
OpenTelemetry (OTel)	Instrumentation and telemetry collection/processing	Metrics, logs, traces, profiles in some ecosystem integrations	Usually no; it forwards telemetry	OTLP and component-specific exporters
Prometheus	Metrics collection, time-series database, recording rules, and alert evaluation	Metrics	Yes, primarily local TSDB; remote-write systems can provide long-term storage	PromQL
Grafana Loki	Log aggregation and storage	Logs	Yes; compressed chunks plus a small label index	LogQL
Grafana	Visualization, exploration, dashboards, alerting, and correlation	Metrics, logs, traces, and other data sources	Usually no; it queries data sources	Data-source query languages such as PromQL and LogQL

A typical deployment is:

Applications / infrastructure
        │
        ├── OTel SDKs, auto-instrumentation, exporters
        ├── Prometheus exporters / /metrics endpoints
        └── log agents such as Grafana Alloy, Fluent Bit, or OTel Collector
                    │
                    ▼
        Collection, enrichment, filtering, batching
       OTel Collector and/or Grafana Alloy
          │             │              │
          ▼             ▼              ▼
      Prometheus       Loki       Trace backend
       (metrics)       (logs)      e.g. Tempo/Jaeger
          │             │              │
          └─────────────┴──────────────┘
                         ▼
                      Grafana
          dashboards • Explore • alerts • correlations

Important distinction: OpenTelemetry is not a replacement for Prometheus, Loki, or Grafana. It is primarily a standard and collection pipeline. Prometheus and Loki are storage/query backends; Grafana is the user-facing analysis layer.

---

2. Observability fundamentals

Observability is the ability to understand a system’s internal state from its externally produced telemetry. The classic signals are:

• Metrics: Numeric measurements over time: request rate, error rate, latency, CPU, memory, queue depth.
• Logs: Timestamped event records, usually containing a message and structured fields.
• Traces: A request’s path through one or more services, represented as spans.
• Profiles: Continuous or sampled measurements of CPU, memory allocation, lock contention, and similar runtime behavior.

A useful incident workflow is:

1. Metrics detect that something changed.
2. Dashboards identify the affected service, region, tenant, or endpoint.
3. Traces show where time or errors were introduced across service boundaries.
4. Logs explain the detailed event, exception, request, or configuration state.

To make this work, propagate common context—especially service.name, environment, deployment version, and trace/span identifiers—across all signals.

---

3. OpenTelemetry

What it is

OpenTelemetry is a vendor-neutral, open-source framework for instrumenting, generating, collecting, processing, and exporting telemetry. It is governed as a CNCF project and is designed to prevent application code from being tightly coupled to one observability vendor.

OpenTelemetry includes:

• APIs: interfaces used by application code
• SDKs: language-specific implementations that create and process telemetry
• Automatic/zero-code instrumentation for supported runtimes and libraries
• The OpenTelemetry Protocol, or OTLP
• The OpenTelemetry Collector
• Semantic conventions for consistent names and attributes
• Exporters and integrations for many backends

What it is not

OpenTelemetry is not:

• A dashboarding product
• A full log-search UI
• A time-series database
• A durable log store
• A replacement for every backend
• Necessarily a complete monitoring solution by itself

Main signals

Metrics

Metrics represent numerical measurements. OTel supports counters, up/down counters, histograms, gauges, and exponential histograms. Examples:

• http.server.request.duration
• request count
• active connections
• queue size
• CPU utilization

Traces

A trace represents one logical operation. It contains spans, and spans can have:

• operation name
• start/end time
• status and status description
• attributes
• events
• links
• parent/child relationships

Trace context is commonly propagated using W3C Trace Context headers such as traceparent.

Logs

OTel logs can carry severity, body, attributes, resource information, timestamps, and trace/span correlation fields. OTel logging adoption varies by language and organization; many deployments still collect application logs using an agent and enrich them with OTel resource attributes.

OpenTelemetry Collector

The Collector is a vendor-neutral service that receives, processes, and exports telemetry. It is commonly used to centralize pipeline behavior rather than putting backend-specific code in every application.

Its pipeline has four principal component types:

1. Receivers — accept telemetry, for example, OTLP over gRPC/HTTP, Prometheus scrape, Jaeger, Zipkin, or syslog.
2. Processors — transform or control data, for example, batching, memory limiting, filtering, resource enrichment, sampling, redaction, and attributes manipulation.
3. Exporters — send data to destinations, such as Prometheus remote writing, Loki-compatible endpoints, OTLP backends, Kafka, or files.
4. Connectors — join pipelines or derive one signal from another, depending on the distribution and version.

A simplified Collector configuration looks like:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  batch:
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert

exporters:
  otlp/traces:
    endpoint: tempo.example.internal:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://prometheus.example.internal:9090/api/v1/write
  debug:
    verbosity: basic

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/traces]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [prometheusremotewrite]

Exact component names and backend support depend on the Collector distribution and version; validate the configuration against the chosen release.

Deployment patterns

• Agent pattern: one Collector/Alloy per node or Kubernetes node, close to workloads. Good for local log collection and metadata enrichment.
• Gateway pattern: a horizontally scaled central Collector layer. Good for centralized sampling, routing, redaction, and export policy.
• Agent plus gateway: agents collect locally and gateways perform shared processing and backend export.

OpenTelemetry with Prometheus and Loki

OTel can complement Prometheus and Loki in several ways:

• Applications emit OTLP to the Collector.
• The Collector exports metrics using Prometheus remote write or exposes a Prometheus-scrapable endpoint, depending on the design.
• The Collector or an agent forwards logs to Loki through a supported Loki/OTLP path.
• Resource attributes become labels or metadata only when deliberately mapped.

Do not automatically convert every OTel attribute into a Prometheus label or Loki stream label. High-cardinality attributes can create severe cost and performance problems.

---

4. Prometheus

What it is

Prometheus is an open-source systems monitoring and alerting toolkit. It collects and stores numerical time-series data and evaluates PromQL expressions and rules.

Prometheus is especially strong for:

• infrastructure and Kubernetes monitoring
• service-level metrics
• alerting on symptoms and saturation
• dynamic service discovery
• dimensional aggregation by labels
• reliable local operation during an outage

Data model

A Prometheus time series is identified by a metric name and a set of labels:

http_requests_total{
  job="checkout",
  instance="10.0.2.17:8080",
  method="GET",
  route="/orders",
  status="200"
}

Every unique label set creates a separate time series. This is powerful, but excessive or unbounded labels cause cardinality explosion.

Avoid labels such as:

• user ID
• request ID
• raw URL with IDs
• email address
• UUID
• arbitrary exception text

Prefer bounded dimensions such as service, environment, region, method, normalized route, and status class.

Collection model

Prometheus primarily uses a pull model:

1. Prometheus discovers or is configured with a target.
2. It periodically sends an HTTP request to the target’s metrics endpoint.
3. The target returns samples in Prometheus exposition format or another supported format.
4. Prometheus stores the samples locally.

Common collection sources include:

• application client libraries
• /metrics endpoints
• node exporter
• kube-state-metrics
• database, cache, load balancer, and cloud exporters
• service discovery
• Pushgateway for carefully scoped short-lived jobs
• remote-write receivers or OTel pipelines, where appropriate

Metric types

• Counter: Monotonically increasing value, such as total requests. Query with rate() or increase().
• Gauge: Value that can increase or decrease, such as memory usage or queue depth.
• Histogram: Buckets plus count and sum, useful for latency distributions and quantiles.
• Summary: Client-side quantiles plus count and sum; less aggregatable across instances than histograms.
• Native histogram: A newer histogram representation that can reduce bucket-management overhead when supported by the complete toolchain.

PromQL examples

Request rate:

sum(rate(http_requests_total[5m])) by (service)

Error percentage:

100 * sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

p95 latency from histogram buckets:

histogram_quantile(
  0.95,
  sum(rate(http_request_duration_seconds_bucket[5m]))
    by (le, service)
)

CPU utilization approximation:

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))

Rules and alerting

Prometheus supports:

• Recording rules: precompute expensive or frequently used expressions.
• Alerting rules: create alert instances when an expression is true.
• Alertmanager: deduplicate, group, route, silence, and notify alerts.

Example alert rule:

groups:
  - name: service-health
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service)
          > 0.05
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "High HTTP error rate for {{ $labels.service }}"

Storage and scaling

A standalone Prometheus server stores data in its local TSDB and is intentionally operationally independent. For long retention, global querying, or high availability, organizations commonly add a remote-write/remote-read compatible system such as Grafana Mimir, Thanos, Cortex, or a managed service.

Common scaling techniques:

• shard targets across multiple Prometheus servers
• use federation for selected aggregates
• remote write to long-term storage
• use recording rules to reduce repeated query cost
• control scrape intervals and label cardinality
• use HA pairs with external deduplication where supported

Prometheus is not ideal as a billing ledger or exact event store. It samples numeric data and is optimized for monitoring and diagnosis rather than immutable financial accounting.

---

5. Grafana Loki

What it is

Loki is a log aggregation system designed around a cost-efficient indexing model. It indexes log-stream metadata—primarily labels—and stores the log content in compressed chunks. Log data can be stored in object storage such as S3 or GCS, or locally for small deployments.

The design is deliberately different from systems that fully index every word in every log line.

Loki’s central concept: streams and labels

A log stream is identified by a unique combination of labels:

{cluster="prod", namespace="payments", app="checkout"}

Every unique combination creates a stream. Loki indexes these labels to locate chunks, then scans the relevant log lines.

Use low-cardinality labels that are useful for nearly every query:

• env
• cluster
• namespace
• app
• job
• region

Do not use high-cardinality values as stream labels:

• request IDs
• trace IDs
• user IDs
• pod IDs when they churn heavily
• IP addresses
• full URLs
• arbitrary error strings

Loki also supports structured metadata for values that should travel with a log entry but should not create a new stream. Trace IDs are a typical example.

Loki architecture

A production Loki deployment commonly includes some combination of:

• Distributor: receives pushed log entries and distributes them.
• Ingester: builds and flushes compressed chunks.
• Querier: evaluates log queries.
• Query frontend: schedules, splits, caches, and parallelizes queries.
• Compactor: compacts indexes and manages retention-related work.
• Index gateway / storage layer: serves index data, depending on the deployment mode.
• Object storage: durable storage for chunks and indexes.
• Ruler: evaluates log-based recording and alerting rules, depending on the setup.

For local experiments, Loki can run as a single binary with filesystem storage. For production, distributed mode and object storage are generally more appropriate.

Log ingestion

Logs can be sent through:

• Grafana Alloy or another agent
• OpenTelemetry Collector
• Fluent Bit or Fluentd
• Promtail in environments still using it, subject to current project and product guidance
• Loki HTTP push APIs
• OTLP-based ingestion is supported by the chosen Loki version and configuration

A collector/agent typically performs:

1. file, container, journald, or socket discovery
2. multiline assembly
3. parsing JSON or other structured formats
4. metadata enrichment
5. label selection
6. batching and retry
7. push to Loki

LogQL examples

All logs from a service:

{app="checkout", namespace="prod"}

Filter lines containing an error:

{app="checkout"} |= "error"

Parse JSON and filter a field:

{app="checkout"} | json | status >= 500

Count errors per service:

sum by (app) (
  count_over_time({namespace="prod"} |= "error" [5m])
)

Extract a rate from structured logs:

sum by (app) (
  rate({namespace="prod"} | json | level="error" [5m])
)

Exact syntax depends on the log format and LogQL capabilities of the deployed version.

Loki alerting

Loki can support log-based rules, such as alerting when error lines or authentication failures exceed a threshold. In practice, prefer metrics for stable SLO alerting when a metric can represent the condition cleanly; use log alerts for event-oriented conditions that are not already modeled as metrics.

Loki performance and cost guidance

• Keep stream labels small, stable, and low cardinality.
• Parse and filter early at the agent when appropriate, but do not discard information needed for investigations.
• Use structured metadata for high-cardinality correlation fields.
• Use retention policies and object-storage lifecycle policies.
• Control ingestion limits and tenant quotas.
• Use query time ranges and selective label matchers.
• Avoid queries that scan all tenants, all streams, and long time ranges.
• Use JSON logs with consistent field names when possible.

---

6. Grafana

What it is

Grafana is open-source visualization and analytics software. It connects to data sources, runs their native queries, renders panels and dashboards, supports Explore workflows, and can evaluate or display alerts.

Grafana can query:

• Prometheus and compatible metric backends
• Loki
• Tempo and other trace backends
• SQL databases
• cloud monitoring services
• Elasticsearch and many plugin data sources

Grafana does not have to own the underlying data. It is usually the presentation and investigation layer.

Core concepts

• Data source: Connection definition for Prometheus, Loki, SQL, tracing, or another backend.
• Dashboard: A saved collection of panels, variables, annotations, and layout.
• Panel: One visualization or table backed by a query.
• Explore: Ad-hoc investigation without first building a dashboard.
• Variables: Dashboard controls such as environment, cluster, namespace, or service.
• Annotations: Events overlaid on charts, such as deployments or incidents.
• Alert rule: A condition evaluated against a query or expression.
• Contact point / notification policy: Where and how alerts are delivered and grouped.

Connecting Prometheus

Typical Prometheus data source URL inside Docker Compose:

http://prometheus:9090

The typical URL in Kubernetes depends on the Service name and namespace, for example:

http://prometheus-operated.monitoring.svc:9090

Grafana panels use PromQL, for example:

sum(rate(http_requests_total[5m])) by (service)

Connecting Loki

Typical Loki data source URL inside Docker Compose:

http://loki:3100

A Grafana Loki data source uses LogQL, for example:

{namespace="$namespace", app="$app"} | json

Correlating metrics, logs, and traces

Correlation works best when services emit consistent resource attributes and identifiers:

• service.name
• service.namespace
• deployment.environment
• service.version
• k8s.cluster.name
• k8s.namespace.name
• k8s.pod.name
• trace_id / span_id

A common workflow is:

1. Click a latency spike in a Prometheus panel.
2. Open related logs in Loki using the service, namespace, and time range.
3. Click a trace ID in a log line.
4. Open the distributed trace in Tempo or another trace backend.

This requires configuring Grafana data links or derived fields and ensuring the identifiers are present in the appropriate data.

Grafana alerting versus Prometheus/Alertmanager

There are two common patterns:

• Prometheus rules + Alertmanager: alerts are evaluated close to Prometheus and routed by Alertmanager.
• Grafana Alerting: Grafana evaluates rules across multiple data sources and manages contact points and notification policies.

Both can be valid. Avoid duplicating the same alert in both systems unless you intentionally want separate semantics. Decide which system is authoritative for each alert class.

---

7. How the four tools fit together

Pattern A: Prometheus-centric metrics, Loki logs, Grafana UI

Applications expose /metrics ──► Prometheus ──► Grafana
Container/node logs ──► Alloy/Fluent Bit ──► Loki ──► Grafana

This is the classic operationally simple stack. Add OTel for application traces and vendor-neutral instrumentation.

Pattern B: OTel Collector as the central pipeline

Applications ──OTLP──► OTel Collector ──► Prometheus-compatible metrics backend
                                      ├──► Loki
                                      └──► Tempo/other trace backend

Prometheus exporters ──scrape──► Prometheus ──► Grafana

This provides centralized enrichment, batching, filtering, redaction, routing, and sampling. It introduces another operational component and should be sized and secured properly.

Pattern C: Grafana LGTM plus OTel

LGTM commonly refers to:

• Loki — logs
• Grafana — visualization and alerting
• Tempo — traces
• Mimir — scalable long-term Prometheus-compatible metrics

OTel supplies instrumentation and collection, while Grafana products provide storage/query backends and UI. A standalone Prometheus can still be used for local metrics collection and alert evaluation.

---

8. Recommended production design

Application instrumentation

1. Use OTel SDKs or zero-code instrumentation.
2. Set service.name explicitly; do not rely on unknown_service defaults.
3. Add environment and version attributes.
4. Propagate W3C trace context across HTTP, messaging, and RPC boundaries.
5. Use normalized route names rather than raw URLs in metric labels.
6. Emit structured logs and include trace/span correlation fields.

Collection

1. Use local agents for node/container logs and local metadata.
2. Use gateway Collectors for centralized policy and export.
3. Enable memory limiting and batching.
4. Configure retries and queues for temporary backend failures.
5. Redact secrets and personal data before export.
6. Use tail sampling or head sampling deliberately for traces.
7. Protect OTLP and backend endpoints with TLS and authentication.

Prometheus

1. Define a bounded label taxonomy.
2. Use recording rules for common service-level queries.
3. Monitor scrape failures and target health.
4. Set retention based on local disk capacity and incident needs.
5. Add durable remote storage for long retention or global views.
6. Use Alertmanager or Grafana Alerting consistently.

Loki

1. Choose a small set of stable stream labels.
2. Put request IDs and trace IDs in structured metadata or log fields, not stream labels.
3. Use object storage for production durability.
4. Set ingestion, query, tenant, and retention limits.
5. Monitor stream count, ingestion rate, chunk flush failures, and query latency.
6. Use tenant isolation when multiple teams or customers share the system.

Grafana

1. Provision data sources and dashboards as code where practical.
2. Use folders, teams, and permissions to separate ownership.
3. Build dashboards around user questions, not every available metric.
4. Include service, environment, region, and time-range variables.
5. Add links between metrics, logs, and traces.
6. Use alert annotations for deployments, incidents, and maintenance.
7. Protect the UI with SSO, TLS, the least privilege, and audit controls.

---

9. Common mistakes

Treating OpenTelemetry as a database

OTel transports and processes telemetry; it does not by itself provide durable search and retention like Loki or Prometheus.

Sending every field as a label

This causes Prometheus cardinality explosions and Loki stream explosions. Labels should be bounded and intentionally selected.

Using logs as the only source of metrics

Logs are useful for event-driven measurements, but counters and histograms are generally more efficient and reliable for rates, latency, and SLOs.

Using metrics as an event store

Metrics are sampled aggregates. They are not a substitute for every individual request or an immutable audit trail.

Scraping high-cardinality application metrics

A metric with labels such as user ID or request ID can create millions of time series. Normalize routes and remove unbounded dimensions.

Poor log formatting

Unstructured multiline logs, inconsistent timestamps, missing severity, and missing correlation identifiers make parsing and investigation harder.

No backpressure strategy

Collectors and agents need bounded queues, memory limits, batching, retry behavior, and a policy for what happens when a backend is unavailable.

Alerting on symptoms without ownership

Every alert should have a clear owner, severity, runbook, notification route, and a condition that reflects user or system impact.

---

10. Minimal tool-selection guide

Choose OpenTelemetry when you need vendor-neutral instrumentation, standardized context, and a flexible collection pipeline.

Choose Prometheus when you need numeric time-series monitoring, PromQL, service discovery, recording rules, and infrastructure alerts.

Choose Loki when you need cost-conscious centralized logs that can be searched efficiently by stable metadata and filtered with LogQL.

Choose Grafana when you need a common UI for dashboards, Explore, visualization, alert management, and cross-signal investigation.

For a small Kubernetes deployment, a sensible starting point is:

OTel SDKs / auto-instrumentation
+ Grafana Alloy or OTel Collector
+ Prometheus for metrics
+ Loki for logs
+ Tempo or another backend for traces
+ Grafana for dashboards and investigation

Start with a small, consistent attribute and label model. The quality of correlation and the control of cardinality matter more than simply installing every component.

---

11. Official references

• OpenTelemetry documentation
• OpenTelemetry Collector
• Prometheus overview
• Prometheus data model
• Grafana Loki documentation
• Loki architecture
• Grafana introduction
• Grafana Prometheus data source
• Grafana Loki data source