Metrics

Prometheus, Mimir, and metrics collection for Kubernetes monitoring

Overview

Metrics provide numerical measurements of your system’s performance over time. The Observability Bundle includes four components that work together to collect, store, and visualize metrics from your Kubernetes clusters:

• Prometheus: Collects and stores short-term metrics
• Grafana Mimir: Long-term metrics storage and querying
• Kube State Metrics: Exposes Kubernetes object state as metrics
• Prometheus Node Exporter: Exposes node hardware and OS metrics

Prometheus

Prometheus is the industry-standard monitoring system and time-series database for cloud-native environments. It automatically discovers services in Kubernetes and scrapes metrics from them.

What Prometheus Monitors

Application Metrics:

• Request rates, latencies, error rates (RED metrics)
• Custom business metrics from your applications
• Metrics from /metrics endpoints

Kubernetes Metrics:

• Pod CPU and memory usage
• Container resource consumption
• Service endpoint availability

Integration:

• Scrapes metrics every 15-30 seconds (configurable)
• Stores data locally for 15-30 days (typical configuration)
• Remote writes to Mimir for long-term retention

PromQL Basics

Prometheus uses PromQL (Prometheus Query Language) for querying metrics.

Example queries:

# CPU usage by pod
sum(rate(container_cpu_usage_seconds_total[5m])) by (pod)

# Memory usage by namespace
sum(container_memory_usage_bytes) by (namespace)

# Request rate for a service
rate(http_requests_total{job="my-service"}[5m])

# 95th percentile response time
histogram_quantile(0.95,
  rate(http_request_duration_seconds_bucket[5m])
)

Configuration

Prometheus is configured via the Observability Bundle’s GitOps workflow. Key configuration options:

enabled: true
valuesObject:
  # Scrape interval
  scrapeInterval: 30s

  # Retention period
  retention: 15d

  # Remote write to Mimir
  remoteWrite:
    - url: http://mimir-gateway/api/v1/push

  # Resource limits
  resources:
    limits:
      memory: 4Gi
      cpu: 2

For detailed Helm chart values, see the Prometheus chart documentation.

Grafana Mimir

Mimir provides long-term, scalable storage for Prometheus metrics. It’s designed to handle billions of active time series across multiple tenants.

Why Mimir?

Long-term Retention:

• Stores metrics for months or years
• Prometheus typically keeps 15-30 days locally
• Historical analysis and capacity planning

Horizontal Scalability:

• Scales to handle millions of samples per second
• Distributed architecture handles large metric volumes
• No single point of failure

Prometheus Compatible:

• Receives data via Prometheus remote write
• Queries with PromQL
• Drop-in replacement for long-term storage

How It Works

Prometheus ──[remote write]──> Mimir ──[PromQL queries]──> Grafana
    │                                                           │
    └─────────────[PromQL queries for recent data]─────────────┘

Grafana can query both Prometheus (recent data) and Mimir (historical data) simultaneously, providing seamless access to both real-time and long-term metrics.

Configuration

enabled: true
valuesObject:
  # Storage backend (S3, GCS, Azure Blob)
  storage:
    backend: s3
    s3:
      endpoint: s3.amazonaws.com
      bucket: my-mimir-metrics

  # Retention period
  limits:
    compactor_blocks_retention_period: 1y

  # Ingestion limits
  ingester:
    ring:
      replication_factor: 3

For detailed configuration, see the Mimir chart documentation.

Kube State Metrics

Kube State Metrics generates metrics about the state of Kubernetes objects. Unlike metrics from the Kubernetes API server, kube-state-metrics focuses on the state of the objects (e.g., deployments, pods, nodes) rather than their resource consumption.

What It Exposes

Deployment Metrics:

• kube_deployment_status_replicas: Number of desired replicas
• kube_deployment_status_replicas_available: Number of available replicas
• kube_deployment_status_replicas_unavailable: Number of unavailable replicas

Pod Metrics:

• kube_pod_status_phase: Pod phase (Pending, Running, Succeeded, Failed)
• kube_pod_status_ready: Whether pod is ready
• kube_pod_container_status_restarts_total: Container restart count

Node Metrics:

• kube_node_status_condition: Node conditions (Ready, MemoryPressure, DiskPressure)
• kube_node_status_allocatable: Allocatable resources per node
• kube_node_status_capacity: Total capacity per node

Use Cases

Cluster Health Monitoring:

# Pods not in Running state
count(kube_pod_status_phase{phase!="Running"})

# Nodes with conditions other than Ready
sum(kube_node_status_condition{condition!="Ready",status="true"})

Capacity Planning:

# Available CPU capacity across cluster
sum(kube_node_status_allocatable{resource="cpu"})

# Pod resource requests vs node capacity
sum(kube_pod_container_resource_requests{resource="memory"}) /
sum(kube_node_status_capacity{resource="memory"})

Deployment Monitoring:

# Deployments with unavailable replicas
kube_deployment_status_replicas_unavailable > 0

# Deployments not at desired replica count
kube_deployment_status_replicas != kube_deployment_spec_replicas

Configuration

Kube State Metrics typically requires minimal configuration:

enabled: true
valuesObject:
  # Resource limits
  resources:
    limits:
      memory: 256Mi
      cpu: 100m

  # Which resources to monitor (default: all)
  collectors:
    - deployments
    - pods
    - nodes
    - services
    - configmaps

For more details, see the Kube State Metrics chart.

Prometheus Node Exporter

The Node Exporter exposes hardware and OS metrics from Kubernetes nodes. It runs as a DaemonSet (one pod per node) to collect node-level performance data.

What It Exposes

CPU Metrics:

• node_cpu_seconds_total: CPU time spent in different modes (user, system, idle)
• Used to calculate CPU usage percentages

Memory Metrics:

• node_memory_MemTotal_bytes: Total memory
• node_memory_MemAvailable_bytes: Available memory
• node_memory_Cached_bytes: Cached memory

Disk Metrics:

• node_filesystem_size_bytes: Filesystem size
• node_filesystem_avail_bytes: Available space
• node_disk_io_time_seconds_total: Disk I/O time

Network Metrics:

• node_network_receive_bytes_total: Bytes received
• node_network_transmit_bytes_total: Bytes transmitted
• node_network_receive_errors_total: Receive errors

Use Cases

Node Resource Monitoring:

# CPU usage by node
100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage by node
100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))

# Disk usage by mount point
100 - ((node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100)

Network Monitoring:

# Network traffic by node
rate(node_network_receive_bytes_total[5m])

# Network errors
rate(node_network_receive_errors_total[5m]) > 0

Disk I/O:

# Disk I/O operations
rate(node_disk_io_time_seconds_total[5m])

Configuration

Node Exporter runs as a DaemonSet with host network and pid namespace access:

enabled: true
valuesObject:
  # Host network access for accurate metrics
  hostNetwork: true
  hostPID: true

  # Resource limits
  resources:
    limits:
      memory: 128Mi
      cpu: 100m

For more configuration options, see the Node Exporter chart.

Visualizing Metrics in Grafana

Once metrics are being collected, Grafana provides powerful visualization capabilities.

Pre-built Dashboards

The Observability Bundle includes dashboards for common monitoring needs:

• Kubernetes Cluster Monitoring: Overview of cluster health and resource usage
• Node Metrics: Detailed node performance (from Node Exporter)
• Pod Metrics: CPU, memory, network by pod
• Deployment Status: Replica counts, pod states

Creating Custom Dashboards

1. Navigate to Grafana (access via ingress URL configured during setup)
2. Click Dashboards → New Dashboard
3. Add a panel and select Prometheus or Mimir as data source
4. Write PromQL queries to visualize your metrics

Example panel configuration:

• Panel Title: “API Request Rate”
• Data Source: Prometheus
• Query: rate(http_requests_total{job="api-service"}[5m])
• Visualization: Time series graph

Alerting

Grafana can alert based on metric thresholds:

1. Create an alert rule in a dashboard panel
2. Define alert conditions (e.g., CPU > 80% for 5 minutes)
3. Configure notification channels (email, Slack, PagerDuty)

Accessing Grafana

To access Grafana and view your metrics:

1. Get the Grafana URL from your Observability Bundle configuration (configured during setup)
2. Log in with the credentials you configured:
   • Username: admin (default)
   • Password: Set during bundle configuration

If you need to retrieve the password:

kubectl get secret grafana -o jsonpath="{.data.admin-password}" -n observability | base64 --decode

For security, change the default admin password and create additional user accounts as needed through the Grafana UI.

Troubleshooting

Prometheus not scraping metrics

Check Prometheus targets:

1. Access Prometheus UI (typically at http://prometheus:9090)
2. Go to Status → Targets
3. Look for targets in “down” state

Common issues:

• Network policies blocking Prometheus
• Service selector not matching pods
• Pods don’t expose /metrics endpoint

Verify metrics endpoint:

kubectl port-forward pod/my-app-pod 8080:8080
curl http://localhost:8080/metrics

High memory usage in Prometheus

Prometheus memory usage scales with:

• Number of time series (unique label combinations)
• Scrape interval (more frequent = more data)
• Retention period (longer = more data stored)

Solutions:

• Reduce retention period (keep 7-15 days, use Mimir for long-term)
• Increase scrape interval (30s → 60s)
• Drop unnecessary metrics using relabeling
• Add more memory to Prometheus pods

Check cardinality:

# Access Prometheus UI
# Go to Status → TSDB Status
# Look for series with high cardinality

Mimir remote write failing

Check Prometheus logs:

kubectl logs -n observability prometheus-server-0 | grep "remote write"

Common issues:

• Mimir gateway not accessible (network/DNS issues)
• Authentication failure (check credentials)
• Rate limiting (ingestion rate too high)

Verify Mimir is running:

kubectl get pods -n observability | grep mimir

Missing node or cluster metrics

Verify kube-state-metrics is running:

kubectl get pods -n observability | grep kube-state-metrics

Verify node-exporter is running on all nodes:

kubectl get pods -n observability -l app=prometheus-node-exporter -o wide
# Should show one pod per node

Check Prometheus is scraping these exporters:

• Access Prometheus UI → Status → Targets
• Look for kube-state-metrics and node-exporter targets

Best Practices

Metric Naming

Follow Prometheus naming conventions:

• Use base unit (seconds, not milliseconds)
• Append _total for counters
• Append _bucket for histograms
• Example: http_request_duration_seconds

Label Usage

Keep cardinality low:

• Avoid high-cardinality labels (user IDs, timestamps)
• Use service name, environment, region as labels
• Don’t create unique label values for every request

Retention Strategy

Balance cost and utility:

• Prometheus: 7-15 days (recent, high-resolution)
• Mimir: months to years (long-term, downsampled)
• Adjust based on storage costs and query patterns

Resource Limits

Right-size resource allocations:

• Prometheus memory scales with active series
• Monitor and adjust based on actual usage
• Use horizontal scaling for very large deployments

Next Steps

• View Logs - Learn about log aggregation with Loki
• Configure Tracing - Set up distributed tracing
• Observability Overview - Return to bundle overview