Observability Overview

Introduction

Observability is the ability to understand the internal state of a system by examining its external outputs. In distributed systems (microservices, cloud-native), observability is critical for debugging, performance optimization, and maintaining reliability.

The Three Pillars of Observability

graph TB
    OBS[Observability] --> LOGS[Logs]
    OBS --> METRICS[Metrics]
    OBS --> TRACES[Traces]

    LOGS --> |What happened| LOG_D[Detailed event records]
    METRICS --> |How much/how fast| MET_D[Numeric measurements over time]
    TRACES --> |Where it went| TRACE_D[Request flow across services]

How They Work Together

graph TB
    INCIDENT[Incident: High Latency] --> METRIC_ALERT[Metric Alert: p99 latency > 2s]
    METRIC_ALERT --> TRACE[Trace: Which request is slow?]
    TRACE --> LOG[Log: What error occurred in that request?]
    LOG --> ROOT_CAUSE[Root Cause: Database query timeout]

Example Workflow:

1. Metrics alert: p99 latency exceeds 2 seconds
2. Traces reveal: requests to payment-service are slow
3. Logs show: database connection pool exhausted
4. Root cause: Database connections not being released (connection leak)

Observability vs Monitoring

Key Insight: Monitoring is a subset of observability. You monitor known conditions; observability helps you investigate unknown conditions.

Observability Stack

graph TB
    subgraph "Data Sources"
        APP[Applications]
        INFRA[Infrastructure]
        CLOUD[Cloud Services]
    end

    subgraph "Collection"
        OTEL[OpenTelemetry Collector]
        PROM[Prometheus]
        FLUENTD[Fluentd / Fluent Bit]
    end

    subgraph "Storage"
        MIMIR[Prometheus / Mimir]
        ES[Elasticsearch / Loki]
        TEMPO[Jaeger / Tempo]
    end

    subgraph "Visualization & Alerting"
        GRAFANA[Grafana]
        PAGER[PagerDuty / OpsGenie]
    end

    APP --> OTEL
    INFRA --> PROM
    CLOUD --> FLUENTD

    OTEL --> MIMIR
    OTEL --> ES
    OTEL --> TEMPO
    PROM --> MIMIR
    FLUENTD --> ES

    MIMIR --> GRAFANA
    ES --> GRAFANA
    TEMPO --> GRAFANA
    GRAFANA --> PAGER

SLI, SLO, SLA

graph TB
    SLI[SLI - Service Level Indicator] --> |Defines| SLO[SLO - Service Level Objective]
    SLO --> |Enforced by| SLA[SLA - Service Level Agreement]

    SLI --> |Example| SLI_D[Request latency, error rate, availability]
    SLO --> |Example| SLO_D[99.9% of requests < 500ms]
    SLA --> |Example| SLA_D[99.9% uptime, or credits/refunds]

SLI (Service Level Indicator)

A quantitative measure of a service aspect:

SLO (Service Level Objective)

The target value for an SLI:

# SLO Example
service: payment-api
slos:
  - name: availability
    sli: "Successful requests / Total requests"
    target: 99.95%
    window: 30d

  - name: latency
    sli: "p99 request latency"
    target: < 500ms
    window: 30d

  - name: error-rate
    sli: "5xx responses / Total responses"
    target: < 0.1%
    window: 30d

Error Budget

graph TB
    SLO_T[SLO: 99.9% availability] --> BUDGET[Error Budget: 0.1%]
    BUDGET --> |43.2 minutes/month| ALLOWED[Allowed downtime]

    ALLOWED --> |Budget remaining > 50%| SHIP[Ship features faster]
    ALLOWED --> |Budget remaining < 50%| SLOW[Slow down, focus on reliability]
    ALLOWED --> |Budget exhausted| FREEZE[Feature freeze, fix reliability]

Golden Signals (Google SRE)

graph TB
    GOLDEN[Golden Signals] --> LATENCY[Latency]
    GOLDEN --> TRAFFIC[Traffic]
    GOLDEN --> ERRORS[Errors]
    GOLDEN --> SATURATION[Saturation]

    LATENCY --> |Time to serve| LAT_D[p50, p95, p99]
    TRAFFIC --> |Demand on system| TRAF_D[Requests/sec, transactions/sec]
    ERRORS --> |Rate of failed requests| ERR_D[5xx rate, error count]
    SATURATION --> |How full the system is| SAT_D[CPU, memory, disk, network]

RED Method (Tom Wilkie)

For request-driven services:

USE Method (Brendan Gregg)

For resources (CPU, memory, disk, network):

Interview Questions

Q1: What are the three pillars of observability?

Answer: (1) Logs—discrete event records with timestamps and context, answering “what happened?” (2) Metrics—numeric measurements aggregated over time, answering “how much/how fast?” (3) Traces—distributed request paths across services, answering “where did the request go and where was it slow?” Together they provide complete system understanding: metrics detect issues, traces locate them, logs explain them.

Q2: What is the difference between monitoring and observability?

Answer: Monitoring tracks pre-defined metrics and alerts on known conditions (dashboard-driven). Observability is the ability to understand any system state by exploring its outputs, even states you didn’t anticipate (query-driven). Monitoring answers “is the system healthy?” Observability answers “why is the system unhealthy?” You monitor known failure modes; you use observability to investigate unknown ones. Modern systems need both.

Q3: Explain SLOs and error budgets.

Answer: An SLO is a target value for a service level indicator (e.g., 99.9% availability). The error budget is the allowed deviation (100% - SLO = 0.1% = 43.2 min/month downtime). Error budgets balance reliability vs velocity: when budget remains, ship features faster; when running low, slow down and focus on reliability; when exhausted, freeze features until reliability improves. This creates a data-driven framework for reliability decisions.

Q4: What are the Golden Signals?

Answer: Google’s four Golden Signals for monitoring: (1) Latency—time to serve requests (p50, p95, p99), (2) Traffic—demand on the system (RPS, concurrent users), (3) Errors—rate of failed requests (5xx rate), (4) Saturation—how full resources are (CPU, memory, disk). For request-driven services, focus on these four. For resources, use the USE method (Utilization, Saturation, Errors). For services, use the RED method (Rate, Errors, Duration).

Q5: How do you implement observability in a microservices architecture?

Answer: (1) Instrument applications with OpenTelemetry SDKs for metrics, traces, and logs, (2) Deploy OpenTelemetry Collector as a sidecar or DaemonSet for data collection, (3) Use Prometheus for metrics storage, Loki or Elasticsearch for logs, Tempo or Jaeger for traces, (4) Visualize in Grafana with dashboards for each service, (5) Set up SLO-based alerts, not threshold-based, (6) Implement correlation IDs across services for log-trace correlation, (7) Use structured logging (JSON) for machine-parseable logs.

Common Mistakes

1. Too many alerts: Alert fatigue—teams ignore important alerts
2. Alerting on symptoms, not causes: Alert on user impact (latency, errors), not CPU spikes
3. No correlation between pillars: Logs, metrics, and traces in separate systems with no links
4. Missing structured logging: Unstructured logs are hard to query and analyze
5. No SLOs: Without SLOs, you can’t make data-driven reliability decisions
6. Over-collecting: Collecting everything without retention policies—cost explosion
7. Ignoring cardinality: High-cardinality metrics (user IDs as labels) overwhelm Prometheus

Summary

Cross-References

• Logging: ELK & Structured Logging — Deep dive into logging
• Monitoring: Prometheus & Grafana — Metrics and alerting
• Tracing: Distributed Tracing — Request tracing across services
• Kubernetes: Pods — Observability for K8s workloads
• CI/CD: Pipelines — Observability in deployment pipelines