REST API Monitoring & Observability

The three pillars of observability — metrics, logs, and distributed traces — with OpenTelemetry instrumentation, health checks, and alerting best practices for production REST APIs.

Last Updated:

Table of Contents

  1. Monitoring vs Observability
  2. The Four Golden Signals
  3. Metrics with Prometheus
  4. Structured Logging
  5. Distributed Tracing with OpenTelemetry
  6. Health Check Endpoints
  7. Tooling: Grafana, Datadog, Jaeger, Postman
  8. Alerting Best Practices
  9. Monitoring Checklist

Monitoring vs Observability

Monitoring tells you when something is wrong — dashboards, alerts, uptime checks. Observability lets you understand why something is wrong, even for failure modes you did not anticipate, by providing rich queryable telemetry.

AspectMonitoringObservability
Question answered"Is the API up?""Why is request #4829 slow?"
DataPredefined metrics & alertsMetrics + Logs + Traces (any query)
Unknown failuresMisses novel failure modesCan debug anything with enough data
ToolsPingdom, UptimeRobot, PagerDutyOpenTelemetry, Grafana, Datadog, Jaeger

You need both. Monitoring catches known problems fast. Observability helps you understand and fix unknown problems. The industry standard in 2026 is OpenTelemetry (OTel) — a CNCF graduated project that unifies metrics, logs, and traces under a single vendor-neutral SDK.

The Four Golden Signals

Google SRE defines four metrics every API must track:

SignalWhat to MeasureAlert Threshold (example)
Latency p50, p95, p99 response time (distinguish success vs error latency) p99 > 500ms for 5 min
Traffic Requests per second (RPS), broken down by endpoint and method RPS drops >50% vs 1h average
Errors 5xx error rate. Also track 4xx separately (client vs server errors). 5xx rate > 1% for 2 min
Saturation CPU, memory, DB connection pool, queue depth — how "full" is the service? DB pool > 90% for 3 min

Metrics with Prometheus & Express

Prometheus is the de facto standard for API metrics collection. The prom-client library exposes metrics at a /metrics endpoint that Prometheus scrapes.

// npm install prom-client express
const express = require('express');
const promClient = require('prom-client');
const app = express();

// Collect default Node.js metrics (event loop lag, GC, heap, etc.)
promClient.collectDefaultMetrics();

// Custom histogram for HTTP request duration
const httpDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5]
});

// Counter for total requests
const httpRequests = new promClient.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status_code']
});

// Middleware — measure every request
app.use((req, res, next) => {
  const end = httpDuration.startTimer();
  res.on('finish', () => {
    const labels = {
      method: req.method,
      route: req.route?.path || req.path,
      status_code: res.statusCode
    };
    end(labels);
    httpRequests.inc(labels);
  });
  next();
});

// Expose metrics endpoint for Prometheus scraping
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  res.end(await promClient.register.metrics());
});

app.listen(3000);

With this in place, Prometheus scrapes /metrics every 15 seconds and Grafana can visualise latency percentiles, error rates, and throughput in real time.

Structured Logging

Logs should be machine-parseable JSON — not plain text strings. Structured logs enable filtering, aggregation, and correlation across millions of events in tools like Datadog, Loki, or CloudWatch.

// npm install pino
const pino = require('pino');
const logger = pino({ level: 'info' });

// Bad — plain text log, impossible to filter
console.log('POST /orders 201 42ms user=alice');

// Good — structured JSON log
logger.info({
  method:      'POST',
  path:        '/orders',
  status:      201,
  duration_ms: 42,
  user_id:     'user_alice',
  request_id:  'req_8f3a2b',
  order_id:    'order_98765'
}, 'Request completed');

// Output:
// {"level":30,"time":1714000000000,"method":"POST","path":"/orders",
//  "status":201,"duration_ms":42,"user_id":"user_alice",
//  "request_id":"req_8f3a2b","order_id":"order_98765","msg":"Request completed"}

Request ID Correlation

Always generate a unique request_id per incoming request and propagate it through every log line and downstream service call. This lets you reconstruct the full journey of a single request across your entire system:

const { randomUUID } = require('crypto');

app.use((req, res, next) => {
  // Use incoming request-id (from upstream) or generate a new one
  req.requestId = req.headers['x-request-id'] || randomUUID();
  res.setHeader('X-Request-ID', req.requestId);

  // Attach to every log in this request's scope
  req.log = logger.child({ request_id: req.requestId });
  next();
});

app.post('/orders', async (req, res) => {
  req.log.info({ user_id: req.user.id }, 'Creating order');
  const order = await orderService.create(req.body);
  req.log.info({ order_id: order.id }, 'Order created');
  res.status(201).json(order);
});

Distributed Tracing with OpenTelemetry

Distributed tracing shows the full execution path of a request across multiple services, databases, and external calls. Each step is a span within a trace. OpenTelemetry is the CNCF standard for vendor-neutral tracing.

// npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node
// tracing.js — load BEFORE anything else

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

const sdk = new NodeSDK({
  serviceName: 'orders-api',
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces'
  }),
  instrumentations: [
    getNodeAutoInstrumentations() // auto-instruments Express, HTTP, pg, redis, etc.
  ]
});

sdk.start();
process.on('SIGTERM', () => sdk.shutdown());

With auto-instrumentation, every Express route, database query, Redis call, and outbound HTTP request is automatically traced — no manual span creation required for the basics.

Manual Spans for Business Logic

const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('orders-api');

async function processOrder(orderData) {
  // Create a custom span for business logic
  return tracer.startActiveSpan('process-order', async (span) => {
    try {
      span.setAttributes({
        'order.user_id':    orderData.userId,
        'order.item_count': orderData.items.length,
        'order.total_usd':  orderData.total
      });

      const inventory = await checkInventory(orderData.items);
      const payment   = await chargePayment(orderData.payment);
      const order     = await createOrder({ ...orderData, paymentId: payment.id });

      span.setStatus({ code: SpanStatusCode.OK });
      return order;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      throw err;
    } finally {
      span.end();
    }
  });
}

Health Check Endpoints

Health checks allow load balancers, Kubernetes, and monitoring tools to determine whether your API is ready to receive traffic.

EndpointPurposeReturns healthy when...
GET /healthGeneral health checkThe process runs
GET /health/liveLiveness (Kubernetes)Process has not deadlocked
GET /health/readyReadiness (Kubernetes)DB connected, cache warm, dependencies up
GET /health/startupStartup probeInitialisation complete
 
// Health check implementation
app.get('/health/live', (req, res) => {
  // Liveness: just confirm the process is alive
  res.status(200).json({ status: 'ok', timestamp: new Date().toISOString() });
});

app.get('/health/ready', async (req, res) => {
  const checks = {};

  // Check database
  try {
    await db.query('SELECT 1');
    checks.database = 'ok';
  } catch (e) {
    checks.database = 'error: ' + e.message;
  }

  // Check Redis cache
  try {
    await redis.ping();
    checks.cache = 'ok';
  } catch (e) {
    checks.cache = 'error: ' + e.message;
  }

  const healthy = Object.values(checks).every(v => v === 'ok');
  res.status(healthy ? 200 : 503).json({
    status: healthy ? 'ok' : 'degraded',
    checks,
    uptime: process.uptime()
  });
});

Return 200 OK for healthy, 503 Service Unavailable for unhealthy. Kubernetes uses liveness to restart stuck pods and readiness to stop routing traffic to pods that are not ready.

Tooling: Grafana, Datadog, New Relic & Postman Monitors

ToolTypeBest ForPricing
Prometheus + Grafana Metrics + Dashboards Self-hosted, cost-effective at scale. Standard for Kubernetes. Free (OSS)
Grafana Tempo Distributed tracing OSS tracing backend, integrates natively with Grafana stack Free (OSS) / Cloud paid
Jaeger Distributed tracing Self-hosted CNCF tracing with full-text trace search Free (OSS)
Datadog Full observability platform Enterprise. Unified metrics, logs, traces, and APM in one UI. Paid (per host)
New Relic Full observability platform APM, browser monitoring, infrastructure. Generous free tier. Free tier / Paid
Postman Monitors Synthetic uptime monitoring Scheduled collection runs. Validates API responses from global regions. Free tier / Paid
OpenTelemetry Collector Telemetry pipeline Vendor-neutral collection, transformation, and routing of all signals Free (OSS)

Postman Monitors: Synthetic Uptime Checks

Postman Monitors run your Postman collections on a schedule from Postman's global infrastructure. The schedule can be every 1 minute, or as infrequent as once per day. Each run validates that your API endpoints return correct responses. Prometheus monitors internal metrics, but Postman Monitors check your API from the outside — the same way your users do.

// Postman collection test script (runs on each monitor execution)
pm.test("Status is 200", () => {
  pm.response.to.have.status(200);
});

pm.test("Response time under 500ms", () => {
  pm.expect(pm.response.responseTime).to.be.below(500);
});

pm.test("Returns valid user object", () => {
  const body = pm.response.json();
  pm.expect(body).to.have.property('id');
  pm.expect(body).to.have.property('email');
  pm.expect(body.email).to.match(/@/);
});

// Set next request's auth token from login response
const token = pm.response.json().access_token;
pm.environment.set("auth_token", token);

Configure monitors via Postman → Collections → Run Collection → Schedule a run. Configure alerts to notify Slack or PagerDuty when assertions fail.

Recommended OSS Stack (2026)

# docker-compose.yml — full observability stack
services:
  app:
    build: .
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    # Receives from app, exports to Prometheus + Tempo + Loki

  prometheus:
    image: prom/prometheus:latest
    # Scrapes metrics from otel-collector and /metrics endpoints

  grafana:
    image: grafana/grafana:latest
    # Dashboards for metrics (Prometheus), traces (Tempo), logs (Loki)

  tempo:
    image: grafana/tempo:latest
    # Stores and queries distributed traces

  loki:
    image: grafana/loki:latest
    # Stores and queries structured logs

Alerting Best Practices

Alert on symptoms, not causes. "p99 latency > 1s" is a symptom alert — it fires when users are affected. "DB query count > 1000/s" is a cause alert — it may fire without impacting users.

Recommended Alert Rules

# Prometheus alerting rules (alerting.yml)
groups:
  - name: api-alerts
    rules:
      # High error rate
      - alert: HighErrorRate
        expr: rate(http_requests_total{status_code=~"5.."}[5m])
              / rate(http_requests_total[5m]) > 0.01
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "API error rate > 1%"

      # High p99 latency
      - alert: HighLatency
        expr: histogram_quantile(0.99,
              rate(http_request_duration_seconds_bucket[5m])) > 1.0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "p99 latency > 1s"

      # Service down (no requests for 2 minutes)
      - alert: ServiceDown
        expr: rate(http_requests_total[2m]) == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "API receiving no traffic"

Runbook Best Practices

  • Every alert should link to a runbook with step-by-step diagnosis and remediation
  • Include a "for" duration to avoid alerting on transient spikes (e.g., for: 2m)
  • Use severity levels: critical (page on-call now), warning (investigate within hours), info (log only)
  • Review and tune alert thresholds quarterly — alert fatigue is as dangerous as no alerts
  • Test your alerts: use amtool alert add or Alertmanager test mode to verify routing

REST API Monitoring Checklist

Use this checklist when you configure monitoring for a new REST API, or when you audit an API that already exists. Each item is independently valuable. Start with the top rows for maximum impact.

✅ItemWhy It Matters
☐ Track four golden signals — latency (p50/p95/p99), traffic (RPS), errors (5xx rate), saturation (CPU/DB pool) Covers the majority of production incidents
☐ Expose /health/ready endpoint checked by load balancer / Kubernetes readiness probe Prevents traffic routing to unhealthy pods
☐ Structured JSON logging with request_id, user_id, method, path, status, duration_ms Enables filtering and correlation across millions of events
☐ Alert: 5xx error rate > 1% sustained for 2 minutes → page on-call First alert for user-facing outages
☐ Alert: p99 latency > 1s for 5 minutes → warning notification Catches degraded performance before users complain
☐ Distributed tracing via OpenTelemetry with auto-instrumentation for HTTP, DB, and cache Root cause analysis for slow or failed requests
☐ Synthetic monitoring — run Postman Monitors or uptime checks every minute from multiple regions Detects outages from the user's perspective, not just internal metrics
☐ Downstream dependency health — track DB query latency and external API response times separately Isolates root cause to your code vs dependencies
☐ Dashboard with SLI/SLO — define availability target (e.g. 99.9%) and track it weekly Aligns engineering effort with reliability goals
☐ Runbooks for every alert — step-by-step diagnosis and remediation linked from alert body Reduces mean time to recovery (MTTR)

See our API Testing guide for how to complement production monitoring with pre-deployment load and contract tests. Our REST API tutorial covers baseline API design practices that make monitoring easier.

Frequently Asked Questions

What is the difference between monitoring and observability?

Monitoring tells you when something is wrong — dashboards and alerts for known failure modes. Observability lets you understand why something is wrong, even for failure modes you did not anticipate, by providing rich queryable telemetry (metrics, logs, traces). Modern production APIs need both.

What is the best tool for REST API monitoring?

It depends on your needs. For self-hosted cost-effective monitoring: Prometheus + Grafana (free, OSS). For all-in-one managed observability: Datadog or New Relic. For synthetic uptime checks: Postman Monitors or Pingdom. Most teams combine Prometheus for metrics with OpenTelemetry for distributed tracing.

What metrics should every REST API track?

Google SRE defines four golden signals. Latency (p50, p95, p99 response time). Traffic (requests per second). Errors (5xx and 4xx rates). Saturation (CPU, memory, DB connection pool). These four cover the vast majority of production incidents.

What is Postman Monitors and how does it differ from Prometheus?

Postman Monitors runs your Postman collections on a schedule from Postman's global infrastructure. It checks that your API returns correct responses from the outside, like a synthetic user. Prometheus collects internal metrics (latency, error rates) from inside your infrastructure. You want both: Prometheus for internal health, Postman Monitors for external uptime verification.

Related Topics

API Testing

Unit, integration, and load testing your REST API before it reaches production.

Error Handling

Consistent error responses that make debugging and monitoring easier.

Rate Limiting

Protect your API with throttling — a key part of saturation monitoring.

Microservices

Distributed tracing is especially important in microservices architectures.

gRPC vs REST

Observability considerations differ between gRPC and REST APIs.

HTTP Headers

Request-ID, tracing headers, and deprecation headers for observability.