Centralized Logging: From ‘grep’ to Useful Dashboards

Quickstart

If you want centralized logging that’s immediately useful (and not a giant “log dump”), do these steps in order. You can implement the first three in a day and see instant improvement during debugging and on-call.

Overview

“Centralized logging” means your logs are collected from every service, stored in one place, and searchable with a shared set of fields. It’s what turns debugging from “SSH into a box and grep” into “click a dashboard, filter a trace, fix the bug.”

This post focuses on the practical side: designing log events, building a minimal schema, collecting and enriching logs, and creating dashboards/alerts that help during incident response. Tooling varies (Elastic/OpenSearch, Loki, Splunk, cloud-native log services), but the principles stay the same.

Core concepts

1) Log event vs log message

A traditional “log message” is a human sentence. A log event is a record with fields. Centralized logging systems work best when your logs behave like events: stable keys, predictable values, and enough context to filter without guessing.

2) Structured logging (and why JSON wins)

You can centralize unstructured text, but you can’t reliably build dashboards and alerts from it. JSON isn’t magical—it's just a shared format that makes parsing and querying predictable. The goal is stable keys (schema) and repeatable meaning (conventions).

3) Correlation IDs: the bridge from “grep” to “trace the request”

The biggest “quality jump” in centralized logging is correlating events. A single request often touches multiple services; without a shared ID, you’re guessing. Add a request_id (or OpenTelemetry trace_id) to every log produced during that request.

4) Cardinality and cost: dashboards don’t like “infinite labels”

Logging systems index fields so you can filter quickly. Some fields (like service or env) are low-cardinality and safe. Others (like user_id, request_id, or raw URLs with IDs) can explode storage and query performance. The trick is to decide what should be indexed vs what should stay as payload.

5) Security and PII: logs are production data

Logs often outlive databases and backups because “retention” is convenient. Treat logs as sensitive: redact secrets, avoid storing PII unless absolutely required, and limit access via roles.

Step-by-step

This is a practical, tool-agnostic workflow for centralized logging. Whether you run Kubernetes, VMs, or serverless, the same pipeline applies: emit structured events → collect reliably → enrich → store → query → dashboard → alert.

Step 1 — Define what “useful” means (before you ship logs)

Step 2 — Implement structured logging at the app boundary

Don’t start by parsing logs downstream. Start by emitting good events at the source. Your services should write newline-delimited JSON to stdout (containers) or to a file (VMs). Keep the schema small, then add fields intentionally.

import json
import logging
import os
import sys
import time
import uuid

SERVICE = os.getenv("SERVICE_NAME", "api")
ENV = os.getenv("ENV", "dev")

class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        payload = {
            "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(record.created)),
            "level": record.levelname,
            "service": SERVICE,
            "env": ENV,
            "message": record.getMessage(),
            # optional context (set these per request/job)
            "trace_id": getattr(record, "trace_id", None),
            "request_id": getattr(record, "request_id", None),
            "route": getattr(record, "route", None),
            "status": getattr(record, "status", None),
            "duration_ms": getattr(record, "duration_ms", None),
        }

        # Attach structured extras safely (avoid huge blobs; redact as needed)
        extra = getattr(record, "extra", None)
        if isinstance(extra, dict):
            payload["extra"] = extra

        # Include exception details in a structured way
        if record.exc_info:
            payload["error"] = {
                "type": record.exc_info[0].__name__,
                "message": str(record.exc_info[1]),
            }

        # Drop nulls to keep logs small and queries cleaner
        payload = {k: v for k, v in payload.items() if v is not None}
        return json.dumps(payload, separators=(",", ":"), ensure_ascii=False)

logger = logging.getLogger("app")
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)
logger.setLevel(logging.INFO)

def handle_request(route: str):
    request_id = str(uuid.uuid4())
    trace_id = request_id.replace("-", "")[:16]  # placeholder; prefer real tracing IDs
    start = time.time()
    try:
        # ... your logic here ...
        time.sleep(0.05)
        logger.info("request completed", extra={"request_id": request_id, "trace_id": trace_id, "route": route, "status": 200, "duration_ms": int((time.time() - start) * 1000)})
    except Exception:
        logger.exception("request failed", extra={"request_id": request_id, "trace_id": trace_id, "route": route, "status": 500, "duration_ms": int((time.time() - start) * 1000)})

handle_request("/health")

Step 3 — Make log levels meaningful (so alerts aren’t noisy)

Log levels are a contract. If everything is INFO, nothing stands out. If everything is ERROR, alerts will be useless. A simple policy is enough:

Step 4 — Collect, enrich, and forward logs reliably

Your collector/agent (Fluent Bit/Fluentd, Filebeat, OpenTelemetry Collector, cloud agents) should do three jobs: collect logs, enrich them with metadata, and forward with buffering/retries. Keep parsing rules small and predictable.

receivers:
  filelog:
    include:
      - /var/log/containers/*.log
    start_at: beginning
    operators:
      # Parse container runtime wrapper (common in Kubernetes)
      - type: regex_parser
        regex: '^(?P<time>[^ ]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) (?P<body>.*)$'
      # Parse JSON emitted by the application
      - type: json_parser
        parse_from: body
        # If parsing fails, keep original body; don't drop the log
        on_error: send

processors:
  resource:
    attributes:
      - key: deployment.environment
        value: prod
        action: upsert
      - key: service.name
        value: api
        action: upsert
  attributes:
    actions:
      # Example redactions (defense-in-depth; also redact in-app)
      - key: http.request.header.authorization
        action: delete
      - key: password
        action: delete

exporters:
  otlp:
    endpoint: logs-backend:4317
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [resource, attributes]
      exporters: [otlp]

Step 5 — Decide storage, retention, and access controls

Centralized logging can get expensive because logs are high volume. Retention is not an afterthought—it's part of the design. Keep retention aligned to your operational needs: recent incidents, audits, debugging windows.

Step 6 — Keep “grep habits”, but upgrade them with structure

Centralized logging doesn’t replace your local debugging muscle memory—it upgrades it. You still filter, you still narrow down, but now you filter by fields. That makes queries safer, faster, and easier to share in incident channels.

# 1) Find recent errors for a deployment (last 10 minutes) and show key fields
kubectl logs deploy/api --since=10m \
  | jq -cr 'select(.level=="ERROR") | {ts:.timestamp, route:.route, status:.status, msg:.message, trace:.trace_id}'

# 2) Follow one request across logs using a trace_id/request_id
TRACE_ID="c6b3a1f2e4d5c6b3"
kubectl logs deploy/api --since=30m \
  | jq -cr --arg tid "$TRACE_ID" 'select(.trace_id==$tid or .request_id==$tid) | {ts:.timestamp, svc:.service, lvl:.level, msg:.message}'

# 3) Quick latency scan: show the slowest request summaries (top 20)
kubectl logs deploy/api --since=15m \
  | jq -cr 'select(.duration_ms != null) | [.duration_ms, .route, .status, .trace_id] | @tsv' \
  | sort -nr \
  | head -n 20

Step 7 — Build dashboards that answer real incident questions

A “useful” logging dashboard is not a wall of charts. It’s a map from symptom → slice → example → fix. Start with a small set of panels that reflect how you investigate issues.

Step 8 — Alerts: fewer, clearer, linked to action

Logs can drive alerts, but alerting directly on raw log volume is noisy. Prefer alerts that represent a user-impacting condition (error rate spike, sustained 5xx on a route, repeated dependency failures), and always attach the query + a runbook link.

Step 9 — Iterate with incidents: logging is a product

After every incident, add one thing that would have made debugging faster: a missing field, a better error code, a new dashboard slice, or a runbook query. That’s how “grep” grows into a reliable operational system.

Common mistakes

Centralized logging usually fails in predictable ways: too much noise, not enough structure, and dashboards nobody trusts. Here are the most common pitfalls and how to fix them quickly.

FAQ

What is centralized logging, in simple terms?

Centralized logging means collecting logs from every service and storing them in one searchable place with consistent fields. Instead of SSH+grep across machines, you filter by service, env, route, status, and trace_id in a shared UI.

Do I need a full observability stack to get value from centralized logging?

No. You can get most of the value with structured logs + a reliable collector + one dashboard. Metrics and traces are great, but centralized logging is often the quickest way to improve incident response and debugging.

Should I log in JSON or keep plain text?

Use JSON for application logs. Plain text is fine for local debugging, but JSON makes parsing and dashboards reliable. Keep the message short and add details as fields.

What fields should every log event include?

At minimum: timestamp, level, message, service, env. For web APIs, add: route, status, duration_ms, and a trace_id/request_id.

How do I avoid huge logging costs?

Reduce noise and control retention. Drop debug logs by default in production, sample chatty events, avoid logging large payloads, and set retention by log type (e.g., request summaries 7–30 days, errors longer).

Can I use logs for alerting?

Yes, but carefully. Alert on conditions that map to impact (error rate spikes, sustained 5xx on a route, repeated dependency failures), and always include a link to the exact query + runbook. Avoid “any ERROR triggers a page.”

What’s the difference between logs, metrics, and traces?

Metrics tell you “how much/how often,” logs tell you “what happened,” and traces show “how a request flowed.” Centralized logging becomes dramatically more powerful when logs include a trace_id to connect events across services.

Cheatsheet

Use this as a quick checklist when setting up centralized logging or auditing an existing setup.

Wrap-up

Centralized logging isn’t about collecting more data—it’s about collecting better data: structured events, consistent fields, and correlation IDs that make incidents boring. Start small: standardize JSON logs, ship them reliably, then build one dashboard and one alert that your team uses every week.

If you want to go deeper, the related posts below cover the broader observability picture, runbooks, and deploy practices that make logs actionable.

Related posts

• Observability 101: Logs vs Metrics vs Traces (And When You Need Each)
• Runbooks That Actually Get Used
• Docker for Developers: Images, Layers, and Caching Explained
• Kubernetes Without Tears: Pods, Services, Ingress in One Story
• CI/CD That Stays Green: Pipeline Patterns That Scale
• GitOps Explained: Deployments You Can Roll Back With Confidence