Metrics vs Logs vs Traces: The Three Pillars of Observability
Your dashboard is green, your logs are 40GB a day, and you still can't say why that one request took nine seconds. The three telemetry types answer three different questions — and knowing which one to reach for is the difference between debugging and guessing.
Green Dashboards, Angry Users
A customer reports that checkout took nine seconds. You open the dashboard: CPU is fine, memory is fine, the p50 latency chart is a flat, reassuring line. Every service reports healthy.
You grep the logs. Seven services, seven log formats, millions of lines, and no way to tell which lines belonged to that request. You find a slow database query in one service, but you can't tell whether it's related or just something that happens all day.
Nothing is broken enough to alert on. The system is telling you it's fine, and the user is telling you it isn't. That gap is the difference between monitoring — watching things you decided in advance to watch — and observability — being able to ask a question you didn't anticipate.
Three data types close that gap, and they're not interchangeable.
Three Questions, Three Answers
Metrics are numbers aggregated over time — request rate, error count, latency histogram, queue depth. They answer is something wrong, and how much?
Logs are timestamped records of discrete events, ideally structured as key-value data rather than prose. They answer what exactly happened in this component?
Traces follow one request across every service it touches, as a tree of timed spans. They answer where did the time go, and which hop failed?
| Metrics | Logs | Traces | |
|---|---|---|---|
| Shape | Numeric time series | Discrete event records | A tree of spans, one request |
| Answers | Is something wrong? How much? | What happened here? | Where did the time go? |
| Cost model | Cheap — cost scales with cardinality, not traffic | Expensive — scales with volume | Moderate — sampled |
| Cardinality | Must stay low | Unlimited | Unlimited (per trace) |
| Retention | Months to years, cheaply | Days to weeks | Days |
| Alert on it? | Yes — this is what alerts are made of | Rarely — on error patterns | No — but sample slow ones |
| Use it to | Detect and quantify | Diagnose within a service | Localize across services |
The workflow that ties them together: a metric tells you error rate jumped at 14:02. A trace shows you that the failing requests all stall on the inventory service's call to Redis. A log line in that service, tagged with the same trace id, tells you exactly which connection error it hit. Detect, localize, diagnose. Skipping a step means guessing.
Metrics: Cheap, and Bounded by Cardinality
A metric is a number with labels, sampled over time. http_requests_total{service="checkout", status="500"}. The storage cost is essentially independent of your request rate — a million requests per second still produce one counter increment per scrape interval. That's why metrics carry your alerts and your long-term trends.
The constraint is cardinality: every unique combination of label values is a separate time series. Add user_id as a label and a million users become a million time series. This is the single most common way teams destroy their metrics backend — not with too many metrics, but with one metric carrying an unbounded label.
⚠️ Never put an unbounded value in a metric label
User ids, request ids, email addresses, full URL paths with ids in them, trace ids. Each unique value is a new time series with its own memory and index cost, and the failure isn't gradual — the backend falls over. Keep labels to bounded sets: service, endpoint template (/orders/{id}, not /orders/8412), status class, region. High-cardinality context belongs in logs and traces, which are built for it.
Two other things worth getting right:
Averages hide everything. A mean latency of 200ms is consistent with every request taking 200ms and with 95% taking 50ms while 5% take 3 seconds. Record histograms and alert on p95 or p99. The average is the one number that reliably describes nobody's actual experience.
Measure user-visible symptoms, not machine internals. CPU at 90% is not an incident if requests are fast; CPU at 20% is an incident if they're failing. The RED method — Rate, Errors, Duration per service — gives you three numbers that actually track whether users are being served. Alert on those; keep the resource metrics for diagnosis.
Logs: Unlimited Detail, Unlimited Bill
Logs are where the specifics live: the stack trace, the malformed payload, the parameter that was null. Nothing else gives you that.
They're also the most expensive telemetry per unit of insight, because volume scales linearly with traffic. Two disciplines keep them useful.
Structure them. A line of prose is unqueryable at scale. Emitted as JSON with consistent field names, the same event becomes searchable, aggregatable, and joinable.
# Unstructured — grep and hope
[2026-08-30 14:02:11] ERROR Payment failed for user 8412 after 3 retries
# Structured — queryable, correlatable, aggregatable
{"ts":"2026-08-30T14:02:11Z","level":"error","event":"payment_failed",
"user_id":8412,"attempts":3,"provider":"stripe","err":"connection_reset",
"trace_id":"4bf92f3577b34da6","span_id":"00f067aa0ba902b7","service":"checkout"}
# Now: count payment_failed by provider over 24h. Or open the trace.Sample the boring ones. Debug and info logs at full volume during an incident are exactly what you want, and the rest of the time they're a bill. Keep errors and warnings at 100%, sample the high-volume success paths, and make the level runtime-adjustable so you can turn detail up on the service you're investigating without redeploying.
🔴 Logging sensitive data is a breach waiting for an audit
Logs get shipped to third-party platforms, replicated, backed up, and read by people with no need to see the contents. A logged password, token, card number, or full request body containing PII is now in all of those places, usually with looser access controls than your database and a retention policy nobody reviewed. Redact at the point of emission — a filter in your logging layer — not in the pipeline downstream, because the pipeline is exactly what you can't fully trust.
Traces: The One That Solves the Nine-Second Checkout
A trace follows a single request across process boundaries. Each unit of work is a span with a start time, a duration, attributes, and a parent — so the whole request forms a tree showing exactly where time was spent and who called whom.
The mechanism that makes tracing work: a trace id created at the edge is passed to every downstream call, conventionally in the W3C traceparent header. Each service creates child spans under the id it received. Break the chain in one service — a background thread, a queue hop, an HTTP client that doesn't inject headers — and every service beyond that point disappears from the trace. Most "tracing doesn't work" problems are broken propagation, not broken instrumentation.
POST /checkout 8,940ms ├─ auth.verify_token 12ms ├─ cart.get_items 31ms ├─ inventory.reserve 8,610ms ← │ ├─ redis.get stock:sku_1183 5,002ms ← timeout, then retry │ └─ postgres.SELECT ... FOR UPDATE 3,601ms ← lock contention ├─ payment.charge 241ms └─ order.create 44ms The p50 dashboard never showed this. One hop, two problems.
That output is the whole argument for tracing. No amount of per-service metrics tells you that this request spent 5 seconds waiting on a Redis timeout inside a service two hops down. The tree does it at a glance.
Sampling is how tracing stays affordable. Head-based sampling decides at the edge — keep 1% of traces — which is simple but throws away most of the slow and failed ones, since they're rare by definition. Tail-based sampling buffers the complete trace and decides after seeing the outcome, keeping everything that errored or exceeded a latency threshold plus a small random baseline. Tail-based costs more infrastructure and is worth it: the traces you want are precisely the unusual ones.
✅ Put the trace id in every log line and every error response
This is the highest-value hour of observability work you can do. One shared field turns three separate tools into one investigation: from a slow trace, jump to that request's logs; from an error log, open the full trace. Return it in a response header too, so a customer support ticket can carry the exact identifier that reconstructs what happened.
Where to Start
Instrumenting everything at once is how observability projects stall. The order matters.
RED metrics per service, and alert on them
Rate, errors, and duration (as a histogram) for every service. This is small, cheap, and it's what tells you something is wrong before a customer does. Alert on symptoms users feel — error rate and p99 latency — not on CPU.
Structured logs with a trace id field
Convert log output to structured JSON with consistent field names, and make sure every line carries the current trace id. Even before you have tracing, this makes logs joinable across services.
Tracing at the boundaries first
You don't need every function instrumented. Spans at service entry points, outbound HTTP calls, database queries, cache calls, and queue publishes cover the overwhelming majority of real latency. Auto-instrumentation gets you most of this without code changes.
Emit through OpenTelemetry, not a vendor SDK
OTel is the vendor-neutral standard for all three signals, and it means your instrumentation isn't a rewrite when you change backends. Run a collector so you can route, sample, and redact centrally rather than redeploying every service to change telemetry behaviour.
What are you trying to find out?
Key Takeaways
- Three signals, three questions. Metrics say something is wrong; traces say where; logs say what exactly. Using one to do another's job is why debugging drags.
- Metrics are cheap until cardinality kills them. Never label with user ids, request ids, or raw paths. Bounded labels only; put the specifics in logs and traces.
- Alert on symptoms, not resources. Rate, errors, and p99 duration reflect user experience. CPU and memory are diagnosis, not detection.
- Averages hide the tail. Record histograms and read percentiles — the mean describes nobody's actual request.
- Structure your logs and redact at emission. Unstructured logs aren't queryable at scale, and sensitive fields end up replicated across systems you don't control.
- Traces need unbroken context propagation. One service that doesn't forward
traceparenterases everything downstream of it. - Tail-based sampling keeps the traces worth keeping — the slow and failed ones are rare, which is exactly why head-based sampling discards them.
- Put the trace id in every log line and error response. It's the single field that turns three tools into one investigation.
References
- OpenTelemetry documentation — the vendor-neutral standard for metrics, logs, and traces, plus auto-instrumentation
- W3C Trace Context — the
traceparentheader format that makes cross-vendor propagation work - Google SRE Book — Monitoring Distributed Systems — the four golden signals and what deserves an alert
- Observability Engineering — Charity Majors, Liz Fong-Jones, George Miranda — the argument for high-cardinality, event-based observability over dashboards