Microservice observability: logs, metrics and traces
When a request crosses ten services, one log file cannot explain a slow checkout. Logs, metrics and traces each answer a different question. Here is how they fit.
In a single application, a slow request is a matter of reading one log. In a system of microservices, the same request passes through a gateway, three services, a queue and two databases. Each one says it was fine. The user still waited eight seconds.
Observability is how you answer "what happened?" without guessing. It rests on three signals.
Three signals, three questions
| Signal | Question it answers | Example |
|---|---|---|
| Metrics | Is something wrong? | Checkout latency doubled at 14:05 |
| Traces | Where is it wrong? | 2.4 seconds were spent in the inventory service |
| Logs | Why is it wrong? | Database connection timed out, pool exhausted |
Each is weak alone. Together they take you from an alert to a cause.
Metrics: the early warning
Metrics are numbers measured over time. They are cheap to store and ideal for dashboards and alerts. For every service, start with three:
- Rate: requests per second,
- Errors: the share of requests that fail,
- Duration: how long requests take, as percentiles and not as an average.
An average hides the slow requests your unhappiest users experience. The 95th and 99th percentiles show them.
Traces: the journey of one request
A trace follows a single request across services. Each service adds a span with its start time and duration, and passes a trace ID to the next service in a request header. The result is a timeline: which service called which, and where the time went.
Traces are what make a distributed system readable.
Logs: the detail
Logs record individual events. They become far more useful when they are structured and when every line carries the trace ID. Then "show me the logs for this slow request" is one filter.
Use one standard to collect them
OpenTelemetry is the vendor-neutral standard for producing all three signals. Its libraries instrument common frameworks, HTTP clients and database drivers automatically, and the data can be sent to the backend of your choice. Instrumenting once and keeping the freedom to change tools later is worth a lot.
A worked example
- An alert fires: the 95th percentile of checkout latency is above two seconds.
- You open a slow trace. Most of the time sits in a single span, the call to the inventory service.
- The metrics for that service show CPU at its limit on one instance.
- The logs for the same trace ID show repeated database connection timeouts.
The cause is an exhausted connection pool. Finding it took four steps and no guesswork.
Where to start
- Structured logs with a request ID in every service.
- Rate, error and duration metrics, with one dashboard per service.
- Tracing on the most important path, such as checkout or login.
- Alerts on what users feel, such as latency and failed requests, not on every CPU spike.
Observability tells you about failures. Handling them well is the other half: see error handling in microservices.