When something goes wrong in production, the first question is always \"what happened?\" Observability is about making our systems transparent enough to answer that question quickly. It rests on three pillars: logs, metrics, and traces.
\nLogs are the most basic form of observability. They are timestamped records of events that happened in the application. A good log entry includes enough context to understand what happened without needing to reproduce the issue.
\nStructured logging, where log entries are formatted as JSON instead of plain text, makes logs much easier to search and filter.
\n// Instead of this\nconsole.log(\"User login failed for user 123\");\n\n// Do this\nlogger.warn({\n event: \"login_failed\",\n userId: 123,\n reason: \"invalid_password\",\n ip: req.ip,\n});Log levels help separate noise from signal. debug for development details, info for normal operations, warn for recoverable problems, and error for failures that need attention. In production, we typically set the level to info or warn to avoid flooding the logs with debug output.
Metrics are numerical measurements collected over time. They tell us how the system is performing at a high level. Common metrics include request rate, error rate, response time percentiles, CPU usage, memory usage, and queue depth.
\nThe RED method is a useful framework for service metrics: Rate (requests per second), Errors (failed requests per second), and Duration (response time distribution). These three metrics give a good overview of a service's health.
\nTools like Prometheus collect metrics by scraping endpoints on our services. Grafana dashboards visualise those metrics and make trends visible. A sudden spike in error rate or a gradual increase in response time becomes obvious on a dashboard.
\nIn a system with multiple services, a single user request might pass through several services before completing. Distributed tracing follows that request through the entire chain, showing how long each service took and where failures or bottlenecks occurred.
\nEach request gets a unique trace ID that is passed along to every service in the chain. Tools like Jaeger or the OpenTelemetry SDK collect these traces and visualise them as a waterfall diagram, making it clear where time is being spent.
\nObservability data is only useful if someone acts on it. Alerts notify the team when something is wrong. Good alerts are based on symptoms, not causes. Instead of alerting when CPU usage exceeds 80 percent, alert when the error rate exceeds a threshold or response times breach the SLA. High CPU might be fine if users are unaffected.
\nAlerts should be actionable. If an alert fires and the response is \"I will look at it later,\" it is either not urgent enough to be an alert or it needs better context to guide the response.
\nStart with logging. It is the simplest to add and provides the most immediate value. Then add key metrics to understand system health at a glance. Add tracing when the architecture involves multiple services and debugging cross-service issues becomes painful. Avoid the temptation to instrument everything upfront. Instrument the critical paths first and expand as needed.
"}},"pageContext":{"slug":"/2025-03-16-observability-and-logging/"}},"staticQueryHashes":[]}