Observability Without Excess: Metrics That Actually Matter

Observability is not logging everything. It's understanding your system through minimal, high-signal data. After building monitoring cloud infrastructure design across FLIR Systems, Lorex Technology, and 1c Platform, I've learned that the teams with the most dashboards are usually the ones that miss the most incidents.
The Three Pillars (Correctly Understood)
1. Metrics: Aggregate numbers (latency percentiles, error rates) 2. Logs: Detailed events for debugging (only when needed) 3. Traces: Request flows across services (expensive, use sparingly)
The mistake most teams make is treating these as equal investments. They're not. Metrics are your daily driver. Logs are your forensic tool. Traces are your deep-debugging last resort. If you're tracing every request in production, you're burning money for signal you'll never use.
What Actually Matters
For most systems, four metrics suffice: • Request latency (p50, p95, p99) • Error rate (4xx, 5xx) • Throughput (requests/sec) • Resource utilization (CPU, memory, disk)
At FLIR, we had 200+ dashboards when I joined. Six months later, we had 12. incident response playbook time dropped 40% because on-call engineers no longer had to hunt through dozens of dashboards to find the relevant one. Less data, faster resolution.
The Sampling Strategy
Sample your traces. Logging everything at scale: • Costs 5-10x more than it should • Creates signal-to-noise problems (useful logs get lost) • Becomes a compliance and data governance liability (unintended data capture)
# Trace only 5% of requests in production, 100% in staging
def should_trace(environment, random_sample):
if environment == 'production':
return random_sample < 0.05
return TrueThe sampling rate should be configurable per service. Critical services (payments, auth) trace at 100%. Background services (reporting, analytics) trace at 1%. The cost difference is enormous, and the debugging value is preserved because you always trace errors at 100%.
The Dashboard You Actually Need
One dashboard. Four panels. Updated every 60 seconds. Anything more and you're drowning in data.
Dashboard discipline saves your on-call engineers hours of frustration during incidents. When the pager goes off at 3am, your engineer doesn't need a 47-panel dashboard. They need four numbers: is the system up, is it fast, is it erroring, is it overloaded. Everything else is a drill-down, not a top-level view.
Alerting: Quality Over Quantity
The most damaging observability problem is alert fatigue. When everything alerts, nothing alerts. At 1c Platform, we inherited a system with 500+ alert rules. 90% were noise. We reduced to 40 rules and incident response playbook improved dramatically.
The alerting philosophy: • Alert on symptoms, not causes (alert on "checkout error rate > 2%" not "CPU > 80%") • Alert on user impact, not system internals (alert on "payment failures" not "database transaction patterns connection pool") • Every alert should be actionable (if you can't fix it, don't alert on it) • Every alert should have a runbook (if there's no runbook, the alert isn't ready)
Structured Logging as a Discipline
When you do need logs, structure them. Unstructured logs are searchable only by grep, which doesn't scale. Structured logs (JSON with consistent fields) are queryable, filterable, and aggregatable.
// Bad: unstructured log// Good: structured log { "timestamp": "2024-06-01T10:30:00Z", "level": "error", "service": "payment-service", "event": "payment_declined", "order_id": "12345", "error_code": "CARD_DECLINED", "latency_ms": 450 } ```
The structured log lets you query "how many payment declines happened in the last hour" in seconds. The unstructured log requires reading every line.
The Cost of Over-Observability
Observability tools charge by ingestion volume. Logging everything at INFO level across a system handling 10k requests/second can cost $50k+/year in tooling alone. That's money spent on data nobody reads. Be intentional about what you ingest.
The best observability strategy is the one your team actually uses. A simple dashboard that every engineer checks daily beats a sophisticated platform that nobody opens.
Related Articles
Production Monitoring Strategy: From Alert Fatigue to Actionable Observability
12 min read
Cloud & DevOpsCI/CD Pipelines That Actually Improve Quality: Lessons from 40% Efficiency Gains
17 min read
Cloud & DevOpsMulti-Cloud Strategy: Avoiding Vendor Lock-In While Maintaining Velocity
8 min read

Sufi Khan Sulaiman
VP Technology & CTO with 25+ years building ecommerce platforms, enterprise systems, and AI solutions
Expertise across ecommerce strategy, cloud architecture, AI & machine learning, DevOps, and technology leadership. Led teams at FLIR Systems, Lorex Technology, 1c Platform, and Genetec.
More Articles
Explore the Full Portfolio
This is the complete portfolio of Sufi Khan Sulaiman, a technology leader specialising in B2B commerce and digital automation. Start from the Home page for the overview, then move through two decades of career experience across FLIR Systems, Lorex Technology, and 1c Platform, and the full catalogue of project case studies spanning headless commerce migrations, AI recommendation engines, and multi-channel fulfilment systems.
The skills and certifications page maps the technical and leadership capabilities behind the work, while the articles and the knowledge base break down the thinking into actionable frameworks. For hands-on learning, the tutorials and applications sections cover practical builds from front-end fundamentals to full-stack web apps.
For consulting engagement, the expertise page outlines service offerings, the ecommerce hub covers platform architecture and automation strategy, and the ecommerce guide (PDF) is a downloadable 55-page field manual. When you are ready to talk, the contact page is the direct line.