Reliability

How it works

1. Define SLOs/SLIs
2. Add redundancy and timeouts
3. Test failures/chaos
4. Automate recovery and alerts

Overview

• Design to tolerate faults and recover quickly; prioritize user-visible correctness.
• Measure with SLIs (availability, latency, error rate) against SLOs; manage risk via error budgets.
• Techniques: redundancy, timeouts/retries/backoff, isolation (bulkheads), graceful degradation, backups/DR.

When to use

• Critical user flows where downtime or data corruption is costly (payments, auth, checkout).
• Systems with strict SLAs/compliance requirements.
• Platforms with many dependencies where partial failures are common.

Trade-offs

• Extra cost for redundancy (multi-AZ/region), added complexity and latency from safety mechanisms.
• Over-engineering can slow delivery; balance with error budgets and stage rollouts.

Patterns

• Health checks + autoscaling; circuit breakers + bulkheads.
• Timeouts with bounded retries and jitter; idempotency for safe replays.
• Read replicas and automated failover; backups with regular restore tests.
• Chaos testing/game days; runbooks and automation for common incidents.

Anti-patterns

• Infinite retries without backoff or idempotency.
• Single points of failure; untested backups/DR; manual-only recovery.
• Coupled deploys creating cascading failures.

📐 Quick Diagram

      Client ▶ LB ▶ App A|B ▶ DB(primary)
                    └──▶ Cache ▶ Fallback reads on DB fail
      

❓ Interview Q&A (concise)

• Q: Reliability vs Availability? A: Reliability is correctness over time; availability is readiness to serve. You can be available but unreliable, and vice versa.
• Q: What is an error budget? A: Allowable unreliability derived from SLO; it guides release pace and risk acceptance.
• Q: Avoid cascading failures? A: Timeouts, circuit breakers, bulkheads, load shedding, and backpressure.

🎯 What is Reliability?

Reliability is like having a safety net for your system - it ensures that even when things go wrong (hardware failures, software bugs, human errors), your system keeps on ticking. A reliable system minimizes downtime and data loss, and maintains consistent performance.

🏗️ Key Concepts

🛠️ Reliability Techniques

📊 Measuring Reliability