How services talk to each other — synchronous REST/gRPC for fast user-facing calls, asynchronous messaging for background work. The circuit breaker prevents cascade failures.
MediumHigh Frequency
Architecture Diagram
How It Works
Microservices communicate via two fundamental styles. The key insight: split your request flow into a synchronous path (user-facing, must be fast) and an asynchronous path (background, can be slow). Protect sync calls with circuit breakers.
Synchronous vs Asynchronous
Synchronous (REST / gRPC)
Service A calls Service B and waits for a response. Simple to understand and debug. But: creates temporal coupling — if B is slow or down, A is stuck. Chains (A→B→C→D) compound latency. Use for user-facing calls that must be fast.
Asynchronous (Message Queue / Events)
Service A publishes an event and continues without waiting. Decoupled: B processes at its own pace. Handles failures via retry + dead letter queue. But: harder to debug, eventual consistency. Use for background work.
Circuit Breaker Pattern
If a downstream service fails repeatedly, stop calling it. The circuit breaker has three states:
Open (tripped) — If error rate exceeds threshold (e.g., 50% in 30s), stop calling. Return fallback/error immediately. Timer starts (e.g., 60s cooldown).
Half-Open (testing) — After cooldown, allow one test request. If it succeeds → Closed. If it fails → Open again.
Other Key Patterns
API Gateway — Single entry point for clients. Handles auth, routing, rate limiting, protocol translation. Kong, AWS API Gateway.
Service Discovery — Services register themselves; others look them up. Client-side (Eureka) or server-side (K8s DNS, Consul).
Saga — Distributed transaction as a sequence of local transactions. Each step has a compensating action for rollback. Order→Payment→Inventory, with cancel-order as compensation.
Sidecar / Service Mesh — Proxy alongside each service handles networking (Envoy/Istio). Circuit breaking, retries, mTLS without code changes.
Key Design Decisions
⚡
Sync vs async for payments: Synchronous = user waits for payment confirmation (slow but certain). Async = instant "order placed!" but payment might fail later (fast but requires compensation). DoorDash chose async + saga: show "order confirmed" immediately, handle payment failure via cancellation.
🔄
Retry strategy: Simple retries can cause thundering herd (all clients retry at once). Use exponential backoff with jitter: delay = min(base * 2^attempt + random_jitter, max_delay). Set a retry budget (e.g., max 3 retries, max 20% of requests are retries).
🔍
Distributed tracing: When a request touches 6+ services, debugging is impossible without tracing. Assign a correlation ID at the gateway, propagate through all calls. Tools: Jaeger, Zipkin, OpenTelemetry. Non-negotiable for microservices.
🏗️
Monolith first: Don't start with microservices. Start monolithic, extract services when you have clear domain boundaries and team scaling needs. A well-structured monolith beats a poorly designed microservices architecture every time.
When to Use
"How do services communicate?" — Discuss sync vs async tradeoffs with concrete examples.
"What happens when Service B is down?" — Circuit breaker + retry with backoff + fallback response.
"How do you maintain consistency across services?" — Saga pattern + outbox for reliable event publishing.
"How does the client know which service to call?" — API Gateway for external clients, service discovery for internal.
Interview signal: Draw the sequence diagram split into sync and async paths. For every async step, explain what happens on failure (saga compensation). Mention circuit breakers for sync calls without being asked — interviewers love hearing about it.
Real-World Examples
Netflix Hystrix / Resilience4j — Netflix invented the circuit breaker library because cascading failures across their ~700 microservices kept causing outages. Every Netflix API call goes through a circuit breaker.
DoorDash — Order flow touches 6 services. Menu + Pricing (sync gRPC, <100ms). Payment + Dispatch + Notification (async via Kafka). Circuit breaker on Menu Service with cached fallback.
Envoy Proxy (Lyft/Istio) — Implements circuit breaking, retries, and load balancing as a sidecar. Services don't need library-level changes. All inter-service traffic routes through Envoy.
Uber — ~2000+ microservices. Strict timeout budgets per call. Request hedging for latency-critical paths. Full distributed tracing with Jaeger (which Uber built).