The Inbox & Outbox Pattern: Reliable Events with Kafka 📬
How to stop losing (and duplicating) events when your service writes to a database and Kafka at the same time.
Practical patterns for retries, idempotency, and observability in Lambda-backed product APIs.
A Lambda-backed API makes a lot of reliability decisions for you — and a few of the most important ones against you. Retries happen whether you designed for them or not, cold starts hit the path you least expected, and a downstream throttle turns into a customer-facing 500. Reliability here is mostly about deciding what happens on the bad path before traffic decides for you. ⚙️
Reliable serverless APIs begin with a written list of what can fail, each with an expected response:
| Failure mode | Expected response |
|---|---|
| Downstream timeout | Fail fast, return 503, let the client retry |
| Duplicate event / retry | Idempotent no-op, same response |
| Downstream throttle | Back off and queue, don't hammer |
| Partial write | Roll back or compensate, never half-commit |
If a failure mode isn't on this list, your handler is improvising under load — the worst possible time to improvise.
Retries only help when handlers tolerate repeated execution. Make the write idempotent first, then turn up retry aggressiveness. A conditional write keyed on a stable request ID is usually enough:
PUT /orders/{orderId}
condition: attribute_not_exists(orderId)
→ 200 on first write
→ 200 (same body) on retry, no duplicate created
Idempotency keys, conditional writes, and stable request identifiers keep a retried request from creating duplicate records or inconsistent customer-visible state.
Lambda APIs work best when synchronous work is limited to validation, a durable state change, and a clear response. Everything else — notifications, exports, analytics, third-party calls — belongs behind a queue or event:
flowchart LR
C[Client] --> API[Lambda: validate +<br/>durable write]
API --> R[200 response]
API -->|emit event| Q[(Queue / EventBridge)]
Q --> N[Notifications]
Q --> An[Analytics]
Q --> Ext[3rd-party calls]A small critical path is faster, cheaper, less cold-start-sensitive, and far easier to keep reliable.
Logs and metrics should describe the API contract, not only infrastructure health. Track validation failures, downstream error classes, retry counts, cold-start-sensitive paths, and queue age — so an alert points to real user impact ("checkout p99 breached", "DLQ growing") instead of a generic "Lambda errored." Instrument what the user experiences, and your alerts start telling you what to actually do.
How to stop losing (and duplicating) events when your service writes to a database and Kafka at the same time.
What makes Kafka a distributed log rather than a queue, and when to reach for RabbitMQ or Redis instead.
Practical patterns for safer ingestion, retries, and medallion-style processing on AWS.