When a customer places an order on a food delivery app, a single tap sets off a chain of activity. The restaurant is notified, a driver is assigned, payment is verified, and the delivery estimate is calculated and updated as the driver moves. Analytics, marketing, and fraud teams each need the same information for their own purposes.
The obvious approach is to handle everything synchronously: when an order arrives, the system notifies the restaurant, finds a driver, charges the card, and updates analytics before replying to the customer. At scale, this causes three problems. Latency increases because the customer waits for the slowest step. Resource contention grows as every request holds servers and database connections open. Tight coupling means a slow or failing analytics service can delay order placement, even though the customer never sees analytics.
Event-driven architecture removes this dependency. The order service publishes a single message, "an order was placed," and returns immediately. Every service that needs that information subscribes to it and processes it independently. A newsroom works the same way: a reporter files a story once, and the website, print and social media teams each use it as they need, without the reporter waiting for any of them.
Kafka partitions and consumer groups
In most large systems, the layer that carries these messages is Apache Kafka. Producers write events to named streams called topics, and consumers read from them. At DoorDash, Kafka is the publish-subscribe layer that provides real-time event delivery across the company [2].
A single stream of billions of events is too much for one machine, so Kafka splits each topic into partitions. This is similar to a supermarket opening several checkout lanes instead of one. Each event goes to one lane, and all lanes are processed in parallel. Events with the same key, such as the same order ID, always go to the same partition, which preserves their order.
n the reading side, a consumer group is a set of workers that share the processing of a topic. Kafka assigns each worker a portion of the partitions, so the group processes events in parallel while keeping the load balanced [3]. Throughput can be increased by adding workers, up to the number of partitions. Other teams, such as analytics, can run their own consumer groups on the same topic and read every event independently.
How DoorDash built Iguazu
Before its current platform, DoorDash relied on several separate pipelines, each built differently and each handling a single type of event. Data passed through multiple hops before reaching the warehouse, and the mix of systems made monitoring difficult, resulting in high latency, significant cost and operational overhead [1].
Its replacement, Iguazu, was described by its architect Allen Wang in a 2022 case study. Over two years, the team scaled it to process "hundreds of billions of events per day with a 99.99% delivery rate" [1]. In a later talk, Wang noted that "the end-to-end latency to Snowflake is reduced from a day to just a few minutes" [4]. A separate DoorDash post on its streaming platform reports that "almost six billion messages are processed each day at an average rate of four million messages per minute, which sometimes peaks at double that rate" [2].
Four design choices stand out:
- Simplified event producing. Services send events to a customized Kafka REST proxy instead of integrating and tuning their own Kafka clients. Tuning batching with a linger time of 50 to 100 milliseconds and the sticky partitioner "reduced the Kafka broker CPU utilization by 30 to 40%" [1].
- Stream processing with Apache Flink. Flink was chosen primarily for its event-time processing, fault tolerance and integrations. A SQL layer allows analysts to define jobs in configuration rather than code [1].
- Standard event format and schema checks. Every event uses a common envelope and payload, and schemas are validated against a schema registry during the build, so incompatible changes fail before deployment [1].
- Independent pipelines and self-serve onboarding. Flink writes Parquet files to S3, and Snowpipe loads them into Snowflake, with a separate job per event type so failures stay isolated. Automated onboarding "reduced the event onboarding time from days to minutes" [1].
Consumer lag and backpressure
DoorDash's article does not cover these topics in detail, but every team operating Kafka at scale has to manage them.
Consumer lag is the gap between the newest event written to a partition and the last event a consumer has processed. If producers write 10,000 events per second and consumers process 8,000, lag grows by 2,000 every second. Short spikes during peak hours are normal; lag that keeps rising indicates that consumers need to be scaled, partitions added or a slow processing step fixed.
Backpressure is the same problem inside a processing pipeline: one stage cannot keep up with the stage before it. In a busy restaurant, if dirty plates arrive faster than the dishwasher can clean them, the kitchen eventually has to slow down. Flink handles this by slowing upstream operators instead of letting data accumulate until something fails. Persistent backpressure points directly to the bottleneck in a job.
Delivery guarantees, idempotency and failure handling
At-least-once delivery and idempotency. Most streaming systems guarantee at-least-once delivery: if there is any doubt that an event was processed, it is delivered again. This prevents data loss but means consumers will occasionally receive duplicates. The solution is to make consumers idempotent, so that processing the same event twice has the same effect as processing it once. A common approach is to assign each event a unique ID and record the IDs already handled, so a repeated payment event for the same order is skipped rather than charged twice.
DoorDash documents its own trade-offs openly. Asynchronous producing can lead to "minor data losses when Kafka is unavailable," which the team mitigates with retries in the proxy and monitoring [1]. On the warehouse side, Flink's checkpointing provides exactly-once delivery of files to S3 [1].
Dead-letter queues. Some events are malformed, with missing fields or unexpected schemas. If a consumer retries them indefinitely, it blocks every event behind it in the partition. A dead-letter queue moves such events to a separate topic after a set number of failed attempts, so processing continues and engineers can investigate later.
Replay. Because Kafka retains events for a configurable period, consumers can be rewound to reprocess data, for example after a bug fix. Kafka retention is limited, which is one reason DoorDash also stores events in S3 to support longer-term backfills [1].
Key takeaways
The same principles apply at smaller scale:
- Move work that does not need an immediate response, such as emails, reports and analytics, onto a queue or stream.
- Give every event a unique ID from the start; adding idempotency later is far more costly.
- Monitor consumer lag before it becomes a problem.
- Define an event format early and validate schemas in the build pipeline.
- Decide in advance where failed events go, so one bad record cannot block a stream.
The most transferable lesson from DoorDash is organizational rather than technical: event processing was built as a shared platform that any team could adopt quickly, rather than a collection of one-off pipelines. That approach is what allowed Iguazu to grow from a few billion to hundreds of billions of events per day [4].
References
- Allen Wang, DoorDash Engineering, Building Scalable Real Time Event Processing with Kafka and Flink (Aug 2022)
- DoorDash Engineering, API-first approach to Kafka topic creation
- DoorDash Engineering, Post on isolating test and production traffic with Kafka consumer groups
- Allen Wang, InfoQ, From Zero to a Hundred Billion: Building Scalable Real Time Event Processing at DoorDash (QCon talk)



