SafeChat AI Builds Real-Time Safety System

DoorDash has built an AI-powered safety system called SafeChat to monitor real-time conversations between customers, delivery workers, and merchants—blocking abusive messages before they reach the other party.

Millions of messages, fractions of a second

Every day, the platform processes over 4 million chat messages, 400,000 phone calls, and 200,000 images exchanged during deliveries. Each message must be classified as safe or unsafe in under a second to avoid disrupting the user experience.

Bruna Pereira, a software engineer who leads DoorDash’s trust and safety team from São Paulo, said the initial instinct was to route every message through a large language model. But with average LLM response times ranging from 2 to 10 seconds—and a daily volume that would make costs unsustainable—the team needed a smarter approach.

“We spent months understanding our data,” Pereira said. “Most messages are safe. Only a small single-digit percent are unsafe. That told us we could build a cheap, fast layer that would be right almost all the time.”

A metal detector for messages

The first layer is a lightweight machine-learning model trained on DoorDash’s own data. It responds in under 100 milliseconds and filters out 90% of messages as clearly safe. Only the remaining 10%—the ones the model flags as potentially unsafe—are sent to an LLM for deeper analysis.

Instead of asking the LLM a simple yes-or-no question, DoorDash asks it to score messages across multiple categories: profanity, threats, sexual content, and disrespect. This scoring system allows for graduated responses. A low-severity message might just be censored. A high-severity threat could trigger an automatic order cancellation and account warnings.

For voice calls, the system transcribes the conversation in real time. If it detects abuse, it can hang up the call immediately. Images go through a similar pipeline, using a commercial vision API as the first layer before escalating to an LLM if needed.

After implementing SafeChat, DoorDash saw a 50% reduction in verbal abuse incidents—a metric Pereira called “real reduction in human harm.”

From system to platform

Once SafeChat was live, other teams at DoorDash started asking for similar moderation tools. Some wanted to screen profile pictures or merchant names. Others needed fraud detection in chat. Instead of rebuilding the system for each use case, the team turned SafeChat into a content-agnostic moderation platform.

The platform lets teams configure their own moderation pipelines without writing code. They can mix and match three types of models: internal models trained on DoorDash’s data, external vendor APIs, and custom LLM prompts. A drag-and-drop interface lets users define conditions, fallbacks, and actions—like blocking a message or canceling an order.

Backtesting is built in. Teams can test their pipelines against historical data, with humans manually labeling results to measure accuracy. Pereira said a thousand labeled examples is usually enough to build confidence before going live.

This flexibility came at a cost. The original SafeChat system was scrapped, though the team kept the trained models and data. “We didn’t throw away the learnings,” Pereira said. “We threw away the system itself.”

For all its sophistication, the platform still relies on human feedback. When agents flag missed abusive messages, the team identifies patterns—like abbreviations the model didn’t catch—and retrains the system. The process is iterative, with the first layer of the model now on its ninth version.

Balancing speed, cost, and accuracy

The internal models are cheap because DoorDash only pays to deploy them, not per call. Training happens once, though the team has retrained the SafeChat model nine times to adapt to new patterns. External models, like vision APIs, are used when they’re already good enough—avoiding the need to rebuild solutions from scratch.

Latency remains a challenge. If an LLM or external API fails, the platform can fall back to another model, even if it’s slower. For chat, where speed is critical, the team avoids relying on LLMs as fallbacks due to their higher latency.

Pereira’s advice to other teams: “If you have a hot path with high volume, put a cheap model in front of the LLM. The economics only work if you ask the LLM only the hard questions.”