Skip to main content

How Instacart's Blueberry AI Helps On-Call Engineers Fix Outages Faster

Instacart's Blueberry AI assistant helps on-call engineers diagnose production incidents in minutes, using years of incident history and multi-agent reasoning directly in Slack.

When the Grocery App Goes Down

There's a special kind of panic that hits when a grocery delivery app starts failing during peak hours. Customers can't order, drivers can't pick up batches, and somewhere in a dark room, an engineer is staring at a dashboard full of red. At Instacart, that engineer now has a new kind of backup: an AI assistant named Blueberry.

Blueberry isn't a flashy chatbot that answers trivia. It's an AI incident response system built to help on-call engineers work through production failures faster. The company rolled it out in April, and it's already handled around 25,000 diagnostics across more than 270 Slack channels. That's a lot of fires put out, or at least triaged.

The Pain of Starting from Zero

If you've ever done on-call work, you know the drill. An alert fires. You open your laptop, and then the real work begins: figuring out which service owns the problem, checking recent deployments, digging through logs and metrics, hunting for documentation, and trying to remember if you've seen something like this before. It's time-consuming and mentally draining, especially at 3 a.m.

Instacart built Blueberry to cut through that noise. Instead of making engineers start from a blank slate, the AI collects context and generates hypotheses before the human even gets their hands on the keyboard. The goal is to turn hours of investigation into minutes of focused troubleshooting.

How Blueberry Actually Works

Blueberry is not a single monolithic bot. It's a system of about ten sub-agents that spin up in parallel when an alert triggers. These agents pull information from internal tools and resources, then write their findings directly into the Slack thread where the engineer is already working.

The system connects to a range of data sources: incident history, service ownership info, logs, deployment records, and other debugging signals. It doesn't just rely on a general-purpose language model. It's wired into Instacart's specific operational knowledge, which makes its suggestions far more relevant than a generic AI could be.

From 60% to 90% Accuracy

The company says Blueberry's diagnostic accuracy jumped from 60% to over 90% after they fed it more than 14 years of incident history. That's a big deal. It means the AI is not just guessing—it's actually learning from past failures and applying that knowledge to new ones.

According to Instacart's CTO Anirban Kundu, the system typically produces initial findings in about three minutes. That's a far cry from the old way, where an engineer might spend twenty minutes just figuring out where to look.

AI as a Force Multiplier, Not a Replacement

One important thing: Blueberry doesn't take action on its own. It doesn't change code or restart services. It gathers information, generates hypotheses, and assists with debugging—but the final call always stays with a human engineer.

That's a deliberate choice. In production environments, you want AI to be a second pair of eyes, not the one holding the scalpel. Instacart's engineering VP Siby Alappatt calls it a "force multiplier"—it makes on-call engineers more effective without removing human judgment from the loop.

Learning from Every Incident

Another smart piece of this system: it remembers. Blueberry integrates past incident cases and team-specific context, so when a similar problem pops up again, engineers can benefit from what was learned before. This turns incident response into a continuous learning loop, not a series of isolated fire drills.

The company also emphasizes that the architecture is "persistent and tool-aware." That means the AI keeps track of the investigation progress across sessions and can call external tools when needed. It's not just a chat window—it's an active participant in the debugging process.

Numbers That Back It Up

Instacart shared some impressive stats: in one month, the system handled about 25,000 diagnostics, with a workflow success rate of 99.9%. It also made more than 58,000 MCP tool calls and adapted to the working styles of about 60 different teams. That's a lot of variation in how teams communicate and solve problems, and Blueberry seems to handle it gracefully.

Software engineering director Alan Wong notes that Blueberry changes the starting point for on-call engineers. Instead of starting from zero, they start with a stack of context—logs, deployment info, related systems—that the AI has already pulled together. That shift alone can save precious minutes when every second counts.

What This Means for the Future of On-Call

Instacart's experience shows that effective AI operations aren't just about the model. The real magic is in the engineering framework around it: operational context, specialized workflows, tool integration, and feedback loops. Blueberry works because it's deeply embedded in Instacart's existing tools and processes, not bolted on as an afterthought.

For other companies dealing with complex systems, this is a promising blueprint. AI won't replace on-call engineers anytime soon, but it can make their jobs a lot more bearable. And when you're woken up at 2 a.m. by a pager, that's worth a lot.

So next time your grocery delivery arrives on time, there's a chance a machine and a human worked together to keep the system running smoothly. Blueberry might not be the hero you see, but it's definitely one of the helpers behind the scenes.

Share this article:

Comments (0)

No comments yet. Be the first to comment!