Databricks 谈 AI 智能体为何难以证明 ROI：速度不是正确的生产力指标

Amber Roberts, Staff Technical Marketing Manager at Databricks, has spent the last several years watching how AI actually gets used inside organizations, not just how it's pitched. In this conversation, she talks through why "faster" is the wrong way to think about agentic AI, why so many companies are struggling to show ROI on their AI investments, and what has to be true for an AI agent to be useful at work instead of just impressive in a demo.

Why speed is the wrong way to measure AI productivity

Most conversations about AI productivity are still about speed: doing the same tasks faster. Why is that the wrong lens for thinking about agentic AI, and what other KPIs should teams consider?

Amber Roberts: Productivity is tied to whatever a team's actual KPIs are, and that's different depending on where you sit. For some teams, the work is detail-oriented and speed isn't the biggest issue. For others, latency is the whole ballgame. Even the word "speed" means different things: speed of development versus speed of inference, for example.

Engineering productivity and marketing productivity aren't the same thing, and they shouldn't be measured the same way. So I'd push back on the idea that there's one right KPI to swap in. It depends on the team, and honestly, it depends on the person. Someone who's learning a domain from scratch is going to use AI differently, and value different things, than someone who already knows exactly what they want to build and just wants to automate the tedious parts.

I'm skeptical of headlines that go all in on "this is the wrong way to think about it." There's a lot of gray areas. Even people on the same team can have different priorities for what they want out of AI, depending on their own expertise.

The AI ROI gap: why adoption is outpacing proof

There's a growing body of research showing companies are adopting AI fast but struggling to show real ROI. Why does that gap exist, and what does it reveal about how most organizations are deploying AI agents today?

Amber Roberts: The people buying AI and the people using it day to day are often different people. The buyers don't want to be late to the game. Their job depends on staying at the cutting edge of technology, even if they're not the ones implementing it. If they see competitors buying and deploying AI, they're going to want to make a deal happen. That's true of any new technology: nobody wants to be the one left behind.

Then, over six months to a few years, you start to see whether it actually built ROI, whether it was worth what was spent. That's the phase a lot of organizations are in right now: looking at what they bought, rolling it out broadly, and figuring out which use cases it's effective for. A lot of it is trial and error. But that trial and error is still better than being six months behind the competition.

Redefining productivity for an agentic world

How do you define productivity in an agentic world, and how is that different from how enterprises have measured it for the last 20 years?

Amber Roberts: The classic definition of productivity is how effectively inputs like labor, time, and capital turn into outputs like goods and services. What's interesting is that AI doesn't output a good, it outputs a service, optimizing something behind the scenes.

The key shift is this: productivity used to be measured by how much work one person could do. Now it's how much work a person can do with a team of assistants at their disposal. We see this even in how we evaluate people, in interviews or take-home assessments. It's no longer impressive to see how much someone produced alone in four hours, because there's a visible difference between that and what someone produces when they're leveraging AI well. That's the new baseline. But it doesn't replace deep knowledge of the subject matter. It just means the same instinct, the same taste, the same expertise, is now applied at a different scale.

Where I see the biggest jump is when people take everything that used to eat their day, the recurring, schedulable, "hire an assistant to do this" work, and automate it so they can focus on the harder, more creative parts of the job. The parts that require actual pushback and judgment, because AI's default mode is agreeable. It'll say "sounds good" to almost anything. That's useful for an assistant, but it's also exactly why the parts of a job that require someone to argue, to push back, to say "no, that's not right," matter more now, not less.

That's also the part that gets missed when people report productivity gains without a quality metric attached. Productivity is technically linked to efficiency, and it assumes you're not sacrificing quality to get there. If a company says they're 40% more efficient, that's really only meaningful if quality held steady. Otherwise you might just be taking the same amount of time to produce the same result, with more moving parts. That tracks with what we're seeing in content specifically: when you push volume up, without extra attention to the details that make something good, quality tends to slip. The tools don't always keep up.

Why business context beats raw task speed

A lot of AI tools can execute a task. What does it actually mean for an agent to "understand business context," and why does that matter more than speed?

Amber Roberts: I think about it like comparing an AI agent to a real human assistant. If you ask an assistant to do something and they don't have your context, your schedule, or a sense of what's happened over the last few months, they can't do the job well. Imagine asking someone to plan an event with no guest list, no inventory, no record of what was done last time, what worked, what didn't, who the vendors are and what they specialize in. You can't just swap that out. That's exactly why people with deep institutional knowledge are so hard to replace: it's far more expensive to lose that person and train someone new than it is to keep them.

That's where the value of business context comes in. You can get a lot of nice demos of "here's how you build this thing," with genuinely good generic instructions. But if you ask, "how do I build an addition onto my house," a generic AI tool will explain how additions generally work. It won't know your zip code, your local building codes, your city's ordinances. It has no information specific to your situation. And that's basically useless if what you're actually trying to build is something for your company, which is the entire reason you'd want AI at work in the first place.

Where the high-value, repeatable use cases are showing up

What are some examples of high-value, repeatable use cases emerging right now?

Amber Roberts: Honestly, there are too many to point to one. Engineering is a good example: scheduling jobs, or using something like Genie Spaces so you don't have to be an engineer to understand the code, you can just type in a question and get slides, documentation, and context back, including who else in the organization is already working on the same problem.

But it's really across every function: health, marketing, sales, finance. Every industry, and every team within it, has automated workflows that used to take a long time and don't anymore. It's a genuinely novel moment. There are opportunities that just haven't been touched or thought of yet, which makes this a pretty unique space to be working in right now.

A concrete example: I recently had to step in for a colleague managing speaker confirmations ahead of a conference. The manual version of that process is emailing each speaker individually, or at best sending out a form and manually compiling responses. Instead, I set up an agent to message all the speakers, log their responses, and flag who hadn't confirmed. A second agent reviewed the outstanding responses and followed up automatically. By the time I checked in, all 137 speakers had been contacted and confirmed, aside from two who happened to be on parental leave. That's the kind of repeatable, cross-functional task that's ripe for this right now.

Measuring success: old metrics still matter, but they're not enough on their own

How should enterprise leaders measure whether an agentic deployment is working? What metrics or signals should replace old productivity measures like tickets closed or hours saved?

Amber Roberts: I don't think traditional metrics go away, they're still how most organizations define success. Engineers especially aren't going to be convinced something's working unless they see their existing metrics improve. So you have to start there: if a task used to take an hour and an agent can do it in five minutes, that's the kind of result that actually convinces people.

Hours saved, tickets closed, these still matter, and they keep a human in the loop. What changes is what you add on top. You can bring in AI-specific metrics: a third-party LLM acting as a judge against criteria you define. For something like a chatbot, you'd still track resolution time and user satisfaction, but now you can also check for correctness, hallucinations, tone, and flag edge cases or anomalies automatically, instead of manually sifting through every piece of feedback yourself.

The most common mistake: scaling before it's ready

What's the most common mistake you see organizations make when they try to scale agentic AI, the thing that turns "faster" into "faster in the wrong direction"?

Amber Roberts: The biggest mistake is trying to scale something before it's working well. A few examples go well, people get excited, and the instinct is "let's scale this." But "scale" has become a bit of a buzzword. The real question is: what specifically do you want to see happen? It has to be evaluated use case by use case. Sometimes the thing doesn't need to be scaled out at all.

A lot of the time, what looks like a scaling problem is actually a readiness problem: it took a lot of iteration just to get one instance working well, and that gets overlooked. It's the same pattern as when a team says, "we have all this data, let's build on it," without checking whether it's the right data. Other times, it's an infrastructure gap. Demos from smaller startups can look great and make scaling look simple, but without the infrastructure to support it, there's nothing to scale against. That's a lot of what optimization work behind the scenes is about: building the infrastructure that makes scaling possible in the first place, not just promising that it will happen.

What a day's work looks like five years from now

What does "a day's work" look like for a knowledge worker five years from now, and what has to be true organizationally for that to happen well?

Amber Roberts: I genuinely can't predict that with confidence. I couldn't have told you two years ago where we'd be today. I read something I wrote two years ago recently and barely recognized how far off it already was.

What I do think holds true is that critical thinking is more important now than it's ever been. There used to be a baseline of trust when you received a document, a video, or a piece of work: you could be nitpicky about specific details, but you generally trusted the source. Now you genuinely don't know what you're getting. You have to go into everything, before you even review it, assuming it could be AI-generated, and AI-generated without context, meaning it could be using outdated information, wrong information, or even a competitor's information without anyone realizing it.

That means everyone is going to have to stay ready to critically evaluate everything they see and everything they put out. I don't think "too polished" reads as trustworthy anymore. People are going to start treating anything that sounds too agreeable, too smooth, as a signal that it might be AI-generated without real judgment behind it. Which is exactly why having actual experts and advocates in the loop, people who know enough to push back, is going to matter more, not less, as this gets more capable.

The real measure of productivity Across every question, the same idea kept surfacing: agentic AI doesn't make old measures of productivity obsolete, it makes them incomplete. Hours saved and tickets closed still matter, but they only tell you what got faster, not what got better. The organizations getting real value aren't the ones chasing speed for its own sake. They're the ones giving agents the business context to make good decisions, keeping humans in the loop for judgment and pushback, and being honest about when something is ready to scale versus when it just looks good in a demo.

See how Genie helps every user ask questions in natural language and get trusted answers grounded in governed enterprise data.