Executives from companies such as Microsoft, Amazon, Anthropic and Stripe agreed on a lot this week about where AI is headed. But one big question was left unresolved: who keeps the relationship with the customer, and the data that comes from it, once an AI agent is in the middle?
That question was a recurring theme at Madrona’s IA40 Summit, the venture capital firm’s annual gathering of AI startups, investors and tech executives, held Wednesday at the Four Seasons Hotel on the downtown Seattle waterfront.
As Madrona Managing Director Matt McIlwain said in his closing remarks, the question of who gets to use the data that comes from people’s engagement with AI systems came up “over and over and over again” during the day.
GeekWire attended the entire day and came up with these takeaways, reducing our own human bottleneck (see below) with lots of back-and-forth with Anthropic’s Claude and other AI tools.
AI agents are coming between companies and their customers: Charles Lamanna, the Microsoft executive vice president who oversees Microsoft 365 and the platform behind Copilot, predicted most business software will end up being used by AI assistants on a person’s behalf, leaving software makers with less power to set prices.
Jean-Denis Greze, CEO of Town, a startup that makes an AI assistant for work, went further. Since about July, he said, AI has been able to operate a browser or a computer almost as well as a person, at a reasonable cost. That means an AI assistant can work through an app’s user interface the way a person does, without needing an API from the software maker, he said.
“Everything is going to become a thin app because the better the AI gets at using the computer, the less the app matters as a unit of software,” Greze said, echoing the term Lamanna had used that morning for software that people open only briefly.
The same shift is under way in retail. Maia Josebachvili, Stripe’s chief revenue officer for AI, said commerce by AI agents on Stripe was roughly flat for eight or nine months and then rose sharply in the past six weeks.
Many merchants make their money by getting shoppers to add to their orders at checkout and by showing them ads, she said, and they lose those chances when an agent does the buying. That won’t work over the long term, Josebachvili said.
That conflict is playing out in real time in a high-profile dispute between two tech giants. Amazon blocked Meta’s Muse assistant from shopping on its site last month.
The data side of the question came up in a session with Anthropic Chief Technology Officer Rahul Patil. Moderator Raphaëlle d’Ornano asked who owns the record of an AI agent’s work, including its mistakes and corrections, and whether it belongs to the customer. She said she has never gotten a clear answer.
Patil didn’t answer directly. He said each company that supplies the software for running agents will work to improve them, and “will use every data that’s available to them to make it better.”
“It’s good for the ecosystem if the agents actually improve,” he said.
Swami Sivasubramanian, Amazon Web Services’ vice president of Agentic AI and Emerging Technologies, said one of the least appreciated parts of making AI agents work is getting them the right company data for the task.
AWS in June announced a service for that purpose, called AWS Context, which maps the relationships in a company’s data for agents to use. Sivasubramanian said it’s meant to work across platforms and cloud providers, and he expects this kind of service to be a basic building block for the next 20 years, as cloud storage and databases were two decades ago.
Most companies aren’t keeping up with the technology: Patil said Anthropic writes about 200 times as much code as it did 18 months ago. Some customers have doubled or tripled their output, he said, but almost none have seen gains like Anthropic’s.
“The bottleneck is actually not AI. The bottleneck is human,” said Archana Vemulapalli, global head of AI product management at Goldman Sachs. A company’s roles and processes were built before AI, she said.
Sivasubramanian said teams inside Amazon were showing him impressive AI agents they had built in two or three weeks. When he asked when the agents could be rolled out, he said, almost every team told him it still had to work out security, identity and monitoring.
McKinsey senior partner Lari Hamalainen said about 40% of companies say AI has increased their profits, but only 6% describe the increase as substantial.
Even as AI becomes more and more capable, there’s still lots of work to do to close the gap between “the goals, the capabilities, and what we all are aspiring to do,” McIlwain said.
Companies are split on depending on a single AI provider: Anthropic’s Patil, notably, argued that large companies spend too much effort preserving the ability to switch between AI providers. They end up building for what all the AI models have in common, he said. They pull people away from work that only their company can do, and they miss the gains when a more capable model arrives.
Carlos Guestrin, co-CEO of the AI startup Noeri, offered an alternative take on the same panel. Intelligence shouldn’t be controlled by one or two companies that own the models, he said, and every company should be able to build and own its own AI systems.
Guestrin is a Stanford computer science professor and former University of Washington professor who founded the Seattle machine learning startup Turi, which Apple acquired in 2016.
Eno Reyes, co-founder and chief technology officer of Factory, a startup that makes AI coding tools, said on a later panel that many businesses don’t see a way to build their future without ceding control to one AI lab. Thomas Dohmke, the former GitHub CEO who now leads the startup Entire, said developers always want choice.
Amazon works with more than one AI lab. It is a major Anthropic investor and partner, and Dan Grossman, Amazon’s vice president of business and corporate development, pointed out on an investor panel that Amazon had launched a managed agents product with OpenAI that week.
Amazon also sells its own product for running AI agents, AgentCore, which overlaps with Anthropic’s Managed Agents.
Companies don’t yet trust AI with important decisions: Zico Kolter, the Carnegie Mellon professor who chairs the safety and security committee of OpenAI’s board, said the ability to control AI systems has to keep pace with their capabilities, which might mean developing them more slowly than is possible.
Sivasubramanian said executives worry about whether an AI agent will always follow their company’s rules and regulations. Companies can keep the creativity of AI models, he said, by pairing them with separate systems that check a model’s work against the company’s rules and catch violations.
The money is concentrated in a few companies: The companies on this year’s IA40 list have raised $410 billion since they were founded, according to Madrona’s data. OpenAI, Anthropic and Databricks account for 92% of it.
Anthropic raised $143 billion of its $161 billion total in the 12 months ending Aug. 15, a figure that includes debt. The company has filed confidentially for an initial public offering and is seeking a valuation of about $2 trillion, Reuters reported this week. Bloomberg reported that it could go public as early as mid-November.
Capital spending by Amazon, Alphabet, Microsoft, Meta and Apple is projected to rise 74% this year to about $742 billion, according to PitchBook figures Madrona presented.
Lamanna said Microsoft’s heaviest AI users among its software developers are each on track to spend more than $1 million a year on AI usage, even with the company’s internal discount.
Madrona’s closing slide said returns on AI are still early for most of the companies using it but look inevitable based on what the earliest adopters are seeing.
