The next hurdle for AI agents: websites that block them

Consumer AI agents are moving beyond answering queries into task completion: Meta’s Muse, Instinct, and ChatGPT’s Dots can book flights, make dinner reservations, and order groceries on a user’s behalf. But the next obstacle is the websites themselves. Amazon recently blocked Muse from its retail site, so the agent could no longer browse or buy from its product catalog, and social media complaints show similar failures elsewhere. Some blocks are deliberate, like Amazon’s; others are collateral damage from conventional anti-bot defenses built to stop spam. For users, the ambiguity is the problem: they can’t tell whether a block was accidental, and they end up frustrated with both the retailer and the agent.

Walmart surfaced the accidental-block case. TechCrunch saw complaints that Muse couldn’t complete a purchase on Walmart‘s site, and NBC reported Walmart ran into the same problem in its own tests. Walmart told TechCrunch the blocks weren’t intentional — the company is a Muse partner, announced at Meta’s Connect developer conference in September. The typical failure mode is a human-verification button that fails when the agent’s session is interrupted, booting the agent out. That points to a mismatch between current human-verification technology and the agent-first web.

Industry players are trying to build a way out. Meta, Walmart, Stripe, Sierra, Genesys, Rocket, NiCE, and Decagon are working on an open standard for how AI agents communicate with businesses, focused specifically on agent-to-agent communication for online commerce. Meta said the goal is to separate good bots acting on behalf of users from bad ones.

Airlines are a major test case, and adoption is uneven. Delta said it has no partnership or integration that enables a third-party AI agent to shop for or book Delta flights on its digital platforms, and that any such move would have to keep “security and the customer experience in mind.” United pointed to its Terms of Use, which bar robots, spiders, or other automatic devices without prior written permission. Brands users have complained about blocking agents include Yelp, eBay, Zillow, Pizza Hut, Adidas, and various airlines. Yelp said it doesn’t permit non-human traffic unless the agent pays for data through its licensing program. eBay said it doesn’t prohibit all third-party shopping agents but restricts unauthorized scraping and model training. Some users reported eBay account suspensions, and one said Google kicked Instinct out while it was cleaning up Gmail.

Cloudflare sits in the middle of this: it offers sites protection from AI agents and changed crawler defaults on September 15 so site owners can block AI training while allowing other bots; existing sites set to block AI bots for security were shifted to block AI agents on pages with ads. Cloudflare said it had no specific data on Muse blocking, pointed to its public Cloudflare Radar hub, which shows rising bot activity but doesn’t separate good bots from bad ones. The company also runs a marketplace where AI bots pay for the data they access and is testing a browser built for AI agents.

Meta’s response has been to lock in brand partnerships for Muse access and maintain a growing list of “connectors” inside the Muse app; many newly announced partners haven’t appeared in the list yet, and a reorganization is likely. As Meta put it: “Turning away a personal agent means turning away the customer behind it.”

The next hurdle for AI agents: getting websites to let them in | TechCrunch

View Original