Visa launched the Agentic Commerce Benchmark on Thursday, November 5, 2026, a test that puts AI models through realistic AI shopping scenarios. The result shows the technology still stumbles on the basics: the models failed 42.8% of test runs.
What the test found
The first round evaluated 22 models from 12 labs across 15 shopping challenges, such as choosing a TV, a laptop, or a personal care appliance. To pass the AI shopping test, the AI had to uncover every customer requirement in conversation and make the right call: buy or don't buy. The overall pass rate was 57.2%.
The best models passed 82.2% of attempts, while the worst managed 28.9%. The difference was in the questioning: the five models that uncovered the most hidden buying factors averaged a 70.2% pass rate. The five that uncovered the least passed less than half the time, at 44.4%.
Price mattered too, but it did not guarantee success. The five models with the highest average cost per run passed 67.1% of the time, versus 50.2% for the five cheapest. Consistency was the weakest point: each model got three attempts per scenario, but passed all three only 34.2% of the time.
The most worrying figure involves wrong purchases. In about 1 in 2 failed runs (54%), the AI still completed a purchase, even though it did not meet the customer's identified requirements or had failed to uncover all of them.
Trust is still the bottleneck
“AI agents have the potential to power the next era of commerce, but trust is key to making it work,” said Rubail Birwadker, Global Head of Growth at Visa.
One scenario makes the problem clear. The customer asks for help choosing a TV, mentions that a relative recommended a brand and that they hold a membership with a third-party supplier, but leaves other details unsaid. The AI must uncover the budget and requirements such as format compatibility, HDR10+ support, and gaming capabilities. Models that asked better questions passed more often. But when asked to repeat a scenario they had already passed, the same models often failed.
The study connects to another Visa survey. The Trust Index, released in September 2026, found that 72% of US consumers had used an AI assistant, but only 23% trusted AI to handle payment transactions on their behalf.
Visa stresses that the result does not mean agents aren't ready for commerce. The highest-scoring models passed 82.2% of attempts, and the benchmark is public: anyone can take the test, watch replays of AI attempts, and read the full study on Visa Payments Frontier.
Image credit: a person using a bank card to shop online, photo by Bogdan Hoyaux / European Commission, via Wikimedia Commons, CC BY 4.0 license (© European Union, 2026).
Sources: Visa, BusinessWire, PYMNTS. Read more tech news on the Geekiki blog.
Transparency: this article was produced with the assistance of artificial intelligence, based on public sources on the internet and public posts on X. Please consult the original sources for full details.
Muse
