Visa has released the first results from its new industry benchmark for artificial intelligence shopping, revealing that AI models failed 43% of realistic test runs. The findings, published in a press release on Thursday, November 5, highlight significant gaps in the current capabilities of AI agents designed to handle consumer purchases.
Visa Agentic Commerce Benchmark Results
The Visa Agentic Commerce Benchmark tested 22 AI models from 12 different laboratories. The goal was to measure how well these systems could identify consumer needs, gather sufficient information to justify a purchase, or decide against buying if no product met the requirements. The test scenarios involved realistic shopping challenges, such as purchasing a television, a laptop, or a personal care appliance.
Of the total test runs, 57.2% passed while 42.8% failed. The performance varied widely depending on the specific model used. Among the top-performing models, 82.2% of test runs passed. In contrast, the worst-performing models only passed 28.9% of their runs. The testing also examined the relationship between cost and performance, finding that the five most expensive models passed 67.1% of test runs, while the five least expensive models passed 50.2%.
A critical finding from the benchmark was that 54% of failed runs still resulted in a purchase. However, these purchases did not meet the customer’s identified requirements, or the AI failed to uncover all necessary details. This suggests that even when AI agents attempt to complete a transaction, they often do so inaccurately.
Trust and Agentic Commerce
The benchmark is designed to create a roadmap for industry players to improve agentic shopping experiences. It can also be used by consumers to evaluate the reliability of different AI tools. Rubail Birwadker, global head of growth at Visa, emphasized the importance of reliability in this emerging sector. “AI agents have the potential to power the next era of commerce, but trust is key to making it work,” Birwadker said. “Visa Agentic Commerce Benchmark is yet another way Visa is working to boost standards across the industry.”
The release of these results comes as AI integration in retail continues to expand. A PYMNTS Intelligence report titled “Will the 2026 Shopping Season Go Agentic?” found that nearly 132 million U.S. adults have bought a product with AI assistance. Despite this adoption, consumers still retain final control over the purchase decision. The benchmark aims to ensure that the AI’s role in product discovery and transaction execution becomes more reliable and transparent.
Source: PYMNTS
