Analysis

Visa released its AI shopping benchmark on Nov. 5, giving merchants a new way to examine how an assistant identifies a customer’s needs before recommending a product. The business question is narrower than whether an agent can complete a trustworthy transaction.

The company reported a 57.2% pass rate across tests involving 22 models from 12 labs and 15 shopping scenarios. That is a result from a controlled experiment, not a failure rate for real purchases. Visa says checkout, payment and order placement are planned extensions, not capabilities assessed in this first round.

Across models and scenarios, Visa said all three attempts passed in 34.2% of cases. That is another measure within this experiment, not evidence of a particular retailer’s customer outcomes or payment performance.

Selection is a defined test

The published methodology describes two AI models: a simulated shopper and the assistant being evaluated. The assistant asks questions and searches a simulated market. Some requirements begin undisclosed and emerge only through relevant questions or actions.

A passing selection must satisfy every answer-key requirement, with those requirements established during the exchange. A decision against purchasing passes only when nothing fits. Merely landing on an appropriate item without discovering the requirements is insufficient.

Each model attempted each scenario three times. Fourteen episodes remain private; one television-shopping episode is public. The reported confidence intervals overlap across leading models, limiting a claim that one supplier clearly wins.

Permission is different evidence

A separate FIS survey released Nov. 5 addresses what people say they would permit. Half of respondents wanted to approve every transaction made by an AI assistant. When asked for their three most important trust conditions, 56% selected access to a person able to review or override decisions.

That research used an online sample of 2,000 U.S. adults supplied by Kantar and surveyed Sept. 16–30. FIS says the data are unweighted and self-reported. These are expressed preferences, not observed shopping behavior or evidence about the participants in Visa’s simulated tests.

The operational implication is a distinct customer-experience question: where does an assistant stop for permission, and how can the customer reach a person? A product recommendation that fits someone’s requirements does not answer whether that person has authorized a transaction. The survey provides context for those design decisions, not a tested implementation or a promised commercial result.

Authentication answers another question

Visa’s Trusted Agent Protocol documentation describes a different control. A merchant reconstructs signed request information and verifies the signature using the agent’s public key. The documentation concerns recognition of agents and authentication of their requests, with additional signatures addressing consumer recognition and payment.

That distinction matters when evaluating a supplier demonstration. Verifying the source and integrity of a request does not establish that the assistant discovered a shopper’s unstated requirements or chose a suitable product. Conversely, a good selection score does not show that a merchant has authenticated the request.

For executives, the useful interpretation is to ask for separate evidence rather than treat one successful demonstration as validation of the whole journey. A supplier can explain its request-verification process; a shopping evaluation can expose how it handles incomplete customer information. Neither account, on its own, establishes a completed payment experience. This is an assessment framework inferred from the different scopes, not a security guarantee or a newly imposed requirement.

Compare research, not percentages

Visa’s September Trust Index announcement adds historical context. It said 72% of respondents had used an AI assistant, while 23% trusted generative AI to handle payments for them. The Harris Poll fielded that research May 26–28, with a total sample of 2,065 U.S. consumers matched to the general adult population using Census data.

Those figures cannot establish that trust rose or fell relative to FIS’s newer findings. The samples, field dates and questions differ. Use of an assistant, willingness to delegate a payment and desire for human intervention are not interchangeable measures.

Both Visa and FIS sell technology serving financial institutions or payments businesses and have commercial interests in adoption. Their studies are useful supplier evidence, not disinterested certification. Digital Transactions’ independent report corroborates the benchmark announcement. Its account presents company-attributed findings, not a separately conducted performance study. The defensible business takeaway is a set of separate evaluation questions, not a forecast of conversion, revenue or customer losses.