Your AI's Confidence Score Won't Catch Products You Don't Carry

Direct Source Verification: This story is aggregated from Forbes (forbes.com). Full reporting rights and copyright belong to the primary publisher.
Retailers increasingly rely on AI to recommend products, but what happens when those recommendations include items they don't actually sell?

Srijith Ravikumar is a Principal Engineer at Amazon building AI search and recommendation systems. Published at ACM CIKM and IP&M.

gettyIn 2024, a Canadian tribunal ordered Air Canada to compensate a customer who had relied on its website chatbot’s advice about bereavement fares, and it rejected the airline’s argument that the chatbot was responsible for its own actions. Retailers face a version of that exposure: AI recommendations for products the store doesn’t carry. A tempting way to decide when to trust those suggestions is a score the model reports about itself.

Ask a recommendation model how confident it is, set a threshold and drop everything below it. That guardrail is easy to build and easy to report upward. In my tests, it barely worked. Across 48 test settings, filtering on stated confidence cut the share of out-of-catalog suggestions by a median of 0.02 percentage points, and in 18 of them, the filtered list came out slightly worse.

My study measured how often suggestions miss the catalog, not what a miss costs. The likely costs are easy to name. A customer taps a suggestion, finds nothing to buy and trusts the feature a little less. A slot that could have sold something goes to an item that can’t. An assistant that names products the store lacks invites complaints. Treat those as hypotheses to check against your own numbers.

This year I tested four widely used large language models from four different developers. Each suggested 10 items from a user’s history, working from memory with no access to the catalog, on three public datasets covering movies, local businesses and retail toys. I checked every suggestion against the catalog it was meant to come from, using a matching method validated against about 200 human judgments, and counted anything not in that catalog as a miss, whether or not it exists elsewhere.

The out-of-catalog rate depended heavily on the catalog. Across the four models and three catalogs, it ran from 0.6% to 61%. Every model stayed under 3% on movies and reached roughly half or more on toys, with identical instructions. A rate measured in one domain tells you little about another, and a vendor demo is one domain.

Confidence barely moved. Each model’s stated confidence on toys stayed within about four points of its confidence on movies, while its out-of-catalog rate rose twentyfold or more. Researchers have already shown that models stating their own confidence tend to be overconfident, a bias you could at least correct for. Here the bias wasn’t stable. Across the 12 model-and-catalog combinations, the models were underconfident in seven and overconfident in five: All four were underconfident on movies, and all four were overconfident on toys. A correction tuned on one catalog would point the wrong way on the next.

One explanation fits the data, though the data don’t prove it. Asked how confident it is, a model seems to rate how good it thinks the suggestion is. It can’t check whether the item sits in your catalog from memory.

The filter I used is split conformal abstention, which uses conformal prediction, a standard statistical technique, to drop the least-confident suggestions while guaranteeing that a set share of the good ones survive. Its best result across the 48 settings was a cut of under two percentage points. The method largely kept the good items, as designed. Its guarantee says nothing about removing bad ones.

I tested only the confidence a model states in words. Token probabilities, agreement across repeated samples and constrained generation are different signals I didn’t test, and they may do better. If your model only picks among candidates retrieved from your catalog, the failure changes shape: The risk becomes a poor pick among real items, and this study doesn’t speak to that.

Within that scope, these are the questions I’d bring to a product review, an AI governance meeting or a decision to fund the next phase.

Is every suggestion checked against the live catalog before a customer sees it? It removes items you can’t sell, though it won’t fix relevance.

How was the out-of-catalog number measured? If items are matched by title, the matching rule can move the answer more than the model does. On identical model outputs, plausible off-the-shelf matching rules disagreed on the toy catalog’s out-of-catalog rate by more than a factor of 10. Ask for a check against a couple of hundred hand-labeled cases, with the remaining error reported next to the number. Having the model return product IDs avoids the problem.

Is a promising fix being tested as an experiment? On the business catalog only, I told two models to recommend from a fixed catalog that I didn’t show them and asked how sure they were that each title was in it. Out-of-catalog suggestions fell from 12.9% to 1.5% for one model and from 14.4% to 8.9% for the other. Each cut is at least three times the filter’s best result. The paper reports the figures and compares the first cut directly against that result. That prompt changed the instruction and the confidence question at once, and it covers two models in one catalog. Before a rollout, ask for a test that changes one prompt element at a time, runs on a held-out sample of the live catalog and reports both the out-of-catalog rate and whether the suggestions stayed relevant.

Was it measured on your own catalog, including the long tail of products customers rarely look for? A benchmark from someone else’s data, or a pilot on your bestsellers, says little about those.

Keep the confidence score for sorting results or deciding where a person looks first. The governance question is a different one: What share of our AI suggestions aren’t in our live catalog, measured across the long tail? The confidence score can’t answer it.​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?

Original Source
https://www.forbes.com/councils/forbestechcouncil/2026/11/18/your-ais-confidence-score-wont-catch-products-you-dont-carry/
Visit Forbes ↗
SHARE STORY:
𝕏 f in

Related Coverage in Technology