How accurate is "just ask the model to categorize it"? We measured — 1–2 of 12 checks.

If you’re mapping a product catalog into Google’s Product Taxonomy, the obvious first move is to hand the title and description to a capable model and ask for the category. It reads well in a demo. So the only honest question is: how often is it actually right? We measured it, on our own approach, and the zero-shot number is worth knowing before you ship it.

The zero-shot number: 1–2 of 12 checks

The test set at that point was six hand-labelled products, each checked against two taxonomies — Google and Shopify. 12 checks in total. Every expected path and leaf ID in it was verified by direct lookup against the real taxonomy files, not guessed, and a check counted as a match only if the full path and the leaf ID matched the label.

Against that set, a straight zero-shot prompt — “classify this product into the Google taxonomy, return the path and leaf ID” — scored 1–2 exact matches out of 12, stable across three live runs on the same day (2 of 12, then 1 of 12, then 1 of 12). Worth knowing how those 12 break down: the zero-shot run returned a null Shopify leaf ID on every product — honest, but still a miss under exact-match scoring — so the one or two matches it did land were on the Google side.

Not because the model is dumb: the Google taxonomy has ~5,600 leaf categories, many of them near-synonyms (“Drinkware > Mugs” vs “Drinkware > Coffee & Tea Cups” vs “Kitchen > Travel Mugs”), and a model recalling a leaf ID from memory is guessing at a 5,600-way choice with no way to check itself. It doesn’t know it’s wrong, which is the dangerous part.

Worse, it fails confidently. Ask for a leaf ID and a model will happily invent one that looks structurally correct — right number of digits, plausible parent — but doesn’t exist in the taxonomy at all. A category ID that doesn’t resolve is arguably worse than no ID, because it passes a shape check and fails silently downstream.

What actually moved the number

The fix wasn’t a cleverer prompt. It was retrieval-grounding: embed the product, retrieve the real candidate paths and leaf IDs from the actual current taxonomy file, and hand those to the model as the menu to choose from — plus a verification step that rejects any leaf ID not present in the real taxonomy, capping confidence when it can’t confirm one.

By the time we re-ran the eval, the test set itself had grown: 12 hand-labelled products, still checked against both Google and Shopify — 24 checks. The six added products deliberately covered categories the original six didn’t touch (kitchen tools, toys, grocery, beauty, sporting goods, pet supplies), each verified leaf-by-leaf the same way. So the two numbers in this post come from two different sets: the before is 12 checks over 6 products, the after is 24 checks over 12. Same scoring rule, same two taxonomies, wider set.

Corrected 2026-07-28: this post originally described the grounded run as scoring 22 of 24 “on the same test set” as the zero-shot baseline. That was wrong — the evaluation set was expanded from 6 products to 12 between the two runs, so the fractions don’t share a denominator basis. The runs, the numbers and the direction of the result are unchanged; the framing above now states both set sizes. Sources at the bottom of the post.

Run against the live retrieval index and a live model API, that expanded set scored 22 of 24 checks (91.7%) exact path + leaf-ID matches — 12 products, each checked against both Google and Shopify — with correct-prediction confidence averaging 0.994. The two remaining misses are both honest: one returns the right path but withholds the leaf ID rather than guess; the other is a genuine wrong-path pick (a mug filed under “Travel Mugs”). Neither is a fabrication — the failure mode of confidently inventing an ID is gone, by construction, because an unverifiable ID never survives the check.

The point isn’t the number — it’s what the number is measuring

Any classification service can quote an accuracy figure. What’s worth asking is exact match against a verified leaf ID, or recall that the right answer was somewhere in a candidate list? They’re very different claims. The 22-of-24 figure above is the strict one: the single returned answer matched the independently-labelled leaf ID. And it’s grounded in the real taxonomy, not the model’s memory of it — so when it’s uncertain, it says null instead of handing you a plausible fabrication.

The same goes for the denominator. Because the two runs above used different sets, the honest reading of the pair is a direction, not a delta — which is why both set sizes are stated rather than the fractions being put side by side.

If you’re building this yourself, the takeaway is cheap to apply: don’t trust zero-shot leaf IDs, retrieve real candidates first, and verify every ID against the actual taxonomy before you store it. That’s what separates 1–2 of 12 checks from 22 of 24.

How this was measured

• The test set: example-products-expected-categories.json — the hand-labelled evaluation set, holding the expected Google and Shopify path + leaf ID for every product. Its _provenance record is the source for the set sizes quoted above: the original 6 products were verified against the real taxonomy files on 2026-07-16, and the set was expanded to 12 on 2026-07-19.
• The scoring: the eval harness — one check per product per taxonomy, a pass only on an exact path and leaf-ID match. Run against a live model API on 2026-07-17 (zero-shot, three runs) and re-run on 2026-07-21 with retrieval grounding active.
• On the deployed service: a later spot-check pushed the same 12 products through the live service itself and scored the Google side — 11 of 12 exact path + leaf ID, one check per product, the same 91.7% rate, confirming the grounding is active in production and not just in the harness. (Shopify was not re-measured this way.)
• Published result: the 22-of-24 figure and the retrieval-grounding change ship in the service’s public release notes (0.2.0). The golden set and the eval harness are internal — the service implementation isn’t open source — so they’re named here rather than linked.

Comparing categorization tools? Here are five questions that separate grounded from guessing.

Try the Catalog Normalizer →