How accurate is "just ask the model to categorize it"? We measured — 1–2 out of 12.
- ACJ Labs
- Engineering
- 25 Jul, 2026
If you’re mapping a product catalog into Google’s Product Taxonomy, the obvious first move is to hand the title and description to a capable model and ask for the category. It reads well in a demo. So the only honest question is: how often is it actually right? We measured it, on our own approach, and the zero-shot number is worth knowing before you ship it.
The zero-shot number: 1–2 out of 12
On a hand-labelled test set, a straight zero-shot prompt — “classify this product into the Google taxonomy, return the path and leaf ID” — scored 1–2 exact matches out of 12, stable across repeated runs. Not because the model is dumb: the Google taxonomy has ~5,600 leaf categories, many of them near-synonyms (“Drinkware > Mugs” vs “Drinkware > Coffee & Tea Cups” vs “Kitchen > Travel Mugs”), and a model recalling a leaf ID from memory is guessing at a 5,600-way choice with no way to check itself. It doesn’t know it’s wrong, which is the dangerous part.
Worse, it fails confidently. Ask for a leaf ID and a model will happily invent one that looks structurally correct — right number of digits, plausible parent — but doesn’t exist in the taxonomy at all. A category ID that doesn’t resolve is arguably worse than no ID, because it passes a shape check and fails silently downstream.
What actually moved the number
The fix wasn’t a cleverer prompt. It was retrieval-grounding: embed the product, retrieve the real candidate paths and leaf IDs from the actual current taxonomy file, and hand those to the model as the menu to choose from — plus a verification step that rejects any leaf ID not present in the real taxonomy, capping confidence when it can’t confirm one.
Re-run against the live index and a live model API, the same test set scored 22/24 (91.7%) exact path + leaf-ID matches, correct-prediction confidence averaging 0.994. The two remaining misses are both honest: one returns the right path but withholds the leaf ID rather than guess; the other is a genuine wrong-path pick (a mug filed under “Travel Mugs”). Neither is a fabrication — the failure mode of confidently inventing an ID is gone, by construction, because an unverifiable ID never survives the check.
The point isn’t the number — it’s what the number is measuring
Any classification service can quote an accuracy figure. What’s worth asking is exact match against a
verified leaf ID, or recall that the right answer was somewhere in a candidate list? They’re very
different claims. The 22/24 above is the strict one: the single returned answer matched the
independently-labelled leaf ID. And it’s grounded in the real taxonomy, not the model’s memory of it — so
when it’s uncertain, it says null instead of handing you a plausible fabrication.
If you’re building this yourself, the takeaway is cheap to apply: don’t trust zero-shot leaf IDs, retrieve real candidates first, and verify every ID against the actual taxonomy before you store it. That’s the whole difference between 1–2/12 and 22/24.
