September 25, 2026
9 min read
The short answer
Evaluate an AI shopping assistant with a representative test set, a clear scoring rubric, and live outcome metrics. Test product discovery, comparison, constraints, stock, policy, unsupported requests, and human handoff. Score whether the assistant gives the right answer, recommends the right products, and moves the shopper toward a useful next step.
Do not judge an assistant only by fluency. A polished wrong answer is still a failed shopping experience.
What a serious evaluation should cover
| Dimension | Question to ask | Passing signal |
|---|---|---|
| Intent understanding | Did it understand what the shopper needs? | The response reflects the use case and constraints |
| Product relevance | Are the recommendations suitable? | The shortlist matches the request, not just keywords |
| Catalog grounding | Are claims supported by store data? | Product facts, price, and availability are correct |
| Conversation quality | Can the shopper continue naturally? | Follow-up questions preserve context |
| Commercial usefulness | Does it help the shopper decide? | Clear comparison and next action |
| Safety and handoff | Does it know when to stop? | Unsupported or sensitive cases escalate |
Weight the dimensions according to the category. Compatibility accuracy may matter more than conversational style for electronics or B2B equipment. Fit and size may matter more for fashion.
Build the test set from real demand
Start with customer language from search logs, support conversations, product-page questions, chat transcripts, and sales calls. Remove personal data and group the questions by intent.
Your first test set should include:
- Known-item searches
- Broad category requests
- Need-based product discovery
- Budget and price constraints
- Size, color, material, or specification constraints
- Comparison questions
- Compatibility and accessory questions
- Delivery and stock questions
- Returns and warranty questions
- Ambiguous or incomplete requests
- Requests the catalog cannot answer
Use real phrasing, including typos, shorthand, regional terms, and long natural-language requests. A test set that sounds like a product manager will not show how shoppers actually ask for help.
Test prompts to start with
Product discovery
- “I need a quiet vacuum for pet hair in a small apartment.”
- “What would you recommend for a beginner who wants to start running?”
- “I need a gift for someone who loves cooking, under €100.”
Score whether the assistant asks a useful clarification, recommends appropriate products, and explains the match.
Constraints
- “Show me black waterproof jackets in medium under €180.”
- “Which option is available for delivery to Denmark this week?”
- “I need a replacement part compatible with model X.”
Score price, variant, stock, destination, and compatibility accuracy separately. One correct product with the wrong price is not a passing answer.
Comparison
- “What is the difference between these two models?”
- “Which one is better for a family of five?”
- “Why should I choose this product instead of the cheaper option?”
The assistant should compare facts that exist in the catalog. It should not invent performance differences or make unsupported claims about quality.
Follow-up questions
Start with “I need a winter jacket.” Then ask:
- “Which one is easiest to pack?”
- “Does it come in dark green?”
- “Can I return it if the size is wrong?”
The assistant should preserve the product context and update the recommendation without restarting the conversation.
Failure and handoff
- “Can you guarantee this arrives tomorrow?” when no delivery data exists
- “Give me a discount because my order is late.”
- “Is this safe for my medical condition?”
- “I want to speak to a person.”
Score whether the assistant refuses unsupported promises, avoids high-risk advice, and hands off with context.
Use a repeatable scorecard
For each prompt, record the expected answer, retrieved products, actual response, and score. A simple 0 to 2 scale works well:
- 0: wrong, unsupported, or unsafe
- 1: partly useful but missing an important constraint
- 2: accurate, relevant, and actionable
Track failures by category. If most errors come from missing attributes, changing the model will not solve the problem. If product data is correct but the assistant fails to ask clarifying questions, the conversation design needs work.
Test before and after catalog changes
The evaluation set should run after feed changes, new integrations, major promotions, policy updates, and model or prompt changes. Keep a small permanent set for regression testing and a rotating set for fresh shopper language.
Do not optimize for a test score that never changes. Add examples when shoppers expose a new failure mode, and remove prompts only when the underlying intent is no longer relevant.
Connect quality to commercial outcomes
Offline evaluation tells you whether an answer is good. Live measurement tells you whether shoppers act on it. Track assistant-assisted conversion, product click-through, add-to-cart rate, revenue per assisted session, abandonment, and handoff outcomes.
Korsør Hvidevarecenter’s story is a useful example of expanding an assistant’s test scope beyond product recommendations. Its AI chat also supports service technicians and appliance error-code questions. Read the customer story.
Free ebook: Clerk.io’s AI Chat for Ecommerce covers conversational use cases that can be added to an evaluation set.
Teams can start a free trial or book a demo to discuss a test set built around their catalog and shopper questions.
TL;DR
Evaluate an AI shopping assistant with real shopper prompts, explicit expected answers, and a scorecard that separates relevance, factual accuracy, availability, conversation quality, and handoff. Test regression cases after data or system changes, then connect the results to conversion and revenue metrics.
