Milly Software
Insights›Knowledge base›When an AI Answer Is Wrong, Fix the Source, Not the Reply
Knowledge base··5 min read

When an AI Answer Is Wrong, Fix the Source, Not the Reply

When your chat assistant gets an answer wrong, the reply came from somewhere. How to use Conversation Replay's knowledge base sources panel to find the entry behind it, read what you find, and fix it at the source.

V
Viet Le
co-founder · Milly Software

A shopper asks how long they have to return a sale item, and the assistant says 30 days. Your policy for sale items is 14. The instinct is to correct that reply. But the reply came from somewhere, and if the somewhere is still wrong, the next shopper who asks gets the same answer.

This post is about finding that somewhere. Conversation Replay shows which knowledge base entries came back when the assistant searched, so you can fix the entry instead of the one conversation. The sources panel shipped on August 25, 2026.

Where a policy answer comes from

For questions about shipping, returns, warranties and the like, the assistant searches your knowledge base. The search returns a ranked handful of entries, and the assistant answers from them. An outdated entry, two entries that contradict each other, or a policy page that doesn't mention sale items can each produce a wrong answer, and keep producing it for the next shopper who asks.

The knowledge base sources panel

Open a conversation in Conversation Replay. Above the transcript, the sources panel lists each knowledge base search the assistant ran in that conversation. For each one you see:

  • The query: the words the assistant searched with, which aren't always the shopper's words.
  • The entries that came back, in rank order, each linked to the knowledge base document you'd edit.
  • Or, when no entry came back, "No knowledge base match for this query".

The links go to the document, not the fragment that matched. A long document is stored in parts, and editing one part would leave the others serving the old text. When several parts of the same document matched, the panel shows it once and says how many sections matched.

The query is worth a look on its own. It's the assistant's wording, not the shopper's: "can I send back the jacket I got in the sale" might be searched as something closer to "return policy for sale items". If the query is a fair reading of the question and the results are still wrong, the problem is in your entries. If the query itself missed the point, that's worth knowing too, and it's the kind of thing the feedback button on the reply is for.

Retrieved, not used

One honest note about the wording, because it matters. The panel says "retrieved". These are the entries that came back for the search, and the assistant doesn't always draw on all of them. The top result isn't automatically the source of the answer.

We could have labelled it "the source of this answer" and it would have looked more useful. It also would have sent you off to edit documents that had nothing to do with the reply. So read the panel as a list of suspects, and then read the entries.

Reading what you find

Most wrong answers fall into one of three cases.

  1. A wrong or outdated entry came back. The 30-day figure is sitting in an old returns entry, or in a general policy that doesn't carve out sale items. Fix the text, or delete the entry if it's been replaced.
  2. No entry came back. The panel says there was no match, and the assistant had nothing of yours to go on. That's a gap: write the entry. The query is a good first draft of its title.
  3. The right entry came back, and the reply was still wrong. That isn't a source problem, and editing the entry won't help. Use the feedback button on the reply in Replay to describe what went wrong. It comes to us for review.

The panel also flags three things worth acting on. An entry marked no longer in the knowledge base was renamed or deleted since that conversation. One marked duplicate title shares its exact title with another entry, so the link is a best guess, and the two are worth merging. And a note that the record was truncated means more entries came back than are listed, so a short list isn't necessarily the whole picture.

Under the list of what came back there's a second section, Reported as used: the entries the assistant said it drew on for that reply. Treat it as a hint, not a measurement. It's the assistant's own report, and when the section says nothing was recorded, that isn't evidence the assistant used none of your entries. Don't delete an entry because it's missing from this list.

Why the source

Feedback on a reply is about one conversation. An entry is what later conversations on the subject search against. When you save an edit to it, the entry is re-indexed for search, so the next search about sale returns runs against the entry that says 14 days.

It also means the knowledge base gets better in the places shoppers push on it, rather than wherever you happened to be writing. If you want a list of where to start, knowledge gaps collects the questions the assistant couldn't answer from your documentation. The sources panel shipped in v2.10.1 and is part of Conversation Replay.

Frequently Asked Questions

How do I find out why the AI chat gave a wrong answer?

Open the conversation in Conversation Replay. The knowledge base sources panel lists each search the assistant ran, the query it searched with, and the entries that came back in rank order, each linked to the document you edit. If the knowledge base had no match, the panel says so.

Does the panel show the source the assistant used?

It shows what was retrieved, not what was used. These are the entries that came back for the search, in order, and the assistant doesn't always draw on all of them. The panel is labelled that way on purpose.

Should I edit the knowledge base or report the reply?

If a wrong or outdated entry came back, or no entry came back, fix the knowledge base: later searches on the subject run against your edited entry. If the right entry came back and the reply still got it wrong, that's not a source problem, and the feedback button on the reply sends it to us for review.

Try Milly Chat

Want to see how this fits your store?
We'll set up a working session.

Get it on Shopify →Talk to sales →