When AI Says "I Created a Ticket," Is It True? Honest Confirmations in Shopify Chat

Tool-using AI chat tools have a specific failure mode that most coverage glosses over. The model calls a tool. The tool returns. Whatever the tool says, the model often paraphrases the user-facing reply with phrases like "I've created a ticket for you" or "You're subscribed to back-in-stock alerts" — even when the underlying API call to the merchant's CRM silently failed and no ticket actually got created.

That kind of false confirmation is the trust break that ages a chat tool fast. The customer waits for the email that never comes; the merchant gets a support call asking why; the widget's reputation as "the AI thing that lies" sticks. v2.4.5 closes this loop with three concrete pieces: intent-aware dedupe, per-platform dispatch tracking, and an AI tool wrapper that lets the model see what actually happened.

The hallucination, specifically

When the AI calls capture_email on a wholesale inquiry routed to HubSpot Tickets, the lead-capture endpoint on our side does several things in sequence:

1. Persist the lead in our database (always succeeds — internal write)
2. Dispatch to whichever upstream adapters the merchant has configured (HubSpot Contact, HubSpot Ticket, Klaviyo, Zendesk, Maestra, custom webhook — possibly several at once)
3. Return a result to the model

Step 2 has multiple potential outcomes per platform: success, validation error (bad list ID, malformed payload), auth error (revoked token), upstream timeout, network failure. Any of these can fail independently — HubSpot Contact succeeds while HubSpot Ticket fails, or vice versa.

If the model only sees "email captured successfully" as the tool result, it doesn't know which dispatches succeeded and which failed. It defaults to confident-sounding confirmations: "I've created a HubSpot ticket for you and the team will be in touch." If the ticket dispatch silently failed, that confirmation is a lie.

Per-platform dispatch results, persisted

The first piece of the fix: every lead row now carries a dispatch_results JSONB column, populated by the capture endpoint:

[
  { "platform": "hubspot_contact", "success": true },
  { "platform": "hubspot_ticket",  "success": false,
    "error": "Invalid pipelineId" }
]

Same shape for every adapter — Maestra, Klaviyo, Zendesk, custom webhook. Two outcomes (success/failure), an optional error string for failures, optional details. The dispatch results live on the lead record, so the merchant can see them in the dashboard alongside everything else about the capture.

The AI tool wrapper grounds the model

The wrapper that hands the capture endpoint's response back to the model now appends a structured note for the model's own reasoning, not for the customer:

Email captured successfully.

[Dispatch outcomes: ✓ hubspot_contact, ✗ hubspot_ticket
 (Invalid pipelineId). Use this to craft an honest customer
 reply — don't claim a ticket/subscription/list signup was
 completed if its dispatch shows ✗. If a ticket dispatch
 failed, fall back to "I've passed your details to the team
 for follow-up" rather than "I've created a ticket".]

The bracketed block is a hint for the model's reasoning, not customer copy. The system prompt instructs the model to read these outcomes and shape its next reply accordingly: confirm only what actually succeeded, hedge on what failed, and route to the merchant for the parts that didn't go through cleanly.

This is a small change but it shifts the model's default from "assume success" to "ground confirmations in evidence." The model still gets to write in its own voice — it just stops claiming things that aren't true.

Intent-aware dedupe windows

The other piece of the v2.4.5 work was related but separate: the lead-capture dedupe logic. Previously, captures were deduped by email within a single global window. Two practical problems with that:

• A customer asking about coupons and getting a code, then coming back ten minutes later with a wholesale question, would have their wholesale capture suppressed as a duplicate — different intent, same email, completely different signal.
• Conversely, a too-tight window meant promotional captures could fire twice for the same coupon ask, issuing two discount codes for one customer (real money lost on real merchants).

The fix scopes the dedupe by intent and uses different windows for different categories:

• Promotional: 24-hour window. The customer who asks for a coupon and gets one isn't served by getting a second code two hours later — and the merchant definitely doesn't want to issue duplicates.
• Business / Support / General: 5-minute window. Only enough to suppress accidental double-submits (customer hitting Send twice). After 5 minutes, the same email asking the same intent is treated as a fresh capture.

Different intents from the same email never suppress each other regardless of window. A wholesale buyer who already signed up for the newsletter still gets their wholesale inquiry through to sales.

✓ / ✗ badges on the lead detail surface

The merchant-facing surface for all this is the lead detail slideout. Each integration that fired for the lead renders with a green checkmark (success) or a red X (failure), with the failure reason visible on hover (in the title attribute).

For the merchant, this is a triage tool. They open a lead, see "HubSpot Contact ✓, HubSpot Ticket ✗ (Invalid pipelineId)," and immediately know what to fix. The ticket pipeline ID in their HubSpot config is wrong; they update it and future captures dispatch cleanly. Without the per-platform breakdown, they'd see the lead in our dashboard and have no way to know it never made it to their ticketing system.

Together, the dispatch tracking, the tool-wrapper grounding, and the dashboard badges make the same data visible to three audiences who need it: the AI (to write honest replies), the customer (to receive truthful confirmations), the merchant (to fix broken integrations). Same payload, three readers.

Frequently Asked Questions