Configuring AI With AI: Natural-Language Configuration Agents on Shopify

Most AI chat widgets ship with a settings page that assumes the merchant knows how to configure AI. Write good system prompts. Tune retrieval. Set conditional rules. Compose the product-feed. The merchant is now expected to learn enough about AI configuration to be productive — which is a strange tax to levy on someone whose job is selling motorcycle helmets.

The bet behind NL Configuration Agents is that AI is the instrument best suited to teaching merchants how to configure AI. Not as a chatbot. As a structured collaborator embedded in the configuration surface itself, emitting one-click apply actions instead of long explanations.

The configuration paradox

AI tools have a built-in onboarding cliff. You can't use them well without an intuition for how the underlying model behaves. You can't build that intuition without using the tool. The merchant who knows the perfect set of AI instructions is also the merchant who didn't need them written down.

The pre-AI workaround is a wizard or a tooltip — pre-baked choices the merchant clicks through. That works until the merchant has a specific store-shaped need that the wizard didn't anticipate ("I sell custom paint, but only the unmixed base counts as the 'product,' the colors are a metafield"). At that point the wizard goes quiet and the merchant is back to reading docs.

The NL agent flips this. The merchant describes their situation in plain English — "I want the widget to greet wholesale buyers differently" — and the agent translates it into actual configuration: an AI Instruction that targets wholesale intent, paired with an applicable conditional rule. The merchant clicks Apply, the config saves, the widget behaves differently the next time a wholesale buyer arrives.

Seven dashboard surfaces, seven specialized agents

Each agent is a specialized system prompt + a route-specific understanding of what configuration shape to emit. v1.9.0 launched five (Mar 31, 2026); v2.0.3 added Corrections + Docs (Apr 1, 2026):

• Product Data — explains why a search returned what it did, suggests Shopify metafield keys to add (e.g. custom.range, stamped.reviews_average), and offers to trigger product sync + embedding regeneration as one-click actions.
• Response — drafts AI Instructions and Quick Questions in plain language. Knows the difference between a suggestion-type quick question and a quiz-trigger one.
• Lead Capture — explains intent classification (promotional / business / support / general), drafts intent-targeted instructions, suggests when to enable coupons vs route to CRM only.
• Rules — translates "hide on checkout pages," "show a different greeting on mobile after 30 seconds," etc. into the actual conditions[] array shape with the right condition types.
• Installation — guides the merchant through Theme App Extension setup, custom-distro auth, headless callback wiring. Knows which install path applies based on store metadata.
• Corrections (v2.0.3) — opens a split view alongside captured corrections; the agent helps diagnose why a correction was needed and suggests upstream fixes (KB additions, AI Instruction tweaks) so the same mistake doesn't recur.
• Docs (v2.0.3) — slide-out panel that surfaces relevant documentation contextual to the merchant's current question. Doesn't require leaving the configuration surface to search docs.

All seven render via a shared AgentChat component. Same conversation UI, same streaming, same suggestion-card rendering — the difference is the system prompt and the per-route render logic for suggestions.

Structured outputs as the contract

The interesting part isn't the chat — it's the write-back. When the agent suggests something concrete, it wraps the suggestion in a fenced code block with a typed label. The frontend parses the fence and renders an apply card:

Drafting AI instructions for wholesale buyers...

```ai_instructions
When the visitor asks about wholesale, B2B, or bulk
pricing, ask for their company name and order volume.
Pass the details to the body field of capture_email so
sales can prioritize the response.
```

```quick_questions
[
  {"text":"Wholesale inquiry","guidance":"Ask for company name and volume"}
]
```

The frontend parses each fence (ai_instructions, quick_questions, metafield_keys, action, etc.) and renders an apply card with a diff preview. One click writes the change to the merchant's config; no separate "copy this and paste it into Settings" step.

Because the contract is the fence, not the conversational wording, the agent is free to explain in any tone, hedge, clarify, ask follow-up questions — none of that affects the write-back. The merchant sees a chatty assistant; the system sees a typed payload. Two audiences, one stream.

Actions the agent can trigger directly

Some configuration changes don't fit a static fence — the merchant might need to do something, not just save something. For those, the agent emits an action fence with a labeled trigger:

Once you've added the metafield keys, the embeddings need
to be regenerated to pick them up.

```action
{"action": "regenerate_embeddings", "label": "Regenerate Embeddings"}
```

The frontend renders the action fence as a button. Click, backend job kicks off, the agent stays open and reports back when it completes. The merchant never opens a separate admin page to do the thing the agent just told them to do — the thing happens where the conversation happens.

Why this isn't "agentic AI"

The word agent is doing dual duty in 2026 — it can mean "an LLM running a multi-step task autonomously" (browsing the web, executing code, deciding next actions on its own), and it can mean "an LLM-backed assistant embedded in a UI surface." The configuration agents are firmly the second.

Every action that mutates merchant config goes through the merchant's click. The agent suggests, the merchant approves, the change applies. There is no "the agent decided to add this rule on its own." Even the action-fence buttons are buttons — the merchant has to press them.

That's a deliberate trust boundary. Configuration changes affect the visitor experience. A bad-faith or hallucinated suggestion that auto-applied would mean the visitor sees a broken widget before the merchant notices. The merchant-in-the-loop pattern keeps that surface stable, even when the underlying model gets things wrong.

The win comes from the agent writing the configuration in the right language, not from the agent applying it autonomously. The merchant still owns the decision.