Reviews (5,787)
The Shopify Inbox AI agent has real potential and is a genuine step up from the previous Inbox (which was limited to live chat management). But several issues still need fixing before it can be used reliably in production.
1. Response language
The AI has replied in English even though the entire conversation was in French.
2. Inconsistency depending on how the question is phrased
Two questions with the same intent, phrased differently, don't give the same answer: one omits relevant information/product, the other doesn't. This is a real reliability concern if the result depends so heavily on the customer's exact wording.
3. Hallucinated non-existent link
On certain requests, the AI tries to share a link to a page that doesn't exist at all on our site. How can this type of URL hallucination be avoided?
4. Scope of learning
Does the training/learning happen only at the conversation level, or does it also benefit future conversations?
5. Disabling the AI mid-conversation
After disabling the AI agent while a conversation was in progress, it kept being handled by the AI despite the box being unchecked. Is this a bug?
6. Limited training options
For training, with Knowledge Base you can suggest answers, but not suggest new questions in the scenarios. Is this planned? We also don't have control over selecting/correcting which data source is used, which is surprising.
Overall, a great tool at first glance, but one that still needs a lot of foundational work. Looking forward to updates. In the meantime, to get reliable answers, I'm forced to use another paid tool instead of this native, free one.
Thank you for taking the time to share such a detailed and thoughtful review. We reviewed the issues you raised against the actual conversations on your store and have shared the findings with our engineering team who are actively digging into the underlying causes to fix them.
To clarify two points: The training/learning will benefit future conversations and also allow our background agent to suggest improved Agent persona if that will be more helpful for representing your business to buyers accurately. Also, yes we are planning to surface more questions in Knowledge Base based on the conversations buyers have with your agent.
We appreciate the thorough testing and candid feedback. This is exactly the kind of input that helps us make the product more reliable.
