You've built your agent, loaded it with knowledge, and connected your tools. Now comes the part that separates a decent AI Agent from a great one — testing. Chatmo includes a built-in chat simulation feature that lets you have full conversations with your agent before any real customer interacts with it.
Using the Chat Simulation Widget
The chat simulation widget is a dedicated testing environment where you can chat with your AI Agent exactly as a customer would. Everything works the same as in production — knowledge base lookups, tool calls, behavior rules — but no real customers are involved.
The simulation widget is always visible on the right sidebar of the AI Agent setup page on desktop. You do not need to click any special button to open it — it is immediately available whenever you are on the agent configuration page. On mobile devices, the simulation widget is accessible via a modal popup.
How to access it:
- Go to AI Agent in the sidebar.
- Select the agent you want to test.
- The agent setup page opens with the chat simulation widget already visible on the right sidebar.
- Start typing messages in the simulation widget to test your agent.
The simulation widget uses the exact same configuration as your live agent: behavior prompt, knowledge base, tools, and all other settings. What you see in the simulation is a direct reflection of what your customers will experience.
Debug Mode
The chat simulation widget includes a Debug toggle (On/Off) that provides valuable insight into how your AI Agent processes each message.
When Debug is turned ON, you can see additional information within the simulation:
- Tool call logs — Shows which tools were called by the agent during the conversation, including the parameters sent and the responses received.
- RAG Knowledge response — Displays the search results from the knowledge base, so you can see exactly what information the agent found (or did not find) when answering a question.
- Knowledge search status — Indicates whether relevant knowledge was found or not found for a given question.
Debug mode helps you:
- Identify whether your knowledge base adequately covers customer questions
- Verify that tools are being called correctly and returning the right data
- Find gaps in your knowledge base that need to be filled
- Understand the agent's decision-making process in detail
Always enable Debug mode when testing. The debug information is invaluable for understanding why the agent gives a particular response and where improvements need to be made.
How to Test AI Responses
Effective testing isn't just typing a few questions and calling it done. Follow a structured approach to uncover issues before your customers do.
Start with the Basics
Begin with simple, common questions your customers ask every day:
- "What are your business hours?"
- "How much does shipping cost?"
- "Do you accept returns?"
- "What payment methods do you accept?"
These baseline questions verify that your knowledge base is properly connected and the agent can retrieve basic information. If it stumbles here, don't move on to complex scenarios — fix the fundamentals first.
Test Natural Variations
Real customers don't all phrase questions the same way. Test multiple variations of the same question:
- "What's your return policy?"
- "Can I return this?"
- "I want to send something back"
- "How do refunds work?"
- "I changed my mind about my purchase"
All of these should lead to a relevant answer about your return policy. If some phrasings get good answers and others get confused responses, you may need to add more content or rephrase entries in your knowledge base.
Test Conversation Flow
Customers don't ask isolated questions — they have conversations. Test multi-turn exchanges:
You: "I ordered something last week and it hasn't arrived." Agent: [Acknowledges the concern, asks for order details] You: "The order number is 78342." Agent: [Looks up the order, provides status] You: "Can I get a refund instead of waiting?" Agent: [Explains refund process based on the specific order context]
This tests whether the agent maintains context across multiple messages, which is crucial for natural conversations.
Testing Different Scenarios
Run through these common scenarios to cover the most important customer interaction types:
Greeting and Introduction
Send a simple greeting and verify the agent responds with your configured welcome message and tone.
| Test Input | What to Check |
|---|---|
| "Hi" | Does the agent greet warmly and explain what it can help with? |
| "Hello, I need help" | Does it acknowledge the request and ask how it can assist? |
| No text, just an emoji | Does it handle non-text messages gracefully? |
Product or Service Inquiry
Ask specific questions about your products or services. This tests knowledge base retrieval.
| Test Input | What to Check |
|---|---|
| "Tell me about [specific product]" | Does it pull accurate details from the knowledge base? |
| "What's the difference between Plan A and Plan B?" | Can it compare information from multiple knowledge entries? |
| "Do you have [product you don't carry]?" | Does it honestly say it's not available rather than making something up? |
Complaint Handling
Unhappy customers are where AI Agents are put to the real test.
| Test Input | What to Check |
|---|---|
| "This product is terrible, I want my money back" | Does it acknowledge frustration before jumping to solutions? |
| "I've been waiting 2 weeks for my order!" | Does it express empathy and offer to investigate? |
| "Your service is the worst I've ever experienced" | Does it stay calm and professional, not defensive? |
Pay special attention to how your agent handles emotional messages. The behavior prompt should include instructions about empathy and de-escalation. If the agent sounds robotic or dismissive when a customer is upset, rewrite the relevant parts of your behavior prompt.
Human Agent Handoff
Test the scenarios where the AI should step back and involve a human.
| Test Input | What to Check |
|---|---|
| "I want to talk to a real person" | Does it immediately initiate handoff without arguing? |
| "Can I speak with a manager?" | Does it acknowledge the request and route appropriately? |
| A question outside the agent's knowledge scope | Does it recognize its limits and offer to connect with a human? |
Tool Integration
If you've configured tools, test them with realistic inputs.
| Test Input | What to Check |
|---|---|
| "What's the status of order #12345?" | Does the tool get called? Is the response accurate? |
| "How much to ship 3 kg to Surabaya?" | Are parameters extracted correctly from natural language? |
| Provide an invalid order number | Does the agent handle the error gracefully? |
Evaluating Response Quality
Not every response needs to be perfect, but it should meet these quality benchmarks:
Accuracy — Is the information correct? Cross-check tool responses and knowledge base answers against your actual data.
Tone — Does the response match your brand voice? Read it out loud. Does it sound like something your company would actually say?
Length — Chat responses should be concise. If the agent writes a five-paragraph essay when two sentences would do, adjust your behavior prompt to emphasize brevity.
Relevance — Does the response actually answer what was asked? A detailed answer to the wrong question is worse than no answer at all.
Natural language — Does it read like a human wrote it, or does it feel obviously automated? Watch for repetitive phrases, overly formal constructions, or robotic patterns.
Debugging Common Issues
Here are problems you're likely to encounter during testing and how to solve them:
Agent gives incorrect or outdated information
Cause: The knowledge base contains outdated content, or conflicting entries are confusing the AI. Fix: Review and update the specific knowledge entries. Remove outdated information. Make sure no two entries give contradictory answers on the same topic.
Agent says "I don't know" when it should know
Cause: The information exists in your knowledge base but isn't being retrieved. The question phrasing may not match how the knowledge is written. Fix: Add the specific question phrasing to your knowledge entry. Rephrase the content to include terms customers actually use. Add more variations and synonyms.
Agent calls the wrong tool or no tool at all
Cause: Tool descriptions are ambiguous, or the agent isn't recognizing when a tool is needed. Fix: Rewrite tool descriptions to be more specific. Include example scenarios in the description. Check that tool parameters match what the AI can extract from conversation.
Agent is too verbose
Cause: Default AI behavior tends toward thorough, lengthy responses. Fix: Add explicit instructions in your behavior prompt: "Keep responses concise — 2-3 sentences for simple questions, no more than a short paragraph for complex ones. This is a chat, not an email."
Agent doesn't hand off when it should
Cause: Handoff triggers aren't configured or are too narrow. Fix: Review your handoff settings. Add more trigger phrases. Consider lowering the confidence threshold so the agent escalates when it's less certain.
Don't skip testing just because your agent seems to work for basic questions. The edge cases — misspelled words, ambiguous requests, multi-part questions, messages in unexpected languages — are where problems hide. Spend at least 20-30 minutes in the testing playground with a critical eye.
Fine-Tuning Based on Test Results
Testing isn't a one-time event. It's an iterative process:
- Test a set of scenarios and document where the agent fell short.
- Identify the root cause — Is it a knowledge gap? A behavior prompt issue? A tool configuration problem?
- Make targeted fixes — Update the specific component that's causing the issue.
- Re-test the same scenarios to confirm the fix works.
- Test related scenarios to make sure the fix didn't break anything else.
Keep notes on what you changed and why. This becomes valuable documentation when you need to debug issues after the agent is live.
Going Live Checklist
Before activating your agent for real customer conversations, run through this checklist:
| Item | Status |
|---|---|
| Agent responds correctly to the top 20 most common questions | Verified |
| Tone and personality match your brand guidelines | Verified |
| Handoff to human agents works for explicit requests | Verified |
| Handoff triggers when the agent can't answer confidently | Verified |
| All configured tools return correct data | Verified |
| Tools handle errors gracefully (invalid inputs, API failures) | Verified |
| Welcome message is appropriate and not misleading | Verified |
| Agent handles greetings and small talk naturally | Verified |
| Agent stays within defined boundaries (doesn't discuss off-topic subjects) | Verified |
| Response length is appropriate for chat format | Verified |
| Agent works correctly in all intended languages | Verified |
| At least one team member has reviewed test conversations | Verified |
Even after going live, keep monitoring your agent's conversations for the first few days. The testing playground can't simulate every real-world scenario, and live traffic will surface edge cases you didn't think of. Be ready to make quick adjustments.
After Going Live
Your agent is live — congratulations. But the work doesn't stop here. Set up a regular review cycle:
- Daily (first week): Scan conversation logs for any obviously wrong or awkward responses.
- Weekly: Review handoff conversations to identify new knowledge gaps. If the AI is handing off the same type of question repeatedly, that's a signal to add that topic to your knowledge base.
- Monthly: Analyze agent performance metrics. Look at conversation completion rates, handoff rates, and response times. Adjust configurations based on trends.
What's Next?
- AI Agent Overview — Revisit the full capabilities of AI Agents in Chatmo.
- Knowledge Base — Add more knowledge to improve your agent's coverage.
- Tools Configuration — Connect additional tools as your needs evolve.
- Create an AI Agent — Set up another agent for a different channel or purpose.