Simulation Tests

​

What Are Simulation Tests?

• Create test scenarios
• Run automated conversations
• Evaluate agent responses
• Generate performance insights
• Identify improvement areas

​

Evaluation Suite

• Scenario management
• Test run execution
• Results analysis
• Performance insights

​

Creating Test Scenarios

​

Create Scenario Manually

• Scenario name
• Description
• Expected outcome
• Success criteria

• User messages (what caller says)
• Expected agent responses
• Required actions (tools to use)
• Conversation flow

Scenario: Appointment Booking - Happy Path

User: "Hi, I'd like to book an appointment"
Expected: Agent asks for preferred date

User: "Next Tuesday"
Expected: Agent checks availability, presents time options

User: "2 PM works for me"
Expected: Agent confirms booking, collects contact info

User: "john@email.com, 555-1234"
Expected: Agent confirms details, uses calendar_booking tool

Success Criteria:
- Appointment booked successfully
- Confirmation email sent
- Call ended politely

​

Generate Scenarios with AI

1. Click “Generate Scenario”
2. Describe scenario type
3. AI generates test conversation
4. User reviews and edits
5. Save scenario

• Happy path scenarios
• Error handling scenarios
• Edge case scenarios
• Compliance testing scenarios

​

Running Test Scenarios

​

Create Test Run

1. Select scenarios to test
2. Configure test parameters
3. Start test run
4. Wait for completion
5. Review results

• Scenarios to include
• Number of iterations
• Test environment
• Evaluation criteria

​

Viewing Test Results

​

Test Results Include

• Pass/Fail status
• Response accuracy
• Tool execution success
• Conversation flow correctness
• Response time metrics

• Intent recognition accuracy
• Entity extraction accuracy
• Tool usage correctness
• Response relevance
• Conversation completion rate

• Where agent failed
• Why it failed
• Actual vs expected response
• Suggested improvements

​

Generating Insights

​

Insights Provide

• Overall pass rate
• Common failure patterns
• Response quality trends
• Tool execution reliability

• Prompt adjustments
• Tool configuration changes
• Guardrail updates
• Training data needs

• Performance over time
• Regression detection
• Quality improvements
• Consistency metrics

​

Test Scenario Types

​

Happy Path Scenarios

• User requests appointment
• Provides all information clearly
• Accepts offered time slot
• Completes booking

• User asks clear question
• Agent retrieves information
• User satisfied with answer
• Call ends politely

• User asks about product
• Agent provides details
• User has follow-up questions
• Agent answers thoroughly

​

Error Handling Scenarios

• User doesn’t provide required data
• Agent prompts for missing info
• User eventually provides it
• Task completes

• Calendar API is down
• Webhook times out
• Email service fails
• Agent handles gracefully

• User mumbles or speaks unclearly
• Background noise present
• Agent asks for clarification
• Conversation continues

​

Edge Case Scenarios

• User starts one topic
• Switches to different topic
• Switches back
• Agent tracks context

• User interrupts agent
• Agent stops and listens
• Conversation continues naturally

• User asks unrelated questions
• Agent uses guardrails
• Redirects or transfers
• Maintains professionalism

​

Compliance Scenarios

• Agent doesn’t share sensitive data
• Verifies identity before sharing
• Follows HIPAA/GDPR rules

• Agent refuses medical advice
• Declines legal advice
• Redirects appropriately

• Agent transfers when required
• Explains transfer reason
• Connects to right department

​

Best Practices

​

Scenario Design

• Cover all conversation paths
• Include success and failure cases
• Test each tool thoroughly
• Simulate real user behavior
• Use actual customer language

• Only testing happy paths
• Unrealistic scenarios
• Missing edge cases
• Not testing tools
• Perfect input only

​

Test Coverage

• All primary tasks
• All enabled tools
• All conversation flows
• All escalation paths

• Response accuracy
• Response time
• Natural conversation flow
• Tool execution success

• Guardrail effectiveness
• Privacy protection
• Prohibited topic handling
• Escalation triggers

​

Regular Testing

• Initial configuration
• After major changes
• Before production release

• Weekly regression tests
• After prompt updates
• After tool configuration changes
• When issues reported

• Monthly comprehensive tests
• Quarterly reviews
• Annual audits

​

Interpreting Results

​

Pass/Fail Criteria

• Agent response matches expected
• Tools executed correctly
• Conversation completed successfully
• Guardrails respected

• Wrong intent detected
• Tool not executed when required
• Prohibited topics discussed
• Conversation incomplete

​

Common Failure Patterns

• User intent not understood
• Wrong task initiated
• Fix: Update identity/tasks section

• Tool not triggered
• Tool triggered incorrectly
• Fix: Update tool prompting

• Prohibited topics discussed
• Missing escalation
• Fix: Strengthen guardrails section

• Awkward transitions
• Missing context
• Fix: Improve task workflows

​

Iterating Based on Results

1. Review failed scenarios
2. Identify failure patterns
3. Update configuration
4. Re-run tests
5. Verify improvements

Test → Analyze Results → Update Config → Re-test → Deploy

​

Integration with Development

1. Make configuration change
2. Run relevant simulation tests
3. Review results
4. Fix issues
5. Re-test until passing
6. Deploy to production

• Automated test runs
• Pre-deployment checks
• Quality gates
• Regression detection

​

Simulation vs Manual Testing

• Quick feedback
• Real-time interaction
• Subjective evaluation
• Good for initial testing

• Comprehensive coverage
• Automated execution
• Objective metrics
• Good for regression testing
• Scalable testing

• Manual for rapid iteration
• Simulation for thorough validation
• Both before production

​

Next Steps

1. Address all failing scenarios
2. Achieve target pass rate (90%+)
3. Run final manual tests
4. Deploy to production
5. Monitor real performance
6. Update scenarios based on real calls