Monitoring Tutorial (25.06)

The monitoring suite is a powerful tool that helps you track, classify, and address issues within your client or tenant environment. This guide explains how to effectively use the classification system, evaluate responses, and leverage monitoring data to improve performance.

Classification System

Adding Classifications

Classifications can be added in two ways:

1. Manual Addition:

   • Open the diagnostics panel by selecting an interaction in the query browser
   • Expand the classification section
   • Either select an existing classification or click "Add Classification"
   • When adding a new classification, provide a descriptive name and specific criteria
   • Classifications can be edited after creation

2. Automatic Evaluation:

   • Classifications are applied automatically when the "Does response answer query" evaluation runs
   • This can be scheduled to run daily or triggered manually

Writing Effective Classification Criteria

When creating classification criteria, leverage the content available to the evaluation model:

• Reference relevant tags:

  • query: Use when classifying based on what the user asked
  • response: Use when classifying based on how the system responded
  • message history: Use when classifying based on the conversation context
  • System prompt elements like data values and skill descriptions

• Examples of effective criteria:

  • "When the message history reflects that the user asked several questions, but the assistant response does not include any direct answers"
  • "When the user requests information in a language different from the system's configured language"
  • "When a specific brand mentioned in message history is automatically appended as a parameter"

Setting Up Daily Evaluations

Configuration Best Practices

1. Set the correct UTC time:

   • Ensure hour and minutes are set to UTC time
   • Use Google to confirm the current UTC time if needed

2. Configure question threshold appropriately:

   • Set to the average number of queries per day (with some buffer)
   • Setting to zero will prevent automatic evaluations
   • Example: If you average 80 queries per day, set threshold to 100

3. Exclude internal domains:

   • Add internal domains like "answerocket" to the exclusion list
   • This prevents evaluations on internal test questions

Using the Monitoring Dashboard

Navigating the Monitor Tab

1. Access the monitoring suite from Skill Studio by clicking the "Monitor" tab
2. Select the relevant assistant from the dropdown at the top
3. View key statistics for the last 28 days:
   • Pass rate
   • Total questions
   • Active users
   • New active users

Understanding Classifications Table

The monitoring dashboard displays classifications in a table format:

1. Successful Questions:

   • Queries that passed evaluation
   • Can still have classifications attached

2. Issue Classifications:

   • Each classification shows:
     • Title and criteria
     • Number of occurrences
     • Number of users impacted
     • Most recent occurrence date
     • Percentage of total classified issues
   • Use these metrics to prioritize issue resolution

Analyzing Classification Details

When you click on a classification:

1. Navigate through examples using arrow keys
2. View all query browser data for each instance
3. Reorganize table columns by dragging
4. Apply filters by date or question type
5. Share the classification link in tickets for troubleshooting

Important Notes

• Test runs are excluded: Interactions from test runs won't appear in classifications to prevent clutter
• Prioritize by impact: Focus on issues affecting multiple users first
• Use classification links: When creating JIRA tickets, include the classification link rather than individual question URLs
• Regular review: Periodically review classifications to identify patterns and recurring issues

Best Practices Summary

1. Create specific, targeted classifications that reference the evaluation context
2. Configure daily evaluations with appropriate thresholds and exclusions
3. Prioritize issues based on occurrence count, user impact, and recency
4. Use classification links in troubleshooting tickets
5. Regularly review and refine your classification system

Note: This guide reflects the monitoring suite functionality as of version 2506. Future updates may enhance these capabilities.