HomeGuidesSDK ExamplesAnnouncementsCommunity
Guides

Monitoring Overview

Suggest Edits

The monitoring suite is a powerful tool that helps you track, classify, and address issues within your client or tenant environment. This guide explains how to effectively use the classification system, evaluate responses, and leverage monitoring data to improve performance.

Classification System

Adding Classifications

Classifications can be added in multiple ways:

  1. From Chat Interface:

    • Click the tag icon beneath any chat response
    • Select existing classifications or create new ones directly from the conversation
    • New classifications are automatically applied to the current interaction

  2. From Query Browser:

    • Open the diagnostics panel by selecting an interaction in the query browser
    • Expand the classification section
    • Either select an existing classification or click "Add Classification"

  3. From Monitoring Page:

    • Navigate to any classification and click "Edit Classification"
    • Modify name and criteria directly from the monitoring interface

  4. Automatic Evaluation:

    • Classifications are applied automatically when the "Does response answer query" evaluation runs
    • This can be scheduled to run daily or triggered manually

Managing Classifications

Editing Classifications:

  • Click into any classification from the monitoring page to modify its name and evaluation criteria
  • Changes are immediately reflected across all monitoring views
  • Classification criteria are preserved even when auto-assignment is disabled

Active/Inactive States:

  • Toggle auto-assignment on/off for any classification
  • Disabled classifications appear grayed out with a "disabled" tag
  • Useful for pausing evaluations on legacy issues or freezing classification groups in time
  • Re-enable by editing the classification and turning auto-assign back on

Unevaluated Questions:

  • System automatically tracks questions that haven't been processed through evaluations
  • "Unevaluated" classification appears when questions exist that haven't been run through daily or manual evaluations
  • Cannot be edited as it's automatically maintained by the system
  • Helps identify gaps in evaluation coverage

Writing Effective Classification Criteria

When creating classification criteria, leverage the content available to the evaluation model:

  • Reference relevant tags:

    • query: Use when classifying based on what the user asked
    • response: Use when classifying based on how the system responded
    • message history: Use when classifying based on the conversation context
    • System prompt elements like data values and skill descriptions

  • Examples of effective criteria:

    • "When the message history reflects that the user asked several questions, but the assistant response does not include any direct answers"
    • "When the user requests information in a language different from the system's configured language"
    • "When a specific brand mentioned in message history is automatically appended as a parameter"

Legacy Admin Tag Migration

  • Legacy admin tags are automatically converted to the new classification system
  • Migrated tags maintain backward compatibility
  • Important: Tagged interactions older than 28 days require adjusting the date filter in classification detail pages to be visible

Setting Up Daily Evaluations

Configuration Best Practices

  1. Set the correct UTC time:

    • Ensure hour and minutes are set to UTC time
    • Use Google to confirm the current UTC time if needed

  2. Configure question threshold appropriately:

    • Set to the average number of queries per day (with some buffer)
    • Setting to zero will prevent automatic evaluations
    • Example: If you average 80 queries per day, set threshold to 100

  3. Exclude internal domains:

    • Add internal domains like "answerocket" to the exclusion list
    • This prevents evaluations on internal test questions

Using the Monitoring Dashboard

Navigating the Monitor Tab

  1. Access the monitoring suite from Skill Studio by clicking the "Monitor" tab
  2. Select the relevant assistant from the dropdown at the top
  3. View key statistics for the last 28 days:
    • Pass rate
    • Total questions
    • Active users
    • New active users

Understanding Classifications Table

The monitoring dashboard displays classifications in a table format:

  1. Successful Questions:

    • Queries that passed evaluation
    • Can still have classifications attached

  2. Issue Classifications:

    • Each classification shows:
      • Title and criteria
      • Number of occurrences
      • Number of users impacted
      • Most recent occurrence date
      • Percentage of total classified issues
    • Use these metrics to prioritize issue resolution

  3. Unevaluated Questions:

    • System-maintained classification for questions not yet processed through evaluations
    • Prompts setup of automated evaluation schedules or manual evaluation runs

Analyzing Classification Details

When you click on a classification:

  1. Navigate through examples using arrow keys
  2. View all query browser data for each instance
  3. Reorganize table columns by dragging
  4. Apply filters by date or question type
  5. Share the classification link in tickets for troubleshooting

Question Browser Integration

View All Questions

  • Access all questions for the current assistant directly from the monitoring page
  • Click "View All Questions" to see comprehensive question history
  • Default 28-day filter can be adjusted for historical analysis

Multi-Select and Bulk Operations

Selecting Multiple Questions:

  1. Use checkboxes on the left side of questions in "View All Questions" view
  2. Select questions from classification detail pages
  3. "Add [X] questions" button appears when questions are selected

Bulk Actions:

  • Add selected questions to existing collections
  • Create new collections from selected questions
  • Useful for building test suites after identifying and fixing issues

Groups and Filtering

Groups Column:

  • New filtering capability by user groups in Query Browser
  • Filter by specific groups (admins, UAT training, etc.)
  • Shows all groups a user belongs to, not just the filtered group
  • Group information included in all data exports

Enhanced Persistence:

  • Questions and responses preserved even after editing or deletion
  • Full conversation history maintained for troubleshooting
  • Multiple versions of edited questions show complete interaction progression

Creating Test Collections from Monitoring

Workflow for Issue Resolution

  1. Identify Issues: Use monitoring dashboard to spot problematic classifications
  2. Analyze Questions: Click "View All Questions" to see comprehensive scope
  3. Select Related Questions: Use multi-select to choose questions affected by the same issue
  4. Create Validation Collection: Add selected questions to a new collection for post-fix testing
  5. Schedule Evaluations: Run tests on the collection to validate fixes

Best Practices for Collections

  • Use descriptive names that indicate purpose (e.g., "Authentication Issues - March 2024")
  • Group questions by issue type for focused testing
  • Create collections before and after fixes for comparison
  • Leverage groups filtering to create user-segment-specific test suites

Important Notes

  • Test runs are excluded: Interactions from test runs won't appear in classifications to prevent clutter
  • 28-day default filtering: Monitoring views default to last 28 days but can be adjusted for historical data
  • Prioritize by impact: Focus on issues affecting multiple users first
  • Use classification links: When creating JIRA tickets, include the classification link rather than individual question URLs
  • Regular review: Periodically review classifications to identify patterns and recurring issues
  • System prompt updates: Enhanced context handling applies to newly created assistants or existing assistants with updated system prompts

Best Practices Summary

  1. Create specific, targeted classifications that reference the evaluation context
  2. Configure daily evaluations with appropriate thresholds and exclusions
  3. Use the new chat interface for immediate classification and feedback
  4. Leverage multi-select to efficiently organize questions into test collections
  5. Take advantage of groups filtering for user-segment analysis
  6. Prioritize issues based on occurrence count, user impact, and recency
  7. Use classification links in troubleshooting tickets
  8. Regularly review and refine your classification system
  9. Adjust date filters when working with historical data or migrated admin tags

Note: This guide reflects the monitoring suite functionality as of version 2507. Future updates may enhance these capabilities.

Updated 10 days ago