HomeGuidesSDK ExamplesAnnouncementsCommunity
Guides
Start typing to search…

Monitor

Prerequisites:

  • Access to Skills Studio
  • Understanding of assistant evaluation concepts
  • Permissions to configure evaluations and classifications

Main Content

Accessing the Monitoring Suite

  1. Navigate to Skills Studio.
  2. Click the Monitoring icon located at the bottom-left corner of the screen.

Overview Dashboard

Upon entering the Monitoring section, you'll see a dashboard containing key performance indicators for each assistant over the past 28 days:

  • Pass Rate
  • Total Questions
  • Active Users
  • New Active Users

Each metric is compared against the previous 28-day period to highlight growth or decline trends.

Classifications Table

Below the KPI cards, you'll find a Classifications Table which includes:

  • Classification names
  • Occurrence counts
  • Number of unique users impacted
  • Last seen date
  • Percentage of total issues

Special rows are always included:

  • Successful: Interactions that passed the evaluation
  • Unclassified: Interactions that did not receive a classification during evaluation
  • Unevaluated: System-maintained classification for questions that haven't been processed through evaluations (cannot be edited)

These give insights into both well-performing interactions and those needing deeper analysis.

Using the Unclassified Tab

To explore unclassified issues:

  1. Click the Unclassified Assistants tab.
  2. A new table will appear, showing:
  • Issues with no classification
  • Summary statistics including occurrences, users, last seen date, and issue percentage

Enhanced Classification Features

Editable Classifications

  • Edit directly from monitoring: Click into any classification to modify its name and criteria
  • In-place editing: No need to navigate away - changes are immediate
  • Preserved criteria: Classification criteria remain intact even when disabled

Active/Inactive States

  • Toggle auto-assignment: Enable or disable automatic classification assignment
  • Visual indicators:
  • Active classifications appear normally
  • Disabled classifications show with gray styling and "disabled" tag
  • Use cases:
  • Pause evaluations on resolved issues
  • Freeze classification groups for historical reference
  • Test new criteria without affecting production

Classifying Interactions

To assign or change a classification:

  1. Click into a specific classification.
  2. Select an interaction to review.
  3. Open the Diagnostics Panel.
  4. In the upper-right corner, expand the Classification section.
  5. Choose an existing classification or click Add Classification.

Enhanced Admin Feedback Panel

You can also classify directly from chat:

  1. Look for the tag icon beneath any chat response
  2. Access the consolidated feedback interface showing:
  • Notes field
  • Sentiment ratings (happy/sad/neutral)
  • All available classifications
  1. Add or modify classifications without opening diagnostics

Adding a New Classification

When creating a new classification:

  • Provide a Name
  • Define the Criteria for that classification

Once added, this classification becomes part of the regular Daily Evaluation process.

Reclassifying Issues

To reclassify:

  1. Select an existing issue.
  2. Open the interaction.
  3. Remove existing classifications.
  4. Add or assign a new one following the same steps as above.

Changes will be reflected in the Monitoring view, and occurrence counts will update accordingly.

Special System Classifications

Unevaluated Questions

  • Automatically maintained by the system
  • Cannot be edited or deleted
  • Appears when questions exist that haven't been processed
  • Helps identify evaluation coverage gaps
  • Prompts setup of automated evaluation schedules

Advanced Monitoring Features

View All Questions

  • Click "View All Questions" to access comprehensive question history
  • Direct access without navigating to Query Browser
  • Default 28-day filter (adjustable for historical data)

Multi-Select for Collections

  1. Use checkboxes to select multiple questions
  2. Click "Add[X] questions" button
  3. Create new collections or add to existing ones
  4. Build targeted test suites from monitoring insights

Groups Filtering

  • Groups column now available in Query Browser
  • Filter by user groups (admin, UAT, training, etc.)
  • Group information included in all exports
  • Analyze behavior patterns by user segment

Configuring Daily Evaluation

To ensure new classifications are evaluated daily:

  1. Navigate to the Assistant in Skills Studio.
  2. Click Settings > Evaluations.
  3. Enable the Daily Evaluations toggle.
  4. Set the Time (UTC) and Question Threshold (max questions per run).
  5. Optionally, exclude specific Domains (e.g., internal email domains) to focus on customer interactions only.

Important Notes

  • 28-day default view: Monitoring shows last 28 days by default
  • Historical data: Adjust date filters to view older interactions
  • Legacy admin tags: Automatically migrated to classifications
  • Tags older than 28 days require date filter adjustment to view
  • Persistent questions: Questions remain in Query Browser even after editing/deletion

Conclusion

The Monitoring suite in Skills Studio empowers you to evaluate assistant performance at a granular level by:

  • Viewing trends and interaction health
  • Creating and applying classifications with flexible management
  • Configuring automated daily evaluations
  • Building test collections directly from monitoring insights
  • Analyzing patterns by user groups

Regular use of this tool helps maintain high-quality user interactions and reveals opportunities for continuous improvement.

Updated 12 days ago