Assistant Monitoring & Classification

This guide provides a walkthrough of the Assistant Monitoring features within Skill Studio, including how to view statistics, evaluate responses, and set up issue classifications. It is intended for users responsible for monitoring assistant performance and improving issue resolution workflows.

Audience: Skill Studio users managing assistant interactions and diagnostics
Prerequisites: Access to Skill Studio, an existing assistant setup, and familiarity with assistant evaluations. Ensure Daily Evaluations is enabled in assistant settings.

Monitoring Your Assistant in Skill Studio

Navigating to Monitoring

1. Navigate to Skill Studio.
2. From the left-hand navigation, select Monitor.
3. You'll see monitoring stats for the currently selected assistant (check in the top-left corner).

Key Metrics Displayed

At the top of the monitoring page, you'll find key performance stats:

• Pass Rate for Evaluations
• Total Questions Asked
• Active Users
• New Active Users

🛈 Tip: Hover over any data point to view additional context.
📊 The Change Information on each card shows performance difference compared to the previous 28 days.

Understanding the Classifications Table

Beneath the overview cards, you'll find a table displaying classifications for assistant interactions.

Default Classifications

• Successful: Interactions that passed the evaluation
• Unclassified: Interactions that did not receive a classification during evaluation
• Unevaluated: System-maintained classification for questions that haven't been processed through evaluations (cannot be edited)

Table Columns Explained

• Issue Type Occurrence: How often this type of issue appeared
• Impacted Users: Number of users affected
• Last Seen: Date the classification last appeared
• % of Total Issues: Proportion of total interactions this issue represents

📘

Note: The monitoring views default to the last 28 days. For interactions older than 28 days (including migrated legacy admin tags), adjust the date filter in classification detail pages.

Creating and Managing Classifications

Step 1: Review Unclassified Interactions

1. Click on the Unclassified group.
2. A list of unclassified questions/interactions will appear.
3. Make sure the Assistant column is filtered to your current assistant.

Step 2: Evaluate the Response

1. Select an issue from the list.
2. Open the Diagnostics panel and go to the Answer tab.
3. In the Evaluation section, check if "Does Response Answer User Query" has run.
   • If not, hit Enter to initiate the evaluation.
4. Once completed, review the prompt and classification outcome.

Step 3: Add a New Classification

Enhanced Admin Feedback Panel

You can now add classifications directly from the chat interface:

1. Click the tag icon beneath any chat response
2. Use the consolidated interface showing notes, sentiment ratings (happy/sad/neutral), and all available tags
3. Select existing classifications or create new ones without opening the diagnostics panel

From Monitoring or Diagnostics

1. Click Add Classification above the classification section.
2. Provide:
   • A clear, specific Classification Name (editable later)
   • Detailed Criteria (e.g., "Fails to respond when queried about payment settings in Skill X")
3. The system will use this data to auto-classify future evaluations.

⚠️

Avoid general criteria like "has an error." Be specific to improve accuracy.

Step 4: Apply and Manage Classifications

• Choose your classification(s) via the Select dropdown.
• Visual indicators show classification status:
  • Gray with "disabled" tag: Classification is inactive (auto-assignment disabled)
  • Red: Active classification running on an evaluation schedule
• Click the classification pill or name to:
  • Edit name or criteria directly from the monitoring page
  • Toggle auto-assignment on/off
  • Save to apply changes

Step 5: Modify Classifications

Editing Classifications

• Click into any classification from the monitoring page
• Modify the name and evaluation criteria in-place
• Changes are immediately reflected across all monitoring views
• Classification criteria are preserved even when auto-assignment is disabled

Active/Inactive States

• Use the toggle to temporarily disable auto-assignment
• Disabled classifications appear grayed out with a "disabled" tag
• Useful for:
  • Pausing evaluations on legacy issues
  • Freezing classification groups in time
  • Maintaining historical classifications without active processing
• Re-enable by editing the classification and turning auto-assign back on

Remove Classifications

• Remove a classification by clicking the X on the pill or the box
• Update classifications anytime by revisiting and adjusting

Special Classifications

Unevaluated Questions

• Automatically maintained by the system
• Appears when questions exist that haven't been processed through evaluations
• Cannot be edited or deleted
• Helps identify gaps in evaluation coverage
• Prompts setup of automated evaluation schedules or manual evaluation runs

Using Monitoring for Collection Building

Direct Access to All Questions

• Click "View All Questions" from the monitoring page
• Access comprehensive question history without navigating to Query Browser
• Default 28-day filter can be adjusted for historical analysis

Multi-Select for Test Collections

1. From classification detail pages, select multiple questions using checkboxes
2. Click "Add[X] questions" button
3. Create new collections or add to existing ones
4. Build targeted test suites for issue validation

Groups Filtering

• New Groups column in Query Browser for filtering by user groups
• Filter by specific groups (admins, UAT training, etc.)
• Group information included in all data exports

Best Practices

Classification Management

• Create specific classifications with clear, actionable criteria
• Use descriptive names that quickly identify the issue type
• Regularly review active vs. inactive classifications
• Leverage the tag icon in chat for immediate classification

Evaluation Coverage

• Monitor the Unevaluated classification to ensure comprehensive coverage
• Set up daily evaluations to process questions automatically
• Use manual evaluations for immediate assessment

Historical Data

• Remember to adjust date filters when reviewing older interactions
• Legacy admin tags are automatically migrated but may require date filter adjustment
• Use classification links in troubleshooting tickets rather than individual question URLs

Workflow Integration

• Build test collections directly from classification results
• Use groups filtering to analyze user-segment-specific issues
• Export data with groups information for external analysis