Assistant Monitoring & Classification

This guide provides a walkthrough of the Assistant Monitoring features within Skill Studio, including how to view statistics, evaluate responses, and set up issue classifications. It is intended for users responsible for monitoring assistant performance and improving issue resolution workflows.

Audience: Skill Studio users managing assistant interactions and diagnostics
Prerequisites: Access to Skill Studio, an existing assistant setup, and familiarity with assistant evaluations. Ensure Daily Evaluations is enabled in assistant settings.

Monitoring Your Assistant in Skill Studio

Navigating to Monitoring

1. Navigate to Skill Studio.
2. From the left-hand navigation, select Monitor.
3. You'll see monitoring stats for the currently selected assistant (check in the top-left corner).

Key Metrics Displayed

At the top of the monitoring page, you’ll find key performance stats:

• Pass Rate for Evaluations
• Total Questions Asked
• Active Users
• New Active Users

🛈 Tip: Hover over any data point to view additional context.
📊 The Change Information on each card shows performance difference compared to the previous 28 days.

Understanding the Classifications Table

Beneath the overview cards, you’ll find a table displaying classifications for assistant interactions.

Default State: Unclassified

• All new interactions start as Unclassified.
• From here, you can begin defining your own classifications to track specific issues.

Table Columns Explained

• Issue Type Occurrence: How often this type of issue appeared
• Impacted Users: Number of users affected
• Last Seen: Date the classification last appeared
• % of Total Issues: Proportion of total interactions this issue represents

Creating and Managing Classifications

Step 1: Review Unclassified Interactions

1. Click on the Unclassified group.
2. A list of unclassified questions/interactions will appear.
3. Make sure the Assistant column is filtered to your current assistant.

Step 2: Evaluate the Response

1. Select an issue from the list.
2. Open the Diagnostics panel and go to the Answer tab.
3. In the Evaluation section, check if “Does Response Answer User Query” has run.
   • If not, hit Enter to initiate the evaluation.
4. Once completed, review the prompt and classification outcome.

Step 3: Add a New Classification

1. Click Add Classification above the classification section.
2. Provide:
   • A clear, specific Classification Name
   • Detailed Criteria (e.g., "Fails to respond when queried about payment settings in Skill X")
3. The system will use this data to auto-classify future evaluations.

⚠️

Avoid general criteria like “has an error.” Be specific to improve accuracy.

Step 4: Apply and Manage Classifications

• Choose your classification(s) via the Select dropdown.
  • Gray: Test classification, not yet active
  • Red: Active classification running on an evaluation schedule
• Click the classification pill to:
  • Edit or toggle on/off
  • Adjust name or criteria
  • Save to apply

Step 5: Remove or Modify Classifications

• Remove a classification by clicking the X on the pill or the box.
• Update classifications anytime by:
  • Revisiting the question
  • Adjusting existing classifications
  • Adding new ones