Monitor

This guide explains how to use the Monitoring section within Skills Studio to view, manage, and classify assistant interactions. It is intended for product analysts, QA engineers, and platform administrators who are responsible for tracking assistant performance and improving evaluation accuracy.

Prerequisites:

• Access to Skills Studio
• Understanding of assistant evaluation concepts
• Permissions to configure evaluations and classifications

---

Main Content

Accessing the Monitoring Suite

1. Navigate to Skills Studio.
2. Click the Monitoring icon located at the bottom-left corner of the screen.

---

Overview Dashboard

Upon entering the Monitoring section, you'll see a dashboard containing key performance indicators for each assistant over the past 28 days:

• Pass Rate
• Total Questions
• Active Users
• New Active Users

Each metric is compared against the previous 28-day period to highlight growth or decline trends.

---

Classifications Table

Below the KPI cards, you'll find a Classifications Table which includes:

• Classification names
• Occurrence counts
• Number of unique users impacted
• Last seen date
• Percentage of total issues

Special rows are always included:

• Successful: Interactions that passed the evaluation
• Unclassified: Interactions that did not receive a classification during evaluation

These give insights into both well-performing interactions and those needing deeper analysis.

---

Using the Unclassified Tab

To explore unclassified issues:

1. Click the Unclassified Assistants tab.
2. A new table will appear, showing:
   • Issues with no classification
   • Summary statistics including occurrences, users, last seen date, and issue percentage

---

Classifying Interactions

To assign or change a classification:

1. Click into a specific classification.
2. Select an interaction to review.
3. Open the Diagnostics Panel.
4. In the upper-right corner, expand the Classification section.
5. Choose an existing classification or click Add Classification.

Adding a New Classification

When creating a new classification:

• Provide a Name
• Define the Criteria for that classification

Once added, this classification becomes part of the regular Daily Evaluation process.

---

Reclassifying Issues

To reclassify:

1. Select an existing issue.
2. Open the interaction.
3. Remove existing classifications.
4. Add or assign a new one following the same steps as above.

Changes will be reflected in the Monitoring view, and occurrence counts will update accordingly.

---

Configuring Daily Evaluation

To ensure new classifications are evaluated daily:

1. Navigate to the Assistant in Skills Studio.
2. Click Settings > Evaluations.
3. Enable the Daily Evaluations toggle.
4. Set the Time (UTC) and Question Threshold (max questions per run).
5. Optionally, exclude specific Domains (e.g., internal email domains) to focus on customer interactions only.

---

Conclusion

The Monitoring suite in Skills Studio empowers you to evaluate assistant performance at a granular level by:

• Viewing trends and interaction health
• Creating and applying classifications
• Configuring automated daily evaluations

Regular use of this tool helps maintain high-quality user interactions and reveals opportunities for continuous improvement.