API Configuration

The API Configuration section within Assistant Settings allows users to customize the Large Language Models (LLMs) their assistant employs for various tasks. These settings ensure the assistant functions precisely as required by defining models for specific roles. This guide explains the available configurations, how they work, and why they are essential.

Audience: Administrators and developers managing assistant configurations.
Prerequisites: Familiarity with assistant features and tenant-wide configuration inheritance.

Main Content

1. Understanding the Role of API Configuration

The API Configuration section defines specific LLM models for different functionalities of the assistant. These models determine the behavior and quality of tasks such as generating responses, analyzing calls, and performing searches.

If no custom configuration is provided, the assistant will default to the tenant-wide configuration. This inheritance ensures a consistent baseline while allowing flexibility for unique assistant needs.

2. Available Configuration Types

Each function of the assistant relies on a distinct model. These can be configured independently to cater to specific requirements. The key configuration options are:

2.1 Chat

Purpose: Powers all chat-related tasks, including generating responses, skill selection, and parameter handling.
Example: Customizing this model improves conversational quality, enabling a tone or style tailored to a specific audience, such as technical professionals or casual users.

2.2 Narrative

Purpose: Handles writing semantic content, such as generating detailed analysis insights or structured artifacts.
Example: Configure this model to produce high-quality performance analysis or insightful reports, ensuring clarity and relevance.

2.3 Evaluations

Purpose: Used to assess the quality of chat responses, enabling continuous improvement.
Example: Setting this model helps in reviewing interactions to ensure accuracy, appropriateness, and user satisfaction.

2.4 Embeddings

Purpose: Determines matches for vector searches, enabling proximity-based data retrieval.
Example: Configure this for search functionalities that rely on understanding user intent, such as knowledge base queries.

2.5 Langsmith Tracing

Purpose: Analyzes LLM calls to provide insights into usage and performance metrics.
Example: Useful for auditing and optimizing how LLM resources are used across different scenarios.

3. Configuring Models

Configurations can be set at two levels:

Assistant-Specific Configuration: Overrides the tenant-wide settings for a particular assistant.
Inherited Configuration: Defaults to the tenant-wide settings unless explicitly changed.

Steps to Configure:

Navigate to Assistant Settings > API Configuration.
Select the configuration type you want to modify (e.g., Chat, Narrative).
Choose a specific LLM model from the dropdown menu.
APIs can be specifically configured in the API Config section of skill studio (Things like temperature and others can be managed here)
Save changes to apply.

Tip: Regularly review the performance of each model to ensure it aligns with the assistant's goals.

Troubleshooting/FAQs

Q1: What happens if no configuration is set?

If no model is defined for a specific function, the assistant will inherit the tenant-wide configuration. This ensures baseline functionality without additional setup.

Q2: Can I use different models for different assistants?

Yes, each assistant can have independent configurations, allowing for diverse functionality across multiple use cases.

Conclusion

Configuring LLM models in the API Configuration section is a powerful way to tailor the assistant's functionality to specific use cases. By understanding and customizing these models, users can significantly enhance the quality, relevance, and effectiveness of their assistant.