Evolving Multi-Agent Systems: The Need for Dynamic Tool Selection

In the realm of artificial intelligence, multi-agent systems are constantly evolving to meet the diverse needs of users. One of the key challenges in this field is the development of dynamic tool selection mechanisms that can adapt to a wide range of queries. This blog post explores innovative solutions and research directions in this exciting area.

Key Areas for Improvement

To enhance the adaptability of multi-agent frameworks, several key areas require focus:

Tool Embedding: Implementing semantic matching of queries to tool capabilities
Intent-based Routing: Guiding tool selection based on query intent
Tool Discovery: Enabling on-the-fly tool creation or discovery
Meta-agents: Orchestrating tool and sub-agent selection dynamically

Spotlight on Intent Classification

Intent classification plays a crucial role in dynamic tool selection. Let’s dive into the internals of SetFit, a powerful framework for few-shot learning, and explore how it can be applied to intent classification and tool selection.

SetFit Internals

SetFit employs a two-stage approach for few-shot learning:

Sentence Transformer Fine-tuning: The model is fine-tuned on a small set of labeled examples using contrastive learning.
Classification Head Training: A classifier is trained on the embeddings produced by the fine-tuned Sentence Transformer.

Implementing Intent Classification with SetFit

Here’s how we can adapt SetFit for intent classification and tool selection:

Prepare a dataset of intents and corresponding tools:

intents_and_tools = [
    ("forecast sales", ["create_prophet_model", "fit_model", "make_future_dataframe", "predict"]),
    ("visualize trend", ["plot_forecast", "plot_components"]),
    ("add seasonality", ["add_seasonality", "fit_model"]),
    ("evaluate model", ["cross_validation", "performance_metrics"])
]

Fine-tune a SetFit model:

from setfit import SetFitModel, SetFitTrainer
from sentence_transformers.losses import CosineSimilarityLoss

# Prepare data
texts, labels = zip(*[(intent, ",".join(tools)) for intent, tools in intents_and_tools])

# Initialize model
model = SetFitModel.from_pretrained("sentence-transformers/paraphrase-mpnet-base-v2")

# Create trainer
trainer = SetFitTrainer(
    model=model,
    train_dataset=(texts, labels),
    loss_class=CosineSimilarityLoss,
    batch_size=16,
    num_iterations=20
)

# Train the model
trainer.train()

Use the model for intent classification and tool selection:

def select_tools(query):
    prediction = model.predict([query])[0]
    return prediction.split(",")

# Example usage
query = "I need to forecast sales for the next quarter"
selected_tools = select_tools(query)
print(f"Selected tools for '{query}': {selected_tools}")

Alternative Classification Heads

While SetFit typically uses a logistic regression classifier, you can experiment with other classification heads:

Neural Network Head

from setfit import SetFitModel
import torch.nn as nn

class NeuralNetHead(nn.Module):
    def __init__(self, input_dim, num_classes):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(input_dim, 64),
            nn.ReLU(),
            nn.Linear(64, num_classes)
        )
    
    def forward(self, x):
        return self.layers(x)

model = SetFitModel.from_pretrained(
    "sentence-transformers/paraphrase-mpnet-base-v2",
    head_params={"head": NeuralNetHead, "input_dim": 768, "num_classes": len(set(labels))}
)

Random Forest Classifier

from sklearn.ensemble import RandomForestClassifier

model = SetFitModel.from_pretrained(
    "sentence-transformers/paraphrase-mpnet-base-v2",
    head_params={"head": RandomForestClassifier, "n_estimators": 100}
)

Multi-label Classification for Multiple Tool Selection

To select multiple tools, we can adapt SetFit for multi-label classification:

Prepare multi-label data:

from sklearn.preprocessing import MultiLabelBinarizer

mlb = MultiLabelBinarizer()
binary_labels = mlb.fit_transform([tools for _, tools in intents_and_tools])

Use a multi-label classification head:

from sklearn.multioutput import MultiOutputClassifier
from sklearn.linear_model import LogisticRegression

model = SetFitModel.from_pretrained(
    "sentence-transformers/paraphrase-mpnet-base-v2",
    head_params={
        "head": MultiOutputClassifier,
        "estimator": LogisticRegression()
    }
)

# Train the model (similar to previous example)

def select_multiple_tools(query):
    prediction = model.predict([query])[0]
    return mlb.inverse_transform(prediction)[0]

# Example usage
query = "Forecast sales and visualize the trend"
selected_tools = select_multiple_tools(query)
print(f"Selected tools for '{query}': {selected_tools}")

SetFit Head: Flexibility in Classification

The SetFit head is the classification component that sits on top of the fine-tuned sentence transformer. It takes the embeddings produced by the transformer and maps them to the output classes (in our case, intents or tools). The head is trained separately from the transformer, allowing for flexibility in choosing different classification algorithms.

Types of SetFit Heads

Logistic Regression (Default)
- When to use: For binary or multi-class classification with linearly separable data.
- Good for interpretability and works well with small datasets.
Neural Network
- When to use: For complex, non-linear relationships in the data.
- Useful when you have a larger number of examples and classes.
Random Forest
- When to use: When you want to capture non-linear relationships and have a mix of categorical and numerical features.
- Good for handling outliers and maintaining good performance with default hyperparameters.
Support Vector Machine (SVM)
- When to use: For binary classification tasks or when you have a clear margin of separation in your data.
- Effective in high-dimensional spaces.

Contrastive Learning-based Training

Contrastive learning in SetFit works by creating positive and negative pairs of sentences. Here’s an example:

from setfit import SetFitModel, SetFitTrainer
from sentence_transformers.losses import CosineSimilarityLoss

# Sample data
intents_and_tools = [
    ("forecast sales", ["create_model", "predict"]),
    ("visualize trend", ["plot_forecast"]),
    ("add seasonality", ["add_seasonality", "fit_model"]),
    ("evaluate model", ["cross_validation", "performance_metrics"])
]

# Prepare data for contrastive learning
train_pairs = []
for i, (intent1, tools1) in enumerate(intents_and_tools):
    for j, (intent2, tools2) in enumerate(intents_and_tools):
        if i == j:
            # Positive pair
            train_pairs.append((intent1, intent2, 1))
        else:
            # Negative pair
            train_pairs.append((intent1, intent2, 0))

# Unpack the pairs
texts1, texts2, labels = zip(*train_pairs)

# Initialize model
model = SetFitModel.from_pretrained("sentence-transformers/paraphrase-mpnet-base-v2")

# Create trainer with contrastive learning
trainer = SetFitTrainer(
    model=model,
    train_dataset=(texts1, texts2, labels),
    loss_class=CosineSimilarityLoss,
    batch_size=16,
    num_iterations=20
)

# Train the model
trainer.train()

# Now, train the classification head
X, y = zip(*[(intent, ",".join(tools)) for intent, tools in intents_and_tools])
model.fit(X, y)

# Use the model
query = "I need to forecast sales for the next quarter"
predicted_tools = model.predict([query])[0].split(",")
print(f"Predicted tools for '{query}': {predicted_tools}")

In this example, we create positive pairs (same intent) and negative pairs (different intents) for contrastive learning. The model learns to minimize the distance between positive pairs and maximize the distance between negative pairs in the embedding space.

Choosing the Right SetFit Head

Start with the default Logistic Regression head. It’s fast, interpretable, and often performs well on a variety of tasks.
If you have a complex task with non-linear relationships, try a Neural Network or Random Forest head.
For binary classification tasks, especially with clear separation, consider an SVM head.
Always validate your choice by comparing performance across different heads using cross-validation.

Pretrained Models and Classifiers Supported by SetFit

SetFit supports a wide range of pretrained models and classifiers for its head:

Pretrained Models

SetFit can use any Sentence Transformer model available on the Hugging Face Hub, including:

BERT-based models (e.g., “bert-base-uncased”)
RoBERTa-based models (e.g., “roberta-base”)
DistilBERT-based models (e.g., “distilbert-base-uncased”)
MPNet-based models (e.g., “sentence-transformers/paraphrase-mpnet-base-v2”)
ModernBERT models (e.g., “nomic-ai/modernbert-embed-base”)
Multilingual models (e.g., “sentence-transformers/paraphrase-multilingual-mpnet-base-v2”)
Domain-specific models (e.g., “sentence-transformers/all-MiniLM-L6-v2”)

Classifiers Supported for SetFit Head

SetFit supports two main types of classification heads:

Scikit-learn based classifiers:
- LogisticRegression (default)
- RandomForestClassifier
- SVC (Support Vector Classifier)
- MLPClassifier (Multi-layer Perceptron)
PyTorch-based differentiable head (SetFitHead):
- A linear layer followed by a softmax activation for multi-class classification
- A linear layer followed by a sigmoid activation for multi-label classification

For multilabel classification, SetFit supports the following strategies:

“one-vs-rest”: Uses OneVsRestClassifier
“multi-output”: Uses MultiOutputClassifier
“classifier-chain”: Uses ClassifierChain

By leveraging these powerful tools and techniques, we can create more flexible and adaptive multi-agent systems capable of handling a wide range of user queries efficiently. The future of AI lies in these dynamic, intelligent systems that can seamlessly adapt to user needs and intents.

References

Microsoft. (2025). Customize Speaker Selection | AutoGen 0.2. Retrieved from https://microsoft.github.io/autogen/0.2/docs/topics/groupchat/customized_speaker_selection/ Pizzocaro, D. (2025, January 13). Introducing smolagents: simple agents that write actions in code. [LinkedIn post]. Retrieved from https://www.linkedin.com/posts/diegopizzocaro_introducing-smolagents-simple-agents-that-activity-7280139201654800385-orVL PromptLayer. (2024, December 16). The Top Agentic Frameworks | How to build AI Agents. Retrieved from https://blog.promptlayer.com/what-are-the-top-agentic-frameworks-ai-agent-agentic-definition-explanation/ Roucher, A. (2025, January 13). Introducing smolagents: simple agents that write actions in code. Hugging Face. Retrieved from https://huggingface.co/blog/smolagents Softweb Solutions. (2024, November 7). Agentic RAG: Types, applications, and implementation guide. Retrieved from https://www.softwebsolutions.com/resources/agentic-rag-types-and-implementation-guide.html Startup News. (2025, January 7). A Developer’s Guide to the AutoGen AI Agent Framework. Retrieved from https://startupnews.fyi/2025/01/07/a-developers-guide-to-the-autogen-ai-agent-framework/ Tarallo, P. (2025, January 13). Introducing smolagents: simple agents that write actions in code. [LinkedIn post]. Retrieved from https://www.linkedin.com/posts/pasqualetarallo_introducing-smolagents-simple-agents-that-activity-7280264588363784192-doP9

AgenticTool Related Posts