Enterprises are increasingly embracing the Model Context Protocol (MCP) mainly to streamline the identification and guidance of agent tool usage. However, researchers from Salesforce have uncovered an alternative application for MCP technology, this time aimed at assessing AI agents themselves. 

The team introduced MCPEval, an innovative method and open-source toolkit based on the MCP system's architecture, designed to evaluate agent performance when utilizing tools. They pointed out that existing evaluation methods for agents are limited because they “often relied on static, pre-defined tasks, thus failing to capture the interactive real-world agentic workflows.”

“MCPEval transcends traditional success/failure metrics by systematically collecting detailed task trajectories and protocol interaction data, offering unprecedented visibility into agent behavior and generating valuable datasets for iterative improvement,” the researchers explained in the paper. “Moreover, as both task creation and verification are fully automated, the resulting high-quality trajectories can be immediately utilized for rapid fine-tuning and continuous enhancement of agent models. The comprehensive evaluation reports generated by MCPEval also deliver actionable insights into the accuracy of agent-platform communication at a granular level.”

MCPEval distinguishes itself by being a fully automated process, which the researchers claimed enables swift evaluation of new MCP tools and servers. It not only gathers information on how agents interact with tools within an MCP server but also generates synthetic data and creates a database for benchmarking agents. Users can select which MCP servers and tools within those servers to test the agent’s performance. 

AI Scaling Hits Its Limits

Power caps, rising token costs, and inference delays are reshaping enterprise AI. Join our exclusive salon to discover how top teams are:

• Turning energy into a strategic advantage

• Architecting efficient inference for real throughput gains

• Unlocking competitive ROI with sustainable AI systems

Secure your spot to stay ahead: https://bit.ly/4mwGngO

Shelby Heinecke, senior AI research manager at Salesforce and one of the paper’s authors, told VentureBeat that obtaining accurate data on agent performance, especially for agents in domain-specific roles, is challenging. 

“We’ve reached a point where, if you look across the tech industry, many of us have figured out how to deploy them. Now, we need to determine how to evaluate them properly,” Heinecke stated. “MCP is a novel idea, a new paradigm. While it's great that agents will have access to tools, we again need to evaluate the agents on those tools. That’s precisely what MCPEval is designed for.”

How it works

MCPEval’s framework incorporates task generation, verification, and model evaluation design. By leveraging multiple large language models (LLMs), users can opt to work with models they are more familiar with, allowing agents to be evaluated through a variety of available LLMs in the market. 

Enterprises can access MCPEval via an open-source toolkit released by Salesforce. Through a dashboard, users configure the server by selecting a model, which then automatically generates tasks for the agent to execute within the chosen MCP server. 

Once the user verifies the tasks, MCPEval then determines the necessary tool calls as ground truth. These tasks are used as the basis for the test. Users choose which model they prefer to run the evaluation. MCPEval can generate a report on how effectively the agent and the test model performed in accessing and using these tools. 

MCPEval not only gathers data to benchmark agents, Heinecke noted, but it can also identify gaps in agent performance. Information obtained from evaluating agents through MCPEval serves not only to test performance but also to train the agents for future applications. 

“We envision MCPEval evolving into a comprehensive platform for evaluating and refining your agents,” Heinecke said. 

She added that MCPEval stands out from other agent evaluators by bringing the testing to the same environment in which the agent will operate. Agents are assessed on how effectively they access tools within the MCP server where they are likely to be deployed. 

The paper highlighted that in experiments, GPT-4 models frequently provided the best evaluation results. 

Evaluating agent performance

The growing necessity for enterprises to start testing and monitoring agent performance has led to a surge of frameworks and techniques. Some platforms offer testing and a variety of methods to evaluate both short-term and long-term agent performance. 

AI agents are designed to perform tasks on behalf of users, often without the requirement for a human to prompt them. To date, agents have proven useful, yet they can become overwhelmed by the sheer volume of tools at their disposal.  

Galileo, a startup, provides a framework that enables enterprises to assess the quality of an agent’s tool selection and identify errors. Salesforce launched capabilities on its Agentforce dashboard to test agents. Researchers from Singapore Management University released AgentSpec to achieve and monitor agent reliability. Several academic studies on MCP evaluation have also been published, including MCP-Radar and MCPWorld.

MCP-Radar, developed by researchers from the University of Massachusetts Amherst and Xi’an Jiaotong University, focuses on more general domain skills, such as software engineering or mathematics. This framework prioritizes efficiency and parameter accuracy. 

Conversely, MCPWorld from Beijing University of Posts and Telecommunications brings benchmarking to graphical user interfaces, APIs, and other computer-use agents.

Heinecke stated that ultimately, how agents are evaluated will depend on the company and the use case. However, it is crucial for enterprises to choose the most suitable evaluation framework for their specific needs. For enterprises, she suggested considering a domain-specific framework to thoroughly test how agents function in real-world scenarios.

“There’s value in each of these evaluation frameworks, and they provide a great starting point as they offer some early indication of how robust the agent is,” Heinecke said. “But I believe the most important evaluation is your domain-specific evaluation and developing evaluation data that reflects the environment in which the agent will be operating.”