top of page
Search

Benchmarking Multi-Agent Frameworks: 5 Libraries, 11 LLMs, 7 Data Platforms

  • mahdinaser
  • May 13
  • 3 min read

When you build a production AI agent today, you face an immediate question: which framework? LangChain. LangGraph. AutoGen. FastAgency. MCP-based tool calling. Each has a public reputation, a Twitter cult, and very little rigorous benchmark data on which one actually performs better at concrete decision-making tasks.

I built a framework to find out empirically. The full codebase is open source: multi-agent-platform-evaluator. It systematically benchmarks 5 agent frameworks across 11 LLMs on a concrete decision task: choosing the right data-processing backend (Pandas, DuckDB, Polars, SQLite, FAISS, Annoy) for a given workload.

This post walks through the design, the libraries, what the framework measures, and the questions it lets you answer.

The Decision Task

Most agent benchmarks over-rotate on toy problems — math word problems, web navigation, multi-hop trivia. These don't predict production behavior. I picked a task I see often in real data systems: given a workload, pick the right backend.

A workload has characteristics: row count, query complexity, cardinality, presence of joins, expected concurrency. A platform has strengths: Polars is fast on single-machine analytics but breaks past memory; FAISS is excellent for vector search but useless for SQL; DuckDB occupies a sweet spot for analytical SQL on disk; SQLite is fine for low-concurrency OLTP.

The agent's job: take workload metadata, optionally call tools (performance history, platform specifications, comparison data), and recommend a platform.

The 5 Frameworks Tested

1. MCP (Model Context Protocol)

Anthropic's standard for tool-augmented LLMs. In this benchmark, MCP is the tool-access layer for a single LLM agent — not a multi-agent orchestrator. It maps cleanly to "single agent, structured tool access, no orchestration overhead."

agent = MCPAgent(
    llm=llm,
    tools=[performance_history_tool, platform_comparison_tool, platform_spec_tool],
)
decision = agent.decide(workload_metadata)

2. LangChain

The grandparent of LLM frameworks. The benchmark uses the standard LangChain function-calling agent.

from langchain.agents import initialize_agent, AgentType

agent = initialize_agent(
    tools=tools,
    llm=llm,
    agent=AgentType.OPENAI_FUNCTIONS,
    verbose=False,
)

3. LangGraph

State-machine framework built on LangChain. Designed for stateful, multi-step decisions where the agent may need to revisit earlier conclusions when new evidence arrives.

from langgraph.graph import StateGraph

graph = StateGraph(AgentState)
graph.add_node("plan", plan_step)
graph.add_node("consult_history", history_lookup)
graph.add_node("decide", decision_step)
graph.add_edge("plan", "consult_history")
graph.add_edge("consult_history", "decide")
app = graph.compile()

4. FastAgency

Lighter-weight orchestration framework. Faster to prototype than AutoGen, less expressive than LangGraph.

5. AutoGen

Microsoft's multi-agent collaboration framework. Multiple specialist agents converse to reach a decision.

from autogen import AssistantAgent, UserProxyAgent

specialist = AssistantAgent("data_specialist", llm_config=llm_config)
validator = AssistantAgent("validator", llm_config=llm_config)
user_proxy = UserProxyAgent("user_proxy")

user_proxy.initiate_chat(specialist, message=f"Decide platform for workload: {workload_metadata}")

The 11 LLMs

The framework runs against models spanning 2B to 70B parameters from the Llama, Qwen, DeepSeek, Mistral, Phi, and Gemma families, served via Ollama for local-only runs so cost doesn't bias results. This breadth lets you cleanly separate "framework effect" from "model effect" — something most agent comparisons fail to do.

What the Framework Measures

  • Regret: performance gap between the agent's choice and the oracle-optimal platform. Primary quality signal.

  • Token cost per decision: input + output tokens across the entire reasoning chain. Multi-agent systems pay 3-5x.

  • Latency per decision: wall-clock from workload-in to platform-out.

  • Decision stability: variance across multiple runs of the same input. Multi-agent systems with conversational drift score worse.

  • Tool-call rate: how often the agent actually invokes tools when available.

The metric most agent benchmarks ignore is decision stability. A multi-agent system that gives different answers to the same input on different runs is a real problem nobody talks about — but it bites you hard in production when the same query routes to different backends across retries.

Reproducing

git clone https://github.com/mahdinaser/multi-agent-platform-evaluator.git
cd multi-agent-platform-evaluator
pip install -r requirements.txt
pip install langchain langchain-community langgraph pyautogen
python app.py

Configuration lives in config/config.yaml. quick_mode: true runs in ~10 minutes; the full benchmark takes 4-8 hours depending on hardware. Results land in experiments/ with statistical summaries.

Why This Matters

If you've been wondering which agent framework to adopt for a production system, the meta-recommendation that emerges from running benchmarks like this is:

Start with the simplest framework that solves your problem. Invest in better tools rather than more agents. Graduate to LangGraph when your decision genuinely needs state, and to AutoGen only when deliberative reasoning materially outperforms structured tool use.

The hype cycle around multi-agent has compressed the perception of these frameworks into "more is better." The data suggests the opposite is often true.

 
 
 

Comments

bottom of page