> ## Documentation Index
> Fetch the complete documentation index at: https://langchain-zh.cn/llms.txt
> Use this file to discover all available pages before exploring further.

# 如何使用轨迹评估来评估你的智能体

许多智能体行为只有在使用真实的LLM时才会显现，例如智能体决定调用哪个工具、如何格式化响应，或者提示修改是否会影响整个执行轨迹。LangChain的 [`agentevals`](https://github.com/langchain-ai/agentevals) 包提供了专门设计用于使用实时模型测试智能体轨迹的评估器。

<Note>
  本指南涵盖开源的 [LangChain](/oss/python/langchain/overview) `agentevals` 包，该包与 LangSmith 集成以进行轨迹评估。
</Note>

AgentEvals 允许你通过执行 *轨迹匹配* 或使用 *LLM 评判器* 来评估智能体的轨迹（确切的消息序列，包括工具调用）：

<Card title="轨迹匹配" icon="equal" arrow="true" href="#trajectory-match-evaluator">
  为给定输入硬编码一个参考轨迹，并通过逐步比较来验证运行。

  适用于测试明确定义的工作流，其中你了解预期行为。当对应该调用哪些工具以及调用顺序有特定期望时使用。这种方法具有确定性、快速且成本效益高，因为它不需要额外的 LLM 调用。
</Card>

<Card title="LLM 作为评判器" icon="hammer" arrow="true" href="#llm-as-judge-evaluator">
  使用 LLM 来定性验证智能体的执行轨迹。"评判器" LLM 根据提示标准（可以包含参考轨迹）来审查智能体的决策。

  更灵活，可以评估效率和适当性等细微方面，但需要 LLM 调用且确定性较低。当你想评估智能体轨迹的整体质量和合理性，而没有严格的工具调用或顺序要求时使用。
</Card>

## 安装 AgentEvals

<CodeGroup>
  ```bash Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  pip install agentevals
  ```

  ```bash TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  npm install agentevals @langchain/core
  ```
</CodeGroup>

或者，直接克隆 [AgentEvals 仓库](https://github.com/langchain-ai/agentevals)。

## 轨迹匹配评估器

AgentEvals 在 Python 中提供 `create_trajectory_match_evaluator` 函数，在 TypeScript 中提供 `createTrajectoryMatchEvaluator` 函数，用于将智能体的轨迹与参考轨迹进行匹配。

你可以使用以下模式：

| 模式                                       | 描述                    | 使用场景                 |
| ---------------------------------------- | --------------------- | -------------------- |
| [`strict`](#strict-match)                | 消息和工具调用顺序完全匹配         | 测试特定序列（例如，授权前进行策略查找） |
| [`unordered`](#unordered-match)          | 允许相同的工具调用以任意顺序出现      | 当顺序不重要时验证信息检索        |
| [`subset`](#subset-and-superset-match)   | 智能体仅调用参考中的工具（无额外调用）   | 确保智能体不超过预期范围         |
| [`superset`](#subset-and-superset-match) | 智能体至少调用参考中的工具（允许额外调用） | 验证至少执行了所需的最小操作       |

### 严格匹配

`strict` 模式确保轨迹包含完全相同的消息顺序和相同的工具调用，但允许消息内容存在差异。当你需要强制执行特定的操作序列时，这很有用，例如要求在授权操作之前进行策略查找。

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  from langchain.agents import create_agent
  from langchain.tools import tool
  from langchain.messages import HumanMessage, AIMessage, ToolMessage
  from agentevals.trajectory.match import create_trajectory_match_evaluator


  @tool
  def get_weather(city: str):
      """获取城市的天气信息。"""
      return f"It's 75 degrees and sunny in {city}."

  agent = create_agent("gpt-4.1", tools=[get_weather])

  evaluator = create_trajectory_match_evaluator(  # [!code highlight]
      trajectory_match_mode="strict",  # [!code highlight]
  )  # [!code highlight]

  def test_weather_tool_called_strict():
      result = agent.invoke({
          "messages": [HumanMessage(content="What's the weather in San Francisco?")]
      })

      reference_trajectory = [
          HumanMessage(content="What's the weather in San Francisco?"),
          AIMessage(content="", tool_calls=[
              {"id": "call_1", "name": "get_weather", "args": {"city": "San Francisco"}}
          ]),
          ToolMessage(content="It's 75 degrees and sunny in San Francisco.", tool_call_id="call_1"),
          AIMessage(content="The weather in San Francisco is 75 degrees and sunny."),
      ]

      evaluation = evaluator(
          outputs=result["messages"],
          reference_outputs=reference_trajectory
      )
      # {
      #     'key': 'trajectory_strict_match',
      #     'score': True,
      #     'comment': None,
      # }
      assert evaluation["score"] is True
  ```

  ```ts TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  import { createAgent, tool, HumanMessage, AIMessage, ToolMessage } from "langchain"
  import { createTrajectoryMatchEvaluator } from "agentevals";
  import * as z from "zod";

  const getWeather = tool(
    async ({ city }: { city: string }) => {
      return `It's 75 degrees and sunny in ${city}.`;
    },
    {
      name: "get_weather",
      description: "Get weather information for a city.",
      schema: z.object({
        city: z.string(),
      }),
    }
  );

  const agent = createAgent({
    model: "gpt-4.1",
    tools: [getWeather]
  });

  const evaluator = createTrajectoryMatchEvaluator({  // [!code highlight]
    trajectoryMatchMode: "strict",  // [!code highlight]
  });  // [!code highlight]

  async function testWeatherToolCalledStrict() {
    const result = await agent.invoke({
      messages: [new HumanMessage("What's the weather in San Francisco?")]
    });

    const referenceTrajectory = [
      new HumanMessage("What's the weather in San Francisco?"),
      new AIMessage({
        content: "",
        tool_calls: [
          { id: "call_1", name: "get_weather", args: { city: "San Francisco" } }
        ]
      }),
      new ToolMessage({
        content: "It's 75 degrees and sunny in San Francisco.",
        tool_call_id: "call_1"
      }),
      new AIMessage("The weather in San Francisco is 75 degrees and sunny."),
    ];

    const evaluation = await evaluator({
      outputs: result.messages,
      referenceOutputs: referenceTrajectory
    });
    // {
    //     'key': 'trajectory_strict_match',
    //     'score': true,
    //     'comment': null,
    // }
    expect(evaluation.score).toBe(true);
  }
  ```
</CodeGroup>

### 无序匹配

`unordered` 模式允许相同的工具调用以任意顺序出现，当你想要验证正确的工具集被调用但不关心顺序时，这很有帮助。例如，智能体可能需要检查城市的天气和活动，但顺序无关紧要。

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  from langchain.agents import create_agent
  from langchain.tools import tool
  from langchain.messages import HumanMessage, AIMessage, ToolMessage
  from agentevals.trajectory.match import create_trajectory_match_evaluator


  @tool
  def get_weather(city: str):
      """获取城市的天气信息。"""
      return f"It's 75 degrees and sunny in {city}."

  @tool
  def get_events(city: str):
      """获取城市中正在发生的事件。"""
      return f"Concert at the park in {city} tonight."

  agent = create_agent("gpt-4.1", tools=[get_weather, get_events])

  evaluator = create_trajectory_match_evaluator(  # [!code highlight]
      trajectory_match_mode="unordered",  # [!code highlight]
  )  # [!code highlight]

  def test_multiple_tools_any_order():
      result = agent.invoke({
          "messages": [HumanMessage(content="What's happening in SF today?")]
      })

      # 参考显示工具调用顺序与实际执行不同
      reference_trajectory = [
          HumanMessage(content="What's happening in SF today?"),
          AIMessage(content="", tool_calls=[
              {"id": "call_1", "name": "get_events", "args": {"city": "SF"}},
              {"id": "call_2", "name": "get_weather", "args": {"city": "SF"}},
          ]),
          ToolMessage(content="Concert at the park in SF tonight.", tool_call_id="call_1"),
          ToolMessage(content="It's 75 degrees and sunny in SF.", tool_call_id="call_2"),
          AIMessage(content="Today in SF: 75 degrees and sunny with a concert at the park tonight."),
      ]

      evaluation = evaluator(
          outputs=result["messages"],
          reference_outputs=reference_trajectory,
      )
      # {
      #     'key': 'trajectory_unordered_match',
      #     'score': True,
      # }
      assert evaluation["score"] is True
  ```

  ```ts TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  import { createAgent, tool, HumanMessage, AIMessage, ToolMessage } from "langchain"
  import { createTrajectoryMatchEvaluator } from "agentevals";
  import * as z from "zod";

  const getWeather = tool(
    async ({ city }: { city: string }) => {
      return `It's 75 degrees and sunny in ${city}.`;
    },
    {
      name: "get_weather",
      description: "Get weather information for a city.",
      schema: z.object({ city: z.string() }),
    }
  );

  const getEvents = tool(
    async ({ city }: { city: string }) => {
      return `Concert at the park in ${city} tonight.`;
    },
    {
      name: "get_events",
      description: "Get events happening in a city.",
      schema: z.object({ city: z.string() }),
    }
  );

  const agent = createAgent({
    model: "gpt-4.1",
    tools: [getWeather, getEvents]
  });

  const evaluator = createTrajectoryMatchEvaluator({  // [!code highlight]
    trajectoryMatchMode: "unordered",  // [!code highlight]
  });  // [!code highlight]

  async function testMultipleToolsAnyOrder() {
    const result = await agent.invoke({
      messages: [new HumanMessage("What's happening in SF today?")]
    });

    // 参考显示工具调用顺序与实际执行不同
    const referenceTrajectory = [
      new HumanMessage("What's happening in SF today?"),
      new AIMessage({
        content: "",
        tool_calls: [
          { id: "call_1", name: "get_events", args: { city: "SF" } },
          { id: "call_2", name: "get_weather", args: { city: "SF" } },
        ]
      }),
      new ToolMessage({
        content: "Concert at the park in SF tonight.",
        tool_call_id: "call_1"
      }),
      new ToolMessage({
        content: "It's 75 degrees and sunny in SF.",
        tool_call_id: "call_2"
      }),
      new AIMessage("Today in SF: 75 degrees and sunny with a concert at the park tonight."),
    ];

    const evaluation = await evaluator({
      outputs: result.messages,
      referenceOutputs: referenceTrajectory,
    });
    // {
    //     'key': 'trajectory_unordered_match',
    //     'score': true,
    // }
    expect(evaluation.score).toBe(true);
  }
  ```
</CodeGroup>

### 子集和超集匹配

`superset` 和 `subset` 模式关注的是调用了哪些工具，而不是工具调用的顺序，允许你控制智能体的工具调用与参考轨迹的对齐严格程度。

* 使用 `superset` 模式时，你想验证执行中调用了几个关键工具，但允许智能体调用额外的工具。智能体的轨迹必须至少包含参考轨迹中的所有工具调用，并且可以包含超出参考的额外工具调用。
* 使用 `subset` 模式来确保智能体的效率，验证智能体没有调用任何超出参考轨迹的不相关或不必要的工具。智能体的轨迹必须仅包含出现在参考轨迹中的工具调用。

以下示例演示了 `superset` 模式，其中参考轨迹仅要求 `get_weather` 工具，但智能体可以调用额外的工具：

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  from langchain.agents import create_agent
  from langchain.tools import tool
  from langchain.messages import HumanMessage, AIMessage, ToolMessage
  from agentevals.trajectory.match import create_trajectory_match_evaluator


  @tool
  def get_weather(city: str):
      """获取城市的天气信息。"""
      return f"It's 75 degrees and sunny in {city}."

  @tool
  def get_detailed_forecast(city: str):
      """获取城市的详细天气预报。"""
      return f"Detailed forecast for {city}: sunny all week."

  agent = create_agent("gpt-4.1", tools=[get_weather, get_detailed_forecast])

  evaluator = create_trajectory_match_evaluator(  # [!code highlight]
      trajectory_match_mode="superset",  # [!code highlight]
  )  # [!code highlight]

  def test_agent_calls_required_tools_plus_extra():
      result = agent.invoke({
          "messages": [HumanMessage(content="What's the weather in Boston?")]
      })

      # 参考仅要求 get_weather，但智能体可以调用额外的工具
      reference_trajectory = [
          HumanMessage(content="What's the weather in Boston?"),
          AIMessage(content="", tool_calls=[
              {"id": "call_1", "name": "get_weather", "args": {"city": "Boston"}},
          ]),
          ToolMessage(content="It's 75 degrees and sunny in Boston.", tool_call_id="call_1"),
          AIMessage(content="The weather in Boston is 75 degrees and sunny."),
      ]

      evaluation = evaluator(
          outputs=result["messages"],
          reference_outputs=reference_trajectory,
      )
      # {
      #     'key': 'trajectory_superset_match',
      #     'score': True,
      #     'comment': None,
      # }
      assert evaluation["score"] is True
  ```

  ```ts TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  import { createAgent } from "langchain"
  import { tool } from "@langchain/core/tools";
  import { HumanMessage, AIMessage, ToolMessage } from "@langchain/core/messages";
  import { createTrajectoryMatchEvaluator } from "agentevals";
  import * as z from "zod";

  const getWeather = tool(
    async ({ city }: { city: string }) => {
      return `It's 75 degrees and sunny in ${city}.`;
    },
    {
      name: "get_weather",
      description: "Get weather information for a city.",
      schema: z.object({ city: z.string() }),
    }
  );

  const getDetailedForecast = tool(
    async ({ city }: { city: string }) => {
      return `Detailed forecast for ${city}: sunny all week.`;
    },
    {
      name: "get_detailed_forecast",
      description: "Get detailed weather forecast for a city.",
      schema: z.object({ city: z.string() }),
    }
  );

  const agent = createAgent({
    model: "gpt-4.1",
    tools: [getWeather, getDetailedForecast]
  });

  const evaluator = createTrajectoryMatchEvaluator({  // [!code highlight]
    trajectoryMatchMode: "superset",  // [!code highlight]
  });  // [!code highlight]

  async function testAgentCallsRequiredToolsPlusExtra() {
    const result = await agent.invoke({
      messages: [new HumanMessage("What's the weather in Boston?")]
    });

    // 参考仅要求 getWeather，但智能体可以调用额外的工具
    const referenceTrajectory = [
      new HumanMessage("What's the weather in Boston?"),
      new AIMessage({
        content: "",
        tool_calls: [
          { id: "call_1", name: "get_weather", args: { city: "Boston" } },
        ]
      }),
      new ToolMessage({
        content: "It's 75 degrees and sunny in Boston.",
        tool_call_id: "call_1"
      }),
      new AIMessage("The weather in Boston is 75 degrees and sunny."),
    ];

    const evaluation = await evaluator({
      outputs: result.messages,
      referenceOutputs: referenceTrajectory,
    });
    // {
    //     'key': 'trajectory_superset_match',
    //     'score': true,
    //     'comment': null,
    // }
    expect(evaluation.score).toBe(true);
  }
  ```
</CodeGroup>

<Info>
  你还可以通过设置 `tool_args_match_mode`（Python）或 `toolArgsMatchMode`（TypeScript）属性以及 `tool_args_match_overrides`（Python）或 `toolArgsMatchOverrides`（TypeScript）属性，来自定义评估器如何考虑实际轨迹与参考轨迹之间工具调用的相等性。默认情况下，只有具有相同参数且调用相同工具的工具调用才被视为相等。访问[仓库](https://github.com/langchain-ai/agentevals?tab=readme-ov-file#tool-args-match-modes)了解更多详情。
</Info>

## LLM 作为评判器评估器

<Note>
  本节涵盖来自 `agentevals` 包的特定于轨迹的 LLM 作为评判器评估器。有关 LangSmith 中通用的 LLM 作为评判器评估器，请参考 [LLM 作为评判器评估器](/langsmith/llm-as-judge)。
</Note>

你也可以使用 LLM 来评估智能体的执行路径。与轨迹匹配评估器不同，它不需要参考轨迹，但如果可用，也可以提供。

### 无参考轨迹

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  from langchain.agents import create_agent
  from langchain.tools import tool
  from langchain.messages import HumanMessage, AIMessage, ToolMessage
  from agentevals.trajectory.llm import create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT


  @tool
  def get_weather(city: str):
      """获取城市的天气信息。"""
      return f"It's 75 degrees and sunny in {city}."

  agent = create_agent("gpt-4.1", tools=[get_weather])

  evaluator = create_trajectory_llm_as_judge(  # [!code highlight]
      model="openai:o3-mini",  # [!code highlight]
      prompt=TRAJECTORY_ACCURACY_PROMPT,  # [!code highlight]
  )  # [!code highlight]

  def test_trajectory_quality():
      result = agent.invoke({
          "messages": [HumanMessage(content="What's the weather in Seattle?")]
      })

      evaluation = evaluator(
          outputs=result["messages"],
      )
      # {
      #     'key': 'trajectory_accuracy',
      #     'score': True,
      #     'comment': 'The provided agent trajectory is reasonable...'
      # }
      assert evaluation["score"] is True
  ```

  ```ts TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  import { createAgent } from "langchain"
  import { tool } from "@langchain/core/tools";
  import { HumanMessage, AIMessage, ToolMessage } from "@langchain/core/messages";
  import { createTrajectoryLLMAsJudge, TRAJECTORY_ACCURACY_PROMPT } from "agentevals";
  import * as z from "zod";

  const getWeather = tool(
    async ({ city }: { city: string }) => {
      return `It's 75 degrees and sunny in ${city}.`;
    },
    {
      name: "get_weather",
      description: "Get weather information for a city.",
      schema: z.object({ city: z.string() }),
    }
  );

  const agent = createAgent({
    model: "gpt-4.1",
    tools: [getWeather]
  });

  const evaluator = createTrajectoryLLMAsJudge({  // [!code highlight]
    model: "openai:o3-mini",  // [!code highlight]
    prompt: TRAJECTORY_ACCURACY_PROMPT,  // [!code highlight]
  });  // [!code highlight]

  async function testTrajectoryQuality() {
    const result = await agent.invoke({
      messages: [new HumanMessage("What's the weather in Seattle?")]
    });

    const evaluation = await evaluator({
      outputs: result.messages,
    });
    // {
    //     'key': 'trajectory_accuracy',
    //     'score': true,
    //     'comment': 'The provided agent trajectory is reasonable...'
    // }
    expect(evaluation.score).toBe(true);
  }
  ```
</CodeGroup>

### 有参考轨迹

如果你有参考轨迹，可以向提示中添加额外的变量并传入参考轨迹。下面，我们使用预构建的 `TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE` 提示，并配置 `reference_outputs` 变量：

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  evaluator = create_trajectory_llm_as_judge(
      model="openai:o3-mini",
      prompt=TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,
  )
  evaluation = evaluator(
      outputs=result["messages"],
      reference_outputs=reference_trajectory,
  )
  ```

  ```ts TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  import { TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE } from "agentevals";

  const evaluator = createTrajectoryLLMAsJudge({
    model: "openai:o3-mini",
    prompt: TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,
  });

  const evaluation = await evaluator({
    outputs: result.messages,
    referenceOutputs: referenceTrajectory,
  });
  ```
</CodeGroup>

<Info>
  要获得关于 LLM 如何评估轨迹的更多可配置性，请访问[仓库](https://github.com/langchain-ai/agentevals?tab=readme-ov-file#trajectory-llm-as-judge)。
</Info>

## 异步支持（Python）

所有 `agentevals` 评估器都支持 Python asyncio。对于使用工厂函数的评估器，可以通过在函数名中的 `create_` 后添加 `async` 来获得异步版本。

以下是使用异步评判器和评估器的示例：

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from agentevals.trajectory.llm import create_async_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT
from agentevals.trajectory.match import create_async_trajectory_match_evaluator

async_judge = create_async_trajectory_llm_as_judge(
    model="openai:o3-mini",
    prompt=TRAJECTORY_ACCURACY_PROMPT,
)

async_evaluator = create_async_trajectory_match_evaluator(
    trajectory_match_mode="strict",
)

async def test_async_evaluation():
    result = await agent.ainvoke({
        "messages": [HumanMessage(content="What's the weather?")]
    })

    evaluation = await async_judge(outputs=result["messages"])
    assert evaluation["score"] is True
```

***

<div className="source-links">
  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/i18n\zh-CN\langsmith\trajectory-evals.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>

  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>
</div>