> ## Documentation Index
> Fetch the complete documentation index at: https://langchain-zh.cn/llms.txt
> Use this file to discover all available pages before exploring further.

# 如何定义汇总评估器

某些指标只能在整体实验级别定义，而不是针对实验的各个运行。例如，您可能希望计算评估目标在整个数据集所有样本上的总体通过率或 F1 分数。这类指标被称为汇总评估器。

## 基础示例

这里，我们将计算 F1 分数，它是精确率和召回率的综合指标。

这类指标只能在整个实验的所有样本上计算，因此我们的评估器接收一个输出列表和一个参考输出列表。

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  def f1_score_summary_evaluator(outputs: list[dict], reference_outputs: list[dict]) -> dict:
      true_positives = 0
      false_positives = 0
      false_negatives = 0

      for output_dict, reference_output_dict in zip(outputs, reference_outputs):
          output = output_dict["class"]
          reference_output = reference_output_dict["class"]

          if output == "Toxic" and reference_output == "Toxic":
              true_positives += 1
          elif output == "Toxic" and reference_output == "Not toxic":
              false_positives += 1
          elif output == "Not toxic" and reference_output == "Toxic":
              false_negatives += 1

      if true_positives == 0:
          return {"key": "f1_score", "score": 0.0}

      precision = true_positives / (true_positives + false_positives)
      recall = true_positives / (true_positives + false_negatives)
      f1_score = 2 * (precision * recall) / (precision + recall)

      return {"key": "f1_score", "score": f1_score}
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  function f1ScoreSummaryEvaluator({ outputs, referenceOutputs }: {
      outputs: Record<string, any>[],
      referenceOutputs: Record<string, any>[]
  }) {
      let truePositives = 0;
      let falsePositives = 0;
      let falseNegatives = 0;

      for (let i = 0; i < outputs.length; i++) {
          const output = outputs[i]["class"];
          const referenceOutput = referenceOutputs[i]["class"];

          if (output === "Toxic" && referenceOutput === "Toxic") {
              truePositives += 1;
          } else if (output === "Toxic" && referenceOutput === "Not toxic") {
              falsePositives += 1;
          } else if (output === "Not toxic" && referenceOutput === "Toxic") {
              falseNegatives += 1;
          }
      }

      if (truePositives === 0) {
          return { key: "f1_score", score: 0.0 };
      }

      const precision = truePositives / (truePositives + falsePositives);
      const recall = truePositives / (truePositives + falseNegatives);
      const f1Score = 2 * (precision * recall) / (precision + recall);

      return { key: "f1_score", score: f1Score };
  }
  ```
</CodeGroup>

然后，您可以将此评估器传递给 `evaluate` 方法，如下所示：

<CodeGroup>
  ```python Python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  from langsmith import Client

  ls_client = Client()
  dataset = ls_client.clone_public_dataset(
      "https://smith.langchain.com/public/3d6831e6-1680-4c88-94df-618c8e01fc55/d"
  )

  def bad_classifier(inputs: dict) -> dict:
      return {"class": "Not toxic"}

  def correct(outputs: dict, reference_outputs: dict) -> bool:
      """行级正确性评估器。"""
      return outputs["class"] == reference_outputs["label"]

  results = ls_client.evaluate(
      bad_classified,
      data=dataset,
      evaluators=[correct],
      summary_evaluators=[pass_50],
  )
  ```

  ```typescript TypeScript theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
  import { Client } from "langsmith";
  import { evaluate } from "langsmith/evaluation";
  import type { EvaluationResult } from "langsmith/evaluation";

  const client = new Client();
  const datasetName = "Toxic queries";
  const dataset = await client.clonePublicDataset(
      "https://smith.langchain.com/public/3d6831e6-1680-4c88-94df-618c8e01fc55/d",
      { datasetName: datasetName }
  );

  function correct({ outputs, referenceOutputs }: {
      outputs: Record<string, any>,
      referenceOutputs?: Record<string, any>
  }): EvaluationResult {
      const score = outputs["class"] === referenceOutputs?.["label"];
      return { key: "correct", score };
  }

  function badClassifier(inputs: Record<string, any>): { class: string } {
      return { class: "Not toxic" };
  }

  await evaluate(badClassifier, {
      data: datasetName,
      evaluators: [correct],
      summaryEvaluators: [summaryEval],
      experimentPrefix: "Toxic Queries",
  });
  ```
</CodeGroup>

在 LangSmith UI 中，您将看到汇总评估器的分数以对应的键名显示。

<img src="https://mintcdn.com/hhh-8c10bf0c/BCyPqRNhtAjzdCmk/langsmith/images/summary-eval.png?fit=max&auto=format&n=BCyPqRNhtAjzdCmk&q=85&s=8a1af552d57ec5a867b24818edef6a1a" alt="summary_eval.png" width="1535" height="122" data-path="langsmith/images/summary-eval.png" />

## 汇总评估器参数

汇总评估器函数必须具有特定的参数名称。它们可以接受以下任意子集的参数：

* `inputs: list[dict]`：对应数据集中单个样本的输入列表。
* `outputs: list[dict]`：每个实验在给定输入上产生的字典输出列表。
* `reference_outputs/referenceOutputs: list[dict]`：与样本关联的参考输出列表（如果可用）。
* `runs: list[Run]`：两个实验在给定样本上生成的完整 [Run](/langsmith/run-data-format) 对象列表。如果您需要访问每个运行的中间步骤或元数据，请使用此参数。
* `examples: list[Example]`：所有数据集 [Example](/langsmith/example-data-format) 对象，包括样本输入、输出（如果可用）和元数据（如果可用）。

## 汇总评估器输出

汇总评估器应返回以下类型之一：

Python 和 JS/TS

* `dict`：形式为 `{"score": ..., "name": ...}` 的字典，允许您传递数值或布尔分数以及指标名称。

目前仅限 Python

* `int | float | bool`：这被解释为可以求平均值、排序等的连续指标。函数名称用作指标的名称。

***

<div className="source-links">
  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/i18n\zh-CN\langsmith\summary.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>

  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>
</div>