> ## Documentation Index
> Fetch the complete documentation index at: https://langchain-zh.cn/llms.txt
> Use this file to discover all available pages before exploring further.

# Hyperbrowser 网页抓取集成

> 使用 LangChain Python 与 Hyperbrowser 网页抓取工具集成。

[Hyperbrowser](https://hyperbrowser.ai) 是一个用于运行和扩展无头浏览器的平台。它允许您大规模启动和管理浏览器会话，并提供易于使用的解决方案以满足任何网页抓取需求，例如抓取单个页面或爬取整个站点。

主要功能：

* 即时扩展能力 - 无需基础设施烦恼，数秒内启动数百个浏览器会话
* 简单集成 - 与 Puppeteer 和 Playwright 等流行工具无缝协作
* 强大的 API - 易于使用的 API 用于抓取/爬取任何网站，以及更多功能
* 绕过反机器人措施 - 内置隐身模式、广告拦截、自动验证码解决和代理轮换

本指南提供了开始使用 Hyperbrowser 网页工具的快速概述。

有关 Hyperbrowser 的更多信息，请访问 [Hyperbrowser 网站](https://hyperbrowser.ai)，或者如果您想查看文档，可以访问 [Hyperbrowser 文档](https://docs.hyperbrowser.ai)。

## 核心功能

### 抓取

Hyperbrowser 提供强大的抓取功能，允许您从任何网页提取数据。该抓取工具可以将网页内容转换为结构化格式（如 markdown 或 HTML），使处理和数据分析变得容易。

### 爬取

爬取功能使您能够自动导航网站的多个页面。您可以设置参数（如页面限制）来控制爬虫探索站点的程度，收集其访问的每个页面的数据。

### 提取

Hyperbrowser 的提取功能使用 AI 根据您的定义模式从网页中提取特定信息。这允许您将非结构化的网页内容转换为您确切要求的结构化数据。

## 概述

### 集成详情

| 工具   | 包                      |  本地 | 可序列化 | JS 支持 |
| :--- | :--------------------- | :-: | :--: | :---: |
| 爬取工具 | langchain-hyperbrowser |  ❌  |   ❌  |   ❌   |
| 抓取工具 | langchain-hyperbrowser |  ❌  |   ❌  |   ❌   |
| 提取工具 | langchain-hyperbrowser |  ❌  |   ❌  |   ❌   |

## 设置

要访问 Hyperbrowser 网页工具，您需要安装 `langchain-hyperbrowser` 集成包，并创建 Hyperbrowser 账户获取 API 密钥。

### 凭据

前往 [Hyperbrowser](https://app.hyperbrowser.ai/) 注册并生成 API 密钥。完成后，设置 `HYPERBROWSER_API_KEY` 环境变量：

```bash theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
export HYPERBROWSER_API_KEY=<your-api-key>
```

### 安装

安装 **langchain-hyperbrowser**。

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
pip install -qU langchain-hyperbrowser
```

## 实例化

### 爬取工具

`HyperbrowserCrawlTool` 是一个强大的工具，可以从给定 URL 开始爬取整个网站。它支持可配置的页面限制和抓取选项。

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_hyperbrowser import HyperbrowserCrawlTool
tool = HyperbrowserCrawlTool()
```

### 抓取工具

`HyperbrowserScrapeTool` 是一个可以从网页抓取内容的工具。它支持 markdown 和 HTML 输出格式，以及元数据提取。

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_hyperbrowser import HyperbrowserScrapeTool
tool = HyperbrowserScrapeTool()
```

### 提取工具

`HyperbrowserExtractTool` 是一个强大的工具，使用 AI 从网页提取结构化数据。它可以根据预定义的模式提取信息。

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_hyperbrowser import HyperbrowserExtractTool
tool = HyperbrowserExtractTool()
```

## 调用

### 基本用法

#### 爬取工具

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_hyperbrowser import HyperbrowserCrawlTool

result = HyperbrowserCrawlTool().invoke(
    {
        "url": "https://example.com",
        "max_pages": 2,
        "scrape_options": {"formats": ["markdown"]},
    }
)
print(result)
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
{'data': [CrawledPage(metadata={'url': 'https://www.example.com/', 'title': 'Example Domain', 'viewport': 'width=device-width, initial-scale=1', 'sourceURL': 'https://example.com'}, html=None, markdown='Example Domain\n\n# Example Domain\n\nThis domain is for use in illustrative examples in documents. You may use this\ndomain in literature without prior coordination or asking for permission.\n\n[More information...](https://www.iana.org/domains/example)', links=None, screenshot=None, url='https://example.com', status='completed', error=None)], 'error': None}
```

#### 抓取工具

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_hyperbrowser import HyperbrowserScrapeTool

result = HyperbrowserScrapeTool().invoke(
    {"url": "https://example.com", "scrape_options": {"formats": ["markdown"]}}
)
print(result)
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
{'data': ScrapeJobData(metadata={'url': 'https://www.example.com/', 'title': 'Example Domain', 'viewport': 'width=device-width, initial-scale=1', 'sourceURL': 'https://example.com'}, html=None, markdown='Example Domain\n\n# Example Domain\n\nThis domain is for use in illustrative examples in documents. You may use this\ndomain in literature without prior coordination or asking for permission.\n\n[More information...](https://www.iana.org/domains/example)', links=None, screenshot=None), 'error': None}
```

#### 提取工具

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_hyperbrowser import HyperbrowserExtractTool
from pydantic import BaseModel


class SimpleExtractionModel(BaseModel):
    title: str


result = HyperbrowserExtractTool().invoke(
    {
        "url": "https://example.com",
        "schema": SimpleExtractionModel,
    }
)
print(result)
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
{'data': {'title': 'Example Domain'}, 'error': None}
```

### 使用自定义选项

#### 带自定义选项的爬取工具

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
result = HyperbrowserCrawlTool().run(
    {
        "url": "https://example.com",
        "max_pages": 2,
        "scrape_options": {
            "formats": ["markdown", "html"],
        },
        "session_options": {"use_proxy": True, "solve_captchas": True},
    }
)
print(result)
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
{'data': [CrawledPage(metadata={'url': 'https://www.example.com/', 'title': 'Example Domain', 'viewport': 'width=device-width, initial-scale=1', 'sourceURL': 'https://example.com'}, html=None, markdown='Example Domain\n\n# Example Domain\n\nThis domain is for use in illustrative examples in documents. You may use this\ndomain in literature without prior coordination or asking for permission.\n\n[More information...](https://www.iana.org/domains/example)', links=None, screenshot=None, url='https://example.com', status='completed', error=None)], 'error': None}
```

#### 带自定义选项的抓取工具

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
result = HyperbrowserScrapeTool().run(
    {
        "url": "https://example.com",
        "scrape_options": {
            "formats": ["markdown", "html"],
        },
        "session_options": {"use_proxy": True, "solve_captchas": True},
    }
)
print(result)
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
{'data': ScrapeJobData(metadata={'url': 'https://www.example.com/', 'title': 'Example Domain', 'viewport': 'width=device-width, initial-scale=1', 'sourceURL': 'https://example.com'}, html='<html><head>\n    <title>Example Domain</title>\n\n    <meta charset="utf-8">\n    <meta http-equiv="Content-type" content="text/html; charset=utf-8">\n    <meta name="viewport" content="width=device-width, initial-scale=1">\n        \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href="https://www.iana.org/domains/example">More information...</a></p>\n</div>\n\n\n</body></html>', markdown='Example Domain\n\n# Example Domain\n\nThis domain is for use in illustrative examples in documents. You may use this\ndomain in literature without prior coordination or asking for permission.\n\n[More information...](https://www.iana.org/domains/example)', links=None, screenshot=None), 'error': None}
```

#### 带自定义模式的提取工具

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from typing import List

from pydantic import BaseModel


class ProductSchema(BaseModel):
    title: str
    price: float


class ProductsSchema(BaseModel):
    products: List[ProductSchema]


result = HyperbrowserExtractTool().run(
    {
        "url": "https://dummyjson.com/products?limit=10",
        "schema": ProductsSchema,
        "session_options": {"session_options": {"use_proxy": True}},
    }
)
print(result)
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
{'data': {'products': [{'price': 9.99, 'title': 'Essence Mascara Lash Princess'}, {'price': 19.99, 'title': 'Eyeshadow Palette with Mirror'}, {'price': 14.99, 'title': 'Powder Canister'}, {'price': 12.99, 'title': 'Red Lipstick'}, {'price': 8.99, 'title': 'Red Nail Polish'}, {'price': 49.99, 'title': 'Calvin Klein CK One'}, {'price': 129.99, 'title': 'Chanel Coco Noir Eau De'}, {'price': 89.99, 'title': "Dior J'adore"}, {'price': 69.99, 'title': 'Dolce Shine Eau de'}, {'price': 79.99, 'title': 'Gucci Bloom Eau de'}]}, 'error': None}
```

### 异步用法

所有工具都支持异步用法：

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from typing import List

from langchain_hyperbrowser import (
    HyperbrowserCrawlTool,
    HyperbrowserExtractTool,
    HyperbrowserScrapeTool,
)
from pydantic import BaseModel


class ExtractionSchema(BaseModel):
    popular_library_name: List[str]


async def web_operations():
    # Crawl
    crawl_tool = HyperbrowserCrawlTool()
    crawl_result = await crawl_tool.arun(
        {
            "url": "https://example.com",
            "max_pages": 5,
            "scrape_options": {"formats": ["markdown"]},
        }
    )

    # Scrape
    scrape_tool = HyperbrowserScrapeTool()
    scrape_result = await scrape_tool.arun(
        {"url": "https://example.com", "scrape_options": {"formats": ["markdown"]}}
    )

    # Extract
    extract_tool = HyperbrowserExtractTool()
    extract_result = await extract_tool.arun(
        {
            "url": "https://npmjs.com",
            "schema": ExtractionSchema,
        }
    )

    return crawl_result, scrape_result, extract_result


results = await web_operations()
print(results)
```

```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
---------------------------------------------------------------------------
```

```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
NameError                                 Traceback (most recent call last)
```

```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
Cell In[6], line 10
      1 from langchain_hyperbrowser import (
      2     HyperbrowserCrawlTool,
      3     HyperbrowserExtractTool,
      4     HyperbrowserScrapeTool,
      5 )
      7 from pydantic import BaseModel
---> 10 class ExtractionSchema(BaseModel):
     11     popular_library_name: List[str]
     14 async def web_operations():
     15     # Crawl
```

```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
Cell In[6], line 11, in ExtractionSchema()
     10 class ExtractionSchema(BaseModel):
---> 11     popular_library_name: List[str]
```

```text theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
NameError: name 'List' is not defined
```

## 在智能体中使用

以下是如何在智能体中使用任何网页工具的方法：

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
from langchain_hyperbrowser import HyperbrowserCrawlTool
from langchain_openai import ChatOpenAI
from langchain.agents import create_agent


# Initialize the crawl tool
crawl_tool = HyperbrowserCrawlTool()

# Create the agent with the crawl tool
model = ChatOpenAI(temperature=0)

agent = create_agent(model, [crawl_tool])
user_input = "Crawl https://example.com and get content from up to 5 pages"
for step in agent.stream(
    {"messages": user_input},
    stream_mode="values",
):
    step["messages"][-1].pretty_print()
```

```python theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
================================ Human Message =================================

Crawl https://example.com and get content from up to 5 pages
================================== Ai Message ==================================
Tool Calls:
  hyperbrowser_crawl_data (call_G2ofdHOqjdnJUZu4hhbuga58)
 Call ID: call_G2ofdHOqjdnJUZu4hhbuga58
  Args:
    url: https://example.com
    max_pages: 5
    scrape_options: {'formats': ['markdown']}
================================= Tool Message =================================
Name: hyperbrowser_crawl_data

{'data': [CrawledPage(metadata={'url': 'https://www.example.com/', 'title': 'Example Domain', 'viewport': 'width=device-width, initial-scale=1', 'sourceURL': 'https://example.com'}, html=None, markdown='Example Domain\n\n# Example Domain\n\nThis domain is for use in illustrative examples in documents. You may use this\ndomain in literature without prior coordination or asking for permission.\n\n[More information...](https://www.iana.org/domains/example)', links=None, screenshot=None, url='https://example.com', status='completed', error=None)], 'error': None}
================================== Ai Message ==================================

I have crawled the website [https://example.com](https://example.com) and retrieved content from the first page. Here is the content in markdown format:

\`\`\`
Example Domain

# Example Domain

This domain is for use in illustrative examples in documents. You may use this
domain in literature without prior coordination or asking for permission.

[More information...](https://www.iana.org/domains/example)
\`\`\`

If you would like to crawl more pages or need additional information, please let me know!
```

## 配置选项

### 通用选项

所有工具都支持这些基本配置选项：

* `url`: 要处理的 URL
* `session_options`: 浏览器会话配置
  * `use_proxy`: 是否使用代理
  * `solve_captchas`: 是否自动解决验证码
  * `accept_cookies`: 是否接受 Cookie

### 工具特定选项

#### 爬取工具

* `max_pages`: 要爬取的最大页数
* `scrape_options`: 抓取每个页面的选项
  * `formats`: 输出格式列表（markdown, html）

#### 抓取工具

* `scrape_options`: 抓取页面的选项
  * `formats`: 输出格式列表（markdown, html）

#### 提取工具

* `schema`: 定义要提取结构的 Pydantic 模型
* `extraction_prompt`: 用于提取的自然语言提示

更多详细信息，请参阅相应的 API 参考：

* [爬取 API 参考](https://docs.hyperbrowser.ai/reference/api-reference/crawl)
* [抓取 API 参考](https://docs.hyperbrowser.ai/reference/api-reference/scrape)
* [提取 API 参考](https://docs.hyperbrowser.ai/reference/api-reference/extract)

***

## API 参考

* [GitHub](https://github.com/hyperbrowserai/langchain-hyperbrowser/)
* [PyPI](https://pypi.org/project/langchain-hyperbrowser/)
* [Hyperbrowser 文档](https://docs.hyperbrowser.ai/)

***

<div className="source-links">
  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/i18n\zh-CN\oss\python\integrations\tools\hyperbrowser_web_scraping_tools.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>

  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>
</div>