用 Browser Use 自动采集网页数据

AI 浏览器自动化，处理复杂动态网站，提取结构化数据。

Market researchers, price monitoring, lead generation from directories, competitive analysis.

Python 3.10+, Browser Use, Playwright, OpenAI/Claude API key

工作流图

Install Browser Use

Define Collection Task

Add Login Handling

Data Validation

Export and Schedule

1
Install Browser Use

pip install browser-use. Set up Playwright browsers: playwright install chromium. Configure your LLM (OpenAI/Claude). Test with a simple navigation task to verify setup is correct.
2
Define Collection Task

Write a natural language instruction: Go to [website], navigate to [page type], for each [item], extract [fields: name, price, description, rating]. Handle pagination by clicking next page button. Stop after 50 items.
3
Add Login Handling

If the site requires login, provide credentials in the task: First, log in with username [USER] and password [PASS]. Then navigate to the data pages. Browser Use will handle the login form automatically.
4
Data Validation

Add a post-processing step that validates extracted data: check for missing fields, validate data types (price is number, rating is 1-5), deduplicate by unique identifier. Use Python pandas for data cleaning.
5
Export and Schedule

Save extracted data to CSV/JSON. For recurring collection, wrap in a cron job or n8n workflow. Add error handling: retry failed pages, log blocked requests, send alert if collection success rate < 80%.