AI-First Scraper
ai-first-scraper-mcp
Plug Claude Desktop, Cursor, or Cline straight into an ad-free web scraper + search engine. Three tools, one line of config.
What it does
Adds three tools to any MCP-compatible agent:
| Tool | What it does |
|---|---|
fetch_page | Fetch one URL → return clean Markdown (HTML or PDF). |
fetch_pages_batch | Fetch up to 25 URLs in parallel → return Markdown for each. |
search_web | Run a web search and return the top-k result pages already converted to Markdown. |
No more "the model called curl and then tried to parse 80kB of ad HTML." Your agent receives clean Markdown ready to reason about.
Backed by the ai-first-scraper and ai-first-search APIs.
Install
Fastest — uvx (no install, runs from PyPI on demand)
// claude_desktop_config.json / cline_mcp_settings.json / ~/.cursor/mcp.json
{
"mcpServers": {
"ai-first-scraper": {
"command": "uvx",
"args": ["ai-first-scraper-mcp"]
}
}
}
Restart your client (Claude Desktop / Cursor / Cline). The three tools above will appear automatically.
Alternative — pip install
pip install ai-first-scraper-mcp
{
"mcpServers": {
"ai-first-scraper": {
"command": "ai-first-scraper-mcp"
}
}
}
Where the config file lives
| Client | Config path |
|---|---|
| Claude Desktop (macOS) | ~/Library/Application Support/Claude/claude_desktop_config.json |
| Claude Desktop (Windows) | %APPDATA%\Claude\claude_desktop_config.json |
| Cursor | ~/.cursor/mcp.json |
| Cline (VS Code) | ~/Library/Application Support/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json |
Point at your own backend (optional)
By default this server calls the public ai-first-scraper.onrender.com and
ai-first-search.onrender.com instances. If you want to self-host, set env
vars in your MCP config:
{
"mcpServers": {
"ai-first-scraper": {
"command": "uvx",
"args": ["ai-first-scraper-mcp"],
"env": {
"SCRAPER_URL": "https://your-scraper.example.com",
"SEARCH_URL": "https://your-search.example.com",
"AFS_TIMEOUT": "60"
}
}
}
}
Verify it works
Open your MCP client and ask the agent:
"Use the search_web tool to find the top 3 recent articles about MCP and summarize them in 5 bullets each."
You should see the agent call search_web, get back Markdown for each result,
and produce the summary without ever touching raw HTML.
Companion projects
- ai-first-scraper — the per-URL Markdown cleaner this MCP server fans out to.
- ai-first-search — search → scrape → markdown pipeline.
- mcp-rec — record & replay any MCP server's traffic for tests and bug reports.
- llm-cache-proxy — local cache for OpenAI/Anthropic API calls.
- promptlocker — lockfile for prompts.
- context-diff — see what blew up your Claude Code context window.
- agentwatch — overlay for browser AI agents.
Develop locally
git clone https://github.com/yubinkim444/ai-first-scraper-mcp.git
cd ai-first-scraper-mcp
uv sync # or: pip install -e .
ai-first-scraper-mcp # speaks MCP over stdio
To test against a local client, point its MCP config at the same command.
License
MIT © yubinkim444
Server Config
{
"mcpServers": {
"ai-first-scraper": {
"command": "uvx",
"args": [
"ai-first-scraper-mcp"
]
}
}
}