#crawling

Web scraping, crawling, and data extraction for AI agents. 5 built-in tools: scrape (clean markdown from any URL), crawl (follow links across entire sites), map (discover all URLs on a domain), extract (structured JSON via schema), and search (web search with optional page scraping). 833ms average latency, single binary, 6 MB RAM. Self-host free or use managed cloud at fastcrw.com with 500 free credits. Firecrawl-compatible API. Works with Claude Desktop, Claude Code, Cursor, Windsurf, and any MCP-compatible client.

Rust MCP server that gives AI agents reliable web access. Bypasses Cloudflare and bot protection using Chrome-level TLS fingerprinting, no headless browser needed. 10 tools for scraping, crawling, search, structured extraction, summarization, and more. Output is clean markdown optimized for LLMs with 67% fewer tokens than raw HTML. 8 of 10 tools work locally without an API key.

Opengraph.io MCP Server — Instant Link Metadata Extraction An MCP server that connects clients to the Opengraph.io web API, fetching rich metadata (titles, descriptions, images, Open Graph tags, etc.) for any URL. It lets clients retrieve structured link preview data. Ideal for chat bots, UIs, and agents that need to unfurl links and display previews or analyze page metadata, using your Opengraph.io App ID. There is also a suite of image generation tools. There are both stdio & https transports available use the npx config for stdio or the url config for HTTPS

Stop writing selectors. Start describing data. UniCrawler ships an MCP (Model Context Protocol) server that exposes UniCrawler’s crawling/parsing/storage capabilities to MCP-capable clients.

Real-time web search and crawling capabilities for AI assistants through Model Context Protocol (MCP)

Olostep MCP Server provides web scraping, crawling, content extraction, and search capabilities through the Olostep API. It's the most reliable and cost-effective on the market. It allows you to extract clean markdown content from web pages (with JavaScript rendering support), discover and map website URLs sorted by relevance, and retrieve structured Google search results including organic results, knowledge graphs, and related questions. The server supports country-specific routing for geo-targeted content and configurable wait times for JavaScript-heavy websites.

Crawlbase Web MCP Server (Model Context Protocol) connects AI agents and large language models (LLMs) with real-time web data. Built on Crawlbase’s proven web scraping and crawling infrastructure, it handles JavaScript rendering, anti-bot protection, and web data extraction at scale — powering Claude, Cursor, and Windsurf. It delivers a reliable, production-ready pipeline of live, structured data for AI workflows and intelligent applications.

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

A Model Context Protocol (MCP) compliant server designed for comprehensive web research. It uses Tavily's Search and Crawl APIs to gather detailed information on a given topic, then structures this data in a format perfect for LLMs to create high-quality markdown documents.

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

A Model Context Protocol (MCP) compliant server designed for comprehensive web research. It uses Tavily's Search and Crawl APIs to gather detailed information on a given topic, then structures this data in a format perfect for LLMs to create high-quality markdown documents.

Discover, extract, and interact with the web - one interface powering automated access across the public internet.

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

MCP (Model Context Protocol) server wrapping Crawl4AI library as functions using Python

MCP server for podcast discovery through web crawling

Mirror of