#crawler

Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM clients.

One API for Search, Crawling, and Sitemaps

Web data toolkit: scrape_url (clean Markdown, optional real-browser JS rendering), crawl (multi-page BFS), extract (tables/links/headings/CSS selectors as JSON), site_map (sitemap discovery). China-hosted.

Web scraping for AI agents. One smart tool fetches any public page and picks the method automatically (direct request, rotating proxy, or full headless browser), getting past anti-bot challenges. Four tools + six prompts, one API key.

Open-source self-hosted technical SEO audit MCP server built on the LibreCrawl engine. Runs full audits inside Claude, Cursor, or Codex — 37 tools, 50+ checks per audit (hreflang, schema.org Rich Results, security headers, image performance, WAF detection on 200-OK pages), chunked-progressive crawler so big sites don't time out, branded PDF + 7 sidecar CSVs. Ephemeral by default — server wipes session, artifacts, and upstream record after every download. MIT.

Web data API that just works. Pass any URL, get clean markdown — Cloudflare, JavaScript, CAPTCHAs handled automatically. No config needed. 97.6% success rate across 500+ domains. Free tier included.

AI-powered website crawling, analysis, and export via MCP. 38 tools for crawl control, browser enrichment capture, WARC/ZIP/single-HTML export, observation timeline, and evidence-backed findings. 4 resources, 4 prompts. Dual-mode: full (38 tools) or code (6 tools). Install: npx crawlio-mcp

Fully decentralized P2P web search engine for LLMs. No API key, no billing, unlimited free search via MCP. Crawls, indexes, and searches the web through a peer-to-peer network. Features offline local search, credit-based incentives, privacy-first design, and 5 MCP tools: search, search_local, fetch_page, crawl_url, network_stats.

Give your AI agent unblockable browser access. Aluvia routes traffic through premium mobile carrier IPs to bypass 403s, CAPTCHAs, and WAFs automatically.

Stop writing selectors. Start describing data. UniCrawler ships an MCP (Model Context Protocol) server that exposes UniCrawler’s crawling/parsing/storage capabilities to MCP-capable clients.

MCP Server for Driflyte (https://console.driflyte.com). The Driflyte MCP Server exposes tools that allow AI assistants to query and retrieve topic-specific knowledge from recursively crawled and indexed web pages. With this MCP server, Driflyte acts as a bridge between diverse, topic-aware content sources (web, GitHub, and more) and AI-powered reasoning, enabling richer, more accurate answers.

Turn any website into LLM-ready, and provide SERP

高等级的免费爬虫MCP, 完全 Markdown 化支持，支持微信公众号等一大批国内网站。A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format. This tool is essential for building knowledge bases, training data collection, and content aggregation for LLM applications. It includes built-in support for the Model Context Protocol (MCP) for enhanced context management.

A powerful MCP server extension providing web search and content extraction capabilities. Integrates DuckDuckGo search functionality and URL content extraction into your MCP environment, enabling AI assistants to search the web and extract webpage content programmatically.

A lightweight MCP server for web searching via DuckDuckGo and extracting page content as Markdown.

A powerful MCP server extension providing web search and content extraction capabilities. Integrates DuckDuckGo search functionality and URL content extraction into your MCP environment, enabling AI assistants to search the web and extract webpage content programmatically.

A powerful Model Context Protocol (MCP) server that intelligently fetches and processes web content with nested URL crawling capabilities. Transform any documentation site or web resource into clean, structured markdown files perfect for AI consumption and analysis.

crawler-mcp-server

A powerful Model Context Protocol (MCP) server implementation with integrations for GitLab, Jira, Confluence, YouTube, and more. This server provides AI-powered search capabilities and various utility tools for development workflows.

mcp server for craw the interview questions in discussion

This project provides a toolset to crawl websites wikis, tool/library documentions and generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.

MCP (Model Context Protocol) server wrapping Crawl4AI library as functions using Python

Bridge the gap between your web crawler and AI language models using Model Context Protocol (MCP). With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously, extracting insights from your web content. Support for WARC, wget, InterroBot, Katana, and SiteOne crawlers is available out of the gate. The server includes a full-text search interface with boolean support, resource filtering by type, HTTP status, and more.

MCP server for podcast discovery through web crawling

Web Content Retrieval (full webpage, filtered content, or Markdown-converted), Custom User-Agent, Multi-HTTP Method Support (GET/POST/PUT/DELETE/PATCH), LLM-Controlled Request Headers, LLM-Accessible Response Headers, and more.

An MCP Server that's also an MCP Client. Useful for letting Claude develop and test MCPs without needing to reset the application.

Model Context Protocol (MCP) Server for Graphlit Platform