mcp-server-webcrawl

Created By

pragmar7 months ago

MCP server tailored to connecting web crawler data and archives

Overview

What is mcp-server-webcrawl?

MCP Server Webcrawl is a tool designed to connect web crawler data and archives with AI language models using the Model Context Protocol (MCP). It allows users to filter and analyze web content either under their direction or autonomously.

How to use mcp-server-webcrawl?

To use mcp-server-webcrawl, install it via pip with the command pip install mcp_server_webcrawl. After installation, configure it through the Claude Desktop settings to connect to your desired web crawler.

Key features of mcp-server-webcrawl?

Compatibility with multiple web crawlers (WARC, wget, InterroBot, Katana, SiteOne)
Full-text search interface with boolean support
Resource filtering by type and HTTP status
Quick MCP configuration
Support for Claude Desktop and upcoming ChatGPT integration

Use cases of mcp-server-webcrawl?

Analyzing web content for research purposes.
Automating data extraction from web archives.
Enhancing AI language model responses with real-time web data.

FAQ from mcp-server-webcrawl?

Is mcp-server-webcrawl free to use?

Yes! It is free and open source.

What are the system requirements?

It requires Claude Desktop and Python version 3.10 or higher.

How do I configure the server?

Configuration is done through the Claude Desktop menu under File > Settings > Developer.

Project Info

Created At

7 months ago

Updated At

7 months ago

Author Name

pragmar

Star

Language

Python

License

View license

Recommend Servers

View All

Blender

@ahujasid

BlenderMCP connects Blender to Claude AI through the Model Context Protocol (MCP), allowing Claude to directly interact with and control Blender. This integration enables prompt assisted 3D modeling, scene creation, and manipulation.

8 months ago

Puppeteer

@modelcontextprotocol

Browser automation and web scraping

8 months ago

Org Mcp

@org-mcp

org-mcp is an Emacs package that implements a Model Context Protocol (MCP) server for Org-mode. It enables AI assistants and other MCP clients to interact with your Org files through a structured API.

2 hours ago

Github

@modelcontextprotocol

Repository management, file operations, and GitHub API integration

5 months ago

Saifsai

a day ago

Firecrawl Mcp Server

@mendableai

Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM clients.

JavaScript

5 months ago

Zhipu Web Search

@BigModel

Zhipu Web Search MCP Server is a search engine specifically designed for large models. It integrates four search engines, allowing users to flexibly compare and switch between them. Building upon the web crawling and ranking capabilities of traditional search engines, it enhances intent recognition capabilities, returning results more suitable for large model processing (such as webpage titles, URLs, summaries, site names, site icons, etc.). This helps AI applications achieve "dynamic knowledge acquisition" and "precise scenario adaptation" capabilities.

6 months ago

Test

@modelcontextprotocol

test

8 days ago

PostgreSQL

@modelcontextprotocol

Read-only database access with schema inspection

8 months ago

302_browser_use_mcp

@302ai

Automatically create a remote browser to complete your specified tasks, developed based on Browser Use + Sandbox. 自动创建一个远程浏览器，完成你指定的任务，基于Browser Use + Sandbox开发。

7 months ago

Guardian MCP

@SkyAstrall

Stops AI from writing duplicate code. Forces Claude/Cursor/Windsurf to analyze existing code first, get approval on plans, then execute step-by-step. Engineering gates ensure no hallucinations or breaking changes.

2 days ago

Hyperliquid Trader Mcp

@patricleehua

一个封装了hyperliquid交易的mcp工具，可以给模型对hyperliquid进行交易

3 days ago

Google Maps

@modelcontextprotocol

Location services, directions, and place details

8 months ago

a day ago

Playwright MCP server

TypeScript

3 months ago

MiniMax MCP

@MiniMax-AI

Official MiniMax Model Context Protocol (MCP) server that enables interaction with powerful Text to Speech, image generation and video generation APIs.

Python

7 months ago

AgentQL MCP Server

@tinyfish-io

Model Context Protocol server that integrates AgentQL's data extraction capabilities.

JavaScript

7 months ago

2 days ago

An MCP service designed for deploying HTML content to EdgeOne Pages and obtaining an accessible public URL.

TypeScript

7 months ago

Langfuse MCP

@therealsachin

Production-ready Langfuse MCP Server with 28 tools for LLM analytics, cost monitoring, and dataset management. Features secure dual-mode operation (readonly & readwrite), real-time cost analysis, trace debugging, and comprehensive usage insights. Deploy instantly with npx for seamless Claude Desktop and other LLM Client integration.

4 days ago

Simplifier Mcp

@simplifier-ag

An MCP Server that enables integration of AI assistants with the Simplifier Low Code Platform. It provides tools and resources for creating and managing Simplifier Connectors and BusinessObjects.

2 days ago

Jina AI MCP Tools

@PsychArch

A Model Context Protocol (MCP) server that integrates with Jina AI Search Foundation APIs.

JavaScript

6 months ago

Search1API

One API for Search, Crawling, and Sitemaps

6 months ago

Serper MCP Server

@garymengcom

A Serper MCP Server

Python

6 months ago

Test222

@modelcontextprotocol

5 days ago

Shipyard MCP

@shipyard

Loop your agents into environment management, and give them access/context to run tests, tail environment logs, start/stop environments, and debug remote issues.

2 days ago

a day ago

6 days ago

高德地图官方 MCP Server

7 months ago

Framelink Figma MCP Server

@GLips

MCP server to provide Figma layout information to AI coding agents like Cursor

TypeScript

8 months ago