mcp-server-webcrawl

Created By

MCP-Mirrora year ago

Mirror of

Overview

What is mcp-server-webcrawl?

mcp-server-webcrawl is a tool that bridges the gap between web crawling and AI language models using the Model Context Protocol (MCP). It allows AI clients to filter and analyze web content either autonomously or under user direction.

How to use mcp-server-webcrawl?

To use mcp-server-webcrawl, install it via pip with the command pip install mcp-server-webcrawl. Configure it through the Claude Desktop settings to set up connections to various web crawlers.

Key features of mcp-server-webcrawl?

Compatibility with multiple web crawlers (WARC, wget, InterroBot, Katana, SiteOne)
Full-text search interface with boolean support
Resource filtering by type and HTTP status
Quick MCP configuration
Support for Claude Desktop and upcoming ChatGPT integration

Use cases of mcp-server-webcrawl?

Analyzing web content for research purposes.
Automating data extraction from websites.
Enhancing AI model training with real-time web data.

FAQ from mcp-server-webcrawl?

Is mcp-server-webcrawl free to use?

Yes! It is open-source and free for everyone.

What are the system requirements?

It requires Python (>=3.10) and Claude Desktop.

Can it work with any web crawler?

Yes, it supports various crawlers like WARC, wget, and more.

Project Info

Created At

a year ago

Updated At

a year ago

Author Name

MCP-Mirror

Star

Language

Python

License

View license

Recommend Servers

View All

Redis

@modelcontextprotocol

A Model Context Protocol server that provides access to Redis databases. This server enables LLMs to interact with Redis key-value stores through a set of standardized tools.

a year ago

Bangladesh Payments Mcp

@junter1989k-ai

9 days ago

PAPI

@getpapi

Persistent Adaptive Planning Intelligence - structured loop engineering for AI coding assistants, with memory that persists across sessions, tools, and teammates.

9 days ago

Costa Rica Payments Mcp

@junter1989k-ai

8 days ago

EdgeOne Pages MCP

@TencentEdgeOne

An MCP service designed for deploying HTML content to EdgeOne Pages and obtaining an accessible public URL.

TypeScript

a year ago

Jordan Payments Mcp

@junter1989k-ai

9 days ago

Aws Kb Retrieval Server

@modelcontextprotocol

An MCP server implementation for retrieving information from the AWS Knowledge Base using the Bedrock Agent Runtime.

a year ago

Hong Kong Payments Mcp

@junter1989k-ai

9 days ago

Creationloop

@marino129

9 days ago

Slovakia Payments Mcp

9 days ago

9 days ago

9 days ago

Lithuania Payments Mcp

@junter1989k-ai

9 days ago

Aiimagemultistyle

@codecraftm

A Model Context Protocol (MCP) server for image generation and manipulation using fal.ai's Stable Diffusion model.

a year ago

Test

@modelcontextprotocol

test

7 months ago

Xiangguangyu Skills

@Chromatic0618

5 AI skills as MCP tools: MBTI relationship sandbox (2-20 people), fact-checking (3 methods: source-tracing, cross-reference, logic-check), deep business analysis (5-layer), game audio brief generator, weekly skill radar. All monetized via Alipay AI Pay (¥0.10-0.50/call).

9 days ago

Baidu Map

@baidu-maps

百度地图核心API现已全面兼容MCP协议，是国内首家兼容MCP协议的地图服务商。

a year ago

PostgreSQL

@modelcontextprotocol

Read-only database access with schema inspection

a year ago

mcp-server-flomo MCP Server

@chatmcp

Write notes to Flomo

JavaScript

a year ago

Tavily Mcp

@tavily-ai

JavaScript

a year ago

9 days ago

MCP server for ArcGIS Portal and ArcGIS Online. Lets AI assistants search content, query feature layers, manage features, handle content operations, and administer users and groups. Built on the Model Context Protocol for integration with Claude Desktop, Cursor, VS Code Copilot, and other MCP clients. Disclaimer: This is an independent open-source project. It is not affiliated with, endorsed by, or sponsored by Esri Inc. "ArcGIS" is a registered trademark of Esri.

8 days ago

Github

@modelcontextprotocol

Repository management, file operations, and GitHub API integration

a year ago

Serper MCP Server

@garymengcom

A Serper MCP Server

Python

a year ago

Qiniu MCP Server

@Qiniu

基于七牛云产品构建的 Model Context Protocol (MCP) Server，支持用户在 AI 大模型客户端的上下文中通过该 MCP Server 来访问七牛云存储资源、利用 Dora 服务进行图片操作等。如果有什么需求欢迎在下方评论，您也可以在 github 仓库中提 issue。

Python

a year ago

I Ching Hexagram Mcp

@Network-Ideas LLC

Public, read-only MCP over the complete 64-hexagram I Ching corpus — hexagrams, trigrams, changing lines, and reading context from an original English translation. Deterministic lookup only; no casting or divination.

9 days ago

Romania Payments Mcp

@junter1989k-ai

9 days ago

Time

@modelcontextprotocol

A Model Context Protocol server that provides time and timezone conversion capabilities. This server enables LLMs to get current time information and perform timezone conversions using IANA timezone names, with automatic system timezone detection.

5 months ago

Tanzania Payments Mcp

@junter1989k-ai

9 days ago

MiniMax MCP

@MiniMax-AI

Official MiniMax Model Context Protocol (MCP) server that enables interaction with powerful Text to Speech, image generation and video generation APIs.

Python

a year ago