mcp-server-webcrawl

Created By
pragmara year ago
Bridge the gap between your web crawler and AI language models using Model Context Protocol (MCP). With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously, extracting insights from your web content. Support for WARC, wget, InterroBot, Katana, and SiteOne crawlers is available out of the gate. The server includes a full-text search interface with boolean support, resource filtering by type, HTTP status, and more.
Overview

what is mcp-server-webcrawl?

mcp-server-webcrawl is an open-source server that bridges the gap between web crawlers and AI language models using the Model Context Protocol (MCP). It allows AI clients to filter and analyze web content, extracting insights either under user direction or autonomously.

how to use mcp-server-webcrawl?

To use mcp-server-webcrawl, install it via pip with the command: pip install mcp-server-webcrawl. You can then run the server using the command: mcp-server-webcrawl --crawler wget --datasrc /path/to/wget/archives/.

key features of mcp-server-webcrawl?

  • Compatibility with Claude Desktop
  • Full-text search interface with boolean support
  • Resource filtering by type and HTTP status
  • Support for various crawlers including wget, WARC, and more
  • Ability to augment your LLM knowledge base
  • ChatGPT support is coming soon

use cases of mcp-server-webcrawl?

  1. Analyzing web content for research purposes
  2. Extracting insights from large datasets collected by web crawlers
  3. Enhancing AI language models with real-time web data

FAQ from mcp-server-webcrawl?

  • Is mcp-server-webcrawl free to use?

Yes! mcp-server-webcrawl is free and open-source.

  • What are the system requirements?

It requires Claude Desktop and Python version 3.10 or higher.

  • Which crawlers are supported?

It supports wget, WARC, InterroBot, Katana, and SiteOne crawlers.

Server Config

{
  "mcpServers": {
    "webcrawl": {
      "command": "mcp-server-webcrawl",
      "args": [
        "--crawler",
        "wget",
        "--datasrc",
        "/path/to/wget/archives/"
      ]
    }
  }
}
Project Info
Created At
a year ago
Updated At
a year ago
Author Name
pragmar
Star
-
Language
-
License
-
Category

Recommend Servers

View All
Verified Tax Vat Global | Tax Year Tools | Ai2fin
@AI2FIN

AI2Fin Tax MCP — free tax rates, GST/VAT, income, company and capital-gains tax for 50+ countries, over the Model Context Protocol The Tax MCP by AI2Fin (2Fin) is a free, public MCP server that gives any assistant or agent — Claude, ChatGPT, Cursor, VS Code, or a custom build — instant access to accurate, source-cited tax data. No login, no API key, no setup beyond pasting one URL: https://taxmcp.ai2fin.com. Every answer cites the national tax authority it came from (ATO, IRD, HMRC, IRS and more) with a verified date, so you always know a number is current — and where to confirm it. What you can do with it: Look up any country's tax rates — GST/VAT standard and reduced rates, top personal income tax rate, and company tax rate for 50+ countries, from Australia and New Zealand to the UK, US, India, Canada, the EU and beyond (tax_rate_lookup). Add or remove GST/VAT on any amount — instant tax-inclusive ↔ tax-exclusive conversion for invoices, quotes, pricing and expense checks, using the right rate for the right country automatically (compute_gst_vat). Compare tax across countries side by side — GST/VAT, income and company tax in one call, for relocation research, market expansion or pricing strategy (compare_countries). Estimate income tax and take-home pay for Australia and New Zealand, computed over effective-dated brackets that roll over automatically at each financial year (income_tax_estimate). Estimate company tax on a profit for Australia, the US, UK, India and Canada — including small-business rates and thresholds, not just the headline figure (company_tax_estimate). Estimate Australian capital gains tax for resident individuals — capital losses, the 12-month CGT discount, and marginal-rate stacking on top of your income, handled correctly (cgt_estimate). Countries only answer when their data is verified against an official source; anything unverified degrades gracefully to "not available" instead of guessing. The server is stateless and private by design — it computes over published rate data only and stores nothing you send. It runs the same single-source dataset behind the free browser calculators at ai2fin.com/tools, so the MCP, the embeddable widgets and the web calculators never disagree. Listed in the official MCP registry as https://taxmcp.ai2fin.com/ . Works with any MCP client over streamable HTTP. General information, not tax advice — every figure links to the authority so you can confirm it. For your own transactions, categories and real tax position, the separate authenticated 2Fin app MCP at api.ai2fin.com/mcp picks up where this one ends.

17 hours ago