crawler-mcp-server

Created By
m1gglea year ago
crawler-mcp-server
Overview

what is crawler-mcp-server?

The crawler-mcp-server is a project designed to facilitate web crawling and data extraction from various sources.

how to use crawler-mcp-server?

To use the crawler-mcp-server, clone the repository from GitHub, configure the server settings, and run the server to start crawling web pages.

key features of crawler-mcp-server?

  • Efficient web crawling capabilities
  • Customizable crawling parameters
  • Data extraction and storage options

use cases of crawler-mcp-server?

  1. Collecting data for research purposes
  2. Monitoring website changes
  3. Aggregating content from multiple sources

FAQ from crawler-mcp-server?

  • What programming language is used for crawler-mcp-server?

The project is primarily developed in JavaScript.

  • Is there a user guide available?

Yes, the repository includes documentation on how to set up and use the server.

  • Can I contribute to the project?

Yes! Contributions are welcome, and you can submit pull requests on GitHub.

Project Info
Created At
a year ago
Updated At
a year ago
Author Name
m1ggle
Star
0
Language
-
License
-

Recommend Servers

View All
Voyei

2 hours ago
Bring your real authenticated browser session to AI coding agents. Local-first MCP server + Chrome MV3 extension. No cloud. No telemetry.
@Cubenest

peek records the user's actual logged-in browser (DOM via rrweb, console events, network metadata, optional response bodies via opt-in Deep capture) through a Chrome MV3 extension. The extension ships events through a native-messaging stdio bridge to a local MCP server (peek-mcp), which persists them to a SQLite database at ~/.peek/sessions.db. AI coding agents (Claude Code, Cursor, Cline, Windsurf) read sessions from the database via 10 MCP tools: Tool What it does list_recent_sessions List recently recorded sessions (id, origin, ts, event count). get_session_summary LLM-readable narrative summary of a session. get_session_console_errors Console errors recorded in a session. get_session_network_errors Failed/notable network requests in a session. get_user_action_before_error Last N user actions before a console error. generate_playwright_repro Generate a runnable Playwright test from a session. get_dom_snapshot Reconstruct the DOM at a given timestamp. query_dom_history Timeline of attribute/text changes for a selector. request_authorization Side-panel consent for write actions (Level 3). execute_action Dispatch a UI action (gated by permission level + destructive blocklist). Why local-first matters Every other "browser session for AI" tool ships to a vendor cloud. peek's SQLite + extension live on the user's machine — no remote endpoints, no telemetry. The privacy policy (docs/peek/PRIVACY_POLICY.md) is the source of truth. Install # 1. Add the MCP server to Claude Code claude mcp add peek -- npx -y @peekdev/mcp # 2. Install the Chrome extension from the Chrome Web Store # (link added once the CWS listing is approved)

a day ago