- Crawler For Llm
Crawler For Llm
高等级的免费爬虫MCP, 完全 Markdown 化支持,支持微信公众号等一大批国内网站。A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format. This tool is essential for building knowledge bases, training data collection, and content aggregation for LLM applications. It includes built-in support for the Model Context Protocol (MCP) for enhanced context management.
Overview
Crawler for LLM with MCP Support
A universal web crawler for LLM frameworks with Model Context Protocol (MCP) integration
A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format. This tool is essential for building knowledge bases, training data collection, and content aggregation for LLM applications. It includes built-in support for the Model Context Protocol (MCP) for enhanced context management.
{
"mcpServers": {
"langgraph-crawler": {
"command": "npx",
"args": ["-y", "@langgraph-js/crawler-mcp@latest"]
}
}
}
Supported Websites
Note: This list is continuously expanding. Feel free to contribute by adding support for more websites.
Documentation & Development
- NPM
- Vitepress Base Website
- Github
- DEV Community (dev.to)
- MDN
- Medium
- SearXNG Search Result
- freeCodeCamp.org
- Docker Hub
- StackOverflow
- Anthropic
Chinese Platforms
- WeChat Public Articles (微信公众号)
- Juejin (稀土掘金)
- The Paper (澎湃新闻)
- Jiemian (界面新闻)
- Huxiu (虎嗅网)
- UISDC (优设网)
- CNBlogs (博客园)
- SSPAI (少数派)
- InfoQ
- CSDN
- Zhihu (知乎)
License
Apache-2.0
Server Config
{
"mcpServers": {
"langgraph-crawler": {
"command": "npx",
"args": [
"-y",
"@langgraph-js/crawler-mcp@latest"
]
}
}
}Project Info
Created At
a year agoUpdated At
a year agoAuthor Name
KongHaYaoStar
-Language
-License
-Recommend Servers
View AllShotapi
@smallhandsome
9 hours ago
Crevio
a day ago
Playwright Mcp
@microsoft
Playwright MCP server
TypeScript
10 months ago
Airtreks Mcp
@SEKeener
10 hours ago
Serper MCP Server
@garymengcom
A Serper MCP Server
Python
a year ago