Crawl4ai Rag

Created By

coleam00a year ago

Overview

what is Crawl4AI RAG?

Crawl4AI RAG is a powerful MCP server that integrates web crawling and retrieval-augmented generation (RAG) capabilities for AI agents and coding assistants, enabling them to scrape web content and utilize it effectively.

how to use Crawl4AI RAG?

To use Crawl4AI RAG, clone the repository from GitHub, set up the necessary environment variables, and run the server using Docker or Python. You can then connect it to your AI applications for enhanced web crawling and content retrieval.

key features of Crawl4AI RAG?

Smart URL detection and recursive crawling
Parallel processing for efficient content scraping
Advanced RAG strategies including contextual embeddings and hybrid search
Knowledge graph integration for AI hallucination detection
Tools for searching code examples and validating AI-generated code

use cases of Crawl4AI RAG?

Enabling AI coding assistants to retrieve relevant documentation and code examples.
Enhancing AI agents with the ability to crawl and analyze web content.
Validating AI-generated code against real-world repositories to prevent hallucinations.

FAQ from Crawl4AI RAG?

Can Crawl4AI RAG handle all types of web content?

Yes! It is designed to crawl various types of web pages and extract relevant information.

Is there a specific setup required for using the knowledge graph features?

Yes, you need to set up Neo4j for the knowledge graph functionalities to work.

How can I customize the crawling and retrieval strategies?

You can configure various RAG strategies in the .env file before running the server.

Try in Playground

Server Config

{
  "mcpServers": {
    "crawl4ai-rag": {
      "command": "python",
      "args": [
        "path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"
      ],
      "env": {
        "TRANSPORT": "stdio",
        "OPENAI_API_KEY": "your_openai_api_key",
        "SUPABASE_URL": "your_supabase_url",
        "SUPABASE_SERVICE_KEY": "your_supabase_service_key",
        "USE_KNOWLEDGE_GRAPH": "false",
        "NEO4J_URI": "bolt://localhost:7687",
        "NEO4J_USER": "neo4j",
        "NEO4J_PASSWORD": "your_neo4j_password"
      }
    }
  }
}

Project Info

Created At

a year ago

Updated At

a year ago

Author Name

coleam00

Star

Language

License

Recommend Servers

View All

Puppeteer

@modelcontextprotocol

Browser automation and web scraping

a year ago

MCP Advisor

@istarwyh

MCP Advisor & Installation - Use the right MCP server for your needs

TypeScript

a year ago

PostgreSQL

@modelcontextprotocol

Read-only database access with schema inspection

a year ago

12 days ago

12 days ago

@modelcontextprotocol

Retrieving and analyzing issues from Sentry.io

a year ago

Papaya Pay Any Bill

@Papaya

Ready for a new way to bill pay? Pay any bill in a snap, right from a chat. Describe or snap a photo of your bill (electric, water, gas, internet, phone, medical, credit card, rent, parking tickets, traffic violations and more) and Papaya reads it, then hands you a secure link to pay by card. Fast, secure, and no juggling twelve logins, with full or partial payments and status updates. Powered by Papaya (papayapay.com).

12 days ago

12 days ago

基于七牛云产品构建的 Model Context Protocol (MCP) Server，支持用户在 AI 大模型客户端的上下文中通过该 MCP Server 来访问七牛云存储资源、利用 Dora 服务进行图片操作等。如果有什么需求欢迎在下方评论，您也可以在 github 仓库中提 issue。

Python

a year ago

EverArt

@modelcontextprotocol

AI image generation using various models

a year ago

Egypt Payments Mcp

@junter1989k-ai

12 days ago

Aws Kb Retrieval Server

@modelcontextprotocol

An MCP server implementation for retrieving information from the AWS Knowledge Base using the Bedrock Agent Runtime.

a year ago

Jina AI MCP Tools

@PsychArch

A Model Context Protocol (MCP) server that integrates with Jina AI Search Foundation APIs.

JavaScript

a year ago

Baidu Map

@baidu-maps

百度地图核心API现已全面兼容MCP协议，是国内首家兼容MCP协议的地图服务商。

a year ago

Framelink Figma MCP Server

@GLips

MCP server to provide Figma layout information to AI coding agents like Cursor

TypeScript

a year ago

mcp-server-flomo MCP Server

@chatmcp

Write notes to Flomo

JavaScript

a year ago

302_sandbox_mcp

@302ai

Create a remote sandbox that can execute code/run commands/upload and download files. 创建远程沙盒，可以执行代码/运行命令/上传下载文件

a year ago

South Africa Payments Mcp

12 days ago

12 days ago

Persistent Adaptive Planning Intelligence - structured loop engineering for AI coding assistants, with memory that persists across sessions, tools, and teammates.

12 days ago

12 days ago

12 days ago

12 days ago

Bangladesh Payments Mcp

@junter1989k-ai

12 days ago

Serper MCP Server

@garymengcom

A Serper MCP Server

Python

a year ago

GBOX Android MCP

@babelcloud

GBOX provides environments for AI Agents to operate computer and mobile devices. Mobile Scenario: Your agents can use GBOX to develop/test android apps, or run apps on the Android to complete various tasks(mobile automation). Desktop Scenario: Your agents can use GBOX to operate desktop apps such as browser, terminal, VSCode, etc(desktop automation). MCP: You can also plug GBOX MCP to any Agent you like, such as Cursor, Claude Code. These agents will instantly get the ability to operate computer and mobile devices.

a year ago

12 days ago

Language models generate; Concordance verifies. The verify tool checks a claim deterministically (no model in the loop) and returns HOLDS / BROKEN / INCOMPLETE with the worked reasoning and a sealed receipt (content_hash + cite_url) that re-fetches byte-identical or not at all. Also: ranked search over an ~11k-record library, seal_fetch to re-verify any receipt, redact to strip PII before text travels, and a sealed connection graph. Runs sovereign/offline too (stdlib-first Python). A public false-positive benchmark covers every domain: the engine has never sealed a falsehood. 38 tools live; remote endpoint at https://narrowhighway.com/mcp.

12 days ago

Framesail AI

@framesail

Official remote MCP server for Framesail AI. Create long-form (faceless YouTube) videos end to end from any MCP client: script, locked character references, storyboard, voiceover, and final video editing — with characters and style held consistent across every shot. Making long-form AI video today means 8+ tabs stitched by hand — an LLM for the script, a voice model, an image model, a video model — with characters drifting between tools and style resetting at every export. Framesail replaces the patchwork: the whole pipeline runs in one place and manages your video's context end to end. Six stages: Style (paste images, videos, or YouTube links and Framesail reverse-engineers the look, voice, and direction), Script (write it yourself or generate it in your narrative style), Reference images (auto-generated for every character, place, and prop), Voiceover (one narrator or many characters, with word-level timing), Storyboard (planned scene by scene), and Editor (captions, music, SFX, then export). No black box: you control every prompt, asset, model, and setting.

12 days ago

Norway Payments Mcp

@junter1989k-ai

12 days ago