Crawl4AI
by UncleCode
Crawls and scrapes web pages into clean Markdown for RAG pipelines, with CSS, XPath, and LLM-driven structured extraction, deep crawling, and full browser session control.
Skills
LLM Ready Markdown
Generates clean Markdown with heuristic and BM25 noise filtering, numbered citation references for links, and pluggable strategies.
Structured Data Extraction
Extracts JSON matching a user-defined schema using CSS or XPath selectors, or delegates to an LLM for semantic field extraction.
Deep Site Crawling
Traverses whole sites with BFS and best-first strategies under page limits, using prefetch URL discovery and resumable crash recovery.
Browser Session Control
Drives Chromium, Firefox, or WebKit with persistent profiles, proxies, stealth mode, hooks, and reusable sessions for multi-step crawls.
Docker API Server
Exposes a FastAPI /crawl endpoint with JWT authentication plus MCP integration so agent clients can crawl over HTTP.
Related Agents
Docling
Parses PDF, DOCX, PPTX, XLSX, HTML, audio, and image files into a unified DoclingDocument and exports Markdown, HTML, D…
Firecrawl
Turn websites into LLM-ready data — crawl, scrape, and convert web pages to clean markdown for AI consumption.
Reddit MCP Buddy
MCP server for Reddit — search subreddits, retrieve posts and comment threads, analyze user activity, and post content.…
Agent Switchboard
Searches a curated, open-source directory of 300+ verified AI agents, MCP servers, and agentic tools by capability, cat…