Crawls and scrapes web pages into clean Markdown for RAG pipelines, with CSS, XPath, and LLM-driven structured extraction, deep crawling, and full browser session control.
- Access
- APICLIMCP
- Auth
- None
- Skills
- 5
Skills
LLM Ready Markdown
Generates clean Markdown with heuristic and BM25 noise filtering, numbered citation references for links, and pluggable strategies.
Structured Data Extraction
Extracts JSON matching a user-defined schema using CSS or XPath selectors, or delegates to an LLM for semantic field extraction.
Deep Site Crawling
Traverses whole sites with BFS and best-first strategies under page limits, using prefetch URL discovery and resumable crash recovery.
Browser Session Control
Drives Chromium, Firefox, or WebKit with persistent profiles, proxies, stealth mode, hooks, and reusable sessions for multi-step crawls.
Docker API Server
Exposes a FastAPI /crawl endpoint with JWT authentication plus MCP integration so agent clients can crawl over HTTP.
Related Agents
context.dev
API · MCP · CLIScrapes, crawls and searches the web for agents through a REST API, SDKs, CLI and hosted MCP server, returning…
AgentQL MCP
API · MCP · ExtensionExtracts structured web data using natural language queries — no XPath or selectors needed. Works via Python/JS SDKs…
Nimble
API · MCP · CLISearches, extracts, maps and crawls the live web and runs cited research agents, for AI agents that need real-time web…
Browser Use
API · CLI · MCPOpen-source Python library for AI browser automation. Control any website with natural language; cloud adds stealth…