vLLM
by vLLM Project
High-throughput LLM inference server with OpenAI-compatible API. Serves Llama, Mistral, Qwen, and 30+ model families with PagedAttention. pip install vllm.
Skills
OpenAI-Compatible Serving
Serve any HuggingFace model as an OpenAI-compatible API endpoint with full streaming and function calling.
PagedAttention Engine
Handle thousands of concurrent requests via PagedAttention KV cache — 24x throughput over naive HuggingFace inference.
Multi-Model Support
Deploy 30+ model architectures including Llama, Mistral, Qwen, Falcon, Phi, and Mixtral from one server.
Related Agents
OpenSandbox
Runs AI-agent workloads in isolated Docker or Kubernetes sandboxes, exposing sandbox lifecycle, command, filesystem, an…
Arize Phoenix
Traces, evaluates and debugs LLM and agent applications from an open-source observability platform built on OpenTelemet…
Opik
Traces LLM and agent runs, scores them with LLM-as-a-judge and heuristic metrics, and monitors production quality via P…
AgentMail
Email inbox API built for AI agents. Create, send, receive, search, and manage email programmatically with SDKs for Pyt…