Agent Switchboard

Tasks

I need to test and evaluate agents

Trace, test and score LLM apps and agents before and after they ship.

29 tools · 13 official

Picked from listings that mention evals, evaluation, llm observability, tracing or prompt testing in their name, description, tags or skills; those in Infrastructure & Ops, Code & DevTools or Orchestration, or with API access, rank higher.

Best fits · 12

  1. Langfuse

    API · CLI

    Open-source LLM observability and analytics. Trace, evaluate, and debug LLM apps with prompt versioning, score…

    VerifiedOfficialby Langfuse · Infrastructure & Ops · 5 skills · API Key

    Why Score traces with custom evaluation rubrics to track quality over time

  2. Future AGI

    API · CLI

    Evaluate, observe, and improve LLM and agent applications from one self-hostable platform — tracing, evals…

    Verifiedby Future AGI · Code & DevTools · 3 skills · API Key

    Why Evaluation Suite: Scores outputs for hallucination, relevance, and task success using configurable evaluators across datasets.

  3. Microsoft PromptFlow

    API · CLI

    Build, evaluate, and deploy LLM applications as visual DAG flows — chain prompts, Python, and tools, then run batch…

    VerifiedOfficialby Microsoft · Orchestration, Code & DevTools · 3 skills · API Key

    Why Batch Evaluation: Executes a flow across a dataset and scores results with groundedness, relevance, and safety evaluators.

  4. OpenLLMetry

    API

    Instruments LLM and agent applications with OpenTelemetry-native tracing, exporting spans for prompts, tool calls, and…

    VerifiedOfficialby Traceloop · Infrastructure & Ops, Research & Knowledge · 3 skills · API Key

    Why Workflow Tracing: Groups multi-step agent runs into one trace so latency and cost attribute per step.

  5. Latitude

    API · CLI · MCP

    Traces, scores, and evaluates production AI agents via OpenTelemetry and SDKs, groups failures into signals, and…

    Verifiedby Latitude · Code & DevTools, Data & Analytics · 4 skills · API Key

    Why Automated Evaluations: Scores sessions with condition sets, LLM judges, or custom JavaScript scripts, and replays regression datasets against fixed…

  6. MCPJam

    API · CLI · MCP

    Tests and evaluates MCP servers and MCP apps for developers, with a playground, OAuth debugger, cross-client evals, and…

    Verifiedby MCPJam · Code & DevTools · 4 skills · No auth

    Why Cross-Model Evals: Runs test cases with expected tool calls across multiple LLMs and tracks accuracy over time to catch MCP server regressions.

  7. LLM Space

    —

    Prototypes, traces, replays, and evaluates LLM agents in a local macOS desktop app, and exports threads as runnable…

    by DeerFlow · Code & DevTools, Orchestration · 4 skills · API Key

    Why Agent Run Tracing: Shows every model call and tool run inside the agent loop as it happens.

  8. Opik

    API · MCP · CLI

    Traces LLM and agent runs, scores them with LLM-as-a-judge and heuristic metrics, and monitors production quality via…

    Verifiedby Comet ML · Infrastructure & Ops, Code & DevTools · 5 skills · API Key

    Why PyTest CI Evaluation: Tests LLM pipelines on every commit through the PyTest integration so evaluation results are recorded per CI run.

  9. Datadog LLM Observability

    API · MCP · CLI

    End-to-end monitoring for LLM applications — trace prompts, evaluate responses, track costs, and debug AI pipelines in…

    VerifiedOfficialby Datadog · Infrastructure & Ops · API Key

  10. Helicone

    API

    Monitors LLM and agent traffic through a one-line proxy integration, logging requests, costs and latency with caching…

    Verifiedby Helicone · Data & Analytics, Infrastructure & Ops · 3 skills · API Key

    Why Session Tracing: Groups multi-step agent calls into sessions so a whole run can be replayed and debugged as one unit.

  11. Arize Phoenix

    API · CLI

    Traces, evaluates and debugs LLM and agent applications from an open-source observability platform built on…

    VerifiedOfficialby Arize AI · Data & Analytics, Infrastructure & Ops · 3 skills · No auth

    Why Evaluation Runs: Runs LLM-as-judge evaluations over captured traces and datasets to quantify quality regressions across versions.

  12. Google Agents CLI

    CLI

    Turns any coding assistant into an expert at creating, evaluating, and deploying AI agents on Google Cloud. An open CLI…

    VerifiedOfficialby Google · Code & DevTools · 3 skills · OAuth 2.0

    Why Evaluation Runs: Runs evaluation suites against an agent and reports quality metrics before it is deployed to production.

See all 29 on Browse, with filters

Official · 13

The ones published by the company behind the product.

See all 13 official on Browse

In stacks

Missing a tool that does this? Suggest a listing.

/tasks/evaluate-agents
I need to test and evaluate agents · 29 fit · 13 official · keywords: evals, evaluation, llm observability, tracing, prompt testing · categories: infrastructure, code-devtools, orchestration · access: api
Trace, test and score LLM apps and agents before and after they ship.
best fits
langfuseapi+cliScore traces with custom evaluation rubrics to track quality over timefuture-agiapi+cliEvaluation Suite: Scores outputs for hallucination, relevance, and task success using configurable evaluators across datasets.promptflowapi+cliBatch Evaluation: Executes a flow across a dataset and scores results with groundedness, relevance, and safety evaluators.openllmetryapiWorkflow Tracing: Groups multi-step agent runs into one trace so latency and cost attribute per step.latitudeapi+cli+mcpAutomated Evaluations: Scores sessions with condition sets, LLM judges, or custom JavaScript scripts, and replays regression datasets against fixed…mcpjamapi+cli+mcpCross-Model Evals: Runs test cases with expected tool calls across multiple LLMs and tracks accuracy over time to catch MCP server regressions.llm-spaceAgent Run Tracing: Shows every model call and tool run inside the agent loop as it happens.opikapi+mcp+cliPyTest CI Evaluation: Tests LLM pipelines on every commit through the PyTest integration so evaluation results are recorded per CI run.datadog-llm-observabilityapi+mcp+cliEnd-to-end monitoring for LLM applications — trace prompts, evaluate responses, track costs, and debug AI pipelines in production.heliconeapiSession Tracing: Groups multi-step agent calls into sessions so a whole run can be replayed and debugged as one unit.arize-phoenixapi+cliEvaluation Runs: Runs LLM-as-judge evaluations over captured traces and datasets to quantify quality regressions across versions.google-agents-clicliEvaluation Runs: Runs evaluation suites against an agent and reports quality metrics before it is deployed to production.
official
langfuse, promptflow, openllmetry, datadog-llm-observability, arize-phoenix, google-agents-cli, langchain-langsmith, agentscope, genkit, cursor-agent-sdk, nvidia-nemo-agent-toolkit, cloudflare-ai-gateway, vercel-ai-gateway
related tasks
/tasks/monitor-apps-in-production44 fitmonitor apps in production/tasks/run-code-in-a-sandbox43 fitrun code in a sandbox/tasks/deploy-an-app52 fitdeploy and host an app/tasks/query-a-database46 fitquery a database
in stacks
/stacks/agents-in-productionarize-phoenix+heliconeWatch your agents in production
more
/browse?task=evaluate-agentsall 29, with filters/tasksevery task/api/tasks/evaluate-agentsthis task as JSONGET · no auth