Tools for evaluating AI agents, compared

Pick a tool by the question you need answered. This page sorts the common evaluation tools by that question, then points you to a detailed comparison for each one.

evalctl answers one question: did this change break what my agent does to a workspace? It runs the agent, looks at the files it wrote, the diffs it made, the commands it ran, and the exit codes, and it scores that. It runs from the command line, needs no Docker and no account, and writes one results file you can commit and another machine can re-check.

Other tools answer different questions. Here is how the field divides.

What are you trying to measure?

Most teams use more than one. evalctl is the fast local check you run first and underneath the heavier tools, not a replacement for them.

The three layers

Evaluation tools stack into three layers. Confusing one layer for another is the most common mistake, so it helps to name them.

1. Record what the agent did. Trajectory formats such as Letta’s Trajectory and Harbor’s ATIF write down an agent’s steps in a standard shape. They don’t run or score anything.
2. Run the agent and score the result. This is where evalctl sits. It drives the run and grades the outcome. A container-based harness like Harbor also lives here, scoring runs inside isolated sandboxes at scale.
3. Manage evaluation across a team. Platforms such as LangSmith and Braintrust store datasets, track history, and support many people over time.

evalctl’s results file belongs to the middle layer. It records how a run was scored, which is a different thing than a recording of the steps (layer one) or a shared history (layer three).

Prompt evaluation and agent evaluation are not the same

A prompt evaluator grades the text a model returns. An agent evaluator grades what an agent did — the files, the diffs, the commands, the exit codes. If your agent only produces a chat reply, a prompt evaluator fits. If it edits code, runs commands, or changes a workspace, the workspace is the thing worth checking, and that is agent evaluation.

evalctl is an agent evaluator. It also handles the plain-text case, but that is not where it is different.

Detailed comparisons

• evalctl and Harbor — a local check pointed at your own repo, next to a container-based harness for benchmark runs at scale.
• evalctl and DeepEval — scoring what an agent did to a workspace, next to scoring the quality of an LLM or RAG application’s outputs.
• evalctl and OpenAI Evals — a local check on your repo, next to grading a model’s answers against a dataset.
• evalctl and Inspect — a single CLI the agent drives, next to a full framework for running models and agents across datasets.
• evalctl and promptfoo — scoring what an agent did, next to scoring the text a model returned.
• evalctl and pytest — why a passing test suite doesn’t tell you what the agent did to make it pass.
• evalctl and LLM-as-judge — when a fixed check beats asking another model, and when a judge still earns its place.
• evalctl and GitHub Actions — how to turn an agent run into a pass/fail check with per-case detail in CI.
• evalctl and LangSmith — a local check that leaves a committable file, next to a hosted platform for datasets, team history, and production traces.
• evalctl and Braintrust — a local check pointed at your own repo, next to a hosted platform for dataset evaluations and comparable experiments.
• evalctl and Patronus — scoring what an agent did to a workspace, next to a hosted platform whose SDK scores LLM output quality with managed and custom evaluators.
• evalctl and Langfuse — a local check that leaves a committable file, next to an open-source observability platform that traces an application and evaluates it.
• evalctl and Arize Phoenix — a single local command, next to an observability server that traces a run step by step and scores its spans.
• evalctl and W&B Weave — scoring what an agent did to a workspace, next to logging dataset evaluations and traces to the Weights & Biases platform.
• evalctl and TruLens — scoring what an agent did to a workspace, next to scoring an application’s generated text with local feedback functions.
• evalctl and SWE-bench — a reusable gate on your own repo, next to a fixed public benchmark that scores agents on curated GitHub issues with their own tests.
• evalctl and lm-evaluation-harness — scoring what an agent did to a workspace, next to measuring a model’s capability on fixed academic benchmarks.
• evalctl and Harbor ATIF — running and scoring an agent, next to a JSON format that records an agent’s run.
• evalctl and Letta Trajectory — running and scoring an agent, next to a token-efficient format that records a run for another agent to read.
• evalctl and Guardrails — an offline gate on what an agent did, next to Guardrails AI and NeMo Guardrails validating an LLM’s inputs and outputs live in the request path.

How to choose in one line

Use evalctl when you want a fast, repeatable local check that scores what your agent did to a workspace and leaves a results file you can commit — with no Docker, no server, and no account. When you need shared datasets, team history, container isolation, or production tracing, add a platform on top. evalctl is built to run underneath one, not instead of it.