> ## Documentation Index
> Fetch the complete documentation index at: https://agentdiff.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> Verify your AI agent performed the expected actions

## What Are Evaluations?

Evaluations let you verify that your AI agent performed the expected actions. Instead of checking if the agent "said the right thing", you check if it **did the right thing** by examining database state changes.

<CardGroup cols={2}>
  <Card title="Mock Evaluation" icon="xmark">
    "Did the agent say it posted a message?" → Maybe ✓
  </Card>

  <Card title="Agent Diff Evaluation" icon="check">
    "Is there a new row in messages table?" → Definite ✓ or ✗
  </Card>
</CardGroup>

## Built-in Evaluations

### Slack Bench

```python theme={null}
suites = client.list_test_suites(name="Slack Bench")
suite = client.get_test_suite(suites.testSuites[0].id, expand=True)
print(f"Slack Bench: {len(suite.tests)} tests")
```

**Coverage:**

* Message sending (5 tests)
* Channel operations (4 tests)
* Reactions (3 tests)
* Threading (4 tests)
* User mentions (4 tests)

### Linear Bench

```python theme={null}
suites = client.list_test_suites(name="Linear Bench")
suite = client.get_test_suite(suites.testSuites[0].id, expand=True)
print(f"Linear Bench: {len(suite.tests)} tests")
```

**Coverage:**

* Issue CRUD (12 tests)
* Labels (6 tests)
* Comments (5 tests)
* Workflow states (8 tests)
* Team operations (5 tests)
* Projects (4 tests)

<Tip>
  **View evaluation suites files on GitHub**: [slack bench](https://github.com/hubertpysklo/agent-diff/tree/main/examples/slack/testsuites) | [linear bench](https://github.com/hubertpysklo/agent-diff/tree/main/examples/linear/testsuites)
</Tip>

## Quick Example

```python theme={null}
from agent_diff import AgentDiff, PythonExecutorProxy, create_openai_tool
from agents import Agent, Runner

client = AgentDiff()

# Get a test from the built-in Slack Bench suite
suites = client.list_test_suites(name="Slack Bench")
suite = client.get_test_suite(suites.testSuites[0].id, expand=True)
test = suite.tests[0]

# Create environment and start run
env = client.init_env(testId=test.id)
run = client.start_run(envId=env.environmentId, testId=test.id)

# Run your agent
executor = PythonExecutorProxy(env.environmentId, base_url=client.base_url, api_key=client.api_key)
agent = Agent(
    name="Test Agent",
    tools=[create_openai_tool(executor)],
    instructions="Use execute_python to interact with Slack at https://slack.com/api/*"
)
await Runner.run(agent, test.prompt)

# Evaluate against assertions
result = client.evaluate_run(runId=run.runId)

print(f"Result: {'✓ PASSED' if result.passed else '✗ FAILED'}")
print(f"Score: {result.score}")
if result.failures:
    print(f"Failures: {result.failures}")

# Cleanup
client.delete_env(envId=env.environmentId)
```

<Tip>
  **Try the built-in benchmarks**: [Slack Bench](https://github.com/hubertpysklo/agent-diff/tree/main/examples/slack/testsuites) | [Linear Bench](https://github.com/hubertpysklo/agent-diff/tree/main/examples/linear/testsuites) | [HuggingFace Dataset](https://huggingface.co/datasets/hubertmarek/linear-bench-mini)
</Tip>

## Evaluation Without Test Suite

You can run evaluations without a pre-defined test suite by passing the expected output explicitly:

```python theme={null}
from agent_diff import AgentDiff, PythonExecutorProxy, create_langchain_tool
from langgraph.prebuilt import create_react_agent as create_agent
from langchain_openai import ChatOpenAI

client = AgentDiff()

# Create environment directly with template
env = client.init_env(
    templateService="linear",
    templateName="linear_expanded",
    impersonateUserId="2790a7ee-fde0-4537-9588-e233aa5a68d1"
)

# Start run (no testId)
run = client.start_run(envId=env.environmentId)

# Run your agent...
model = ChatOpenAI(model="gpt-4o")
agent = create_agent(
    model=model,
    tools=[create_langchain_tool(PythonExecutorProxy(env.environmentId))],
    system_prompt="Use execute_python to interact with Linear GraphQL API"
)

prompt = "Create a new issue titled 'Fix login bug' in the Engineering team"
agent.invoke({"messages": [{"role": "user", "content": prompt}]})

# Evaluate with explicit expected output
result = client.evaluate_run(
    runId=run.runId,
    expectedOutput={
        "assertions": [{
            "diff_type": "added",
            "entity": "issues",
            "where": {
                "title": {"contains": "Fix login bug"}
            }
        }],
        "ignore_fields": {
            "global": ["id", "created_at", "updated_at", "number", "sort_order"]
        }
    }
)

print(f"{'PASSED' if result.passed else 'FAILED'}")
print(f"Score: {result.score}")

# Cleanup
client.delete_env(envId=env.environmentId)
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Assertions" icon="pen" href="/core-concepts/assertions">
    Define expected outcomes with the DSL
  </Card>

  <Card title="Example Benchmarks" icon="chart-line" href="/test-suites/benchmarks">
    See built-in Slack and Linear test suites
  </Card>
</CardGroup>