Quickstart

Get Agent Diff running in 5 minutes

Want to skip setup? Copy one of the Jupyter notebooks:

• Diffs Demo — Post messages, create issues, capture diffs
• Evaluations Demo — Run assertions against agent diffs
• Linear Bench — Run the 40-task benchmark from HuggingFace

Choose Your Setup

Hosted (Recommended)

Step 0: Get your API key

1. Sign up at agentdiff.dev
2. Get your API key from the dashboard

Step 1: Set up the environment variables

.env

AGENT_DIFF_API_KEY=
AGENT_DIFF_BASE_URL=https://api.agentdiff.dev

export AGENT_DIFF_API_KEY=
export AGENT_DIFF_BASE_URL=https://api.agentdiff.dev

Self-Hosted

Run Agent Diff locally with Docker

Step 1: Start the Backend (Self-Hosted)

Clone & Run

git clone https://github.com/hubertpysklo/agent-diff.git
cd agent-diff/ops
docker-compose up

Backend is now running at http://localhost:8000

Step 2: Install the SDK

Python

pip install agent-diff
# or with uv
uv add agent-diff

npm install agent-diff
# or with pnpm
pnpm add agent-diff

Step 3: Create Your First Environment

Where do these values come from? The templateName and impersonateUserId must match data in the template. See Environments & Templates for available templates and user IDs (e.g., U01AGENBOT9 is the agent bot user in slack_default).

Python (Hosted)

from agent_diff import AgentDiff

# For hosted version, pass explicility or add environment variables
client = AgentDiff(
    base_url="https://api.agentdiff.dev",
    api_key="ak_your_api_key"  # From dashboard
)

# Create isolated Slack environment
env = client.init_env(
    templateService="slack",
    templateName="slack_default",
    impersonateUserId="U01AGENBOT9",  # User your agent will act as
    ttlSeconds=3600  # Auto-cleanup after 1 hour
)

print(f"Environment: {env.environmentId}")
print(f"API URL: {env.environmentUrl}")

from agent_diff import AgentDiff

client = AgentDiff()  # Defaults to http://localhost:8000

# Create isolated Slack environment
env = client.init_env(
    templateService="slack",
    templateName="slack_default",
    impersonateUserId="U01AGENBOT9",
    ttlSeconds=3600
)

print(f"Environment: {env.environmentId}")
print(f"API URL: {env.environmentUrl}")

import { AgentDiff } from 'agent-diff';

// For hosted version
const client = new AgentDiff({
  baseUrl: 'https://api.agentdiff.dev',
  apiKey: 'ak_your_api_key'  // From dashboard
});

// Create isolated Slack environment
const env = await client.initEnv({
  templateService: 'slack',
  templateName: 'slack_default',
  impersonateUserId: 'U01AGENBOT9',
  ttlSeconds: 3600
});

console.log(`Environment: ${env.environmentId}`);
console.log(`API URL: ${env.environmentUrl}`);

import { AgentDiff } from 'agent-diff';

const client = new AgentDiff();  // Defaults to http://localhost:8000

const env = await client.initEnv({
  templateService: 'slack',
  templateName: 'slack_default',
  impersonateUserId: 'U01AGENBOT9',
  ttlSeconds: 3600
});

console.log(`Environment: ${env.environmentId}`);
console.log(`API URL: ${env.environmentUrl}`);

Step 4: Run Your Agent & Capture Diff

Python

from agent_diff import PythonExecutorProxy, create_openai_tool
from agents import Agent, Runner

# Start run (takes "before" snapshot)
run = client.start_run(envId=env.environmentId)

# Create code executor that intercepts API calls
python_executor = PythonExecutorProxy(env.environmentId, base_url=client.base_url, api_key=client.api_key)
python_tool = create_openai_tool(python_executor)

# Your agent with the tool
agent = Agent(
    name="Slack Assistant",
    instructions="""Use execute_python tool to interact with Slack API 
    at https://slack.com/api/*. Authentication is handled automatically.""",
    tools=[python_tool]
)

# Run the agent
response = await Runner.run(agent, "Post 'Hello World!' to #general")

# Get the diff (what changed)
diff = client.diff_run(runId=run.runId)

print("Inserts:", diff.diff['inserts'])  # New messages created
print("Updates:", diff.diff['updates'])  # Records modified
print("Deletes:", diff.diff['deletes'])  # Records removed

# Cleanup
client.delete_env(envId=env.environmentId)

import { TypeScriptExecutorProxy, createVercelAITool } from 'agent-diff';
import { generateText } from 'ai';
import { openai } from '@ai-sdk/openai';

// Start run (takes "before" snapshot)
const run = await client.startRun({ envId: env.environmentId });

// Create code executor that intercepts API calls
const executor = new TypeScriptExecutorProxy(env.environmentId, client.getBaseUrl());
const tool = await createVercelAITool(executor);

// Run AI with the tool
const result = await generateText({
  model: openai('gpt-4o'),
  tools: { execute_typescript: tool },
  prompt: "Post 'Hello World!' to Slack channel #general",
  maxSteps: 5
});

// Get the diff (what changed)
const diff = await client.diffRun({ runId: run.runId });

console.log('Inserts:', diff.diff.inserts);  // New messages created
console.log('Updates:', diff.diff.updates);  // Records modified  
console.log('Deletes:', diff.diff.deletes);  // Records removed

// Cleanup
await client.deleteEnv(env.environmentId);

Example Output

{
  "inserts": [
    {
      "__table__": "messages",
      "message_id": "1732645891.000200",
      "channel_id": "C01GENERAL99",
      "user_id": "U01AGENBOT9",
      "message_text": "Hello World!",
      "created_at": "2025-11-26T15:31:31"
    }
  ],
  "updates": [],
  "deletes": []
}

You’ve just run an agent against an isolated Slack replica and captured the exact changes it made!

What Just Happened?

1. Environment Created: A fresh PostgreSQL schema was created with seed data (users, channels, messages)
2. Before Snapshot: start_run captured the initial state
3. Agent Executed: Your agent’s API calls were intercepted and routed to the isolated environment
4. Diff Computed: The system compared before/after states to produce a deterministic diff

Next Steps

Example Benchmarks

See built-in evaluation suites for Slack and Linear

Environments

Understand environments and templates in depth

Run Evaluations

Add assertions and run benchmarks against your agent

Integrate with Agents

Connect your AI agent framework