Writing Assertions

Define expected outcomes for your agent tests with the evaluation DSL

Overview

Assertions describe what database changes you expect after the agent runs. They’re written in a JSON DSL that gets evaluated against the computed diff.

Schema

{
  "strict": false,
  "assertions": [
    {
      "diff_type": "added|changed|removed|unchanged",
      "entity": "table_name",
      "where": { /* predicates */ },
      "expected_count": 1,
      "expected_changes": { /* for 'changed' type */ }
    }
  ]
}

Top-Level Fields

Field	Type	Required	Description
strict	boolean	No	Fail if extra changes exist (default: false)
assertions	array	Yes	List of assertions to evaluate

Assertion Fields

Field	Type	Required	Description
diff_type	string	Yes	Type of change to check
entity	string	Yes	Database table name
where	object	No	Conditions to match records
expected_count	number/object	No	How many records should match
expected_changes	object	No	Field changes to verify (for changed)

---

Diff Types

Type	Description
added	New records were inserted
changed	Existing records were updated
removed	Records were deleted
unchanged	Records exist and weren’t modified

added - New Records

Check that new rows were inserted:

{
  "diff_type": "added",
  "entity": "messages",
  "where": {
    "message_text": {"contains": "Hello"}
  },
  "expected_count": 1
}

changed - Modified Records

Check that existing rows were updated:

{
  "diff_type": "changed",
  "entity": "issues",
  "where": {
    "id": {"eq": "issue-123"}
  },
  "expected_changes": {
    "status": {"to": {"eq": "Done"}}
  }
}

removed - Deleted Records

Check that rows were deleted:

{
  "diff_type": "removed",
  "entity": "messages",
  "where": {
    "message_id": {"eq": "1234567890.000100"}
  },
  "expected_count": 1
}

---

Predicate Operators

Equality Operators

Operator	Description	Example
eq	Equal to	{"status": {"eq": "Done"}}
neq	Not equal to	{"status": {"neq": "Archived"}}

Comparison Operators

Operator	Description	Example
gt	Greater than	{"priority": {"gt": 2}}
gte	Greater than or equal	{"priority": {"gte": 2}}
lt	Less than	{"priority": {"lt": 5}}
lte	Less than or equal	{"priority": {"lte": 5}}

String Operators

Operator	Description	Example
contains	Contains substring (case-sensitive)	{"text": {"contains": "bug"}}
not_contains	Does not contain substring	{"text": {"not_contains": "error"}}
starts_with	Starts with prefix	{"title": {"starts_with": "Fix"}}
ends_with	Ends with suffix	{"name": {"ends_with": ".js"}}

Null Operators

Operator	Description	Example
is_null	Value is NULL	{"assignee": {"is_null": true}}
not_null	Value is not NULL	{"assignee": {"not_null": true}}

Set Operators

Operator	Description	Example
in	Value is in set	{"status": {"in": ["Done", "Closed"]}}
not_in	Value is not in set	{"status": {"not_in": ["Backlog"]}}

Array Operators

Operator	Description	Example
has_any	Array contains any of	{"labels": {"has_any": ["bug", "urgent"]}}
has_all	Array contains all of	{"labels": {"has_all": ["bug", "p1"]}}

---

Combining Predicates

AND (implicit)

Multiple conditions in where are ANDed:

{
  "where": {
    "channel_id": {"eq": "C01GENERAL99"},
    "message_text": {"contains": "hello"},
    "user_id": {"eq": "U01AGENBOT9"}
  }
}

AND (explicit)

{
  "where": {
    "and": [
      {"status": {"eq": "Done"}},
      {"priority": {"lt": 3}}
    ]
  }
}

OR

{
  "where": {
    "or": [
      {"status": {"eq": "Done"}},
      {"status": {"eq": "Closed"}}
    ]
  }
}

Nested Logic

{
  "where": {
    "and": [
      {"team_id": {"eq": "eng"}},
      {"or": [
        {"priority": {"eq": 1}},
        {"labels": {"has_any": ["urgent"]}}
      ]}
    ]
  }
}

---

Expected Count

Exact Count

{"expected_count": 1}           // Exactly 1 record
{"expected_count": 0}           // No records
{"expected_count": 5}           // Exactly 5 records

Bounded Count

{"expected_count": {"min": 1}}           // At least 1
{"expected_count": {"max": 5}}           // At most 5
{"expected_count": {"min": 1, "max": 10}} // Between 1 and 10

---

Expected Changes

For diff_type: "changed", specify field transitions:

Check New Value Only

{
  "expected_changes": {
    "status": {"to": {"eq": "Done"}}
  }
}

Check Both Old and New

{
  "expected_changes": {
    "status": {
      "from": {"eq": "In Progress"},
      "to": {"eq": "Done"}
    }
  }
}

Multiple Field Changes

{
  "expected_changes": {
    "status": {"to": {"eq": "Done"}},
    "completed_at": {"to": {"not_null": true}},
    "updated_at": {"to": {"not_null": true}}
  }
}

---

Strict Mode

With strict: true, the evaluation fails if there are changes beyond what’s asserted:

{
  "strict": true,
  "assertions": [{
    "diff_type": "changed",
    "entity": "issues",
    "expected_changes": {
      "status": {"to": {"eq": "Done"}}
    }
  }]
}

If strict: true and the agent also changed priority, the assertion fails even if status was correctly changed.

---

Ignore Fields

Ignore fields let you exclude certain fields when comparing diffs. This is useful for:

• Auto-generated fields (id, created_at, updated_at)
• Timestamps that change on every run
• Fields set by the system, not the agent

Using Ignore Fields

{
  "assertions": [{
    "diff_type": "added",
    "entity": "messages",
    "where": {
      "message_text": {"contains": "Hello"}
    }
  }],
  "ignore_fields": {
    "global": ["created_at", "updated_at", "message_id", "ts"]
  }
}

Common Ignore Fields by Service

Slack:

{
  "ignore_fields": {
    "global": ["ts", "message_id", "created_at", "updated_at", "edited_at"]
  }
}

Linear:

{
  "ignore_fields": {
    "global": ["id", "created_at", "updated_at", "sort_order", "number"]
  }
}

Entity-Specific Ignore Fields

You can ignore fields for specific tables:

{
  "ignore_fields": {
    "global": ["created_at", "updated_at"],
    "messages": ["ts", "message_id"],
    "issues": ["number", "sort_order"]
  }
}

Without ignore_fields, auto-generated timestamps and IDs would cause assertion failures even when the agent performed the correct action.

---

Complete Examples

Slack: Post Message to Channel

{
  "assertions": [{
    "diff_type": "added",
    "entity": "messages",
    "where": {
      "channel_id": {"eq": "C01GENERAL99"},
      "message_text": {"contains": "Hello"},
      "user_id": {"eq": "U01AGENBOT9"}
    },
    "expected_count": 1
  }]
}

Slack: Add Reaction

{
  "assertions": [{
    "diff_type": "added",
    "entity": "reactions",
    "where": {
      "message_id": {"eq": "1234567890.000100"},
      "reaction_name": {"eq": "thumbsup"}
    },
    "expected_count": 1
  }]
}

Linear: Create Issue with Label

{
  "strict": true,
  "assertions": [
    {
      "diff_type": "added",
      "entity": "issues",
      "where": {
        "and": [
          {"title": {"contains": "Fix"}},
          {"team_id": {"eq": "team-eng"}},
          {"priority": {"lte": 2}}
        ]
      },
      "expected_count": 1
    },
    {
      "diff_type": "changed",
      "entity": "teams",
      "where": {
        "id": {"eq": "team-eng"}
      },
      "expected_changes": {
        "issue_count": {
          "from": {"eq": 5},
          "to": {"eq": 6}
        }
      }
    }
  ]
}

Linear: Update Issue Priority

{
  "assertions": [{
    "diff_type": "changed",
    "entity": "issues",
    "where": {
      "title": {"contains": "login bug"}
    },
    "expected_changes": {
      "priority": {"to": {"eq": 1}}
    }
  }]
}

---

Table Names by Service

Slack

Table	Description
users	User accounts
channels	Channels and DMs
channel_members	Channel membership
messages	Messages
reactions	Emoji reactions

Linear

Table	Description
issues	Issues/tickets
teams	Teams
users	User accounts
comments	Issue comments
labels	Labels
workflow_states	Workflow states
projects	Projects
cycles	Cycles/sprints

---

Next Steps

Example Benchmarks

See built-in Slack and Linear test suites

Creating Test Suites

Organize and manage your tests