> For the complete documentation index, see [llms.txt](https://ask.birdie.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ask.birdie.ai/frontline-intelligence/disputes.md).

# Disputes

## Overview

Disputes give agents and supervisors a structured way to challenge AI-generated criteria classifications they believe are incorrect.

When Birdie's AI evaluates an interaction and flags a criterion as violated, the agent or their supervisor may disagree. Disputes capture that disagreement formally: a written justification is submitted for each contested criterion, a reviewer accepts or rejects each one, and the outcome corrects the quality record.

Beyond fixing individual classification errors, disputes are a **calibration signal**. A criterion that receives many disputes and has a high acceptance rate is likely producing inaccurate results — and its AI instructions should be reviewed and recalibrated.

***

## How it works

{% stepper %}
{% step %}

### An agent or supervisor identifies an incorrect classification

After reviewing an interaction where the AI flagged a criterion violation they disagree with, they open a dispute directly from the evaluation view.
{% endstep %}

{% step %}

### They write a justification for each contested criterion

The dispute screen lets them confirm which criteria to contest and write a clear explanation for each. All selected criteria must have a written justification before submitting.
{% endstep %}

{% step %}

### A reviewer accepts or rejects each criterion

An internal reviewer goes through the dispute queue, reads the interaction and justifications, and records a decision — accepted or rejected — for each criterion individually.
{% endstep %}

{% step %}

### Accepted disputes correct the quality record

When a criterion dispute is accepted, Birdie removes the AI's original classification from that interaction. The decision becomes part of the quality record and affects any score computed from that criterion.
{% endstep %}
{% endstepper %}

***

## Roles

| Role                            | Permission         | What they can do                                                                              |
| ------------------------------- | ------------------ | --------------------------------------------------------------------------------------------- |
| **Agent / Supervisor**          | `disputes:submit`  | Submit disputes for AI classifications they believe are incorrect                             |
| **Internal Reviewer / Manager** | `disputes:manager` | Review disputes; accept or reject each criterion; access the Disputes dashboard and analytics |

{% hint style="info" %}
Contact your Admin to confirm which role you have been assigned.
{% endhint %}

***

## Submitting a Dispute

Open a dispute from the interaction's evaluation view.

{% stepper %}
{% step %}

### Select the criteria to dispute

From the interaction's evaluation view, select one or more criteria whose AI findings you disagree with, then click **Dispute**.
{% endstep %}

{% step %}

### Review the interaction context

The dispute screen opens as a full-screen overlay. The **left column** displays the original interaction with two tabs:

* **CONVERSATION** — the full transcript with date headers and rating badge.
* **INFO** — interaction metadata: Ingested ID, Posted at, Language, Source, Author, Sentiment, Rating.

Use this context to confirm the criteria you want to contest.
{% endstep %}

{% step %}

### Confirm criteria and write justifications

The **right column** lists all pre-selected criteria as cards with checkboxes.

* Check or uncheck criteria as needed.
* For each checked criterion, a text area appears — write a clear justification explaining why you believe the AI finding is incorrect.
* All checked criteria must have a non-empty justification before you can submit.

The footer shows how many criteria are currently selected.
{% endstep %}

{% step %}

### Submit

Click **Submit dispute**. A confirmation message appears and you are returned to the previous page. The dispute is now in the queue with status **Pending**.
{% endstep %}
{% endstepper %}

{% hint style="info" %}
You can select multiple criteria from the same interaction in a single dispute submission.
{% endhint %}

***

## Reviewing Disputes

Internal reviewers access the dispute queue from the **Disputes** page.

{% stepper %}
{% step %}

### Open the review queue

Navigate to the **Disputes** page. The header shows a **"Review disputes (N)"** button indicating how many disputes are pending. Click it to open the review workflow.
{% endstep %}

{% step %}

### Select an area

In the first step of the review modal, choose the **operational area** you want to review from the searchable dropdown. Only areas with pending disputes are shown. Click **Review disputes** to proceed.
{% endstep %}

{% step %}

### Review each dispute

For each dispute in the queue:

* **Left column** — dispute details including the Dispute ID, submitter information, and two tabs:
  * **INFO** — quality score, interaction ID, team, evaluation time, and other metadata.
  * **CONVERSATION** — the original interaction transcript.
* **Right column** — the list of disputed criteria. For each criterion card you will see:
  * The criterion text and ID.
  * The submitter's justification.
  * **Accept** and **Reject** toggle buttons — select one for each criterion.
  * An optional **comment** field for reviewer notes.

The footer shows your progress (e.g., "2 out of 4 criteria reviewed"). The **Review next** button is disabled until all criteria in the current dispute have a decision.
{% endstep %}

{% step %}

### Finish the session

Once all disputes in the selected area have been reviewed, click **Finish session**. All decisions are saved and the disputes are marked as **Reviewed**.
{% endstep %}
{% endstepper %}

{% hint style="warning" %}
Accepting a criterion dispute removes the AI's original violation label from that interaction. This action directly affects quality scores and AI classification history.
{% endhint %}

***

## Editing a Reviewed Dispute

Reviewers can modify a previously reviewed dispute at any time.

1. Navigate to the **Disputes** page → **Dispute History** tab.
2. Find the dispute row and click the **⋯ (three-dot menu)** → **Edit**.
3. The review modal reopens with the existing decisions pre-filled.
4. Adjust any decisions or reviewer comments.
5. Click **Save changes**.

***

## Deleting a Dispute

1. Navigate to the **Disputes** page → **Dispute History** tab.
2. Click the **⋯ (three-dot menu)** on the dispute row → **Delete**.
3. Confirm the action in the dialog.

{% hint style="danger" %}
Deletion is permanent and cannot be undone.
{% endhint %}

***

## Disputes Dashboard

The **Disputes** page (`/disputes`) is the central hub for managing and analyzing contestations. It has two tabs: **Dispute History** and **Analytics**.

### Dispute History Tab

A searchable, filterable table of all disputes in your organization.

#### Filters

| Filter           | Description                                  |
| ---------------- | -------------------------------------------- |
| **Search**       | Filter by Dispute ID                         |
| **Date range**   | Filter by submission date                    |
| **Submitted by** | Filter by the user who submitted the dispute |
| **Reviewed by**  | Filter by the user who reviewed the dispute  |

#### Table Columns

| Column              | Description                                            |
| ------------------- | ------------------------------------------------------ |
| **Dispute ID**      | Unique code within your organization (e.g., `DSP-847`) |
| **Criteria**        | Number of criteria included in the dispute             |
| **Submission date** | Date the dispute was submitted                         |
| **Revision date**   | Date the dispute was last reviewed                     |
| **Submitted by**    | Name and email of the person who submitted it          |
| **Reviewed by**     | Name and email of the internal reviewer                |
| **Actions**         | Edit or Delete the dispute                             |

***

### Analytics Tab

Use this tab to identify patterns in disputes and evaluate the health of your AI calibration.

#### KPI Cards

| KPI                 | Description                                                                     |
| ------------------- | ------------------------------------------------------------------------------- |
| **Total disputes**  | Total disputes submitted in the selected period, with trend vs. previous period |
| **Dispute rate**    | Percentage of AI evaluations that were disputed                                 |
| **Acceptance rate** | Percentage of disputed criteria accepted by reviewers                           |

{% hint style="info" %}
**Dispute rate:** the denominator is the count of distinct interactions with at least one AI-generated criterion finding in the selected period — not total interactions or manual evaluations.

**Acceptance rate:** `accepted criteria ÷ (accepted + rejected criteria) × 100`. Pending (unreviewed) criteria are excluded.
{% endhint %}

#### Charts

* **Dispute rate by criteria** — horizontal bar chart showing dispute volume and rate per criterion. Sortable by name or count.
* **Accepted vs. Rejected** — grouped vertical bar chart comparing accepted and rejected decisions per criterion across the selected period.

#### All Disputed Criteria Table

| Column             | Description                                       |
| ------------------ | ------------------------------------------------- |
| **Criteria**       | Criterion name — click to open the details drawer |
| **Collection**     | The collection this criterion belongs to          |
| **Count**          | Total AI evaluations for this criterion           |
| **Disputes count** | Number of times this criterion has been disputed  |
| **Dispute rate**   | `Disputes count ÷ Count × 100`                    |
| **Accepted count** | Number of disputes accepted for this criterion    |
| **Accept rate**    | `Accepted count ÷ Disputes count × 100`           |

#### Criterion Details Drawer

Clicking a row in the analytics table opens a side drawer with five tabs:

| Tab                   | Content                                                                                      |
| --------------------- | -------------------------------------------------------------------------------------------- |
| **DISPUTE**           | Summary metrics for this criterion: dispute count, dispute rate, accepted count, accept rate |
| **OVERVIEW**          | Criterion name, description, related reason, collection, and weight — read-only              |
| **FILTERS**           | Criterion-level filter configuration                                                         |
| **ADVANCED SETTINGS** | Advanced criterion settings                                                                  |
| **AI INSTRUCTIONS**   | The AI-generated subject and refined description used for this criterion                     |

The drawer footer includes a **"Duplicate to recalibrate"** action, which creates a copy of the criterion for adjustment when dispute patterns suggest the AI instructions need refinement.

***

## Using Disputes as a Calibration Signal

Disputes are more than a correction mechanism — they are the clearest signal you have that a criterion's AI instructions may need to be updated.

**When to consider recalibration:**

* A criterion consistently receives a high number of disputes, **and**
* A large share of those disputes are accepted by reviewers.

This pattern indicates the AI is reliably producing outcomes that human reviewers consider wrong. The problem is usually in the criterion's instructions — the AI instructions may be too broad, too strict, or based on examples that no longer reflect the reality of your interactions.

**What to do:**

1. Open the Analytics tab and sort the disputes table by **Dispute count** or **Accept rate**.
2. Identify criteria with high dispute counts and high acceptance rates.
3. Click the criterion to open the details drawer and review the **AI INSTRUCTIONS** tab.
4. Use the **"Duplicate to recalibrate"** action to create a copy for adjustment without affecting live evaluations.
5. Update the criterion instructions and monitor the dispute rate after the new version activates.

***

## Business Rules

* A dispute must contain at least one criterion. Empty disputes cannot be submitted.
* Each disputed criterion requires a written justification. The submit button stays disabled until all selected criteria have a non-empty reason.
* A dispute has two possible statuses: **Pending** (awaiting review) or **Reviewed** (a decision has been recorded).
* Each criterion within a dispute is independently marked as **Accepted** or **Rejected** by the reviewer.
* When a criterion dispute is accepted, Birdie removes the AI's original classification label from that interaction.
* Disputes can be edited after review. Reviewers can reopen and change their decisions or comments at any time from the Dispute History tab.
* Disputes are scoped to your organization — each dispute receives a unique code (e.g., `DSP-001`) within your organization.

***

## Permissions

| Action                               | Required Permission |
| ------------------------------------ | ------------------- |
| Submit a dispute                     | `disputes:submit`   |
| View the Disputes page and analytics | `disputes:manager`  |
| Review, accept, or reject disputes   | `disputes:manager`  |
| Edit a reviewed dispute              | `disputes:manager`  |
| Delete a dispute                     | `disputes:manager`  |

***

## FAQs

**Why is the Submit button disabled?**

The Submit button stays disabled until every checked criterion has a written justification. Make sure all selected criteria have a non-empty reason filled in.

**Can I dispute more than one criterion in a single submission?**

Yes. You can select multiple criteria from the same interaction and submit them together as one dispute.

**What happens when a dispute is accepted?**

When a reviewer accepts a criterion dispute, Birdie removes the AI's original classification from that interaction. This directly corrects the quality record and affects any score computed from that criterion.

**Can a reviewer change their decision after submitting a review?**

Yes. Reviewers can edit any previously reviewed dispute from the **Dispute History** tab using the Edit action.

**Who can see the Disputes page?**

Only users with the `disputes:manager` permission can access the Disputes page and analytics. Users with `disputes:submit` can submit disputes from the interaction view but do not have access to the dashboard.

**Can I filter disputes by status?**

Yes. Use the filters in the Dispute History tab. The status (Pending vs. Reviewed) is visible in each row.

***

## Related Articles

* [Criteria](/frontline-intelligence/criteria.md)
* [Manual Evaluation](/frontline-intelligence/manual-evaluation.md)
* [Frontline Intelligence — Product Overview](/frontline-intelligence/product-overview.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://ask.birdie.ai/frontline-intelligence/disputes.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.