Disputes
Overview
Disputes give agents and supervisors a structured way to challenge AI-generated criteria classifications they believe are incorrect.
When Birdie's AI evaluates an interaction and flags a criterion as violated, the agent or their supervisor may disagree. Disputes capture that disagreement formally: a written justification is submitted for each contested criterion, a reviewer accepts or rejects each one, and the outcome corrects the quality record.
Beyond fixing individual classification errors, disputes are a calibration signal. A criterion that receives many disputes and has a high acceptance rate is likely producing inaccurate results — and its AI instructions should be reviewed and recalibrated.
How it works
Roles
Agent / Supervisor
disputes:submit
Submit disputes for AI classifications they believe are incorrect
Internal Reviewer / Manager
disputes:manager
Review disputes; accept or reject each criterion; access the Disputes dashboard and analytics
Contact your Admin to confirm which role you have been assigned.
Submitting a Dispute
Open a dispute from the interaction's evaluation view.
Review the interaction context
The dispute screen opens as a full-screen overlay. The left column displays the original interaction with two tabs:
CONVERSATION — the full transcript with date headers and rating badge.
INFO — interaction metadata: Ingested ID, Posted at, Language, Source, Author, Sentiment, Rating.
Use this context to confirm the criteria you want to contest.
Confirm criteria and write justifications
The right column lists all pre-selected criteria as cards with checkboxes.
Check or uncheck criteria as needed.
For each checked criterion, a text area appears — write a clear justification explaining why you believe the AI finding is incorrect.
All checked criteria must have a non-empty justification before you can submit.
The footer shows how many criteria are currently selected.
You can select multiple criteria from the same interaction in a single dispute submission.
Reviewing Disputes
Internal reviewers access the dispute queue from the Disputes page.
Review each dispute
For each dispute in the queue:
Left column — dispute details including the Dispute ID, submitter information, and two tabs:
INFO — quality score, interaction ID, team, evaluation time, and other metadata.
CONVERSATION — the original interaction transcript.
Right column — the list of disputed criteria. For each criterion card you will see:
The criterion text and ID.
The submitter's justification.
Accept and Reject toggle buttons — select one for each criterion.
An optional comment field for reviewer notes.
The footer shows your progress (e.g., "2 out of 4 criteria reviewed"). The Review next button is disabled until all criteria in the current dispute have a decision.
Accepting a criterion dispute removes the AI's original violation label from that interaction. This action directly affects quality scores and AI classification history.
Editing a Reviewed Dispute
Reviewers can modify a previously reviewed dispute at any time.
Navigate to the Disputes page → Dispute History tab.
Find the dispute row and click the ⋯ (three-dot menu) → Edit.
The review modal reopens with the existing decisions pre-filled.
Adjust any decisions or reviewer comments.
Click Save changes.
Deleting a Dispute
Navigate to the Disputes page → Dispute History tab.
Click the ⋯ (three-dot menu) on the dispute row → Delete.
Confirm the action in the dialog.
Deletion is permanent and cannot be undone.
Disputes Dashboard
The Disputes page (/disputes) is the central hub for managing and analyzing contestations. It has two tabs: Dispute History and Analytics.
Dispute History Tab
A searchable, filterable table of all disputes in your organization.
Filters
Search
Filter by Dispute ID
Date range
Filter by submission date
Submitted by
Filter by the user who submitted the dispute
Reviewed by
Filter by the user who reviewed the dispute
Table Columns
Dispute ID
Unique code within your organization (e.g., DSP-847)
Criteria
Number of criteria included in the dispute
Submission date
Date the dispute was submitted
Revision date
Date the dispute was last reviewed
Submitted by
Name and email of the person who submitted it
Reviewed by
Name and email of the internal reviewer
Actions
Edit or Delete the dispute
Analytics Tab
Use this tab to identify patterns in disputes and evaluate the health of your AI calibration.
KPI Cards
Total disputes
Total disputes submitted in the selected period, with trend vs. previous period
Dispute rate
Percentage of AI evaluations that were disputed
Acceptance rate
Percentage of disputed criteria accepted by reviewers
Dispute rate: the denominator is the count of distinct interactions with at least one AI-generated criterion finding in the selected period — not total interactions or manual evaluations.
Acceptance rate: accepted criteria ÷ (accepted + rejected criteria) × 100. Pending (unreviewed) criteria are excluded.
Charts
Dispute rate by criteria — horizontal bar chart showing dispute volume and rate per criterion. Sortable by name or count.
Accepted vs. Rejected — grouped vertical bar chart comparing accepted and rejected decisions per criterion across the selected period.
All Disputed Criteria Table
Criteria
Criterion name — click to open the details drawer
Collection
The collection this criterion belongs to
Count
Total AI evaluations for this criterion
Disputes count
Number of times this criterion has been disputed
Dispute rate
Disputes count ÷ Count × 100
Accepted count
Number of disputes accepted for this criterion
Accept rate
Accepted count ÷ Disputes count × 100
Criterion Details Drawer
Clicking a row in the analytics table opens a side drawer with five tabs:
DISPUTE
Summary metrics for this criterion: dispute count, dispute rate, accepted count, accept rate
OVERVIEW
Criterion name, description, related reason, collection, and weight — read-only
FILTERS
Criterion-level filter configuration
ADVANCED SETTINGS
Advanced criterion settings
AI INSTRUCTIONS
The AI-generated subject and refined description used for this criterion
The drawer footer includes a "Duplicate to recalibrate" action, which creates a copy of the criterion for adjustment when dispute patterns suggest the AI instructions need refinement.
Using Disputes as a Calibration Signal
Disputes are more than a correction mechanism — they are the clearest signal you have that a criterion's AI instructions may need to be updated.
When to consider recalibration:
A criterion consistently receives a high number of disputes, and
A large share of those disputes are accepted by reviewers.
This pattern indicates the AI is reliably producing outcomes that human reviewers consider wrong. The problem is usually in the criterion's instructions — the AI instructions may be too broad, too strict, or based on examples that no longer reflect the reality of your interactions.
What to do:
Open the Analytics tab and sort the disputes table by Dispute count or Accept rate.
Identify criteria with high dispute counts and high acceptance rates.
Click the criterion to open the details drawer and review the AI INSTRUCTIONS tab.
Use the "Duplicate to recalibrate" action to create a copy for adjustment without affecting live evaluations.
Update the criterion instructions and monitor the dispute rate after the new version activates.
Business Rules
A dispute must contain at least one criterion. Empty disputes cannot be submitted.
Each disputed criterion requires a written justification. The submit button stays disabled until all selected criteria have a non-empty reason.
A dispute has two possible statuses: Pending (awaiting review) or Reviewed (a decision has been recorded).
Each criterion within a dispute is independently marked as Accepted or Rejected by the reviewer.
When a criterion dispute is accepted, Birdie removes the AI's original classification label from that interaction.
Disputes can be edited after review. Reviewers can reopen and change their decisions or comments at any time from the Dispute History tab.
Disputes are scoped to your organization — each dispute receives a unique code (e.g.,
DSP-001) within your organization.
Permissions
Submit a dispute
disputes:submit
View the Disputes page and analytics
disputes:manager
Review, accept, or reject disputes
disputes:manager
Edit a reviewed dispute
disputes:manager
Delete a dispute
disputes:manager
FAQs
Why is the Submit button disabled?
The Submit button stays disabled until every checked criterion has a written justification. Make sure all selected criteria have a non-empty reason filled in.
Can I dispute more than one criterion in a single submission?
Yes. You can select multiple criteria from the same interaction and submit them together as one dispute.
What happens when a dispute is accepted?
When a reviewer accepts a criterion dispute, Birdie removes the AI's original classification from that interaction. This directly corrects the quality record and affects any score computed from that criterion.
Can a reviewer change their decision after submitting a review?
Yes. Reviewers can edit any previously reviewed dispute from the Dispute History tab using the Edit action.
Who can see the Disputes page?
Only users with the disputes:manager permission can access the Disputes page and analytics. Users with disputes:submit can submit disputes from the interaction view but do not have access to the dashboard.
Can I filter disputes by status?
Yes. Use the filters in the Dispute History tab. The status (Pending vs. Reviewed) is visible in each row.
Related Articles
Last updated