Oral Argument Transcript Hallucination Detector
System / Product: Hallucination check for Whisper transcripts of oral argument audio (transcription_was_hallucinated)
Model Type: Traditional ML (hand-fitted linear regression with a fixed decision threshold)
Version: v1.0
Date: 2026-11-12
Owner: Case Law team
Status: [x] Production [ ] Deprecated
1. Purpose
1.1 What this model does
CourtListener transcribes oral argument audio with OpenAI's whisper-1 speech-to-text model (cl.audio.tasks.transcribe_from_open_ai_api). Sometimes Whisper "hallucinates": it stops transcribing the audio and repeats one phrase over and over, usually the case name we send as the prompt (see #7473, where a transcript was mostly "U.S. Attorney General" on a loop).
This model is a binary classifier that runs on each new transcript. It flags transcripts that have far fewer unique words than an audio of that length normally has. It takes two inputs:
audio.duration: the length of the audio in seconds, as reported by Whisper (rounded up).audio.stt_transcript: the transcript text. The feature is the number of unique space-separated tokens,len(set(stt_transcript.split(" "))).
It returns True (hallucinated) or False. The rule is:
expected_unique_words = 376 + 0.435 * duration
hallucinated = duration >= 500 and unique_words <= 0.5 * expected_unique_words
In words: the transcript is flagged when it has half or fewer of the unique words that the regression line predicts for that duration.
The output is not shown to users as text. It sets Audio.stt_status:
False:STT_COMPLETE(1). The transcript is indexed and shown.True:STT_HALLUCINATION(3, label "Transcription does not match audio"). The transcript text and Whisper metadata are still saved, but they are not indexed in search or shown on the oral argument page.
The model does not fix hallucinations, does not find which segments are wrong, and does not judge transcript accuracy beyond this one failure mode.
1.2 Intended users
There are no direct users. The flag controls what CourtListener shows to everyone who uses oral argument data:
- Website: the oral argument page (
cl/audio/views.py) shows transcript segments only whenstt_status == STT_COMPLETE. Otherwise it says "Transcript data is not available." - Search: only
STT_COMPLETEtranscripts are indexed in Elasticsearch (cl/search/documents.py), and only they are used for search snippets (cl/lib/elasticsearch_utils.py). - REST API (v4) and bulk data:
stt_statusandstt_transcriptare exposed as-is. API users get the flag and the flagged text, so it is up to them to filter onstt_status. - Operations: engineers re-run flagged audio with
manage.py transcribe --status-to-process 3(Section 6.1).
1.3 Scope and limitations
- In scope: English oral argument recordings from federal and state appellate courts, transcribed by OpenAI
whisper-1. - Out of scope:
- Audio shorter than 500 seconds. These always return
False, because there were too few short recordings to fit the model. Short hallucinated transcripts are markedSTT_COMPLETE. - Other kinds of hallucination, such as invented but varied text, wrong speakers, or a short repeated loop inside an otherwise good transcript. The docstring says these "may require other approaches".
- Other transcription engines. The coefficients were fit on
whisper-1output.STT_SELF_HOSTED_WHISPERexists as a source but this code path does not use it.
- Audio shorter than 500 seconds. These always return
2. Base Model
| Field | Details |
|---|---|
| Base model name | None. Ordinary least squares regression, lm(unique_words ~ duration) (R notation) |
| Version / checkpoint | Coefficients hard-coded in cl/audio/utils.py: intercept 376, slope 0.435, tolerance 0.5, minimum duration 500 s |
| Provider | In-house |
| License | Same as CourtListener (AGPL-3.0) |
| Link to production model location | cl/audio/utils.py transcription_was_hallucinated |
The model it checks is OpenAI whisper-1, called with language="en" and the case name as prompt. Any change to that model or its call parameters affects this model (Section 6.3).
2.1 Model Modifications
- Continued pretraining
- Finetuning
- Prompting only (generative models)
- Other: a two-parameter linear regression, fit once by hand. The tolerance (
0.5) and minimum duration (500s) were chosen by looking at a scatter plot, not by optimizing a metric.
3. Data
3.1 Pretraining Dataset (if applicable — continued pretraining only)
N/A
3.2 Training Dataset
- Source: 1,377 oral argument transcripts produced by
whisper-1during the first transcription runs in June 2024 (#4129), part of #440. - Fields used:
durationand the unique-word count ofstt_transcript. - Selection: not documented. These were the transcripts available when the model was built, so they reflect the courts and dates that were transcribed first.
- Cleaning: none recorded. Hallucinated transcripts were included in the fit. They pull the line down slightly, which makes the threshold a little more lenient.
- Availability: the dataset was not saved. It can be rebuilt approximately from production (
Audiorows withstt_source=1created before 2024-06-22), but transcripts that were later re-run have changed since.
3.3 Prompt Design and Versioning (if applicable — generative models only)
N/A for this model. One related prompt choice in the Whisper call: when an audio is re-run because it was flagged (stt_status == STT_HALLUCINATION), the case name prompt is left out (prompt=omit). Manual testing showed this gives a clean transcript in many cases (#4137).
3.4 Validation and Test Dataset
There is no held-out validation or test set. The threshold was checked by eye on a scatter plot of the training data, with the decision boundary drawn in red (#4129 comment):

Automated tests (cl/audio/tests.py, test_hallucination_detection) cover two hand-picked cases:
- A real hallucinated transcript (audio 92035, 1,779 s, "I. THE UNCANNY COUNTESS" repeated) must return
True. - A normal transcript must return
False.
Known gaps:
- No labeled ground truth, so precision and recall are unknown.
- The data was not split by court, era, recording quality, or duration bucket.
- The data was not saved, so the fit cannot be reproduced exactly.
3.5 Label Documentation (if applicable — traditional ML and finetuned models only)
The regression is unsupervised with respect to the hallucination label: it predicts unique-word count, not "hallucinated". No transcripts were labeled. Points far below the line were judged to be hallucinations by looking at them, according to the author ("Those points below the red line would be tagged as hallucinations, and they indeed are", #4129).
4. Training & Evaluation
4.1 Design Decisions and Rationale
- Unique words vs. duration. In normal speech, vocabulary grows steadily with recording length, and in this data the relationship was "linear and very stable". A loop of one phrase keeps the unique-word count almost flat no matter how long the audio is, so these transcripts fall far below the line.
- A line instead of a set of
ifrules. A line with a tolerance band does the same job as several duration/word-count rules but is shorter to define and explain (#4129). - Tolerance of 50%. This is a deliberate trade-off. The author noted some false negatives at 0.5 and said a tighter tolerance would risk false positives. The open question at the time was which error is worse: a hallucination shown as a good transcript, or a good transcript hidden. 0.5 favors not hiding good transcripts.
- Minimum duration of 500 s. Few recordings in the data were this short, so the line is not reliable there. These audios are never flagged.
- Cheap and deterministic. It runs in the same database transaction that saves the transcript, with no external calls.
- Whitespace tokenization.
split(" ")is case-sensitive and keeps punctuation attached ("Court" and "court." count as different words). This adds noise but was accepted for simplicity. The same tokenization was used to fit and to predict, so the coefficients match it.
4.2 Metrics and Evaluation
No quantitative metrics (precision, recall, F1) have been computed, because there are no labels.
Production flag rate (CourtListener API, 2026-11-12):
stt_status |
Count |
|---|---|
STT_COMPLETE (1) |
103,550 |
STT_HALLUCINATION (3) |
724 |
| All audio | 104,569 |
About 0.7% of transcribed audio is currently flagged. These counts are after re-runs, so audios that were flagged and then fixed by a re-run count as complete. The most recent flagged audios (for example 107410, 1,884 s) show the model is still catching new cases.
TODO: Build a labeled sample (suggested: about 200 flagged and 200 unflagged transcripts, stratified by court and duration, including some under 500 s) and report precision and recall. Without it, the false negative rate is unknown.
4.3 Failure Analysis
Known and expected failure modes:
- False negatives (hallucination missed):
- Audio under 500 s is never checked.
- Partial hallucinations, where the audio is transcribed correctly for a while and then loops, can stay above 50% of the expected vocabulary. The test example for audio 92035 is a mix of a loop and invented text.
- Varied but invented text ("This location, some of them are just peculiar people who have never heard of chiseled fish...", audio 92035) adds unique words and hides the loop.
- The author saw some false negatives on the training plot at the 0.5 tolerance.
- False positives (good transcript hidden):
- Long recordings with a lot of silence, music, or recess time, or recordings where only a small part is speech, have fewer unique words than expected.
- Short arguments with an unusually small vocabulary.
- Re-run failures: if a flagged audio is still flagged after a re-run without the prompt,
transcribe_from_open_ai_apilogs"Audio %s: hallucination was not corrected"and leaves it asSTT_HALLUCINATION.
No systematic review of flagged audio has been recorded.
5. Known Limitations
- Flagged text is still public through the API.
stt_transcriptandAudioTranscriptionMetadataare saved for flagged audio and returned by the REST API and bulk data. API users must checkstt_statusthemselves. - Fixed coefficients. The line was fit on 2024 data and never refit. Changes in courts, recording quality, or the Whisper model may move the true line.
- Duration comes from Whisper.
audio.durationis set fromtranscript.duration. If Whisper reports a wrong duration, the check uses it as-is. - One failure mode only. It catches repetition loops, not wrong words, wrong speakers, or invented text with normal vocabulary. A
STT_COMPLETEstatus does not mean the transcript is accurate. The site keeps its general disclaimer: "Transcripts generated by AI. Please verify by listening." - Undocumented threshold trade-off. The 0.5 tolerance was chosen by eye. Its false negative and false positive rates are unknown.
- Potential bias: courts with poor recording quality, long silences, or a very different speaking style (for example very short arguments) are more likely to be misclassified. Not measured.
- Hallucination risk: this model is a mitigation for Whisper hallucinations. Its own output is a deterministic formula, so it cannot hallucinate.
6. Deployment and Monitoring
6.1 Deployment Setup and Data Dependencies
- Where it runs: inside the Celery task
cl.audio.tasks.transcribe_from_open_ai_api, in the same transaction that saves the Whisper result. Code:cl/audio/utils.py. - Upstream: audio scrapers (Juriscraper oral argument sites) create
Audiorows.reenqueue_pending_audio_tasksandmanage.py transcribesend them to Whisper. - Downstream:
Audio.stt_statuscontrols Elasticsearch indexing (cl/search/documents.py), search snippets (cl/lib/elasticsearch_utils.py), and the oral argument page (cl/audio/views.py, changed in #7487 to hide flagged transcripts). - Retry flow: flagged audio is not retried automatically. An engineer runs
manage.py transcribe --status-to-process 3. That call sends no case name prompt. If the new transcript passes, it becomesSTT_COMPLETE. A newAudioTranscriptionMetadatarow is created on every run, and the page uses the latest one.
6.2 Monitoring Plan
Current signals:
- Error log
"Audio %s: hallucination was not corrected"when a re-run is still flagged. stt_status=3count through the API or database. There is no dashboard or alert.- User reports, such as #7473.
Suggested:
- Track the share of new transcripts flagged per month. A sudden change suggests Whisper or recording changes.
- Review a few flagged and unflagged transcripts each quarter.
6.3 Re-evaluation Criteria and Process
Triggers for unscheduled re-evaluation:
- Any change to the transcription model (for example, moving off
whisper-1or to a self-hosted Whisper) or its call parameters (prompt, language). - A clear change in the flag rate, or user reports of looping transcripts that were not flagged.
- Adding audio from new courts or sources with different recording conditions.
- A labeled sample is built (Section 4.2). Refit the coefficients and tune the tolerance on it.
Scheduled cadence: To be decided (suggested: yearly).
7. Version History
| Version | Date | Author | Summary of changes |
|---|---|---|---|
| v1.0 | 2024-06-22 | grossir | Initial release with STT_HALLUCINATION status (#4129) |
| v1.0 | 2024-06-26 | grossir | Re-runs of flagged audio drop the case name prompt (#4137). Model unchanged. |
| v1.0 | 2026-06-24 | grossir | Flagged transcripts hidden from the oral argument page and search snippets (#7487). Model unchanged. |
8. Output Disclosure
This model's output does not meet the disclosure definition. It produces no content for users. It only decides whether AI-generated content (the Whisper transcript) is shown. The transcript itself is already labeled as AI-generated on the site ("Transcripts generated by AI. Please verify by listening.") and through stt_source in the API.
Flagged audio shows the same "Transcript data is not available." message as audio that was never transcribed. This was a product decision in #7473: for end users, the reason does not matter.
9. Contacts
| Role | Name | Contact |
|---|---|---|
| Feature owner | Case Law team | bill@free.law |
| Model author | Gianfranco Rossi (grossir) | gianfranco@free.law |
| AI team contact | Rachel Gao | rachel@free.law |