Model Card — RECAP Opinion Case Name Extractor

System / Product: Case Name Extractor for RECAP documents imported as opinions (classify_case_name_by_llm)
Model Type: Out-of-the-box LLM
Version: v1.0
Date: 2026-11-02
Owner: Case Law team
Status: [x] Production [ ] Deprecated

1. Purpose

1.1 What this model does

When a RECAP document (a PACER filing, usually from a district court) is imported into the case law database as an opinion, the new OpinionCluster gets its case names from the docket. Docket titles often differ from the caption on the document itself: they are truncated, all caps, list consolidated cases, or are not the right name for that particular decision.

This model reads the first 2,000 characters of the extracted opinion text and returns two strings:

case_name_full: the case caption copied verbatim from the document. Connectors such as "vs" or "against" are normalized to "v.", and ALL CAPS is converted to normal capitalization.
case_name: a Bluebook-style short case name derived from case_name_full. For example, it keeps only the first party on each side, uses surnames only, and drops "et al.", "Inc." and "of America".

The output is structured (JSON validated against a Pydantic model). case_name_full is meant to be extractive; case_name is a rule-based abstractive transformation of it. Both values overwrite OpinionCluster.case_name_full and OpinionCluster.case_name.

The model does not classify whether the document is an opinion, does not set case_name_short, and does not change the docket's case names.

1.2 Intended users

The output goes into CourtListener's case law database, so it reaches everyone who reads that data: the website (opinion pages, search results, citation display), the REST API, bulk data exports and replication customers. It is also indexed in Elasticsearch, so it affects case name search.

The original docket-derived names stay on the Docket (case_name, case_name_full), but the cluster keeps no record of its pre-LLM values or of the fact that an LLM produced them. See Section 8.

1.3 Scope and limitations

In scope: English-language federal court documents from PACER, mostly district court orders and memorandum opinions, imported through recap_document_into_opinions (triggered by scrape_pacer_free_opinions and the recap_into_opinions command).
Out of scope:
- Documents with no usable extracted text (empty plain_text is skipped).
- Documents whose caption is not in the first 2,000 characters, such as those with long cover pages or a header that comes after a long preamble.
- Deciding whether a document is an opinion. Upstream code filters documents by citation count and hash checks, not by this model.
- Short case names (case_name_short) and party or attorney extraction.
- State courts, appellate courts and other ingestion paths. The prompt was only tuned on RECAP district court documents.
Not run in local development (DEVELOPMENT=True and not TESTING), because the dev API key has no quota.

2. Base Model

Field	Details
Base model name	`gpt-4o-mini`
Version / checkpoint	`gpt-4o-mini` (unpinned alias; resolves to OpenAI's current snapshot)
Provider	OpenAI, called through the `instructor` library (`openai/gpt-4o-mini`)
License	OpenAI API terms of use (proprietary)
Link to production model location	https://platform.openai.com/docs/models/gpt-4o-mini

⚠️ OpenAI is deprecating gpt-4o-mini in January (see CL#8200). Replacing it triggers a re-evaluation (Section 6.3).

2.1 Model Modifications

Continued pretraining
Finetuning
Prompting only (generative models): no weights modified. Behavior is shaped by the system prompt in Section 3.3.
Other

3. Data

3.1 Pretraining Dataset (if applicable — continued pretraining only)

N/A

3.2 Finetuning Dataset (if applicable — finetuned models only)

N/A

3.3 Prompt Design and Versioning (if applicable — generative models only)

Location: cl/corpus_importer/prompts/system.py (CASE_NAME_EXTRACT_SYSTEM), versioned in git. It has not changed since it was introduced in #6290 (2025-10-03).

Call parameters (cl/corpus_importer/tasks.py, cl/lib/llm.py):

Parameter	Value
System prompt	`CASE_NAME_EXTRACT_SYSTEM`
User prompt	`sub_opinion.plain_text[:2000]` (raw text only; no docket name or other metadata)
Response model	`CaseNameExtractionResponse` (`case_name: str \| None`, `case_name_full: str \| None`)
Temperature	0.0
Max completion tokens	300

Design: The prompt was written by the Case Law team through repeated manual runs on recent RECAP-into-opinion candidates (see #3790). It has two parts:

Part I, caption extraction: about 24 rules aimed at failure modes seen during development. Return the caption verbatim; never swap or flip parties; do not shorten "United States of America"; keep titles, aliases and capacity language; join multiple cases with ;; do not include docket numbers or court names; normalize ALL CAPS only after extraction.
Part II, Bluebook shortening: rules adapted from Bluebook Rule 10.2. It adds special handling for Social Security cases, where claimants are named by first name and last initial (e.g. William Z. v. Bisignano), and for compound or Spanish surnames (use the last surname, keep hyphenated blocks).

The prompt uses zero-shot instructions; it has no few-shot examples. Few-shot examples and a shorter, restructured prompt were suggested during review (#3790). An alternative prompt was tested on about 1,000 documents but was not adopted (Section 4.1).

Red-teaming: None recorded.

3.4 Validation and Test Dataset

There is no formal held-out test set.

During development (#3790), the prompt was run on the most recent ~1,000 RECAP documents that were imported, or eligible to be imported, as opinions. The results were compared by hand against the docket case name. Each prompt variant was run twice to check consistency: results differed on about 5–10 of the 1,000 documents between runs. The sample is attached to this #3790 comment: data_sample.csv (1,006 rows; columns recap_id, docket_name, plain_text, plus the model's outputs under the old 7-field schema: is_opinion, case_name_short, case_name, case_name_full, case_name_match, needs_ocr, error). These are model predictions, not hand-verified labels. The output from flooie's earlier test run was never attached.

Known gaps:

No ground-truth labels for case_name or case_name_full, so accuracy cannot be computed.
The sample is a single recent time window, so it may under-represent older documents, OCR-extracted text and less common courts.
The sample was not stratified by court, document type (order or memorandum opinion), case type (for example Social Security, prisoner, bankruptcy appeal) or OCR status.
Leakage is not a concern because no training or finetuning was done. The prompt rules were tuned on the same documents they were judged on, though, so the informal results are likely optimistic.

TODO (#8325): Build a labeled set (suggested: 300–500 clusters stratified by court and case type, with hand-verified case_name and case_name_full) before evaluating the replacement model.

3.5 Label Documentation (if applicable — traditional ML and finetuned models only)

N/A

4. Training & Evaluation

4.1 Design Decisions and Rationale

Use an LLM instead of heuristics. Docket titles can't be cleaned reliably with heuristics. Inferring names from the defendants in the document description was considered and judged hard to get right (#3790). An LLM reading the caption on the document itself handles the variety of district court caption layouts.
Model: gpt-4o-mini. Chosen for low cost and latency. The task runs once per imported document, at scraper volume. Larger or reasoning models (GPT-5) were discussed but not adopted. They would need different prompting.
First 2,000 characters only. The caption is almost always at the top of the document. This keeps cost per call low.
Structured output through instructor and Pydantic. Gives a typed response. Both fields are optional strings, so validation only catches malformed or wrongly typed output, not wrong names. On a validation failure, instructor re-asks the model (production Sentry shows up to 4 attempts). If it still fails, the task fails and leaves the cluster unchanged (see Section 5 on error handling).
Prompt choice. Two prompts were compared on the same ~1,000 documents. In that run, the case-law team's prompt (then also asking for is_opinion) flagged ~673 documents as opinions; the alternative flagged ~225. The team's prompt was kept because it had been through more rounds of iteration. The is_opinion and related fields were later dropped from the response model, so the production model only extracts names.
Overwrite instead of storing a separate field. The LLM output replaces the cluster's case names directly. No separate "LLM-derived" field or provenance flag was added.

4.2 Metrics and Evaluation

No quantitative metrics (exact match or similar) have been recorded. Evaluation so far has been qualitative: a person compared outputs against the document caption and the docket name on the ~1,000-document sample, plus a run-to-run consistency check (about 0.5–1% of documents differed between identical runs).

Issue tracking pending work: #8325):

4.3 Failure Analysis

Failure modes seen during development, which explain many of the prompt's rules:

Parties swapped or the case name inverted.
Truncated captions, with dropped parties, official titles or "of America".
Surnames that change between case_name_full and case_name (typos, wrong token picked from compound names).
Social Security captions reduced to a single initial.
Docket numbers or court names mixed into the case name.
Consolidated cases collapsed into one name.

Sentry issues observed in production (Sentry, last 90 days as of 2026-11-02) and needing further analysis are tracked here 8324

5. Known Limitations

Overwrites without provenance: OpinionCluster.case_name and case_name_full are replaced, and the previous values and the fact that they came from an LLM are not recorded on the cluster. The docket keeps its original names, which is the only way to recover them.
Partial responses blank a field: if only one of case_name or case_name_full comes back, the other is saved as "".
No grounding check: output isn't checked against the source text, so a hallucinated or mangled caption is saved as-is.
Output token cap truncates long captions: max_completion_tokens=300 is too low for some captions. 18 production events were truncated (see Section 4.3), and those clusters keep their docket names.
Limited context: only the first 2,000 characters are sent. Captions further down are missed, and the model may pick up a cited case's name instead.
OCR quality: OCR-extracted text (extracted_by_ocr=True) may hold garbled captions, and the prompt asks the model to "fix" capitalization, which can hide OCR errors.
Unpinned model alias: gpt-4o-mini is not pinned to a dated snapshot, so OpenAI can change its behavior without any change on our side.
Bluebook coverage: the shortening rules are a subset of Bluebook Rule 10. Edge cases (unions, mandamus, popular names, in rem cases) depend on the model's general knowledge and have not been tested.
Potential bias:
- The surname rules ("use the last surname") assume Western or Spanish naming order. Names from cultures that put the family name first, or that have no surname, may be shortened incorrectly.
- Pro se and Social Security claimants identified only by first name and initial are more likely to be mishandled. These are often more vulnerable litigants.
- Mitigation so far: explicit prompt rules for Social Security and compound surnames. Not measured.
Hallucination risk: the main risk is names that look plausible but are wrong (swapped parties, misspelled surnames, invented parties when the caption is missing). Mitigations: temperature 0; schema validation; prompt rules telling the model not to hallucinate and to check surnames. There is no automated detection.
Error handling doesn't match the library's behavior: instructor 1.17.0 (instructor/v2/core/retry.py) catches every exception from the API call and from parsing except IncompleteOutputException, and re-raises it as InstructorRetryException, keeping the original as __cause__. InstructorRetryException doesn't subclass pydantic.ValidationError or any OpenAI error. As a result:
- the task's except ValidationError branch (log, then return) never runs. Validation failures fall through to except Exception, are sent to Sentry, and make the task fail.
- Celery autoretry_for never matches, so rate limits and OpenAI outages don't get Celery's 5 backoff retries. The cluster keeps its docket-derived names, and nothing re-queues the task.
- The tests (cl/corpus_importer/tests.py) mock call_llm and raise raw RateLimitError / ValidationError, so they pass without exercising the real behavior.
- Verified locally against a fake OpenAI server: both a 429 and a schema-invalid response came back as InstructorRetryException, with __cause__ of RateLimitError and ValidationError respectively.
- Fix (outside this card): in call_llm or the task, catch InstructorRetryException and branch on __cause__, or have the task retry on InstructorRetryException.
Stale docstring: the task docstring says the task "verifies if the document is an opinion". It no longer does, since is_opinion was removed from the response model.

6. Deployment and Monitoring

6.1 Deployment Setup and Data Dependencies

Where it runs: Celery task cl.corpus_importer.tasks.classify_case_name_by_llm in the CourtListener repo. It calls OpenAI through cl.lib.llm.call_llm with the OPENAI_CASE_LAW_INFERENCE_KEY API key.
Upstream: recap_document_into_opinions creates the OpinionCluster and Opinion (copying names from the docket), then queues this task. The opinion text comes from Doctor extraction (plain_text, possibly OCR). Callers: the scrape_pacer_free_opinions daily scraper chain and the recap_into_opinions backfill command.
Downstream: updated cluster names feed the Elasticsearch index (through signals), opinion pages, the REST API (v4), bulk data and replication.
Retries: the task declares Celery autoretry_for (5 retries, exponential backoff) on OpenAI RateLimitError, APIConnectionError, ConflictError, InternalServerError and APIError. This doesn't work in production: instructor wraps those errors in InstructorRetryException, so Celery never retries the task (see Section 5). The only retries that happen are instructor's single re-ask and the OpenAI SDK client's own built-in retries.
Labeling of AI output: none at present (see Section 8).

6.2 Monitoring Plan

Current signals (Sentry and logs):

LLM - Response validation error (fingerprint llm-casenames-validation-error): never fires in production. Validation failures arrive wrapped in InstructorRetryException (see Section 5). They show up instead as an InstructorRetryException Sentry issue with culprit classify_case_name_by_llm. Example: COURTLISTENER-EG1, 2 events, 2026-09-29 to 2026-09-30, now resolved.
LLM - No case name returned: both fields empty. The log includes the raw response and the cluster's current names.
Unexpected instructor and OpenAI exceptions are sent to Sentry with capture_exception.
Successful updates are logged at INFO with the cluster URL.
Current error volume per issue is in Section 4.3.

Again, this part depends on further analysis of (#8324)

6.3 Re-evaluation Criteria and Process

Triggers for unscheduled re-evaluation:

Base model change or deprecation. This applies now: gpt-4o-mini is being deprecated (CL#8200).
Any change to CASE_NAME_EXTRACT_SYSTEM, the input window (2,000 chars), max_completion_tokens or the response schema.
Changes upstream in Doctor text extraction or OCR, or new ingestion paths (other courts, state sources).
A sustained rise in validation errors or empty responses, or user reports of wrong case names on RECAP-sourced opinions.

Scheduled cadence: To be decided (suggested: yearly, or whenever OpenAI announces a deprecation).

7. Version History

Version	Date	Author	Summary of changes
v1.0	2025-10-03	quevon24 (implementation), flooie (prompt)	Initial release (#6290)

8. Output Disclosure

This model's output meets the disclosure definition. The case names it produces replace the source data (the docket title) and are shown as the case's name across the site, the API and bulk data. Users rely on them to identify and cite the case without checking the document caption.

Current state: not disclosed. Nothing in the UI, API or bulk data shows that a cluster's case name was produced by an LLM, and the cluster has no provenance field. This is a gap against the AI policy.

Proposed remediation (TODO, to be discussed in #8321):

9. Contacts

Role	Name	Contact
Feature owner	Case Law team	bill@free.law
AI team contact	Rachel Gao	rachel@free.law