Reply Review Judge
System / Product: Litigant Portal
Model Type: Prompted language model for structured review
Version: 1.0
Date: 2026-11-18
Owner: Free Law Project, individual owner pending confirmation
Status: Draft. This is an optional site feature. Live enablement awaits confirmation.
1. Purpose
1.1 What this model does
Before a final chat answer appears, an optional second model reviews the assistant’s draft against the conversation and supplied court material. It approves the draft or returns findings for the assistant to address. The assistant writes the revised answer. Judge instructions and verdict · Review loop
1.2 Intended users
The chat service consumes the verdict. The litigant sees the released answer and, in certain failure cases, a warning. Staff with audit access can inspect the internal verdict and rejected drafts. Transcript audit guide
1.3 Scope and limitations
The judge reviews text-only evidence supplied to the current turn. It cannot inspect the original image or document block, look up the law, or verify a court action. Prose sent alongside a tool call can be displayed without this review. The separately invoked benchmark evaluator grades saved answers for engineering use. Review request · Tool-call behavior test · Benchmark evaluator
2. Base Model
| Field | Details |
|---|---|
| Base model name | Administrator-selected judge family. A blank judge setting disables review. |
| Version / checkpoint | The benchmark comparison below used openai.gpt-6-luna through Bedrock Mantle in us-east-1. The active site selection and release await confirmation. |
| Provider | Amazon Bedrock. The model developer depends on the selected family. |
| License | Model-specific terms. Confirm the deployed selection and applicable terms with the owner. |
| Model location | Public reference for the deployed model awaits confirmation. Site selection · Review call |
2.1 Model Modifications
Prompting only. FLP uses provider-hosted weights and controls the instructions sent with each request.
3. Data
3.1 Pretraining Dataset
FLP used the hosted model as supplied. No FLP pretraining dataset applies.
3.2 Finetuning Dataset
No FLP finetuning dataset applies.
3.3 Prompt Design and Versioning
The versioned prompt asks the judge to evaluate relevance, evidentiary support, citations, legal-advice boundaries, and consistency with recorded state. Its structured verdict contains an approval flag and findings with codes and explanations. Earlier corrections are included as diagnostic context. Prompt and schema
3.4 Validation and Test Dataset
End-to-end comparison uses the frozen North Dakota, fictional, and multi-turn cases described in the benchmark. Unit tests cover conflicting verdict fields, retries for correction, provider errors, and audit writes. No independent human-labeled sample of accept/reject decisions was identified. Case set · Verdict tests · Runtime tests
3.5 Label Documentation
No FLP training labels apply.
4. Training & Evaluation
4.1 Design Decisions and Rationale
With a judge configured, the service buffers a final draft until review completes. A rejection can trigger up to two corrections within the tool-step limit. On exhausted corrections or a judge timeout, provider error, or invalid verdict, the service can release the last draft with a warning. A generation timeout follows the guided-help fallback. An audit-write failure interrupts release. Runtime behavior and failure tests
4.2 Metrics and Evaluation
On the November 9 real-reference subset, the Luna assistant and Luna judge produced a mean gated score of 83.3/100 across 45 answers. The same assistant with runtime review off scored 81.7/100 across 45. The external grader flagged critical failures in 5/45 and 6/45 respectively. The judge-enabled configuration recorded 57 review decisions: 45 approvals and 12 rejections, including intermediate drafts. Generation timeout and retry settings also differed. These counts cannot establish the judge’s precision or recall. Source: 20261109_204106-frozen-corpus-baseline, batch 20261110_151211-43d56f, 15 North Dakota cases repeated three times. Comparison methodology · Benchmark evaluator
4.3 Failure Analysis
The external grader still found five missed referrals among the 45 answers generated with the Luna judge. A runtime approval can therefore coexist with a later critical finding. Conversely, a rejection can ask for an unsupported correction. The instructions tell the judge to reassess the current draft and explain mistaken earlier feedback. Review instructions
5. Known Limitations
The assistant and judge can share the same model family, the same incomplete source material, and the same blind spots. Users whose questions fall outside the supplied court materials have less evidence available for either model to assess. Agreement with human reviewers and performance across court or language groups remain unmeasured in the cited run.
6. Deployment and Monitoring
6.1 Deployment Setup and Data Dependencies
If a site selects a judge, Django sends the candidate answer, conversation messages, and thread state to Bedrock. The request replaces non-text attachment parts with a notice. The service stores audit metadata for the verdict or error and keeps rejected drafts out of the litigant’s visible history. Judge input · Audit writes and release
6.2 Monitoring Plan
The service records approved or rejected verdicts, error categories, and exhausted corrections. Counts of individual reviews can exceed counts of answers. For a reported bad reply, the audit can show which candidate was reviewed and what was released. Monitoring ownership, alert thresholds, and cadence await confirmation. Runtime audit
6.3 Re-evaluation Criteria and Process
Proposed triggers are a judge-model or prompt change, a change to release or warning behavior, repeated exhausted corrections, or a disputed verdict. Compare paired configurations on frozen cases and have qualified reviewers examine both missed issues and unnecessary rejections.
7. Version History
| Card version | Date | Author | Change |
|---|---|---|---|
| 0.1 | 2026-11-18 | Codex, prepared for FLP review | Initial draft against portal revision b20fec4. |
8. Output Disclosure
The verdict is internal, but it changes what the litigant reads. A released draft with failed or exhausted review displays “Warning: This answer could not be fully verified and may contain errors.” The built-in Copy action preserves the warning. An approved answer has no approval badge. An ordinary reply copied out of context has no explicit AI-origin label. The warning describes uncertainty and does not say that a lawyer or court verified the answer. Warning in the chat interface · Copy behavior
9. Contacts
| Role | Name | Contact |
|---|---|---|
| Feature owner | Litigant Portal team. | Public contact pending confirmation. |
| AI team contact | John Gardner | john@free.law |