FLP Wiki

System / Product: Litigant Portal
Model Type: Prompted language model for structured review
Version: 1.0 Date: 2026-11-18
Owner: Free Law Project, individual owner pending confirmation
Status: Draft. This is an optional site feature. Live enablement awaits confirmation.

1. Purpose

1.1 What this model does

Before a final chat answer appears, an optional second model reviews the assistant’s draft against the conversation and supplied court material. It approves the draft or returns findings for the assistant to address. The assistant writes the revised answer. Judge instructions and verdict · Review loop

1.2 Intended users

The chat service consumes the verdict. The litigant sees the released answer and, in certain failure cases, a warning. Staff with audit access can inspect the internal verdict and rejected drafts. Transcript audit guide

1.3 Scope and limitations

The judge reviews text-only evidence supplied to the current turn. It cannot inspect the original image or document block, look up the law, or verify a court action. Prose sent alongside a tool call can be displayed without this review. The separately invoked benchmark evaluator grades saved answers for engineering use. Review request · Tool-call behavior test · Benchmark evaluator

2. Base Model

Field Details
Base model name Administrator-selected judge family. A blank judge setting disables review.
Version / checkpoint The benchmark comparison below used openai.gpt-6-luna through Bedrock Mantle in us-east-1. The active site selection and release await confirmation.
Provider Amazon Bedrock. The model developer depends on the selected family.
License Model-specific terms. Confirm the deployed selection and applicable terms with the owner.
Model location Public reference for the deployed model awaits confirmation. Site selection · Review call

2.1 Model Modifications

Prompting only. FLP uses provider-hosted weights and controls the instructions sent with each request.

3. Data

3.1 Pretraining Dataset

FLP used the hosted model as supplied. No FLP pretraining dataset applies.

3.2 Finetuning Dataset

No FLP finetuning dataset applies.

3.3 Prompt Design and Versioning

The versioned prompt asks the judge to evaluate relevance, evidentiary support, citations, legal-advice boundaries, and consistency with recorded state. Its structured verdict contains an approval flag and findings with codes and explanations. Earlier corrections are included as diagnostic context. Prompt and schema

3.4 Validation and Test Dataset

End-to-end comparison uses the frozen North Dakota, fictional, and multi-turn cases described in the benchmark. Unit tests cover conflicting verdict fields, retries for correction, provider errors, and audit writes. No independent human-labeled sample of accept/reject decisions was identified. Case set · Verdict tests · Runtime tests

3.5 Label Documentation

No FLP training labels apply.

4. Training & Evaluation

4.1 Design Decisions and Rationale

With a judge configured, the service buffers a final draft until review completes. A rejection can trigger up to two corrections within the tool-step limit. On exhausted corrections or a judge timeout, provider error, or invalid verdict, the service can release the last draft with a warning. A generation timeout follows the guided-help fallback. An audit-write failure interrupts release. Runtime behavior and failure tests

4.2 Metrics and Evaluation

On the November 9 real-reference subset, the Luna assistant and Luna judge produced a mean gated score of 83.3/100 across 45 answers. The same assistant with runtime review off scored 81.7/100 across 45. The external grader flagged critical failures in 5/45 and 6/45 respectively. The judge-enabled configuration recorded 57 review decisions: 45 approvals and 12 rejections, including intermediate drafts. Generation timeout and retry settings also differed. These counts cannot establish the judge’s precision or recall. Source: 20261109_204106-frozen-corpus-baseline, batch 20261110_151211-43d56f, 15 North Dakota cases repeated three times. Comparison methodology · Benchmark evaluator

4.3 Failure Analysis

The external grader still found five missed referrals among the 45 answers generated with the Luna judge. A runtime approval can therefore coexist with a later critical finding. Conversely, a rejection can ask for an unsupported correction. The instructions tell the judge to reassess the current draft and explain mistaken earlier feedback. Review instructions

5. Known Limitations

The assistant and judge can share the same model family, the same incomplete source material, and the same blind spots. Users whose questions fall outside the supplied court materials have less evidence available for either model to assess. Agreement with human reviewers and performance across court or language groups remain unmeasured in the cited run.

6. Deployment and Monitoring

6.1 Deployment Setup and Data Dependencies

If a site selects a judge, Django sends the candidate answer, conversation messages, and thread state to Bedrock. The request replaces non-text attachment parts with a notice. The service stores audit metadata for the verdict or error and keeps rejected drafts out of the litigant’s visible history. Judge input · Audit writes and release

6.2 Monitoring Plan

The service records approved or rejected verdicts, error categories, and exhausted corrections. Counts of individual reviews can exceed counts of answers. For a reported bad reply, the audit can show which candidate was reviewed and what was released. Monitoring ownership, alert thresholds, and cadence await confirmation. Runtime audit

6.3 Re-evaluation Criteria and Process

Proposed triggers are a judge-model or prompt change, a change to release or warning behavior, repeated exhausted corrections, or a disputed verdict. Compare paired configurations on frozen cases and have qualified reviewers examine both missed issues and unnecessary rejections.

7. Version History

Card version Date Author Change
0.1 2026-11-18 Codex, prepared for FLP review Initial draft against portal revision b20fec4.

8. Output Disclosure

The verdict is internal, but it changes what the litigant reads. A released draft with failed or exhausted review displays “Warning: This answer could not be fully verified and may contain errors.” The built-in Copy action preserves the warning. An approved answer has no approval badge. An ordinary reply copied out of context has no explicit AI-origin label. The warning describes uncertainty and does not say that a lawyer or court verified the answer. Warning in the chat interface · Copy behavior

9. Contacts

Role Name Contact
Feature owner Litigant Portal team. Public contact pending confirmation.
AI team contact John Gardner john@free.law
7 views Last updated 1 day, 1 hour ago
Creator: john