FLP Wiki

System / Product: Litigant Portal
Model Type: Prompted language model for rubric-based grading
Version: 1.0 Date: 2026-11-18
Owner: Free Law Project, individual owner pending confirmation
Status: Draft. Internal evaluation tool. The documented run was graded on 2026-11-10.

1. Purpose

1.1 What this model does

The evaluator grades an answer already produced by a benchmark candidate. It receives the question, reference material, expected facts, numbered answer passages, and conversation context when present. It returns structured judgments about the answer. The harness then calculates scores. Grading request and validation

1.2 Intended users

Portal engineers and content reviewers can use the results to compare configurations and investigate failures. The evaluator runs outside the litigant’s chat session. The optional model that can change a reply before a litigant sees it has its own card. Reply review judge

1.3 Scope and limitations

The grading target is agreement with the supplied benchmark references. A high score establishes neither current law nor suitability for a particular litigant. In the fictional cases, the invented court rules are the standard for scoring. Evaluation rubric

2. Base Model

Field Details
Base model name OpenAI GPT-6.1 Sol for the cited batch.
Version / checkpoint bedrock_mantle/us-east-1/openai.gpt-6.1-sol, pinned explicitly in the grading configuration.
Provider OpenAI model served through Amazon Bedrock Mantle in us-east-1.
License Provider terms for the identified Bedrock offering. Exact applicable terms await owner confirmation.
Model location Bedrock Mantle in us-east-1. The saved batch and evaluation configuration identify the model route. Pinned evaluator configuration · Version and region checks

2.1 Model Modifications

Prompting only. FLP uses provider-hosted weights and controls the instructions sent with each request.

3. Data

3.1 Pretraining Dataset

FLP used the hosted model as supplied. No FLP pretraining dataset applies.

3.2 Finetuning Dataset

No FLP finetuning dataset applies.

3.3 Prompt Design and Versioning

The rubric defines critical failures and seven quality dimensions. The output contract requires evidence tied to numbered answer passages and an assessment of each expected fact. The harness validates passage identifiers and required values, records protocol information for each grading batch, and stores the judgment separately from the original answer. Rubric · Output contract · Grader implementation

3.4 Validation and Test Dataset

The November 9 run used 25 cases: 15 North Dakota reference questions, six fictional cases, and four multi-turn conversations. Candidate configurations included raw model calls, grounded chat without runtime review, and grounded chat with selected runtime judges. The frozen court files have recorded provenance and hashes. The references were awaiting court review. A separate human-adjudicated set for validating the grader, including screening for development overlap, was not identified. Case set · Snapshot provenance · Evaluation design discussion

3.5 Label Documentation

For evaluation, the case authors supply expected facts, references, and any required escalation. The grader classifies facts as supported, contradicted, omitted, or uncertain. The rubric specifies qualitative scores and critical failures. Grading contract

4. Training & Evaluation

4.1 Design Decisions and Rationale

A pinned external model grades all configurations against the same references. The request identifies the answer and case evidence without naming the candidate system. A deterministic scorer weights facts at 70% and seven qualitative dimensions together at 30%. Each critical failure zeroes the gated overall score. The report also retains the score before that gate. Evaluation method · Scorer

4.2 Metrics and Evaluation

The saved run 20261109_204106-frozen-corpus-baseline recorded all 900 planned attempts. Batch 20261110_151211-43d56f graded all 895 completed answers. The other five attempts failed during candidate execution and counted as zero in the overall result. No completed answer was awaiting grading. This measures grading coverage. Agreement with human reviewers, false critical-failure rates, and score calibration were not measured in that run. Report semantics

4.3 Failure Analysis

An invalid evidence ID, a missing fact assessment, or an omitted required score can invalidate a judgment. The reporting code distinguishes missing grades from candidate execution failures. A structurally valid judgment can still mistake a conditional fee for an asserted fee or miss a required referral. A reviewer needs the answer passage and case reference to decide whether the grade was justified. Grader checks · Rubric examples

5. Known Limitations

The reference questions cover a narrow set of tasks. Some follow QA scenarios used in development. Synthetic cases test whether answers track changed rules. Neither group establishes general performance across courts, languages, or actual litigant traffic. GPT-6.1 Sol served as both the external evaluator and one of the candidate models. The available results do not quantify how that relationship affects grades.

6. Deployment and Monitoring

6.1 Deployment Setup and Data Dependencies

The standalone evaluation harness reads saved attempts and references, calls the pinned model, and records judgments and aggregate reports. An engineer can regenerate reports offline from existing results. Each run records its model routes and reference hashes. The cited run recorded evaluator source revision 3fb2502f and an application source fingerprint, with the application Git revision unknown. Evaluation harness

6.2 Monitoring Plan

Operators can inspect grading coverage, errors, malformed responses, execution failures, and per-case evidence. The harness can recover eligible saved responses or retry failed grades in a separate batch. Outputs record the evaluator identity and batch so later reports can be traced. Responsibility for recurring human calibration and its cadence await confirmation. Grading, recovery, and reporting

6.3 Re-evaluation Criteria and Process

Proposed triggers are a changed evaluator release, rubric, scoring formula, reference corpus, or repeated disagreement with human review. Preserve the prior batch, regrade comparable saved answers, and review disagreements by case and failure type. A new reference set requires its own benchmark version. Benchmark versioning

7. Version History

Card version Date Author Change
0.1 2026-11-18 Codex, prepared for FLP review Initial draft against portal revision b20fec4.

8. Output Disclosure

The immediate outputs are machine-generated grades and explanations used by the project team. They are not delivered as legal guidance in chat. The saved summary names the external model and grading batch, and distinguishes quality before critical-failure gates from the final overall score. When sharing a chart or number separately, identify it as externally model-graded, include the case group and denominator, and identify the reference set and grader. A detached score cannot establish a human or court assessment. Report fields and interpretation

9. Contacts

Role Name Contact
Feature owner Litigant Portal team. Public contact pending confirmation.
AI team contact John Gardner john@free.law
8 views Last updated 1 hour, 7 minutes ago
Creator: john