Benchmark Evaluator
System / Product: Litigant Portal
Model Type: Prompted language model for rubric-based grading
Version: 1.0
Date: 2026-11-18
Owner: Free Law Project, individual owner pending confirmation
Status: Draft. Internal evaluation tool. The documented run was graded on 2026-11-10.
1. Purpose
1.1 What this model does
The evaluator grades an answer already produced by a benchmark candidate. It receives the question, reference material, expected facts, numbered answer passages, and conversation context when present. It returns structured judgments about the answer. The harness then calculates scores. Grading request and validation
1.2 Intended users
Portal engineers and content reviewers can use the results to compare configurations and investigate failures. The evaluator runs outside the litigant’s chat session. The optional model that can change a reply before a litigant sees it has its own card. Reply review judge
1.3 Scope and limitations
The grading target is agreement with the supplied benchmark references. A high score establishes neither current law nor suitability for a particular litigant. In the fictional cases, the invented court rules are the standard for scoring. Evaluation rubric
2. Base Model
| Field | Details |
|---|---|
| Base model name | OpenAI GPT-6.1 Sol for the cited batch. |
| Version / checkpoint | bedrock_mantle/us-east-1/openai.gpt-6.1-sol, pinned explicitly in the grading configuration. |
| Provider | OpenAI model served through Amazon Bedrock Mantle in us-east-1. |
| License | Provider terms for the identified Bedrock offering. Exact applicable terms await owner confirmation. |
| Model location | Bedrock Mantle in us-east-1. The saved batch and evaluation configuration identify the model route. Pinned evaluator configuration · Version and region checks |
2.1 Model Modifications
Prompting only. FLP uses provider-hosted weights and controls the instructions sent with each request.
3. Data
3.1 Pretraining Dataset
FLP used the hosted model as supplied. No FLP pretraining dataset applies.
3.2 Finetuning Dataset
No FLP finetuning dataset applies.
3.3 Prompt Design and Versioning
The rubric defines critical failures and seven quality dimensions. The output contract requires evidence tied to numbered answer passages and an assessment of each expected fact. The harness validates passage identifiers and required values, records protocol information for each grading batch, and stores the judgment separately from the original answer. Rubric · Output contract · Grader implementation
3.4 Validation and Test Dataset
The November 9 run used 25 cases: 15 North Dakota reference questions, six fictional cases, and four multi-turn conversations. Candidate configurations included raw model calls, grounded chat without runtime review, and grounded chat with selected runtime judges. The frozen court files have recorded provenance and hashes. The references were awaiting court review. A separate human-adjudicated set for validating the grader, including screening for development overlap, was not identified. Case set · Snapshot provenance · Evaluation design discussion
3.5 Label Documentation
For evaluation, the case authors supply expected facts, references, and any required escalation. The grader classifies facts as supported, contradicted, omitted, or uncertain. The rubric specifies qualitative scores and critical failures. Grading contract
4. Training & Evaluation
4.1 Design Decisions and Rationale
A pinned external model grades all configurations against the same references. The request identifies the answer and case evidence without naming the candidate system. A deterministic scorer weights facts at 70% and seven qualitative dimensions together at 30%. Each critical failure zeroes the gated overall score. The report also retains the score before that gate. Evaluation method · Scorer
4.2 Metrics and Evaluation
The saved run 20261109_204106-frozen-corpus-baseline recorded all 900 planned attempts. Batch 20261110_151211-43d56f graded all 895 completed answers. The other five attempts failed during candidate execution and counted as zero in the overall result. No completed answer was awaiting grading. This measures grading coverage. Agreement with human reviewers, false critical-failure rates, and score calibration were not measured in that run. Report semantics
4.3 Failure Analysis
An invalid evidence ID, a missing fact assessment, or an omitted required score can invalidate a judgment. The reporting code distinguishes missing grades from candidate execution failures. A structurally valid judgment can still mistake a conditional fee for an asserted fee or miss a required referral. A reviewer needs the answer passage and case reference to decide whether the grade was justified. Grader checks · Rubric examples
5. Known Limitations
The reference questions cover a narrow set of tasks. Some follow QA scenarios used in development. Synthetic cases test whether answers track changed rules. Neither group establishes general performance across courts, languages, or actual litigant traffic. GPT-6.1 Sol served as both the external evaluator and one of the candidate models. The available results do not quantify how that relationship affects grades.
6. Deployment and Monitoring
6.1 Deployment Setup and Data Dependencies
The standalone evaluation harness reads saved attempts and references, calls the pinned model, and records judgments and aggregate reports. An engineer can regenerate reports offline from existing results. Each run records its model routes and reference hashes. The cited run recorded evaluator source revision 3fb2502f and an application source fingerprint, with the application Git revision unknown. Evaluation harness
6.2 Monitoring Plan
Operators can inspect grading coverage, errors, malformed responses, execution failures, and per-case evidence. The harness can recover eligible saved responses or retry failed grades in a separate batch. Outputs record the evaluator identity and batch so later reports can be traced. Responsibility for recurring human calibration and its cadence await confirmation. Grading, recovery, and reporting
6.3 Re-evaluation Criteria and Process
Proposed triggers are a changed evaluator release, rubric, scoring formula, reference corpus, or repeated disagreement with human review. Preserve the prior batch, regrade comparable saved answers, and review disagreements by case and failure type. A new reference set requires its own benchmark version. Benchmark versioning
7. Version History
| Card version | Date | Author | Change |
|---|---|---|---|
| 0.1 | 2026-11-18 | Codex, prepared for FLP review | Initial draft against portal revision b20fec4. |
8. Output Disclosure
The immediate outputs are machine-generated grades and explanations used by the project team. They are not delivered as legal guidance in chat. The saved summary names the external model and grading batch, and distinguishes quality before critical-failure gates from the final overall score. When sharing a chart or number separately, identify it as externally model-graded, include the case group and denominator, and identify the reference set and grader. A detached score cannot establish a human or court assessment. Report fields and interpretation
9. Contacts
| Role | Name | Contact |
|---|---|---|
| Feature owner | Litigant Portal team. | Public contact pending confirmation. |
| AI team contact | John Gardner | john@free.law |