Current status: working Python rules baseline, browser review flow and synthetic evaluation. An optional structured-output LLM adapter is implemented and tested with mocked responses; a live model comparison has not yet been run. The public demo replays recorded Python outputs and says so explicitly.
The problem
An incident notice can contain useful facts, missing details and contradictory statements in the same document. This prototype asks a narrow question: can a reviewer prepare a structured summary while keeping the original evidence visible?
Motor claims are an illustrative use case. This is an independent project, not an insurer commission or a claim about any company's existing systems. It does not determine fault, coverage or payment.
A complete review loop
- Select a synthetic notice, including complete, incomplete, conflicting and free-form examples.
- Inspect seven fields: date, time, location, vehicle, description, damage and parties.
- Select a source quote to highlight it in the notice. Missing or conflicting fields stay unresolved.
- Edit a value and acknowledge review flags. Human corrections remain separate from the extraction.
- Approve the intake summary and export JSON containing the method, source evidence, edits and review timestamp.
What is implemented
The core is a dependency-free Python extraction engine and a loopback HTTP service. Label-based rules provide an inspectable baseline. The model adapter uses a constrained JSON schema; downstream validation rejects unsupported values and quotes that do not occur in the source. Model errors and refusals do not silently become successful extractions.
The public version contains four recorded examples. The local server additionally accepts new text. Model calls require explicit server configuration and user confirmation, and are capped per server session. No API key is shipped to the browser.
Evaluation, including the failures
The aggregate includes missing-field decisions and should not be read as extraction accuracy on real documents. On the labelled cases, the baseline matches 126/126 field decisions. On the two free-form cases, it matches only 3/14. This is the baseline's central limitation and the reason for evaluating a language model next.
The fixtures were authored during development, include repeated structures and have no independent annotation. They are a regression suite, not an unseen benchmark. Literal quote matching proves occurrence, not semantic correctness. There is no measured insurer time saving or real-claims performance claim.
Next experiment
Freeze a separately annotated set of new incident families before tuning prompts. Compare the rules, raw model extraction and model extraction with evidence validation on the same notices. Report field correctness, unsupported facts, abstention, latency and actual model usage. The existing score is not reused as an independent test.
Built with AI coding assistance. Abdirahman is reviewing and developing the implementation as a learning and portfolio project. The illustrative attachment rule is versioned and visible in the demo.