← All benchmarks

Ivo-micro1 ContractBench

Professional Work · 2026-10-13

Ivo and micro1's ContractBench has an agent redline a real Word contract with tracked changes, comments and escalations, as an in-house attorney would, using the represented side's playbook and commercial brief. It has 350 tasks from 35 US and UK scenarios: 175 opening redlines and 175 responses to a counterparty markup, with five attorneys per scenario writing pass-or-fail rubric items weighted 3/2/1 by priority. A panel of three judge models grades each item by strict majority. A task scores the earned share of rubric weight. The score is the mean over three rollouts per input, counting only rollouts that end in a gradeable redline. Runs that hit the token limit or fail the validity gate are left out and lower the completion rate instead. This is not the 2026 arXiv ContractBench on observation contracts.

Top models (higher is better)