We start this week’s Wrap with some surprising data about Grok’s performance on contract review. But first, a quick announcement: Artificial Lawyer is now on editorial holiday until Thursday Oct 29 and won’t be publishing news until then.

This site and its founder will be in Boston, USA, to help moderate ClioCon’s Enterprise track and is taking the opportunity to get some R&R before Legal Innovators UK on Nov 4 + 5, and then returning to America for Legal Innovators New York on Nov 17 + 18. (See below for your express tickets).

Spilling some legal AI tea in Boston. How it might look….!

Ivo has developed a new benchmark with data lab, micro1. They tested seven leading AI models across 350 real-world contract review tasks to ‘see how well they could make the same judgment calls as experienced attorneys’. The benchmark evaluated whether they could prioritize important issues, make appropriate edits, and recognize when a decision required human input, they explained.

According to the results, all seven models scored below 60% on weighted attorney-defined criteria, with even the top-performing model reaching just 58.3%.

And that’s not massively surprising, because to get above moderate success rates on complex legal work you really need more ‘context’, i.e. specific reference data, rulebooks and precedent documents that are very closely fitted to the matter, and subtle oversight that can correct initial outputs.…

Because you read this