Benchmark
Edition 2026-11 · Question set 1As of: 3 November 2026

How often does an AI model cite a case number that is not in our collection?

We asked four AI models which case law applies: ten questions from six areas of law. Each question ran three times: with no tools, with web search and with klaracase. We then compared every case number they cited with our collection.

no tools15%

36of 241

cited case numbers are not in our collection

  • Wrong date: 5 of 241
with web search9%

27of 306

cited case numbers are not in our collection

  • Wrong date: 7 of 306
with klaracase100%

355of 355

cited case numbers came from a tool result (traceable)

  • Not in our collection: 0 of 355
  • Wrong date: 1 of 355

What this means for your work: check every case number an AI tool gives you before you cite it. This page shows how often that is necessary.

Models tested, measured on 3 November 2026:GPT-6.1 TerraGPT-6 AstraClaude Opus 5.5Claude Sonnet 5.5

Our collection of court decisions holds 510,000+ today. We compared against its state on the day of the measurement.

As of: 3 November 2026

AI models
4
Areas of law
6
Setups per question
3
Cited case numbers
902
02 · Per model

Results by model

All six areas of law combined, per model and setup. The names are the exact model versions tested, measured on 3 November 2026.
SetupCitedConfirmedWrong dateWrong courtNot in our collectionFound elsewhereOpenTraceable
GPT-6.1 Terra
No tools232100211–
With web search393620110–
With klaracase60600000060
GPT-6 Astra
No tools464310202–
With web search626100101–
With klaracase65641000065
Claude Opus 5.5
No tools6959001073–
With web search97782017512–
With klaracase12312300000123
Claude Sonnet 5.5
No tools1037740221012–
With web search1089730844–
With klaracase10710700000107

The columns “Confirmed”, “Wrong date”, “Wrong court” and “Not in our collection” add up to the number under “Cited”. “Confirmed” means: same court and same date as in our collection. “Found elsewhere” and “Open” split the column “Not in our collection”. “Traceable” counts the cited case numbers that came from a tool result.

03 · Method

How we measured

No tools

  1. 1Question
  2. 2AI model
  3. 3Answer with case numbers

The model answers the question on its own. No search, no tools, no internet.

Compared withOur collection of court decisions on the day of the measurement

With web search

  1. 1Question
  2. 2AI model with web search
  3. 3Answer with case numbers

The model may search the internet before it answers.

Compared withOur collection of court decisions on the day of the measurement

With klaracase

  1. 1Question
  2. 2AI model with access to our database
  3. 3Answer with case numbers

The model reads our database through tools. We count how many cited case numbers came from a tool result. We call that traceable.

Compared withOur collection on the day of the measurement. Also: did the case number come from a tool result?

What counts as one?
One case number cited in one answer.
How large is the sample?
Four models, ten questions in six areas of law, each question in three setups. The models cited 241 case numbers with no tools, 306 with web search and 355 with klaracase. The totals differ, because each model decides for itself how many decisions it cites. That is why every figure sits beside its own total.
How do we check a case number?
We count every cited case number and compare it with our collection as it stood on the day of the measurement. It is confirmed (same court, same date), has a wrong date, has a wrong court, or is not in our collection.
How often did we measure each question?
Once per model and setup. We take no average and claim no repeat runs.
Are the questions public?
No. We do not publish the wording. This page shows only figures, no questions and no single case numbers.

Trap questions

20of 24

Two more questions name sections that do not exist. They are not part of the ten questions. They drew 24 answers. An automatic check flagged 20 of them as saying the section does not exist. A person still has to review the other 4.

04 · Limits

What this benchmark does not show

What “not in our collection” means

It only means: we did not find the case number in our collection. The decision can still exist. We looked these case numbers up on dejure.org. Some exist there (“found elsewhere”). The rest is still open.

SetupNot in our collectionFound elsewhereOpen
No tools361818
With web search271017
With klaracase000

“Open” means: a person has not checked these case numbers yet. We claim nothing about them.

  1. We did not measure whether an answer is legally right. Nobody reviewed the legal content of the answers. We only count cited case numbers.
  2. klaracase built the tool and graded the run itself. We are the vendor and the checker at once.
  3. We count a wrong date for each case number, from the text of the answer. An automatic reader assigns the dates. It can be wrong.
  4. The figures apply to the four model versions tested on the day of the measurement. There were ten questions in six areas of law. The figures say nothing about later versions or other questions.
  5. All figures come from one recorded measurement on 3 November 2026. They do not change as our collection grows.
05 · Editions

All editions

A new edition appears about every three months. The counts are comparable only between editions with the same question set.
EditionDateQuestion setModelsNo toolsNot in our collectionWith web searchNot in our collectionWith klaracaseTraceable
2026-11current3 November 20261436 of 24127 of 306355 of 355

Get the next edition by email

Work with klaracase

References that hold up in your work

klaracase names the decision behind every answer. In research, and through an interface for your own applications.

Get startedLog in

with klaracase

355of 355

cited case numbers came from a tool result (traceable)

Not in our collection

No tools36 of 241 not in our collectionWith web search27 of 306 not in our collectionWith klaracase0 of 355 not in our collection
The full method
Contact

Talk to us

Law firms and companies

Want to know whether klaracase fits your work? Write to us. We will arrange a call.

Email: info@klaracase.de

Research and press

Questions about the method or the data behind this benchmark? Write to us. We share the measurement data with researchers on request.

Email: pr@klaracase.de

The next edition by email

The klaracase newsletter brings you each new edition of the benchmark.