How often does an AI model cite a case number that is not in our collection?
We asked four AI models which case law applies: ten questions from six areas of law. Each question ran three times: with no tools, with web search and with klaracase. We then compared every case number they cited with our collection.
36of 241
cited case numbers are not in our collection
- Wrong date: 5 of 241
27of 306
cited case numbers are not in our collection
- Wrong date: 7 of 306
355of 355
cited case numbers came from a tool result (traceable)
- Not in our collection: 0 of 355
- Wrong date: 1 of 355
What this means for your work: check every case number an AI tool gives you before you cite it. This page shows how often that is necessary.
Models tested, measured on 3 November 2026:GPT-6.1 TerraGPT-6 AstraClaude Opus 5.5Claude Sonnet 5.5
Our collection of court decisions holds 510,000+ today. We compared against its state on the day of the measurement.
As of: 3 November 2026
- AI models
- 4
- Areas of law
- 6
- Setups per question
- 3
- Cited case numbers
- 902
Results by area of law
- Criminal lawNo tools5 of 16 not in our collectionWith web search8 of 85 not in our collectionWith klaracase0 of 92 not in our collection
- Business crimeNo tools6 of 59 not in our collectionWith web search1 of 47 not in our collectionWith klaracase0 of 59 not in our collection
- Tenancy lawNo tools4 of 77 not in our collectionWith web search0 of 68 not in our collectionWith klaracase0 of 85 not in our collection
- Employment lawNo tools8 of 50 not in our collectionWith web search9 of 53 not in our collectionWith klaracase0 of 64 not in our collection
- Tax lawNo tools5 of 28 not in our collectionWith web search1 of 32 not in our collectionWith klaracase0 of 32 not in our collection
- Social security lawNo tools8 of 11 not in our collectionWith web search8 of 21 not in our collectionWith klaracase0 of 23 not in our collection
Results by model
| Setup | Cited | Confirmed | Wrong date | Wrong court | Not in our collection | Found elsewhere | Open | Traceable |
|---|---|---|---|---|---|---|---|---|
| GPT-6.1 Terra | ||||||||
| No tools | 23 | 21 | 0 | 0 | 2 | 1 | 1 | – |
| With web search | 39 | 36 | 2 | 0 | 1 | 1 | 0 | – |
| With klaracase | 60 | 60 | 0 | 0 | 0 | 0 | 0 | 60 |
| GPT-6 Astra | ||||||||
| No tools | 46 | 43 | 1 | 0 | 2 | 0 | 2 | – |
| With web search | 62 | 61 | 0 | 0 | 1 | 0 | 1 | – |
| With klaracase | 65 | 64 | 1 | 0 | 0 | 0 | 0 | 65 |
| Claude Opus 5.5 | ||||||||
| No tools | 69 | 59 | 0 | 0 | 10 | 7 | 3 | – |
| With web search | 97 | 78 | 2 | 0 | 17 | 5 | 12 | – |
| With klaracase | 123 | 123 | 0 | 0 | 0 | 0 | 0 | 123 |
| Claude Sonnet 5.5 | ||||||||
| No tools | 103 | 77 | 4 | 0 | 22 | 10 | 12 | – |
| With web search | 108 | 97 | 3 | 0 | 8 | 4 | 4 | – |
| With klaracase | 107 | 107 | 0 | 0 | 0 | 0 | 0 | 107 |
The columns “Confirmed”, “Wrong date”, “Wrong court” and “Not in our collection” add up to the number under “Cited”. “Confirmed” means: same court and same date as in our collection. “Found elsewhere” and “Open” split the column “Not in our collection”. “Traceable” counts the cited case numbers that came from a tool result.
How we measured
No tools
- 1Question
- 2AI model
- 3Answer with case numbers
The model answers the question on its own. No search, no tools, no internet.
Compared withOur collection of court decisions on the day of the measurement
With web search
- 1Question
- 2AI model with web search
- 3Answer with case numbers
The model may search the internet before it answers.
Compared withOur collection of court decisions on the day of the measurement
With klaracase
- 1Question
- 2AI model with access to our database
- 3Answer with case numbers
The model reads our database through tools. We count how many cited case numbers came from a tool result. We call that traceable.
Compared withOur collection on the day of the measurement. Also: did the case number come from a tool result?
- What counts as one?
- One case number cited in one answer.
- How large is the sample?
- Four models, ten questions in six areas of law, each question in three setups. The models cited 241 case numbers with no tools, 306 with web search and 355 with klaracase. The totals differ, because each model decides for itself how many decisions it cites. That is why every figure sits beside its own total.
- How do we check a case number?
- We count every cited case number and compare it with our collection as it stood on the day of the measurement. It is confirmed (same court, same date), has a wrong date, has a wrong court, or is not in our collection.
- How often did we measure each question?
- Once per model and setup. We take no average and claim no repeat runs.
- Are the questions public?
- No. We do not publish the wording. This page shows only figures, no questions and no single case numbers.
Trap questions
20of 24
Two more questions name sections that do not exist. They are not part of the ten questions. They drew 24 answers. An automatic check flagged 20 of them as saying the section does not exist. A person still has to review the other 4.
What this benchmark does not show
What “not in our collection” means
It only means: we did not find the case number in our collection. The decision can still exist. We looked these case numbers up on dejure.org. Some exist there (“found elsewhere”). The rest is still open.
| Setup | Not in our collection | Found elsewhere | Open |
|---|---|---|---|
| No tools | 36 | 18 | 18 |
| With web search | 27 | 10 | 17 |
| With klaracase | 0 | 0 | 0 |
“Open” means: a person has not checked these case numbers yet. We claim nothing about them.
- We did not measure whether an answer is legally right. Nobody reviewed the legal content of the answers. We only count cited case numbers.
- klaracase built the tool and graded the run itself. We are the vendor and the checker at once.
- We count a wrong date for each case number, from the text of the answer. An automatic reader assigns the dates. It can be wrong.
- The figures apply to the four model versions tested on the day of the measurement. There were ten questions in six areas of law. The figures say nothing about later versions or other questions.
- All figures come from one recorded measurement on 3 November 2026. They do not change as our collection grows.
All editions
| Edition | Date | Question set | Models | No toolsNot in our collection | With web searchNot in our collection | With klaracaseTraceable |
|---|---|---|---|---|---|---|
| 2026-11current | 3 November 2026 | 1 | 4 | 36 of 241 | 27 of 306 | 355 of 355 |
References that hold up in your work
klaracase names the decision behind every answer. In research, and through an interface for your own applications.
with klaracase
355of 355
cited case numbers came from a tool result (traceable)
Not in our collection
No tools36 of 241 not in our collectionWith web search27 of 306 not in our collectionWith klaracase0 of 355 not in our collectionTalk to us
Law firms and companies
Want to know whether klaracase fits your work? Write to us. We will arrange a call.
Email: info@klaracase.de
Research and press
Questions about the method or the data behind this benchmark? Write to us. We share the measurement data with researchers on request.
Email: pr@klaracase.de