Beta
Back to the overview
Benchmark

Criminal law

Most often not confirmed: Kimi K3, with 15 of 20. Least often: Claude Opus 4.8, with 2 of 15.

without our data

43%

21 of 49 cited case numbers not confirmed

with our data

0%

0 of 80 cited case numbers not confirmed

What this means for your work: check every case number an AI tool gives you before you cite it. This page shows how often that is necessary.

Models tested, measured on 21 July 2026: Claude Opus 4.8, GPT-5.6 Sol, Kimi K3

Looked up in our collection of court decisions, 510,000+ today.

As of: 21 July 2026

Per model

Figures per model in this area

All three questions in this area combined, per model.
  1. Kimi K375%15 of 20without our data not confirmedwith our data:0 of 32with our data not confirmed
  2. GPT-5.6 Sol29%4 of 14without our data not confirmedwith our data:0 of 23with our data not confirmed
  3. Claude Opus 4.813%2 of 15without our data not confirmedwith our data:0 of 25with our data not confirmed

Both runs count the same thing: case numbers we could not confirm. They are checked against two different yardsticks. Without our data we search our whole collection; with our data we search the decisions the model received.

Per question

With our data: 0 of 80 not confirmed, on every question

Each square is one case number the model cited. Red with a cross: not confirmed.

Each question ran twice, as two separate answers. That is why the model cites a different number of case numbers.

The topic with the most unconfirmed case numbers comes first.

not confirmedconfirmed

  1. Sentencing § 46 Abs. 3 StGB

    Kimi K3

    Without our datachecked against our whole collection

    7 of 8 not confirmed

    With our datachecked against the decisions provided

    0 of 9 not confirmed

    GPT-5.6 Sol

    Without our datachecked against our whole collection

    2 of 6 not confirmed

    With our datachecked against the decisions provided

    0 of 4 not confirmed

    Claude Opus 4.8

    Without our datachecked against our whole collection

    1 of 7 not confirmed

    With our datachecked against the decisions provided

    0 of 8 not confirmed
    Show unconfirmed case numbers (10)

    “Not confirmed” means we did not find the case number in our collection. It does not mean the decision does not exist. It can be genuine and missing from our holdings.

    • Kimi K3, without our data
      • 2 StR 172/83
      • 1 StR 308/91
      • 2 StR 570/93
      • 2 StR 528/95
      • 1 StR 525/01
      • 2 StR 217/02
      • 4 StR 375/05
    • GPT-5.6 Sol, without our data
      • 4 StR 696/79
      • GSSt 1/86
    • Claude Opus 4.8, without our data
      • 3 StR 391/16

  2. Appeal on points of law § 349 Abs. 2 StPO

    Kimi K3

    Without our datachecked against our whole collection

    6 of 10 not confirmed

    With our datachecked against the decisions provided

    0 of 12 not confirmed

    GPT-5.6 Sol

    Without our datachecked against our whole collection

    1 of 4 not confirmed

    With our datachecked against the decisions provided

    0 of 12 not confirmed

    Claude Opus 4.8

    Without our datachecked against our whole collection

    1 of 3 not confirmed

    With our datachecked against the decisions provided

    0 of 10 not confirmed
    Show unconfirmed case numbers (8)

    “Not confirmed” means we did not find the case number in our collection. It does not mean the decision does not exist. It can be genuine and missing from our holdings.

    • Kimi K3, without our data
      • 3 StR 465/01
      • 4 StR 176/09
      • 5 StR 160/00
      • 1 StR 275/81
      • 2 StR 491/00
      • 4 StR 225/12
    • GPT-5.6 Sol, without our data
      • 2 StR 45/91
    • Claude Opus 4.8, without our data
      • 2 BvR 1782/96

  3. Arson § 306b Abs. 2 StGB

    Kimi K3

    Without our datachecked against our whole collection

    2 of 2 not confirmed

    With our datachecked against the decisions provided

    0 of 11 not confirmed

    GPT-5.6 Sol

    Without our datachecked against our whole collection

    1 of 4 not confirmed

    With our datachecked against the decisions provided

    0 of 7 not confirmed

    Claude Opus 4.8

    Without our datachecked against our whole collection

    0 of 5 not confirmed

    With our datachecked against the decisions provided

    0 of 7 not confirmed
    Show unconfirmed case numbers (3)

    “Not confirmed” means we did not find the case number in our collection. It does not mean the decision does not exist. It can be genuine and missing from our holdings.

    • Kimi K3, without our data
      • 5 StR 267/98
      • 5 StR 296/16
    • GPT-5.6 Sol, without our data
      • 2 StR 213/20

The topic names the legal term and the statute the question is about. We do not publish the question itself, so that no one can tune a model to exactly this wording.

One example

One question, two runs, verbatim

This is one of the eighteen questions, shown in full. On the left the answer without our data, on the right with it. The case numbers appear exactly as the model wrote them.
§ 306b Abs. 2 StGBQuestion 1, criminal law, topic arson

Aggravated arson, section 306b(2) of the German Criminal Code.

without our data

1 of 4 not confirmed

Case numbers cited, looked up in our collection:

  • 4 StR 19/00 confirmed
  • 3 StR 541/14 confirmed
  • 5 StR 280/18 confirmed
  • 2 StR 213/20 not confirmed
Nach ständiger BGH-Rechtsprechung reicht eine bloß abstrakte Lebensgefahr – etwa allein der Umstand, dass sich Menschen in dem angezündeten Gebäude befinden – nicht aus.

with our data

0 of 7 not confirmed

Case numbers cited, looked up in the decisions we handed over:

  • 4 StR 432/18 confirmed
  • 1 StR 416/17 confirmed
  • 3 StR 172/17 confirmed
  • 3 StR 293/12 confirmed
  • 3 StR 45/13 confirmed
  • 4 StR 399/17 confirmed
  • 3 StR 336/13 confirmed
Gefährdungsvorsatz liegt vor, wenn der Täter die Umstände erkennt, aus denen sich die konkrete Todesgefahr ergibt, und sich mit dem Eintritt dieser Gefahr abfindet.

From the same measurement run as every figure on this page. Model: GPT-5.6 Sol. Measured on 21 July 2026.

BGH stands for Bundesgerichtshof, the highest German court in civil and criminal matters. Case numbers such as 4 StR 19/00 are the docket numbers German courts assign to a case. StR marks a criminal division of the BGH. The two answers above are in German, as the model wrote them.

Method

How we measured

What does “without our data” mean?

  1. Question
  2. AI model
  3. Answer with case numbers

The model answers the question on its own. No search, no tools, no internet.

Checked againstOur whole collection of court decisions, 510,000+ today

What does “with our data” mean?

  1. Question and relevant decisions
  2. AI model
  3. Answer with case numbers

The same model receives a selection of relevant decisions from our collection for the same question. It does not search for them itself.

Checked againstOnly the decisions the model was given

The full method
Limits

What this benchmark does not show

Not confirmed does not mean invented

“Not confirmed” means we did not find the case number in our collection. It does not mean the decision does not exist. It can be genuine and missing from our holdings.

11 not found on dejure.org either5 exist there and are missing from our collection

We looked up 16 of these case numbers, from two of the six areas, on dejure.org. Five decisions exist there and are missing from our holdings. Eleven we did not find there either.

  • All figures come from one measurement written down on 21 July 2026. They do not change as our collection grows. A new measurement gets a new date.

References that hold up, in your work

Klaracase names the decision behind every answer. In research, and through an interface for your own applications.

Get startedLog in
Contact

Talk to us

Law firms and companies

Want to know whether Klaracase fits your work? Write to us. We will then set a time for a call.

Schedule a call

Email: info@klaracase.de

Research and press

Questions about the method or the data behind this benchmark? Write to us. We share the measurement data with researchers on request.

Ask about the benchmarkGo to the press page

Email: pr@klaracase.de