Many companies launched website chatbots on a quiet assumption: if the AI gets something wrong, that is the AI’s problem. A German court has now rejected that idea.
On 12 May 2026, the Higher Regional Court of Hamm (case 4 UKl 3/25) ruled in a case brought by a consumer association against a provider of aesthetic treatments. Asked about the qualifications of the company’s managing directors, both doctors, the chatbot on its website said they were specialists in plastic and aesthetic surgery. They held no such qualification. The chatbot also cited two specialist titles that do not exist.
The company argued that the answers could not be attributed to it, because the AI works statistically, no person checks each answer, and users know AI can make mistakes. The court disagreed.
What the court decided
- The operator defined what the chatbot was for and had enough influence over it, so the answers count as the operator’s own statements. The court noted that the company was able to reprogram the bot easily after the complaint.
- The “black box” nature of AI does not change that responsibility.
- A generic notice that AI can make mistakes is not a reliable defense, because many users trust computer-generated answers.
- The company should have foreseen the question and prevented false answers before the chatbot went live, since doing so took little effort.
The judgment is not final. The court allowed an appeal to Germany’s Federal Court of Justice because of the case’s fundamental importance. It follows a direction set in 2024, when a Canadian tribunal ordered Air Canada to honor a fare policy its chatbot had described, rejecting the argument that the chatbot was responsible for its own statements.
Where trusted data was missing
Nothing was hacked. A customer asked a predictable, high-stakes question, and the assistant answered without an authoritative source and without limits on what it would claim. That combination, open-ended answers with no grounding for critical facts, is common in chatbots built quickly from a website crawl or a general-purpose model.
Lessons learned
- Answer only from approved sources. Ground the assistant in curated content with an owner and a review date for each source, and show citations with every answer.
- Define what it must not answer. Credentials, legal terms, medical claims, prices, and policy exceptions should come from a verified page or a person, not from generated text.
- Test the questions customers will ask. Build an evaluation set that includes the risky, predictable questions, and run it before launch and after every change.
- Keep the knowledge current. Stale content produces wrong answers just as reliably as invented content. Assign someone to keep it fresh.
- Log answers with their sources. You will need to find and fix a bad answer quickly, and show what the assistant relied on.
What this means for your AI program
A customer-facing assistant is a publishing channel. What it says, your business says. Our Knowledge Agent Launchpad builds assistants that answer from your approved content with citations, permission-aware retrieval, and an agreed accuracy target, and AI Trust & Guardrails adds the evaluation and monitoring that keep them that way.
This article summarizes public reports of the ruling and is not legal advice.