Show HN: LLM Hallucination Detector – Works with GPT, Claude, and Local Models
It checks for signs of hallucinated or unreliable output using a multi-method approach (overconfidence patterns, factual density, coherence, contradictions, etc).
What it does:
Works with GPT, Claude, local models (e.g., Mistral, DialoGPT)
Outputs a hallucination probability (0.0–1.0)
Flags overconfident or uncertain language
Scores factual density, coherence, and contradictions
Compares responses to context (if provided)
Fully framework-agnostic — no extra dependencies
Built for production + research workflows
Benchmarked on 1,000+ samples:
F1: 0.75
AUC-ROC: 0.81
Fast: ~0.2s per response
Comes with plug-and-play examples:
OpenAI, Anthropic, local models
Flask API
Custom scoring configs
I’m giving this away free under MIT. Would love feedback, issues, PRs — or just to know if it helps you build safer LLM apps.
GitHub: https://github.com/Mattbusel/LLM-Hallucination-Detection-Scr...
I’m excited to share this lightweight hallucination detector I built to help identify unreliable or “hallucinated” outputs from LLMs like GPT, Claude, and various local models.
It uses multiple methods — from spotting overconfidence and contradictions to scoring factual density and coherence — to give a hallucination probability score for any generated response.
It’s framework-agnostic, fast (~0.2s per response), and designed for both research and production use. Plus, it’s completely free under the MIT license.
I’d love to hear your thoughts, feedback, and if you find it useful for your projects. Happy to answer questions or discuss how it works under the hood!
Thanks for checking it out!
From among the analyses the tool makes, it makes sense to me that contradictions can be detected, since that doesn't require knowledge of the real world. I'm very interested in how you do this detection ("Logical inconsistencies") in practice. Likewise for "Logical progression".
Two questions:
1. Since "overconfidence" is treated as a red flag, won't applying your tool as a filter cause LLM response precision to drop, often unnecessarily? The safest answer an LLM can give to "When was the Eiffel Tower built?" is surely along the lines of "The Eiffel Tower may or may not have been built at some time in the past."
2. I don't see how this tool can detect the kind of hallucination that (a) involves no contradiction and (b) requires knowledge of the world. These come up often. Examples: Citing plausible-sounding but nonexistent court cases, calling plausible-sounding but nonexistent methods in an API.
To your points:
1. Overconfidence and precision You're right that filtering on overconfidence alone could tank precision. That’s why the tool doesn’t treat it as a strict red flag, it’s one of several signals, and the final hallucination score is a weighted combination of multiple metrics (confidence, density, contradictions, progression, etc.). Overconfident phrasing tends to correlate with hallucinations in aggregate, but the idea is never to penalize all confident answers, just to flag the ones where that confidence is unjustified by the context or content.
2. Detecting hallucinations that require world knowledge Absolutely, those are the hardest cases. This tool doesn’t solve that. Instead, it acts as a proxy evaluator:
Factual density gives a rough measure of “how many claims are being made”
Overconfidence vs. ambiguity highlights where a model might be bluffing
Logical coherence and contradiction checks flag when a model violates internal structure (not ground truth) But it won’t catch the subtle world-knowledge misses (like fake court cases or made-up API calls) unless you pair it with a grounded context or use external validators.
The long-term hope is: use this tool to raise suspicion, not declare judgment. It's a cheap sanity layer, a “weak oracle” that’s fast, pluggable, and good enough to catch the dumb stuff before you escalate to expensive validators or human review.