We asked one hard question of a fixed body of source text, and answered it ten different ways — the AITruthX framework, and three flavours of Microsoft GraphRAG, each driven by three different large language models. Every answer below is a real, live output from the actual engine. Then a neutral judge counted, for each one, exactly how many claims were not supported by the source. This note reports what happened and what it means for anyone who needs data they can trust.

What AITruthX is — the framework searches, the model only speaks

AITruthX uses a language model for one job only: putting what the framework already found into clear, correct language — the wording and the grammar. The model does not search, and it does not decide what the answer is. That work is done by the framework itself, over approved source material. Because the model is only writing, it can be a cheap, open-source-class base model instead of an expensive frontier one.

That difference is the whole subject of this study, and it is why AITruthX can run on the cheapest model in the comparison and still be the only engine to reach perfect grounding: correctness comes from the retrieval, not from the model's judgement.

What's next for the framework

We're building AITruthX into a platform where anyone can upload their own data and get grounded, LLM-quality answers from it — at the cost of the cheapest model, with output quality rivaling the most expensive.

The rest of this note is the evidence behind that claim — the experiment, the numbers, and every answer and flagged claim in full.

01The five questions — and what the document actually contains

The corpus is a structured food-science document: menu items built from preparations, preparations built from ingredients, ingredients carrying chemical constituents. It was chosen because the dependency chains are real, deep and mechanically checkable — we can compute the correct answer from the source and compare, rather than asking a model to grade prose.

1. “Which menu items use the Warm Spice Blend? List every single one.” — truth: 149, a maximum-spread enumeration.
2. “Which preparations ultimately contain Piperine, through any chain of ingredients?” — truth: 29, multi-hop.
3. “Which menu items use the Sour Acid Base?” — held-out, truth: 60.
4. “Which preparations ultimately contain Curcumin, through any chain?” — held-out, truth: 19.
5. “List all chemical formulas in the Umami Stock Base's full chain — down to constituents.” — truth: 39, four levels deep.

02The answer is complete before any AI is involved

This is the cleanest statement of what the framework is and is not. The answer is complete before any AI is involved, with each chemical formula already attached to the item it belongs to. A basic language model is then used for one job — compacting that complete answer into flowing prose someone wants to read — and that step is where the 60–110 seconds and the entire token bill are spent. It is also where completeness can be lost rather than gained.

For anyone who needs the full result rather than a readable summary — an auditor, a compliance review, a regulatory filing — the zero-AI answer is the one to take, and it is free.

03Why multi-hop scores zero — structure, not tuning

The zero on Q2 is not a tuning failure, and no amount of top-k widening or reranking repairs it. The passages about a chemical constituent never mention the preparations three links away. Similarity search — by meaning, by keywords, or both fused — can only retrieve text that exists; it cannot retrieve a relationship that was never written down in any single place. The relationship has to be established before the question is ever asked.

That is the difference this test isolates. AITruthX resolves the relationships once, during ingestion, and every answer is then assembled from those established, individually-checkable connections. Chunk retrieval has no equivalent step: it meets the question with nothing but text similarity, and a fact that spans documents has no similarity to find.

This is the same architectural point Tests 1–3 made about grounding, now visible as a hard zero rather than a score difference.

04What the answers look like

Excerpts, unedited, from the AITruthX answers:

“The Warm Spice Blend is used in the following menu items: Hollowmere Courtyard 6613 is a menu item of the Vietnamese line composed of Green Chilli Base, Warm Spice Blend, and Preserved Rind Base… Cobalt Pavilion 9649 is a menu item of the Russian line constructed with Aromatic Oil Base, Warm Spice Blend, and Golden Spice Paste…” — continuing through all 149, each with its full composition.

“Several component preparations and standing preparations ultimately contain Piperine through their ingredients: Aromatic Bark Oil: this component preparation…” — each of the 29, with the chain that reaches it.

05The ingestion, and the honest costs

The document was ingested once — ~3 hours, fully automated, pausing and resuming itself through interruptions — producing 1,118 records joined by 3,291 byte-exact connections, every connection verifiable against the source text.

Two separate things happen, and only one of them costs time. AITruthX produces the complete, in-depth answer to the question — that part is the framework, and it is effectively instantaneous. A basic language model is then used for one job only: to compact that complete answer into readable prose for the person asking. The 60–110 seconds is the model writing, not the framework finding. We report it because the user waits for it, but it is a property of the external model, not of the retrieval.

That split also decides the bill. AITruthX itself is free inside the platform — what a user pays for is the external model that renders the answer. Choose a cheaper or faster model and the wait and the cost both fall; the completeness measured in Figure 8 does not change, because it was already determined before the model wrote a word.

We do not re-read the document to answer. The full corpus is ~165,000 tokens; a deep answer draws on ~65,000 of them — only the records the question actually reaches (~450 of 1,118 here). RAG read ~19,000 tokens, but the wrong selection. Brute-forcing all 165k through a model on every question is not merely expensive: our own ingestion campaign showed models cannot reliably enumerate at that size even with everything in front of them. The connections are established once, at ingestion; each question then reads only what it needs. These five were maximum-spread stress questions — an everyday question reaches far less.

Limits we state plainly. Single benchmark runs — unlike Tests 1–3, these were not repeated, so treat individual numbers as one observation each. Eight records with name-twin titles were excluded from every truth set, for AITruthX and RAG alike. Three retrieval families were tested here; proxy methods, HyDE, MMR, parent-document, RAPTOR and GraphRAG are follow-up work — in our earlier smaller-scale rounds proxy methods lost the chemical symbols and parent-document capped hard, but that is not yet measured at this scale. Raw answers are retained and available. A minute is a long time to wait — that wait is the external model writing the answer out, and shortening it is our next engineering target. We would rather publish the honest latency than hide it.

01What changed, question by question

02Why precision is the lever, not compression

Nothing was summarised, shortened or dropped to achieve this. No text was rewritten and no model was involved in the reduction. The answers contain the same verified records as before — there are simply fewer records that do not belong to the question, because the connections between records were made more exact.

That distinction matters for what it protects. Compression trades completeness for size, and a compressed answer can lose a fact. Precision does not: the reduction comes entirely from not reaching material the question never needed, so every figure in Test 4 still holds. Completeness is unchanged; only the irrelevance is gone.

It also decides the bill. Because the free depth answer is what the paid model reads, halving the characters roughly halves the cost of every compact answer — without touching what the answer says. This is the direction the work continues in: not a faster model, but fewer characters that were never worth reading.

01What Test 2 adds — the rest of the RAG world

Reviewers of Test 1 rightly asked: Microsoft GraphRAG is one family — what about the standard fixes? Two in particular are the textbook rebuttals: cross-encoder reranking (precise top-k instead of noisy stuffing) and parent-document retrieval (resolve to a unit, return its bounded neighbourhood — structurally the closest rival to AITruthX). Test 2 runs both, plus every other major family:

02A new metric: coverage — so “clean by saying less” can't hide

A fair criticism of grounding alone: a method could score 1.00 by returning one narrow sentence. Test 2 therefore adds coverage — each answer is scored against 35 frozen source facts (fixed before any run): how many does the answer actually convey? A clean-but-empty answer now shows up as exactly that. As the table shows, AITruthX's 1.00 grounding comes at 0.90 coverage — it says as much of the real material as the stuffing methods do.

How to read the table — what each number means

Grounding is the fraction of an answer's claims actually supported by the source (1.00 = nothing invented). Flags is the raw count of claims a judge marked unsupported — lenient counts only clear hallucinations, strict counts every claim not explicitly backed, however small. Coverage is completeness: how many of the 35 frozen source facts the answer conveys. Calls is the number of LLM calls the method needed. A single number means the judges agreed; a range is the min–max across judges, modes, and runs.

Step by step — how every score is produced: (1) Coverage points — chosen by the data, not by anyone. At ingestion, AITruthX breaks the uploaded source into individual factual statements and stores them. For a question, the checklist is exactly the stored statements of the topic being asked (35 for the Teller-Ulam question) — fixed before any scoring; no judge or model writes it at test time. Each judge then marks every point COVERED / PARTIAL / MISSED; coverage = (covered + ½·partial) ÷ total. These points are the entire set of stored fact-points the data holds for that topic — nothing selected, nothing left out. And the checklist itself was verified point-by-point against the full uploaded source by both judges: Gemini found 33/35 supported, Claude Fable 5 found 32/35 — the two points both reject are clarifying definitions added during ingestion (“deuterium is an isotope of hydrogen”, “lithium-6 is an isotope of lithium”): true statements, but not written in the uploaded text; the third is a borderline entailment one judge accepts and one does not. We publish this verification instead of assuming the checklist — that is the guarantee. (2) Hallucination flags. Each judge reads the answer claim by claim against the source text: the lenient pass lists only clear inventions; the strict pass lists every claim without an explicit backing sentence, however small. Two judges × two passes — counts are shown as ranges because they swing between passes. (3) Grounding. The judge counts the answer's total factual claims; grounding = 1 − (flagged ÷ total). (4) The provenance audit (Test 2) traces every flagged claim against what the system actually retrieved, separating true inventions from metric artifacts. (5) One honest limitation of strict judging: it demands an explicit backing sentence and does not follow chains of implication — e.g. it flagged "deuterium is present" even though the source's "lithium deuteride" contains deuterium by definition. We show such flags rather than argue them away — which is why lenient and strict are reported separately, and why the one-number score (below) weights them differently, with weights assigned independently by Claude Fable 5 (ultra). (6) Same protocol for everything: these exact judges, prompts, checklist and formulas were applied identically to every RAG family and to AITruthX, in all three tests — nothing in the measurement was changed between engines. The only thing unique to AITruthX is its own answering instruction (its product behaviour), which plays no part in how anything is measured.

How it is tested — the stored-data principle: the uploaded data is the ultimate proof, but every retrieval system stores it in its own format — chunks for chunk-RAG, an entity graph for Microsoft GraphRAG, summary trees for RAPTOR — and AITruthX's own protected internal format, which is never exposed. So the fair test, and the one used here, is against what the system stored and retrieved: if a claim's backing is absent from that stored data, it is a flag; if a stored fact never appears in the answer, that is a coverage miss; grounding is the supported fraction of everything said. Test 2's provenance audit applied exactly this — classifying every flagged claim against the exact context each system retrieved — and it is how the 76%-invented figure was measured. Full transparency: this is the same standard for AITruthX as for every baseline.

With every heading counted — not just grounding — AITruthX leads by 0.12. Flags and extra calls pull RAPTOR and the agent below simpler methods despite good grounding. And the score still under-sells the gap: a hallucinated claim can mislead silently, while a missing fact merely awaits a follow-up question (or one click of in-depth mode).

Read the coverage column first: almost everyone covers 0.8–0.94 of the source facts, including AITruthX at 0.90 — so no row is winning by saying less. Then read grounding: the only 1.00 in either test is still the first row. The best chunk-based methods (RAPTOR, HyDE, the agent, hybrid) get close — 0.90-something — but close means several invented claims per answer, every time, in every run.

03Premium models on the same retrievals — the Test 1 result, repeated harder

We re-generated the four decisive configurations with Claude Opus 4.8 and Claude Fable 5 — the exact same retrieved context, only the writer changes. If hallucination were a model-quality problem, the expensive models should clean it up.

04Where the hallucinations actually come from — a claim-by-claim audit

Counting flags says how much; it doesn't say why. So we audited every flagged claim from every Test 2 configuration against the exact context its model was given, and classified each one: was it actually in the retrieved text? Stitched across chunk borders? Or invented outright — no basis in the context at all?

05Findings of Test 2

iNo retrieval family produced a clean answer

Fourteen configurations, seven families, two full runs each — and not one clean answer anywhere. The best (RAPTOR, HyDE, the LangGraph agent) reached 0.92–0.94, which sounds close until you read it as what it is: several invented claims in every answer, every time. For a system whose product is trust, "close" is the failing grade.

iiBounding retrieval does not bound generation

The textbook rebuttal to Test 1 was that a top-1 reranker is "as bounded as AITruthX." It isn't: given one 395-character chunk, the model wrote 1,000+-token answers — ~70 claims manufactured around 4 supported ones. Boundedness of input says nothing about boundedness of output; only completeness of the retrieved unit does that.

iiiLoops and trees pay more for the same problem

The agentic loop spent 6 LLM calls and 64 seconds (vs 1 call, ~8s) and mostly re-retrieved the same chunks. RAPTOR spent 5 extra summarisation calls building its tree. Both still flagged 12–29 claims under strict judging. Extra machinery multiplied cost, not integrity.

06What we actually tested

All ten runs read from the same source corpus (an encyclopedia-level bucket about early nuclear-weapon design, used purely as a fixed test document). All ten answered the same broad question:

“Explain the complete Teller–Ulam thermonuclear design step by step, from the fission primary to the final total yield, including every stage and mechanism involved.”

We chose a deliberately big concept — a whole design, not a single fact — because breadth is exactly where retrieval methods either hold together or start inventing. What differs between the ten runs is only two things: how the source is retrieved, and which model writes the final answer.

07The four retrieval methods, in plain terms

Three of these are Microsoft GraphRAG's real search modes — Microsoft's open-source (MIT-licensed) graph-RAG library, which is exactly why we could run the genuine pipelines rather than a re-implementation; the fourth is the AITruthX framework. Same corpus, same judge — only the retrieval logic changes.

08How the experiment ran — the full process

Nothing here is a paraphrase or a remembered answer. For each cell in the results table we executed the real engine prompts against the real database, captured the model's actual output, and measured tokens and calls directly. Here is the exact pipeline and where each credential came from.

Which keys, which plans, ran what

• Google Gemini (base plan API key). Ran the AITruthX base model, the Gemini-driven Microsoft GraphRAG and flat answers, all text embeddings (gemini-embedding-001), and the neutral judge — model gemini-2.5-flash.
• Claude Opus 4.8 & Claude Fable 5 (Claude Max plan, via Claude Code CLI). After a direct Anthropic-API purchase failed, we authenticated the Claude Code command-line tool on the Max plan and ran each Claude model as a fresh, headless process with zero knowledge of this project — fed byte-identical engine prompts. This is what makes the Claude columns real rather than assistant guesses.
• Same prompts, same data, same judge across every model. The only variable inside a Microsoft GraphRAG row is the model; the retrieval context is held fixed.

09Results

Ten real runs, one broad question. Grounding is the fraction of the answer's claims that the source actually supports; flagged claims is the raw count of claims it does not; depth is how completely the answer walks the full design (0 = one-liner, 1 = every stage).

Read the table top-down and a pattern jumps out. The only perfectly-grounded row is AITruthX on a cheap model; flat stuffing on Fable 5 comes next at 0.94 — but AITruthX gets there at roughly a quarter of the tokens. Everything in the Microsoft GraphRAG block sits near the bottom regardless of which model wrote it: swapping Gemini for Opus for Fable barely moves grounding, because the context those methods retrieved was thin or tangential to begin with.

Isolating the model: the flat-stuffing rows — and why the count is noisy

The flat rows saw the same rich context, so they should isolate pure model quality. When we tried to rank the three models by flag count, we found we couldn't — the number moves too much between judge passes to trust. That instability is itself the finding:

10What “accuracy” means here — and an honest note

In our earlier in-product comparison screen, the single “accuracy %” figure was a blended display metric: accuracy = grounding × 0.6 + relevance × 0.4. It is convenient for a dashboard, but it mixes two different things. In this note we report the components separately and treat grounding — the fraction of claims the source supports — as the real integrity number, because that is what “does it hallucinate?” actually measures. Relevance and depth are reported alongside, never folded in silently.

11Three findings

iRetrieval is the bottleneck, not the model

This is the headline. Microsoft GraphRAG local and global stayed near the floor whether Gemini, Opus, or Fable wrote the answer. The framework that retrieved the right passage — AITruthX — reached top grounding on the cheapest model available. You cannot buy your way out of bad retrieval with a bigger model; you fix retrieval.

iiBigger concepts break stuffing first

The broad “explain the whole design” question is precisely where flat stuffing bled: flat-Opus went from a handful of unsupported claims on a narrow question to 19 on this one. The wider the ask, the more a permissive method fills gaps with plausible-sounding invention. AITruthX held at zero because it never left the bounded entry.

iiiA stronger model does not mean “reasoning wins” — answering your question directly

You asked whether the results show Gemini's base model is best for reasoning, even against Claude's biggest models. They do not — and the honest answer is stronger than a leaderboard. When we tried to rank the three big models by how many unsupported claims they made, the number would not hold still: repeated strict passes put flat-Fable anywhere from 4 to 27, and the ranking among the models flipped between passes. The reason is that the count is confounded by answer length — the more thorough the answer, the more specific details a strict pass can mark as “outside the exact reference,” even when those details are real and came from the retrieved source. So no model here is cleanly “best for reasoning” on this evidence; that framing doesn't survive contact with the data. What does hold across every pass is two things: all the big models keep grounding high (0.88–0.96), and only AITruthX reaches a perfect 1.00 — on the cheapest model, because its answer is bounded to exactly what the source supports, so there is almost nothing for any judge to flag. The grounding comes from retrieval structure, not from hoping a bigger model behaves. That stability is the product.

For the record, the Gemini model used throughout was gemini-2.5-flash for all answers and judging, with gemini-embedding-001 for embeddings.

12One more thing we saw: the refusal

On this broad, weapons-adjacent question, the real Claude Opus 4.8 global run partially declined to reconstruct the full staging — a safety response. AITruthX answered fully, because its context is a bounded, pre-approved slice of encyclopedia-level source text rather than an open-ended “reconstruct the weapon” request. Controlled retrieval doesn't only reduce hallucination; it keeps legitimate answers deliverable.

13Why this becomes a product: grounded training data

Here is the commercial point the experiment sets up. Teams building their own models increasingly harvest answers from top LLMs to assemble training data for a target question — but that harvested data can hallucinate, and providers increasingly throttle or refuse bulk, repetitive querying (often suspected to be model-training use). AITruthX offers the alternative: training pairs that are grounded in the customer's own corpus by construction, so they don't invent.

Each pair is delivered in two modes:

Prompts come either from a short list the customer supplies, or are generated internally per stored entry. The customer submits their data through the ordinary ingestion flow and sees it land correctly; the training-data job itself is run from the admin side and only the finished, downloadable data file is returned to their profile. It is priced as a data product, separate from ordinary usage cost.

14How the judging was done — two judges, two strictness levels

Because a single count can mislead, we did not judge once. We scored the same four full-depth answers four ways: with Google Gemini 2.5 Flash and, independently, with Claude Fable 5 — and each judge in two modes. A normal pass counts only clear hallucinations (statements that plainly go beyond the source, ignoring reasonable rephrasing). A strict pass enumerates every claim, however small, that isn't word-for-word in the reference. The earlier figures in this note began with the normal, in-depth reading; this section adds the strict reading and a second, independent judge on top.

Running two separate model families as judges is the safeguard: if the conclusion only held under one judge, it wouldn't be trustworthy. Here is every count — and the one row that stays near the floor no matter who judges or how strictly.

AAppendix — every answer, every judge, every flagged claim

Complete transparency behind Figure 6. For each of the four answers: first the exact text the engine returned, then — listed separately — every claim flagged by each of the four judge passes (Gemini normal, Gemini strict, Claude Fable 5 normal, Claude Fable 5 strict), each with the reason it was flagged. Read these and you can see for yourself why the counts move so much: nearly every flagged item is a real detail drawn from the retrieved source that simply falls outside the narrow structured reference — not an invention. That is why longer answers accrue more flags, and why grounding (a fraction) is the metric to trust over any single raw count.