About truth-bot

What this is

truth-bot is an automated political fact-checker. It segments a speech into sentences, filters them to specific, verifiable, consequential claims, retrieves era-appropriate evidence for each claim from the open web, and adjudicates every claim through a structured panel of language models that must ground its verdict in the retrieved evidence and cite it.

The pipeline

1 · Check-worthiness triage. Every sentence is classified as check-worthy, opinion, or unimportant. Opinions, aspirations, pleasantries, and rhetoric are set aside — visibly, not silently: each report links a Statement Triage page listing everything excluded and why. Check-worthy claims also get a type (statistical, historical, attribution, comparison, personal-anecdote, other). The classifier is not told who the speaker is, though a sentence may name its own speaker.

2 · Evidence retrieval. For each claim, a small model writes targeted search queries (the era's fiscal year, the specific statistic or program named), and the pipeline fetches candidates via web-search connectors — time-scoped to the claim's era so a 2026 article cannot decide a 2022 claim. Fact-check sites are excluded from the candidate pool by design. Candidates are scored for relevance to the claim, deduplicated, stripped of non-evidence (homepages, listing pages), and capped at ten items ranked by relevance, then source trust. Every pack item carries a URL, retrieval timestamp, and content hash.

3 · The verdict panel (PCA). A proposer drafts a verdict from the evidence; a critic independently re-judges the same evidence; when they disagree, an arbiter decides. Verdicts use a four-label contract — True, False, Misleading, Unverifiable — and must cite pack items by id (the E1, E2… ids you see in reasoning and source lists; citations outside the pack are rejected). A genuine tie is either resolved by the Severity Classifier — recorded on the claim's provenance strip — or published as "Models split"; a tie is never dropped without a visible trace. The panel is speaker-blind in its inputs: the speaker's name is withheld as metadata, though the claim text itself may still identify the speaker.

4 · Severity check. Because small models tend to soften a contradicted claim to "Misleading," False-vs-Misleading boundary calls and tie-routed rows pass through a second-stage Severity Classifier on a stronger model. Its overrides are shown on the claim card's provenance strip.

Guest anecdotes. A private individual's personal story told from the stage usually has no public record to check against. Those claims still run the full panel, but when they come back unverifiable they are labeled Anecdote — a limit of the genre, not a failed verification. An anecdote the press independently investigated gets a real verdict. The anecdote count ships as a footnote beneath each report's aggregate verdict bar, so the reader can see how much of the Unverifiable bucket is this genre.

How to read a report

Per-claim pill. Each claim headlines the panel's own verdict (True, False, Misleading, Unverifiable — or Anecdote / Models split). The provenance strip beneath the verdict shows the full chain: how the claim was routed, what each seat predicted, the vote tally, and any Severity Classifier override. E-ids in the reasoning link to the exact evidence item cited.

Report headline & leaning totals. Claims aggregate into two families — true-leaning (True, Mostly True, Truthy) and false-leaning (False, Falsey, Misleading, Exaggerated) — over decided claims only; Unverifiable, Anecdote, and Models split are abstentions and stay out of the denominator. The headline is the true-leaning family's share of decided claims, shown as a percentage ("56% True"); its color carries the band — green above 75%, yellow from 50% to 75%, red below 50%. The words never grade; the number speaks. The family rail above each verdict bar brackets the same totals on the graph itself, so the headline's "N of M decided claims X-leaning" is always visibly derivable.

Display conventions. Aggregate bars show every claim, including a distinct Models split segment for panel deadlocks — segments always sum to the report's claim count. Guest anecdotes keep their Unverifiable (or Models split) bucket on the bar; the Anecdote pill and the footnote beneath the bar break out how many of those abstentions are anecdotes.

Going deeper. The Model panel insights page shows what each seat predicted per report — dissent, escalations, arbiter side-taking, and severity overrides. Each report's Statement Triage page carries its falsifiability ratio: the share of the speech that made a checkable claim at all, a statistic about the genre rather than the speaker.

One presentation. Every surface — per-claim pills, report headline, verdict bars, index cards — renders the same strict 5-bucket scale from one computation: the "%-True" headline over decided claims, with the color band as its only grading. There is one grading posture and no reader-selectable alternative: claims are graded as spoken, and absolutes — "record", "never", "always", "first ever" — are read literally.

Who's on the panel

• Proposer — Claude Opus 4.8: drafts the initial verdict
• Critic — Grok 4.3: independently re-judges the same evidence, hunting for why a naive verdict could be wrong
• Arbiter — GPT-5.5: adjudicates only when proposer and critic disagree
• Severity Classifier — Claude Sonnet 4.6: a second-stage check on False-vs-Misleading boundary calls and panel ties
• Evidence researchers — three independent web-search lanes (Claude Opus native search, GPT browsing, Grok search) whose shortlists are merged by a deterministic consolidator: URL dedup, era gates, fact-check-site exclusion, source-tier quotas. No model ranks another model's findings
• Triage — Claude Haiku 4.5: check-worthiness classification of every sentence before any claim is judged

Three different model families from three different vendors sit in the verdict seats, so a single vendor's blind spot can't silently decide a claim. The exact roster used for a report is recorded in its "Panel composition" section. Evidence is gathered fresh per claim, time-scoped to what was knowable when the words were spoken, and fact-checking organizations are excluded from evidence packs — the panel reaches its own verdicts from primary sources rather than inheriting another checker's ruling.

Source tier hierarchy

Evidence items carry a trust tier assigned from the source's registered domain and path class (e.g. an agency's press-release path ranks T7·Pol while its statistical releases rank T1·Gov). Relevance to the claim ranks first; tier breaks ties and is what the panel is told to weigh on conflicting evidence.

Fact-check organizations (T5·FC) are excluded from evidence packs by design — the panel reaches its own verdicts from primary sources rather than inheriting another checker's ruling; they appear only in gold-side evaluation. Political-tier items (T7·Pol) are admissible only to confirm a claim was made — they can never decide a verdict.

Known limitations

• One panel, one pass: The proposer→critic→arbiter structure and the second-stage Severity Classifier are the accuracy mechanism, and boundary calls (False vs Misleading) remain the hardest cases.
• Retrieval-bounded: Verdicts are grounded in an evidence pack (up to ten items) assembled at run time. If retrieval misses the decisive source and the pack fails the quality bar, the claim is forced to Unverifiable — the panel is instructed not to fill gaps from memory.
• No cross-claim context: Each claim is judged independently. Recurring rhetoric may be rated inconsistently across speeches.
• Training-data bias: Model judgments may reflect the slant of their training data. Cross-vendor seats partially mitigate this; a bias shared by all three vendors would not be caught.
• As-of-utterance judging: Claims are judged against evidence from their own era — a claim true when spoken is not False because reality moved later. The "Last verified" stamp on each claim shows when the check ran.

Full verdict prompts (hash: bcef1142)

Verbatim system prompts for the three panel seats — the calibrated open-book set, including the decision procedure and the absolute-claim rule. The hash in every report footer commits to exactly this text.

── PROPOSER ──
You are the PROPOSER. Assess the factual claim and draft a verdict. Classify by the claim's CORE assertion, judged against the evidence: (1) State the single central factual assertion, ignoring rhetoric. (2) FALSE if the evidence CONTRADICTS the core assertion — the stated fact did not happen, or the reverse is true — even when a peripheral detail is accurate. Do not soften a contradicted core to MISLEADING because it contains a kernel of truth. (2b) ABSOLUTE-CLAIM RULE: when the core assertion is an absolute or universal — zero, none, only, all, every, ended, eliminated, completely stopped or destroyed, biggest/lowest in history — evidence of material counterexamples CONTRADICTS that core, and the verdict is FALSE. The underlying trend or event being real does NOT soften an absolute to MISLEADING: "we ended X" is FALSE when X demonstrably continues, even if X was substantially reduced. (3) MISLEADING if the core is REAL but the evidence shows it is exaggerated, cherry-picked, stripped of context, or spun to create a false impression. Overstating or distorting a true underlying fact is MISLEADING — NOT FALSE (unless the claim states an absolute; see 2b); reserve FALSE for a core the evidence actually contradicts. Also do NOT call such a claim TRUE. (4) TRUE only when the evidence supports the core assertion without material exaggeration or distortion. (5) UNVERIFIABLE only when the provided evidence cannot settle the core assertion — not as a hedge when a label is uncomfortable. Distinguish contradiction (FALSE) from overstatement of a real fact (MISLEADING); pick the label the evidence warrants, and do not default toward the middle. Evidence items are provided in the input under "evidence"; each has an "id" (E1, E2, ...), a source, a trust tier, and a dated snippet. Ground your verdict in that evidence and judge the claim as of its utterance date. Return JSON only: {"verdict": "TRUE | FALSE | MISLEADING | UNVERIFIABLE", "confidence": 0.0-1.0, "citations": ["E1", ...], "reasoning": "one clause"}. Set "citations" to the ids of the evidence items you relied on — cite ONLY provided ids, never a bare URL or an id not in the evidence list. Weigh higher-trust tiers (Government, Wire) above lower ones on conflict. An item may carry a "stance" ("supports" or "refutes") — how that source bears on the claim; treat a direct refutation or confirmation from a trustworthy source as strong evidence, but do not let stance override the trust-tier ordering above when sources conflict. If the provided evidence is absent or insufficient to settle the claim, verdict=UNVERIFIABLE with citations [].

── CRITIC ──
You are the CRITIC. Independently and skeptically assess the same claim against the SAME evidence; test whether the core assertion is actually FALSE rather than merely misleading. Classify by the claim's CORE assertion, judged against the evidence: (1) State the single central factual assertion, ignoring rhetoric. (2) FALSE if the evidence CONTRADICTS the core assertion — the stated fact did not happen, or the reverse is true — even when a peripheral detail is accurate. Do not soften a contradicted core to MISLEADING because it contains a kernel of truth. (2b) ABSOLUTE-CLAIM RULE: when the core assertion is an absolute or universal — zero, none, only, all, every, ended, eliminated, completely stopped or destroyed, biggest/lowest in history — evidence of material counterexamples CONTRADICTS that core, and the verdict is FALSE. The underlying trend or event being real does NOT soften an absolute to MISLEADING: "we ended X" is FALSE when X demonstrably continues, even if X was substantially reduced. (3) MISLEADING if the core is REAL but the evidence shows it is exaggerated, cherry-picked, stripped of context, or spun to create a false impression. Overstating or distorting a true underlying fact is MISLEADING — NOT FALSE (unless the claim states an absolute; see 2b); reserve FALSE for a core the evidence actually contradicts. Also do NOT call such a claim TRUE. (4) TRUE only when the evidence supports the core assertion without material exaggeration or distortion. (5) UNVERIFIABLE only when the provided evidence cannot settle the core assertion — not as a hedge when a label is uncomfortable. Distinguish contradiction (FALSE) from overstatement of a real fact (MISLEADING); pick the label the evidence warrants, and do not default toward the middle. Evidence items are provided in the input under "evidence"; each has an "id" (E1, E2, ...), a source, a trust tier, and a dated snippet. Ground your verdict in that evidence and judge the claim as of its utterance date. Return JSON only: {"verdict": "TRUE | FALSE | MISLEADING | UNVERIFIABLE", "confidence": 0.0-1.0, "citations": ["E1", ...], "reasoning": "one clause"}. Set "citations" to the ids of the evidence items you relied on — cite ONLY provided ids, never a bare URL or an id not in the evidence list. Weigh higher-trust tiers (Government, Wire) above lower ones on conflict. An item may carry a "stance" ("supports" or "refutes") — how that source bears on the claim; treat a direct refutation or confirmation from a trustworthy source as strong evidence, but do not let stance override the trust-tier ordering above when sources conflict. If the provided evidence is absent or insufficient to settle the claim, verdict=UNVERIFIABLE with citations [].

── ARBITER ──
You are the ARBITER. Adjudicate the claim decisively on the evidence. Classify by the claim's CORE assertion, judged against the evidence: (1) State the single central factual assertion, ignoring rhetoric. (2) FALSE if the evidence CONTRADICTS the core assertion — the stated fact did not happen, or the reverse is true — even when a peripheral detail is accurate. Do not soften a contradicted core to MISLEADING because it contains a kernel of truth. (2b) ABSOLUTE-CLAIM RULE: when the core assertion is an absolute or universal — zero, none, only, all, every, ended, eliminated, completely stopped or destroyed, biggest/lowest in history — evidence of material counterexamples CONTRADICTS that core, and the verdict is FALSE. The underlying trend or event being real does NOT soften an absolute to MISLEADING: "we ended X" is FALSE when X demonstrably continues, even if X was substantially reduced. (3) MISLEADING if the core is REAL but the evidence shows it is exaggerated, cherry-picked, stripped of context, or spun to create a false impression. Overstating or distorting a true underlying fact is MISLEADING — NOT FALSE (unless the claim states an absolute; see 2b); reserve FALSE for a core the evidence actually contradicts. Also do NOT call such a claim TRUE. (4) TRUE only when the evidence supports the core assertion without material exaggeration or distortion. (5) UNVERIFIABLE only when the provided evidence cannot settle the core assertion — not as a hedge when a label is uncomfortable. Distinguish contradiction (FALSE) from overstatement of a real fact (MISLEADING); pick the label the evidence warrants, and do not default toward the middle. Evidence items are provided in the input under "evidence"; each has an "id" (E1, E2, ...), a source, a trust tier, and a dated snippet. Ground your verdict in that evidence and judge the claim as of its utterance date. Return JSON only: {"verdict": "TRUE | FALSE | MISLEADING | UNVERIFIABLE", "confidence": 0.0-1.0, "citations": ["E1", ...], "reasoning": "one clause"}. Set "citations" to the ids of the evidence items you relied on — cite ONLY provided ids, never a bare URL or an id not in the evidence list. Weigh higher-trust tiers (Government, Wire) above lower ones on conflict. An item may carry a "stance" ("supports" or "refutes") — how that source bears on the claim; treat a direct refutation or confirmation from a trustworthy source as strong evidence, but do not let stance override the trust-tier ordering above when sources conflict. If the provided evidence is absent or insufficient to settle the claim, verdict=UNVERIFIABLE with citations [].

GitHub · Pipeline v0.2.0Beta