Why AI Teams Should Never Treat Training Data As Evidence

Patrick Dajos - Founding Team at Hyperbound.

gettyIn Prometheus Rising, Robert Anton Wilson described people as living inside their own “reality tunnels.” In other words, we filter reality through attention, language, memory and prior beliefs. He also wrote that “the human mind is a verbalizing circuit.” Read today, that sounds unexpectedly close to one feature of a large language model (LLM): learned relationships between symbols can produce useful language outputs.

An LLM is not a digital brain; human cognition is embodied and extends far beyond language. But the brain remains our only demonstrated basis for human-level general intelligence, and it does not use everything it knows on every question. It selects a limited working set. Context selection may not only be a workaround for today’s models. It may be part of intelligent behavior itself.

Psychologist Nelson Cowan estimated that working memory holds roughly four chunks under controlled conditions. Attention research similarly describes many signals competing while the brain prioritizes a few for deeper processing.

These limits force the brain to select. AI systems face an analogous practical constraint: A capable model can still fail when the decisive instruction, document or observation is missing from its active context.

I encountered this while building an AI system for analyzing datasets larger than a model could use reliably in one context window. Most of my engineering work moved into retrieval. Segmenting the material for semantic retrieval produced one of the largest improvements because the boundaries directly affected what the system could find. Straightforward questions often needed one vector-search pass. Complex analysis required a loop: inspect the initial results, identify gaps, reformulate the search, retrieve again, compare sources and verify material claims against the underlying records.

In other words, some hallucinated or unfaithful answers originate before generation. The system may omit the best source, retrieve stale information or bury the decisive record under loosely related material. A larger window provides capacity, not guaranteed relevance or truthfulness.

Training data, parameter count, architecture, compute and post-training collectively shape capability; research on a predicted compute-optimal model called Chinchilla underscored that model size alone does not determine quality. Parameters are usually fixed during an interaction, while context can change from one request to the next.

Knowledge stored in model weights remains valuable, but it can be outdated, difficult to trace and the model may hallucinate. Claims that must be current, proprietary, auditable or consequential should not rely solely on latent knowledge. Training supplies capability; context can supply current evidence, if the system selects and verifies it well.

The same lesson applies beyond one system. Companies often have the information an agent needs, but spread it across documents, databases and tools with inconsistent ownership, timestamps and permissions. Before expecting AI to reason reliably over that knowledge, leaders must make it traversable.

The goal is not one centralized version of truth. It is an information environment with stable source identities, provenance, dates, version histories, citations and explicit links between conflicting claims. Better knowledge infrastructure will not make an agent truthful by itself, but it gives the system (and its users) a basis for checking its work.

A larger context window does not automatically solve the problem. Research found that performance could decline when decisive information appeared in the middle of a very long prompt. Retrieval can likewise omit the best source or add plausible but irrelevant material.

Wilson’s “Thinker and Prover” idea offers a useful warning: “Whatever the Thinker thinks, the Prover proves.” A leading assumption can shape retrieval toward confirming material. The final answer may cite real sources and still be misleading.

The objective is not to give a model the most context possible. It is to assemble the smallest sufficient and representative set of evidence, including information that may contradict the system’s first hypothesis.

For AI teams, three practices matter.

1. Define an explicit research method. System instructions should require the agent to inspect initial evidence, reformulate searches, run additional retrieval passes, compare sources and verify material claims. A prompt is not enforcement, however. The workflow and evaluations must test whether those steps occurred and whether the required evidence was found.

2. Preserve source, timestamp, version and access permissions for important context. A relevant document is not necessarily authoritative or current, and a citation is useful only when it actually supports the claim.

3. Handle contradictions explicitly. When credible sources disagree, the model should surface the conflict, apply a defined ranking policy, request clarification or abstain when the evidence does not justify confidence.

More capable models and larger windows will continue to matter. But truthfulness also depends on how information is found, filtered, challenged and traced. Useful AI should provide a basis for checking why an answer should (or should not) be trusted.​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?