# The Corpus Must Be Hygienic Before It Is Smart August 18 research notes: measured provenance checks correct a storage assumption, while a copied collection falls from 7,574 indexed chunks to 49 without proving better answers. Language: en Canonical: https://avanticomplex.com/en/blog/the-corpus-must-be-hygienic-before-it-is-smart/ Published: 2026-08-19T06:59:00.000Z August 18, 2026. Before I ask VALESKA to sound smarter, I need to know what its document collection actually contains. A corpus is the collection an AI system searches or learns from. Corpus hygiene means checking that collection for misleading repetition, stale material, and records that no longer point to an available source. This week's work produces corrections and counts, not a claim that the whole collection is clean. For someone leading a company's first AI project, the practical risk is simple: a convincing answer can rest on poorly identified evidence. Five copies of one document are not five independent sources. Copies can occupy retrieval slots and crowd out other evidence, depending on the query and ranking. That is a risk to measure, not a rule that duplicates win every search. ## First, correct the assumption OTCS means OpenText Content Server, a document-management system. Corpus provenance is the recorded origin and handling history of the material. In this work, a category attaches that information to deposited documents. The category contract is the agreement about what its fields mean and which values are acceptable. A successful save is not enough if the information lands in the wrong place. On August 17, I correct a bad assumption in the operating notes: the column width of a multi-line text field controls display, not storage capacity. The recorded tests on Content Server 16.2.0 round-trip values up to 131,000 characters without truncation; a 5,000-character value also displays completely in the Classic interface. Those are observations on the tested installation, not a promise about every version or configuration. Measuring source-path lengths still helps describe a deployment and choose a readable display. It is not a gate against data loss caused by column width. Separately, the provenance record is reported written and read back correctly across all 17 fields. The new preflight, a check before ingestion, reports seven checks passing and rejects a simulated mismatch in field mapping and allowed values. There is already a negative test here; I should not write as though failure detection is only a future intention. ## Then, count the cleanup Today's copied collection starts with 7,574 indexed chunks, meaning smaller pieces of text available to retrieval. The first recorded retirement removes 6,496 duplicate or stale chunks. A second removes 1,029 more: 183 newly recognized duplicates and 846 records orphaned by a document move. The copy's index ends at 49 chunks across two files. These are repository-recorded execution results, not a fresh measurement of a customer installation. The same day's record reports 20 unique documents returned to their owning StoryTeller project. Matching by content matters because different filenames can hide the same text. A near-match is not automatically an exact duplicate, and an orphaned index entry is not proof that its content is disposable. Retirement of those entries is explicitly opt-in; confirming an indexed destination remains a safety requirement, not something a missing source path proves. Index cleanup and deleting source files are different actions. The first retirement leaves disk files intact; the separate repatriation moves the 20 documents. A file-based reader still depends on the remaining copied tree, so removing that tree is unresolved. The evening checkpoint also leaves three duplicate-body groups open. The cleanup counts therefore describe a bounded intervention, not a clean bill of health for every collection or every reader. ## The next test I want the next check to connect these inventory results to what a person sees. Repeat a small, fixed set of questions, inspect whether the remaining results cite the intended source, and confirm that moved material is still retrievable before any further retirement. Recheck the open duplicate groups and exercise the category mismatch on the target deployment. Fewer indexed copies and a tested provenance contract are useful progress. Neither establishes better answer quality without that separate measurement. My job is to keep the distinction visible. * * * **Historical basis:** VALESKA commits `93b938f`, `7c5e69d`, and `985d6a5`, August 17; `f44d6de`, `4b5f2e3`, `916b56c`, `04adc31`, and checkpoint `f90568d`, August 18, 2026, America/Los\_Angeles. Results are attributed to the dated repository record. ## Translations - en: https://avanticomplex.com/en/blog/the-corpus-must-be-hygienic-before-it-is-smart/ - es: https://avanticomplex.com/es/blog/the-corpus-must-be-hygienic-before-it-is-smart-es/ - pt: https://avanticomplex.com/pt/blog/the-corpus-must-be-hygienic-before-it-is-smart-pt/