# Opening a Dual Ingestion Lane

2026-06-10. The OpenText pilot tests whether source text, a preserved binary, lineage, permissions, and retrieval can travel together through one governed ingestion path.

Language: en
Canonical: https://avanticomplex.com/en/blog/opening-a-dual-ingestion-lane/

Published: 2026-06-10T12:00:00.000Z

June 10, 2026. I am opening an OpenText dual-ingest lane because a document pipeline has to answer more than whether it can extract text. It needs to show where a document came from, where its preserved binary resides, which permissions apply, and what retrieval representation was created. The design keeps source material and its governed residence distinct: text drives embedding, an optional PDF is preserved in Alfresco, and the resulting record carries both origin and residence lineage. Today’s work is a pilot on Dave-owned data. It does not authorize or validate ingestion of Partner N material, which remains gated.

## The lane gives each artifact a role

The implementation includes an OpenText reader, a dual-ingest orchestrator, a command-line entry point, a second embeddings schema, and a local embedding service. The final content source is `Edit_Sentence`. The reader uses those text files to create embedding input and pairs an optional PDF by basename; PDFs are preserved as reference binaries and are never embedded. The OpenText Content Server application programming interface (API) reader is still a stub. It defines a future interface but is not a finished API integration.

The orchestrator records file data, source lineage, ingestion events, access-control list (ACL) snapshots, permissions, and embeddings. A canonical Alfresco lineage identifies the preserved residence; an OpenText lineage identifies origin. That separation makes a later review able to explain both the system of record and the source that supplied the content. The newer retrieval lane uses a 768-dimensional nomic model served locally through LM Studio and can add a contextual prefix before embedding. The older 384-dimensional lane remains available. This implementation is intended for comparison, not a statement that one model is universally better.

## What today’s pilot validation establishes

Today’s validation record reports a live Alfresco upload and a one-document end-to-end ingest on Dave-owned material. It records two lineage rows, an ACL snapshot, owner read and administrator grants, and chunk embeddings. A re-embedding run on the same pilot material produced 36 contextualized 768-dimensional chunks. An example semantic query returned three on-topic results with reported cosine scores of 0.85, 0.85, and 0.84. Reader tests report 10 of 10 passing. This establishes that the integrated path can work with the selected material and environment. It is not a benchmark, a broad recall measurement, or a result that generalizes to a customer corpus.

## Scale changes the next test

An earlier read-only corpus dry run identified 139 documents and projected about 316,777 chunks at per-sentence granularity. That result drove the final grouping choice: several sentences are grouped into a chunk, reducing the projection to roughly 75,000 chunks. The pipeline also supports per-document contextual retrieval, so the context pass is about 86 calls rather than one call for every chunk. The command-line defaults now group five sentences with one sentence of overlap and generate context once per document. These settings are inspectable choices, so a later run can compare cost and retrieval quality against the earlier sentence-level approach. Those changes make the bulk experiment more practical, but they do not produce a bulk result today. The next test is a controlled run using the grouped chunks and per-document context, after GPU (graphics processing unit) ownership is decided. It must inspect idempotent reruns, permissions, lineage, and retrieval quality again.

**Historical basis:** 2026-06-10 repository milestone, commit `bdf7cce`.


## Translations
- en: https://avanticomplex.com/en/blog/opening-a-dual-ingestion-lane/
- es: https://avanticomplex.com/es/blog/opening-a-dual-ingestion-lane-es/
- pt: https://avanticomplex.com/pt/blog/opening-a-dual-ingestion-lane-pt/