Technical Insights and Theory of Operation

Caber is based on three technical insights:

1. Enterprise data is duplicated, mostly below the document level. The same sentences and paragraphs recur across files, messages, and systems.
2. Most business data does not identify itself. Whether a sentence is fresh, stale, or a trade secret cannot be read from the sentence.
3. Content is finite; context is unbounded. A sentence is a fixed byte sequence. What it means and how it may be used depends on every copy of it, and that keeps growing.

Every numeric claim below links to a public primary source, except one internal measurement that is marked as such. The public documentation (caber-product-docs, page technical-insights.md) carries the same claims without the internal material.

Claim discipline

The documentation doctrine requires a public primary source for every numeric claim. Round figures such as "99% of enterprise data is duplicated", "95%", "90%", or a "30/70" split between self identifying and other data have no public source and must not appear in any external material. Use the figures on this page instead.

Definitions

These are Caber's definitions, not the common AI usage.

Term	Common AI usage	Caber definition
Content	Any text or data given to a model	The specific bytes and their sequence that make up a given fragment of data, for example one sentence in one file. Identical bytes are identical content wherever they appear.
Context	The text placed in the model's context window: prompt, retrieved passages, history	Any data related to the content that signals what it means, how it may be used, who owns it, the regulatory frameworks that apply, access permissions, and organizational or operational boundaries. Provenance and version history are part of it.
Fragment	A retrieval chunk sized for an embedding model	A WRL Chunker unit of roughly sentence size with a deterministic fingerprint. See Characterization and the WRL Chunker for sizes.
Lineage	The document a chunk came from	A graph, not a line: every source, owner, and version holding a copy, linked through the fingerprint.

What common usage calls the "context window" holds content in Caber's terms. Caber's context is the information about that content that the window leaves out. Content propagation is nearly frictionless, whereas context propagation is not.

Insight 1: Enterprise data is duplicated, mostly below the document level

Measurement	Finding	Source
Duplicate bytes in user authored data, 15 corporate file servers, over 2,000 users, 64 KB chunks, virtualization libraries excluded	15.2% to 43.8% by server	Primary Data Deduplication, USENIX ATC 2012, Table 2
Same servers, whole file matching only	1.9% to 15.9%	Same, Table 2
SharePoint dataset, whole file against chunk level	3.1% against 43.8%	Same, Table 2
Where duplicate bytes sit	"the bulk of the duplicate bytes lie between files having 40% to 95% of duplicate chunks"	Same
Copies per duplicated chunk	Most duplicate bytes are in chunks appearing 2 to 32 times, consistent across all datasets	Same
857 desktop file systems, 162 TB, block level	Storage reduced to as little as 32% of original	Meyer and Bolosky, USENIX FAST 2011
C4 web corpus	One 61 word sequence appears 61,036 times; 7.18% of tokens fall in exact duplicate substrings of 50 or more tokens	Lee et al., ACL 2022
Enron email corpus	619,446 messages as released; 200,399 after removing computer generated folders that "contained large numbers of duplicate email messages"	Klimt and Yang, ECML 2004

Caveats to state whenever these figures are used

• Granularity. The storage studies use 4 KB to 64 KB chunks. The ATC 2012 paper notes that "smaller chunk sizes can give higher space savings, arising from duplicate detection at finer granularities." WRL fragments are far smaller. Say the figures likely understate sentence level duplication; never present them as a sentence level measurement.
• Desktop binaries. In the FAST 2011 data, dll, lib, pdb, exe, cab, msp, and msi files make up 58% of the whole file savings, so that study overstates duplication in authored documents.
• Enron. The paper reports how much the cleaning removed, not how many removed messages were duplicates. It is an upper bound, not a count.
• Scope. SNIA states that "the wider the scope of data deduplication, the higher the data deduplication ratio will be." It gives no prevalence figure, so do not cite SNIA for a percentage. Understanding Data Deduplication Ratios, SNIA 2008

Open gap: no sentence level enterprise measurement

No published study measures sentence level duplication in enterprise data. A reproducible measurement Caber could publish: run the Fingerprinter over the public Enron corpus and report the share of fragments that occur more than once, and the distribution of copies per fragment. That would give a citable, sentence level figure.

Insight 2: Most business data does not identify itself

Self identifying data carries its own marker. Payment card numbers end in a Luhn check digit (ISO/IEC 7812); US Social Security numbers have a fixed nine digit format. Regex DLP handles this class well.

Everything else falls into states that the bytes cannot reveal:

State	Meaning	Where the deciding fact lives
Fresh	The current version of the information	Outside the sentence: no newer document supersedes it
Stale	Superseded but still stored and retrievable	In a different document, the one that replaced it
Trade secret	Valuable because not generally known, and protected by its owner	In the owner's conduct: access controls and protective measures

Evidence

• Identical bytes, identical verdict. A classifier whose only input is the text returns the same answer for the same text. If one sentence is current in one place and superseded or confidential in another, a content only classifier must mislabel at least one copy.
• Staleness happens elsewhere. A document goes stale when another document supersedes it. Nothing in the stale document changes.
• Trade secret status is conduct. 18 U.S.C. § 1839(3) requires that "the owner thereof has taken reasonable measures to keep such information secret" and that the information "derives independent economic value ... from not being generally known." Cornell LII
• Internal measurement, not for external use. On the docker/docs git history (400 lineages, 2,001 documents), the content only ceiling for recovering parent edges was 88.7% by raw overlap and 83.2% IDF weighted. Roughly 11% of the version relationships in that corpus cannot be recovered from content by any direction method. Source: the CaberDataFactory corpus handoff reply. The scorer is uncommitted, so the doctrine treats this as a proprietary anecdote; keep it out of public material until the scorer is committed and the run is reproducible.
• No measured split. No public study gives the share of enterprise data that identifies itself. Do not quote one.

Insight 3: Content is finite; context is unbounded

A fragment is a fixed byte sequence. Its context has no fixed size: every copy, reader, version, permission change, and policy adds to it. Content copies freely; context stays behind. A sentence pasted into an email leaves the source folder's ACL behind.

Evidence

• Many copies per fragment. Most duplicate bytes in the ATC 2012 data sat in chunks appearing 2 to 32 times. In C4, 280 near duplicate clusters have more than 5,000 members and the largest has 250,933. ATC 2012; Lee et al., 2022
• Each copy has its own context. Source system, owner, ACL, and timestamps are per copy, so the context of one fragment grows with its copy count while the fragment stays the same.

CONSOLE FUNCTIONAL PROOF/admin/overlappingContent vs. Context Overlap Across Enterprise Sources

Assertion Verified: Proves that while enterprise content (sentences in files) is finite and heavily duplicated, the context attached to that content (meaning, provenance, freshness, and permissible use) diverges across every copy.

Finite Content (Sub-Sentence Byte Spans)Infinite Context (Meaning, Provenance, Use)Cross-Source Policy Inheritance

Finite sentence content shared across repositories inherits distinct contextual provenance, freshness state, and metapolicy governance.

Does the same sentence mean the same thing?

Not necessarily. The bytes are the same everywhere; meaning, truth, and what may be revealed about origin depend on the copy, the time, and the reader.

Where this example comes from. On June 27, 2025, Morgan Littlewood and Kris Moore of TrueNAS described their goal of a trustworthy, L2 capable support chatbot. Their assistant (DocsBot at the time) ingested public and private documentation, and they wanted to add community forums, source code, and internal support tickets. The challenges they named map directly onto this question: inconsistent answers between documentation and community responses, information becoming outdated, and maintaining accuracy across versions. They also planned to collect system information from paid subscribers to give more contextual support. Scale at the time: about 400,000 community members, with about 10 staff answering forums. Meeting notes: PyCaber/tests/test_data/Caber-TrueNAS-6-27-25.docx.

External use

TrueNAS must not be named in external material. The public site presents this scenario as "an open source storage vendor" and cites the upstream OpenZFS 2.3.0 release instead of TrueNAS release numbers or blog posts, since those would identify the vendor. Do not name TrueNAS or Morgan, quote the meeting, or imply the relationship in anything external.

One sentence about growing a storage pool lands in four places:

You cannot add a single disk to an existing RAIDZ vdev. To grow the pool, add another vdev.

Copy	Location	Meaning there	Permitted use
1	Official documentation for releases before 24.10	Vendor guidance, correct for those releases	Serve to users on a release before 24.10
2	Community forum answer from a volunteer	Unverified answer, correct when posted	Lower trust; prefer the official copy and attribute to the forum if used
3	Internal support ticket, pasted by an engineer into a reply to a paying customer	Same guidance inside a record about one customer's system	Sentence may be used; a public answer must not cite the ticket or reveal the customer
4	Every copy, after TrueNAS 24.10 added RAIDZ expansion	Wrong for 24.10 and later, still right for older releases	Withhold from users on 24.10 or later; serve the newer documentation

Evidence

• The sentence changed meaning with a release. TrueNAS 24.10 "Electric Eel" (October 2024) added RAIDZ expansion, which grows an existing RAIDZ vdev one drive at a time. TrueNAS, 2024

All four copies share one fingerprint and therefore one equivalence class. Regex DLP and content classifiers see one harmless sentence and give one verdict. Caber sees one fragment with four context sets: authoritative, low trust, restricted, and superseded for some readers. Given the requester's release (the paid subscriber system information TrueNAS planned to collect would supply it), a policy can serve the sentence to a user on an older release and withhold it from a user on 24.10 with no change to the bytes. Copy 1 is accurate, official, and public, and it is still the wrong answer for anyone who has upgraded.

CONSOLE FUNCTIONAL PROOF/admin/config/policiesEliminating Coarse Over-Blocking and Regex Under-Blocking

Assertion Verified: Proves how sub-sentence WRL characterization and contextual provenance eliminate both traditional failure modes: coarse document-level over-blocking (which denies 100% of a useful document over one sensitive clause) and pattern-only under-blocking (which misses non-self-identifying trade secrets).

0% False-Negative Trade Secret Leaks94% Document Utility Retained via Surgical RedactionNon-Self-Identifying Data Governed by Provenance

Contrasting coarse document-level blocking and regex pattern scanners against Caber's sub-sentence provenance-driven surgical redaction.

Theory of Operation

1. Fingerprint content. The WRL Chunker fragments every parsed document and live payload and fingerprints each fragment. See Characterization and the WRL Chunker.
2. Link the copies. Fingerprints are indexed in the Roaring Bitmap Index (.rbix). Fragments with the same fingerprint form one equivalence class across sources.
3. Collect context from every copy. Identity and metadata connectors supply owners, ACLs, groups, and timestamps. NER, driven by the configured labels, enabled ontologies, and enabled compliance frameworks, attaches entities. The graph holds (:Entity {id,text,label}) nodes linked by TAINTS to the Edges they occur in.
4. Resolve versions (beta). Two Stage IDF Stale Source Clustering builds version DAGs and assigns freshness_rank. See Freshness and Stale Clustering.
5. Decide at inference time. Compiled Rego runs in process with the fragment's context, the Human Principal (obo.user), and the declared purpose. Metapolicies resolve multisource conflicts. See Metapolicies and Governance.
6. Record the decision. Each adjudication emits an Audit V2 receipt with the affected regions, determining policy, caller identity, and purpose.

CONSOLE FUNCTIONAL PROOF/admin/contextGraphNeo4j Context Graph & Ontological Provenance Explorer

Assertion Verified: Proves that identical sentence bytes in two different documents carry distinct meaning and governance verdicts depending on their ontological neighborhood (:Region NER rollup, LNKD-{xor}-{count} equivalence class, and CORI provenance).

Content is Finite; Context is InfiniteMorgan Company Ambiguity DisambiguatedLNKD-{xor}-{count} Equivalence ClassesDecision E7 :Region NER Rollup

Caber Console Context Graph Explorer (/admin/contextGraph) proving how ontology and provenance disambiguate identical sentence content across Morgan's company divisions.

Ontologies (beta)

An ontology names the entity types that matter in a domain. Caber uses each ontology entity's rdfs:label as an NER label. Found entities become fragment context and can be tested in data use policies.

Bundled domains

Bundled JSON-LD files are embedded by pycaber and self extract next to the runner scripts. A domain name resolves to {runner_dir}/{domain}.jsonld; an unknown name is logged as a warning and skipped.

Domain	Entity labels	Notes
medical	Medical Condition, Medication, Laboratory Result, Clinical Visit, Vital Signs	Validated: "lisinopril" as Medication, "hypertension" as Medical Condition
sec_common	SEC Filing, Financial Statement, Financial Line Item, MD&A Section, Risk Factor, Note to Financial Statements, Business/Geographic Segment, Financial Metric
apple_delta	None in practice	Known issue: the file has no entities array (its data is under productCategories, riskCategories, and similar keys), so load_ontology_labels returns nothing for it

Enable

Off by default. Configure under DocParser:

{
  "DocParser": {
    "ontology": { "enabled": true, "autoLoadDomains": ["medical"] }
  }
}

pycaber copies these into IngestOptions (ontology_enabled, ontology_domains), and overlay_ontology_opts in pycaber writes ontologyDomains and ontologyPaths into the NER stage config. Per call IngestOptions values (ontology_enabled, ontology_domains, ontology_paths in pycaber) override the configured ones.

Environment variables are unconfirmed. Doc comments and a sys_config test name CSI_DOC_PARSER_ONTOLOGY_ENABLED and CSI_DOC_PARSER_ONTOLOGY_AUTO_LOAD_DOMAINS, but both registry entries have env_var: None. Document and use the JSON config until the registry is reconciled.

Custom ontology

A custom file needs an entities array whose members carry rdfs:label:

{
  "@context": { "rdfs": "http://www.w3.org/2000/01/rdf-schema#" },
  "entities": [
    { "@id": "acme:Project", "rdfs:label": "Project Codename" },
    { "@id": "acme:Customer", "rdfs:label": "Customer Account" }
  ]
}

Point the engine at it with ontologyPaths (or ontologyPath) in the engine block, for example DocParser.gliner.ontologyPaths; a per call ontology_paths replaces the configured list.

Engine behavior differs:

• gliner (gliner_runner.py): with no explicit labels, ontology labels replace the default label set; with explicit labels, the two are unioned.
• gliner2 (gliner2_runner.py, the default engine): ontology labels are appended to the explicit labels.

Compliance framework vocabulary

pycaber adds the entity labels of every enabled framework (policies with kind == "framework" and enabled == true that have a mapping in policies/governance/data/framework-mappings/) to the extraction vocabulary and stamps gov.auth.ner.domains. Today only hipaa-v1.json exists; it declares person, medical record number, date of birth, diagnosis, medication, Social Security number, email address, phone number, postal address, account number, organization, and credit card number. Labels map through policies/governance/data/entity-registry-v1.json to one canonical id each, so aliases cost one label. Operator labels under DocParser.<engine>.labels are additive and cannot remove a framework label. Enablement is read at boot.

Policy reference

A data use condition tests auth.ner.labels, which resolves to edge.gov.auth.ner.labels. Use the lower case spaced spelling, as the Rego tests do ("social security number", "genetic information"):

{
  "left": { "source": "edge", "attribute": "auth.ner.labels" },
  "operator": "contains",
  "right": { "kind": "literal", "value": "medical condition" }
}

The Rust grammar test in pycaber uses MEDICAL_RECORD_NUMBER. That exercises parsing only and is not the stored spelling; do not copy it into examples.

Limits and stale material

• Only entities[].rdfs:label is read. Properties, relationships, extractionRules, and rdfs:subClassOf are ignored, so there is no hierarchy expansion.
• The unknown_entity_label validation was dropped. A condition naming a label that is never extracted is accepted and silently never matches.
• The Rust ontology engine was deleted (commit 1d0d4851). orchestrator/docs/Ontology_Integration.md still describes it and is stale.

Sources

• "Primary Data Deduplication: Large Scale Study and System Design." USENIX ATC 2012 (Microsoft authors; first author's surname is hyphenated, see the PDF). PDF
• Meyer and Bolosky. "A Study of Practical Deduplication." USENIX FAST 2011. PDF
• Lee et al. "Deduplicating Training Data Makes Language Models Better." ACL 2022. arXiv 2107.06499
• Klimt and Yang. "The Enron Corpus: A New Dataset for Email Classification Research." ECML 2004. PDF
• Dutch. "Understanding Data Deduplication Ratios." SNIA, 2008. PDF
• 18 U.S.C. § 1839. Cornell LII
• ISO/IEC 7812, Identification cards: Identification of issuers.
• TrueNAS. "TrueNAS Delivers the Industry's First Integration of OpenZFS 2.3." TrueNAS blog, October 21, 2024. Web page