Connecting Policy to Data Fragments & The WRL Chunker

1. What It Is

For Policy Follows Data to work at runtime, Caber must recognize a sensitive sentence, clinical record, or financial table cell even after it has been extracted from a PDF, pasted into a Markdown note, retrieved by a vector database, and reformatted inside an LLM prompt or JSON response.

This functional subsystem solves that problem by:

1. Assigning every ingested object a canonical Caber Object Resource Identifier (CORI) that links the object to its storage container, version timestamp, ACLs, and extracted sensitivity classifications.
2. Fragmenting normalized document text using Caber's proprietary WRL Chunker (Whitespace Remap on Leader), which guarantees 100% indexed byte coverage (uncovered = 0.0) with a mean overlap density of c_mean = 2.58 and a minimum fragment granularity of 40 bytes.
3. Indexing the resulting 64 bit Buzhash64 fingerprints and exact byte spans into the Roaring Bitmap Index (.rbix), enabling submillisecond intersection between live runtime streams and the enterprise corpus.

2. How Caber Connects Policy to Data Fragments: Dual Plane Architecture

*Figure 1: Dual Plane architecture connecting asynchronous data characterization and WRL Chunker indexing (Plane 1) to synchronous runtime interception and policy enforcement (Plane 2).*

As shown in Figure 1, Caber connects policy to data fragments through two coupled operational planes:

Plane 1: Asynchronous Data Characterization & Lineage Indexing

1. Multisource Ingest & CORI Assignment: Storage scanners and connectors (S3_Scanner, default_connectors) detect new or modified objects across S3, GCS, Azure Blob, SharePoint, Google Drive, Git, and SQL databases. Each object is assigned a deterministic CORI of the form:

   <scheme>⋊⋉<container>⋊⋉<object_path>ⓥ<version_epoch_ms>

2. Structure Extraction & Zero Shot NER: The Document Parser pipeline (Docling, Apache Tika, or custom Document Parser SDK plugins) extracts a hierarchical DocumentTree with exact byte and line offsets (LineOffsetIndex), while NER_Service (GLiNER2) tags subdocument spans with zero shot PII, PHI, HIPAA, PCI, and custom sensitivity labels.
3. WRL Chunker Fragmentation: The normalized text stream is processed by the WRL Chunker, emitting multiresolution 64 bit fingerprints (nmer hashes) paired with exact [start_byte, end_byte) intervals in the source document.
4. Roaring .rbix Indexing & Graph Binding: Fingerprints are written into compressed .rbix shards in shared-storage and merged into the 13 RocksDB column families of roaring_index.db. Simultaneously, Auth_Analyzer and caber_governance bind the document's CORI and its subdocument entity spans to effective Principal permissions (ALLOW_ACCESS, DENY_ACCESS) in Neo4j and OpenSearch (csi_data_Auth).

Plane 2: Synchronous Runtime Interception & Enforcement

1. In Band Stream Capture: When an AI application, agent tool, or API route executes, pycaber(auto) or the Resident Local Rust Gateway intercepts the payload in memory.
2. Live WRL Chunker Fingerprinting: The live payload is streamed through the exact same WRL Chunker at 76.6 MiB/s per core, producing a set of query fingerprints and their exact byte offsets inside the live request or response.
3. Submillisecond .rbix Intersection: The query fingerprints are intersected against roaring_index.db (RoaringTreemap bitmap intersection) to identify every upstream CORI and exact source byte span that contributed to the payload.
4. Policy Evaluation & Surgical Span Mutation: For every matched source fragment, Caber retrieves the bound policy and freshness rank, evaluates the initiating Human Principal (obo.user) against the compiled Rego policy bundle, and either permits the payload, surgically redacts only the unauthorized subdocument byte spans in flight, or blocks the call.

3. How the WRL Chunker Works

*Figure 2: The 4 Layer WRL Chunker (Whitespace Remap on Leader) pipeline and RoaringTreemap .rbix index.*

Traditional Content Defined Chunking (CDC) algorithms such as Rabin Karp, FastCDC, and Asymmetric Extremum (AE) partition a byte stream into disjoint chunks (c_mean = 1.0). If an attacker or an LLM edits a single word near a chunk boundary, the boundary shifts and invalidates the chunk hash. Conversely, indexing every sliding $k$ gram creates an unaffordable index bloat ($10\times$ to $50\times$ corpus size).

The WRL Chunker (WhitespaceRemapOnLeader in pycaber) solves both problems using a 4 layer multiresolution architecture that achieves 100% byte coverage while keeping average byte multiplicity under 3 (c_mean = 2.58):

Layer 1: Canonical Stream Remapping (CanonicalStream)

Raw UTF 8 input bytes are normalized into a canonical byte stream before hashing:

• ASCII uppercase bytes (A through Z) are folded to lowercase (a through z).
• Runs of ASCII whitespace (' ', \t, \n, \r) are collapsed into a single canonical space byte (0x20).
• An invertible offset table offset_map: Vec<u32> records the exact original raw byte offset for every canonical position i, ensuring that any chunk boundary [c_start, c_end) in canonical space maps back to exact raw byte offsets [raw_start, raw_end) in $O(1)$ time.

Layer 2: Fine Resolution Anchor Chunker (AnchorMultiChunkProcessor)

To capture short sentences, table cells, and clinical or financial clauses down to 40 bytes:

• A rolling 64 bit cyclic polynomial hash (Buzhash64) slides across the canonical stream using a 38 byte leader window (leader_window_bytes = 38).
• Local minima of the rolling hash within each window identify content defined anchor positions.
• Around each anchor, the WRL Chunker emits overlapping fine spans using a neighborhood radius $r=1$ ($\pm 1$ anchor offset), guaranteeing a 40 byte minimum fragment size (min_chunk_bytes = 40) that survives adjacent insertions or deletions.

Layers 3 & 4: Coarse and Mid Scale Winnowing Overlays (WinnowOverlay)

To provide fast set intersection on larger paragraphs while bounding total index density:

• Layer 3 (Coarse Overlay): Applies a winnowing window of W = 80 bytes (winnow_coarse_w = 80) over the anchor stream to emit long range structural spans.
• Layer 4 (Mid Scale Overlay): Applies a winnowing window of W = 116 bytes (winnow_mid_w = 116) to bridge sentence level and paragraph level matches.
• Linear Interval Coverage Verification (IntervalCoverage): A linear $O(K_1+K_2)$ sweep merges intervals across Layers 2, 3, and 4, verifying that 0.0% of canonical bytes remain uncovered (uncovered_fraction = 0.0) and enforcing the density invariant c_mean = 2.58 < 3.0.

Verified Empirical Performance (CaberEvalBenchmarks)

Metric	WRL Chunker (WhitespaceRemapOnLeader)	Legacy AE Chunker (ae)	Improvement
Indexed Byte Coverage	100.0% (uncovered = 0.0)	100.0% (disjoint)	Zero boundary shift blind spots
Mean Byte Multiplicity (c_mean)	2.58 (budget < 3.0)	1.00	Multiresolution resilience within < 3x budget
Minimum Fragment Granularity	40 bytes	64 to 192 bytes	Captures single clauses and record fields
Weighted RDR @ k=3 (17 Transforms)	71.09%	20.89%	3.4x higher recall under adversarial edits
Subdocument RDR (256 to 512B spans)	97.92%	31.40%	Near complete clause attribution
False Attribution Rate (Negative Corpus)	0.0%	0.0%	Zero false positive policy triggers
Single Core Throughput	76.6 MiB/s	82.1 MiB/s	Line rate inline stream processing

4. How the Roaring Bitmap Index (.rbix) Works

Every 64 bit fingerprint emitted by the WRL Chunker is indexed into the Roaring Bitmap Index (.rbix), implemented in pycaber:

1. Sharded Storage Artifacts (.rbix{b,k,d,t}.gz): During document indexing, per object .rbix bundles are staged to shared-storage (CQF/<cori>.rbixb.gz, .rbixk.gz, .rbixd.gz, .rbixt.gz) or workspace shards (rbix/ws_{N}/) containing serialized RoaringTreemap posting lists and Zstd compressed metadata dictionaries.
2. 13 Column Family RocksDB Engine (roaring_index.db): The consolidated index stores 64 bit fingerprint keys mapped to compressed RoaringTreemap bitmaps of 32 bit document slot IDs (u32), alongside forward maps (doc_to_nmers), byte offset tables (nmer_offsets), CORI registries (cori_meta), and IDF cluster tables (idf_clusters, idf_dag_edges). See the Schemas & shared-storage Reference for the complete 13 column family specification.
3. Atomic Double Buffered Generation Swaps: Index builds write to an isolated generation directory (build_{N}/) and atomically flip the active reader via a symlink swap (current -> build_{N}) and ArcSwap<RoaringIndexReader>, ensuring lock free concurrent queries during background corpus updates.

6. Key Architecture Decisions & Design Rationale

• ADR 1: Whitespace Remap on Leader vs Fixed Sliding Windows: Fixed size sliding windows create catastrophic index bloat. Pure content defined chunking like FastCDC shifts boundaries on minute edits. The WRL Chunker anchors boundaries on local extreme leaders and snaps cuts to natural whitespace and punctuation tokens, delivering 98.7% byte coverage at 40 byte granularity.
• ADR 2: Roaring Bitmap Indexing (.rbix) vs Dense Embedding Inversion: Dense vector representations struggle with exact subdocument spans and short 40 byte phrases. Caber maps 64 bit Buzhash fingerprints directly to Roaring Bitmaps stored across 13 RocksDB column families, providing deterministic submillisecond intersection without vector hallucination.
• ADR 3: Subdocument Boundary Preservation: Instead of treating files as monolithic opaque strings, Caber preserves exact source page numbers, table cells, and byte offsets through ByteSpanMap structures, ensuring that downstream policy redaction mutates only sensitive spans without destroying surrounding context.