Stale Source Detection & Two Stage IDF Clustering

1. What It Is

Enterprise storage estates rarely hold a single clean copy of important documents. Instead, they accumulate exported spreadsheets, emailed contract drafts, departmental wiki copies, and staging snapshots. When a RAG pipeline or AI agent retrieves fragments from these copies, two critical failures occur if the platform cannot distinguish canonical sources from stale replicas:

1. Knowledge Drift: The AI model answers using superseded clauses, outdated pricing tables, or revoked procedures from an older draft.
2. Authorization Drift: An older draft sitting in a shared folder may have permissive ACLs, whereas the current authoritative document in a restricted vault has strict executive only permissions (or vice versa).

Caber solves both problems through Two Stage Inverse Document Frequency (IDF) Clustering and Directed Version Lineage DAG (EditCostTree) construction. Every indexed document is grouped into its content equivalence cluster, ordered into a directed evolution graph, and assigned a topological freshness_rank (0 for canonical head versions, > 0 for stale ancestors).

2. How It Fits Into the Bigger Picture

Figure 1: Two Stage IDF clustering pipeline from .rbix nmer postings to EditCostTree DAG direction and freshness_rank assignment.

Two Stage IDF Clustering runs automatically during every index generation build (Link_Data / Caber runtime) and feeds three downstream consumers:

• Multisource Metapolicies (metapolicy.py & caber_governance): Filters matched source objects to canonical (freshness_rank == 0) versions during policy evaluation, or fails closed across the entire cluster if an AI response was synthesized purely from stale sources.
• Runtime RAG & Agent Interception (caber-core): Suppresses or flags stale context chunks before they pollute LLM prompts.
• CaberUserConsole (/admin/overlapping & /admin/contextGraph): Displays document version clusters, similarity scores, and directed parent to child evolution arrows so operators can remediate shadow copies.

3. How Two Stage IDF Clustering Works

Stage 1: Container & Path Version Grouping (CoriPolicy)

Every indexed object carries a structured CORI:

<scheme>⋊⋉<container>⋊⋉<object_path>ⓥ<version_timestamp>

Stage 1 (CoriPolicy) groups all snapshots that share an identical <scheme>⋊⋉<container>⋊⋉<object_path> prefix. Within each same path group, snapshots are ordered deterministically by their ⓥ<version_timestamp> epoch, immediately linking linear storage revisions (such as S3 object versions or Git file commits) without requiring pairwise set comparisons.

Stage 2: Cross Container IDF Weighted Similarity (cluster_by_similarity)

To detect copies that were renamed, moved across buckets (for example, copied from SharePoint into S3 or embedded into a Confluence page), or edited, Stage 2 computes IDF weighted nmer similarity across all documents sharing at least minNmerMatches fingerprints in .rbix:

1. Corpus Wide Nmer IDF Weighting: For each 64 bit WRL Chunker fingerprint $h$ in a corpus of $N$ documents, Caber computes its Inverse Document Frequency weight:

   $$\text{idf}(h)=\ln (1+\frac{N}{\text{df}(h)})$$

   Rare clauses (such as a specific indemnification paragraph or patient diagnosis) receive high IDF weight, while common formatting phrases receive low weight.

2. Boilerplate & Template Hub Filtering (JunkThresholds): Before clustering, Caber identifies "junk hubs" (corporate email footers, standard NDA boilerplate, or empty form templates that match thousands of unrelated files) using two statistical thresholds configured in Link_Data.junkStDevs:

   • Absolute In Degree (junkStDevs.absolute = 2.5): Flags documents whose match in degree exceeds ${\mu }_{\text{deg}}+2.5{\sigma }_{\text{deg}}$.
   • Relative Density (junkStDevs.relative = 1.5): Flags documents whose ratio of match in degree to total nmers exceeds ${\mu }_{\text{rel}}+1.5{\sigma }_{\text{rel}}$. Flagged boilerplate nmers are excluded from driving false transitive cluster merges.

3. Pairwise IDF Similarity (sim_idf > t_idf): For any pair of documents $(A,B)$ with nmer sets $S_A$ and $S_B$, the IDF weighted Jaccard similarity is:

   $${\text{sim}}_{\text{idf}}(A,B)=\frac{\sum _{h\in S_A\cap S_B}\text{idf}(h)}{\sum _{h\in S_A\cup S_B}\text{idf}(h)}$$

   Pairs exceeding the threshold $t_{\text{idf}}$ (default 0.35) are merged into connected components via cluster_by_similarity.

Directed Version Lineage DAG (EditCostTree)

Once a connected cluster of related documents is formed, Caber determines which document is the canonical head (freshness_rank = 0) and which documents are stale ancestors or forks (freshness_rank > 0) by constructing a minimum cost directed spanning DAG (EditCostTree):

• Asymmetric Edit Direction Scoring (idfDirectionMethod): Documents normally grow and refine over time (accumulating sections, signatures, or audit notes). For candidate edge $u\to v$, Caber computes the IDF weighted insertion cost (nmers in $v\setminus u$) versus deletion cost (nmers in $u\setminus v$).
• Continuous Timestamp Confidence Modulation (idfTimestampModulated & idfMetadataBeta): Storage modification timestamps (mtime) can be noisy when files are bulk copied during backups. When idfTimestampModulated = true, Caber combines the content edit cost with a storage metadata ordering penalty weighted by $\beta =\text{idfMetadataBeta}$:$$\text{Cost}(u\to v)={\text{EditCost}}_{\text{idf}}(u\to v)+\beta \cdot max(0,\tanh \left(\frac{\mathrm{\Delta }{t}_{u,v}}{\tau }\right))$$
• Topological Freshness Ranks: The terminal sinks of the resulting DAG (documents that supersede earlier versions without being superseded themselves) receive freshness_rank = 0 (Canonical / Fresh). All upstream ancestors receive freshness_rank = 1, 2, ... proportional to their topological distance from the canonical head.

4. Why It Matters: Fail Closed Stale Source Governance

Retrieval Scenario	Matched Documents in Cluster	freshness_rank	Governance Behavior
Fresh Canonical Match	contract_v3_final.pdf (rank=0) and contract_v1_draft.pdf (rank=2)	Contains rank == 0	Metapolicy evaluates permissions of the canonical rank == 0 source (contract_v3_final.pdf); outdated draft permissions are ignored.
Stale Only Match	Fragment matches only contract_v1_draft.pdf (rank=2) because clause was deleted in v3	All matches have rank > 0	Fail Closed: Caber flags the fragment as stale_only, either evicting the stale chunk from the AI context or requiring the caller to hold permissions across every version in the lineage cluster (intersect).
Divergent Fork Match	Two parallel branches branch_A.docx (rank=0) and branch_B.docx (rank=0)	Multiple rank == 0 heads	Metapolicy intersects permissions across both canonical heads (Allow(A) ∩ Allow(B)).

6. Key Architecture Decisions & Design Rationale

• ADR 1: Content Grounded Lineage vs Metadata Only Ordering: Relying solely on filesystem timestamps or object version tags fails when files are migrated between cloud buckets or copied across file shares. Caber uses asymmetric edit cost over WRL Chunker nmers as the primary topological signal, with storage timestamps serving only as a secondary modulation factor.
• ADR 2: Two Stage IDF Filtering vs Naive Jaccard Matrix: Pairwise Jaccard comparison across an entire enterprise corpus is O(N^2). Stage 1 groups paths deterministically via CoriPolicy in O(N) time, and Stage 2 only evaluates candidate pairs sharing high IDF rare nmers in .rbix, dropping boilerplate hubs via JunkThresholds.
• ADR 3: Fail Closed Stale Only Retrieval: When an AI prompt matches only an obsolete document version whose text was deleted from the canonical head, Caber fails closed by either evicting the chunk from the model prompt or requiring the caller to satisfy permissions across all cluster ancestors.