Architecture Overview & Unified Data Flow

4. End to End Data Flow Walkthrough

Plane 2: Synchronous SDK & Gateway Interception Flow

1. Zero Code Instrumentation (pycaber.auto): Calling import pycaber.auto activates pycaber_wrap monkeypatch hooks across 15 HTTP/WebSocket/gRPC frameworks (FastAPI, Flask, Django, Starlette, AIOHTTP, Tornado, Quart, Falcon, Bottle, Pyramid, Sanic, Litestar, Grpcio, Websockets, Requests/HTTPX) and 5 AI/agent frameworks (LangChain/LangGraph, LlamaIndex, CrewAI, AutoGen, OpenAI/Anthropic/MCP clients).
2. Streaming WRL Characterization: Request and response payloads stream through streaming_core::WrlProcessor at 76.6 MiB/s per core, extracting 64 bit ChunkId fingerprints and exact [start_byte, end_byte) coordinates.
3. OBO Identity & Protocol Detection: caber_identity and caber_auth extract On Behalf Of (obo.user) JWT/HMAC delegation chains across A2A, MCP (tools/call, resources/read), and LLM API boundaries, binding every tool invocation and completion to the initiating human principal.

5. Architectural Pillar & Package Mapping

Architectural Pillar	Rust Crates (PyCaber/crates/)	Active Python Packages (PyCaber/python/)	Storage Backends & Queues	Runtime Binaries & Listeners
Transport, Discovery & Coordination	caber_transport, coordinator, caber_hub, ingest_stage, caber_mcp	pycaber_connector, pycaber_types	SQLite (Fold), PostgreSQL + AWS SQS (Split), shared-storage	caber-hub (/v1/dependencies), caber-core (QueueIngestor on csi-s3event-q)
Fingerprinting, WRL & RBIX Indexing	streaming_core, roaring_index, rocksdb_storage, base_utils	pycaber (PyO3 bindings)	RocksDB (roaring_index.db 13 CFs), .rbixb + .json sidecars	caber-core (in memory + RocksDB reader), caber-cli
Document Parsing & Zero Shot NER	ingest_core, document_parser_client, content_typer, sys_config	pycaber_parser (markdown_runner.py, gliner2_runner.py, docling_runner.py)	shared-storage (ParserCache/, redirect-{hash}/)	scripts/parser_service.py (:8765 GLiNER2, :8766 Docling), document_parser_container
Lineage, Context Graph & Governance	graph_builder, caber_governance, caber_auth, caber_identity, caber-core	pycaber, pycaber_wrap (pycaber_graph, pycaber_graph_lib, pycaber_ingest decommissioned into caber-core)	Neo4j (bolt://neo4j:7687), OpenSearch (csi_doc_graph, csi_audit_trail)	caber-core (HTTP :8080, gRPC :50051, Redis Pub/Sub, QueueIngestor)