Connector SDK & CLI

To enforce zero trust policies based on Principal identity and live data attributes, Caber connects to external enterprise systems across four domains:

• SaaS Application Event Logs and Activity Streams (audit and activity feeds from Slack, GitHub, Jira, Confluence, Salesforce, Google Workspace, and Microsoft 365)
• External Identity Providers (user, group, role, and OBO delegation mappings from Okta, Microsoft Entra ID, AWS IAM Identity Center, Keycloak, and SPIFFE or SPIRE)
• Metadata and Data Catalog Sources (classification labels, sensitivity tags, and data ownership records from Collibra, Alation, Apache Atlas, Snowflake Horizon, and AWS Glue)
• Object and Document Storage Sources (enterprise corpora in Amazon S3, Google Cloud Storage, Azure Blob Storage, SharePoint, Google Drive, Box, and relational or vector stores)

Figure 1: Connector Architecture Ingesting SaaS Streams, IdPs, and Object Storage with Cursor Checkpointing

## 1. Connector Architecture & System Communication

Caber connectors operate as standalone ingestion agents or in process service tasks that bridge enterprise systems to the core Rust processing engine (caber_core and streaming_core).

Data Filtering & Normalization

• VRL Transformation: Ingestion feeds apply Vector Remap Language (vrl_engine) scripts to strip noise and normalize proprietary vendor JSON into canonical Caber event envelopes.
• Pre Filter Rules: Connectors can drop unmonitored event types, non document file extensions, and healthcheck noise before allocating memory envelopes.

Connector Configuration

Connectors are declared under CONFIG_SCHEMA.json inside sections Identity_Sources, Object_Sources, and Metadata_Sources. Each connector instance specifies:

• enabled (boolean): Activation state.
• sync_interval_seconds (integer): Polling frequency for batch or cursor endpoints.
• credential_ref (string): Secret name in AWS Secrets Manager or local environment.
• vrl_rule_path (string): Path to custom field remap logic.

2. Four Connector Base Interfaces

Connector Family	Python Base Class (pycaber.connector_sdk)	Rust Trait (caber_connector_sdk)	Target Configuration Section	Output Destination
SaaS Event Log Connector	BaseEventLogConnector	EventLogConnector	Metadata_Sources / Object_Sources	Emits AuditEventRecord and activity spans to csi-audit-logs, csi-traces, and Neo4j provenance nodes
Identity Provider Connector	BaseIdentityConnector	IdentitySourceConnector	Identity_Sources	Emits PrincipalIdentityRecord and group graphs to csi-identity-cache and Neo4j (:Principal)-[:MEMBER_OF]->(:Group)
Metadata and Catalog Connector	BaseMetadataConnector	MetadataSourceConnector	Metadata_Sources	Emits ResourceMetadataPatch (classification, owner, tags, policy id) to csi-document-db and Neo4j
Object and Document Connector	BaseObjectConnector	ObjectSourceConnector	Object_Sources	Streams RawDocumentEnvelope to shared-storage (raw/) and triggers DocParser plus WRL Chunker indexing

3. Exhaustive caber-connector CLI Reference

Global Command Options

• -v, --verbose: Enable verbose debug logging during command execution.
• -h, --help: Display help and command usage information.
• -V, --version: Print binary version string.

caber-connector rules test

Executes a VRL rule file against a sample JSON payload to test routing conditions, schema transformations, and output envelopes.

• Syntax:

  caber-connector rules test <file> -i <input> [-v]

• Parameters:
  • <file> (string, required): Path to .vrl rule file.
  • -i, --input <input> (string, required): Sample JSON input, specified as either an inline JSON string or a file path to a .json document.
  • -v, --verbose (flag, optional): Print intermediate compilation AST and execution timings.
• Evaluation Output:
  • MATCH: Emits transformed canonical event envelope formatted as pretty printed JSON.
  • DROPPED: Displays rule routing condition evaluation explaining why the input was dropped.
  • NO OUTPUT: Explains transformation branches that resulted in empty emits.
• Exit Codes:
  • 0: Rule executed without runtime errors.
  • 1: File not found, invalid JSON input, or VRL execution error.

4. Comprehensive Rust SDK Function & Method Signatures

The Rust SDK (caber_connector_sdk) provides the core trait hierarchy, deduplication engine, and lifecycle primitives implemented by connectors: