Services, SDKs, Ingest, and Policies Configuration

This page documents the service, document parser, ingest pipeline, graph linker, scanner, connector, and policy configuration sections in pycaber.

DocParser

Configures the PyCaber Document Parser, including the per format parser router (DocParser.formatRouter), the sparse extraction fallback gate (DocParser.formatRouter.fallbackGate), and the Named Entity Recognition engine (DocParser.nerEngine).

Key	Derived Environment Variable	Type	Default	Description
DocParser.host	CSI_DOC_PARSER_HOST	string	"0.0.0.0"	Bind address for the Document Parser HTTP/gRPC service.
DocParser.port	CSI_DOC_PARSER_PORT	integer	8084	Listen port for the Document Parser service (CSI_DOC_PARSER_PORT kept exception).
DocParser.workers	CSI_DOC_PARSER_WORKERS	integer	4	Number of concurrent worker threads or subprocesses (CSI_DOC_PARSER_WORKERS kept exception).
DocParser.mode	CSI_DOC_PARSER_MODE	string	"Standalone"	Execution topology ("Standalone" for router_fold, "Internal" or "External" for router_split).
DocParser.nerEngine	CSI_DOC_PARSER_NER_ENGINE	string	"gliner2"	Active Named Entity Recognition backend ("gliner2" zero shot transformer default, "spacy", "regex", or "disabled").
DocParser.glinerEndpoint	CSI_DOC_PARSER_GLINER_ENDPOINT	string	"http://127.0.0.1:8765"	Endpoint URL for the host GPU gliner2_model_server.py daemon.
DocParser.doclingEndpoint	CSI_DOC_PARSER_DOCLING_ENDPOINT	string	"http://127.0.0.1:8766"	Endpoint URL for the host GPU docling_serve_wrapper.py daemon (verified via GET /caber/attest).
DocParser.maxDocumentBytes	CSI_DOC_PARSER_MAX_DOCUMENT_BYTES	integer	104857600	Maximum input file size in bytes (100 MB default) accepted before streaming rejection.
DocParser.ocrEnabled	CSI_DOC_PARSER_OCR_ENABLED	boolean	true	Enables optical character recognition inside docling for scanned PDFs and image inputs.
DocParser.formatRouter.defaultChain	CSI_DOC_PARSER_FORMAT_ROUTER_DEFAULT_CHAIN	array<string>	["docling", "tika"]	Ordered parser chain evaluated when an input format does not match an explicit key in DocParser.formatRouter.routes.
DocParser.formatRouter.routes	CSI_DOC_PARSER_FORMAT_ROUTER_ROUTES	object	See below	Case preserved map from canonical format name (markdown, pdf, docx, html, csv, json, txt) to an ordered array of parser lane names.
DocParser.formatRouter.overrideAllowlist	CSI_DOC_PARSER_FORMAT_ROUTER_OVERRIDE_ALLOWLIST	array<string>	["markdown", "docling", "pymupdf4llm", "tika", "plain_text"]	Allowlist of parser lane identifiers that callers may request via per request parser_override headers.
DocParser.formatRouter.fallbackGate.minTextChars	CSI_DOC_PARSER_FORMAT_ROUTER_FALLBACK_GATE_MIN_TEXT_CHARS	integer	200	Minimum extracted non whitespace Unicode character count required from a fast non docling parser lane when input byte length meets minInputBytes.
DocParser.formatRouter.fallbackGate.minInputBytes	CSI_DOC_PARSER_FORMAT_ROUTER_FALLBACK_GATE_MIN_INPUT_BYTES	integer	10240	Minimum raw input document size in bytes (10 KiB) at which Caber runtime enforces the minTextChars threshold and triggers a single hop docling retry with a {sha}.redirect.json pointer (gate-v1:floor={minTextChars},min={minInputBytes}).

Default DocParser.formatRouter.routes Table

{
  "DocParser": {
    "nerEngine": "gliner2",
    "formatRouter": {
      "defaultChain": ["docling", "tika"],
      "routes": {
        "markdown": ["markdown"],
        "pdf": ["pymupdf4llm", "docling", "tika"],
        "docx": ["docling", "tika"],
        "pptx": ["docling", "tika"],
        "xlsx": ["docling", "tika"],
        "html": ["docling", "tika"],
        "txt": ["plain_text"]
      },
      "overrideAllowlist": [
        "markdown",
        "docling",
        "pymupdf4llm",
        "tika",
        "plain_text"
      ],
      "fallbackGate": {
        "minTextChars": 200,
        "minInputBytes": 10240
      }
    }
  }
}

Process_API, API_Tap, and ApiMap

Configures runtime API traffic inspection, sidecar tap ingestion, and API topology discovery.

Key	Derived Environment Variable	Type	Default	Description
Process_API.bindAddress	CSI_PROCESS_API_BIND_ADDRESS	string	"0.0.0.0:8082"	Listen address for the Process_API inline evaluation and ingestion service.
Process_API.maxConcurrentRequests	CSI_PROCESS_API_MAX_CONCURRENT_REQUESTS	integer	1024	Maximum in flight requests processed concurrently before applying HTTP 429 backpressure.
Process_API.requestTimeoutMs	CSI_PROCESS_API_REQUEST_TIMEOUT_MS	integer	5000	Per request evaluation timeout in milliseconds.
API_Tap.bufferCapacity	CSI_API_TAP_BUFFER_CAPACITY	integer	8192	Bounded ring buffer capacity for asynchronous mirrored HTTP and gRPC traffic frames.
API_Tap.dropOnFull	CSI_API_TAP_DROP_ON_FULL	boolean	true	Drops mirrored tap frames without blocking the caller when the ring buffer is full.
API_Tap.sampleRate	CSI_API_TAP_SAMPLE_RATE	number	1.0	Fraction of non sensitive traffic frames (0.0 to 1.0) sampled for schema discovery.
ApiMap.discoveryIntervalSecs	CSI_API_MAP_DISCOVERY_INTERVAL_SECS	integer	60	Interval in seconds for aggregating observed endpoints and flushing discovery records to csi_api_map.
ApiMap.inferOpenApiSchema	CSI_API_MAP_INFER_OPEN_API_SCHEMA	boolean	true	Automatically infers JSON Schema and OpenAPI parameter types from observed request and response payloads.

S3_Scanner, SQL_Scanner, and Kafka_Scanner

Configures batch and streaming data source scanners built on caber_connector_sdk and coordinator.

Key	Derived Environment Variable	Type	Default	Description
S3_Scanner.concurrency	CSI_S3_SCANNER_CONCURRENCY	integer	16	Number of parallel object download and chunking workers per bucket scan job.
S3_Scanner.includePrefixes	CSI_S3_SCANNER_INCLUDE_PREFIXES	array<string>	[]	Optional list of S3 key prefixes to include (empty array scans the entire bucket).
S3_Scanner.excludePrefixes	CSI_S3_SCANNER_EXCLUDE_PREFIXES	array<string>	[]	List of S3 key prefixes skipped during bucket enumeration.
S3_Scanner.incrementalCheckpoint	CSI_S3_SCANNER_INCREMENTAL_CHECKPOINT	boolean	true	Skips objects whose ETag and LastModified timestamp match the last committed checkpoint in OpenSearch.
SQL_Scanner.fetchSize	CSI_SQL_SCANNER_FETCH_SIZE	integer	5000	Number of rows fetched per cursor batch when scanning relational tables.
SQL_Scanner.maxConnections	CSI_SQL_SCANNER_MAX_CONNECTIONS	integer	8	Maximum connection pool size per registered database source.
SQL_Scanner.sampleRowsLimit	CSI_SQL_SCANNER_SAMPLE_ROWS_LIMIT	integer	0	Optional row cap per table (0 scans all rows).
Kafka_Scanner.consumerGroupId	CSI_KAFKA_SCANNER_CONSUMER_GROUP_ID	string	"caber-scanner"	Kafka consumer group identifier used for offset tracking.
Kafka_Scanner.autoOffsetReset	CSI_KAFKA_SCANNER_AUTO_OFFSET_RESET	string	"latest"	Starting offset policy ("earliest" or "latest") when no committed offset exists.
Kafka_Scanner.maxPollRecords	CSI_KAFKA_SCANNER_MAX_POLL_RECORDS	integer	500	Maximum messages fetched per poll cycle.

ENV_Manager, Dashboard, and Auth_Analyzer

Configures environment lifecycle management, the administrative dashboard API, and authentication graph analysis.

Key	Derived Environment Variable	Type	Default	Description
ENV_Manager.reconcileIntervalSecs	CSI_ENV_MANAGER_RECONCILE_INTERVAL_SECS	integer	30	Interval in seconds between service health and container state reconciliation loops.
ENV_Manager.noRestart	CSI_NO_RESTART	boolean	false	Disables automatic container or worker restarts on crash (CSI_NO_RESTART kept exception).
Dashboard.bindAddress	CSI_DASHBOARD_BIND_ADDRESS	string	"0.0.0.0:8080"	Listen address for the backend dashboard and control plane API server.
Dashboard.sessionTtlSecs	CSI_DASHBOARD_SESSION_TTL_SECS	integer	3600	Administrator session token lifetime in seconds.
Auth_Analyzer.anomalyThreshold	CSI_AUTH_ANALYZER_ANOMALY_THRESHOLD	number	0.85	Risk score threshold (0.0 to 1.0) for flagging anomalous privilege escalation or cross tenant OBO delegation paths.
Auth_Analyzer.windowMinutes	CSI_AUTH_ANALYZER_WINDOW_MINUTES	integer	60	Sliding time window in minutes for aggregating authentication and token exchange events.

Compliance_Policies, Identity_Sources, and Object_Sources

These three top level sections store structured, case preserved declarations loaded via Caber runtime and the policy engine:

• Compliance_Policies: Map of named zero trust and data protection policies. Each policy specifies mode ("enforce", "audit", "disabled"), matching classifications or CORI tags (c<id> / g<id>), allowed principal roles or groups, and target action ("ALLOW", "REDACT", "BLOCK"). Downgrading a policy mode (for example from "enforce" to "audit") is classified as OperationClass.PolicyModeDowngrade in sys_config_control and requires two distinct PlatformAdmin approval grants.
• Identity_Sources: Map of configured identity providers (Okta, EntraID, AWS_IAM, OIDC, LDAP) with OIDC issuer URLs, JWKS endpoints, group claim mappings, and sync intervals. Mutating an entry under Identity_Sources is classified as OperationClass.IdentityProviderChange and requires two distinct PlatformAdmin approval grants.
• Object_Sources: Map of registered external data sources (s3, minio, gcs, azure_blob, cloudtrail, postgres, snowflake). Loaded exclusively through Caber runtime(), which validates each entry's CORI identifier (leading 'c' or 'g', <= 10 lowercase ASCII alphanumeric characters, 'gss' and 'cs3aws' reserved), preserves scheme fidelity via ObjectSourceEndpoint.authority() (endpoint) and ObjectSourceEndpoint.url() (connection_url), and populates the lock free ArcSwap SourceRegistry.