Core Engine and Dependencies Configuration

This page documents the core engine configuration sections in pycaber: AWS, GLOBAL (including the GLOBAL.sdk.* identity and instance registry settings and GLOBAL.txnTokenConfig), Dependencies, WhitespaceRemapOnLeader, rbix, RKCQF, and ObjectSourceEndpoint scheme preservation (endpoint vs connection_url).

Every registered parameter derives its environment variable mechanically via ConfigEntry.derived_env_var() (CSI_<SECTION_UPPER_SNAKE>_<KEY_UPPER_SNAKE>). See Environment Variables and Governance for the derivation rules and the 7 kept legacy exception variables.

AWS

Controls AWS region selection and S3 client tuning for cloud and hybrid deployments.

Key	Type	Default	Sensitivity	Required In	Description
AWS.region	string	"us-west-1"	none	ecs-ec2, ecs-fargate, eks	AWS region for S3 and cloud service calls. Overridden by CSI_AWS_REGION.
AWS.s3MaxRetries	integer	20	none	Optional	Maximum retry attempts for transient S3 client errors (CSI_AWS_S3_MAX_RETRIES).

GLOBAL

Controls platform wide identity, storage mode, chunking algorithms, cryptographic keys, SDK runtime governance (GLOBAL.sdk.*), and transaction token enforcement (GLOBAL.txnTokenConfig).

Core GLOBAL Parameters

Key	Type	Default	Sensitivity	Required In	Description
GLOBAL.accountId	string	""	internal	all	Caber tenant or customer account identifier (CSI_GLOBAL_ACCOUNT_ID).
GLOBAL.chunkingAlgorithm	string	"AE"	none	Optional	Content defined chunking algorithm ("AE", "Rabin", "TTTD", "Rsync").
GLOBAL.configPollIntervalSecs	integer	0	none	Optional	Interval in seconds for polling remote configuration revisions (0 disables polling).
GLOBAL.coriId	string	""	internal	all	Canonical Organization and Resource Identifier prefix (CSI_GLOBAL_CORI_ID).
GLOBAL.csiBucket	string	"caber-data"	none	all	Default shared storage bucket name (CSI_GLOBAL_CSI_BUCKET).
GLOBAL.debug	boolean	false	none	Optional	Enables verbose debug logging across core Rust and Python modules (CSI_GLOBAL_DEBUG).
GLOBAL.defaultTenantId	string	"default"	none	Optional	Fallback tenant identifier when single tenant mode is active (CSI_GLOBAL_DEFAULT_TENANT_ID).
GLOBAL.deploymentMode	string	"docker-compose"	none	all	Deployment target topology ("docker-compose", "ecs-ec2", "ecs-fargate", "eks", "standalone").
GLOBAL.enableMultitenancy	boolean	false	none	Optional	Enforces strict per tenant index and graph namespace isolation (CSI_GLOBAL_ENABLE_MULTITENANCY).
GLOBAL.encryptionKeyId	string	""	secret	Optional	KMS or vault key identifier used for envelope encryption (CSI_GLOBAL_ENCRYPTION_KEY_ID).
GLOBAL["localPrefix"]	string	"/var/tmp/"	none	Optional	Base directory prefix for local filesystem storage mode (CSI_GLOBAL_LOCAL_PREFIX).
GLOBAL["localStorage"]	boolean	false	none	Optional	Routes object reads and writes to the local filesystem under get_global_output_directory() (CSI_GLOBAL_LOCAL_STORAGE).
GLOBAL.logFile	string	""	none	Optional	File path for structured log output (CSI_LOG_FILE kept exception).
GLOBAL.logLevel	string	"INFO"	none	Optional	Minimum log severity ("TRACE", "DEBUG", "INFO", "WARN", "ERROR", overridden by CSI_LOG_LEVEL).
GLOBAL.maxChunkSize	integer	65536	none	Optional	Upper bound in bytes for content defined chunks (CSI_GLOBAL_MAX_CHUNK_SIZE).
GLOBAL.minChunkSize	integer	4096	none	Optional	Lower bound in bytes for content defined chunks (CSI_GLOBAL_MIN_CHUNK_SIZE).
GLOBAL.serviceApiKey	string	""	secret	Optional	Shared service authentication key for internal microservice calls (CSI_GLOBAL_SERVICE_API_KEY).
GLOBAL.targetChunkSize	integer	16384	none	Optional	Target average chunk size in bytes for content defined chunking (CSI_GLOBAL_TARGET_CHUNK_SIZE).
GLOBAL.tlsVerify	boolean	true	none	Optional	Enforces TLS certificate verification on outbound HTTPS connections (CSI_DISABLE_SSL_VERIFY inverts this flag).

GLOBAL.sdk.* Identity, Instance Registry, and Auto Instrumentation

The GLOBAL.sdk subtree configures the PyCaber Unified Auto Instrumentation SDK (pycaber.auto and caber_sdk), controlling how SDK instances register with the platform, authenticate, and patch third party libraries and agent frameworks.

Key	Derived Environment Variable	Type	Default	Sensitivity	Description
GLOBAL.sdk.autoApprove	CSI_GLOBAL_SDK_AUTO_APPROVE	boolean	true	none	Automatically approves newly registered SDK instances in the instance registry without waiting for manual administrator approval in CaberUserConsole. Set to false in strict zero trust production environments.
GLOBAL.sdk.instanceExpirySecs	CSI_GLOBAL_SDK_INSTANCE_EXPIRY_SECS	integer	300	none	Heartbeat lease duration in seconds before an silent SDK instance transitions from ACTIVE to EXPIRED.
GLOBAL.sdk.expiredRetentionHours	CSI_GLOBAL_SDK_EXPIRED_RETENTION_HOURS	integer	24	none	Number of hours an EXPIRED SDK instance record is retained in OpenSearch before background compaction purges it.
GLOBAL.sdk.seedApiKey	CSI_GLOBAL_SDK_SEED_API_KEY	string	""	secret	Bootstrap API key presented by SDK instances during initial registration (POST /v1/sdk/register) when dynamic key provisioning is enabled.
GLOBAL.sdk.requireApiKey	CSI_GLOBAL_SDK_REQUIRE_API_KEY	boolean	false	none	Requires every inbound SDK telemetry, policy evaluation, and heartbeat request to present a valid registered API key.
GLOBAL.sdk.noPatchModules	CSI_GLOBAL_SDK_NOPATCH_MODULES	array<string>	[]	none	Comma separated list or JSON array of Python module names (out of the 15 supported HTTP/WS/gRPC libraries) that pycaber.auto must skip when registering wrapt.when_imported hooks.
GLOBAL.sdk.agentFramework.crewai	CSI_GLOBAL_SDK_AGENT_FRAMEWORK_CREWAI	array<string>	[]	none	Module or class prefixes to skip when pycaber.auto instruments crewai (Crew.kickoff, Agent.execute_task, BaseTool.run).
GLOBAL.sdk.agentFramework.langchain	CSI_GLOBAL_SDK_AGENT_FRAMEWORK_LANGCHAIN	array<string>	[]	none	Module or class prefixes to skip when pycaber.auto instruments langchain_core (Runnable.invoke, BaseTool.invoke, BaseChatModel._generate).
GLOBAL.sdk.agentFramework.langgraph	CSI_GLOBAL_SDK_AGENT_FRAMEWORK_LANGGRAPH	array<string>	[]	none	Module or node prefixes to skip when pycaber.auto instruments langgraph (Pregel.invoke, CompiledGraph, ToolNode).
GLOBAL.sdk.agentFramework.llamaindex	CSI_GLOBAL_SDK_AGENT_FRAMEWORK_LLAMAINDEX	array<string>	[]	none	Module or class prefixes to skip when pycaber.auto instruments llama_index.core (BaseQueryEngine.query, BaseRetriever.retrieve, FunctionTool.call).
GLOBAL.sdk.agentFramework.autogen	CSI_GLOBAL_SDK_AGENT_FRAMEWORK_AUTOGEN	array<string>	[]	none	Module or class prefixes to skip when pycaber.auto instruments autogen / autogen_agentchat (ConversableAgent.generate_reply, initiate_chat, BaseChatAgent.on_messages).

GLOBAL.txnTokenConfig

Configures outbound HTTP and gRPC host filtering for Transaction Token (Txn-Token) and On Behalf Of (OBO) header propagation in caber_sdk:

Key	Derived Environment Variable	Type	Default	Description
GLOBAL.txnTokenConfig.allowHosts	CSI_GLOBAL_TXN_TOKEN_CONFIG_ALLOW_HOSTS	array<string>	["*"]	Glob patterns of destination host or host:port authorities eligible to receive X-Caber-Txn-Token and OBO headers.
GLOBAL.txnTokenConfig.denyHosts	CSI_GLOBAL_TXN_TOKEN_CONFIG_DENY_HOSTS	array<string>	[]	Glob patterns of destination authorities that must never receive internal transaction tokens (evaluated before allowHosts, useful for third party SaaS APIs).

Dependencies

Defines connection endpoints, credentials, listen ports, and logical database table or index mappings for external infrastructure services (elastic-search, neo4j, shared-storage, kafka, postgres, redis, milvus).

Shared Dependency Schema (Dependencies.<service>.*)

Key Pattern	Type	Sensitivity	Description
Dependencies.<service>.url	string	internal	Primary service endpoint URL or host:port address (for example Dependencies.elastic-search.url, Dependencies.neo4j.url, Dependencies.shared-storage.url).
Dependencies.<service>.username	string	credential	Authentication username for the backing service.
Dependencies.<service>.password	string	secret	Authentication password or secret token for the backing service.
Dependencies.<service>.bucket	string	none	Bucket name for object storage services (Dependencies.shared-storage.bucket, default "caber-data").
Dependencies.<service>.region	string	none	Cloud region override for shared-storage.
Dependencies.<service>.useSsl	boolean	none	Enables TLS transport for the service connection.
Dependencies.<service>.verifyCerts	boolean	none	Enforces server certificate chain verification when useSsl is true.
Dependencies.<service>.caCertPath	string	none	Path to a custom PEM CA bundle for verifying the service certificate.
Dependencies.<service>.dbTables	object	none	Map of logical Caber table aliases to physical OpenSearch index names or relational table names (case preserved in raw_json).
Dependencies.<service>.listenPorts	object	none	Map of logical listener names (http, Bolt, grpc, metrics) to TCP port numbers.

Standard OpenSearch Logical Tables (Dependencies.elastic-search.dbTables)

Logical Key	Default Physical Index	Purpose
api_map	"csi_api_map"	Discovered API endpoints and registered data source discovery records.
auth_log	"csi_auth_log"	Edge authentication, OBO token exchange, and policy evaluation audit records.
chunk_index	"csi_chunks"	Content defined chunk metadata, SHA 256 digests, and CORI references.
comparison_log	"csi_comparison_log"	Passive parity comparison records emitted by Caber runtime.
doc_metadata	"csi_doc_metadata"	Document level ingest status, parser provenance, and NER entity summaries.
sdk_instances	"csi_sdk_instances"	Active and expired pycaber.auto SDK instance registration records.

Object_Sources Scheme Preservation (endpoint vs connection_url)

When configuring object storage connectors under Object_Sources or Dependencies.shared-storage, pycaber enforces a strict distinction between bare network authorities and scheme qualified URLs through ObjectSourceEndpoint:

• ObjectSourceEndpoint.authority() (endpoint field): Always returns the bare host:port authority with any leading http:// or https:// scheme stripped (for example "minio.example.com:9000" or "s3.us-west-1.amazonaws.com"). Used by low level socket dialers and legacy S3 SDK builders that construct the scheme separately from useSsl.
• ObjectSourceEndpoint.url() (connection_url field): Preserves or reconstructs the full scheme qualified URL ("http://minio.example.com:9000" or "https://s3.us-west-1.amazonaws.com"). When the raw configuration specifies an explicit http:// or https:// prefix, ObjectSourceEndpoint records that scheme losslessly so connection_url never silently upgrades an explicit http:// MinIO dev endpoint to https:// or strips TLS from a production endpoint.

WhitespaceRemapOnLeader

Configures deterministic whitespace normalization and byte offset remapping when electing a canonical leader chunk across near duplicate text segments.

Key	Derived Environment Variable	Type	Default	Description
WhitespaceRemapOnLeader.enabled	CSI_WHITESPACE_REMAP_ON_LEADER_ENABLED	boolean	true	Enables byte span remapping across runs of collapsible ASCII and Unicode whitespace when mapping follower chunks onto a canonical leader chunk.
WhitespaceRemapOnLeader.maxDriftBytes	CSI_WHITESPACE_REMAP_ON_LEADER_MAX_DRIFT_BYTES	integer	512	Maximum cumulative byte offset drift tolerated between follower and leader before falling back to an independent chunk registration.
WhitespaceRemapOnLeader.preserveNewlines	CSI_WHITESPACE_REMAP_ON_LEADER_PRESERVE_NEWLINES	boolean	true	Treats line break boundaries (\n and \r\n) as structural anchors during whitespace collapse.