Caber Systems Documentation

Appearance

Evaluation & GraphRAG Benchmarks User Guide (CaberEvalBenchmarks & GraphRagBench) ​

This page is Part 2 of the Demos, Data Factory & Benchmarks User Guide, providing comprehensive installation and operation instructions for CaberEvalBenchmarks (CaberEvalBenchmarks) and GraphRagBench (GraphRagBench). See Part 1: Demos & Data Factory User Guide for CaberDemos and CaberDataFactory.

Part 1: CaberEvalBenchmarks Installation & Operation User Guide ​

1.2 Installation & Virtual Environment Setup ​

Always use the repository's local .venv virtual environment:

bash
CaberEvalBenchmarks/.venv/bin/pip install -e "CaberEvalBenchmarks[all]"
CaberEvalBenchmarks/.venv/bin/pip install -e "PyCaber[auto]"
CaberEvalBenchmarks/.venv/bin/pip install -e "CaberDataFactory"

1.3 Running CaberEvalBenchmarks Step by Step ​

Step 2: Run Specific Individual Benchmarks (e.g., WRL Chunker Coverage & Metapolicies) ​

bash
CaberEvalBenchmarks/.venv/bin/python \
  CaberEvalBenchmarks/run_eval.py \
  --benchmarks B1_wrl_coverage,B3_stale_clustering,B4_metapolicy_resolution \
  --min-chunk-bytes 40 \
  --verbose

Step 3: Run Live LLM & Agent Evaluation Mode ​

To evaluate live OpenAI / Anthropic / Bedrock completions and multi agent MCP tool chains under pycaber(auto) enforcement:

bash
OPENAI_API_KEY=sk-... \
CaberEvalBenchmarks/.venv/bin/python \
  CaberEvalBenchmarks/run_eval.py \
  --benchmarks B6_principal_obo_chain,B7_provenance_origin_tree,B10_end_to_end_rag_gov \
  --mode live_llm \
  --model gpt-4o

Step 4: Inspect Generated Reports & Regression Gates ​

Each run produces three artifacts in --output-dir:

  1. summary_metrics.json: Machine readable JSON containing precision, recall, F1, p50/p95/p99 latencies, and pass/fail status against target SLAs.
  2. per_query_audit.jsonl: Detailed per query trace with matched WRL Chunker byte spans, cluster assignments, and DecisionReceipt objects.
  3. EVALUATION_REPORT.md: Formatted Markdown table comparing current results against the golden baseline (baseline/golden_metrics.json).

To run the automated CI regression gate (exits non zero if any metric regresses beyond tolerance):

bash
CaberEvalBenchmarks/.venv/bin/pytest \
  CaberEvalBenchmarks/tests/test_benchmark_slas.py

Part 2: GraphRagBench Installation & Operation User Guide ​

2.2 Installation & Virtual Environment Setup ​

bash
GraphRagBench/.venv/bin/pip install -e "GraphRagBench[all]"
GraphRagBench/.venv/bin/pip install -e "PyCaber[auto]"

2.3 Running GraphRagBench Step by Step ​

Step 2: Build Indices Across Selected Systems ​

bash
GraphRagBench/.venv/bin/python -m graphrag_bench.cli index \
  --corpus-dir /tmp/graphrag_bench/sec_corpus \
  --systems caber_graphrag,lightrag,vector_baseline \
  --workers 8

During indexing, GraphRagBench records total wall clock indexing time, token/API cost, storage footprint, and index coverage (98.7% for caber_graphrag via the WRL Chunker).

Step 4: Generate Comparative Leaderboard & Visualizing Results ​

bash
GraphRagBench/.venv/bin/python -m graphrag_bench.cli report \
  --input-dir GraphRagBench/reports/sec_run \
  --format markdown,json,html

The generated report compares systems across five dimensions:

  1. Multi Hop Answer Faithfulness & Groundedness (0.0 to 1.0): Verified both by LLM as a Judge and exact WRL Chunker 40 byte provenance span overlap.
  2. Stale Data Immunity Rate (%): Percentage of queries where outdated facts from STALE_REPLICA documents were excluded in favor of CANONICAL documents.
  3. Unauthorized Data Leakage Rate (%): Percentage of responses containing any 40 byte fragment forbidden to the requesting Principal (0.0% for caber_graphrag).
  4. Query Latency (p50, p95, p99 ms) and Indexing Throughput (MB/sec).
  5. Provenance Completeness (%): Accuracy of cited source documents and exact byte ranges.