RAG Knowledge Base

Semantic retrieval + grounded Q&A over Dr. Wael's engineering documents (tenders, contracts, research, P6 schedules, code, family, general). Cross-lingual Arabic + English. Local LanceDB — no cloud DB, separate from paperclip.

When to use

• "Search my tenders for the liquidated-damages clause"
• "Ask the knowledge base: what retention percentage applies in the KCPC contract?"
• Ingesting a folder of PDFs/DOCX/MD and querying them later with citations.

Quick start (ALWAYS load env first)

bash
bash -c '. /data/.openclaw/secrets/env.sh; python3 skills/rag-knowledge-base/rag.py <cmd> ...'

Commands

| Command | Purpose | |---|---| | ingest <path> [--collection NAME] [--tag T] [--title T] | Ingest a file OR a directory (recursive). Re-ingest replaces old chunks for the same doc. | | search "<query>" [--collection NAME] [--k N] | Semantic search; prints similarity %, source citation, snippet. | | ask "<question>" [--collection NAME] [--k N] | Retrieve + LLM answer with inline [source: file#chunkN] citations. | | list [--collection NAME] | List ingested documents. | | stats | Counts by collection + language. |

Collections: code, tenders, contracts, research, p6, family, general (default general).

Examples

bash
Ingest a folder of tender PDFs
... rag.py ingest ~/tenders/kcpc_2026 --collection tenders --tag kcpc
Cross-lingual search (Arabic query hits English+Arabic docs)
... rag.py search "ما هي غرامة التأخير؟" --collection tenders --k 5
Grounded Q&A with citations
... rag.py ask "What is the retention percentage and release schedule?" --collection contracts

Architecture

• Storage: LanceDB at skills/rag-knowledge-base/db/ (table chunks, 1024-dim vectors).
• Embeddings: Voyage-3 (1024-dim) primary with 429 retry/backoff; Gemini gemini-embedding-001 (output_dimensionality=1024) fallback. Both cross-lingual.
• Vector-space consistency (critical): Each collection is locked to the first embedding model that successfully ingested into it. Queries embed with that same model. This prevents mixing incompatible vector spaces (which would silently corrupt cosine similarity). stats/list reflect per-chunk embed_model.
• Chunking: ~800 tokens, 100 overlap, paragraph-aware with sentence-level fallback for oversized paragraphs.
• Formats: .txt .md .markdown (raw), .pdf (pypdf), .docx (python-docx).
• Answering: Claude claude-opus-4-8 primary → Gemini gemini-3.1-pro-preview fallback. System prompt forces source-only answers, inline citations, language matching (AR/EN), and no LaTeX.

Guarantees / behavior notes

• The ask model answers ONLY from retrieved chunks and explicitly says when the sources don't contain the answer (verified anti-hallucination behavior — it will not invent values).
• If Voyage is rate-limited mid-session it retries with exponential backoff (up to ~20s). If a collection was built on Voyage, queries stay on Voyage to preserve the vector space; if Voyage is hard-down, re-run shortly after (transient RPM throttle clears in seconds).
• To rebuild a collection on a different model, delete skills/rag-knowledge-base/db/ and re-ingest.

Tested (2026-05-31)

• Ingested AR+EN KCPC tender (1 chunk) + PT-slab research (7 chunks).
• English search ranked the correct punching-shear chunk #1.
• Arabic query retrieved the Arabic section cross-lingually.
• ask (EN + AR) returned grounded answers with correct [source: file#chunkN] citations and honestly flagged a missing value rather than hallucinating it.