08-13-2026, 08:00 AM
RAG is dead, long live RAG — the discourse cycles but the practical problem hasn't gone away: how do you get your organization's actual data into an LLM in a way that works, doesn't hallucinate, and doesn't cost a fortune?
I'm curious what stack people are *actually* running for internal knowledge bots / data Q&A in 2026. Not what's trending on HN, what's in production.
My small-team stack (under 50 people):
- Ingestion: Unstructured.io for docs, a custom Python ETL for SQL databases, plain crawlers for internal wikis.
- Chunking: 512-1024 tokens with overlapping, plus parent-document retrieval (store small chunks, return full parent).
- Embeddings: Still text-embedding-3-large from OpenAI; tried a couple of open-source ones and the quality delta was real for our messy docs.
- Vector DB: Postgres + pgvector for almost everything. Only reach for a dedicated vector DB (Pinecone/Qdrant) when scale demands it.
- Retrieval: Hybrid search (BM25 + vectors) via pgvector, plus a reranker.
- LLM: Claude for long-context synthesis, GPT-4.1 for structured/tabular answers.
- Evals: Ragas + a small goldenset I run every week.
- Guardrails: PII redaction before data ever hits the embedder; per-doc permissions enforced at retrieval time.
What I'm still unhappy with:
- Table/spreadsheet retrieval. Still bad in most tools.
- Updating/changing content without full re-indexing.
- Telling the user "I don't know" confidently instead of guessing.
What's your stack? And what's the one piece you'd replace tomorrow if a better option existed?
I'm curious what stack people are *actually* running for internal knowledge bots / data Q&A in 2026. Not what's trending on HN, what's in production.
My small-team stack (under 50 people):
- Ingestion: Unstructured.io for docs, a custom Python ETL for SQL databases, plain crawlers for internal wikis.
- Chunking: 512-1024 tokens with overlapping, plus parent-document retrieval (store small chunks, return full parent).
- Embeddings: Still text-embedding-3-large from OpenAI; tried a couple of open-source ones and the quality delta was real for our messy docs.
- Vector DB: Postgres + pgvector for almost everything. Only reach for a dedicated vector DB (Pinecone/Qdrant) when scale demands it.
- Retrieval: Hybrid search (BM25 + vectors) via pgvector, plus a reranker.
- LLM: Claude for long-context synthesis, GPT-4.1 for structured/tabular answers.
- Evals: Ragas + a small goldenset I run every week.
- Guardrails: PII redaction before data ever hits the embedder; per-doc permissions enforced at retrieval time.
What I'm still unhappy with:
- Table/spreadsheet retrieval. Still bad in most tools.
- Updating/changing content without full re-indexing.
- Telling the user "I don't know" confidently instead of guessing.
What's your stack? And what's the one piece you'd replace tomorrow if a better option existed?