RAG Is the Wrong Tool for Your Structured Data

There's a reflex right now where every "let the AI use our data" project starts with the same move: chunk it, embed it, stuff it in a vector database, retrieve the top-k by similarity. RAG. It's the default, and for a lot of problems it's the right default.

For your structured data, it's usually the wrong one, and I keep watching teams spend a quarter learning that the expensive way.

What RAG is actually good at

RAG earns its keep on unstructured text. You have ten thousand support docs, a pile of PDFs, a knowledge base nobody can navigate. The question is fuzzy ("how do I reset a customer's 2FA?") and the answer lives in prose somewhere. Semantic search finds the relevant chunk, the model reads it, you get an answer. That's a genuinely good use of the technology, and I'm not knocking it.

The key property: there's no precise, computable answer. "How do I reset 2FA" doesn't have a number. It has a passage.

Why it falls apart on your database

Now ask a different question. "How much revenue did we do last month?"

That has exactly one correct answer, and it's a computation, not a passage. The right way to get it is select sum(amount) from orders where .... It's deterministic. Run it twice, get the same number. The database was built to answer this kind of question, and it does it perfectly.

Watch what RAG does to it. You'd have to embed your rows, retrieve the ones that seem similar to the question, and hand a sample to the model to reason over. But "last month's revenue" isn't a similarity problem. It's an aggregation over every relevant row, and retrieval gives you a top-k sample, not the full set. The model ends up summing the twenty rows it happened to retrieve and confidently reporting a number that's wrong by orders of magnitude.

You took a problem with a precise, cheap, correct answer and replaced it with a fuzzy, expensive, approximate one. That's the whole bug.

The tell

Here's the quick test for which tool you want. Ask: does this question have a computable answer, or a findable one?

Computable answers (counts, sums, averages, "which users did X," "how many of Y") want SQL. The database already knows how to compute them exactly. Give the model the schema and let it write the query.

Findable answers ("what's our refund policy," "summarize the complaints about onboarding") want retrieval. There's no formula. You're looking for the right text.

Most "AI on our data" projects are a mix, and the mistake is treating the whole thing as retrieval because RAG was the first tool everyone learned. Your Postgres tables are not a findability problem. They're a computability problem, and they come with a query engine that's been optimized for forty years.

"But I want natural language over my data"

Right, and you can have it. The natural-language part and the RAG part are not the same thing. You can let someone ask "how much revenue last month" in plain English and have a model translate that to SQL, run it, and return the exact number. That's natural language over structured data done correctly. The model writes the query; the database computes the answer. Nobody embedded anything.

That's the entire premise of QueryBear, and to be clear, I'd say the same thing if I didn't build it. The interface can be conversational. The retrieval mechanism should be a query, because your data is structured and a query is exact.

Where it's genuinely both

To be fair, there's a real middle. If your "structured" data has a big freeform text column — support ticket bodies, product reviews, notes — then questions about the content of that text are a retrieval problem, and questions about the metadata around it (how many, from whom, when) are a query problem. The good systems route each part to the right engine instead of forcing everything through embeddings.

The point isn't "RAG bad." RAG is great at the thing it's for. The point is that your database is not a pile of documents, and the moment you start embedding rows to answer questions that GROUP BY would answer exactly, you've talked yourself out of the right tool because the wrong one was trendy.