What Structured Questions Actually Need That Vector Search Can't Give Them

A finance analyst asks an AI assistant how many customers in the Northeast exceeded their quarterly quota. The system answers with confidence, cites a few documents, and gets it wrong. That failure is the predictable result of pointing a similarity engine at a counting problem.

Most enterprise AI stacks were built around vector search and retrieval-augmented generation, tools designed to find documents that mean roughly the same thing as a question. But the business questions companies actually need answered, revenue by region, headcount trends, metric comparisons, belong to a completely different category. They are about computing exact numbers across related tables.

The mix-up makes sense once you see how it happened. Both tasks take natural language as input. Both show up in the same product demos, often in the same chat window. Vector databases are mature, well-funded, and heavily marketed, so when a company builds an AI interface, the vector search pipeline becomes the default tool reached for, whether the question in front of it is "summarize this contract" or "what was revenue in a given region last quarter. The two problems look similar from the outside. They are not solved the same way, and the rest of this piece explains why.

How vector search works

Vector search fails structured business questions for a mechanical reason: the system has no arithmetic in it anywhere. It is not that today's implementations are young or under-tuned. The approach itself was never built to count, sum, or compare.

Here is what actually happens under the hood. An embedding model, something like BERT, turns a chunk of text into a fixed-length list of numbers, a vector. A search query gets converted into its own vector. The system then measures the geometric distance between the query vector and every stored vector, and returns whichever ones sit closest. That is the entire operation: find neighbors in a high-dimensional space. Nothing in that process computes an answer. It retrieves documents that resemble the question.

The problem gets worse on the generation side, where a large language model takes those retrieved documents and writes a response. LLMs don't handle digits as numbers. They break them into subword tokens chosen by statistical likelihood, which destroys the positional structure that arithmetic depends on. So when a model produces something that looks like a calculation, it is predicting a plausible sequence of tokens, not running a sum. The output can look exactly like a correct answer and still be invented.

Dense vector embeddings also carry no concept of operators like "greater than" or "between these dates." A query embedding for "sales greater than a certain threshold" does not encode that threshold as a direction the system can evaluate. The vector just sits somewhere in space, and distance metrics can't tell you whether a number clears a bar.

Relational structure disappears too. Vector search returns similar items from one undivided space. Its geometry represents similarity, not table relationships, so it has no built-in way to follow a foreign key from one table to another. Joins require knowing that this customer ID in one table matches that customer ID in another. A similarity score carries none of that information.

Where vector search fails on structured business questions

They are the most common query types a business user submits, and vector search misses nearly all of them.

Take the aggregation example again: how many customers in the Northeast exceeded their quarterly quota. That's a structured query over metadata and stored values, and it needs a count. Vector search returns documents that talk about customers, regions, and quotas in similar language. It does not return a number, and the similarity score attached to each result tells you how related the text is, not whether it actually contains the answer the user needs.

Multi-hop questions expose the gap even more clearly. Find all contracts signed by subsidiaries of accounts flagged for review. Answering that means walking a chain: account to subsidiary to contract. That is join logic, full stop, and no amount of embedding quality turns a chain of relationships into a nearest-neighbor lookup. Individual documents retrieved along the way might each be relevant on their own, yet the one piece of information that actually connects them can go missing.

The functions that feel this hardest are finance, legal, compliance, and operations, the teams that need numbers they can act on directly, without running a separate check behind the AI's work. A single wrong figure in that setting can fail a reconciliation, trigger a compliance exception, or send a budget decision in the wrong direction.

What makes this worse is how the errors present themselves. A high similarity score reads as confidence. Users come away learning that the system sounds sure of itself, not that it is right, and that is close to the worst possible signal to train people on when the stakes involve real money or real risk.

What structured questions require

A structured business question carries three requirements, and none of them have anything to do with semantic similarity.

The first is exact computation. A revenue figure or a headcount number cannot be approximately correct. It has to be arithmetically correct, and the system producing it needs to show how it got there, not just state a number and move on.

The second is relational logic. A headcount trend means joining an HR system to a time dimension. A revenue-by-region figure means joining a transactions table to a geography hierarchy. The system has to walk those table relationships on purpose, following defined keys, not guess at them based on which records happen to look alike.

The third is governed schema context, and this one trips up systems that otherwise look like they're working fine. A column named "ARR" in one table might include professional services revenue. The same label in a different table might not. Without something that resolves which definition applies, a query can be written in flawless SQL and still answer the wrong business question. Technically correct, practically useless.

Put together, these three requirements describe something no amount of better embeddings or longer context windows can deliver on their own. They call for a layer of the stack that knows what the data means, how it connects, and how to compute with it exactly. That's the subject of the next section.

The semantic layer's missing infrastructure for AI-ready structured queries

A semantic layer placed between raw data and the AI querying it is the one piece of infrastructure built to satisfy all three requirements at once. It is the connective tissue that makes a structured answer trustworthy in the first place, not overhead bolted onto a pipeline for compliance reasons.

Start with metric definitions. A semantic layer exposes certified definitions, ARR, headcount, churn rate, as structured context that a language model can query directly. When a finance leader asks what Q3 ARR was, the answer resolves against the same definition used in the board deck, not whatever a raw column happens to contain in whichever table the model found first.

The same logic applies to joins. A semantic layer encodes which keys connect which tables, which hierarchies apply, and which time dimension counts as canonical. That means the SQL generated in response to a question is structurally sound before it ever runs, rather than being a best guess about which tables seem related.

Governed access works the same way. Permissions get enforced at the moment of the query, under the actual user's identity, so the answer only includes what that person is cleared to see. It isn't inherited from a broad service account that happens to have access to everything.

The gap between where companies think they stand and where they actually stand on this is large. The overwhelming majority of organizations say a solid data foundation is essential for AI to work well. Only about half believe they actually have one, and the semantic layer tends to be the most overlooked piece of that foundation, often treated as a nice-to-have.

Skipping this layer lets AI agents bypass governance. They write SQL straight against raw schemas, and that SQL can run without error while still answering the wrong question, with no record anywhere that lets anyone catch or correct the mismatch after the fact.

Governance gaps, not model limitations, stall AI initiatives on structured data

They are a surface symptom of a governance problem that exists no matter which retrieval method a company picks. AI doesn't fix weak governance. It amplifies it.

A survey by Drexel University and Precisely, cited by KPMG, found that a majority of organizations name a lack of data governance as the main obstacle holding back their AI initiatives, ahead of model quality, compute cost, or talent shortages.

AI models are genuinely good at spotting patterns across large amounts of data, but they only work with what they're given. Inconsistent metric definitions, unclear lineage, and siloed schemas don't disappear just because an AI system is now in the loop. They get amplified, because the model treats whatever it receives as ground truth and builds on top of it without question.

The mechanism is straightforward enough to picture. A company running Tableau, Power BI, and a few analyst notebooks often finds that revenue is calculated slightly differently in each one. Once AI agents start writing SQL against raw schemas, they inherit those inconsistencies and reproduce them at a speed and scale no team of human analysts could match.

Fixing this takes governance that runs continuously, not governance that happens on a quarterly audit cycle. Classification, lineage tracking, quality checks, and policy enforcement need to live inside the data pipelines themselves. Manual review, done periodically, simply cannot keep pace with how fast an AI agent can generate and run new queries.

The clock on this is not abstract. Gartner projects that a substantial share of agentic AI projects will be canceled by 2027, due to costs that keep climbing, unclear business value, or risk controls that were never built to handle this much automated querying. That's a forward deadline for the organizations that haven't closed the gap yet, not a historical note about ones that already failed.

How hybrid architectures bridge structured and unstructured retrieval

The strongest challenge to everything argued so far is a fair one: relational databases are actively adding vector capabilities, and that is narrowing the line between the two retrieval approaches.

SQL Server 2025 now includes a native vector data type, supporting cosine, Euclidean, and dot-product distance metrics, with the DiskANN algorithm handling graph-based index traversal for fast lookups. But the VECTOR type itself explicitly does not support mathematical operations, constraints, or general-purpose data manipulation. It stores and compares vectors. It does not compute with them.

Academic work is formalizing this convergence too. Research on Vector-augmented Analytical Queries, accepted to ICDE 2026, combines vector similarity search with relational operators, filters, aggregates, and joins across multiple tables in a single query. The Exqutor optimizer built for this work shows real performance gains when cardinality estimation is applied correctly to these hybrid workloads, letting the system better predict how much work a query actually involves before running it.

This kind of hybrid setup solves real problems. Finding supplier parts that look visually similar to a product image, while filtering by date range and joining to a pricing table, is a case where combining vector and relational querying delivers something neither approach gets close to alone.

But none of this dissolves the original issue. Embedding vector search inside a relational engine still depends on the structured half of the query, the joins, the aggregations, the metric definitions, being governed and correctly defined. Running a vector search join against a raw, ungoverned schema column just produces an approximate similarity match attached to an answer nobody has verified.

The technology is also younger in practice than the research papers suggest. As of April 2026, there is no official large-scale vector dataset or dedicated benchmarking tool built for SQL Server 2025's vector features. Most production testing relies on public datasets and custom scripts assembled by whoever is running the benchmark. The convergence between vector and relational querying is architecturally real. At enterprise scale, it is still operationally young.

How trusted data access and data products change AI agent querying

Governance and semantic context only scale across a company when data is published as a governed product, not handed out as raw infrastructure that every team has to interpret on its own.

The model works by connecting supply and demand directly. A data provider publishes a dataset once, complete with its definitions, lineage, and quality checks, and that same dataset gets reused across finance, operations, and AI agents without anyone having to redo the governance work. Analysts and automated systems both get real-time access to data that already carries its own documentation, and none of it requires opening up uncontrolled access to get there.

In June 2026, Ataccama announced a native integration with the Databricks Marketplace built around this idea. Its MCP Trust Layer gives AI agents operating inside Databricks a combined semantic and quality layer to work from, business glossary terms, catalog metadata, and quality scores that explain what a dataset means and where it came from, plus live quality scores, monitoring results, and anomaly signals that let an agent judge whether a dataset is trustworthy before it acts on it.

Snowflake Marketplace shows the scale this model has reached: hundreds of data providers, offering thousands of live, AI-ready data products, agents, and integrated SaaS tools, all available to connect to directly. That volume only works because the governance travels with the product instead of needing to be reconstructed by each company that uses it.

Alation's research on this marketplace approach backs up what that savings actually looks like: 90% faster implementation of new use cases, along with a meaningfully lower total cost of ownership and stronger trust and compliance outcomes. Governance gets built once, at the point a data product is published, instead of being reinvented by every team and every AI agent that later tries to query it.