Structural Metadata vs Semantic Metadata for AI Consumers

Semantic metadata answers the questions structural metadata never even attempts: what does this data represent, who owns it, what business process depends on it, and how should it actually get calculated. Some people call it business metadata. Same idea.

Take a column called custltv12m. Structurally, that's DECIMAL(10,2), and that's all a system knows about it. Describe it semantically instead: "Customer Lifetime Value, 12 Month. Predicted revenue from a customer over the next twelve months, calculated using historical purchase patterns and engagement scores, governed by finance policy FP-2024-03, reported in USD at the transaction level." Same column. One version is usable, the other is a guess waiting to happen.

I think of semantic metadata as a contract: a label just names a thing, while a contract tells you what it means, how to use it, and what rules bind it. Ask an agent for "revenue" with no semantic layer behind it and you've handed it a coin flip with four sides: gross or net, USD or local currency, transaction-level or rolled up. One word, four defensible answers, and the agent has no way to know which one you wanted.

Data lake research has been circling this for years now, treating field names and plain explanations and ownership info as functionally distinct from the technical plumbing underneath. Without that semantic layer, structured data is opaque to an AI system in a very particular way. It can see the columns. It can see the values sitting in them. It just can't tell you whether a calculation is right, only whether the SQL happens to parse.

Worth being precise here: this has nothing to do with vector embeddings or unstructured retrieval. Semantic metadata means structured, queryable meaning attached to structured data, something you look up rather than approximate through a similarity score.

How AI agents actually consume data, and why they cannot fill in missing meaning

Human analysts patch over thin metadata all the time, and most don't even notice they're doing it. They pull up the wiki. They ping the engineer who built the table. They remember, half-consciously, a Slack thread from six months ago explaining why one column behaves strangely.

An agent gets none of that. It can't tap a colleague on the shoulder, and it usually can't tell when its own guess is wrong, which is the part that actually matters.

So it guesses, and each guess stacks on the last one. Ask an agent what regional revenue looked like last quarter with no semantic layer underneath it, and it has to guess what "regional" means, guess which table holds revenue, guess how that revenue gets calculated. What comes back is syntactically clean SQL and a confident-looking number. Nothing in the output tells you the number might be wrong.

A 2025 medical study makes this concrete. Researchers tested a RAG chatbot two ways: restricted to curated, high-quality reference material, and running as baseline GPT-4 with no such grounding. Restricted, hallucinations dropped to near zero. Unrestricted, baseline GPT-4 fabricated answers for 52% of questions that fell outside its reference set. Same model both times. The only thing that changed was whether the data carried enough context to ground an answer.

That's the actual mechanism behind confident-but-wrong. It's a silent failure: no error thrown, just a result that looks exactly like a correct one would look.

Agentic systems make this worse. Agents chain tool calls together, one step feeding the next, and a bad inference early in that chain doesn't get flagged. It gets used instead. The error propagates through every step after it, and by the time a human looks at the output, the damage is baked in five layers deep.

Why inheriting human-ready data infrastructure reproduces this failure at scale

Most enterprise data was built for one audience: human analysts. Formats consistent enough for a person to read, values accurate enough for someone to sanity-check by eye, documentation sitting in a wiki that a new hire can page through in an afternoon.

Clean for humans and interpretable by AI are different standards, and treating them as interchangeable is where most of this goes wrong. The first assumes a reader who already carries institutional memory into the room. The second gets none of that.

The scale here isn't small. RAND Corporation research puts the failure rate for AI projects reaching real production above 80%, roughly double the failure rate of ordinary IT projects. S&P Global surveyed over 1,000 enterprises in 2025 and found 42% had abandoned most of their AI initiatives that year, up from 17% the year before. Separately, the average organization scrapped 46% of its AI proof-of-concepts before anything shipped.

What's actually causing it isn't much of a mystery once you dig in. Informatica's CDO Insights 2025 survey ranked data quality and readiness as the top obstacle to AI success at 43%, ahead of technical maturity, ahead of skills gaps. A 2026 report from Cloudera and Harvard Business Review Analytic Services found that only 7% of enterprises consider their data completely ready for AI.

Seven percent.

Gartner projects 60% of AI projects will be abandoned by 2026, for the same underlying reason: nobody made the data AI-ready before pointing a model at it. Demos work when someone controls the environment by hand and crafts the prompts carefully. Production fails when live data carries no shared meaning, and nobody's crafting anything at three in the morning when an agent fires off its two-thousandth query of the day.

IBM's research found 82% of enterprises deal with workflow disruptions traced straight back to siloed data. Silos are a structural headache, and the deeper issue is that silos, almost by definition, share no contract with each other. Nobody agreed on what "customer" means across those two systems, so nothing downstream can trust either one.

What a semantic layer does structurally, and how it solves the meaning problem

A semantic layer sits between raw data and whatever's consuming it, human or otherwise. It defines how tables relate, standardizes what a metric actually means across the company, bakes business rules directly into the layer itself, and hands an agent a pre-validated business entity instead of a raw schema it has to decode on its own.

The agent ends up reasoning about customer, revenue, churn, rather than reverse-engineering column names and data types every single time it runs a query.

In practice that means several things working together. Plain-language descriptions on tables and columns. Metric definitions specifying calculation method, currency, time grain, exclusions. Entity relationships mapping how customers connect to transactions and transactions connect to regions. Business rules enforced as constraints the query layer actually checks, rather than policy language sitting unread in Confluence. Provenance, so you can trace a value back through every transformation it passed through and know exactly when it landed.

Joe DosSantos, VP of Enterprise Data and Analytics at Workday, put it plainly at Atlan's Re:Govern keynote. Workday's own financial reporting agent couldn't answer a single question it was asked. "We started to realize that we were missing this translation layer," he said. A named executive, in a real production environment, running straight into the structural metadata ceiling.

The market has caught up. Gartner's 2025 Hype Cycle for BI and Analytics moved the semantic layer into essential infrastructure territory. BigDATAwire reporting has roughly 40% of enterprise leaders naming the absence of semantic context as a major blocker to getting operational AI working at all. And industry efforts toward a shared, structured language for semantic metadata reflect a broader push to stop every vendor from inventing its own private dialect.

The distinction between a BI semantic layer and one built for AI agents

BI-oriented semantic layers, think dbt MetricFlow, AtScale, Looker's LookML, mostly solve query translation. They turn a business question into consistent SQL that a dashboard can run against reliably every time someone refreshes it.

That's a real problem, and those tools solve it well. But it's a narrower problem than it looks: consistent metric definitions for human-driven BI, inside one specific tool's walls.

What they typically don't do matters more the longer you sit with it. They don't unify meaning across data sources that speak different dialects before a query even gets written. Nor do they encode the ontologies and entity relationships an agent needs to chain several reasoning steps together. Permissions rarely get scoped to the actual person asking, at the moment they ask, rather than inherited from a shared service account nobody's audited in a year. Identity, intent, and lineage don't get logged together at the volume agents generate. And sensitivity rarely gets evaluated at the point two fields get joined, only at the level of one column sitting by itself.

BI semantic layers live inside specific tools. An AI semantic layer has to sit in the middle of everything, so every agent, every app, every analyst pulls from the same set of definitions. Underneath all of it is the harder problem: connecting data across sources that were never built to agree with each other, before any query gets written. That's ontology engineering, a genuinely different discipline from building a metric store. Semantic fragmentation across teams consumes a significant share of data engineering time, and it's the reason most AI programs stall out right after the pilot goes well.

What governance looks like when it is built into the semantic layer rather than applied afterward

The old governance playbook applies access controls at the warehouse or pipeline level, writes the rules down in a policy doc, and audits after the fact. That playbook assumes a human user, authenticated once, trusted from that point forward to interpret their own scope correctly. Agents break both halves of that assumption at once, and nobody's rewritten the playbook yet.

Permissions have to get checked at query time, scoped to whoever or whatever is actually asking, rather than inherited wholesale from one broad service account every agent in the building shares. Sensitivity has to get evaluated at the moment data actually gets combined. Two columns, each harmless alone, can produce something sensitive the instant they're joined, and structural metadata has no way to catch that. Semantic metadata, carrying the relationship context, does.

Audit logs need identity, intent, and lineage together, or they're close to worthless. "Service account X ran query Y" tells you nothing useful once agents are firing off thousands of queries an hour and something's already gone wrong. Write actions need even stricter handling than reads: an agent that misreads something gives you a bad answer, but an agent that writes a wrong inference back into a live system causes actual damage.

There's a case for a live query layer over static pipelines here too. Pipelines assume schemas and sources hold still. Agents don't get that luxury; they act on data that's changing while they're acting on it, so freshness and permissions need checking live, rather than pulled from whatever a pipeline happened to cache last night. A 2025 survey of over 1,000 senior leaders found 98% had hit AI-related data quality problems, and only 46% felt confident their data actually met their AI goals. Governance bolted on at the model or prompt layer hasn't closed that gap, and closing it will take more than another patch at the prompt level.

What building an AI-ready semantic layer actually requires in an existing enterprise

The most common mistake I see is treating semantic readiness like a cleanup task, something you bolt onto pipelines that already exist and call it done. It has to be designed for AI consumption from the start, not patched in after the fact.

No rip-and-replace required, which is the good news most people don't expect. The semantic layer sits on top of the warehouses, SaaS tools, and operational systems already running, and exposes one unified interface with meaning and governance built in rather than stacked on top later.

Research has found that organizations seeing real returns from AI were far more likely to have redesigned their data workflows before ever touching model selection. Sequence matters: the programs that actually worked put 50 to 70% of their project timeline and budget into data readiness, extraction, normalization, governance metadata, quality gates, retention controls, rather than into fine-tuning a model.

What an AI-ready semantic layer actually needs to have: shared metric definitions and a business glossary encoded as queryable contracts, not wiki pages nobody opens. Entity relationships and ontological context an agent can traverse instead of a schema it has to guess at. Permission enforcement checked at query time against real user identity. Lineage and audit logging fine enough to match how agents actually operate, at their volume, at their speed. Live access to current data instead of a snapshot that's already stale by the time an agent acts on it.

Gartner's 2025 report projects that by 2028, 15% of daily work decisions will get made autonomously by agentic AI. The semantic layer is what makes those decisions trustworthy as well as fast, and that distinction is going to matter more every year. Platforms like Peaka sit on top of existing infrastructure to provide exactly this: one query interface carrying semantic context, metric definitions, table descriptions, and query-time governance, without forcing anyone into a warehouse migration to get there.

Most enterprises aren't short on data. They're short on data that carries the meaning an agent actually needs to use it correctly, and right now, that describes most companies.