Open Source Data Catalog Options for AI Infrastructure

There is a hard lesson embedded in the AI adoption numbers from 2025, and it has nothing to do with model quality. MIT Project NANDA tracked generative AI deployments across nearly a hundred organizations and found that 95% saw zero measurable return. The culprits were data readiness, workflow integration, and absent outcome definition. Meanwhile, Accenture research across two thousand companies found that only 7% had reached the data-readiness level required to scale advanced AI, and IBM's 2025 CEO Study put enterprise-scale AI initiatives at just 16% of the field. The pattern underneath all of these numbers is the same: organizations layer new AI tooling on top of existing data chaos and compound the fragmentation rather than resolve it.

What this means practically is that the selection criterion for a data catalog has shifted. The question is no longer whether the catalog helps human analysts find datasets. The question is whether it makes those datasets trustworthy and interpretable for agents operating at machine speed. That distinction drives every evaluation that follows.

What AI Agents Actually Need From a Data Catalog That Human Analysts Never Did

A human analyst can spot an obvious error. She can apply business context to a suspicious number, recognize that last quarter's revenue figure looks off because of an acquisition mid-period, and walk down the hall to ask a colleague what an ambiguous column name actually means. An AI agent cannot do any of these things. It returns a confidently wrong answer at machine speed, and the answer is syntactically valid SQL with a result that does not mean what anyone thinks it means.

Valid query and correct business answer are not the same thing. This is the central problem.

There are four requirements that separate an AI-ready catalog from what was adequate for the analyst era.

The first is semantic context: table descriptions, metric definitions, and business relationships embedded in the catalog as machine-readable objects. Not documentation that a human might eventually read. Objects an agent can query.

The second is governance enforcement at query time. Permissions must be scoped to the real end user's identity, not assumed from a service account with broad access across the estate. Sensitivity must be evaluated at the point where fields are combined in a live query, not only at the individual field during ingestion.

The third is lineage with intent. Column-level lineage from source through transformation through consumption, not just for compliance but so an agent can trace why a number looks the way it does before it reasons on top of it.

The fourth is audit logging at agent-level volume. Identity, intent, and lineage captured together per query. When thousands of agent calls are executing per minute, a log that captures only what SQL ran is functionally meaningless.

A 2024 IBM Institute for Business Value survey found that only 29% of technology leaders strongly agreed their enterprise data met the quality, accessibility, and security standards needed to scale generative AI. The 62% of organizations that cited lack of data governance as their primary AI inhibitor, per the 2025 Drexel and Precisely Data Integrity Trends report, are feeling the downstream consequence of that gap. Agentic AI escalates the stakes further: a single compromised agent reasoning across sensitive data and chaining tool calls is not a bad answer. It is a blast radius.

These four requirements become the scoring rubric applied to every catalog reviewed below.

How the Semantic Layer Became Non-Negotiable Infrastructure, and What That Means for Catalogs

Gartner elevated the semantic layer to essential infrastructure in its 2025 Hype Cycle for BI and Analytics. The market is projected to grow from $2.71 billion in 2025 to $7.73 billion by 2030, a 23.3% compound annual growth rate driven almost entirely by AI adoption pressure. The Futurum Group forecasts that growth rate doubling from 16% in 2026 to 30% by 2031 as autonomous agents move from experiment to production workloads.

What a semantic layer actually does is translate physical database schemas into business context. When gross margin is defined in the semantic layer, both a BI tool and an AI agent return the same result, consistently, regardless of how the underlying tables are physically structured. Without it, agents infer meaning from column names and schema structure. That process produces plausible-sounding answers. It does not produce reliable ones.

The open standards milestone worth noting: the Open Semantic Interchange specification finalized in early 2026, creating a vendor-neutral standard for sharing semantic definitions across tools and agents. dbt Labs' open-sourcing of MetricFlow moved in the same direction, pushing metric definitions out of proprietary configurations and into shared, accessible objects.

The implication for catalog evaluation is direct. A catalog that stores metadata but cannot expose semantic definitions as machine-readable, queryable objects is not AI-ready. It is a human-oriented index with an AI veneer.

Why the 2025 Consolidation Wave Makes Open Source Catalogs More Attractive, Not Less

The commercial catalog landscape consolidated sharply through 2025. Snowflake, Atlassian, ServiceNow, and Salesforce each acquired significant players in the space. The pattern after acquisition is consistent: roadmaps slow, support attention migrates to the acquirer's priorities, and customers of the acquired product find themselves waiting on a tool that no longer moves the way it used to.

Paid catalog pricing at scale runs from roughly thirty to two hundred dollars per user per month for major commercial options. Open source turns that into an infrastructure and platform engineering cost that scales differently and stays inside the organization's control.

Five reasons teams choose open source in this environment: cost control, no vendor lock-in, regulatory preference for self-hosting, customization depth, and a community iteration pace that can outrun any single vendor's roadmap. The consolidation argument cuts both ways, though. Open source projects can stall if maintainer interest wanes. Community health and release cadence matter as much as the license.

The real selection criterion entering 2026 is this: the catalog chosen now becomes the context layer that AI agents query at runtime. Locking into a product whose roadmap is now subordinate to an acquirer's strategic priorities is not just a procurement risk. It is a structural risk to the AI infrastructure built on top of it.

OpenMetadata: Strong Semantic Foundations and a Governance Model Built for Agents

OpenMetadata is one of the two most capable open source catalog options available, with a large and active community and a release cadence that has remained consistent. Its Unified Metadata Graph supports a substantial connector library, Elasticsearch-powered search, and an Activity Feeds interface that functions as a central home screen for data teams.

Version 1.8, released in June 2025, introduced data contracts: machine-readable schemas, SLAs, and quality guarantees that can be enforced automatically rather than simply documented. That is the most direct answer to the AI-readiness problem in any open source catalog released to date. It turns "this dataset is probably fine" into a machine-verifiable guarantee. An agent can check contract status before trusting a result.

On the four-requirement rubric: semantic context is strong. Table and column descriptions, glossary terms, and metric definitions are first-class catalog objects, not free-text annotations that exist for human readers. Governance is integrated at the catalog layer, with role-based access control, data masking, and policy enforcement. Lineage is end-to-end at the column level across sources, transformations, and BI consumption. The conversational interface and programmatic API give agents a surface to query catalog metadata at runtime.

The honest consideration: self-hosting at scale requires meaningful platform engineering investment. The managed cloud offering reduces that operational burden but reintroduces some of the cost dynamics that made commercial alternatives less attractive in the first place. Teams need to price that tradeoff honestly before committing.

DataHub: The Lineage and Metadata Graph Leader, With Governance Still Maturing

Originally built at LinkedIn and now one of the most widely deployed open source catalogs in production at large enterprises, DataHub's metadata graph architecture makes it exceptionally strong at entity relationships and impact analysis. Understanding how a change upstream propagates to downstream consumers is where it genuinely differentiates.

On the four-requirement rubric: lineage is the standout. Column-level lineage across a wide range of sources and transformation layers, with impact analysis at the entity level that few tools match. The programmatic API is well-documented and accessible enough for agents to call at runtime to retrieve metadata context. The contributor base is large and active, which reduces the stagnation risk that plagues smaller open source projects.

The gaps are real, though. Metric definitions and business logic are less natively structured than in OpenMetadata. How useful the semantic context actually is depends heavily on how diligently teams have populated descriptions, which is a human process with human variability. Governance enforcement is present but requires more configuration to reach query-time enforcement comparable to what purpose-built governance tooling provides.

Best fit: organizations that need deep lineage and impact analysis as the primary use case, with governance layered on from separate tooling. Less suited as a standalone semantic context layer for agents without additional investment.

Apache Atlas: Mature Governance Lineage, but an Architecture Designed Before Agents Existed

Apache Atlas is part of the Hadoop ecosystem and widely deployed in organizations that built their data infrastructure around Hive or HBase. Governance was a core design goal from the start, not something bolted on later. Classification, tagging, and lineage tracking across Hadoop-native sources are reliable and mature.

The problem is the architecture. Atlas was designed around human-driven classification workflows. The REST API exists, but it was not built for the query volume or the low-latency response that agents require at runtime. Semantic business context is not a first-class object in Atlas; the catalog stores technical metadata well and business meaning requires external tooling layered on top. Active development pace has slowed relative to OpenMetadata, and community contributions have contracted.

The honest assessment: Atlas is a viable governance and compliance record-keeping layer for organizations already deep in the Hadoop stack. It requires significant supplementation to serve as an AI agent's context source. Teams that are not already committed to the Hadoop ecosystem have better starting points.

Amundsen and Marquez: Lightweight Discovery Tools That Stop Short of Agent-Ready Governance

Amundsen, originally from Lyft, and Marquez, which grew out of the OpenLineage project, both prioritized fast, intuitive data discovery for human analysts. They solved that problem well. Amundsen's search-first UX, table popularity signals, and social metadata showing who uses what reduced time-to-find meaningfully for data teams. Marquez's OpenLineage-native architecture makes it a strong lineage capture tool for pipeline-centric teams running Airflow or Spark; job-level and dataset-level lineage is reliable.

Neither was designed for what the moment now requires.

Both tools share the same AI-readiness gaps: semantic definitions are not exposed as structured, queryable objects that agents can consume at runtime; governance enforcement is minimal, with no query-time permission evaluation against a real user identity; and audit logging is not designed for agent-level query volume.

Marquez's OpenLineage integration is genuinely valuable as a lineage input into a broader stack. It functions best as a component feeding a more capable catalog, not as a standalone AI infrastructure layer. Both tools are honest solutions to a 2019 to 2022 problem. Human data discovery at scale was the ceiling that mattered then. It is not the ceiling that matters now.

Where Every Open Source Catalog Hits the Same Ceiling: Semantic Enforcement at the Data Layer

Every catalog reviewed can store metadata. The divergence is in whether that metadata is enforced at query time or merely documented for human reference.

The shared ceiling is structural: open source catalogs are metadata repositories that sit alongside the data layer. They describe what data means. They do not intercept queries to enforce that meaning, or verify permissions at the moment an agent fires a query against live data. A catalog can mark a column as PII-sensitive. If the query layer does not enforce that classification at runtime under the actual requesting user's identity, the classification is advisory, not protective.

Accenture's research found that only 30% of what it categorized as "data reinventors" had achieved a unified logical data view across systems and ecosystems. Among the broader population, that figure dropped to 4%. The catalogs alone are not closing that gap.

Two specific failure modes worth naming. First, the freshness problem: a catalog reflecting last night's ingestion is accurate for a dashboard; an agent making a live decision on that metadata may be reasoning from a stale description of a schema that has already changed. Second, sensitivity at combination: open source catalogs generally evaluate field sensitivity during ingestion. They do not evaluate what happens when two fields are joined in a live query, which is where the majority of real privacy exposure actually occurs.

This ceiling is not a configuration problem. It reflects a design assumption that the catalog and the query layer are separate systems. That assumption was reasonable for human analyst workflows. For agentic workflows, it is a liability.

What AI-Ready Data Infrastructure Looks Like When the Catalog and the Query Layer Are Unified

The architectural move is to stop treating the catalog as a separate registry that describes data and the query engine as a separate system that retrieves it. Unify them. Expose semantic context, governance, and lineage through the same interface that answers the query.

In practice, this means four things working together rather than in sequence.

Semantic definitions, including table descriptions, metric calculations, and business relationships, are enforced at query time. They are not stored as documentation that a human might read and an agent might ignore.

Permissions are evaluated against the real end user's identity at the moment the query executes. Not inherited from a service account with broad access across the estate, which is how most current architectures actually work when agents are in the loop.

Audit logs capture identity, intent, and lineage together per query. At agent-level volume, a log that records only what SQL ran is noise. A log that records who asked, why the agent asked, and what data context was used to answer is infrastructure.

Sensitivity is evaluated at the combination point, when fields are joined in a live query, not only at the individual field during ingestion. That is where exposure actually happens.

This architecture does not require replacing existing warehouses or pipelines. It operates on top of current infrastructure as a query-time enforcement layer, which means it can be adopted incrementally without a rip-and-replace mandate. The catalogs reviewed in this piece represent meaningful progress on the metadata and discovery dimensions of this problem. The ones that will define AI-ready infrastructure in 2026 are the ones that close the gap between describing data and governing it at the moment an agent reaches for it.