Modern Data Stack Ecosystem Tools Overview

Ingestion tools pull data out of SaaS apps, operational databases, event streams, and APIs, then load it into a warehouse or lakehouse. No manual scripts, no custom connectors built in-house from scratch — that's the pitch, and for the most part, it holds up.

There's a fork early on: batch ingestion, which syncs on a schedule, versus streaming, which moves data event by event in near real time. Which one you need depends entirely on how fresh the downstream use case has to be. A monthly finance report can run on a batch sync from last night, but an agent deciding whether to flag a transaction as fraud cannot.

Fivetran and Airbyte dominate the batch and EL side, with broad coverage of SaaS sources out of the box. Stitch covers similar ground with a lighter footprint. For streaming, Kafka and Confluent are the standard, and Debezium handles change data capture, watching operational databases for row-level changes and streaming them out as they happen.

The connector count actually matters here, more than most buyers expect going in. A tool that covers 500 SaaS sources out of the box is worth more in practice than one that covers 50 and forces a team to write custom connectors for the gap. That gap is where engineering time quietly disappears.

Yet none of this makes data meaningful. Ingestion moves rows from point A to point B, but it doesn't tell anyone what those rows mean, how they relate to each other, or whether they can be trusted. That's the next layer's job, and skipping straight past it is where a lot of stacks start to go sideways.

Storage: the warehouse and lakehouse landscape in 2025

Cloud warehouses hold structured and semi-structured data at scale. Lakehouses do that too, but they also handle unstructured data and run machine learning workloads natively, without exporting data somewhere else first.

A survey of more than 2,000 data teams found Snowflake leading adoption at 41%, up 3 points year over year. Databricks came in at 32%, up 12 points, by far the largest jump in the market. BigQuery sat at 24%, up 2 points, while Redshift dropped to 18%, down 8 points and still falling. That Databricks number is the one worth sitting with; nothing else in the survey moved that fast in a single year.

Multi-warehouse setups aren't an edge case anymore, either. Thirty-eight percent of companies now run more than one warehouse at the same time. That's mainstream architecture, and it means the semantic and governance layers can't be built to serve one platform. They have to work across all of them, or they're not doing their job.

Snowflake and Databricks are converging on each other from opposite directions. Snowflake started in structured analytics and has been building out toward machine learning, while Databricks started in ML and has been building toward structured analytics and BI. Give it a few more product cycles and the line between them gets hard to find.

Open table formats are what make this convergence workable. Delta Lake, from Databricks, and Apache Iceberg, now supported broadly, including by Snowflake and BigQuery, both give teams time travel, schema evolution, and the ability to read the same table from different engines without copying it. That interoperability is what keeps a multi-warehouse world from turning into a data-duplication nightmare.

BigQuery's case is straightforward: managed, serverless, no infrastructure to babysit, and it pairs naturally with Vertex AI for teams already in the Google Cloud ecosystem. Redshift's decline says something too, because being tightly coupled to AWS used to be enough of a reason to pick a warehouse. It isn't anymore.

Still, none of this solves anything on its own. A warehouse full of raw tables with no shared definitions isn't an asset. It's a liability sitting there waiting for the next two layers to clean it up.

Transformation: turning raw loads into trusted, modeled data

The old ETL pattern transformed data before loading it, usually on a separate staging server that became its own bottleneck. Modern stacks flip that: load first, then transform inside the warehouse, where the compute already lives. That's the ELT shift, and it's why transformation tools look so different than they did a decade ago.

dbt is the dominant tool in this layer, by a wide margin. It's SQL-based, version-controlled, and it brought software engineering habits into data work that didn't exist before: modular models, automated tests, documentation that lives next to the code instead of in a wiki somewhere. dbt models have also become something more than transformation logic; they're often the first place business logic gets formally written down at all.

Other tools fill in around it. SQLMesh offers an alternative approach built around incremental processing and environment management, Spark-based pipelines still handle large-scale transformation jobs on lakehouses, and Dataform serves the same role natively inside Google Cloud.

Good transformation work produces a layered model: staging tables feeding intermediate models feeding marts, so that whoever queries the data, whether that's a dashboard, a data scientist, or an AI agent, hits clean, tested, clearly named entities instead of raw source tables with cryptic column names.

Even so, a well-named table isn't the same as a well-understood one. Naming a column revenue doesn't tell anyone which revenue definition it uses, how it should be joined to other tables, or who's allowed to see it. Transformation builds structure, not meaning, and conflating the two is where a lot of AI data problems actually start.

Testing deserves its own line here, because skipping it is where things quietly break. Untested transformation models are where silent data quality failures come from. What gets blamed on an AI agent hallucinating is very often just a bad upstream model that nobody tested before three other models built on top of it.

The semantic layer: where raw structure becomes machine-readable meaning

A semantic layer sits between transformed data and whoever's consuming it, and it defines the things a table alone can't: what a metric means, how dimensions relate, what the correct join paths are, what grain the data sits at, who's allowed to query what. One governed place for all of it, instead of five different definitions scattered across five dashboards.

This layer exists because SQL joins and column names don't explain themselves, especially not to a language model. Peaka, for instance, addresses this as a federated data platform that builds a semantic layer with governance directly into every query against connected sources. An LLM has no institutional memory, and it can't turn around and ask a colleague what "revenue" means in this specific company's context, so it guesses, based on whatever table and column names happen to be in front of it.

That guessing is where most AI analytics failures actually come from, and it's worth being precise about this: they're semantic failures, not hallucinations. The model picks the wrong table, joins two tables at the wrong grain, or aggregates a metric incorrectly. Analysis from Omni on AI BI deployments found that a governed semantic layer heads off the majority of these errors before a query even runs.

The category has matured fast enough that analysts are formally recognizing it. The 2025 GigaOm Radar classified the semantic layer as a mature category for the first time, moving it out of emerging territory. Gartner did something similar, elevating the semantic layer to essential infrastructure in its 2025 Hype Cycle for BI and Analytics.

On tooling: Looker's LookML gives Google's ecosystem a governed semantic model that plugs directly into Gemini, and internal Google testing found LookML cuts data errors in generative AI natural language queries by roughly two-thirds. AtScale takes a vendor-agnostic approach, working across Snowflake, BigQuery, Databricks, and others, and it was named a leader in the 2025 GigaOm Radar for semantic layers and metric stores. dbt's MetricFlow keeps semantic definitions inside the dbt project itself, traveling with the models they describe. And in 2025, dbt Labs, Snowflake, and Salesforce, among others, launched the Open Semantic Interchange, an effort to standardize semantic definitions in vendor-neutral YAML so a metric defined once doesn't need to be redefined for every tool that touches it.

The direction for 2026 points toward semantic-first AI agents: agents that reason directly against a governed model instead of depending on a human to write the query first. Under that model, the semantic layer stops being a nice-to-have for BI and becomes the prerequisite for letting an agent anywhere near structured data at all. Gartner's forward projection puts real numbers on this: organizations that build semantics into the reasoning layer could see accuracy improve by up to 80% and costs drop by 60% by 2027.

Governance: making policies enforced rather than documented

Governance used to mean a policy document somewhere in a shared drive that nobody read twice. That's over: the EU AI Act took effect in 2024, with enforcement starting in 2025 for prohibited practices and extending to high-risk AI systems in 2026. Governance is a legal obligation now, not a best practice a team gets to when there's time.

There's a real gap opening up here worth naming directly. Eighty-four percent of business leaders, per Salesforce, believe AI agents can make data more accessible, and seventy-nine percent of corporate strategists, per Gartner, consider AI and analytics critical to business success. That confidence is running well ahead of actual governance maturity at most organizations, and that gap is exactly where the risk sits.

In an AI-driven stack, governance tooling has to do a specific set of things well. Automated classification and PII detection matter because human tagging simply can't keep pace with the volume of queries agents generate. Lineage tracking matters because when an agent's output feeds a business decision, someone needs to trace it back: which source, which transformation, which model. Access policies need enforcement at query time, tied to the actual user's identity, not waved through under a broad service account. Sensitivity evaluation has to happen at the point two fields get joined together, because a join can reveal something neither field shows on its own. Audit logs need to capture identity, intent, and lineage as one connected record; at agent-scale query volume, a log without that structure is just noise.

On the vendor side, Collibra is a prominent enterprise governance platform with broad system coverage and compliance capabilities. Microsoft Purview serves Azure-native environments, while Atlan takes an active metadata approach. Informatica offers broad enterprise coverage, and Alation focuses on catalog-driven governance and data stewardship. For organizations whose data mostly lives in one place, the platforms themselves now cover a lot of this ground: platform-native governance tooling inside Snowflake, Unity Catalog inside Databricks.

That last point is worth being blunt about. If the large majority of an organization's data sits in one warehouse, platform-native governance now handles most of what used to require a standalone tool. Standalone platforms earn their keep once data spans multiple systems or multiple clouds, which, given that 38% of companies run multiple warehouses, is most enterprises by now. Teams building things incrementally and watching budget can also look at open-source options: Apache Atlas, OpenMetadata, Egeria, and Amundsen, though that route means lower cost but more operational work to keep them running.

One principle AI workloads expose more than anything else: an agent reading data and an agent writing data do not carry the same risk. Write actions need tighter governance than reads, full stop, because treating them the same is how a minor agent error turns into a data integrity problem.

Observability: knowing when the data that feeds your stack has broken

Data observability tracks five things: freshness, whether a pipeline actually ran; volume, whether the expected number of rows showed up; schema, whether a column changed shape or vanished; distribution, whether the values inside a column started drifting from their normal range; and lineage, which downstream tables and dashboards get hit when something upstream breaks.

As stacks get more modular, one broken upstream table doesn't stay contained — it ripples. Observability is how a team catches that break before a dashboard, or worse, an agent, uses the bad data and reports it with total confidence.

A 2025 survey of data teams found dbt tests combined with custom checks are still the dominant approach, used by 42% of teams. Great Expectations, the open-source option, comes in at 15%. Monte Carlo sits at 11%, Elementary at 8%, Datafold at 7%, and 17% of teams run no monitoring solution at all. That's not a rounding error; that's roughly one in six teams with zero visibility into whether their data is even correct.

Warehouse-native catalogs are becoming the anchor point for observability rather than a separate bolt-on system. Unity Catalog inside Databricks and Polaris inside Snowflake both emerged in 2025 as the source-of-truth layer that new observability setups get built around, with lineage tracked as a native feature instead of stitched in after the fact.

There's a direct line from this to AI quality, too. Analysis of AI engineering practice has made the point plainly: RAG quality is a data quality problem before it's ever an LLM problem. Running quality checks against whatever gets retrieved and fed to a model isn't optional, and observability tooling needs to reach into that AI data layer, not stop at the warehouse boundary.

That 17% with no monitoring at all matters more than it used to, because a stale or partial answer from an agent looks exactly as confident as a correct one. There's no visual cue, no hedge in the tone. When a human misses a data problem, it's a bad chart; when an agent acts on it, it's a decision made on broken information, and nobody finds out until later.

How the layers connect — and where stacks actually break down

Think of the stack as a chain, where each layer's output quality sets the ceiling for whatever comes next. Bad ingestion poisons transformation before it even starts. Undefined semantics make governance impossible to enforce, because there's nothing precise to enforce against, and missing observability means every failure above it propagates in silence until someone notices a wrong number three reports downstream.

The actual break points tend to cluster in a few predictable spots. Between ingestion and transformation, it's schema drift: a source system changes shape and nobody tells the data team until a model starts failing. Between transformation and the semantic layer, it's business logic defined informally inside a BI tool instead of in a governed model, so two dashboards answer the same question two different ways and nobody can say which one's right. Between the semantic layer and governance, it's permissions set at the table level in the warehouse but not at the metric level in the semantic layer, so an agent can query a derived metric it was never supposed to touch. Between governance and observability, it's lineage tracked for compliance paperwork but not built for incident response, so a team knows what happened after the fact but has no fast way to find what broke.

The multi-warehouse reality makes all of this harder. With 38% of organizations running more than one warehouse, governance and semantic tools that only understand one platform are already behind. That's not a future problem; it's a current one.

AI workloads act as an amplifier on top of all this. Agents query at a volume and pace that exposes seam failures a human would have caught by instinct before they mattered. The weakest interface in the stack becomes the highest-frequency point of failure, simply because agents hit it far more often and far faster than any analyst would.

There's an architectural mismatch underneath all of it, too. A pipeline built around stable schemas and scheduled batch refreshes doesn't fit an agent that queries at runtime and needs current data right now, not tomorrow morning. The stack has to support live query access alongside batch pipelines, not one or the other. A semantic layer with real governance behind it, sitting on top of a stack built for that kind of access, is what actually closes the gap between a system that reports information and one that agents can safely act on.