Knowledge lakes and semantic layers have been round for a very long time – every residing in their very own walled gardens, tightly coupled to pretty slim use instances. As knowledge and analytics infrastructure migrates to the cloud, many are difficult how these foundational expertise elements match within the fashionable knowledge and analytics stack. On this article, we’ll dive into how a knowledge lakehouse and a semantic layer collectively upend the standard relationship between knowledge lakes and analytics infrastructure. We’ll learn the way a semantic lakehouse can dramatically simplify cloud knowledge architectures, get rid of pointless knowledge motion, and cut back time to worth and cloud prices.

The Conventional Knowledge and Analytics Structure

In 2006, Amazon launched Amazon Internet Companies (AWS) as a brand new approach to offload the on-premise knowledge middle to the cloud. A core AWS service was its file knowledge retailer and with that, the primary cloud knowledge lake, Amazon S3, was born. Different cloud distributors would introduce their very own variations of cloud knowledge lake infrastructure thereafter.

For many of its life, the cloud knowledge lake has been relegated to taking part in the position of dumb, low cost knowledge storage – a staging space for uncooked knowledge, till knowledge could possibly be processed into one thing helpful. For analytics, the information lake served as a holding pen for knowledge till it could possibly be copied and loaded into an optimized analytics platform, sometimes a relational cloud knowledge warehouse feeding both OLAP cubes, proprietary enterprise intelligence (BI) software knowledge extracts like Tableau Hyper or Energy BI Premium, or all the above. On account of this processing sample, knowledge wanted to be saved a minimum of twice, as soon as in its uncooked kind and as soon as in its “analytics optimized” kind. 

Not surprisingly, most conventional cloud analytics architectures appear like the diagram under:

As you’ll be able to see, the “analytics warehouse” is liable for a majority of the features that ship analytics to customers. The issue with this structure is as follows:

1. Knowledge is saved twice, which will increase prices and creates operational complexity.
2. Knowledge within the analytics warehouse is a snapshot, which implies knowledge is immediately stale.
3. Knowledge within the analytics warehouse is usually a subset of the information within the knowledge lake, which limits the questions customers can ask.
4. The analytics warehouse scales individually and in another way from the cloud knowledge platform, introducing extra prices, safety issues and operational complexity.

Given these drawbacks, you would possibly ask “Why would cloud knowledge architects select this design sample?” The reply lies within the calls for of the analytics customers. Whereas the information lake may theoretically serve analytical queries on to customers, in observe, the information lake is just too gradual and incompatible with standard analytics instruments. 

If solely the information lake may ship the advantages of an analytics warehouse and we may keep away from storing knowledge twice!

The Start of the Knowledge Lakehouse

The time period “Lakehouse” noticed its debut in 2020 with the seminal Databricks white paper “What’s a Lakehouse?” by Ben Lorica, Michael Armbrust, Reynold Xin, Matei Zaharia, and Ali Ghodsi. The authors launched the concept that the information lake may function an engine for delivering analytics, not only a static file retailer.

Knowledge lakehouse distributors delivered on their imaginative and prescient by introducing excessive pace, scalable question engines that work on uncooked knowledge recordsdata within the knowledge lake and expose an ANSI customary SQL interface. With this key innovation, proponents of this structure argue that knowledge lakes can behave like an analytics warehouse, with out the necessity for duplicating knowledge.

Nevertheless, it seems that the analytics warehouse performs different important features that aren’t happy by the information lakehouse structure alone, together with:

1. Delivering “pace of thought” queries (queries in beneath 2 seconds) persistently over a variety of queries.
2. Presenting a business-friendly semantic layer that permits customers to ask questions without having to write down SQL.
3. Making use of knowledge governance and safety at question time.

So, for a knowledge lakehouse to actually change the analytics warehouse, we want one thing else.

The Position of the Semantic Layer

I’ve written so much in regards to the position of the semantic layer within the fashionable knowledge stack. To summarize, a semantic layer is a logical view of enterprise knowledge that leverages knowledge virtualization expertise to translate bodily knowledge into business-friendly knowledge at question time. 

By including a semantic layer platform on high of a knowledge lakehouse, we are able to get rid of the analytics warehouse features altogether as a result of the semantic layer platform:

1. Delivers “pace of thought queries” on the information lakehouse utilizing knowledge virtualization and automatic question efficiency tuning.
2. Delivers a business-friendly semantic layer that replaces the proprietary semantic  views which might be embedded inside every BI software and permits enterprise customers to ask questions without having to write down SQL queries.
3. Delivers knowledge governance and safety at question time.

A semantic layer platform delivers the lacking items that the information lakehouse is lacking. By combining a semantic layer with a knowledge lakehouse, organizations can:

1. Remove knowledge copies and simplify knowledge pipelines.
2. Consolidate knowledge governance and safety.
3. Ship a “single supply of reality” for enterprise metrics.
4. Cut back operational complexity by preserving the information within the knowledge lake.
5. Present entry to extra knowledge and extra well timed knowledge to analytics customers.

The Semantic Lakehouse: Everyone Wins

Everyone wins with this structure. Shoppers get entry to extra fine-grained knowledge with out latency. IT and knowledge engineering groups have much less knowledge to maneuver and remodel. Finance spends much less cash on cloud infrastructure prices. 

As you’ll be able to see, by combining a semantic layer with a knowledge lakehouse, organizations can simplify their knowledge and analytics operations, and ship extra knowledge, quicker, to extra customers, with much less value.