When writing about satellite loading, we need also to talk about hub and link loading. They happen at the same time. Before Data Vault 2.0 a hub or link surrogate key was an integer. Nowadays modern data vaulters would use hash keys instead to allow parallel loading. Others are going back to the roots and […]

Finally, we reach our goal of talking about Data Vault Objects. This is the new area which I’m working and learning from and probably you might be interested as well. Data Vault consists of mainly 3 objects: Hub Link Satellite I’m not going to discuss what Data Vault is. There are lots of resources in […]

Creating a history layer (or Persistent Staging Area from Roeland Vos) is a safeguarding layer. All data arrives unmodified from the source. People in the Data Vault Space would probably suggest creating a Source Vault if I don’t want to build a proper Raw Vault with good Business Keys. I considered that option too. I […]

What is the fastest way to ingest data? To persist or to stream? A view comes close to streaming data. Persisting is always kind of stop-and-go in my view. I wanted to wait until the end of my little template story. But I couldn’t resist to talk about it now. I’m too excited about it! […]

Usually for a start, somebody starts with an easy template. The stage table is the most difficult one. The first running version was done in a few days. But perfecting it and adding more features has taken us months! The stage table is used everywhere. So, I’m starting with it. Where do we use a […]

WhereScape consists of 2 applications: WhereScape 3D WhereScape RED WhereScape 3D is a newer application and is the way to go for the future. With 3D I can dig into my source system, profile it, document it and design my future design and at the end it spills out any objects (including any needed intermediate […]

How to build a data warehouse? I used to work with SSIS on Microsoft SQL Server to generate all my pipelines. This was fast and convenient and a lot of people in the industry are working like that. Define a source, do some transformations and load them to a target table. Very easy, very fast. […]

In my previous post I documented roughly a data warehouse and some data modeling styles. In this post I want to discuss, how I came to the conclusion how I want to ingest my data and serve them to my user. So what we know for sure is, that we have some sources and our […]