BTW, we need a DWH. Yesterday!

Data is the new bacon. Or is it the new oil? Maybe gold? Regardless, it seems to be valuable. This is why we need our own data warehouse (DWH). Now!

At upday we’ve been collecting anonymous user data since the beginning, in order to improve user experience — particularly content recommendations. For example, we categorise articles and track the topics that users read about, then with some fancy algorithms we try to predict and satisfy our users but also strive to surprise them with articles they could be interested in (we call it serendipity).

Another use case for collected data is to understand how our users behave within the app. You can do field research, engage test audiences and ask friends, but in reality the user always acts differently. And very often the differences are surprising.

Since we originate from a media company, we have a strong motivation to support journalism. This is why we also need to know which articles have been read and how often, in order to share revenue with the original content publishers. Unsurprisingly, the publishers we cooperate with have a similar opinion.

Given all these different use cases, it was clearly time for a big data solution. We have a lot of strong software engineers, but none of them are specialists for this kind of technology. A few of us ventured to dip our toes into this topic. So we formed a small data platform team and started researching and experimenting the current state-of-art in this domain.

In a series of articles we would like to share our experiences with you and invite you on our journey towards a functional custom data warehouse solution. See us fail, despair, hope, fail again and maybe finally succeed in our undertaking.

Let our journey begin…

A Journey Towards a Custom Data Warehouse Solution Part 1 : Getting Started