The Technical Specs
The main objective of Funda was to create a single Kubernetes-based data runtime, where jobs are scheduled, and data is transformed. So that everyone could use the data for his or her own purpose. This data infrastructure needed to be set up in Microsoft Azure. The team migrated Funda’s use cases from the Hadoop platform and old BI server to Google Cloud Platform and integrated best practices around dbt (data build tool) and data quality. Data Engineers and Data Scientists can now create new pipelines without working on underlying cloud infrastructure. Data is accessible via an SQL interface, fulfilling the “self-service” criteria. Teams can access various data marts depending on their role, and creating new use cases is easy and quick.
Guillermo, Analytics Engineer at Xebia Data and AI, specifies: “Data is now both replicated from Funda’s systems and micro-batched from different event streams into BigQuery. However, other surrounding services were deployed in Microsoft Azure, where most of Funda’s applications reside. In Azure, the team setup Airflow on Azure Kubernetes Service (AKS) to orchestrate all of Funda’s data pipelines. These pipelines could run in three different setups, all of them running on AKS. These setups are: (1) Python scripts, (2) Spark jobs, (i3) dbt transformations. The data pipelines platform is implemented with GitOps methodology, where all code resides in Git and is automatically deployed to Kubernetes by Flux GitOps agent. For self-service reporting, Google Data Studio was chosen as the preferred tool for its easy out of the box integration with BigQuery. Cloud Infrastructure management was implemented with infra-as-code principles using Terraform.”
A New Way of Working
The new platform also introduced a new, more flexible, more robust way of working. Spiros shares: “We don’t have to understand the new data platform in detail; as end-users, all we need to know is how to use it. Xebia implemented the platform and trained us on it. We also changed our way of working. For instance, before, we didn’t have a production environment to run batch jobs. So, whenever we had to create a new data application, we needed to figure it out from scratch. Now, we know that whenever we build something and need to run it, we ‘Dockerize’ it, and it will run. Before, deploying and scheduling were challenging and caused significant delays. Today, creating an application is done in no time!”
Guillermo adds his experience: “Every consultant wants to understand the needs of the end-users as soon as possible to start adding value as quickly as possible. Funda really helped me become familiar with its way of working. I gained insight into how data was behaving, made recommendations for testing, trained the team and any new hires, and worked closely with the marketing team. This collaboration was very fruitful. Some of its requests we could even deliver on the same day. That’s way ahead of what I’ve seen in other businesses.”
.svg)
