Core Services of the Data Platform:
- ETL to onboard data sources to the platform: on-prem databases (Oracle, Teradata), cloud data sources (PostgreSQL, Cassandra) and third-party data providers (e.g., Google Analytics, Adobe Analytics & Salesforce)
- Provides a data science stack (exploration & development) for data scientists and a data analytics stack for data analysts
- (SQL based) data exploration and data catalog
- A runtime environment for hosting Analytical, ML, and AI applications (e.g., Spark, Tensorflow models)
The new platform integrated with multiple data sources and can quickly add new ones. It covered cloud benefits such as scalability, reliability, real-time availability, and cost-effectiveness. Dashboards visualized performance, and dedicated alerting notified the team immediately of incidents.
Migration from AWS to GCP
In 2019 the client decided to move the data platform from AWS to Google Cloud Provider (GCP). The migration required refactoring the cloud-native services (a more specific change to the infrastructure code created in Hashicorp Terraform). It also required moving the data sets from the AWS S3 buckets to GCP Cloud Storage. The migration proved the portability of the platform’s cloud-agnostic, analytical, ML, and AI applications. Configured with the proper libraries and access to required data sets, they can execute on a runtime environment.
The GCP Data and Analytics Platform
The new platform aligns data-science and analytics with the business's needs. It can be used to search and identify customer segments to target with campaigns and can also run analyses without interfering with existing data in corporate systems. The data platform facilitates data analysis on demand. This on-demand way of working has replaced standard reporting.
After the analysis by the data scientists, the cluster cleans up automatically, saving costs. Each data scientist has a personal data science stack to develop predictive models. Benefits are isolation, data traceability, and scalability.
It also helps data scientists to apply mature software engineering and DevOps principles. They can use version control for their code, embed code quality checks, and use automated deployment pipelines.
