A Custom Solution Deployed Without a Hiccup
The project had many challenges, including prediction quality, timeliness, system stability, and assortment evolution. There were also critical requirements, such as tracking any irregularities — predictions that lie outside the bandwidth of historical demand must be flagged and inspected. So, accurate monitoring was also paramount to the project’s success. Reliability was also extremely important. As the replenishment of stores depends on the system’s predictions, any hiccup could disrupt critical processes. And finally, gaining the trust of stakeholders within AH and proving the project could reliably deliver quality predictions was also an ongoing effort and process.
But the biggest challenge, by far, was the massive scale of the project. The number of predictions — one for each item in the assortment, for every store, up to 50 days into the future — amounted to over a billion forecasts every day. With only a few hours available to process that data and create forecasts, the enormous demand would put a strain on even the most modern cloud services.
Together with the supermarket’s in-house Machine Learning engineers and Data Scientists, GoDataDriven deployed a custom solution on Albert Heijn’s Azure cloud infrastructure and orchestrated it using Airflow. The set-up consisted of three phases: data loading and feature generation, model training, and generating predictions. The data loading and feature generation were done using dbt (data build tool) with a Spark engine hosted in a Databricks environment.
The Overall Result
Every night, production models are provided with the most recent data to create predictions for the coming days — up to 50 days into the future. The machine learning models are a huge improvement over the existing forecast models in two important respects: first, the number of manual adjustments needed has been greatly reduced; second, the new forecasts are more accurate than before, by a percentage greater than 5%. The effects on on-shelf availability and loss are still being collected, but overall, it’s the right step in the right direction for the pioneering supermarket chain — one that improves its overall operations with better forecasting and contributes to a better future for the world.
