Productionizing machine learning pipelines @ ING
In this blog, a data Engineer will tell you about the project she has been working on at ING. She will tell you about the challenge, the solution, and the result.
Challenge
Differences between two platforms.
Under the guidance of Giuseppe D’Alessio, ING’s IT Chapter is charged with deploying machine learning pipelines into a Hadoop production platform. However, its data scientists develop their machine learning models on Hathi, an experimentation platform. Because of the differences between the two platforms, transferring code as a package between them is often difficult.
Solution
Closing the gap between ING’s data scientists and its platform team.
So ING asked Xccelerated to help the organization deploy machine learning products using Kubernetes. After a full month of data engineering boot camp, Xccelerated’s young professionals (who already had several years of software development experience under their belts) were ready to join ING’s existing Hathi platform team. Since then, Xccelerated’s data engineers have helped close the gap between ING’s data scientists and its platform team. First, we created awareness about using containers in machine learning environments by planning and facilitating a training session focused on Kubernetes.
By sharing knowledge about Docker (including how to write, build and run Docker images) we were able to go in depth about how to create Kubernetes pods to gain insights on deployments and services.
Result
Production-ready projects in an experimentation platform.
Within the first quarter, a successful proof of concept for deploying machine learning models with Kubernetes was created. Using Python, the team developed a tool that continuously integrated with GitlabCI by default, so the data scientists could create production-ready projects in an experimentation platform while constantly testing the code. And since the tool was developed in Python, everyone can contribute new features. Users can also access their logs from GitlabUI, monitor their deployments and easily detect any issues. Manual steps are automated in the process.
Our next step? We’ll be teaching more data scientists and platform engineers on how to use the new tools. Meanwhile, the happy collaboration between ING and our data engineers has led to permanent offers.
“Drawing on an exceptional capacity to innovate, Xccelerated helped us deploy several machine learning models and provided insight for improving my team’s way of working.”
Are you a software engineer and became interested in becoming a data or Machine Learning engineer?
Xccelerated is an initiative within Xebia Group, accelerating careers of young professionals with 1 to 3 years of relevant work experience in the fields of data & AI. Our 13-month advanced training programs integrate hands-on learning and skill development with working at one of our partner companies, like Heineken, KLM, ING, Vattenfall, Randstad Group or FedEx.