How this data engineering training program got me started right away
At Xccelerated we believe software engineering is the foundation to become a data engineer, so we put an emphasis on consolidating principles and best practices rather than tools. Twice a year, a new group of software engineers start their data journey at Xccelerated and enter the 13-month program.
Last year, Jorrick started his journey from software engineering to data engineering with Xccelerated. He shares his story below.
It was around July and I had been trying to find the perfect job for quite a while now. All of my side-jobs taught me one thing; my interests are working on complex data problems and coming up with elegant high-tech solutions built with the latest technologies. Sadly no Software Engineering job came up that ticked all the boxes.
At that point, a friend told me to look into Data engineering which I had never really considered yet. Working with the latest buzz words, such as kubernetes, nosql, airflow, spark, it almost seemed too perfect. However, for inexperienced data engineers there are hardly any junior positions. Luckily for me, Xccelerated bridges this gap as they hire experienced software engineers and teach them the art of data engineering.
My start as a data engineer
The data engineering bootcamp is a very intensive month. Every day we started at 9 and studied new data engineering related material, had a great lunch at 12, played some ping-pong and then worked till 5. During the bootcamp we worked with a lot of new technologies such as Spark, Hive, AWS, transforming true software engineers to highly skilled data engineers. After this month, I felt I had enough knowledge about a lot of the technologies used by larger companies to start as a data engineer.
Data Engineering project
One of the first data engineering projects I did during this month was developing a demo-application to show the power of Kubernetes using Python. The project would be represented by a user interface closely copying the game Whac-a-mole.
First, two dockerized Python flask apps were created. One would route the traffic from outside of the Kubernetes application to the pods and the other one would simply only respond to pings and kill commands. Second, we set up the Kubernetes deployments such that it can easily scale the number of pods running and automatic restart for when a pod is unresponsive. Finally, the Whac-a-mole user interface fires a kill command to the dockerized pods when a mole would be hit with the mouse. This resulted in a demo where Kubernetes would automatically load a new mole pod within 10 seconds after it had been killed.
Working at the client
Everyone in the Netherlands knows Schiphol of course, as it transports more than 70 million people on a yearly basis, at least when there is not a global pandemic as is going on now. The idea that my work here can reach and impact a large amount of people and often even close friends makes it even more interesting!
Residents near Schiphol make a lot of complaints about the noise of the aircrafts. Completely understandable, considering the noise that one airplane can make. One of the main reasons that they would complain was because they never know beforehand how much noise they could expect. This is where my work comes in. Our team is responsible for predicting noise levels in the areas around Schiphol, which will finally be shown in a smartphone application. The application makes predictions on the basis of weather forecasts, the time of day and historical noise measurements to predict the future noise levels.
Schiphol Amsterdam
Although this project might seem easy at first glance, I can assure you this is not a one-man job. As data engineers, we set up several Python and Scala projects to ingest data. These processes are set up to be orchestrated by Airflow, which also launches Kubernetes Pods to process raw weather data inside a docker image. We set up a Cassandra database for data storage. Most important, we set up a Python API which fetches the latest weather info. This information, together with GPS coordinates of the user, is passed onto the model created by the data scientists. Finally, the app returns the predictions back to the smartphone app.
The version is currently in closed beta and soon will be published to the app stores
“I am very grateful for both Schiphol and Xccelerated who provided me with this opportunity. Schiphol offers a great place for trainees as we have a great team and especially a lot of seniority (from GoDataDriven) in our team. I learned so much already in the first half year that I am really looking forward to the years to come at Schiphol.”
Are you a software engineer and became interested in becoming a data engineer?
Become a Data Engineer through the Xccelerated Cloud Engineering program. In 13 months, we will teach you many topics mentioned above and connect you with one of our partner organizations.
The program starts on October 1st. Apply now, or find more information about our organization and the full training program.