Re-engineer Your Career From Software To Data Engineer
As a software engineer, do you ever feel like there’s something more out there to explore? Are you fed up with backend Java development? Do you feel the urge to do something different?
That is exactly what former software engineer Barend Garvelink felt when he became a data engineer. Want to find out how that worked out? In Barend’s words, this is his story…
Like a lot of other software engineers, I knew a lot about many different technologies. For years I worked on Java backend development and on mobile Android platforms. I knew all the tricks and after a while I felt like I was repeating myself. So, a few years back I started focusing on data technologies.
By broadening my horizon, I was able to change my career and become a data engineer.
The choice between Data Scientist and Data Engineer
Fifteen years ago, when I was studying software engineering, big data wasn’t really a thing, let alone data science or big data engineering. Now there’s an entirely new field of expertise around data, and it’s a challenging one, at that.
There’s been a lot of buzz around data scientists—and the shortage of them—over the past few years. According to an article published in 2012 by the Harvard Business Review, it was going to be the sexiest job of this century. As it turns out, data engineers are at least as hard to find.
I like to say that data scientists work in the data and data engineers work on the data. What that means is one group focuses on the mathematical side while the other focuses on the IT side. As engineers we prepare the data and make sure that the output of the data scientists’ work is usable to support the business. That means ensuring the data is clean, and the IT systems are robust, scalable, and well-tested.
These skill sets have limited overlap. So, now there’s a massive shortage of data engineers, too. That means if you become one, you’ll find challenging job opportunities around every corner.
Working as a Data Engineer
As a software engineer, I worked for Xebia for many years. But after I specialized in data technologies, I transferred to its sister company GoDataDriven, a company that helps other companies to become data-driven. I became a data engineer on the job and learned a lot from both my colleagues and online courses.
Now I work for a diverse group of clients on all kinds of data challenges. Since our projects are often part of innovation programs, they get a lot of attention within organizations, even from the board.
Complex Problems
The problems you need to solve in data engineering are more complex than the ones in back-end software engineering. As a backend developer, you’re building an application. As a data engineer, you’re building the application and you’re also responsible for the data the application uses, the cloud infrastructure it’s on, monitoring, and the security of the data. So, a broad software engineering skill set is imperative.
The everyday tasks get commoditized into SaaS products, so the remaining challenges get even more complex. Working with the feed from millions of energy meters or other sensor networks that provide new data every few seconds requires knowledge far beyond the requirements of a typical backend development gig.
As a backend software engineer, we mostly ignore computational complexity (“Big-O”). Computers are so fast that it’s pointless to spend much time thinking about it. This is not the case in data engineering, where the “n” gets large enough for the “O” to matter. You need to know about complexity, distributed computing, and consistency. The best solution to a problem might require hours to run, the second-best solution might require weeks.
Creativity, Skills, and Knowledge
Using my creativity, skills, and knowledge to come up with the best solution was one of the things I enjoyed most about “regular” software engineering—until I felt I was repeating myself. Now it’s one of the things I love about data engineering. Since this is such a complex and relatively new area, there are new techniques and new ideas to explore every week. It will be years before I get bored or feel I’m repeating myself. I have a whole new world to explore.