I’m Dennis Eilers and I work as a software / data engineer in the field of text mining and natural language processing at a large e-commerce company. As a lecturer I also teach data science topics at a university of applied sciences and arts. I’m glad you’re interested in my story. 🙂

When I started my data science journey by working as a student assistant at the university, I was simply thrilled about the idea of self-learning systems which make their own decisions based on what they have learned from the past. Developing those systems by myself and applying them to challenges with a real-world impact became the goal for my future carrier. During my time at the university, I learned the maths behind a ton of algorithms, how to code in R and Python and how to apply and interpret statistical test which helped me to write my first papers and finally get my Ph.D. And now?

I learned to love the academic way of thinking but during discussions with friends who have a software engineering background I often asked myself if I am really on track for the second part of my goal: Application with real world impact. The work of software engineers sounded so much more impactful to me than what I was doing. While spending my time on optimizing a single hyperparameter of a model training, they use a framework to implement a completely new service with backend and frontend which creates real customer value. In fact, I found myself optimizing things with almost no real-world impact in my view. As a statistician I would say I had a significant impact but not a relevant one. I was dissatisfied and asked myself if it is really worth what I am doing?

Without a concrete answer I decided to end my academic path at this point to explore the “real world” by working in a startup as a data scientist. As expected, it was a culture shock. In summary I would say it was all about execution. While I was previously able to live out my perfectionism to the fullest, I was now forced to accept 99% in order to achieve quick wins or meet deadlines. My work shifted from bare bone algorithm optimization and interpreting statical tests to the application of tools and frameworks for specific use cases. I also focused more on deployment and operation of the result than on the result itself. And a little to my own surprise, I really enjoyed this kind of work. But does this meet my goal of real impact? Of course, I now had 10X more countable results on my desk but only applied for specific use cases. Didn’t my academic findings have more influence after all because they are generally valid and could possibly be useful for a variety of later special cases? Or is it just my way of always wanting what I don’t have right now…

Of Couse, it is always an ongoing journey, but with a little distance I found some insights over the past years which helped me to better understand my thoughts back then and to better assess myself and what I am doing to be more satisfied overall. One of the main realizations was that the field of technology is build up on several layers of abstractions. Maybe an obvious thing to notice but it helped me to better understand my work and my ultimate question about real impact. Because it is nothing wrong with working on the improvement of an algorithm as well it is nothing wrong with applying the algorithm or even just using the tool which applies the algorithm under the hood. If impact is important for you any abstraction level is fine. The only thing we should focus on and the only factor by which we should evaluate our impact and our work is value creation. If this means optimizing a hyperparameter or developing a new training algorithm to achieve 0.1% improvement in accuracy it is completely fine if the results can be applied at scale and the actual impact on real world challenges is relevant. If it means to apply a standard model and achieve reasonable results for a specific use case its completely fine too if your company for example is now able to better understand customer needs. At some point we all stand on the shoulders of giants and we should always use what already exists to maximize our impact. But on all layers, there are plenty of opportunities to use some techniques from downstream and have meaningful impact on the upstream of your work. Therefore, it is always helpful to have the big picture in mind. Which value do I want to achieve in my current situation, and which is the highest level of abstraction to create this value?

What is also important, there is no need to find your exact place on the ladder of abstractions. Each new challenge and each new opportunity requires only your ability to assess what it needs to be done and to be honest by yourself if this meets your interest and passion. But of course you need experience for this and it is probably a journey that never ends, so try out as much as you can and find satisfaction with the impact of your choice.