What Charles Darwin Teaches You About Being A Kick-Ass DataOps Professional
Strike the balance between stability and innovation
We live in a weird time.
If you are new to the data field, you’ll probably stumble upon the concept of the modern data stack. You’ll find people hyped about tools built less than five years ago.
At the same time, you may meet people who support data platforms as old as your parents. These people work in rock-solid enterprises and swear by ancient technologies.
You see the risk in investing in such new tooling, no matter how cool. But you also don’t want to bet on skill sets, which may become obsolete soon.
The only way to stay relevant is by constantly learning new things. However, it's important to control the speed of change and learn from mistakes to minimise the damage.
In this issue of Data Gibberish, you’ll learn what Darwin teaches you about being a resilient and adaptive DataOps professional.
Here’s what we are discussing:
Who Darwin is
What happens when you are too slow
What are the consequences of being too fast
How to strike the balance
Reading time: 6 minutes
Meet Darwin
I know I don’t need to tell you who Charles Darwin but let me frame our discussion.
Charles Darwin (1809-1882) was an English naturalist and biologist known for his theory of evolution by natural selection. His work revolutionised our understanding of life on Earth, proposing that all species have descended from a common ancestor.
During his voyage on the HMS Beagle, he collected samples that contributed to his groundbreaking ideas. His book "On the Origin of Species" presented the theory of evolution, stating that the fittest organisms are more likely to survive and pass on their traits.
Darwin's influence spans biology, earth science, and beyond, making him one of the most influential figures in the history of science. It may surprise you, but you can borrow and apply these lessons to your evolution.
Let’s dig a little deeper.
Improving Too Slowly
The tale of the dodo is a famous example of a species that went extinct due to its lack of adaptability. Native to the island of Mauritius, the dodo had no natural predators and, as a result, did not develop a fear of humans.
When European explorers arrived, the dodos were easily hunted due to their trusting nature, leading to their rapid extinction within a century. This story serves as a poignant reminder of the importance of adaptability to changing environments and the potential consequences of failing to evolve.
Now, let’s switch gears and talk about a more recent story.
Meet Noah, a data engineer who has spent years utilising antiquated technology. Noah sticks with batch ETL jobs that load data into on-prem Hadoop clusters and doesn’t care about the hype called “The modern data stack”.
Noah’s CTO, Kevin, has been planning a move to the cloud for a while. Keving follows the company’s global initiative to bring maintenance costs down and increase sharing capabilities. This project will also allow the business to sell AI-based data products effortlessly.
Noah knows nothing about AWS, Snowflake or LLMs. The learning curve is too steep, and after a month of reading, he said he couldn’t do Kevin’s project.
But Noah had similar stories before, and the company is dropping Hadoop. Last week, Kevin told his data engineer the company is hiring a consultant. The business will, unfortunately, let Noah go when they complete the migration.
Noah failed to adapt and keep up to speed with the changing world of data engineering. He became a dinosaur on the way to extinction.
OK, I know what you’re thinking:
I’ll never become obsolete. I’ll learn and be on the edge of technology.
But this is just one side of the coin. Let me tell you about the other before rushing into conclusions.
Moving Too Quickly
Have you heard about the tragic death of a zebra at the Milwaukee County Zoo?
This story acts as a poignant example of an animal facing danger due to its speed. In this unfortunate incident, the zebra died after running into a containment fence post in its enclosure.
This case highlights the potential risks that living creatures, even those known for their swiftness, can encounter when their speed leads to unforeseen dangers.
I was like that zebra. Dashing into complete turnarounds, I was constantly deploying untested code in the name of a bright future.
As you can imagine, my code was often buggy. And I’m not talking about wrong metrics or missing fields. Let me give you just two examples that could cost me my job:
I once ruined the entire data infrastructure for days because of a mistake.
Another time, I generated a $8000 Redshift bill in just one night!
Fortunately, these mistakes didn’t cost me my job because I work with nice people. But I needed to learn from my mistakes and change my behaviour.
I could have prevented these mistakes if I had slowed down and checked what I was doing before pushing to production. I was in a hurry to build a better world for my peers, but what I did was slow everybody down.
Not slow, not fast. What are you trying to say?
Summary
It is not the strongest of the species that survives, not the most intelligent that survives. It is the one that is the most adaptable to change.
— Charles Darwin
This quote from Darwin’s theory is particularly relevant in the fast-paced world of data engineering, where new technologies and methodologies are emerging.
If you want to thrive as a DataOps professional, you need to be willing to adapt and evolve. This means being open to new ideas, experimenting with new tools and techniques, and improving your skills constantly.
However, it's vital to strike a balance between innovation and stability. If you rush into making drastic changes without proper testing, you risk deploying buggy code and breaking things your colleagues need to do their jobs.
By embracing change, testing thoroughly, continuously improving, and making decisions, you can thrive as a data practitioner and help your organisation stay ahead of the curve.
So, how can you apply Darwin's teachings to your work in the data?
Your Turn
To wrap this up, I want to give you a few tips on how not to “win” The Darwin Awards:
Continuously improve: Keep honing your skills and seeking new opportunities to learn and grow. Attend conferences, take online courses, and take part in online communities. Stay up-to-date with the latest trends and best practices.
Test thoroughly: Before making any changes to your data models, ensure these make sense. Test if your ideas and code serve you and your users.
Learn from your mistakes: Take the most out of failure. Learn your lessons, and strive to become a better professional after failure.
What are your tips? How do you stay relevant in the ever-changing field of data? What resources would you recommend to the community?
Did you enjoy that piece? Follow me on LinkedIn for daily updates.