The Counterintuitive Approach to Fostering DataOps Culture That Actually Works
How to prove value early and secure buy-in later
Hi there,
As a data engineer, you know how hard it is to get DataOps going in your company. You've faced pushback from bosses. You've struggled to show why your work matters. You've dealt with limited money and resources.
But what if the usual ways of doing DataOps are holding you back? What if a different method could help you show the value of DataOps faster? What if it could help you get the investment and resources you need?
Today, I will share an unconventional strategy that delivers tremendous value to stakeholders and increases the chances of investing in DataOps.
Reading time: 14 minutes
🤯 The Problems with Usual Approaches
Usual approaches to data engineering often put perfection before speed. You take time for each task, doing things "the right way". This approach can lead to several issues:
Long development cycles and delayed value
Over-engineered solutions that cost too much to maintain
Solutions incapable of adapting to changing business needs
Focus on technical excellence instead of business value
Complex data pipelines and rigid data structures
Lack of flexibility in responding to changing business needs
Missed business chances and less competitiveness
Failure to deliver value to stakeholders
Slow adoption and limited support from higher-ups
Lack of visibility and understanding of DataOps benefits
Think about it: When did you last try to start a new DataOps project? Were you met with doubt and resistance from bosses? Did you spend hours writing a proposal outlining the benefits and return on investment? Did it fall flat? Why?
Because usual approaches often put long-term stability before short-term results. This can be a hard sell to stakeholders who want quick wins.
🏰 Introducing the Unusual Method
When looking for a buy-in, what worked for me was putting speed before quality. It's like building a sandcastle on the beach. It's quick to build and looks impressive quickly. You know it won't last forever, but you can easily change or rebuild parts of it. Each time you make one, you get better at it.
You can apply the same approach to your data engineering work. You build impressive data products quickly, show immediate value, and improve your work over time. You know from the start that you might need to rebuild some parts. However, the immediate impact and lessons learned are valuable.
For this article, let's call this approach The Sandcastle Tactic.
The unusual method is a data-driven approach that prioritises speed and business value. This approach recognises perfection isn't always necessary; sometimes, "good enough" can be good enough.
Key aspects of the Sandcastle Tactic:
Focuses on delivering results quickly and step-by-step
Lets data engineers show value to stakeholders
Gets investment and resources by showing immediate impact
Drives business outcomes through fast implementation
Makes trade-offs to deliver results quickly
Needs a deep understanding of the data pipeline
Prioritises tasks effectively
Focuses on delivering high-quality results within time limits
Communicates the risks and potential failures
This approach requires a change in thinking. You need to be willing to let things break in the long run, knowing you'll have to fix them later. You need to put short-term speed and business value before long-term stability and perfection. This might feel uncomfortable, but it's necessary. It's how you get the buy-in and resources you need to build a lasting DataOps practice.
Now, you need to be upfront about the potential for failure. Warn stakeholders that some things will break while you do your best. This honesty is crucial. It sets realistic expectations and prepares the organisation for the inevitable bumps in the road.
🙌 The Benefits of the Unusual Method
The Sandcastle Tactic offers several significant benefits. These can transform your data engineering practices and deliver real value to your organisation.
Faster time-to-value is perhaps the most immediate and noticeable benefit. By prioritising speed over perfection, you can deliver results quickly and show the value of your efforts in a much shorter timeframe.
This rapid delivery can help build momentum for your DataOps projects and secure buy-in from stakeholders who might otherwise be sceptical of long-term projects with unclear outcomes.
Increased agility is another crucial advantage of this approach. The tactic allows you to respond quickly to changing requirements and priorities.
This flexibility can be a significant competitive advantage, enabling your organisation to adapt to new opportunities or challenges faster than bogging down by rigid, slow-moving data processes.
Improved collaboration is a natural outcome of the Sandcastle Tactic. By involving stakeholders in the prioritisation process and communicating regularly about your work, you can ensure that everyone is aligned on the project's goals and objectives.
This alignment can lead to smoother project execution, fewer misunderstandings, and better overall outcomes.
🧭 Step-by-Step Guide to Implementation
I understand this unusual method may look a bit chaotic to you. However, putting it into practice needs a structured approach. Here's a step-by-step guide to get you started:
Identify Critical Areas for Quick Wins 🏆
Look at your data pipeline and find the critical parts that deliver the most value. Focus on areas where you can quickly show the value of DataOps efforts—a specific business goal, a pain point, or a chance for growth. By targeting these high-impact areas, you can show results sooner and build momentum for your projects.
For example, you might find a manual reporting process that takes several days each month. By automating this process, you can quickly show time savings and improved accuracy. This improvement provides a clear win that stakeholders can easily understand and appreciate.
Prioritise Tasks and Communicate Risks 📊
Be ruthless about prioritising tasks based on business value. Triaging tasks means learning to say no to non-essential requests, even if they seem important at first.
Communicate clearly with stakeholders about the risks involved in this approach. Explain that by moving quickly, some things may break. But the trade-off is worth it for the speed of delivery and value creation.
Create a prioritisation matrix that considers the potential impact and the effort required for each task. Share this matrix with stakeholders to help them understand your decision-making process and the trade-offs involved.
Involve Management in the Prioritisation Process 🤝
When faced with multiple tasks, ask upper management for guidance. They might initially ask you to prioritise all of them. But it's your job to explain that you can't work on all tasks at once without risking failure across the board.
Explain the impact of trying to do too many things simultaneously. Your managers are experienced people. They know how this works. They just may now know it applies to data engineering, too.
Communicate the Value and Risks of DataOps Efforts 📣
Use data storytelling to convey the value of your efforts and potential risks. Create a narrative that resonates with stakeholders and higher management.
Highlight how your work has improved efficiency, reduced costs, or enabled new business opportunities. At the same time, be clear about the potential for failure and the need for resources to address issues as they arise.
Develop a dashboard that shows the successes and challenges of your DataOps efforts. This visual representation can help stakeholders quickly grasp your work's impact and understand the process's ongoing nature.
Deliver Results Quickly and Iterate ♻️
Don't wait for perfection before delivering results. Instead, focus on quickly releasing a working solution, even if it's not perfect. Then, refine and enhance your work over time.
This approach allows you to show value early and often, which can help build confidence in your DataOps efforts. It also allows you to gather feedback and make adjustments based on real-world usage, leading to better outcomes in the long run.
Implement a rapid prototyping approach, where you deliver a minimum viable product (MVP) within a short timeframe, then gather feedback and iterate. These MVPs allow stakeholders to see progress quickly and provide valuable insights for future improvements.
🔎 Things to Be Aware Of
Balancing Short-Term Goals with Long-Term Vision ⚖️
Understanding that The Sandcastle Tactic isn't about sacrificing long-term success for short-term gains is crucial. Instead, it's about creating a balance, allowing you to show value quickly while building towards your long-term vision.
By prioritising short-term speed and business value, you can create a snowball effect that builds momentum and support for your DataOps projects. This approach allows you to show the value of your efforts, secure investment and resources, and create a culture of data-driven decision-making. Over time, this can speed up your progress towards long-term goals by ensuring continued support and resources for your DataOps efforts.
To maintain this balance:
Regularly reassess your long-term goals and ensure your short-term actions align with them
Create a roadmap that shows how your quick wins contribute to larger strategic objectives
Allocate a portion of your time and resources to foundational work that supports long-term stability
The Importance of Transparency 🪟
While The Sandcastle Tactic emphasises speed, it's crucial to be clear about the risks and potential failures. Regularly communicate with stakeholders about the trade-offs you're making and the possible consequences.
This transparency serves two purposes. First, it sets realistic expectations. And second, it helps stakeholders understand the need for resources to address issues as they arise.
By being upfront about the challenges, you're building trust and preparing the organisation for the inevitable bumps in the road. This honesty can increase stakeholder confidence in your approach, as they appreciate your foresight and risk management.
To maintain transparency:
Hold regular stakeholder meetings to discuss progress, challenges, and risks
Create a risk register that outlines potential issues and mitigation strategies
Be proactive in communicating both successes and failures
The Importance of Saying "No" ⛔
When implementing The Sandcastle Tactic, learning to say "No" is a crucial skill. You can't do everything at once, and trying to do so will only dilute your efforts and increase the risk of failure. Prioritise tasks based on business value and be prepared to say "No" to non-essential requests, even if they seem important.
Declining a request might feel uncomfortable initially, but delivering results quickly and effectively is necessary. Remember, you're not saying "No" to the request forever! You're saying "No" to the request at this time.
Communicate your priorities clearly with stakeholders and management, explaining why you focus on certain tasks and not others. This transparency can help manage expectations and avoid conflicts down the line.
To say no effectively:
Develop a clear prioritisation framework and share it with stakeholders
Offer alternatives or compromises when saying no to a request
Explain the opportunity cost of taking on additional tasks
The Role of Management in the Unusual Method 👔
Management plays a crucial role in the success of The Sandcastle Tactic. You must involve them in prioritisation and ask for guidance and support. By involving management in these decisions, you ensure that everyone is aligned on the goals and objectives of the DataOps initiative.
This alignment is crucial for securing the resources and support you need to succeed. Additionally, management can help remove obstacles, advocate for your work to other parts of the organisation, and provide a high-level perspective to ensure your efforts contribute to the company's strategic goals.
To involve management effectively:
Schedule regular check-ins to discuss priorities and progress
Provide clear, concise updates that focus on business impact
Seek their input on strategic decisions and trade-offs
💣 Potential Risks and Mitigation Strategies
While the unusual method can be highly effective, it has risks. It's essential to be aware of these potential pitfalls and have strategies to mitigate them.
Long-Term Stability ⏳
One of the primary risks is compromising long-term stability. The focus on quick delivery can sometimes lead to temporary solutions, usustainable in the long term. To mitigate this risk:
Implement temporary solutions with a clear plan for long-term stability
Document your quick fixes and their limitations
Create a roadmap for replacing temporary solutions with more robust ones over time
Allocate regular time for refactoring and improving existing solutions
Technical Debt 💩
Another significant risk is the creation of technical debt. The pressure to deliver quickly can sometimes lead to cutting corners or implementing less-than-ideal solutions. Over time, this technical debt can accumulate, making your systems harder to maintain and evolve. To address this:
Keep a detailed log of technical debt
Allocate time in each sprint or work cycle to address technical debt
Prioritise addressing technical debt that has the highest impact on system stability or performance
Educate stakeholders on the concept of technical debt and its long-term implications
Stakeholder Misalignment ⛈️
There's also the risk of misalignment with stakeholders. The unusual method's focus on speed and iteration can sometimes be at odds with stakeholders who expect perfect, polished solutions. To mitigate this risk:
Communicate your approach and its benefits upfront
Set realistic expectations about the iterative nature of the work
Regularly demonstrate the value you deliver, even if solutions are not yet perfect
Involve stakeholders in the prioritisation and decision-making process
Do you enjoy this article? Make sure you like my latest post on Facebook!
👍👎 When to Use (and When Not to Use) This Method
The Sandcastle Tactic is most effective in scenarios where speed and business value are critical.
Here are some examples:
When trying to secure initial investment and buy-in from stakeholders
When facing intense competition and needing to demonstrate quick results
When exploring new technologies or approaches where quick iteration is valuable
In startup environments or new product development where time-to-market is crucial
When dealing with rapidly changing business requirements or market conditions
My unconventional approach allows you to move quickly, demonstrate value, and adapt to changing circumstances. It's particularly handy when you need to prove the concept of DataOps or show the potential of a new data initiative.
However, there are scenarios where The Sandcastle Tactic may not be suitable:
When working on critical systems that require long-term stability and reliability
When dealing with systems that need to comply with strict regulatory requirements
When resources and budget are minimal, and short-term goals may compromise long-term success
In environments where the cost of failure is exceptionally high (e.g., healthcare systems, financial trading platforms)
When working on foundational infrastructure that will be difficult to change later
In these cases, you need a more traditional, careful approach. The potential risks of rapid, iterative development outweigh the benefits of quick delivery.
When deciding whether to use The Sandcastle Tactic, consider the following:
The criticality of the system you're working on
The potential consequences of errors or instability
The regulatory environment and compliance requirements
The resources available to you
The culture and expectations of your organisation
The potential value of quick delivery versus perfect execution
The long-term implications of your choices
The Sandcastle Tactic is about making calculated trade-offs to deliver value quickly while being transparent about the risks. It's a powerful tool, but like any tool, you need to use it in the proper context and with a clear understanding of its limitations.
🏁 Summary
The unusual method in practice means:
Taking shortcuts when possible
Using existing solutions instead of building new ones
Focusing on delivering results quickly, even if it means sacrificing some quality
Occasionally, taking the time to do things a bit slower and with better quality
Prioritising speed over quality but not sacrificing quality entirely
By embracing the Sandcastle Tactic, data engineering teams can deliver results quickly, improve collaboration, and drive business outcomes. This approach can help data engineers to drive business value and demonstrate the impact of their efforts.
The Sandcastle Tacitc has its challenges. You must manage stakeholder expectations, balance short-term goals with long-term vision and strategy, and ensure long-term stability. By overcoming these challenges and leveraging the power of data storytelling, you can create a culture of data-driven decision-making that drives business value.
So, consider the unusual method if you want to shake things up and try a new approach to DataOps. It may be the key to unlocking the full potential of your data engineering team.
Until next time,
Yordan
📚 Picks of the Week
What is the Return on Investment (ROI) of your DataOps efforts? Check this mind-blowing article that shows the value of technical tasks. (link)
Being a data engineer can mean different things depending on the organisation and your preferences. What kind of data engineer are you? (link)
Do you know what is the worst part about data engineering? It’s full of vendors who want to sell you their products. I have my own selection approach, but truly liked this one, too. (link)
❓ Ask Me Anything
Do you want to learn anything about data and data engineering? Don’t miss the opportunity to participate in this month’s Ask Me Anything.
Maybe you have something bigger in mind? Reply to this email and reach out to me. I love chatting with friendly people.
😍 How Am I Doing?
I love hearing you. How am I doing with Data Gibberish? Is there anything you’d like to see more or less? Which aspects of the newsletter do you enjoy the most?
Use the links below, or even better, hit reply and say hello. Be honest!
