Over the last decade, the growth in online technologies and tech-based services has resulted in an explosive growth in data. I saw this first hand, building high performance databases at Oracle, and then building petabyte-scale storage and database services at Google. As the volume of data being generated continues to increase, with the availability of these scalable infrastructure solutions, data storage and retrieval are no longer the biggest challenges.
The biggest challenge facing our connected world today is making sense of this data with ever-increasing data complexity. For businesses, there is an urgent need for cross-functional teams to quickly discover data-driven insights to inform business decisions. An organization’s success is often tied to the quality and velocity of this decision-making.
It’s not just that the data itself is complex, and is scattered across different parts of the organization. The end-to-end data journey to inform decision-making is also complex, involving multiple steps: from data gathering to visualization and finally, communication of insights.
Current data workflows for decision-making are complex, error-prone and brittle, impacting decision-making velocity.
Each of these steps involves different, disconnected teams of people, with different skill sets, and increases workflow complexity.
Our 2021 research report — The State of DataViz — found that the team members involved in the above workflow use two, three, four, and sometimes even more tools to stitch together an end-to-end flow to analyze data and communicate insights. In fact, there were more than 180 unique tools named by respondents, optimized for different roles with different skills (see color-coded diagram below), that are involved in the data workflows.
Different folks with different expertise use different tools to get their work done.
It's painfully obvious that the current data exploration, analysis, and visualization ecosystem forces users to hop between tools and creates barriers between people who play critical roles in data-driven decision-making. This error-prone and often brittle process slows down organizational velocity. No one feels empowered to make decisions with high confidence because each person (or sub-team) has only a piece of the entire data puzzle.
Organizations can’t simply rely on separate data science or analyst teams to independently be the single source of fast data analysis and insights. Everyone in a team, irrespective of role or expertise, needs to be able to work together to make sense of their data.
We’ve noticed through our user research, customer stories, and market analysis, that the following roles sitting in their own departments, along with their collaborators such as product managers, team leads, and business owners, need to work together as a collective, to gain insights for better data-based decisions:
Data scientists
Data engineers
Data architects
Business and financial analysts
Data visualization (dataviz) developers
Business Intelligence / Operational dashboard developers
Observable uses the term data practitioners as an inclusive identity covering all of these people who work together on data.
The Data Practitioner umbrella includes multi-disciplinary roles coming together to get their work done.
These data practitioners need to be able to be productive in their individual work where they can utilize their expertise, as well as be productive when working collaboratively with cross-functional colleagues. The current tool ecosystem doesn't meet the market need to enable this collective set of people with different expertise in one place, and allow them to focus on getting the work done, without wasting precious time and resources wrangling with disjoint tools.Teams will have a huge business advantage when they can collaborate across these roles without friction and make decisions quickly. That is the future of data work!
Data collaboration for data practitioners is an active, iterative, and transformative process. In fact, data collaboration can become an organizational practice focused on better outcomes for the entire business. Each organization must decide and then live the practice. Similar to DevOps, data collaboration is about the symphony that happens between the people, the processes, the tools, and the culture involved.Companies need a new way to thrive in this new world!
A new generation of data tools is needed that allow each of the cross-functional experts within the data practitioner umbrella, to maximize their effectiveness and seamlessly progress from ideas and data analysis to insights and communication in one space, without tool hopping or hitting a wall with their colleagues. The most empowering part of data collaboration is that each organization can determine its own practices and processes, since it’s not a static deliverable or tied to glueing together disparate tools.
Observable Data Canvas simplifies workflows and enables data practitioners to be effective in one space.
This is the future of data collaboration, and Observable is building this future! If you identify as a data practitioner, join your community on Observable, and help us build this future together.
What are your thoughts on the future of data work? Let us know with the hashtag #FutureOfDataWork on Twitter, or drop a comment below.
If you’re interested in following along with the latest updates on the community, you can subscribe here.If you have used Observable in a way that you think would help other data practitioners, we’d love to hear from you! You can reach us at community@observablehq.com or join us at one of our upcoming Meetups.