Time to Focus on the Twin Pillars of Big Data 2.0

While many executives claim to loath the term Big Data, they should nonetheless appreciate that the term has resonated from Wall Street to Main Street. Data may not provide insights or answers to all questions, but wouldn’t we rather have good data available to inform our decisions? The great benefit of the Big Data wave of this decade, from my perspective, has been the elevation of the discussion, and the liberation of data from the exclusive purview of a small community of specialists. Ich bin ein Data Analyst. We are all data analysts and beneficiaries now.
In the same vein as the debate over Big Data as a term, another debate has swirled around what constitutes Big Data. For some, Big Data means only new forms of data, such as social media data or unstructured data. From my perspective, Big Data refers to the ability to access rapidly and cost effectively all types of data–new, old, large, small, or of any variety–by leveraging new tools and business processes. In this context, Big Data implies a vital new approach that is applicable to all data challenges and opportunities. The Bastille has fallen. The data warehouse will live on, but as one element of a comprehensive data strategy, which also includes data lakes, data hubs, centers of excellence, and analytic sandboxes. Rather than one size fits all, organizations must think in terms of what mix of approaches will be most effective based on the kind of business questions and analyses they need.
Today, we are moving rapidly into the world of Big Data 2.0. If the emerging years of Big Data were about the ability to cost-effectively process larger volumes, greater varieties, and rapidly moving data in motion, the formative years of Big Data will be about creating new businesses and business models that are data driven. As data driven business models emerge and mature, businesses must focus their attention on the two foundational pillars of Big Data 2.0:
Data Integration. According to the industry research firm Wikibon, 52% of Big Data tool investment is now being spent on technologies for ingesting and organizing data so that it can be more readily accessible and prepared for analysis. Data preparation has always been a big challenge for data professionals, with many organizations reporting that 75%-80% of their energies are devoted to up-front data engineering. Big Data approaches democratize data integration by enabling non-technical users to directly access the data they need for analysis. As a result, businesses have more options to choose from and more approaches to consider. Developing an effective data integration framework becomes the first step in deriving business value from their data.
Data Governance. Having the data you need in a structure which you can access and analyze it is critical, but managing that data going forward is essential to avoiding chaos and undoing the results of your hard won efforts. Data Governance is fast becoming the glue in the Big Data life cycle, delineating the roles and responsibilities of every individual within a business that accesses, analyzes, reports on, or derives new data, and governing processes that ensure data quality, data integrity, and a single source of truth. Data Governance is like the Constitution. It is a living body of rules and rights that govern data from production to consumption across its lineage. We the People. Think of Data Governance as the human side, or soft side, of data.
How businesses organize, utilize, analyze, and benefit from the data at their disposal will manifest itself over time. Patience is essential. The process will be evolutionary, playing out over a decade or more. A new generation of data professionals, nurtured on Big Data approaches, will supplement and supplant data traditionalists. With the benefit of time, the impact of Big Data, like the impact of the Internet, will be seen as stages of a larger Information Revolution — making the world a much smaller place. As the bard of our time Bob Dylan astutely observed, “The Times They are a Changin.”
This article was originally published in the The Wall Street Journal March 12, 2015.