“Big Data should not only be framed as part of the digital revolution of the past few decades,” writes Irving Wladawsky-Berger, “it’s also a major part of the scientific revolution of the past few centuries. Scientific revolutions are launched when new tools make possible all kinds of new measurements and observations, e.g., the telescope, the microscope, spectrometers, DNA sequencers. Our new Big Data tools now have the potential to usher an information-based scientific revolution.”[1] I appreciate the fact that Wladawsky-Berger recognizes that the Big Data Era is a continuation of humankind’s thirst for understanding. Almost every ancient civilization kept accurate enough records to understand the movements of the sun, the moon, and the planets. As recordkeeping improved, so did humankind’s understanding of the world around them. Thanks to the technologies referred to by Wladawsky-Berger that understanding is only going to get better.
Lots of people are tired of hearing and reading about the buzzword “big data.” Some people feel that it has been overhyped and that the promises vendors have made about what big data analytics can do for companies have fallen short. Karsten Scherer (@twoARguys), head of analyst relations globally for TEKsystems, reflects this sentiment when he writes, “Big Data has arrived at a critical juncture. Organizations that jumped at its potential to revolutionize business are now increasingly disappointed by Big Data projects’ implementation challenges. Many business leaders are also surprised by how their organizational frameworks have been ill-equipped to handle the insights generated by even the most successful Big Data and business intelligence (BI) initiatives.”[2] Scherer then asks, “As Big Data projects tax the limits of organizations’ IT abilities — and challenge businesses’ organizational structures — should companies continue to invest in them?” That’s an excellent question that deserves a serious answer.
As Wladawsky-Berger points out, the digital revolution that sparked the Big Data Era is only a few decades old and the Era itself is only a few years old. There is still time for big data analytics to prove their value. Wladawsky-Berger points out that one technology that will greatly enhance the value that can be derived from big data is the Internet of Things (IoT). The IoT remains in its infancy, but most analysts believe it will have a tremendous impact on how the world operates. As Wladawsky-Berger notes, “The Internet of Things is truly taking off, already comprising about 25 billion smart devices, a number that’s expected to more than double over the next decade, and continue to keep growing for the foreseeable future. These devices generate massive amounts of data, with a lot more to come.” Scherer admits, “Big Data shows immense potential but the technological and organizational challenges of implementing it are huge.” From the beginning of time, people have understood that knowledge endows them with an advantage. Sir Francis Bacon is given credit for coining the adage, “Knowledge is power.” Scherer points out that Mohammed’s father-in-law once stated, “Without knowledge action is useless and knowledge without action is futile.” Scherer notes, “While predictive analytics and the like were probably not on Abu Bakr’s mind when he stated that … over a millennium ago, his maxim certainly applies here. As with most technology trends and business innovations, end-user organizations are challenged to cross the chasm between the benefits they hope to achieve from Big Data and its successful integration with the reality of the people and systems they have in place.”
To get across the chasm described by Scherer, Eric Carr, Senior Vice President of Engineering at Guavus, believes that companies must be able to ride three Big Data waves (volume, variety and velocity).[3] He explains:
“While each element is important, when looking at production implementations, many organizations have primarily focused on the first wave, volume, while finding ways to report on and process the huge volumes of data being created. As customers and their competencies in the big data technology stacks matures, we see them moving to focus more on the second V, Variety, and then third on Velocity: fusing the wide variety of data sources and keeping pace with the rapid velocity of data created to deliver timely insights. … As the market matures, we are seeing businesses riding from one wave to the next: that’s when things will really start to get exciting. Arguably, the next wave will be when we start to see more machine-driven control loops and predictive analytics, helping to push us closer to a data-driven, value-centric future.”
Sumit Sadana (@SanDiskDataCtr), Executive Vice President and Chief Strategy Officer at SanDisk, steps back and looks at how the Big Data Era itself has gone through several phases. Not surprisingly, Sadana’s phases have some similarities to Carr’s waves. “In the first phase,” he writes, “we saw the advent of software technologies like Hadoop and NoSQL for handling extremely large amounts of data. (This phase, of course, is by no means over.)”[4] What he means is that data scientists are still creating news ways of handling and analyzing the volume of data discussed by Carr (his first wave). Sadana reports that some databases are so large they can’t be moved to the cloud. “In fact,” he writes, “some of the largest and technologically advanced companies in the world are increasingly shuttling entire storage systems from one place to another just to be able to share data.”
Sadana asserts that the second phase of the Big Data Era involved Carr’s second wave — variety. He writes, “The second phase began with the proliferation of reliable, economical sensors and other devices for harvesting real-world data. Software apps for mining video streams, images, handwriting forms and other ‘dark’ data belong in this category too: Without them, the data, from a practical perspective, wouldn’t exist.” Integrating all of that data has been a challenge. Fortunately, artificial intelligence platforms (especially, cognitive computing systems) have emerged to address this problem. Nevertheless, Sadana insists, more can be done; which brings us to his third phase:
“The third phase will focus on infrastructure. Simply put, we need new hardware, software, networking and data centers designed to manage the staggering amounts of data being generated and analyzed by the first two innovations. Hyperscale data centers, software-defined networking and new storage technologies represent the first steps in what will be a tremendous cycle of innovation.”
As noted above, one of new technologies that is going to generate much more data than we’ve ever had to cope with before is the Internet of Things. Cisco believes that the IoT (which is primarily envisioned as a machine-to-machine network) will eventually merge with the World Wide Web (which is primarily a person-to-person network) to create the Internet of Everything. It is little wonder that Sadana believes we are going to need technologies to meet the challenges this connectivity will create. The term “big data” may eventually lose its luster, but whatever term replaces it will surely reflect the fact that insight systems that improve decision-making (like cognitive computing) are at its heart.
Footnotes
[1] Irving Wladawsky-Berger, “The Big Data Era Is Here,” The Wall Street Journal, 6 February 2015.
[2] Karsten Scherer, “You’ve come a long way, Big Data … but still have far to go,” TEKsystems Blog, 26 March 2015.
[3] Eric Carr, “Riding the Waves of Big Data,” Datanami, 21 April 2015.
[4] Sumit Sadana, “The Third Phase of Big Data,” Re/code, 22 April 2015.
