The ExaFlood

Article Tools

• June-July 2007 Issue
• Post a Comment
• Print This Article
• Email This Article to a Friend
• Save This Article
• Order a Reprint

“And, behold, I, even I, do bring a flood of waters upon the earth to destroy all flesh, wherein is the breath of life, from under heaven; and everything that is in the earth shall die.” That’s how the Old Testament heralds the great flood in Genesis 6:17. It’s an interesting story of animals, two by two, with some interesting parallels in the technology world. I make the comparison because we are about to get hit with another flood of epic proportions — a flood of data that will swamp us all.

Here’s the situation in a nutshell. The sheer amount of data we computer-using, Internet-addicted online travelers are creating increases logarithmically each year. Soon, this flood of information will reach the new levels that are more than 3 million times the information contained in the total number of books published in all of human history. That’s an amount of data called an Exabyte. And we simply do not have the storage or the tools or the mental capacity to easily manage such a massive flood of data.

To be sure, much of this data will be in the form of the high-definition videos that populate YouTube and movies that are illegally swapped on Bit Torrent. But there will also be an increase in spam, in e-mail in general, and in web pages and blogs.

Research firm IDC calculates that the amount of digital data being created increased six-fold in the past year, passing 161 exabytes (billions of gigabytes) in 2006. And that amount will grow to 988 exabytes by 2010, less than three years away.

IDC further calculates the growth of the data, much of it generated and communicated via the Internet, at an annual rate of 57 percent. Only 48 million people routinely logged onto the Internet in 1996. Last year, there were 1.1 billion users on the Internet, with over 100 million in the United States alone. IDC expects another 500 million users to come online by 2010.

That’s a level that staggers the imagination. It also creates some real challenges for accountants and their clients. Consider just four things that will have to be done immediately to prepare for the flood that is just three years away:

• E-mail screening will need to be increasingly aggressive to eliminate spam.
• The days in which employees could use the office computer to browse the web, shop or download music must come to an end almost immediately. Your network just won’t hold up to the stress.
• The creation and use of document management and archiving systems will be critical for any company that doesn’t want to become overwhelmed by its own data.
• In spite of all of these efforts, the cost of storage of company data (including the records required under a myriad of federal and state laws) will increase. Substantially.

The ExaFlood will have other effects as well. It will change the way we get our news and information, and what we use to filter that information to make it more manageable. We’ll put more faith in web browsers to help manage information flows and new communications tools that can integrate all of those flows into formats we can use more easily.

None of this will really be enough. I’m no techno-Chicken Little, but I do believe we are in for a downpour even if the sky does not fall. Already, I find that people who keep up with Internet and technology trends for a living are suffering from fatigue — worn down by the sheer volume of the data we have to deal with each day. And the flood hasn’t even started yet.