Rethinking Big Data as Transient Data

Earlier this year, I stumbled upon an intriguing Verge article about self-destructing electronics and almost forgot about it. But recently, after strolling through the tech tweets about big data, social media, and the multi-billion-dollar Snapchat offers, I remembered that Verge article and a new brainstorm hit me. What about the trends toward self-destructing social data? Thankfully, I’m not the first one to recognize this pattern.

Hi, my name is Jay, and I’m an IBM TRIRIGA information developer at IBM. In a previous post, I discussed the epic Switch SuperNAPs. With the accelerating trends in big-data analytics and its skyrocketing demand for high-density computing, let’s think against the expected grain and pose the less-predictable questions.

Snapchat CEO, Evan Spiegel

Snapchat CEO, Evan Spiegel (

Will we generate social data faster than we can handle technologically and sociopolitically? Are we already there? If we must slow the pace, then what if we defined an expiration date or time limit for all of our social data? Instead of big data, what if we generated self-destructing data, ephemeral data, or transient data?

What is big data?

Before we delve into transient data, let’s start on the same page with big data. While the Wikipedia article about big data offers a more-technical definition, here are a couple of more-practical explanations from Forbes, Gartner, and IBM that highlight the role of social media.

“Last year, Gartner defined big data as ‘high-volume, high-velocity, and/or high-variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization’… Technological innovation and processing speed are fundamental to systems that leverage big data to produce insights. Big-data sources are often linked to social media data exclusively, but can also include RFID data, logistics, production, and retailer scanner data — even weather or traffic patterns. Big data is about integrating data and analyzing patterns.”

“Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone. This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals to name a few. This data is big data.”

Big data infographic

Big data infographic (

What are the assumptions of big data?

While the Wikipedia article about big data discusses several critiques of its theoretical approach and scientific execution, the most-reasonable real-world assumption is that big data is eternal. Naturally, because big data is assumed to be permanent, one of the most — if not “the” most — sensitive sociopolitical issues regarding big data is privacy.

For example, here are several perspectives on the idea of “eternal data” from a CNN article about Google, a Bloomberg Businessweek article about Snapchat, and a Popular Science article also about Snapchat.

“You’ve done it, said it, clicked it, searched it, Googled it. You can never undo it or unclick it. It stays there forever. Unless the people demand that government order a stop to it… The European Commission has a new privacy proposal known as the ‘Right to be forgotten.’ It would allow Internet users in 27 countries of the European Union to demand Internet companies delete their personal data… I followed the instructions and with some difficulty eventually downloaded pages upon pages of personal material about myself from Google. What I was looking for was a simple, shall we say beautiful, button telling Google not to save anything I don’t explicitly want it to save. But there was no such button.”

“At the moment, the default setting for almost everything people share online is that it will live for eternity in the cloud. In ‘Delete: The Virtue of Forgetting in the Digital Age’, Viktor Mayer-Schönberger, a lawyer and a professor at the University of Oxford, argues that this inevitably creates problems for individuals and societies that need the ability to forget in order to move forward. ‘A perfect memory,’ he writes, ‘can be paralyzing, trapping people in the past and discouraging them from trying new challenges.’ “

“Most of what we do online is permanent, much more permanent than we realize when creating it. A diary entry you wrote on LiveJournal in 2002 is still online, somewhere. Your first tweets are still out there. You can dig up emails, logs of IM chats, photos posted on Flickr, status updates on Facebook. (Or, more recently, a meme-ified instant hero had his image shattered by decades-old arrests.) Our output is quickly covered over by new items, but that doesn’t make it go away.”

Snapchat logo

Snapchat logo

What is self-destructing or transient data?

Looking outside of social media, the relatively-quiet implementation of “self-destructing data” has been around for almost a decade. For example, since its release in 2004, Google’s Gmail featured the automatic deletion of messages that have been sitting in the trash for more than 30 days.

But within social media, especially in the last few years, the rising attraction of “self-destructing data” has been harder and harder to ignore as demonstrated by the multi-million-photo battle between Snapchat and Facebook. If this sociopolitical trend carries weight, then giving users the flexibility to define an expiration date or time limit for their social data or “transient data” — whether it’s a few seconds, days, or years — might throw an unexpected curveball at big-data assumptions and approaches. Or it might not.

In this light, here are several “self-destructive” excerpts from the aforementioned Verge article about self-destructing electronics, the aforementioned Bloomberg Businessweek article about Snapchat, and the aforementioned Popular Science article also about Snapchat.

“Right now, electronics are built to last. Soon, however, their lifespan might be more fleeting — courtesy of burgeoning research into ‘transient electronics,’ or devices meant to serve a specific function before completely dissolving into their environment over a predetermined span of weeks, months, or even years. In fact, your own body might be one of the first places these devices are deployed…

‘The goal of the electronics industry has always been to build durable devices that last forever and perform consistently,’ John Rogers, Ph.D., a materials scientist at the University of Illinois at Urbana-Champaign, told The Verge. ‘But imagine the possibilities once you start thinking about electronics that disappear in a way that’s programmable and controlled.’ Rogers is leading a team of materials scientists and biomedical engineers from across the country in the development of such gadgets.”

“Mayer-Schönberger argues that all information created online should come with customizable expiration dates. Not every piece of data would have to expire after a few seconds as photos on Snapchat do. ‘The key,’ says Mayer-Schönberger, ‘is to include some form of a self-destruct button in everything created online and to give consumers the power to tinker with the settings from the outset’…

Destroy-your-own media is becoming much more readily available. In the summer of 2012 a team of data security experts in San Francisco launched Wickr, a free mobile app that allows users to send each other an array of impermanent media—including self-destructing text messages, videos, audio files, and PDFs. Like Snapchat, users customize how long their messages will live on the recipient’s device (lasting up to several days) before disappearing… Nico Sell, a co-founder of Wickr [says,] ‘It’s natural for us to go back to it for things like communicating with our friends and family, and not having to think about the fact that the Internet is forever. Ephemeral data is the future.’ “

“And that’s what’s so freeing about Snapchat. It allows messaging to be ephemeral and experimental. You don’t necessarily want to take an Instagram shot of your weird neighbor or post a Vine of yourself sprinting up and down a rare empty subway car. Those are silly moments, and they don’t need to be part of your online portfolio. They’re not embarrassing, either, although they can be, and that’s okay. The stakes are so much lower with a photo or video that’ll only exist for nine seconds; a lousy snap earns a shrug rather than judgment, and a great one is a fantastic surprise. Your future employer will read your tweets, but won’t see your snaps…

It sounds ridiculous, but there is something lovely and beautiful and very today about visual art that exists only on a four-inch cellphone screen for a few seconds. Internet culture is overwhelming; there’s too much of everything. Snapchat is the one service that you wish there was more of. You wish you could save it. You wish you could keep experiencing it. When was the last time you went to a concert and weren’t surrounded by people taking photos and videos on their phones? Non-recorded experience is a hard thing to find in 2013.”

Snapchat app

Snapchat app

What’s the big deal? Why should I care?

Like the cyclical evaporation and precipitation of water across an expansive ocean, it’s entirely possible that the cyclical creation and deletion of transient social data will have no significant impact upon the vast digital tides of big data. But even so, sociopolitically speaking, transient data might still address the majority of our privacy concerns.

On the other hand, like an extended drought that dries up a running river, it’s remotely possible that the extended deletion of transient social data will overwhelmingly outweigh its creation and slow down or dry up the surging digital streams of big data. But again, this slowing down or drying up of big data might not matter to us as long as our privacy concerns are addressed.

Snapchat companion apps

Snapchat companion apps

Interestingly enough, with a quick trip to the Google Play store, you can already find Snapchat companion apps that offer you the ability to capture and save your transient snaps, which defeats the unique purpose of Snapchat and clearly indicates that many folks still don’t get it. Of course, from this anti-transient perspective, there’s little danger of big data slowing down or drying up.

IBM Lotus Notes

IBM Lotus Notes

Nevertheless, as an IBMer, I’d love to see IBM pursue not only the number-crunching big-data solutions in business analytics and private clouds, but also the socially-responsible possibilities of exploring transient-data solutions through its IBM Connections social media platform and IBM Notes social business applications. But even as an IBMer, I’d still love to see other companies like Google, Twitter, or Instagram, even Apple or Samsung, seriously consider the opportunities and benefits of tackling transient data as an alternative to big data.

IBM social business benefits

IBM social business benefits

Taken further, especially in this post-Snowden era, such transient-data solutions would transcend the simple automatic deletion of trashed messages, and allow us to define expiration dates or time limits for received messages, sent messages, photo attachments, wiki pages, blog entries, forum topics, community files, video uploads, comments and statuses, even tags and likes, and any other type of social media that we’ve received, read, written, posted, or shared.

In such a future, if you can imagine it, transient social data might feel as natural as a friendly chat in the park — temporary and forgotten — without the fear of being recorded or knowing that recordings will last for eternity.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.