The amount of digital data we are creating every day is mind-boggling. But the photos we take, tweets we send, places we check into and other digital detritus is often siloed into proprietary formats from various companies stored on disparate sites across the web.
Camlistore, which stands for Content-Addressable Multi-Layer Indexed Storage, is an open-source project from Livejournal creator and current Googler Brad Fitzpatrick that aims to store your digital life in an easy-to-search, privately controlled environment. Fitzpatrick spoke about the project, which started as his 20 percent time project back in 2011, last night at Google’s Seattle office.
“I’m very concerned about making sure I have all my stuff in 50, 60 years or however long I live,” said Fitzpatrick, who created the LiveJournal blogging service as a student at the University of Washington starting in 1999.
As sites like Justin-TV and Lavabit die off and the data they store disappears, Fitzpatrick wanted a way to store that data on his own securely using multiple backups. But he also wanted to make that data readable as technology changes, letting you read your data even after coding languages die out.
“I built it for me,” Fitzpatrick said. “This is how I want to manage my own data and this is what I want in 20, 30, 40 years. I don’t know what else will survive, so at least I can trust something.”
But he knew that building a project on his own wouldn’t lead to something sustainable, so he made it open source to ensure that it continues to be developed.
“I want to get to a point where I want to use some new service … and it turns out somebody else already wrote an importer for it,” he said.
The importers may be the biggest barrier for Camlistore mass adoption right now. While Twitter and Foursquare imports are built in, many other data sources don’t automatically get fed into Camlistore. Fitzpatrick hopes one day users will be able to get a full set of data about anything they do online, from the number of likes they get on a picture of their baby to the email they sent to their boss.
However, storing all that data may seem like a huge investment. All those photos and videos take up huge amounts of space, and storing multiple copies may be hard on the wallet.
But Fitzpatrick warns against getting rid of things too soon. He was looking at photos he had taken with an early digital camera and noticed that he had used the smallest file size possible to save on storage.
“I’m so pissed at myself because I didn’t have the foresight in high school that storage prices were dropping so much,” he said. “I’m never making that mistake again. I keep everything in the highest resolution possible because I know in a year or two I just won’t care.”
Camlistore is about more than just data storage, though. Fitzpatrick noted last night that it also lets you use data from Twitter and Swarm (nee Foursquare) so you can keep a record of where you were when you said things. You can even import data from apps like Runtracker or even Google’s location history to keep track of where you’ve been.
Then, you can combine that data with photos you took on your DSLR to match up timestamps and figure out where photos were taken. Since all that data is stored on Camlistore in a standard format, it’s simple to combine data for a better picture of your life at any point in time.
Fitzpatrick built the project in Go, the open-source programming language created at Google. His work on the project eventually led him to joining the Go team at Google. However, he doesn’t see Camlistore as something from Google.
“I kind of don’t want it to be a Google product,” he said. “I think people have enough fear of big companies and data centralization that I think it’s kind of better a bit on the side. … Camlistore is naturally paranoid.”
Camlistore is currently available for anyone to use, but it’s still a little rough around the edges with its 0.9 release.
You can watch Fitzpatrick’s full talk, recorded as a Google Hangout, below.