re live data. Processing that much live data without changes breaking the whole thing is a Hard problem. Very impressive work on their part. I'll give them that.
re storage platform. I'd have to look into it to Haystack or F4 to see if they were truly unique or incremental improvements on existing stuff. Cold Storage, though, I *did read up on. The work and decisions that went into that were brilliant. Every little detail and optimization they could. Plus, although minor precedents existed, I don't think I saw anyone else thinking or working in the direction they worked. Truly innovative.
re datacenter. I count most of that as more incremental. I've looked at their publications on their architecture. They might strip unnecessary stuff out of a blade, put in a RAM sled for a cache, use a simpler protocol than TCP/IP for internal comms, and so on. The kind of stuff everyone in whole datacenter (and supercomputer) industry does. Most of it is obvious & has plenty precedent. Now, if you know a few specific tech they used that were extremely clever (little to no precedent) please share. Example would be whoever started installing servers into shipping containers for mass production then easy installation... that was brilliant stuff. Was either Facebook or Microsoft.
So, you've given two good examples of Hard problems Facebook solved. Brilliant thinking went into solving them. No doubt a company working some miracles on a regular basis.
I built Haystack with two other engineers and then founded the Everstore team which works on Haystack/F4/Cold Storage. I moved off of managing Everstore to focus on Traffic in fall of 2011, so F4 and Cold Storage happened after my time, but I worked on Haystack and the storage management layer above it for five years.
There were a few things that were interesting or novel about Haystack. CDNs were only giving us a 80% hit rate on photos, and even back in 2007 when we first starting working on Haystack, that was an immense number of misses to serve from the origin. That heavily informed the design goals of Haystack, which were to have a very I/O efficient userland filesystem that used direct I/O to squeeze as much iops from the underlying drives as possible. We also wanted zero indirect I/O, so we used an efficient in-memory index so that all seeks on disk were to service production reads versus indirect index blocks or other metadata. Lastly, Haystack is very very simple. We made it as simple as we could and eschewed anything that looked too clever. I actually think its simplicity is one of the things that made it Hard. It would have been much easier to design something more complex.
If you have any questions about the Haystack paper after reading it, I'd be happy to answer what I can. Just send me an email.
Regarding the datacenters, you're kind of simplifying it, but that's okay. There's a lot more orchestration, automation, and integration, and I wish I could give you a tour of one of our OCP datacenters so you can see it firsthand. There's plenty of things I wish I could talk about on the datacenter front that I feel are truly amazing and without precedent, but I can't until they are public. There's an excellent chance that they will become public and likely even part of OCP in the future though. I'm really proud of how much of Facebook's datacenter architecture has been shared with the world for free. It gives me warm fuzzies all the time.
Anyways, thanks for the fun thread, and let me know if you ever want to talk storage sometime.
The paper was straightforward on what mattered. I liked that your team did their homework on prior efforts, used object-based storage (I pushed it in late 90's), used most proven filesystem (one I'm using it now), kept data structures fairly simple for modern stuff, and wisely used compaction. All good stuff.
I actually do have ideas on where to go from there and that's general. I'd probably charge Facebook for the info, though, given the increase in performance and savings they might generate. I might contact you in the future on that.
If true, that would be an epic burn of my comment. Yet, the only person's name that kept popping up has a LinkedIn that doesn't mention Google. He worked at Microsoft, Amazon, etc. Might not be the main inventor, though. Who are you referring to?
re storage platform. I'd have to look into it to Haystack or F4 to see if they were truly unique or incremental improvements on existing stuff. Cold Storage, though, I *did read up on. The work and decisions that went into that were brilliant. Every little detail and optimization they could. Plus, although minor precedents existed, I don't think I saw anyone else thinking or working in the direction they worked. Truly innovative.
re datacenter. I count most of that as more incremental. I've looked at their publications on their architecture. They might strip unnecessary stuff out of a blade, put in a RAM sled for a cache, use a simpler protocol than TCP/IP for internal comms, and so on. The kind of stuff everyone in whole datacenter (and supercomputer) industry does. Most of it is obvious & has plenty precedent. Now, if you know a few specific tech they used that were extremely clever (little to no precedent) please share. Example would be whoever started installing servers into shipping containers for mass production then easy installation... that was brilliant stuff. Was either Facebook or Microsoft.
So, you've given two good examples of Hard problems Facebook solved. Brilliant thinking went into solving them. No doubt a company working some miracles on a regular basis.