I don't know how good Heartbeat is these days. I'm just going based on our experience back then and GitHub's stated experience now.

We (in a new place) use our own fail over code now, but we're running databases instead of file servers -- and you usually don't want to STONITH a database.

For file servers we use Gluster, and it works great for us (I'm sure there are reasons GitHub are doing active-passive DRBD) and you don't need STONITH since it's shared-nothing.

There's still plenty of things to keep in mind with Gluster, though. If you use a replicated setup for redundancy, and you get a split network, you can very well end up with an inconsistent state because you can get into situations where you write to different replicas from different clients and replication doesn't succeed.

Gluster will just throw it's hands up during self-heal if that happens, and you'll need to manually resolve it.

Your filesystem structure won't break, but your files certainly can.

I'll just point out that experience with heartbeat from 2001 is rather outdated.

That would be similar to comparing 2.4 kernels problems to the most recent 3.7 kernel. Likely not an overly useful anecdote.

For the record we use heartbeat at work with no issues such as this.

While I agree with you, I think the anecdote is still useful because it shows that the problems of the present are really problems of the past too.

Software quality improves, sure, but you can still learn from the past.

I'd say its more an implementation issue than software. And that this is a solved issue even 10+ years ago with heartbeat.

Our heartbeat links are segregated from the public network with separate network cards/switches. So the specific issue github hit here isn't what we would have encountered. We do have quad nic cards for a reason in the systems we run, this issue github hit is one of the various reasons you don't run heartbeat over the public topology. It will bite you in the ass no matter what cluster software you are running.

Unless you also have a disk heartbeat over shared fibre/scsi, or maybe serial but same difference. Depending upon the public network though is a lost cause.