I've reached the same conclusion for our suite of UI tests (which are based on Cucumber + Capybara + SitePrism) - Phantom has random issues that don't occur with any other browser/driver (e.g. elements not being found even if you add a long wait - our app is rich client side Angular so I suspect some JS issue is happening) and it's too hard to debug what is going on compared to Firefox + Xvfb (where you can just redirect the display to a real X display and see what is going on).
Yes, with selenium you can actually watch the entire test happen and see where things go wrong. Very easy to debug and understand but PhantomJS makes it really tough to find out why something failed because its headless. Sometimes I would try to take screenshots to figure out what was going on and even that would fail at times.
Maybe you'd be able to run more instances with PhantomJS compared to running Xvfb and a real browser because of lower memory consumption. I don't know if this is true.
