Are you bottlenecking on high iowait? or something else?
just one random bit to try... Obviously, I have no insight into your system and I'm not saying I know more than you or anything, but I've been seeing more situations lately where I had massive latency but reasonable throughput and the disks mostly looked okay wrt. smart, and I mostly just wanted to write about it:
(this is a new server built out of older disks that appears to have the problem. It's not so bad that I get significant iowait when idle, but if you try to do anything, you are in a world of hurt.)
Check out the await value. re-do the same command with a '1' after /dev/sdd and it will repeat every second. If sdd consistently has a much worse await, it is what is killing your RAID. Drop the drive from the raid. If performance is better, replace the drive. If performance is worse (and with raid z2, it should be worse if you killed the drive) the drive was fine.
(Of course you want to do the usual check with smart and the like before this)
The interesting part of this failure mode that I have seen is that /throughput/ isn't that much worse than healthy. You get reasonable speeds on your dd tests. but latency makes the whole thing unusable.
just one random bit to try... Obviously, I have no insight into your system and I'm not saying I know more than you or anything, but I've been seeing more situations lately where I had massive latency but reasonable throughput and the disks mostly looked okay wrt. smart, and I mostly just wanted to write about it:
[lsc@mcgrigor ~]$ sudo iostat -x /dev/sda /dev/sdb /dev/sdc /dev/sdd Linux 2.6.18-371.3.1.el5xen (mcgrigor.prgmr.com) 01/16/2014
avg-cpu: %user %nice %system %iowait %steal %idle
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %utilsda 0.70 75.11 35.66 1.38 4568.62 611.67 139.85 0.36 9.61 0.53 1.95
sdb 0.46 75.10 35.62 1.39 4566.77 611.67 139.89 0.22 5.89 0.45 1.66
sdc 0.80 75.14 35.63 1.35 4569.63 611.63 140.10 0.64 17.18 0.57 2.10
sdd 0.46 75.09 35.62 1.40 4566.60 611.63 139.87 0.13 3.47 0.40 1.49
(this is a new server built out of older disks that appears to have the problem. It's not so bad that I get significant iowait when idle, but if you try to do anything, you are in a world of hurt.)
Check out the await value. re-do the same command with a '1' after /dev/sdd and it will repeat every second. If sdd consistently has a much worse await, it is what is killing your RAID. Drop the drive from the raid. If performance is better, replace the drive. If performance is worse (and with raid z2, it should be worse if you killed the drive) the drive was fine.
(Of course you want to do the usual check with smart and the like before this)
The interesting part of this failure mode that I have seen is that /throughput/ isn't that much worse than healthy. You get reasonable speeds on your dd tests. but latency makes the whole thing unusable.