The information I've seen about Sibyl (the Google ML system, not the genomics package (http://sybil.sourceforge.net/documentation.html)) says it is basically doing logistic regression using a parallel algorithm (Collins, Schapire, Singer) with a transpose on each iteration. Without knowing more about the problem sizes and data sparsity/irregularity, I expect the transpose to be a significant expense. I'd be happy to read more if you have access to further technical information, but it's not clear how this comment relates to your previous question about CG and deep learning. As it relates to HPGMG, I think my previous response covers the important performance dimensions. I'd be happy to discuss further over email.

Both logistic regression and deep learning are basically just big conjugate gradient minimizers.

What I meant by asynchronous is that not all terms in a gradient are required to be summed in the same step.

The transpose step in Sibyl is implemented in the Shuffle and Reduce phases. The filesystem is used to hold the temporary data. Nevertheless even for large systems, very few steps are required, and step times are reasonable, even compared to modern supercomputers. This is a tribute primarily to the design of sibyl and the implementation of MapReduce at Google.

This is all explained in online versions of the Sibyl presentation. I really wish more people from DOE who write modern solvers would pay attention to this stuff.