are generated using GCP, AWS, and an NVidia DGX-1, all using exactly the capabilities any ordinary user has on those platforms. The K80 distributed training results are AWS.
There's also a very useful set of suggestions for how to tune TensorFlow for best performance both, and scripts that repeat the benchmarking results: https://www.tensorflow.org/performance/
I see that since my comment, Microsoft has updated the claims in the cited page. It's still not true that there are two versions, but I'm glad you're trying to provide more detail. I'd like to stick a big [citation needed] on the claim that the internal version is much faster.
At the time Mu Li did his performance analysis of MXnet vs Tensorflow, we hypothesized that gRPC overhead was one of the reasons that MXnet was showing better scaling numbers than TF. That turns out to not have been very correct - there were several things that the TF team identified that closed the scalability gap to a pretty narrow degree around the 1.0 release. I don't feel confident that gRPC is much of an impediment to scalability. (I'm also not saying that it isn't -- just that I don't think there's a lot of evidence one way or another).
I'd love it if the CNTK team or someone else were to publish high-quality, head-to-head scalability numbers using the best practices and scripts identified in the TensorFlow performance guide, and using the equivalent CNTK best practices. It benefits everyone when Microsoft and Google work hard to out-do each other. :) (And throw in MXNet as well, with Amazon's best guidance.)
Thanks for the clarification. gRPC is slow. We have in-house experiments showing on RDMA-capable networks an optimized implementation can achieve significant speed up over gRPC. And I bet Google's internal version is even faster.
MxNet has a highly efficient network stack that's open source; Caffe2 uses gloo, which is open source; CNTK primarily uses Open MPI, NCCL and soon NCCL2.0. I think it's fair that Google also open source the internal network stack because it is the key to scaling.
Most convolutional networks are not a stress test for scaling because the model size/computation ratio is too low. Use a speech model that has many fully connected layers, or VGG16/19, the communication cost will dominate, and that's when CNTK's 1-bit SGD and Block Momentum really shine.
Publish those results? It'd be very interesting to see. And, it sounds like you think there are benchmarks missing from the existing common set of things people are measuring -- what's a very specific network you'd like to see added to the mix? VGG16 doesn't fall into my radar of "modern and applicable" in the days of ResNet.
From the benchmarks available, and not knowing what your in-house experiments show, I don't believe that the "internal network stack" is key to scaling. The scalability numbers shown on tensorflow.org/performance are very reasonable: From 902 images/sec to 1783 (1.97x) going from 32->64 K80 GPUs on Amazon for Inception v3, and 565->981 (1.7x) for ResNet-512. I'd love to be proved wrong.
That 1.7x scaling on ResNet-512 would be a great point of comparison, for example. From my student Hyeontaek's results, I actually suspect that there are scheduling improvements that could make up some of that difference, not networking improvements.
As I'm sure you know, of course, and are just fishing for, the reason that code links against gRPC externally is because trying to extract Google's internal networking code from the full internal software codebase would be ridiculous. I think it's far more likely to see the other direction, with everything settling on gRPC -- gRPC is actually newer, and in general, more feature-ful, than Stubby: https://cloudplatform.googleblog.com/2016/08/gRPC-a-true-Int...
are generated using GCP, AWS, and an NVidia DGX-1, all using exactly the capabilities any ordinary user has on those platforms. The K80 distributed training results are AWS.
There's also a very useful set of suggestions for how to tune TensorFlow for best performance both, and scripts that repeat the benchmarking results: https://www.tensorflow.org/performance/
I see that since my comment, Microsoft has updated the claims in the cited page. It's still not true that there are two versions, but I'm glad you're trying to provide more detail. I'd like to stick a big [citation needed] on the claim that the internal version is much faster.
At the time Mu Li did his performance analysis of MXnet vs Tensorflow, we hypothesized that gRPC overhead was one of the reasons that MXnet was showing better scaling numbers than TF. That turns out to not have been very correct - there were several things that the TF team identified that closed the scalability gap to a pretty narrow degree around the 1.0 release. I don't feel confident that gRPC is much of an impediment to scalability. (I'm also not saying that it isn't -- just that I don't think there's a lot of evidence one way or another).
I'd love it if the CNTK team or someone else were to publish high-quality, head-to-head scalability numbers using the best practices and scripts identified in the TensorFlow performance guide, and using the equivalent CNTK best practices. It benefits everyone when Microsoft and Google work hard to out-do each other. :) (And throw in MXNet as well, with Amazon's best guidance.)