It's not really racist to say something like that especially when there is strong supporting evidence.

Racism is where you advocate treating somebody differently because of these differences. So it would be racist to say "White men are too slow therefor should be banned from taking part in the olympics"