But as soon as you redistribute it without the source, the people who get that copy don't have the same rights. This is like the old saying: "your right to swing a fist ends at my jaw"; rights should not give you the power to take away others' rights.
I suppose it depends on whether you think there is an inherent right to access the source code of every software you use. The whole positive vs. negative rights issue comes into play big time.