There's a difference between "building from source" and "obtaining software from a trusted party". Your suggestion (trusting the packagers) implies the latter and is almost irrelevant to the former. If one trusts a team to catch possible issues, one may trust the binary this team builds as well.