Ok I just had another idea, so thought to write it here, in case PG gives it a look. How about showing votes after you took an action on a particular post. Say you upvoted a post, or replied to it, then you are eligible to seeing the points that it has.

This helps because even if in hindsight, I would know if the comment was the general consensus or a popular one or not?