The proposed negative impact is that votes from "bad" users would be counted less. So it can't be that your best option is to not participate: if you stop voting, you already punish yourself worse than this algorithm would.