One reason is that if they are storing the password in a recoverable format, it tells you something about how much you can trust the security of that site in other ways. Even if you're using a different password, you may be much more hesitant about what other information you are willing to put into a site that doesn't take security seriously.