I sympathize with your position, but the problem is that you're forcing them to break the HTTP semantics (since a GET request - clicking a link - should be a safe action[1], without side effects like unsubscribing people), which then prevents anyone else from relying on those behaviors. This is why products like Web Accelerators died, and current prefetching features in browsers must only load URLs that are specifically marked as safe.

Personally, I think if you voluntarily subscribed, it's not too much of an effort to click a button after the link to confirm the subscription. If you didn't, then you should mark them as spam regardless of their unsubscription link.

[1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.1...