Because usually you trust the x server.

This is saying that most client libraries are exploitable by a malicious x server.

A vulnerability in the x server itself, allowing a client to take control of it, would be rather more worrying to most people.

Yes, but how likely it is for your X server to get compromised? How often do you connect to your machine using an untrusted X server?

This is like saying your car can be stolen by anyone who has its key. Yes, it most likely can. Or that your house is vulnerable to anyone who's inside it.

This is likely why it's not been noticed before, I'd imagine the practical consequences of this are sort of moot: If the X Server is untrustworthy, you've likely got bigger problems.

I'll admit, my first thought was X11 Forwarding, as I understand you could have a bogus X server to cause junk to happen on clients on a remote server with X11 Forwarding enabled, which you'd already need credentials for. Again, if this was viable, the attacker having SSH credentials is a bigger problem.

I can't see any way for this to be really damaging in the wild, but I'm not a security guy so I could be totally off the mark.

Well, once your X server is compromised, you're basically In The Matrix.

Even if you fix all the buffer overflow and related bugs discussed here, connecting to a potentially lying X server is still a world of pain.