Seems to me that pluggable USB is no different from installable software. So why not just the same sort of sandbox + permissions model that iOS/Android apps live in?
Qubes OS is evolving in this direction for all peripheral devices (even some of the ones on the motherboard such as NICs). USB is taking more time, but controllers will in future automatically be sandboxed in Service VMs. For now, a Qubes user can quickly create their own USBVM that uses the system IOMMU (if present) to reassign USB controllers to that VM; Likewise, you can also assign particular USB controllers to individual VMs. These VMs can be selected for autostart at boot time with a couple mouse clicks.
A more granular and hence flexible way to sandbox and handle USB devices will be coming to Qubes by way of Xen's PV USB feature.
Discussed in other thread: https://news.ycombinator.com/item?id=8115002