I'm pretty sure it's a bad idea because each time you connect and disconnect tools, you are hitting a full cache miss on the full context, that will be probably more expensive than having these tools in place from the very beginning.

Does anyone know what the impact on prompt caching is?

I like the idea of dynamic tool/skill sets, but wonder if it needs to be part of the new session workflow where you would choose a group to activate rather than this dynamic.