I have almost the same problem. With Poetry, I managed to work around it using this technique, involving a combination of groups and extras: https://github.com/lucaspar/poetry-torch

It's a hacky workaround, but it seems to work so far. It would be much nicer to see this solved in a better way, though!