I have a Linux box with two 3090s and it's been great for running Qwen3.6 27b. I lowered the power on each card down to 250w, and then built a small ducting/fan system to vent the waste heat outside. The machine is pretty much silent, and I'm still getting 110 tokens per second out of it for coding tasks.
How useful is the second 3090 in this setup? I run the 5-bit quantized model on a single 3090. Does the second 3090 allow you to use the full precision model instead or a less aggressive quantization by splitting the layers? What about running the 35B model instead?
More memory means less aggressive quantization, more concurrent requests, and larger context windows. I also get a boost in tokens per second (not double, about 1.5x compared to a single GPU).
The 35B model is an MoE (mixture of experts), which uses only a subset of parameters at a time. The 27b one is slower but has way better performance.
https://github.com/tedivm/qwen36-27b-docker