The transformer architecture is fundamentally unsuitable for local inference, while being efficient at scale. It's a fun experiment to try, but that's about it.