Having the client/server split has made this a lot easier to reason about. I store client state on the client and server state stays on the server.
When I said I don't use sessions, I should have specified I don't use them for application state (or roles and stuff).