Depends on who "we" are, what you're talking about is a thing for inference providers doing batched inference and similar stuff. If you run one inference requests locally, you can actually step-by-step debug LLM output, just there is a ton of steps. But there is nothing "inherently random" or non-deterministic involved here, just optimization strategies for the large inference servers that makes it "impossible".

> we cannot step debug an llm's output to find out what happened

We absolutely can with mechanistic interpretability & companies like Anthropic, OpenAI, Meta, and Google do precisely this do debug their models.

I'll give you that it's not wrapped up in nice product UX, but these are market choices first and technical limitations second.