Thanks, that's a much better example of what I was trying to explain. Again, I'm projecting humanity onto a machine, but it's not a big stretch to imagine that the interaction of attention heads over high-dimensional vectors produces "decisions" that can't be articulated in the moment, only described after the fact in text form.
There's enough behavioural research to show that it is the case. For ex:
https://www.cambridge.org/core/journals/behavioral-and-brain...