Wow, I can't remember the last time I ran a shell directly on a prod machine. The very thought feels so dirty now.

When you inherit your environment and are whipping it into shape, shit happens.

Whether it's shell literally, or "shell" as in, the window from which you ultimately perform shell-like things, there's still a place you perform these tasks and knowing when you are in simulation or actual is an important state to track.

How do you know what's really going on on those machines and in their respective environments assuming they are geographically distinct?

Any advice/tutorials on how to set this up? Ideally I'd like just that ssh session to change the text to red for prod boxes but then go back to my normal iTerm theme after exit...

In the .bashrc/.zshrc of the remote machine add this:

    echo '\033]11;#440000\007'

On the local machine add this to reset the terminal color:

    ssh () {/usr/bin/ssh "$@"; echo '\033]11;#000000\007'}

You can muck around with your bashrc, if you use that shell.

Hmm good idea. I haven't had any issue with it but it's trivial to change background colors. Probably not a bad idea. Thanks for the tip!