> Isn't sending data "somewhere" pretty common. Unless this is middleware in python ecosystem, data is going to go a logger, database, console, web page, file. Am I misunderstanding it? It seems you are dismissing it as something one doesn't need to worry about, because it is not done much...
The point is not to dismiss the encoding. The point is that the OP is confusing the character representation (Unicode) and the I/O encoding (utf-8). Yes, sending data somewhere is common, and when you do that, you are taking your Unicode string and encoding it using whatever encoding is appropriate (usually utf-8).
As for whether you should worry about what the encoding is, modern systems (including Python 3, but not Python 2!) use utf-8 everywhere by default, and save you the headache of specifying the encoding or passing it around. One important exception is a Linux process which hasn't had the locale variables set (normally this is done by the PAM environment module, but in a number of situations all environment variables might be stripped from the process, leaving it with what's known as a "POSIX C" locale, which is kind of a broken anachronism). Generally that leaves the system (not just Python) open to all kinds of brokenness, so keep the locale set by not stripping LANG and LC_ALL from your environment.
> Completely ignorant about it, but how would Python 3 know when reading from stdin what the encoding is? Or what about when reading sys.argv?
The output of locale.getpreferredencoding() is used for environment variables, command line arguments, and I/O streams. More generally, you can access and change the encoding of any character stream using its .encoding attribute (e.g. sys.stdin.encoding).
> Python guesses the system encoding using logic that looks at the locale environment variables
This is not an implementation detail, but means that when Python chooses a different encoding than you thought, your program crashes spectacularly (and not on start, but on some unicode operation further down the line).
The same program will run differently depending on who starts it. This should go on top on all Python 3 tutorials, since it's not obvious at all what happened, especially not for a beginner. Non-conformant environment variables and file names causes all sorts of weird problems.
I can recommend Armin Ronacher's unicode tutorials for Python 3. It is what saved my sanity when I first encountered it.
...your program crashes spectacularly (and not on start, but on some unicode operation further down the line).
ISTM that it could be a good idea to try all potentially-dangerous unicode-related operations immediately on startup. That might be a good idea for a package or even an addition to the stdlib.
The point is not to dismiss the encoding. The point is that the OP is confusing the character representation (Unicode) and the I/O encoding (utf-8). Yes, sending data somewhere is common, and when you do that, you are taking your Unicode string and encoding it using whatever encoding is appropriate (usually utf-8).
As for whether you should worry about what the encoding is, modern systems (including Python 3, but not Python 2!) use utf-8 everywhere by default, and save you the headache of specifying the encoding or passing it around. One important exception is a Linux process which hasn't had the locale variables set (normally this is done by the PAM environment module, but in a number of situations all environment variables might be stripped from the process, leaving it with what's known as a "POSIX C" locale, which is kind of a broken anachronism). Generally that leaves the system (not just Python) open to all kinds of brokenness, so keep the locale set by not stripping LANG and LC_ALL from your environment.
> Completely ignorant about it, but how would Python 3 know when reading from stdin what the encoding is? Or what about when reading sys.argv?
Great question! Python guesses the system encoding using logic that looks at the locale environment variables (most commonly LANG and LC_ALL) defined by POSIX (or on Windows, detects the console or uses ANSI). For more information, take a look at https://docs.python.org/3.4/library/locale.html#locale.getde... and https://docs.python.org/3.4/library/sys.html#sys.stdin. It's also possible to override the I/O encoding with the PYTHONIOENCODING environment variable.
The output of locale.getpreferredencoding() is used for environment variables, command line arguments, and I/O streams. More generally, you can access and change the encoding of any character stream using its .encoding attribute (e.g. sys.stdin.encoding).