If have just been writing a lot of C code to support unicode in a new project. It reads UTF-8 converts to UTF-32 does lots of clever stuff and spits UTF-8 out once again.
Here's my take: UTF-8 should, rightly, be the only interchangeable text format of choice for the right-minded individual. The other UTFs should be internal formats used in-memory or on disk cache / database / etc
Why? Simple: UTF-8 rocks!
It's a beautifully designed, backwardly compatible, nifty piece of back-of-a-napkin genius. Simple to code and decode (once you understand it), simple to check with a regular expression (once you realise it is essentially just a token with a set number of chars), and simple to add the wealth of the world's characters to your app with a reasonable amount of code. Plus, it's compact in the way Huffman coding is compact (at least from a western perspective).
Also, no-one should ever be using (char ❄) in the 21st century unless it is to temporarily hold a UTF-8 string before converting to wchar.
Why? Because you almost certainly don't have a (char ❄), you have a UTF-8 sequence (^^^see above). Sadly this makes your memory mapped files slightly redundant. But don't fret, this is the future: convert them to 32-bits and release them. Be happy that you can now treat any character sequence, in common usage, in the whole of humanity like an array.
As for UTF-16, why bother? It's neither compact, clever, nor big enough to hold every character on the internet. 💩 needs more than 16 bits and everyone, now, needs to support a poop with eyes.
tl;dr: Share UTF-8 promiscuously, keep UTF-32 for private moments. Don't dally with UTF-16, she's an old tease and can't handle poop.
UTF-16 will handle poop just fine. UTF-16 handles higher characters in a manner just like UTF-8. Pile-of-poo is encoded as D83D DCA9 in UTF-16, The same size as it is in UTF-8.
There may be a distinct size advantage for some asian cultures to using UTF-16 instead of UTF-8 as it will allow for encoding more of the glyphs without having to add more overhead bits. How much this saves in reality I'm not sure.
I wouldn't use UTF-16 unless having to work with a legacy system. A lot of software claiming to handle UTF-16 is broken and really only works with UCS-2. You have to worry about endianess and so on.
If you develop an application for the international market you should probably go with UTF-8 as well. Maybe if you develop only for the Asian market it's a bit different. But in my experience significant amounts of text usually come in some form of data or markup format (HTML, XML, JSON, etc.) and usually those markup formats are defined in the ASCII subset. So UTF-8 still wins. Just take a random page from the Japanese Wikipedia and encode it in UTF-8 and in UTF-16. You'll see that UTF-8 almost always wins.
The advantage of moving to one standard, plus the backwards compatibility for most of the internet will outweigh any small size advantage that UTF-16 will have on real file sizes for some countries / cultures.
A saving of ⅓ on a text file will barely be noticed in a world seemingly governed by Moore's law in nearly every future metric.
(also my bad for using UTF-16 where I meant 16-bit unicode character arrays :))
Here's my take: UTF-8 should, rightly, be the only interchangeable text format of choice for the right-minded individual. The other UTFs should be internal formats used in-memory or on disk cache / database / etc
Why? Simple: UTF-8 rocks!
It's a beautifully designed, backwardly compatible, nifty piece of back-of-a-napkin genius. Simple to code and decode (once you understand it), simple to check with a regular expression (once you realise it is essentially just a token with a set number of chars), and simple to add the wealth of the world's characters to your app with a reasonable amount of code. Plus, it's compact in the way Huffman coding is compact (at least from a western perspective).
Also, no-one should ever be using (char ❄) in the 21st century unless it is to temporarily hold a UTF-8 string before converting to wchar.
Why? Because you almost certainly don't have a (char ❄), you have a UTF-8 sequence (^^^see above). Sadly this makes your memory mapped files slightly redundant. But don't fret, this is the future: convert them to 32-bits and release them. Be happy that you can now treat any character sequence, in common usage, in the whole of humanity like an array.
As for UTF-16, why bother? It's neither compact, clever, nor big enough to hold every character on the internet. 💩 needs more than 16 bits and everyone, now, needs to support a poop with eyes.
tl;dr: Share UTF-8 promiscuously, keep UTF-32 for private moments. Don't dally with UTF-16, she's an old tease and can't handle poop.
note: ❄ = asterisk :)