I've been doing a lot of the encoding work on Rails 3. Curious what your issues are so I can be sure we get it right for Rails 3

The latest problem I hit seems to be related to logging. It goes away if I replace line 99 in buffered_logger.rb to read:

   @log.write(old_buffer.join.force_encoding(Encoding.default_external))

Without this ad-hoc "fix" (well it's not a fix, I was using it just to confirm the source of the bug) I'm seeing:

   Encoding::UndefinedConversionError: "\xC4" from ASCII-8BIT to UTF-8 

all over the place (every time the log is flushed).

p.s. just to clarify, I was testing an app that currently runs in production on ruby 1.8.7 + rails 2.3.5 with rails 2.3.6 and ruby 1.9.2-head. 30 or so tests fail and about 70 have errors. I'll be sure to report any new bugs I find.

I'd be interested to learn where the BINARY data is coming from in the first place. For reasons I outlined in a recent blog post force_encoding is almost never a reliable general-purpose solution

I run into the same problem when trying to submit any unicode data through ActiveRecord. I will try this fix and see if it helps.