Is message passing that you're referring to different than the Ruby concept of messages? Objects respond to messages in Ruby, and it is encouraged to think of it that way rather than calling functions.
Is that actually the case in practice? I haven't used Ruby enough to know, but my experience so far is that most of the time the approach is still mostly just plain method calling in practice. I vaguely recall Rails moving away from some usage of method_missing (which I'd argue is one explicit example of message passing vs method calling).
He said "[...] it is encouraged to think of it that way rather than calling functions."
This is what Sandi Metz talks about when she speaks about message/responsibility centric design vs data-centric design in OOP. The conceptual difference is said to lead to different designs. You don't start out your design by thinking about what data you hold but you think about which responsibilities/roles you have and create objects around those by thinking how they might talk to each other to reach their goals.
(I'm currently reading POODR by Sandi Metz, so this is still new to me, if I got anything wrong there then please feel free to correct me)
I personally don't know how it is implemented under the hood, but it is important to understand as a Ruby programmer that methods are not just functions. In fact functions are not really first class entities in Ruby. Instead you have a mishmash of procs, blocks, instance methods and class methods and they are all different to a certain degree. It's one of the big frustrations of programming in Ruby sometimes.
But when it comes down to it what really is message passing? You have a reference to an object, a method name and a collection of parameters. When you dispatch the message you look for the method on the object and run the function passing in the parameters. Does it matter if you optimise the situation by placing the parameters on a stack and using a dictionary with a reference to the method? Or if you optimise by doing static analysis to look up the method and replace the dispatch with a direct call to the method? Or if you optimise further by inlining the code altogether?
The point is not how it is implemented but how the programmer thinks of it. Object Oriented programming is a thing humans do, not a thing computers do. Object Oriented Programming Languages have features that make it more convenient to do Object Oriented programming. Whether the code is OO or not is only up to the programmer, not the compiler/interpreter.
Having said all that, is Ruby my first choice for (what I understand to be) Alan Kay's model of OOP? No. But then I really don't know what my first choice would be. It is possible that the language hasn't been written yet ;-)
