Long ago, some people said computers would never beat humans at chess, and they were wrong. A human brain contains no magical components. It runs on physics. Whatever it does, can in principle be done by some other machine that runs on physics. I won't be surprised if someone finds a general way to replace metamathematical insight with clever search-and-pruning over a large but simply structured space, like it happened with chess.
Also, the limits of computation have been given an upper bound. If it's not computable by a Turing machine with a finite memory tape (or anything else that is mappable to a Turing machine with a finite memory tape), it's not computable by a physical computer.
Steve Yegge started exploring the metaphysics of computer programs before he stopped blogging; it makes for interesting reading.
There's a reason why my argument never mentioned Turing computability :-) If a human brain uses quantum effects to get cognitive bonuses, you can build a quantum computer that does same. If the brain uses super-duper unified field theory, you can build a machine to take advantage of that, and so on. The only assumption I make about the human brain is that it doesn't run on magic juju that's available exclusively to humans by the orders of Cthulhu, and I think that assumption is quite warranted.
(As far as I know, neurons can be approximated well enough by classical computers and you don't even need to go quantum, but that's another story.)
Am not going to argue whether a machine can "really" be alive, "really" be self-aware. Is a virus self-aware? Nyet! How about oyster? I doubt it. A cat? Almost certainly. A human? Don't know about you tovarishch, but I am. Somewhere along evolutionary chain from macromolecule to human brain self-awareness crept in. Psychologists assert it happens automatically whenever a brain acquires certain very high number of associational paths. Can't see it matters whether paths are protein or platinum.
> If it's not computable by a Turing machine ... it's not computable by a physical computer.
By a physical Turing equivalent computer. True.
The breakthrough will come when we'll be able to overcome the damnation of step-by-step-ness (which is the primitive expression of causality).
