The singularity (as conceived pre-Kurzweil) is the horizon beyond which we can't make any reasonable predictions about the future. Recursively self-improving AI is merely one potential path to that. But GP is right that the meme that the singularity is ridiculous tends to go along with the meme that intelligence is ineffable and can't be reduced to an algorithm in a computer.
I think what most people mean is, "even if you don't believe that anything other than squishy brains can ever recursively do what the brain does".