Yes, the overflow either segfaults your program, or doesn't. It's not undefined. Computers don't perform random operations. Just say "it will produce buffer overflow" - that is absolutely a defined operation: the CPU will execute a load from an address that hasn't been defined at that place in code and will either contain leftover data, or refer to an unmapped page, triggering a segfault. This isn't quantum mechanics where the underlying physical reality is inherently undefined until you measure it (caveat: as far as we know, at least), this is just insufficient knowledge of hidden variables. I understand that the compiler might emit instructions under various assumptions, which is what we mean by "undefined behaviour" when those assumptions don't hold, but once there's actual machine code running on the actual CPU, everything is absolutely perfectly defined.
All the solutions and problems in computing stem from the fact that we're programming completely deterministic Turing machines that do exactly what they're told.
That's a very narrow view of the problem. There are so many moving parts in a program that overwriting an arbitrary chunk of stack is effectively random. Do you object to the term "random number generator" too?
Just saying "or doesn't" is not a useful description. The vast majority of horrible broken behaviors fit inside of that bucket, where it doesn't segfault but then goes on to do the wrong thing at an unexpected place.
And because we're using an optimizing compiler, you can't give a simple description like "overflows a buffer". When the compiler assumes your code is correct, it might output instructions that behave in 'impossible' ways when fed invalid data. For example an if/else that takes neither branch, or both branches, or code that verifies a number has a certain value yet outputs a completely different value. You can only make assertions about what a particular compile will do, and that's obsolete information immediately.
All the solutions and problems in computing stem from the fact that we're programming completely deterministic Turing machines that do exactly what they're told.