People in similar cultures and societies tend to roughly align, but its still leaky within a society and can be very misaligned across different societies.
The problem of alignment with AI is that there's potentially a power imbalance.
When a small number of people in a society are misaligned they can be dealt with. Say there's a murder for example, and that's not okay in their culture, those around may banish them, imprison them, etc.
That doesn't work with something that can cause drastically more damage than a single human can. If a misaligned AI hacks NORAD and launches our nukes, it only took one misalignment issue and there's no dealing with that after the fact.
