People in similar cultures and societies tend to roughly align, but its still leaky within a society and can be very misaligned across different societies.
The problem of alignment with AI is that there's potentially a power imbalance.
When a small number of people in a society are misaligned they can be dealt with. Say there's a murder for example, and that's not okay in their culture, those around may banish them, imprison them, etc.
That doesn't work with something that can cause drastically more damage than a single human can. If a misaligned AI hacks NORAD and launches our nukes, it only took one misalignment issue and there's no dealing with that after the fact.
Well, we don't. Just look at all the wars going on right now. In light of that, does it seem like a good idea to introduce a bunch of even less aligned agents into the world?
"Grey goo" nanobots are another example of artificial agents that aren't aligned with humanity, that we should probably try to avoid creating.
Humans already come out aligned basically the vast majority of the time. Our whole system is run on a spectacular amount of just “I trust you not to screw me over.”
I couldn’t disagree more strongly. Biology implies competition, both within and outside one’s species. Factionalism is a direct consequence of our evolution, so to say that it is not aligned with humanity’s interests is incoherent. You might dislike it, but in a real sense factionalism is part of us.
There is a difference between humans and humanity.
My argument isn't whethwr factionalism is a part of us. My argument is that different faction often, or even just sometimes, aren't aligned with each other.
Alignment isn't inherent in the system. Sometimes groups are aligned, sometime they aren't. When a single misalignment can cause catastrophic damage you have a problem.
How do we align humans?