Trolley problem. What's worse than all of humanity dead from ASI?
Sam Altman: "Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could."
most of humanity dead because some morons decided to try to kill everyone just in case the server farm evolved the inclination and the capacity to try to kill everyone...
Trolley problem: on the left track is a class of schoolchildren. On the right track is a a computer which a Harry Potter fanfic author argues has magic powers and the marketing department of Apple kind of agrees. Do you let Eliezer persuade you to kill the kids?
no one dead because we're willing to use our means to enforce a restriction on a scenario that is likely on our trajectory to kill everyone. We also use serious threats to prevent nuclear proliferation that endangers everyone
Serious threats which have completely failed to persuade North Korea et al to stop pursuing nukes. The closest we've actually come to a nuclear exchange was "using serious threats" during the Cuban missile crisis, and the US is currently losing a war nominally to prevent nuclear proliferation to a state that fortunately doesn't actually have the capacity to retaliate except at very short range. You might remember a few other nominally intelligence-lead claims about weapons of mass destruction too not amounting to very much either...
And in this case we're not talking about bombing sites if they appear to be used for uranium enrichment which isn't exactly an everyday need, we're talking about bombing them if they appear to be optimised to contain lots of computers. You don't even have to be an AI sceptic to imagine a policy of threatening to launch missile strikes on Chinese businesses if they build out capacity in a way distant observers don't like might be more likely to result in millions of deaths than the whatever runs on the Chinese business' computers...
Sam Altman: "Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could."