That assumes AIs can enhance their intelligence given fixed computational resources. We don't actually know the shape of the curve for optimization power on the X-axis and necessary FLOPs on the Y-axis. If it's anything larger than linear, AIs will not self-improve in a "singularity" type fashion, but in fact hit a point of severely diminished returns relatively quickly. Judging by the computational complexity of machine-learning algorithms, it's actually entirely possible that curve is exponential (meaning that AGI is DEXPTIME-complete), in which case the only question is how far we are towards the diminishing-returns point.

This objection further assumes that the end-point after consuming all low-hanging fruit is a pre-singularity level AI. If a self-improving AI runs into algorithmic limits only after figuring out how to turn the solar system into computronium, that's still worth calling a "singularity".

Does the singularity concept actually require some sort of quickly growing or even asymptotic curve for intelligence improvement? I don't see why that would be a requirement. Even if AIs are only slightly more intelligent than humans, it seems like many of the effects associated with the singularity could occur. If AIs were more intelligent than humans to the same extent humans are more intelligent than the next most intelligent species on Earth, surely we would consider that to resemble the proposed singularity.

But that assumes running AI on some architecture where FLOPs have any real meaning. If we grow an AI that's externally accessible, is that less of an achievement than simulating the same on a general purpose computer? Still assuming it could achieve singularity that is...

Except that we already have examples of semi-autonomous non-AGI software that successfully acquires additional computational resources via parasitism.

Further, beyond the task of software improvement, the task of improving hardware is itself already heavily software-mediated when performed by humans, and will likely be a prime candidate for applying pre-AGI AI.

There are likely other ways to attack the problem, such as not focusing on hardware improvements per-se, but instead on the cycle time for getting the improvements deployed (eg. somehow eliminating or cutting the size/cost/construction time of new Fabs).

This idea of singularity relies on the assumption that intelligence scales easily. If we have a machine that is more intelligent than the average human, or even more intelligent than the most intelligent human. Will it be more intelligent than a group of intelligent people? How many? If you have an AI running on a supercomputer that is more intelligent than the most intelligent human you might still need a few billion supercomputers to outmatch the human race.

What messy about this is figuring out exactly how to define intelligence, and what it really means to be more intelligent than someone. If somebody said that an AI was more intelligent than the most intelligent human, then how would you figure out if that was true? If it was true, what exactly would this AI be able to do that the human couldn't do?

How does a difference in the level of intelligence of a particular entity compare to the effective intelligence of having lots of less intelligent entities? Having lots of people means that you can apply intelligence towards a lot of tasks, but perhaps all of them together will never come up with something that a greater intelligence could come up with.

Thinking about it, it seems to me that you have to apply it against an actual difficult task. Take a task like "build a power-producing fusion reactor" that nobody knows how to do yet. Would we be able to figure out how to do that faster if there was some superior intelligence applied to it? Or does the huge number of engineering problems associated with building such a thing respond better to a large number of more ordinary intelligence?

This assumes that a single greater-than-human AI doesn't figure out how to get humans onto its side. Given that it will not be a cartoon supervillain, that task will actually be incredibly easy: find ways to give humans what they want in exchange for doing what the AI wants.

Sure, then it will become just another productive person/citizen. We work together because we benefit from it. Working against another intelligent being is significantly harder in the long term.

edit: And the collective is significantly more intelligent than the individual (for some but definitely not all reasonable definitions of intelligent and collective).

>Sure, then it will become just another productive person/citizen.

Only in the sense that Stalin or Pinochet was a productive citizen. Powerful people can mobilize others to their own ends without those ends being at all good or moral for everyone who is not so mobilized.

You can disagree with his methods, but saying Stalin wasn't productive is just plain wrong.

From agrarians to victors of ww2 in ~20 years is pretty impressive by any measure.

It's only one data point, but observe how much more intelligent humans are than the next most intelligent species on Earth.

We are much more intelligent because we have efficient ways to interchange information. It allows us to use our collective intelligence. Our singular intelligence isn't that much more evolved. ("Intelligence is in our language")

Sorry for reviving old thread.

And observe how large our brains have to be to achieve that, and how many calories it requires.

I don't understand how this is relevant to the conversation at hand. I had assumed that we were discussing this topic with the assumption that we can make full use of first world resources, so we should have a capable, (mostly) reliable power source, food, materials, etc. readily available.

> Human-level AI is possible.

> Human-level AI will likely be practical this century.

Both are unknown.

>Human-level AI is possible.

Well we exist, so we know human-level intelligence can be implemented in physics. So we certainly know AI is possible.

I agree to that. But the question of whether human-level I is capable of designing human-level AI remains open still.

Well, design is a slippery word. We are capable of producing children that grow up to be intelligent.

In fact, I think it is quite clear that we are capable of designing systems that are more intelligent than an unaided human (or even group of humans). One of the first such systems were writing. I also tentatively claim that new means of recording and arranging knowledge (which we do today by means of computers) is another step towards superhuman intellect in the strictest sense of "more than mere human".

I think it is an open question what will come out of our playing with genes directly. I certainly expect we'll able to produce some change in the human condition, I hope some of that change will be for the "objectively" better. I have no idea if it will directly impact our intelligence.

Even if we never figure out a formal computational design for human-level AI, I find it hard to believe that we wouldn't be able to achieve the same ends with genetic engineering and computer interfaces to biological computers (i.e. brains).

> I agree to that. But the question of whether human-level I is capable of designing human-level AI remains open still.

"Designing" isn't necessary for creating (OTOH, whether we can artificial select for it and avoid selecting for dead ends may be an issue.)

You are correct. But it's not 100% certain whether we would be capable of setting up a world simulation that would create intelligent agents of the human level. And much less certain, (1) are we able to set up such simulation in the next couple of centuries, and (2) will the simulation produce these intelligent agents after having been running for only a couple of centuries (wall clock time).

Btw, Crystal Nights by Greg Egan is a fascinating short story: http://ttapress.com/553/crystal-nights-by-greg-egan/

Playing a little bit devil advocate here ;)

We have no mean to know. Maybe the subjective experience is an inherent part of our intelligence, and not just a side effect. And I do not see any way of applying the scientific method to the subjective experience.

Nethertheless, I really hope that the future will prove me wrong.

Please correct if I'm wrong, but sounds like you think human intelligence requires something outside of known physics, something "magical", to work?

I'm agnostic about this question. We do not have the necessarry tools to answer right now. One possible way I can see to do that would be to map a human brain neuron by neuron and simulate it. We are not here yet.

> Human-level AI will likely be practical this century.

Given how far away from that goal we are now, after 85 previous years of research, I would not hold my breath for reaching that goal in the next 85 years.

While I'm in the camp of "people who say 'X will happen in the next Y years' are really saying 'X will happen before I die'", I think it is unreasonable to treat the last 50-85 years as a failure for AI.

There are two things holding back AI (which from a singularity-standpoint just means "a machine capable of general problem-solving", and not "a machine possessing the ineffable characteristics of human consciousness"). First, we don't know how. Second, we might not have the computational capacity.

While we still don't know how, the bar is getting lower and lower as time goes on. Increasing computational capacity means that we can start with worse and worse initial versions of a general intelligence. Maybe a really special genius could write a general problem solver which ran on a 386 (even if it lacked the self awareness to feel the pain of running on a 386). Maybe we're too dumb to ever write a general problem solver even if we had an infinately fast computer with unlimited memory.

But even if programmers have made no progress since the 1960s, we are definitely not in the same position.

"While I'm in the camp of "people who say 'X will happen in the next Y years' are really saying 'X will happen before I die'"..."

That depends on who is saying it. I've noticed that when actual researchers say, "We hope this will provide a treatment for reticularly homicidal brain-worms within 5 years", they mean they have no idea if or when any such practical material benefits will appear. If they say, "We will be able to get effectively unlimited amounts of energy from russet potatos within 20 years," they mean they don't believe it's ever going to happen.

If the difficulty is exponential, as I rather expect it to be (in my own completely unfounded opinion), then lowering the bar a given amount doesn't help much. And as far as increasing computational capacity goes, we're already deep into the realm where communication costs outweigh actual computational costs, which makes it a whole different ball game going forward.

If you want a further speculation, I suspect that modern statistical methods like those behind th' Goog, self-driving cars, and what-not will not get too much closer to a super intelligence than conventional heuristic search. They're capable of doing a lot of things easily, that only humans could do heretofore, but they're not actually more powerful. So, "a machine capable of general problem-solving" is easily conceivable in the same sense that you are capable of general problem solving, but I can't see a singularity on the other side of that.

Relevant xkcd: http://xkcd.com/678/.

> (even if it lacked the self awareness to feel the pain of running on a 386)

Now we know the real motivation behind pretty much every sci-fi storyline involving AI revolts.

2014-2*85=1860 -- so I take it your statements means that discovering AI in the next 85 years would be a bigger change/achievement than the changes we see in the world today from the world of 1929, that was founded on work started in 1860?

So, having a look at wikipedia, we find: "Inspired by Babbage's difference engine plans, Per Georg Scheutz built several difference engines from 1855 onwards, one of which was sold to the British government in 1859." -- that's very convenient. We can cheat a little, and say that in 1929, there'd been about 86 years worth of research into computing -- and so by extension, achieving AI by 2100 is a greater feat than the NSAs new data center was as seen from 1929?

Perhaps. Makes for some interesting thought experiments.

How do you even quantify "how far away from that goal we are now"? Do you have a percentage?