>"We are obviously still some distance from asking an AI to build an entire phone in one prompt."
What a great idea for a new AI benchmark!
We already have general knowledge benchmarks, instruction following benchmarks, "Humanity's Last Exam", and several software engineering benchmarks.
Why not create a new AI hardware engineering benchmark where some of the questions could be around building an entire smartphone, and one of those questions (which will obviously fail at this point in time, but perhaps not in the future) would be to:
create an entire smartphone in a single prompt.
?
We could subsequently diagnose a given LLM (or other AI's) hardware engineering abilities by diagnosing when, where, why and how it failed, when asked to create an entire smartphone!
Which could help train better models, at least in terms of hardware engineering capabilities...
(Also, side note: For a hardware engineering company, asking a job candidate the rather open-ended question of "tell me how you would design a smartphone?" would be a great open-ended interview question, leading to many other subsequent hardware questions / explorations of other areas of related knowledge potentially relevant to the candidate's assessment...)
What a great idea for a new AI benchmark!
We already have general knowledge benchmarks, instruction following benchmarks, "Humanity's Last Exam", and several software engineering benchmarks.
Why not create a new AI hardware engineering benchmark where some of the questions could be around building an entire smartphone, and one of those questions (which will obviously fail at this point in time, but perhaps not in the future) would be to:
create an entire smartphone in a single prompt.
?
We could subsequently diagnose a given LLM (or other AI's) hardware engineering abilities by diagnosing when, where, why and how it failed, when asked to create an entire smartphone!
Which could help train better models, at least in terms of hardware engineering capabilities...
(Also, side note: For a hardware engineering company, asking a job candidate the rather open-ended question of "tell me how you would design a smartphone?" would be a great open-ended interview question, leading to many other subsequent hardware questions / explorations of other areas of related knowledge potentially relevant to the candidate's assessment...)