I helped start the practice at ITA Software (acquired by Google last year) of using puzzles as part of the hiring process. We ultimately even put our puzzles in the Boston subway as recruiting ads. My assessment, after more than a decade of doing it:
- Good puzzles are actually a talent attractor; many smart people found out about our company via our puzzles.
- Good puzzles are ones that scale well; i.e., where the basic problem is pretty easy and can be solved by a decent programmer in a few hours, but with harder variants that can take much longer.
- "Take-home" puzzles (as opposed to in-person whiteboard tests) weed people out who don't really like to program and/or can't finish things; this is a useful filter. If someone complains that "they shouldn't have to spend three hours writing code to get an interview," that itself is a pretty good counter-indicator. (Yes, we are all busy. But you're talking about starting a relationship with the company that may last 10+ years. You can do three hours of prep work for your interview. And you have to code a fun puzzle in the language of your choice, not some subroutine in a 30-year-old COBOL banking system.)
- It seems that in-person whiteboard or locked-in-a-room tests are pretty poor indicators of success. Many good programmers put in this situation significantly underperform their true abilities.
- It's not a great idea to evaluate someone purely on the basis of puzzle-solving ability.
- Many one-liner puzzles are bad indicators, because you either need to "know the trick" or have memorized the answer. Many, but not all, Microsoft and Google interview questions I hear about -- I've never interviewed at either place myself -- sound like they fall into this category. (E.g., from a recent article I read about Google: "You're reduced to the size of a nickel and thrown into a blender. How do you get out?" I don't care if you're clever enough to answer that; I just want you to able to write enormous amounts of high-quality code for me.)
Im my experience, the best way to find out if someone is a good programmer is to talk to him/her at length about something he/she has built. Can he/she talk in detail about how it worked? What challenges were involved? What tradeoffs were made? And, above all, was there passion behind the work and these choices?
A good interview for me was one where the candidate came in having written or contributed to a large system -- for fun or work -- and was excited to tell me about it. It didn't really matter whether it was relevant to the work we did (airfare search). Anecdotally, it seemed like people who really loved to code generally worked out pretty well. People with great-looking resumes that didn't love to code -- but maybe loved other things, like arguing over which language, operating system, architecture, or business strategy was better -- usually didn't produce much.
If the first thing you want to do when you wake up in the morning is code something, you're probably going to do well as a coder on a hard-core software team. Otherwise, not. Everything I personally did on programmer-hiring strategy was a proxy for figuring out whether someone was like that or not.
Maybe I'm weird, but I find puzzles to be a turnoff. I love writing code, and I enjoy working through a math book.
But compared to coding and math, puzzles feel sterile. I don't get to build anything, and I rarely get the kind of insights I would get from figuring out a proof. (Raymond Smullyan's puzzles are a marvelous exception to this.) Even cryptanalysis is more fun, because there's a real opponent.
I know lots of amazing programmers who love puzzles, and that's cool. But if you only hired puzzle lovers, you'd miss out on a lot of good people.
You're not weird. Puzzles can definitely be fun, but, in my experience, it doesn't mean you're a lousy programmer if you can't solve one/some.
Everyone likes to use Google as the example of the elite programming company. I've read the books about them, used the products, followed their progress over the years. So, sure, a part of their big success is hiring the 'smartest' college grads.
However, I've also worked with Ph.D.'s and even interviewed them for positions lower than ones I've held (I have not yet finished college - some day...). And I can tell you that book smarts != execution smarts.
I feel the same way about puzzles. I have never used puzzles or asked 'small' one off programming questions. Instead I've always been interested more in grep'ing the concepts and if the personality is the proper fit. In my mind, anyone can open up a book and read about specifics about O(log n). But how does a person persist at solving a problem and how do they (can they?) change direction if their solution sucks?
I feel like I value creativity over anything and they can creatively solve a puzzle, cool. But if they can creatively solve a task on the system I'm building and are a great fit to the team, then better.
Same here. My impression is that the puzzle lovers turn out to be the algorithm geeks. You need a bunch of those, but I can't see how a room for of complexity analysis experts necessarily creates well-structured software.
I'm an API design geek. I've never seen a cool puzzle about API design, or something remotely related.
i'm pretty sure one of quora's questions is/was designing an api for something in python (probably time, given how bad the current api is).
also, replying more to the other post above - you can get insights from puzzles. a lot depends on the puzzle (which is why ita is getting praise here). for example, the ita puzzle i remember taught me a lot about handling data in large numbers of dimensions (in particular, it made me think about the "curse of dimensionality" which was something i was aware of before, but hadn't considered in detail; it also led me to understanding some tools for dealing with it - locality sensitive hashing being the most interesting at the time).
Most puzzles are not very interesting. :) But some are surprisingly rich, and those are the good ones to put in front of candidates in my opinion. E.g., I worked on one puzzle before I left ITA that involved laying out train track pieces to create a complete circuit within a rectangular region. (I don't believe this one has ever been used, but I could be wrong about that.) It was pretty interesting to solve this heuristically, but it turned out to involve something much deeper: the theory of so-called "lattice animals" -- an active area of research I knew nothing about.
Lattice animals include polyominoes, which are probably familiar to most HN readers, but subsume other graph-related objects as well. I didn't know much about these, and ended up reading a bunch of cool papers on them as part of working on the puzzle. So at least for me, good puzzles have catalyzed some learning.
I don't think that's too weird. I am never very interested in solving puzzles. Myst was a major turnoff, for instance. The Project Euler problems are usually "meh".
My mathematical preference is continuous math; my preference for using mathematics is to apply them into real problems.
I'm not saying that all puzzles are pointless, but I think most interview puzzles are kind of an empty exercise that only a certain type of person likes, and that only people who know the trick will get.
For a good programmer, a 3 hour take home test is like a $300+ application fee. Even if the puzzles are reasonable, filters work both ways. For a recent graduate or a mediocre programmer, it's probably worth the time for a shot at the job, but talented people will have plenty of options at good companies that don't make them jump through extra hoops. Perhaps it's fine if you pay a lot higher than average or offer significant perks, but I could see a strategy like this unwittingly driving away the top tier of candidates if you aren't careful.
That is a reasonable theory, and I was always personally very concerned about that, but in practice it didn't seem to be the case. It seemed that the opposite occurred: people who didn't even want jobs did the puzzles just for fun, and asked us if they got the right answer.
Of course, we couldn't easily do A/B experiments, so we'll never really know how the true attract:alienate ratio.
Trying to look at it objectively, I would probably be somewhat put off by being required to do a puzzle to "prove myself", but would then find the specific puzzles offered to be pretty interesting, and worth playing around with. Perhaps it's not tantamount to a $300 tax on ones life it is seemed fun and valuable in its own right?
I have less disdain for pre-interview development filters and have seen some companies do it pretty well. I think your assumption about people who would find it fun could hold some merit. But I do believe the parent poster is right, those really good developers are rarely going to bother. They are usually to busy in the first place. They may see your company and think oh cool I like these guys and then hit the pre-interview filter and decide it to much investment for just a chance and that's the core of it, on both sides it's a numbers game, we can say well they did not really want it or we are only getting the ones that really want it, but you just may be getting the ones that are not working the odds out in their head. Or you could be getting the ones that are desperate and will invest the time for a chance. There are a lot of assumptions in these type of tests.
I think in the end, most people just want to have their abilities evaluated fairly. And even if I'm not a total puzzle freak, I do love to code, and given time I can figure most of these puzzles out. So if I were looking for a job at your (former) company, this would not constitute a barrier for me. On the other hand, if I were verbally asked a question and told to work it out on a whiteboard, this would not be an accurate indicator of my abilities or my potential. Add to that the fact that my handwriting is poor, and I would be nervous, and I'm quite sure a verbal description of pascal's triangle would almost certainly elude me, whereas if I were shown a document with the written output, I think I could come up with a naive solution in 5-10 mins. I'm good at sitting down by myself with a written problem and working it out. As a result, I've always done well on standardized tests. In fact, I suspect this is the reason for some of the indignation at the whiteboard testing. People who have always gotten good grades, coded well, done well on standardized tests are suddenly told that they're not smart enough. I can understand how it would rankle.
It isn't like that because paying money to someone in exchange for merely being evaluated is an easy way to fall for a scam: the evaluator has a significant incentive to offer this deal to people who will fail.
It is much more like having to travel 90 minutes each way to get to where they want you to interview. That could be very offputting if you do enough interviews that three hours each adds up to a big number, but at least you don't need to worry about them being tempted to rip you off.
I interviewed at ITA in 2007, and ITA did the puzzle process better than all but a few other companies I've interviewed at, for the reasons you mention. But don't discount the in-person whiteboard puzzle. I was told I had the second highest score on the puzzle solution I sent in, but I flopped the whiteboard puzzle part of the interview, and ended up not being hired. I think this was a good thing - I didn't have the algorithm skills and knowledge of the other people in the QPX group (QPX is ITA's main business, their search engine - they actually recommended in advance that I interview for their reservation system group based on my resume, but I decided to go for the more brainy one). If I didn't have the whiteboard interview, I would have gotten in and probably regretted it later (much later, I realized that I would have regretted working at ITA anyway, or anywhere in Boston, but that's totally unrelated).
During the same job search, I also noticed Facebook had puzzles. I solved one and got invited to interview but never went. I wouldn't have applied otherwise. I guess this is another lesson - if you put public puzzles out there, expect people to send solutions without necessarily being excited to work at your company.
I once was asked the SAME puzzle at two interviews (for different companies) two weeks apart.
I'd never seen it before the first interview. In that first interview, I broke it down quickly into its component parts, and figured out where the trick had to be -- and from there quickly solved it on the spot.
In the second interview I spent a couple minutes trying to remember the solution from the first interview, and completely flubbed it. It wasn't the first part of that second interview that I flubbed, either, including arguing with one of the interviewers and basically telling him he was wrong.
Oddly enough, I got the job with the second company and worked there for almost six years. For the record, I love doing those kinds of puzzles. But I do video game development, so maybe puzzles are a bit more relevant to my industry (in that 80%+ of what we're doing is figuring out how to solve puzzles in the form of how to do something with less CPU or why a particular video card is behaving in a certain way).
I saw those puzzle ads frequently in Kendall/MIT and Harvard (what, 6-7 years ago?). It was very clever recruiting on your and ITA's part, and a much better way of using brain teasers than as part of an in-person whiteboard session.
I don't think you give yourself enough credit in the success of the puzzles. They were really well-conceived problems - just difficult enough without requiring too much domain knowledge. Very 'catchy', if that makes sense.
Thanks. We were very aware of the marketing aspect of the puzzles, and designed them accordingly. The ones that made it into the T were ones that had visual appeal as well. It wasn't just coders, either; marketing and graphic design worked played a big role as well.
"You're reduced to the size of a nickel and thrown into a blender. How do you get out?"
If I were asked such a question in an interview, my response would be: "Dunno. Let's talk about some real-world problems you're facing right now instead."
I dislike trivial questions, especially during an interview. I'm a professional, not a circus animal.
3 hours of prep work X 5 applications (1 application a day for a week) = 15 hours of unpaid work per week, or 60 hours of unpaid work per month. Given that a typical job search lasts multiple months, this is not an insignificant workload to saddle potential employees with. However, if your company is special in some way and people feel it's worth the rigamarole to apply there, then the market will bear it.
> I don't care if you're clever enough to answer that;
Bingo.
To me, this is the kicker statement for the industry. Solving fake problems like nickel-in-a-blender is for the clever. But I sure as hell don't want clever code solutions; I want rock-solid, reliable, fundamental solutions.
Anyone who feels these questions are an indicator of programmatic success is measuring the wrong things.
To be fair, the "nickel in a blender" type of problems have been actively banned from engineering interview at Google for many years. That doesn't mean that there aren't a few holdouts that still use them, but there is an active effort to eliminate them.
I sit on an engineering hiring committee and I can't really think of the last time I saw one of the ridiculous puzzle questions show up in an engineering interview. The WSJ article was pretty ridiculous.
Wasn't implying these to be solely a Google issue, but rather something that a material number of firms in the tech industry like to follow in interviewing.
Sorry, but I'm not familiar with the WSJ article you've referenced. Link?
but i suspect that ita were exceptional. first, your puzzles were interesting - i imagine it's quite hard to think of interesting puzzles (how did you do it?). second, ita had a very good reputation so people were very motivated to work there. third, you may have been solving harder problems (and looking for more problem-oriented programmers) than many other companies.
i have done your puzzles for fun, but i would still be annoyed at having to do half-baked puzzles if i were suddenly out of a job and applying to a bunch of places.
I agree that the puzzles definitely have to be well-designed. There were a few of us -- myself included, for a while at least -- who created most of the puzzles.
We also never put a puzzle out to the world until a bunch of us had solved it. There was a long gestation period, and there's a "puzzles-discuss" mailing list for working out the kinks (and, in many cases, rejecting certain puzzles entirely). So, yes, you have to make a real commitment to do it well.
On the other hand, developing the puzzles was a lot easier than most of the coding on our real system, so it's all relative. It was actually really fun to do this stuff, and a nice change of pace. One of ITA's best developers (Justin Boyan) would take it upon himself to create the "perl postcard" solution for each problem. Another (Jim Rees) would always make an incredibly fast C++ version; it usually ended up being not only the fastest of ours, but faster than all the submitted solutions as well. I learned things just by reading his puzzle solutions. :)
I found the process rewarding beyond the hiring benefits, and I'm sure I'm not the only one. (Well, I hope so, at least.)
I'm jealous:) sounds like a lot of fun. When they came out, I used the ITA puzzles to shake the rust of my lisp skills and even considered sending some of the solutions in and applying, but that would have been a major career change.
In any case, kudos and thanks to you - those puzzles were a lot of fun, and I always wondered where they came from.
Personally, I failed the ITA puzzle. (Either I failed it or they looked at my resume post-fact and decided I wasn't smart or qualified enough... probably a combination).
So, here is my opinion as a non-halfway-decent programmer.
I thought puzzle was fairly fun, but I like programming for two reasons:
* Building useful knowledge. Learning a new language, library, or algorithm I want to play with, programming is the natural way to learn.
* Building cool stuff that people like. Having an awesome product when I'm done with it gives the feeling that I've done a good job and produced something of value.
Programming puzzles don't really do either of these things for me. My motivation for spending three hours doing a puzzle is that I would like to talk to a person. I am basically assured that the puzzle will be used as a filter for the company to avoid talking to people. It will run against a test suite and if it fails it will not be looked at by anybody. This seems very impersonal. Is this the experience I should expect from the rest of the company if I'm hired?
I've got a Git-hub account with some of my code in it. I've spent time on it and enjoyed writing all of that code. I published it to the internet so I'm (at very least) not ashamed of it. Why can't you look at that to determine if I can program? Why do I have to write code specifically for your particular company?
This isn't an attack on ITA, it seems to work for them. But they are a unique case of a company where it seemed worth it for me to spend the time to do the puzzle. Any other company I would have just skipped it unless I genuinely had nothing better or more interesting to do (of which I almost always do).
>If someone complains that "they shouldn't have to spend three hours writing code to get an interview," that itself is a pretty good counter-indicator. (Yes, we are all busy.
When I applied at ITA they didn't even acknowledge the program I had written to go along with the application. Not even a quick 'it ran' or 'had the right output but we hired somebody else'.
I'm glad I didn't get the job because not having enough time to evaluate or even run a programming test you gave is also a pretty good counter-indicator of a good company to work for. And when I wrote back to express this the guy sheepishly admitted they had so many exceptional applicants so they just went with an Ivy-league graduate. Wow.
If you are going to require a programming test, be prepared to evaluate it.
If you are going to require a programming test, be prepared to evaluate it.
I agree. It is possible the reviewer didn't think your code would run, couldn't get it to compile, etc. But, nevertheless, you should have received a fuller response than that. If I were still COO at ITA, I would ask someone to look into your particular case to understand what happened.
Like jsnell, I spent weeks’ worth of nights beating my head against the problem I selected, and the solution I eventually came up with was not good enough to earn me an interview.
I gather that someone with a stronger theoretical CS background than myself would have spent an hour looking at the puzzle, said “Ah, this can be modelled as a graph and solved by applying the Flibblewhiz Heuristic!”, spent another hour coding up a first draft of the solution, and a third hour optimizing and cleaning up the code.
My ego has survived the blow of learning that I am not awesome enough to work for ITA, but I wish I had figured that out sooner.
I don't think I had a satisfying solution to any of the puzzles in 3 hours. In the most extreme case, I saw some people spend a couple of weeks solving the two-time pad problem. Even after knowing how to cancel out the pad.
Ah, fair enough, there must have been some simpler ones there as well. It just was not the kind of thing I would have hacked on for fun :-)
I was thinking more of all the cool search problems like Strawberry Fields, Traversing the T or Palindromic Pangram, where coming up with some solution is easy, a naive exhaustive search for the optimal solution was not feasible, and where verifying the validity of any heuristics / pruning was not necessarily trivial.
Are you responsible for the puzzle involving the historical sites search with a google instant style interface? This was a long time ago and it's no longer in the archives.
I worked on that one for a looong time, and what bugged me was the screenshot of the expected interface had a result returned in like 0.05675ms or something, and I couldn't figure out how to get it that fast. It took be forever to convince myself that it was just a mock and probably should have read 0.05675s (seconds instead of milliseconds).
I didn't work on that one, no, but I wouldn't assume the time was fake either. :)
We often had internal solutions (and externally-submitted solutions as well) that were orders of magnitude faster than the average. There was a very wide range of execution times. E.g., we had solutions to the Queens & Knights puzzle (in the archive) that ran in tens of milliseconds, if I recall correctly. Typical solutions we received took orders of magnitude longer.
Usually when one solution was a million times faster than another, it was because the fast solution exploited a clever algorithm or data representation -- e.g., representing an 8x8 chess board mask as a single 64-bit register rather than as an array of bytes.
Ah thanks! However, I'm worried I may take another crack at it now!
I may have missed that the time was the time taken for the search algorithm to find results, I think I was under the impression that should be the time for the request/response to happen. Hmm.
I must personally say thank you for starting those puzzles. Over the years I have implemented many of them as fun weekend projects, but one puzzle in particular stuck with me: Sling Blade Runner. On and off the last ~four years I have worked on the sling blade runner puzzle creating dozens of different approaches for the problem. From genetic algorithms, visualizers, various traveling salesmen "solutions", stuffing the graph in the cpu cache and letting it run for a month to document how far the dumb solution wouldn't get, custom algorithms doing wacky things such as spherical gravity water flow simulations and many more I have pushed that data set through a lot of things. When I read books and papers and constantly working on the SBR problem in the back of my head. Generic graph theory can be a boring topic, but for some reason with this problem and data set to play with it has been something that has stuck with me. The generic problem of solving the longest path in a directed circular graph is NP so program solutions are all about find something good enough given limited time or exploiting issues with the data set to actually solve it. There is a lot of payoff in the industry for good enough solutions to this problem (which I am not actually involved in, I hack on webkit during the day). It is the puzzle the keeps on giving and as I encounter new techniques on the job or in reading I often end up trying later in the evening to see how well/fast they apply to this problem. I moved to Boston with an existing job (same one I am at now) so I never really got to go through the whole interview process at ITA, but always thought it would be fun to help design those puzzles. So again thanks for the puzzles, especially Sling Blade Runner!
I got asked 3 puzzles for my current job. It was quite annoying. One, I knew off the cuff. The other 2 were bad questions. One was caught in the "if you don't make these assumptions your wrong" trap. For the other, the "correct" answer was actually wrong. It really pissed me off.
Funnily enough, the only time I've been asked a genuine logic puzzle in an interview was at ITA (don't remember the details, but it was something about ants walking on a 1d line and bouncing off each other). It even had some kind of a "either you see it or you don't" aspect as well.
But the programming puzzles on the website were just awesome, even though I just solved them because they were fun, not for applying for a job. I've been surprised that nobody else has done the same thing, but your explanation of how much work went into the puzzles kind of explains that :-)
Just to clarify, ITA's puzzles weren't all of mathematical / algorithmic tricks variety.
I did one of them just for fun 3-4 years ago:
"implement a scalable multi-user chat server without using a framework like Netty/ACE/EventMachine". Unfortunately, at the time I couldn't move to Boston, so I never submitted my solution.
Likewise there were puzzles that would appeal to web developers (e.g., build a typeahead web applications), machine learning/AI folks/etc...
It seems you like you still hire based on puzzle as a first filter (based on on your careers page). Just wondering, what is the ratio of people who apply with only a resume vs those who solve the puzzle (I understand you probably can't reveal). And, more importantly, are those on the puzzles side show to be more successful on the interview?
But the puzzle process, as originally conceived, (in 2000!) was not necessarily supposed to be a hard filter. As an applicant, you had several ways to stand out, and doing a great job on a puzzle was one. Having someone who already worked for us say you were awesome was another. Being an open source superhero was another. Etc.
In actual implementation, though, it probably was/is used as a hard filter.
In actual implementation, though, it probably was/is used as a hard filter.
It always becomes a hard filter, there is a lot of ego in development and test are an area where ego's shine through. It's my major beef with quiz exercises in interviews, they always become the hardest filter, no matter how much we try to contain them.
- Good puzzles are actually a talent attractor; many smart people found out about our company via our puzzles.
- Good puzzles are ones that scale well; i.e., where the basic problem is pretty easy and can be solved by a decent programmer in a few hours, but with harder variants that can take much longer.
- "Take-home" puzzles (as opposed to in-person whiteboard tests) weed people out who don't really like to program and/or can't finish things; this is a useful filter. If someone complains that "they shouldn't have to spend three hours writing code to get an interview," that itself is a pretty good counter-indicator. (Yes, we are all busy. But you're talking about starting a relationship with the company that may last 10+ years. You can do three hours of prep work for your interview. And you have to code a fun puzzle in the language of your choice, not some subroutine in a 30-year-old COBOL banking system.)
- It seems that in-person whiteboard or locked-in-a-room tests are pretty poor indicators of success. Many good programmers put in this situation significantly underperform their true abilities.
- It's not a great idea to evaluate someone purely on the basis of puzzle-solving ability.
- Many one-liner puzzles are bad indicators, because you either need to "know the trick" or have memorized the answer. Many, but not all, Microsoft and Google interview questions I hear about -- I've never interviewed at either place myself -- sound like they fall into this category. (E.g., from a recent article I read about Google: "You're reduced to the size of a nickel and thrown into a blender. How do you get out?" I don't care if you're clever enough to answer that; I just want you to able to write enormous amounts of high-quality code for me.)
Im my experience, the best way to find out if someone is a good programmer is to talk to him/her at length about something he/she has built. Can he/she talk in detail about how it worked? What challenges were involved? What tradeoffs were made? And, above all, was there passion behind the work and these choices?
A good interview for me was one where the candidate came in having written or contributed to a large system -- for fun or work -- and was excited to tell me about it. It didn't really matter whether it was relevant to the work we did (airfare search). Anecdotally, it seemed like people who really loved to code generally worked out pretty well. People with great-looking resumes that didn't love to code -- but maybe loved other things, like arguing over which language, operating system, architecture, or business strategy was better -- usually didn't produce much.
If the first thing you want to do when you wake up in the morning is code something, you're probably going to do well as a coder on a hard-core software team. Otherwise, not. Everything I personally did on programmer-hiring strategy was a proxy for figuring out whether someone was like that or not.