How many golf balls can fit in a 747?

If you’re left scratching your head, you’re in good company. 

Abstract questions like this one have been popular as a way of assessing developers for technical roles. You might have heard one or two of them before: How many window panes are there in Manhattan? How many piano tuners are there in the world? How many light bulbs are sold in the U.S. each week?

This style of question is purportedly designed to reveal how a person thinks about a problem critically, as opposed to actually locking down a “correct” answer. (Our best guess is that seven million golf balls can fit in that 747, but don’t take our word for it).

But there are two problems with using brain teasers to evaluate technical talent. First off — developers hate them. And second: They don’t actually measure a developer’s technical skills. Does knowing how many piano tuners exist in the world actually translate to the ability to build an application with, say, Java?

A lot of people associate the popularity of this style of interviewing with the rise of the tech sector. But its history is a little more complicated, and the reaction to (and against) this philosophical approach to skill assessments has led us into new territory — one which we think will change the future of the technical interview for better. 

The Past: Thomas Edison and Brain-Teaser Challenges

“Brain teaser” questions became popular in technical interviews during the rise of the modern tech industry in the 90s and 2000s. But this type of interview question actually predates the tech industry by quite some time.

Thomas Edison famously asked arcane trivia questions to gauge an applicant’s knowledge, including questions like “Where is the Sargasso Sea?” or “Why is cast iron called Pig Iron?”

In the early 21st century, innovative tech companies like Microsoft and Google adopted this a similar approach to screen candidates. Like Edison’s lab, these companies strived to be centers of innovation, and both valued unorthodox approaches to problem solving. This style of question was seen as a gauge of candidates’ creativity, problem-solving skills, adaptability, and capacity to innovate.

As newer tech startups emerged, many copied interviewing styles and questions directly from pioneers like Microsoft. This had the effect of transforming somewhat niche and arcane trivia questions into the industry-standard.

But if brain-teaser questions were once so popular as an interviewing strategy, why have they fallen out of favor? 

1. They’re Not Actually Relevant to the Role at Hand

Brain teaser questions don’t give hiring managers the opportunity to evaluate a candidate’s ability to work with the required technologies. They also don’t simulate the day-to-day working reality of a technical role.

2. They Don’t Indicate Skill Level or Proficiency 

A candidate’s answer to a question about golf balls fitting in an airplane doesn’t help hiring managers understand if they have basic, intermediate, or advanced skills in specific technical competencies. The only real insight this style of question illuminates is the candidate’s ability to answer that specific brain teaser — and, perhaps, think a bit abstractly.

3. They Introduce Unconscious Bias. 

While brain-teasers might seem like a creative way to assess talent, they have the potential to become problematic. That’s because brain teasers require knowledge and techniques that candidates might not have access to, which introduces unconscious bias to the hiring process.

The Present: Algorithm Problems — and the Problem with Algorithms

Over time, companies came to understand that they needed to orient their hiring processes around technical skills. Starting in the mid-2010s, there was a discernible shift toward whiteboarding and interview questions about algorithms and data structures, all in an attempt to better vet for technical know-how. 

A far cry from the arcane trivia questions of the past, these problems are designed, in theory, to test practical technical skills. Companies using this approach tend to focus on the candidate’s solution to gain a more holistic understanding of their technical chops. For example, a hiring manager might ask an engineering candidate how to invert a binary tree.

This newer approach to interviewing shifted the focus from abstract thinking to a higher-level, almost academic application of programming knowledge and engineering skills. 

But assessing developers on theoretical, rather than pragmatic, coding questions presents its own challenges. 

1. Developer Push Back

In 2017, prominent software engineers took to Twitter to confess that they would fail a whiteboard interview. The subject of their criticism? Tough algorithmic challenges that focus on academic and theoretical coding concepts that they’d rarely encounter on the job. They may have studied those concepts in school, but developers don’t need to have them memorized — because they rarely, if ever, use them in their day-to-day projects. 

Another issue is that many algorithm challenges also deprive developers of the problem-solving tools they would have in a real-world scenario. On the job, developers rely on online resources like GitHub and Stack Overflow to source code, solutions, and bug fixes. Their IDE also saves them time by handling a lot of the coding minutiae. Interviews that take these tools away from developers prioritize the assessment of inconsequential details instead of job-relevant skills and knowledge.

2. Candidate Experience

A 2018 survey by Dice found that 42 percent of respondents’ most dreaded part of the entire hiring process is standing at a whiteboard with a big problem to solve. Dice theorizes that this is tied to fears of “feeling dumb” or looking inadequate. 

Competition in hiring has increased the need for employers to provide an improved candidate experience for developers. Even during hiring slowdowns, developers and engineers are still the most in-demand professionals in the world and receive more inquiries from recruiters than their white-collar peers. This gap between supply and demand affords developers the opportunity to be highly selective in their search process. Candidate experience has become a key determining factor in many career decisions, and as a result, companies of every size are reconsidering how their technical interviews impact the developer’s overall experience.

3. Diversity, Equity & Inclusion

In the past few years, the diversity in tech movement has also ignited calls to make interviewing more equitable. In 2014, major tech companies including Google, Facebook and LinkedIn started publicly sharing the demographic makeup of their workforces for the first time. As part of that movement, advocates also urged companies to shift away from interviews that reward academic computer science backgrounds over applicants with real-world skills in a push to level the playing field for economic opportunities in tech roles.

4. Rise of Bootcamps and Self-Taught Developers

The rise of bootcamps and independent learning has already had a significant impact on the developer workforce. From 2013 to 2020, the number of developers graduating from bootcamps increased by 1,046 percent. As a result, nearly one in six Gen Z developers said they leveraged bootcamps to learn new skills. And 71 percent used YouTube to drive their skills development.

It’s becoming increasingly common for the best candidate for a position to be a bootcamp grad or self-taught coder. Focusing technical interviews on tough, theoretical concepts prevents this talent pool from job opportunities for which they are otherwise a strong fit. 

The Future: A Movement Founded on Real-World Skills

There’s never been more at stake in hiring the right talent with the right skills. While it’s hard to predict the future, what we do know is that the world of development is vast, nuanced — and growing. Year after year, computer science and programming roles have dominated lists for the most in-demand positions in the world. Between 2020 and 2030, the demand for developers will grow by 22% — with competition remaining sky high.

To adapt to this competitive but uncertain future, companies need to evolve their hiring processes to focus on the skills a developer will actually use on the job. Testing and supporting those skills in a candidate-first way will be the difference between a closed and open job posting.

This transition is very much underway. Winning companies are shifting their focus from theoretical applications of knowledge to real-world skills tested in an environment that’s as close to real-world job scenarios as possible.

Skills-based hiring is the process of assessing candidates based on concrete skills through means like coding and problem-solving assessments, virtual pair programming, and take-home projects. This approach benefits developers by providing transparency into what skills they need, what tech they would work with, and the types of technical challenges they’d be solving should they accept an offer. Skills-first hiring also benefits employers, as it gives companies clearer insight into a candidate’s ability to do the job at hand.

What’s more: Developer roles are changing as fast as the technology industry itself. New roles, skills, and technologies are emerging every day that change the skills and developers technical teams need to hire for.

As roles and technologies change at a rapid clip, the assessments used to evaluate developers also have to change. Developers are excited to demonstrate their technical abilities through interviews and projects grounded in their actual skill sets. And savvy technical teams are giving them the opportunity to do so.