Free for 90 days: Sign up now and get 90 days managed free after your first deposit. See offer details

Limited time offer: Sign up today and get up to 1 year managed free. See details

Get your next rollover managed free in 2018. Learn how to enroll

Engineering

The Evolution of the Betterment Engineering Interview

Betterment’s engineering interview now includes a pair programming experience where candidates are tested on their collaboration and technical skills.

Mike Reust

By Mike Reust
VP, Head of Engineering, Betterment  |  Published: August 2, 2016

Betterment’s revamped hiring process allows engineers to demonstrate their technical aptitude and collaborative problem-solving ability.

If invited onsite, applicants will work alongside an interviewer for a pair programming experience.

Recruits can also provide feedback and speak openly with non-interviewing members of the team.

reust-jana pair programming engineering interview

Building and maintaining the world’s largest independent robo-advisor requires a world-class team of human engineers.

This means we must continuously iterate on our recruiting process to remain competitive in attracting and hiring top talent.

As our team has grown impressively from five to more than 50 engineers (and this was just in the last three years), we’ve significantly improved our abilities to make clearer hiring decisions, as well as shortened our total hiring timeline.

Back in the Day

Here’s how our interview process once looked:

  1. Resumé review
  2. Initial phone screen
  3. Technical phone screen
  4. Onsite: Day 1
    • Technical interview (computer science fundamentals)
    • Technical interview (modelling and app design)
    • Hiring manager interview
  5. Onsite: Day 2
    • Product and design interview
    • Company founder interview
    • Company executive interview

While this process helped in growing our engineering team, it began showing some cracks along the way.

The main recurring issue was that hiring managers were left uncertain as to whether a candidate truly possessed the technical aptitude and skills to justify making them an employment offer.

While we tried to construct computer science and data modelling problems that led to informative interviews, watching candidates solve these problems still wasn’t getting to the heart of whether they’d be successful engineers once at Betterment.

In addition to problems arising from the types of questions asked, we saw that one of our primary interview tools, the whiteboard, was actually getting in the way; many candidates struggled to communicate their solutions using a whiteboard in an interview setting.

The last straw for using whiteboards came from feedback provided by Betterment’s Women in Technology group. When I sat down with them to solicit feedback on our entire hiring process, they pointed to the whiteboard problem-solving dynamics (one to two engineers sitting, observing, and judging the candidate standing at a whiteboard) as unnatural and awkward.

It was clear this part of the interviewing process needed to go. We decided to allow candidates the choice of using a whiteboard if they wished, but it would no longer be the default method for presenting one’s skills.

If we did away with the whiteboard, then what would we use?

The most obvious alternative was a computer, but then many of our engineers expressed concerns with this method, having had bad experiences with computer-based interviews in the past.

After spirited internal discussions we landed on a simple principle: We should provide candidates the most natural setting possible to demonstrate their abilities. As such, our technical interviews switched from whiteboards to computers.

Within the boundaries of that principle, we considered multiple interview formats, including take-home and online assessments, and several variations of pair programming interviews.

In the end, we landed on our own flavor of a pair programming interview.

Today: A Better Interview

Here’s our revised interview process:

  1. Resumé review
  2. Initial phone screen
  3. Technical phone screen
  4. Onsite:
    • Technical interview 1
      • Ask the candidate to describe a recent technical challenge in detail
      • Set up the candidate’s laptop
      • Introduce the pair programming problem and explore the problem
      • Pair programming (optional, time permitting)
    • Technical interview 2
      • Pair programming
    • Technical interview 3
      • Pair programming
    • Ask-Me-Anything session
    • Product and design interview
    • Hiring manager interview
    • Company executive interview

While an interview setting may not offer pair programming in its purest sense, our interviewers truly participate in the process of writing software with the candidates.

Instead of simply instructing and watching candidates as they program, interviewers can now work with them on a real-world problem, and they take turns in control of the keyboard.

This approach puts candidates at ease, and feels closer to typical pair programming than one might expect. As a result, in addition to learning how well a candidate can write code, we learn how well they collaborate.

We also split the main programming portion of our original interview into separate sections with different interviewers. It’s nice to give candidates a short break in between interviews, but the main reason for the separation is to evaluate the handoff. We like to evaluate how well a candidate explains the design decisions and progress from one interviewer to the next.

Other Improvements

We also streamlined our question-asking process and hiring timeline, and added an opportunity for candidates to speak with non-interviewers.

Questions

Interviews are now more prescriptive regarding non-technical questions. Instead of multiple interviewers asking a candidate about the same questions based on their resumé, we prescribe topics based on the most important core competencies of successful (Betterment) engineers.

Each interviewer knows which competencies (e.g., software craftsmanship) to evaluate. Sample questions, not scripts, are provided, and interviewers are encouraged to tailor the competency questions to the candidates based on their backgrounds.

Timeline

Another change is that the entire onsite interview is completed in a single day. This can make scheduling difficult, but in a city as competitive as New York is for engineering talent, we’ve found it valuable to get to the final offer stage as quickly as possible.

Discussion

Finally, we’ve added an Ask-Me-Anything (AMA) session—another idea provided by our Women in Technology group.

While we encourage candidates to ask questions of everyone they meet, the AMA provides an opportunity to meet with a Betterment engineer who has zero input on whether or not to hire them. Those “interviewers” don’t fill out a scorecard, and our hiring managers are forbidden from discussing candidates with them.

Ship It

Our first run of this new process took place in November 2015. Since then, the team has met several times to gather feedback and implement tweaks, but the broad strokes have remained unchanged. As of July 2016, all full-stack, mobile, and site-reliability engineering roles have adopted this new approach. We’re continually evaluating whether to adopt this process for other roles, as well.

Our hiring managers now report that they have a much clearer understanding of what each candidate brings to the table. In addition, we’ve consistently received high marks from candidates and interviewers alike, who prefer our revamped approach.

While we didn’t run a scientifically valid split-test for the new process versus the old (it would’ve taken years to reach statistical significance), our hiring metrics have improved across the board.

We’re happy with the changes to our process, and we feel that it does a great job of fully and honestly evaluating a candidate’s abilities, which helps Betterment to continue growing its world-class team.

For more information about working at Betterment, please visit our Careers page.

More from Betterment:

Determination of largest independent robo-advisor reflects Betterment LLC’s distinction of having highest number of assets under management, based on Betterment’s review of assets self-reported in the SEC’s Form ADV, across Betterment’s survey of independent robo-advisor investing services as of March 15, 2016. As used here, “independent” means that a robo-advisor has no affiliation with the financial products it recommends to its clients.

Recommended Content

View All Resources
How We Built Two-Factor Authentication for Betterment Accounts

How We Built Two-Factor Authentication for Betterment Accounts

Betterment engineers implemented Two-Factor Authentication across all our apps, simplifying and strengthening our authentication code in the process.

Betterment’s Approach to Financial Advice: An Overview

Betterment’s Approach to Financial Advice: An Overview

Achieving your financial goals is only possible if you plan effectively. Saving enough, choosing the right accounts, deciding when you can buy a house or when to retire—all of these are essential decisions even before you build an optimal portfolio.

How We Engineered Betterment’s Tax-Coordinated Portfolio™

How We Engineered Betterment’s Tax-Coordinated Portfolio™

For our latest tax-efficiency feature, Tax-Coordinated Portfolio, Betterment’s solver-based portfolio management system enabled us to manage and test our most complex algorithms.

Start your investment plan

I am

years old and

  • Not Retired
  • Retired

.

My annual income is

.

Experience the new way of investing. Sign up today.

Start investing smarter

Get started

Refer a friend or family member and get up to 1 year managed free

Refer a friend

How would you like to get started?

Your first step toward a smarter investing future starts here.

Create a Betterment account

Go ahead and join the smart, modern way to invest.

See what we can do for you

Tell us a bit about yourself, and we’ll show you the benefits of investing with us.

Get a free investing checkup

Help us get a sense of your investing approach and see how you could improve.

Transfer a 401(k) or an IRA

Move an existing retirement account into a Betterment IRA.

Download the mobile app

Enjoy the Betterment experience anywhere on the go.

For more information and disclosures about the Betterment Resource Center, click here. | See our contributors.