Big change is more stressful than small change for people and software systems alike. Dividing a big software project into small pieces is one of the most effective ways to reduce the risk of introducing bugs.
As we incorporated Two-Factor Authentication (2FA) into our security codebase, we used a phased rollout strategy to validate portions of the picture before moving on. Throughout the project, we leaned heavily on our collaborative review processes to both strengthen and simplify our authentication patterns.
Along the way, we realized that we could integrate our new code more easily if we reworked surrounding access patterns with 2FA in mind. In other words, the 2F itself was relatively easy. Getting the surrounding A right was much trickier.
Lead software engineer Chris LoPresto (right) helped lead the team in building Two-Factor Authentication and App Passwords for Betterment accounts.
What We Built
Two-factor authentication is a security scheme in which users must provide two separate pieces of evidence to verify their identity prior to being granted access. We recently introduced two different forms of 2FA for Betterment apps:
- TOTP (Time-based One-Time Passwords) using an authenticator app like Google Authenticator or Authy
- SMS verification codes
While SMS is not as secure as an authenticator app, we decided the increased 2FA adoption it facilitated was worthwhile. Two authentication factors are better than one, and it is our hope that all customers consider taking advantage of TOTP.
To Build or Not To Build
When designing new software features, there is a set of tradeoffs between writing your own code and integrating someone else’s. Even if you have an expert team of developers, it can be quicker and more cost-efficient to use a third-party service to set up something complex like an authentication service.
We don’t suffer from Not Invented Here Syndrome at Betterment, so we evaluated products like Authy and Duo at the start of this project. Both services offer a robust set of authentication features that provide tremendous value with minimal development effort. But as we envisioned integrating either service into our apps, we realized we had work to do on our end.
Betterment has multiple applications for consumers, financial advisors, and 401(k) participants that were built at different times with different technologies. Unifying the authentication patterns in these apps was a necessary first step in our 2FA project and would involve far more time and thought than building the 2F handshake itself. This realization, coupled with the desire to build a tightly integrated user experience, led to our decision to build 2FA ourselves.
Validating the Approach
Once we decide to build something, we also need to learn what not to build. Typically the best way to do that is to build something disposable, throw it away, and start over atop freshly learned lessons. To estimate the level of effort involved in building 2FA user interactions, we built some rough prototypes.
For our TOTP prototype we generated a secret key, formatted it as a TOTP provisioning URI, and ran that through a QR code gem. SMS required a third-party provider, Twilio, whose client gem made it almost too easy to text each other “status updates.” In short order, we were confident in our ability to deliver 2FA functionality that would work well.
The quick ramp-up time and successful outcome of such experiments are among the reasons we value working within the mature, developer-friendly Rails ecosystem. While our initial prototypes were naive and didn’t actually integrate with our auth systems, they formed the core of the two-factor approaches that ultimately landed in our production codebase.
Introducing Concepts Before Behaviors
Before 2FA entered the picture, our authentication systems performed several tasks when a Betterment user attempted to log in:
- Verify the provided email address matches an existing user account
- Hash the provided password with the user’s salt and verify that it matches the hashed password stored for the user account
- Verify the user account is not locked for security reasons (e.g., too many incorrect password attempts)
- Create persistent authorization context (e.g., browser cookie, mobile token) to allow the user in the door
Our authentication codebase handled all of these tasks in response to a single user action (the act of providing an email and password). As we began reworking this code to handle a potential second user action (the act of providing a login challenge code) the resultant branching logic became overly complex and difficult to understand. Many of our prior design assumptions no longer held, so we paused 2FA development and spun our chairs around for an impromptu design meeting.
With 2FA requirements in mind, we decided to redesign our existing password verification as the first of two potential authentication factors. We built, tested, and released this new code independently. Our test suite gave us confidence that our existing password and user state validations remained unchanged within the new notion of a “first authentication factor.”
Taking this remodeling detour enabled us to deliver the concept of authentication factors separately from any new system behaviors that relied on them. When we resumed work on 2FA, the proposed “second authentication factor” functionality now fell neatly into place. As a result, we delivered the new 2FA features far more safely and quickly than we could have if we attempted to do everything in one fell swoop.
Adding App Passwords
Betterment customers have the option of connecting their account to third-party services like TurboTax and Mint. In keeping with our design principle of authorization through impossibility, we created a dedicated API authentication strategy for this use case, separate from our user-focused web authentication strategy. Dedicated endpoints for these services provide read-only access to the bare minimum content (e.g., account balances, transaction information).
This strict separation of concerns helps to keep our customers’ data safe and our code simple. However, in order to connect to third-party services, our customers still had to share their account password with these third parties. While these institutions may be trustworthy, it is best to eliminate shared trust wherever possible when designing secure systems. Because these services do not support 2FA, it was now time to build a more secure password scheme for third-party apps.
We started by designing a simple process for customers to generate app passwords for each service they wish to connect. These app passwords are complex enough for safe usage yet employ an alphabet scheme easily transcribed by our customers during setup. We then rewrote our API authentication code to accept app passwords and to reject account passwords for users with 2FA enabled.
Our customers can now provide (and revoke) unique read-only passwords for third party services they connect to Betterment. Crucially, our app password scheme is compatible right out of the gate with the new 2FA features we just launched.
Slicing Up Deliverables
Building 2FA and app passwords involved a complex set of coordinated changes to sensitive security-related code. To minimize the level of risk in this ambitious project, we used the feature-toggling built into our open-source split-testing framework TestTrack. By hiding the new functionality behind a feature flag, we were able to to launch and validate features over the course of months before publicly unveiling them to retail customers.
Even experienced programmers sometimes resist the “extra” work necessary to devise a phased approach to a problem. Sometimes we struggle to disentangle pieces that are ready for a partial launch from pieces that aren’t. But the point cannot be overstated: Feature flags are our friends. At Betterment, we use them to orchestrate the partial rollout of big features. We validate new functionality before unveiling it to our user base at large. By facilitating a series of small, testable code changes, feature flags provide one of the most effective means of mitigating risks associated with shipping large features.
At the beginning of the 2FA project, we created a feature flag for the engineers working on the project. As the project progressed, we flipped the flag on for Betterment employees followed by a set of external beta testers. By the time we announced 2FA in the release notes for our mobile apps, the “new” code had been battle tested for months.
Help Us Iterate
The final step of our 2FA project was to delete the aforementioned feature flag from our codebase. While that was a truly satisfying moment, we all know that our work is never done. If you’re interested in approaching our next set of tricky projects in a nimble, iterative fashion, apply here.
More from Betterment: