Betterment Engineering

CI/CD: Standardizing the Interface Dec 7, 2018 12:00:00 AM Meet our CI/CD platform, Coach and learn how we increased consistent adoption of Continuous Integration (CI) across our engineering organization. And why that's important. This is the second part of a series of posts about our new CI/CD platform, Coach. Part I explores several design choices we made in building out our notifications pipeline and describes how those choices are emblematic of our overarching engineering principles here at Betterment. Today I’d like to talk about how we increased consistent adoption of Continuous Integration (CI) across our engineering organization, and why. Our Principles in Action: Standardizing the Interface At Betterment, we want to empower our engineers to do their best work. CI plays an important role in all of our teams’ workflows. Over time, a handful of these teams formed deviating opinions on what kind of acceptance criteria they had for CI. While we love the concern that our engineers show toward solving these problems, these deviations became problematic for applications of the same runtime that should abide by the same set of rules; for example, all Ruby apps should run RSpec and Rubocop, not just some of them. In building a platform as a service (PaaS), we realized that in order to mitigate the problem of nurturing pets vs herding cattle we would need to identify a firm set of acceptance criteria for different runtimes. In the first post of this series we mention one of our principles, Standardize the Pipeline. In this post, we’ll explore that principle and dive into how we committed 5000 line configuration files to our repositories with confidence by standardizing CI for different runtimes, automating configuration generation in code, and testing the process that generates that configuration. What’s so good about making everything the same? Our goals in standardizing the CI interface were to: Make it easier to distribute new CI features more quickly across the organization. Onboard new applications more quickly. Ensure the same set of acceptance criteria is in place for all codebases in the org. For example, by assuming that any Java library will run the PMDlinter and unit tests in a certain way we can bootstrap a new repository with very little effort. Allow folks outside of the SRE team to contribute to CI. In general, our CI platform categorizes projects into applications and libraries and divides those up further by language runtime. Combined together we call this a project_type. When we make improvements to one project type’s base configuration, we can flip a switch and turn it on for everyone in the org at once. This lets us distribute changes across the org quickly. How we managed to actually execute on this will become clearer in the next section, but for the sake of hand-wavy-expediency, we have a way to run a few commands and distribute CI changes to every project in a matter of minutes. How did we do it? Because we use CircleCI for our CI pipelines, we knew we would have to define our workflows using their DSL inside a .circleci/config.yml file at the root of a project’s repository. With this blank slate in front of us we were able to iterate quickly by manually adding different jobs and steps to that file. We would receive immediate feedback in the CircleCI interface when those jobs ran, and this feedback loop helped us iterate even faster. Soon we were solving for our acceptance criteria requirements left and right — that Java app needs the PMD linter! This Ruby app needs to run integration tests! And then we reached the point where manual changes were hindering our productivity. The .circleci/config.yml file was getting longer than a thousand lines fast, partly because we didn’t want to use any YAML shortcuts to hide away what was being run, and partly because there were no higher-level mechanisms available at the time for re-use when writing YAML (e.g. CircleCI’s orbs). Defining the system Our solution to this problem was to build a system, a Coach CLI for our Coach app, designed according to CLI 12-factor conventions. This system’s primary goal is to create .circleci/config.yml files for repositories to encapsulate the necessary configuration for a project’s CI pipeline. The CLI reads a small project-level configuration definition file (coach.yml) located in a project’s directory and extrapolates information to create the much larger repo-level CircleCI specific configuration file (.circleci/config.yml), which we were previously editing ourselves. To clarify the hierarchy of how we thought about CI, here are the high level terms and components of our Coach CLI system: There are projects. Each project needs a configuration definition file (coach.yml) that declares its project_type. We support wordpress_app, java_library, java_app, ruby_gem, ruby_app, and javascript_libraryfor now. There are repos, each repo has one or more projects of any type. There needs to be a way to set up a new project. There needs to be a way to idempotently generate the CircleCI configuration (.circleci/config.yml) for all the projects in a repo at once. Each project needs to be built, tested, and linted. We realized that the dependency graph of repository → projects → project jobs was complicated enough that we would need to recreate the entire .circleci/config.yml file whenever we needed to update it, instead of just modifying the YAML file in place. This was one reason for automating the process, but the downsides of human-managed software were another. Manual updates to this file allow the configuration for infrequently-modified projects to drift. And leaving it up to engineers to own their own configuration lets folks modify the file in an unsupported way which could break their CI process. And then we’re back to square one. We decided to create that large file by ostensibly concatenating smaller components together. Each of those smaller components would be the output of specific functions, and each of those functions would be written in code and be tested. The end result was a lot of small files that look a little like this: https://gist.github.com/agirlnamedsophia/4b4a11acbe5a78022ecba62cb99aa85a Every time we make a change to the Coach CLI codebase we are confident that the thousands of lines of YAML that are idempotently generated as a result of the coach update ci command will work as expected because they’re already tested in isolation, in unit tests. We also have a few heftier integration tests to confirm our expectations. And no one needs to manually edit the .circleci/config.yml file again. Defining the Interface In order to generate the .circleci/config.yml that details which jobs to run and what code to execute we first needed to determine what our acceptance criteria was. For each project type we knew we would need to support: Static code analysis Unit tests Integration tests Build steps Test reports We define the specific jobs a project will run during CI by looking at the projecttype value inside a project’s coach.yml. If the value for projecttype is ruby_app then the .circleci/config.yml generator will follow certain conventions for Ruby programs, like including a job to run tests with RSpec or including a job to run static analysis commands like Rubocopand Brakeman. For Java apps and libraries we run integration and unit tests by default as well as PMD as part of our static code analysis. Here’s an example configuration section for a single job, the linter job for our Coach repository: https://gist.github.com/agirlnamedsophia/4b4a11acbe5a78022ecba62cb99aa85a And here’s an example of the Ruby code that helps generate that result: https://gist.github.com/agirlnamedsophia/a96f3a79239988298207b7ec72e2ed04 For each job that is defined in the .circleci/config.yml file, according to the project type’s list of acceptance criteria, we include additional steps to handle notifications and test reporting. By knowing that the Coach app is a ruby_appwe know how many jobs will need to be run and when. By writing that YAML inside of Ruby classes we can grow and expand our pipeline as needed, trusting that our tests confirm the YAML looks how we expect it to look. If our acceptance criteria change, because everything is written in code, adding a new job involves a simple code change and a few tests, and that’s it. We’ll go into contributing to our platform in more detail below. Onboarding a new project One of the main reasons for standardizing the interface and automating the configuration generation was to onboard new applications more quickly. To set up a new app all you need to do is be in the directory for your project and then run coach create project --type $project_type. -> % coach create project --type ruby_app 'coach.yml' configuration file added -- update it based on your project's needs When you run that, the CLI creates the small coach.yml configuration definition file discussed earlier. Here’s what an example Ruby app’s coach.yml looks like: https://gist.github.com/agirlnamedsophia/2f966ab69ba1c7895ce312aec511aa6b The CLI will refer back to a project’s coach.yml to decide what kind of CircleCI DSL needs to be written to the .circleci/config.yml file to wire up the right jobs to run at the right time. Though our contract with projects of different types is standardized, we permit some level of customization. The coach.yml file allows our users to define certain characteristics of their CI flow that vary and require more domain knowledge about a specific project: like the level of test parallelism their application test suite requires, or the list of databases required for tests to run, or an attribute composed of a matrix of Ruby versions and Gemfiles to run the whole test suite against. Using this declarative configuration is more extensible and more user friendly and doesn’t break the contract we’ve put in place for projects that use our CI platform. Contributing to CI Before, if you wanted to add an additional linter or CI tool to our pipeline, it would require adding a few lines of untested bash code to an existing Jenkins job, or adding a new job to a precarious graph of jobs, and crossing your fingers that it would “just work.” The addition couldn’t be tested and it was often only available to one project or one repository at a time. It couldn’t scale out to the rest of the org with ease. Now, updating CI requires opening a PR to make the change. We encourage all engineers who want to add to their own CI pipeline to make changes on a branch from our Coach repository, where all the configuration generation magic happens, verify its effectiveness for their use-case, and open a pull request. If it’s a reasonable addition to CI, our thought is that everyone should benefit. By having these changes in version control, each addition to the CI pipeline goes through code review and requires tests be written. We therefore have the added benefit of knowing that updates to CI have been tested and are deemed valid and working before they’re distributed, and we can prevent folks from removing a feature without considering the impact it may have. When a PR is merged, our team takes care of redistributing the new version of the library so engineers can update their configuration. CI is now a mechanism for instantly sharing the benefits of discovery made in isolated exploration, with everyone. Putting it all together Our configuration generator is doing a lot more than just taping together jobs in a workflow — we evaluate dependency graphs and only run certain jobs that have upstream changes or are triggered themselves. We built our Coach CLI into the Docker images we use in CircleCI and so those Coach CLI commands are available to us from inside the .circleci/config.yml file. The CLI handles notifications, artifact generation, and deployment triggers. As we stated in our requirements for Coach in the first post, we believe there should be one way to test code, and one way to deploy it. To get there we had to make all of our Java apps respond to the same set of commands, and all of our Ruby apps to do the same. Our CLI and the accompanying conventions make that possible. When before it could take weeks of both product engineering and SRE time to set up CI for an application or service within a complex ecosystem of bash scripts and Jenkins jobs and application configuration, now it takes minutes. When before it could take days or weeks to add a new step to a CI pipeline, now it takes hours of simple code review. We think engineers should focus on what they care about the most, shipping great features quickly and reliably. And we think we made it a little easier for them (and us) to do just that. What’s Next? Now that we’ve wrangled our CI process and encoded the best practices into a tool, we’re ready to tackle our Continuous Deployment pipeline. We’re excited to see how the model of projects and project types that we built for CI will evolve to help us templatize our Kubernetes deployments. Stay tuned.

How We Develop Design Components in Rails Jun 28, 2018 12:00:00 AM Learn how we use Rails components to keep our code D.R.Y. (Don’t Repeat Yourself) and to implement UX design changes effectively and uniformly.. A little over a year ago, we rebranded our entire site . And we've even written on why we did it. We were able to achieve a polished and consistent visual identity under a tight deadline which was pretty great, but when we had our project retrospective, we realized there was a pain point that still loomed over us. We still lacked a good way to share markup across all our apps. We repeated multiple styles and page elements throughout the app to make the experience consistent, but we didn’t have a great way to reuse the common elements. We used Rails partials in an effort to keep the code DRY (Don’t Repeat Yourself) while sharing the same chunks of code and that got us pretty far, but it had its limitations. There were aspects of the page elements (our shared chunks) that needed to change based on their context or the page where they were being rendered. Since these contexts change, we found ourselves either altering the partials or copying and pasting their code into new views where additional context-specific code could be added. This resulted in app code (the content-specific code) becoming entangled with “system” (the base HTML) code. Aside from partials, there was corresponding styling, or CSS, that was being copied and sometimes changed when these shared partials were altered. This meant when the designs were changed, we needed to find all of the places this code was used to update it. Not only was this frustrating, but it was inefficient. To find a solution, we drew inspiration from the component approach used by modern design systems and JavaScript frameworks. A component is a reusable code building block. Pages are built from a collection of components that are shared across pages, but can be expanded upon or manipulated in the context of the page they’re on. To implement our component system, we created our internal gem, Style Closet. There are a few other advantages and problems this system solves too: We’re able to make global changes in a pretty painless way. If we need to change our brand colors, let’s say, we can just change the CSS in Style Closet instead of scraping our codebase and making sure we catch it everywhere. Reusable parts of code remove the burden from engineers for things like CSS and allows time to focus on and tackle other problems. Engineers and designers can be confident they’re using something that’s been tested and validated across browsers. We’re able to write tests specific to the component without worrying about the use-case or increasing testing time for our apps. Every component is on brand and consistent with every other app, feels polished, high quality and requires lower effort to implement. It allows room for future growth which will inevitably happen. The need for new elements in our views is not going to simply vanish because we rebranded, so this makes us more prepared for the future. How does it work? Below is an example of one of our components, the flash. A flash message/warning is something you may use throughout your app in different colors and with different text, but you want it to look consistent. In our view, or the page where we write our HTML, we would write the following to render what you see above: Here’s a breakdown of how that one line, translates into what you see on the page. The component consists of 3 parts: structure, behavior and appearance. The view (the structure): a familiar html.erb file that looks very similar to what would exist without a component but a little more flexible since it doesn’t have its content hard coded in. These views can also leverage Rails’ view yield functionality when needed. Here’s the view partial from Style Closet: You can see how the component.message is passed into the dedicated space/ slot keeping this code flexible for reuse. A Ruby class (the behavior aside from any JavaScript): the class holds the “props” the component allows to be passed in as well as any methods needed for the view, similar to a presenter model. The props are a fancier attr_accessor with the bonus of being able to assign defaults. Additionally, all components can take a block, which is typically the content for the component. This allows the view to be reusable. CSS (the appearance): In this example, we use it to set things like the color, alignment and the border. A note on behavior: Currently, if we need to add some JS behavior, we use unobtrusive JavaScript or UJS sprinkles. When we add new components or make changes, we update the gem (as well as the docs site associated with Style Closet) and simply release the new version. As we develop and experiment with new types of components, we test these bigger changes out in the real world by putting them behind a feature flag using our open source split testing framework, Test Track. What does the future hold? We’ve used UJS sprinkles in similar fashion to the rest of the Rails world over the years, but that has its limitations as we begin to design more complex behaviors and elements of our apps. Currently we’re focusing on building more intricate and and interactive components using React. A bonus of Style Closet is how well it’s able to host these React components since they can simply be incorporated into a view by being wrapped in a Style Closet component. This allows us to continue composing a UI with self contained building blocks. We’re always iterating on our solutions, so if you’re interested in expanding on or solving these types of problems with us, check out our career page! Addition information Since we introduced our internal Rails component code, a fantastic open-source project emerged, Komponent, as well as a really great and in-depth blog post on component systems in Rails from Evil Martians.

Supporting Face ID on the iPhone X Nov 3, 2017 12:00:00 AM We look at how Betterment's mobile engineering team developed Face ID for the latest phones, like iPhone X. Helping people do what’s best with their money requires providing them with responsible security measures to protect their private financial data. In Betterment’s mobile apps, this means including trustworthy but convenient local authentication options for resuming active login sessions. Three years ago, in 2014, we implemented Touch ID support as an alternative to using PIN entry in our iOS app. Today, on its first day, we’re thrilled to announce that the Betterment iOS app fully supports Apple’s new Face ID technology on the iPhone X. Trusting the Secure Enclave While we’re certainly proud of shipping this feature quickly, a lot of credit is due to Apple for how seriously the company takes device security and data privacy as a whole. The hardware feature of the Secure Enclave included on iPhones since the 5S make for a readily trustworthy connection to the device and its operating system. From an application’s perspective, this relationship between a biometric scanner and the Secure Enclave is simplified to a boolean response. When requested through the Local Authentication framework, the biometry evaluation either succeeds or fails separate from any given state of an application. The “reply” completion closure of evaluatePolicy(_:localizedReason:reply:) This made testing from the iOS Simulator a viable option for gaining a reasonable degree of certainty that our application would behave as expected when running on a device, thus allowing us to prepare a build in advance of having a device to test on. LABiometryType Since we’ve been securely using Touch ID for years, adapting our existing implementation to include Face ID was a relatively minor change. Thanks primarily to the simple addition of the LABiometryType enum newly available in iOS 11, it’s easy for our application to determine which biometry feature, if any, is available on a given device. This is such a minor change, in fact, that we were able to reuse all of our same view controllers that we had built for Touch ID with only a handful of string values that are now determined at runtime. One challenge we have that most existing iOS apps share is the need to still support older iOS versions. For this reason, we chose to wrap LABiometryTypebehind our own BiometryType enum. This allows us to encapsulate both the need to use an iOS 11 compiler flag and the need to call canEvaluatePolicy(_:error:) on an instance of LAContext before accessing its biometryType property into a single calculated property: See the Gist. NSFaceIDUsageDescription The other difference with Face ID is the new NSFaceIDUsageDescriptionprivacy string that should be included in the application’s Info.plist file. This is a departure from Touch ID which does not require a separate privacy permission, and which uses the localizedReason string parameter when showing its evaluation prompt. Touch ID evaluation prompt displaying the localized reason While Face ID does not seem to make a use of that localizedReason string during evaluation, without the privacy string the iPhone X will run the application’s Local Authentication feature in compatibility mode. This informs the user that the application should work with Face ID but may do so imperfectly. Face ID permissions prompt without (left) and with (right) an NSFaceIDUsageDescription string included in the Info.plist This compatibility mode prompt is undesirable enough on its own, but it also clued us into the need to check for potential security concerns opened up by this forwards-compatibility-by-default from Apple. Thankfully, the changes to the Local Authentication framework were done in such a way that we determined there wasn’t a security risk, but it did leave a problematic user experience in reaching a potentially-inescapable screen when selecting “Don’t Allow” on the privacy permission prompt. Since we believe strongly in our users’ right to say “no”, resolving this design issue was the primary reason we prioritized shipping this update. Ship It If your mobile iOS app also displays sensitive information and uses Touch ID for biometry-based local authentication, join us in making the easy adaption to delight your users with full support for Face ID on the iPhone X.

Women Who Code: An Engineering Q&A with Venmo Aug 12, 2015 12:00:00 AM Betterment recently hosted a Women in Tech meetup with Venmo developer Cassidy Williams, who spoke about impostor syndrome. Growing up, I watched my dad work as an electrical engineer. Every time I went with him on Take Your Child to Work Day, it became more and more clear that I wanted to be an engineer, too. In 2012, I graduated from the University of Portland with a degree in computer science and promptly moved to the Bay Area. I got my first job at Intel, where I worked as a Scala developer. I stayed there for several years until last May, when I uprooted my life to New York for Betterment, and I haven’t looked back since. As an engineer, I not only love building products from the ground up, but I’m passionate about bringing awareness to diversity in tech, an important topic that has soared to the forefront of social justice issues. People nationwide have chimed in on the conversation. Most recently, Isis Wenger, a San Francisco-based platform engineer, sparked the #ILookLikeAnEngineer campaign, a Twitter initiative designed to combat gender inequality in tech. At Betterment, we’re working on our own set of initiatives to drive the conversation. We’ve started an internal roundtable to voice our concerns about gender inequality in the workplace, we’ve sponsored and hosted Women in Tech meetups, and we’re starting to collaborate with other companies to bring awareness to the issue. Cassidy Williams, a software engineer at mobile payments company Venmo, recently came in to speak. She gave a talk on impostor syndrome, a psychological phenomenon in which people are unable to internalize their accomplishments. The phenomenon, Williams said, is something that she has seen particularly among high-achieving women—where self-doubt becomes an obstacle for professional development. For example, they think they’re ‘frauds,’ or unqualified for their jobs, regardless of their achievements. Williams’ goal is to help women recognize the characteristic and empower them to overcome it. Williams has been included as one of Glamour Magazine's 35 Women Under 35 Who Are Changing the Tech Industry and listed in the Innotribe Power Women in FinTech Index. As an engineer myself, I was excited to to speak with her after the event about coding, women in tech, and fintech trends. Cassidy Williams, Venmo engineer, said impostor syndrome tends to be more common in high-achieving women. Photo credit: Christine Meintjes Abi: Can you speak about a time in your life where ‘impostor syndrome’ was limiting in your own career? How did you overcome that feeling? Cassidy: For a while at work, I was very nervous that I was the least knowledgeable person in the room, and that I was going to get fired because of it. I avoided commenting on projects and making suggestions because I thought that my insight would just be dumb, and not necessary. But at one point (fairly recently, honestly), it just clicked that I knew what I was doing. Someone asked for my help on something, and then I discussed something with him, and suddenly I just felt so much more secure in my job. Can you speak to some techniques that have personally proven effective for you in overcoming impostor syndrome? Asking questions, definitely. It does make you feel vulnerable, but it keeps you moving forward. It's better to ask a question and move forward with your problem than it is to struggle over an answer. As a fellow software engineer, I can personally attest to experiencing this phenomenon in tech, but I’ve also heard from friends and colleagues that it can be present in non-technical backgrounds, as well. What are some ways we can all work together to empower each other in overcoming imposter syndrome? It's cliché, but just getting to know one another and sharing how you feel about certain situations at work is such a great way to empower yourself and empower others. It gets you both vulnerable, which helps you build a relationship that can lead to a stronger team overall. Whose Twitter feed do you religiously follow? InfoSec Taylor Swift. It's a joke feed, but they have some great tech and security points and articles shared there. In a few anecdotes throughout your talk, you mentioned the importance of having mentors and role models. Who are your biggest inspirations in the industry? Jennifer Arguello - I met Jennifer at the White House Tech Inclusion Summit back in 2013, where we hit it off talking about diversity in tech and her time with the Latino Startup Alliance. I made sure to keep in touch because I would be interning in the Bay Area, where she’s located, and we’ve been chatting ever since. Kelly Hoey - I met Kelly at a women in tech hackathon during my last summer as a student in 2013, and then she ended up being on my team on the British Airways UnGrounded Thinking hackathon. She and I both live in NYC now, and we see each other regularly at speaking engagements and chat over email about networking and inclusion. Rane Johnson - I met Rane at the Grace Hopper Celebration for Women in Computing in 2011, and then again when I interned at Microsoft in 2012. She and I started emailing and video chatting each other during my senior year of college, when I started working with her on the Big Dream Documentary and the International Women’s Hackathon at the USA Science and Engineering Festival. Ruthe Farmer - I first met Ruthe back in 2010 during my senior year of high school when I won the Illinois NCWIT Aspirations Award. She and I have been talking with each other at events and conferences and meetups (and even just online) almost weekly since then about getting more girls into tech, working, and everything in between. One of the things we chatted about after the talk was how empowering it is to have the resources and movements of our generation to bring more diversity to the tech industry. The solutions that come out of that awareness are game-changing. What are some specific ways in which companies can contribute to these movements and promote a healthier and more inclusive work culture? Work with nonprofits: Groups like NCWIT, the YWCA, the Anita Borg Institute, the Scientista Foundation, and several others are so great for community outreach and company morale. Educate everyone, not just women and minorities: When everyone is aware and discussing inclusion in the workplace, it builds and maintains a great company culture. Form small groups: People are more open to talking closely with smaller groups than a large discussion roundtable. Building those small, tight-knit groups promotes relationships that can help the company over time. It’s a really exciting time to be a software engineer, especially in fintech. What do you think are the biggest trends of our time in this space? Everyone's going mobile! What behavioral and market shifts can we expect to see from fintech in the next five to 10 years? I definitely think that even though cash is going nowhere fast, fewer and fewer people will ever need to make a trip to the bank again, and everything will be on our devices. What genre of music do you listen to when you’re coding? I switch between 80s music, Broadway show tunes, Christian music, and classical music. Depends on my feelings about the problem I'm working on. ;) IDE of choice? Vim! iOS or Android? Too tough to call.