One way we earn our pay as advisors is by helping customers improve their take-home returns. While the quality of our advice is important, so is how effectively we are delivering it. There are numerous ways we communicate with our customers, and we build our service to encourage positive investing behavior.

If we want to know we’re being successful at helping customers invest better, we need to rigorously test changes in the design of our guidance. We always hope to help our customers’ investing behaviors, but testing ensures that we’re being successful. These design changes may occur visually or as part of your experience on our website or in our mobile app.

We commonly use randomized controlled trials (RCT)—a framework used in many scientific contexts. RCTs aim to test if a change to an experience i.e., “treatments”—are reliable improvements, or deteriorations compared to the status quo. In pharmaceuticals, RCTs help researchers discover which drugs are effective. In public policy, RCTs help social scientists learn which policies lead to improved outcomes. At Betterment, RCTs help us to develop designs and features that help customers make better investing decisions.

Today, we’ll review three examples of Betterment’s service that resulted from extensive RCT-based testing: email notifications, in-app notifications, and Tax Impact Preview—a Betterment feature that allows you to estimate the tax impact of a transaction. In each case, we’ve used analysis based on an RCT to build an experience that solves for both the positive and negative aspects of customers’ typical investing behavior.

How We Use Randomized Controlled Trials

RCTs randomly assign people to one of two groups: for all intents and purposes let’s say the groups are either:

• A,” a control group (generally the status quo), or
• B,” a new treatment that we think might improve outcomes

If outcomes in the treatment group B are reliably better (have robust statistical significance), we can have confidence that the only difference between the groups (the treatment) likely caused it—not random chance.

When you use an RCT to check your hypothesis, you’re making decisions based on how the world really works. We think it’s one of the best ways to create an investing experience that optimizes humans’ real behaviors.

A Case Study: How Peers Impact Retirement Saving

In a recent and very relevant example, economists and employers wanted to see if they could increase saving rates among employees in employer-sponsored retirement accounts (e.g. 401(k)s).

Conventional wisdom is that when consumers know what their peers are doing, they are influenced by their behavior, and can be influenced to behave better.

The researchers in the study wanted to motivate employees to contribute to retirement plans. They constructed and ran an RCT, and found that providing the employees with information about how much their peers were saving had a very large effect on their own behaviors—it decreased saving!

If the researchers hadn’t run an RCT, they wouldn’t have known about the unintended effects of how people are discouraged from upward social comparisons with their peers. Good intentions are not enough.

In a wealth management setting, advisors are constantly trying new approaches to coaching and encouraging their clients, but they often have very little power to analyze the success of their new tactics. Leveraging technology and our large, diverse customer base, Betterment is able to iterate time and time again to help improve behavioral outcomes over time.

3 Tested Features That Improve Betterment Customers’ Investing Behavior

If you frequent the investment advisor conference circuit (I don’t recommend it), conventional wisdom says that advisors should proactively contact customers when the market drops.

“Don’t wait for unsettled clients to call in a panic. Pick up the phone, send them emails get in touch with them when things get rough and help eliminate the fear.”

The idea is that advisors should preempt customer calls, because part of an advisor’s value lies in explaining what’s going on when the market does something anxiety-provoking. It’s important for the advisor to call the customer first and not vice versa, because in the case of the latter, chances are higher customers will have already made a rash decision.

There is another possibility, however. Perhaps people are good at being long-term investors. It’s entirely possible that they didn’t even know the market had dropped. If this is the case, by proactively bringing it to their attention, advisors are actually causing the anxiety.

From May 21, 2013 to June 24, 2013, the S&P 500 fell 6.05%, largely because of concerns that the Federal Reserve might raise interest rates (our 70% stock portfolio, the average of our customers, fell 6.43%). June 20, 2013 was the largest single-day drop since November 2011.

Following the conventional wisdom, we carefully crafted an outbound email and post talking about what was happening in markets, and why customers should remain invested. However, because we wanted to know whether we were truly helping rather than hindering our customers, we also created a control group of customers who did not receive the email.

What We Learned

We found that customers who read the email engaged in all kinds of activities: deposits, withdrawals, and allocation changes, and they were three times more likely to make ad-hoc deposits. But there was more to it than that.

We analyzed data after the test was run, trying to understand what had driven these different behaviors. In general, customers who were highly engaged (i.e., those logging in more, and therefore more likely to know about the drop) were the ones most likely to react to market moves.

We did help these engaged customers by proactively contacting them, as they were already worried and looking for direction. However, the other un-engaged customers also received an email. It’s possible that we provoked these less-engaged customers into worrying and checking their account when they wouldn’t have done so otherwise.

The diagram below illustrates how this worked. Each box represents a hypothetical person in our customer base.

This chart is for illustrative purposes only.

From this test, we decided that we should target help to customers who need it most, but not worry the ones who don’t. As a result, Betterment uses a more targeted way of communicating during market stress – we use active engagement as a filter. We then target those customers specifically.

Another way we can deliver high priority messages to customers through both our website and smartphone app is with high-priority notification upon login.

In both August of 2015 and January of 2016 markets experienced significant corrections. We set up an RCT that would test high-priority notifications of three different messages across customer groups, and left a control group that received no message. Among those customers in the groups that received in-app notifications, upon logging into their Betterment accounts on their phones or computers they received a form of this message:

We tested three different messages: Two positive, forward-looking ones (one with a no-action message, one with an action message), and one with an explanatory “why” focus.

Condition Title Description
Control None None
Don’t worry Don’t worry, stay the course. Temporary losses are more common now because you’ve been investing with us for a short time. The smartest actions to take are to stay calm and stay the course.
What should I do What should I do when my returns are decreasing? It’s a good time to make a deposit. Deposits rebalance your account tax-free and set you up to take advantage of potential future growth. If you can’t make a deposit, the best thing to do is stay the course.
Why am I losing Why am I losing money? While unsettling, short-term losses are completely normal, and you shouldn’t focus on them. What’s important is to be invested correctly for your time horizon (our advice helps you with this)—and to not react to temporary fluctuations.

What Happened?

To measure results of this test, we defined a negative behavior as a customer changing portfolio stock or bond allocation, making a large withdrawal, or removing all the funds from an account.

A positive behavior, on the other hand, was defined as a customer’s deposit. A deposit could help put our customer’s goal back on track and allow us to rebalance without selling, thus reducing that customer’s tax burden.

We found that the in-app notifications worked: They decreased negative behaviors by 12% for existing customers and 37% for new customers. This included a modest reduction in those making allocation changes (17%), and a dramatic reduction in customers defunding their accounts entirely: 44% for existing customers and 53% for new customers.

However, two of the notification messages didn’t work as desired, specifically in terms of cash flows among existing customers, “Why am I losing money?” and “Don’t worry, stay the course” led to decreased rates of ad hoc deposits and increased rates of withdrawals. These effects were relatively small, but if we’d continued to use these messages over time, the cumulative effect could have been significant.

Without testing our ideas, we wouldn’t have known about this problem, and might have unwittingly proceeded with one of these counterproductive messages in the future.

And of course, this method of communicating during market drops was ideally targeted because it had no impact on customers who didn’t log in at all.

3. Tax Impact Preview: What Customers Do With Timely Information

While the prior two examples involved how we help our customers be better investors through our communications, our service also has built-in features that are intended to help drive better investing behavior.

Betterment’s Tax Impact Preview allows our customers to see the estimated tax impact of an allocation change or withdrawal before they go through with it. Investors usually don’t know what taxes will result from their actions, and Tax Impact Preview provides an immediate and actionable estimate for them.

When we launched TIP, we initially tested if it would impact our customers’ propensity to (1) change their allocations, and (2) withdraw money. The more allocation changes a customer makes, the lower their returns tend to be. Conversely, most customers make withdrawals because they need the money to spend, which is fine.

So we hoped to reduce harmful allocation changes more than we reduced acceptable withdrawals. Over the course of 20 days, about half of all customers saw Tax Impact Preview when making a decision, and the other half didn’t.

How Allocations Changed

We found that customers who were given access to Tax Impact Preview executed about 14% fewer allocation changes than those who didn’t have access to it. Now, because Betterment works extremely hard to minimize the tax impact of changes for customers by using TaxMin, it’s entirely possible a customer saw a tax impact of $0, which wouldn’t deter them from making a change. So we further broke our customers down by the size of the tax impact they were presented with. At a top level, customers who were shown an anticipated tax of$5 or more were 62% less likely to complete an allocation change as those who were not.

How Tax Impact Preview Influenced Withdrawals

We can also measure the effect of Tax Impact Preview on withdrawals. Withdrawals are a more complicated case than allocation changes, since the entire point of investing is to withdraw at some point, hopefully with gains. In fact, the overwhelming majority of Betterment withdrawals occur because customers need to use the funds. So we’d hope to see a limited impact of Tax Impact Preview on withdrawals, compared to allocation changes.

Analysis of the withdrawal data found no statistically significant difference in withdrawals between customers with and without Tax Impact Preview. So Tax Impact Preview was influencing allocation change decisions, but not withdrawal decisions.

We’re confidently testing our way to better investing practices

We hope that all advisors put their clients’ best interest first. We certainly do whenever we make design changes, but the fact is that some proportion of those changes may not work for a subset of our customers, or may not encourage good investing behavior. And there is no way of knowing that’s the case without a testing framework.

And that’s why good intentions are not enough.

We believe that the standard of good advice should include a measure of efficacy: that the advice did actually affect and improve the individual’s outcome. Ideally, such measures would show improvement relative to either not receiving advice, or compared to advice already received. Ultimately, it might turn out that the most effective “advice” is to simply do it for the individual, as we’ve seen with programs like Save More Tomorrow. And sometimes making better behavior the default (while allowing people to opt-out) is the best option.

On my desk at home I have a quote printed out from Abraham Lincoln: “The best way to predict the future is to create it.” I’m comfortable predicting that the standards by which we assess advisors will be raised to include how well they test and genuinely change the outcomes for their customers. By investing in rigorous testing of our advice, algorithms, and design, we’re creating a future where we can have high confidence we’re continually improving and doing right by our customers.

Analysis included in this article is based on Betterment research and is not intended to convey specific outcomes for any individual. This information is for educational purposes only and does not consider individual circumstances or future outcomes.