Using online experiments to test improvements to consumer welfare

The Behaviouralist partnered with Open Banking to conduct a large-scale online experiment testing the effect of showing consumers’ their account balance at the point of payment on their economic welfare, decision-making, and purchasing experience. 

Open Banking Implementation Entity (OBIE) offers consumers an alternative payment method for making online purchases by allowing them to pay-by-bank rather than with a debit or credit card. OBIE wanted to test the optimal design of this payment screen from a consumer-welfare perspective, to see whether they could deter consumers from making harmful financial decisions, without adding unwanted friction to their payment process.

We ran an online randomised controlled trial (RCT) with a representative sample of 7,500 UK adults

To investigate the impact of showing consumers’ their bank balance before check-out, we created a fully interactive online shop to realistically replicate the online shopping experience, including the payment process. To also test income effects, participants were randomly assigned a budget of either £100 or £250 with an overdraft of £100, and asked to make purchases across four product category areas that they could choose. During each of the four shopping’ rounds they were shown six popular products in that category and were able to purchase the products they liked.

Testing the effect of different payment screens

Once they selected their desired products, participants would check-out and be taken to a payment screen. Their choice of payment method (by debit card or using Open Banking’s pay-by-bank) varied between the four rounds - see figures below.

During the payment stage, participants were randomly exposed to one of five different versions of the pay-by-bank option. In addition to the control condition that did not display a balance statement, the different versions varied the way in which the account balance was displayed, and varied the size and placement of the ‘continue’ and ‘cancel’ buttons.

Creating a realistic decision-making environment 

When it is not practically feasible to run field experiments (such as testing different payment screens on consumers making real transactions), online experiments which recreate the user experience are a valuable alternative. Aside from recreating the payment and purchase interface, trying to re-create the consumer mindset of a “real” purchase is very important to gain reliable experimental results.

We did this in several ways:

• Participants were informed they would have a chance of receiving the products they chose to incentivise them to make realistic and considered purchase decisions.
• In addition to the products they selected, consumers would receive any leftover balance in cash form if they won the draw.
• These incentives, as well as pricing the products in the online store at 50% off their RRP, helped replicate the spend versus save trade-off that consumers face in real life.
• To incentivise fiscal responsibility, participants were told that going into their overdraft would reduce their chances of ‘winning’ the products and cash by 50%.
• There were several rounds of comprehension checks to ensure participants understood these conditions.

Result: Displaying account balances improved the consumer experience using Open Banking, made people more aware of their account balance, and less likely to go into overdraft.

Most participants (75%) believed that banks should display consumers’ balances at the point of purchase, and those who experienced this in our study were more likely to choose to pay-by-bank when they had the choice.

Being shown their bank statements made participants more aware of their balance at the end of the shopping experience, and more likely to consider how much money they had left when making purchases. The effect was most pronounced for those consumers who had the least experience making purchases online; they were the most likely to choose to pay using the bank option when being shown their balance statement.

In addition, displaying account balances helped prevent irresponsible purchase decisions by making consumers more likely to cancel transactions and less likely to go into overdraft. Displaying a pop-up to warn participants that a purchase would send them into overdraft was particularly effective, leading to a 265% increase in the number of cancelled payments.

The importance of online survey experiments for enhancing consumer protection

By partnering with us, Open Banking were able to test alternative payment screens to empower consumers to make responsible purchase decisions while maintaining the usability and convenience of their payment option. Conducting this online experiment enabled them to discover the effects of different solutions without the risk and uncertainty of testing such changes live and putting actual customers’ finances on the line.. As a result, they could make customer-facing changes with confidence.

We have run similar survey experiments for other projects, where we have recreated interfaces to improve consumer experience and welfare. For instance, we worked with Mozilla to trial different browser choice screens and with Which? to investigate the impact of online fake reviews. If you would like to explore how we might work together on a similar project, please don’t hesitate to get in touch!