A bank considering the rollout of new retail technology naturally wants a new service to help foster customer loyalty and drive more business to its branches.

But predicting consumer reaction to something new can be tricky. For every story of a paradigm-shattering success like the iPod, there's an example of a budget-shattering failure like New Coke.

In 2002, as Wells Fargo was considering moving to envelope-free automatic teller machines, where a check scanner replaced the use of a deposit envelope, it was concerned about customers' reaction. Would they embrace the technology and use ATMs more to make deposits, or would their distrust of it cause them to use the teller window more?

Before committing to a revamp of its thousands of ATMs, Wells wanted some idea of how the move would go over with the machines' users, said Dina C. Fenton, its senior vice president of distribution strategies.

"We wanted to test the pre-versus-post impact of the change so we could understand how customers use the ATM and how it changes their deposit activity," Fenton said.

For help, Wells turned to software called Test & Learn from Applied Predictive Technologies.

"The founding premise of the company is that testing new ideas before you implement them is important, and doing it well is very hard. So, there is a need for software to help," said Patrick O'Reilly, APT's president.

Basically, it is the application of the scientific method to the development of products and services.

"It's about creating a test group and a control group, and measuring the change between them," O'Reilly said.

The idea is that, rather than rolling out a new product and relying on anecdotal evidence to decide whether it is working, a company can use APT's software to generate a granular analysis of the impact a single change has on all manner of things, including customer retention, new account opening and traffic at neighboring branches.

APT's proprietary software system does this by integrating with the bank's internal systems to develop a regularly updated database that can be used to measure and record changes in performance.

The APT system is licensed on a software-as-a-service basis, and the setup takes a good amount of effort. But once installed, it is a tool that can be used over and over, O'Reilly said.

"Banks can use it to support any decision they can test," he said. "Operating hours, capital investments … these can all be tested. This is not meant to be a one-off solution. It is meant to support an ongoing, broader capability."

And because the software is maintained by APT, not housed on the bank's own servers, customers are always using the latest version of the program, O'Reilly said.

In 2003, Wells Fargo made the decision to enlist APT on a trial basis for a pilot rollout of its envelope-free ATMs, installing 400 of the new machines across the San Francisco Bay Area.

"It took several months to set up," said Fenton, describing a process that included as many as half a dozen APT employees and a handful of Wells employees. She estimated that from the time the project was initiated to the end of the four-month pilot program, nearly a year had elapsed.

Part of the preparation and execution of the pilot project involved Wells officials satisfying themselves that APT had sufficient data security systems in place.

The Test & Learn service requires the bank to transmit data to APT regularly to keep the database current and to allow for analysis of changes over time.

Of course, in an age where mishandled customer data can generate embarrassing headlines and an erosion of trust, handing over data to a third party is no small consideration.

Wells had several executives visit APT's data-storage facilities to be sure data security was sound.

APT, of Arlington, Va., has tried to set up a program that can operate with minimal risk to sensitive data.

"We never take identifiable customer information," O'Reilly said. "Each customer gets a dummy ID, and no phone number or other information is available to us."

The setup process was lengthy. However, in the space of that year Wells was able to complete the integration of the Test & Learn software into its own system, turning a vast Wells data warehouse into a ready source of information on customer behavior. "We had monthly data templates and processes implemented after that, so if we wanted to test something else, we didn't have to pull our data all over again," Fenton said.

Judging the pilot program a success, Wells agreed to license the Test & Learn technology from APT. "We have used it for a number of projects centered around customers' ATM experience," Fenton said. The program has also proved useful in other settings in the past several years, she said.

Fenton said Wells has used the Test & Learn program to gauge the effectiveness of changing branch hours, the impact of remodeling branches and methods of alleviating capacity issues in a startup branch.

At a corporation like Wells, the program scales well, Fenton said. The original project, which rolled out envelope-free ATMs, involved a test group of several hundred machines.

However, she said, the bank has used the program for analyses where test subjects numbered only a handful as well as for large studies of consumer behavior that analyzed customers by the millions.

APT's Test & Learn software may not be suitable for every bank mulling a revised retail delivery strategy.

O'Reilly concedes that the Test & Learn analysis becomes less useful as the number of subjects in the control group declines. This means that a bank with only a handful of ATMs or branches is not likely to get much statistically meaningful information from using it.

"Our client base is regional-to-national banks," O'Reilly said.

But size isn't the only variable a bank might want to consider before investing in APT's program.

Fenton said she would hesitate to recommend the service to a bank that did not already have a good internal system for capturing and retaining customer data.

"It is important to have your own analytical framework and a good strong data warehouse in place before you engage with them," she said.

Because the system is also designed to be operated by the bank itself once setup is complete, a bank needs someone with the skill to operate the system and translate the results, which "need a little repackaging to be presented to someone at the senior management level," Fenton said.

While implementing a program such as Test & Learn may represent an initial burden, those who study retail delivery processes find that testing of this sort is underused by companies it would benefit.

"Too many business innovations are launched on a wing and a prayer — despite the fact that it's now reasonable to expect truly valid tests," Thomas H. Davenport, professor of information technology and management at Babson College in Babson Park, Mass., wrote in Harvard Business Review. "Companies that equip managers to perform small-scale yet rigorous experiments don't only save themselves from expensive mistakes," Davenport wrote, "they also make it more likely that great ideas will see the light of day."

Rob Garver, who covered regulatory issues as an American Banker reporter from 1999 to 2003, is a freelance writer in Springfield, Va.

What do you think about this? Send us your feedback. Click Here.


Subscribe Now

Authoritative analysis and perspective for every segment of the payments industry

14-Day Free Trial

Authoritative analysis and perspective for every segment of the industry