We Like to Test Things Here – Bring Your Measuring Tape to Work Every Day

Don’t guess on something you can measure

In an Amazon All-Hands meeting, a biannual all-employee summit in Seattle, Amazon Founder & CEO Jeff Bezos explained a principle that is core to how we do business here. In essence, he told us: don’t guess on something you can measure. He clarified with an example – what would be the point of sitting in a business meeting arguing about how tall the ceiling is? If you ask 10 people in the room, you’ll end up with 10 different heights. It’s far more effective to simply pull out a measuring tape and get the facts. That anecdote plays into a cultural norm at Amazon: we like to test things here – to measure things that aren’t necessarily the norm to measure – all in the name of applying data to our decisions. I’ll give you a few examples of how that’s played out in our HR Shared Services team.

In contact centers it’s generally taken as an article of faith that first contact resolution (FCR) yields a better customer experience than requiring multiple contacts. Makes sense. But because Amazon is prone to questioning things, we wanted to see if this was the case for employees contacting our Employee Resource Center (ERC), and particularly to understand how much more satisfied employees were if we resolved their question on the first vs. second or third contact. So we pulled out some historical survey data and analyzed it to identify what percentage agreed with the statement, “My experience was frustration-free,” segmented by how many times they had to contact us to resolve their issue. The axiom held true, but what surprised us was how quickly experience deteriorated – we lost about 900 basis points each time. Well, there was our empirical evidence, our burning platform, for doing all we could to drive resolution of a contact the first time, at least in this specific service.

Another example relates to translation of our core anchor question, the “frustration-free” scale I wrote about previously. One challenge we immediately found was how hard it was to create a culturally and linguistically relevant translation of that construct. So in some of our supported countries, we engaged in a little experiment. We had three different Amazonians translate the question into their native tongue (unaware of each other’s translations), dropped them into an internal voting tool and sent the link out to a couple hundred employees. Through that method we quickly identified the translation that was most resonant in-country (there was always one clear ‘winner’), which helped us avoid the bias inherent in having a single, un-vetted attempt.

Finally, to earn my ACES Black Belt (Amazon Customer Excellence System – our customer-oriented version of Lean/Six Sigma and process improvement), I completed a project that analyzed the impact of service levels—how quickly we answer the phone—on customer experience. Specifically, I sought to analyze at what point the time spent on hold began to significantly deteriorate scores on the “frustration-free” question respondents answered after the call. I got the chance to geek out on a host of statistical and analytical tools (Hypothesis Testing including tools like Chi Square and 2-Sample T-Tests, Design of Experiment, Correlation and Regression, ANOVA/ANOM, run charts, Pareto, and more), which I’m only slightly embarrassed to admit I absolutely loved. A notable finding was that as a caller’s hold time moved from about 200 to 300 seconds, and particularly beyond – the variation in satisfaction increased quickly. In other words, after three to five minutes, the caller’s experience was going to be much more dependent on variables that were harder for us to control, such as the experience level of the agent, how much empathy was shown, the time constraints of the caller (they might be on a short break in a fulfillment center). So, although service level wasn’t necessarily the primary driver of a frustration-free experience, a long hold time made the experience unpredictable, more subject to the vagaries of circumstance, and thus a feature of the experience we should continue to protect wherever possible.

We continue to measure, test, and experiment. I enjoy this stuff – I sometimes call it “professional play” - the chance to make data and our employee experience a playground where we find unusual ways to Learn and Be Curious about how employees really experience our services. It’s one of my favorite features of this job, team, and company (that and the Amazon narrative – but I’ll simply point you to a great article about that cultural norm rather than write an article about it).

If you’re attending SSON’s Shared Services & Outsourcing Week later this month, please say hello! I’ll be presenting there about these and other lessons we learned from Amazon’s focus on customer experience, and how we applied them to our HR Shared Services employee experience.