Better remote user testing

Moderating mistakes I’ve made and how I fix them

Working in Healthcare IT, I’ve had the luxury of doing lots of remote user testing. For the past 5 years, working both B2C and B2B, I’ve regularly been able to share a working prototype with users via WebEx, give them keyboard & mouse control, and observe how they interact.

I set the test up to be a thorough exploration. My test plans outline clear objectives grounded in specific questions. I list success metrics to score after each test so there is a concrete measurement of a design’s successes and weak points. I print out all my questions separately using large font so that I’m not fumbling about while on the call. I’m ready.

The testing session goes well. My adrenaline is up! Then I listen to the recording and I get to really hear ME.

Could be better.

Listening to recordings provides an excellent opportunity to learn from my own mistakes. I have enough experience to know not to “lead the witness” or skew questions in a way that forces a specific result. The grosser pitfalls of moderation haven’t been an issue for well over a decade. But there’s ever more polish to achieve as a moderator.

Isn’t improvement the point of user testing in the first place?

Let go of “perfect”

Foremost in my priorities of improvement is letting go of the notion that I should be doing it perfectly or that perfect even exists. Perfectionism is a set up for failure. It implies a destination to be reached whereupon I can coast along just being wonderful. Nonsense. I focus on constant progress and never stop listening for how to be better.

Ummmm

To combat the dreaded “Umm” in my speech patterns, I employ a couple of techniques. First, have a script for words I want to say. This includes not only an intro script and my test questions but also responses, statements, or follow up questions. These are words I already know perfectly well, but having them printed out in front of me provides an additional memory cue at the moment.

Second, I shut my mouth. I have realized the “Umm” is coming out of my mouth because I’m thinking fast and my mouth is open. When my mouth is shut, no noise comes out. No Um!

Beware of unconscious feedback to the user

It’s natural to want approval and to be liked. If a user hears my voice rise after they interact with a design in an intended manner, they will sense my approval and will go for more of it thus skewing the test results. Even worse, if they hear the disappointment in my tone they may feel bad and not wish to participate again. The scripted responses which prevent me from saying “Um” also help me to keep a neutral voice. The classic “you’re helping us find out what we needed to know” is a great go-to.

I also keep my finger on the meeting’s mute button so that a tester cannot hear breathing changes such as sighs or sharp inhales. I unmute only when I have to talk.

Also, I set up a mirror next to my monitor. By seeing myself I can see my body language. The same way people can “hear the smile” in a voice over the phone, they can “hear” my body language in my vocal inflection. This also helps to minimize saying “Um” because I can see if my mouth is hanging open.

Users are not my friends

I’ve listened to recordings and heard myself getting chummy with the testers. Working with a fixed customer base I’ve had the chance to remote test with the same people on multiple occasions. While doing a contextual inquiry across all customers over the last year, I also personally met a number of these users. I sat at their desks, saw their kids’ drawings on the wall, and watched them juggle tasks. They feel like my friends. Who doesn’t want more friends? It feels good!

It’s good to see myself as part of their healthcare team and be determined to make that work better. It’s good to have a working relationship so that they know I am really invested in listening deeply to their pain points. But these users are not my friends.

When my human instinct to make friends kicks in, the test becomes a two-way conversation in which talking about me fights for equal time with talking about the user. Testing should never be a two-way street. For the duration of the meeting, it’s all about them.

These are specific “friend seeking” behaviors I correct for:

• I crack jokes when I’m nervous to get approval. They’re not investing time out of their day to have a chuckle or to make me feel secure. They want to be part of making the software better.
• Mirroring what a tester is saying by using examples from my own life has never, in retrospect, proved to be useful. I’m better off just listening and repeating what I’ve heard them say to verify listening. Again, a test is not a two-way street and it’s their experience that matters in this conversation, not mine.

Friendly, not friend-seeking, practices that have helped in testing:

• If I’m meeting with a repeat tester, I go back to the previous test recording and make sure I have their job description correct and can say it back to them in their own words.
• Remember any specific issues they have mentioned which impact their work. This could be a recent wrist surgery or knowing that their workspace is so sunny in the afternoon that they can’t see their computer monitor.
• Stay curious. Maintain “don’t know mind” and be open to hearing whatever the user has to share. A conversation that starts with evaluating designs can lead to new insights about workflow pain points that the customer has not yet had a chance to vocalize.

Stay teachable

As some flaws are mitigated, new ones arise. When reviewing a test recording to hear the feedback I always find myself jotting notes on ways to improve my moderating. Practice makes progress.

A note from the Plain English team

Did you know that we have four publications? Show some love by giving them a follow: JavaScript in Plain English, AI in Plain English, UX in Plain English, Python in Plain English — thank you and keep learning!

We’ve also launched a YouTube and would love for you to support us by subscribing to our Plain English channel

And as always, Plain English wants to help promote good content. If you have an article that you would like to submit to any of our publications, send an email to submissions@plainenglish.io with your Medium username and what you are interested in writing about and we will get back to you!