Usability testing or UX testing used to be something that only hi-tech companies would engage in. Today, demand for prototyping and testing is so high, it’s hard to keep up (which we couldn’t be happier about!). However, usability testing in an almost entirely remote world does raise new concerns around data privacy, security, and inclusivity.
In the “old days”, researchers might have conducted testing in a controlled lab setting, with cameras to record participant’s screens and reactions. Data could be stored on an external hard drive without the use of servers or the Cloud. But in the context of a global pandemic, the old days of in-person testing are gone, which means everything is done through digital tools like Zoom, Lookback, GoToMeeting, and others. So what should you be thinking about when planning your next remote user testing study?
In this post we’re covering:
-
Five best practices for usability testing
-
How to conduct remote user testing, particularly mobile testing
-
Which digital tools are best for mobile testing and prototyping
-
How to handle data storage and privacy
-
How remote testing impacts inclusivity
Usability Testing Best Practices
-
Plan Ahead: Successful UX testing requires preparation, especially remote user testing. This includes not only planning who to research (through careful recruiting) and creating testing scripts, discussion guides, consent forms, and screening questionnaires, but also planning out technology, instructions for participants, and having backups ready should technology fail.
-
Test the Test: In addition to detailed planning, it’s important to test your usability test as well as the technology you plan to use. This can involve running a pilot test with other stakeholders or colleagues to make sure the discussion guide/script and sequence is clear––as well as being willing to adapt your guide after the first test with a user.
-
Establish Rapport: When participants feel comfortable, you will see and hear more about their true needs. Do this by easing tension, reassuring and encouraging participants, providing clear instructions prior to the test, and thanking them for their time.
-
Do Not Lead: A moderator’s job is to listen and observe during a usability test. It is critical not to lead the participant or inadvertently “give them the answer”, as this can skew results. Embrace silence while continuing to encourage your participants, without giving anything away.
-
Encourage Thinking Aloud: This is a standard usability testing best practice, but we need to include it. It is so critical to have participants think aloud as they walk through a prototype, to tell us what they are thinking and to get a full understanding of what’s working and what’s not.
How to Conduct Remote Usability Testing
Build a Prototype
A good usability test starts with a good prototype! Choosing your prototyping tool is important, especially in the case of mobile testing. These are just a few of the tools (among many) that we like to use for creating prototypes; they all work well, but of course, each tools has its own pros and cons.
Once you’ve created a prototype, you’ll want to test it several times to make sure all the links work and there are no dead ends. This is part of the “testing the test” best practice. Get a colleague to run through it to make sure everything works before you start testing with real users.
For mobile testing, we typically create prototypes that users can open directly on their phone (versus having to share our screen with them and trying to simulate a mobile experience on a desktop, for example). This can be done with most of the design prototyping tools out there, and we sometimes even get users to save the prototype link to their home screens with an app icon (here’s an example of how to do this with Invision). This process allows a more ‘true-to-life’ experience, with participants opening the app directly from their phone, as they would normally.
View this post on Instagram
Click through to see more UX tools on our Instagram…
Prepare Your Test Protocol
Once your prototype is built––and hopefully you’ve recruited participants (we usually recommend testing with 5–8 participants)––you’re ready to plan your test. Begin by planning out the specific tasks you’ll ask participants to accomplish, and any questions you want to ask in a test protocol and/or discussion guide. As you create these protocols, it’s also normal to sometimes return to your prototype and add an additional screen here or there, to ensure user flows are as complete as possible.
Prepare Participants in Advance
We suggest scheduling a quick call with participants in advance to set them up with the technology. Ensure they are able to join the web-conferencing tool, and that video and audio are all working. When using a tool like Lookback, you’ll also need to walk participants through installing the plug-in to their browser. If it’s a mobile usability test, have them download the appropriate apps in advance (see the next section for choosing the right tool). This additional preparation can go a long way to ensure a smooth remote testing experience, and it will also help to establish rapport and put participants at ease.
Choose Your User Testing Tool & Run the Test
With remote usability testing, deciding which tool to use is everything. For example, you could run a usability test using simple web conferencing software like Zoom. In this approach, you might send participants the prototype link and the meeting link in advance. They can join, share their screen and video, and click through the prototype as you walk them through the test protocol (and you can record them!). This approach is easy, inexpensive, requires very little learning, and is best for desktop/web prototypes. Alternately, you can select a tool specifically built for usability testing like Lookback.io (our personal favourite). That said, mobile usability testing, conducted remotely, is another beast in and of itself, and the process should be considered separately.
So what’s the best way to conduct mobile testing remotely? Well, there are two main approaches: either the participant is actually using their phone for the test, or they are engaging in a simulated experience on a computer.
Approach 1. (Most Recommended) Choose a tool like Lookback.io that offers mobile apps for iOS and Android. Ask participants to download the app to their phone in advance of the test and send them the prototype link to open on their phone. Then, when the test begins, they can join directly through the Lookback.io app. On your end, you’ll be able to see their screen and video while they interact with the prototype directly on their phone.
Approach 2. If you don’t want to use a tool like Lookback, you can have participants mirror their phone screens to their laptops using a tool like Mirroring360 or Reflector 4. In this case, participants will still open the prototype on their phone, mirror their screen to their laptop, and join any web-conferencing tool that allows recording (like Zoom or GoToMeeting) to run the testing session. One logged in, participants can share their screen with you (which is the mirrored screen from their phone). Video and recording would occur through Zoom, and the participant’s interactions would be on their phone.
At Outwitly, we prefer Approach 1, as it is less burdensome for participants both from a bandwidth demand/internet perspective, and from a user experience standpoint (in terms of the instructions users are required to follow to participate in the test).
Inclusivity & Remote Testing
When conducting any kind of UX research or design research, inclusivity is always important to consider. Of course, remote testing does require that participants have a certain aptitude and comfort level with technology up-front. Participants will also need more recent smartphone models, devices, and a strong internet connection, which presents important questions around accessibility and inclusivity. For example, the type of mobile remote testing described above might exclude individuals from certain socioeconomic backgrounds, older and less tech savvy generations, or those living in rural areas without high-speed internet. Unfortunately, the COVID-19 pandemic has only exacerbated this “digital divide.” We haven’t found the perfect solution yet––these challenges were easier to manage pre-pandemic with in-person research–– but these new issues are certainly on our radar. Career Foundry has a fantastic guide to conducting inclusive UX research that we highly recommend.
Data Privacy and Ethics
With remote usability testing, data privacy and research ethics are still crucial factors that research teams must plan for and seek consent for early on. For example, when using digital/remote testing tools, user and testing data needs to be stored somewhere. Depending which country you work in, you may find that many of these digital apps are storing data on servers that are off-shore (many house their servers in Europe and the US, for example).
A few data privacy and ethical best practices for remote testing and UX research include:
-
Reducing the amount of Personally Identifiable Information (PII) you collect. Lookback.io, for example, automatically collects participants’ full names, email addresses, and IP addresses. Depending on how a user is sharing their screen, other personal information can be unintentionally captured as well (like passwords or credit card numbers). In general, there are a couple ways to get around this:
-
Test with prototypes rather than real websites, to limit the amount of information that can be entered into certain fields;
-
Ask all participants to not enter their full names or email addresses when prompted to do so; and,
-
Simply try to avoid asking questions or including tasks that encourage users to reveal more PII than is strictly necessary from a UX testing perspective.
-
-
Identifying where and how data will be stored and destroyed, and explicitly informing participants of this. Consent forms are obviously crucial for any UX testing or research work you perform. They let the participant know the purpose of the research and how findings will be used, as well as giving concrete steps to withdraw or ask questions throughout the process. Consent forms are also the right space to explain to participants, down to the finest detail, how and where their PPI will be stored, how it will be used, and how it will be destroyed following the completion of the project. Other privacy information you may want to add to the consent form might include:
-
Server locations for all digital tools being used, including the prototype testing tool and video-conferencing tool(s);
-
The name of the person(s) who will be storing the raw data files–and whether these files will be password protected or otherwise secured;
-
The method(s) being applied to ensure full anonymity for all participants; and
-
Where the research findings, based on the data being collected, will be shared or presented following the completion of the project.
-
Of course, not all projects will require the same level of consideration regarding data privacy and storage. It really depends on the nature of the project and the types of information you are gathering from participants. We’ll leave it to your discretion!
Remote usability testing isn’t exactly a new frontier, but it’s certainly much more common today than ever before. With these tips and best practices in your back pocket, you should be off to a great start. Happy testing!
Resources we like…
User Experience Magazine’s Primer on Remote Usability Testing
