Now that the news is out about our new Workstyle apps, it seems like a good time to start talking about some of the things we’re learning as we go through the process of designing and developing these apps. In this first (of hopefully many) posts, I’d like to focus on one area in particular: how we’re making usability testing part of our design and development cycles.
When I worked on the agency side of user experience design I learned one thing very quickly. I learned that usability testing is the hardest thing to sell to potential clients. That is, until they observe their first session. As soon as someone sees a real user interacting with their product, all doubts about the value of usability testing gets replaced by regret that they hadn’t started doing it sooner. There is simply no better methodology for uncovering (sometimes very expensive) usability problems with your product.
Setting the details of recruiting, script writing, and interviewing aside for another post, from a technical perspective doing usability testing on desktop web applications is pretty simple, thanks to software like Morae and Silverback. There is, however, no straight-forward, single solution for doing usability testing on mobile devices. I recently went through the process of setting up our own mobile usability testing process for Jive’s Workstyle apps, so I thought I’d share some of what we learned about what makes a good setup.
First, as with all design projects, it is important to define your constraints. What is essential, and what is nice to have in this process? For us, a couple of things emerged as essential:
- It needs to be a mobile lab. To make mobile usability testing as useful as possible, we need to (duh!) be able to go where users are. We need to visit them in their offices or homes, or wherever they would normally use their devices. This means that we need a solution that is lightweight enough to take anywhere. This leads to the second point…
- We need to be able to record all sessions. This proved trickier than we thought (more on that later). If we’re going to be truly mobile, it is essential to record each session so that the rest of the team can view the videos later. This should include video of the phone, how the user interacts, as well as a picture-in-picture view of the user’s face so that you can see their reactions.
There are other things one can compromise on, but for mobile usability testing we felt that these two points were non-negotiable. So with those requirements defined, we set out to find the perfect solution.
After reviewing and trying out every prototyping tool known to man, we settled on Proto.io as a good solution for most projects. There are so many great options out there that it’s hard to go wrong on a choice of tool. That said, Proto.io has all of the key features that we believe are important to make a mobile usability test useful:
- Full interactivity. To get the best feedback prototypes need to feel as real as possible. There are tools that allows for hotspots on static mockups to approximate a real app, but that’s not truly interactive. An app doesn’t feel real unless you can select dropdowns, type into text fields, and perform the basic functionality of the app. Proto.io is the only tool that I’ve seen that allows for this much interactivity in a mobile prototype. The best part of this is their companion app that creates a true native experience that doesn’t require loading a URL in a mobile browser.
- Flexibility in the animations/transitions supported. It’s important to try out many different solutions to see how they feel (to us and to users), and Proto.io supports any interaction I threw at it (including the fairly scarce “long tap” action).
- Mix and match built-in components with our own visual assets. Proto.io handles this pretty well, which means we can make the fidelity of the prototype as high as our hearts desire.
This isn’t a post on how to use Proto.io, but Smashing Magazine recently published a great introduction called Mobile Prototyping With Proto.io, and it’s definitely worth checking out.
Mobile usability testing rig
If you’re going to record, you’re going to need a camera. So I got lost in mobile usability testing rig guides for days. I was on the verge of giving up and just go with remote testing instead, because it just seemed like such a hassle. But I persisted because I realized it just isn’t a good option for testing prototypes. If you’d like to do part IDI (in-depth interview) and part usability testing, you need a way to be in the room with users and dig deep into certain areas.
I’ve seen some crazy setups — my favorite and weirdest is probably MailChimp’s “hug your laptop” idea. It’s a brilliant hack, but I’m worried about the ability of non-tech savvy users to get comfortable with this, so I needed another solution.
One of the more annoying issues you’ll have to sort out is focus and brightness. With autofocus and automatic brightness adjustment, most cameras are way too smart for this kind of dynamic recording. What ends up on the screen is often blurry and/or awash in a sea of bright light — especially since there are fingers in the way all the time. So you’ll need to download your camera’s OEM software to be able to adjust focus and brightness manually (we used Logitech Camera Settings).
With that out of the way, it was on to the next challenge — how to record the sessions.
Recording mobile usability tests
Recording proved much harder than I expected it to be. It turns out it’s easy to record a desktop screen and a built-in camera (see Silverback, Morae). It’s also easy to record a mobile screen natively (see Camtasia) — but of course that doesn’t record taps on the screen so you can’t tell what a user is doing. There are tools that can do that kind of interactive recording, but it uses a custom browser, and in most cases we need to test native apps.
The final thing I couldn’t figure out is how to record a mobile screen and a built-in camera (for picture-in-picture faces) at the same time.
We were stuck.
The solution we came up with seems like quite a hack, but it ended up working really well, and I’ll definitely use it again. The setup (for Macs) is as follows:
- First, you’ll have to purchase iGlasses for Mac to have better control over which camera is used by what software. Without this your Mac will get really confused. It’s not fun.
- Open QuickTime, create a new movie recording, and set the camera to the mobile testing rig’s camera. Don’t hit record, just let it sit there. You’re just using this as a viewer, not a recorder. Minimize all other windows.
- Start up Silverback and create a new usability session as you normally would for a desktop app.
The Silverback recording will now record the desktop (which has the QuickTime viewer dialed into the testing rig camera), as well as a picture-in-picture using the webcam. Voila!
Here’s what it looks like when it’s all set up and recording:
With this setup we accomplished all our goals. Everything you see in that photo can easily fit in my backpack so we can take it wherever we want. And since we’re recording all sessions the entire team is able to view the sessions once they’re all done. I do usually make highlight videos, but hey, if someone wants to watch 5 hours of usability testing on a Friday night, who am I to judge!
I hope this guide will spark some ideas on how to get mobile usability testing going in your own work. And please, if you have a better solution for any of these steps, let me know!