Mobile usability testing: open source evolution call-to-action

and | July 26, 2012

2010 was “the year mobile con­nected the world,” and ever since, we’ve all been on fast-forward to evolve our mobile design, devel­op­ment and test­ing prac­tices. The dig­i­tal expe­ri­ences we craft are increas­ingly used on a wide vari­ety of devices in a wide vari­ety of con­texts. But the aver­age task suc­cess rates for peo­ple using mobile sites and apps is still way lower com­pared to big screens (62 per­cent for mobile vs. 84 per­cent for desk­top accord­ing to Jakob Nielsen’s Mobile Usabil­ity Update). And cus­tomer expec­ta­tions of fluid dig­i­tal expe­ri­ences that morph to every con­text show no signs of turn­ing back.

So, test­ing our cre­ations in device con­text is NOT optional. As with the desk­top envi­ron­ment, mobile test­ing often needs to be recorded so we can remem­ber what hap­pened, spot the pain points and com­mu­ni­cate the oppor­tu­ni­ties that cul­mi­nate in a stel­lar mobile expe­ri­ence. But by nature, mobile usage hap­pens out in the world rather than in a con­trolled lab with fancy equip­ment, so mobile usabil­ity test­ing equip­ment must also be pre­pared to work out in the real world.

Thus we set about arm­ing our cus­tomer research with portable mobile record­ing capa­bil­i­ties only to find that there really wasn’t an obvi­ous solu­tion. It would be great if there were some sim­ple way to record screen activ­ity on all sorts of hand­held devices, but we’re just not there yet. And, in keep­ing with Lokion’s keep-it-simple usabil­ity approach (no two-way mir­rors, wired mics or cam­eras drop­ping out of the ceil­ing), we weren’t exactly thrilled with the obtru­sive out-of-the-box options on the mar­ket at the time.

So, we made our own.

We knew what we wanted:

  • Flex­i­bil­ity to cap­ture users’ screen activ­ity, hands and facial expres­sions on their own devices in any con­text — i.e. light­weight, portable sup­port for two cameras
  • Unob­tru­sive yet styl­ish design allow­ing nat­ural device use while rep­re­sent­ing the value of our pro­fes­sional ser­vices (after all, the qual­ity of mobile usabil­ity research is way more about the skills of the team than the tools they use)
  • Qual­ity, focused video out­put with­out bloated costs or file sizes

We stud­ied our options and learned from oth­ers pub­lish­ing about their DIY mobile test­ing giz­mos (ex: Mr Tappy sells for $289 with­out cam­era and Mac­Gyver always finds a way). We were par­tic­u­larly inspired by Harry Brignull’s post about mak­ing a home­made “sled” in a day for less than $10.

Two years later, after mak­ing and using our own mobile usabil­ity test­ing sled, hear­ing feed­back from our cus­tomers, a follow-up about our mobile sled from Mr. Brignull, plus grow­ing con­ver­sa­tions across the Web about approaches to afford­able home­made mobile usabil­ity test­ing tools, we’d like to share details about our design as a call-to-action for fur­ther evolution.

Lokion’s mobile sled design (Let’s begin!):

Step 1: Gather sup­plies. Effec­tive and afford­able.
Sup­ply list:

Acrylic sheet, 5.6mm up to $95 per sheet (less with­out cutouts and engrav­ings, makes 7 sleds)
2 Web cams $3-$20 each
Screws  (search: 90190A057) $6.45 for 1 pack of 10
Wash­ers  (search: 96765A105) $3.94 for 1 pack of 100
Painters tape $4.15 per roll
Rub­ber grip spray $5.98 per can


$28-$63 per sled with cameras

Tools (if you don’t already have them):

1 Heat gun $25
Drill $20
Drill bits $5-$10

Step 2: Make sled tem­plate. Place order.

Cre­ate an account at and choose Make to get started. We decided to use acrylic for our sleds because it is light­weight yet durable. We played with sev­eral dif­fer­ent thick­nesses before land­ing on 5.6mm as the right bal­ance for lev­ity and stur­di­ness. Thicker gets in the way and thin­ner top­ples over when cam­eras are attached. We made some clear, but landed on bright red ‘cause… well, it’s our style.

Next design your tem­plate. Check out Ponoko’s 2D Design Rules. Then either down­load a blank tem­plate from Ponoko or feel free to use ours (which you can down­load from Thin­gi­verse)! Our tem­plate pro­duces 7 sleds, each with a series of 5 large holes in the neck for tuck­ing away cam­era cables (they also help make the sled lighter). We exper­i­mented with var­i­ous widths and shapes, but — first thought, best thought — stuck with a sim­ple nar­row rectangle.

With design tem­plate ready, log in and select your acrylic and click the “Upload your design” but­ton top-right. After upload­ing your file, click the “Click here to add a mate­r­ial” link in the “Your 2D laser mate­ri­als” box. You will be pre­sented with a series of drop-down menus:

  1. Choose a mate­r­ial type” — select “plastic”
  2. Choose a mate­r­ial” — select “Acrylic — (your color)”
  3. Spec­ify the thick­ness of the acrylic (we use 5.6mm)
  4. Spec­ify the over­all size of the sheet of acrylic (note: there are 3 sizes p1, p2, and p3 which must match your tem­plate; we use P3 — 790mm x 384mm).
  5. When you are fin­ished, click the “Add this mate­r­ial” button.

Ver­ify the recap of your selec­tions and the price of your order. If every­thing looks good, click the “Make it” but­ton on the left to place your order. In our expe­ri­ence, it takes about 2 weeks to receive your order.

Step 3: Assem­ble and cus­tomize. The fun part!

Once you have gath­ered all your sup­plies and received your order from Ponoko, you are ready to assem­ble your sled(s). Here’s how:

  • Drill two holes: Lay one flat piece of acrylic face down on a table. Drill the first hole in the upper left cor­ner where you will attach the cam­era that records the user’s face. Next, drill a hole in the cen­ter of the sled (between the top two cable holes if you’re using our tem­plate) where you will attach the cam­era that cap­tures the user’s inter­ac­tion with the device. (Note: We chose to man­u­ally drill our cam­era mount holes because we were still exper­i­ment­ing with cam­era place­ment. Now that we’re more cer­tain about that, we may inte­grate these cuts into the Ponoko template.)
  • Bend into shape: Plug in your heat gun and get ready to bend. You are going to make two bends in your sled. The first bend will make the arm that holds the cam­era over the device (about 110-degrees, between the top two and bot­tom three cable holes if you’re using our tem­plate). The sec­ond bend will make the flat shelf to hold the test device (about 100-degrees, just beneath the bot­tom three cable holes if you’re using our tem­plate). To make your bend, care­fully run your heat gun slowly over both sides of a bend area. When the acrylic starts to get soft, it is ready to bend. You should not have to force it. If it doesn’t bend eas­ily, keep heat­ing. Hold the bent acrylic in place for a minute or so to allow time to re-solidify. You may have to adjust the bends some by reheat­ing to ensure you get the right posi­tion for your cam­eras to be angled appro­pri­ately. Don’t for­get to turn off the heat gun!
  • Tape and spray: We tried sev­eral fin­ish­ing touches (ex: glu­ing on no-slip padding) to help keep devices well posi­tioned and set­tled on a thin tex­tured spray sur­face. To apply your sur­face, first tape off the areas where you don’t want grip adhe­sive. Spray vis­i­ble sur­face with a light, even coat of plas­tic grip spray. Let that dry for about an hour and repeat. Do this at least four times to get a good thick grippy sur­face so devices won’t slide around. When the last coat of spray is almost dry, SLOWLY remove the tape.
  • Mount cam­eras: There are lots of small afford­able cam­era options. Beware: many do not focus well on objects nearer than 40cm (16 in). We chose a sim­ple round web­cam with basic focus set­tings and LED lights to help when ambi­ent light­ing isn’t ideal out in the field (care­ful using lights as they can cause glare). To mount your cam­eras, first remove the round base that comes on the cam­eras by unscrew­ing the small screw in the bot­tom. Insert the post of the cam­era into the holes you drilled. It should be a snug fit. Next, get the wash­ers and screws you ordered (the screw used in the cam­era base isn’t long enough) and attach the cam­era securely.
  • Adjust and tweak: Plug it in. Give it a whirl. See how it per­forms. Adjust the angle of the bends as needed to get just the right cam­era focus and angles with­out get­ting in the way. You’ll want the sled with cam­eras attached to feel nat­ural held in one hand hold­ing a mobile device and also still be sta­ble when sit­ting on a table. These angles are the advan­tage of using a sled rather than a tri­pod or doc­u­ment cam­era as they keep the device sta­ble within cam­era view. It’s a bit tricky. Play with it until you’re happy.

We’ll save tips on hook­ing up your sled cam­eras to record­ing soft­ware (we run our record­ings through Morae for real-time obser­va­tion, task analy­sis and clip edit­ing) for another post.

We already envi­sion lots of improve­ments to our mobile sled design:

  • A wider-base ver­sion for tablets and/or a table­top ver­sion for all devices which may be more com­fort­able for length­ier ses­sions and facil­i­tate more nat­ural device usage move­ments (ex: eas­ier switch­ing from por­trait to land­scape device ori­en­ta­tion, more com­plex multi-touch ges­tures, tilt­ing and shak­ing, etc.)
  • Smaller cam­eras with bet­ter gain/exposure/contrast/focus options (ex: abil­ity to pan, zoom, and adjust set­tings from a sep­a­rate observer computer)

Let us know on Thin­gi­verse if you use our design, improve upon it or make your own.  We wel­come ideas, sug­ges­tions, ref­er­ences, resources and ques­tions. That’s what’s so great about open-source design, right?! Let’s share and evolve so we can all cre­ate more usable, use­ful and adapt­able dig­i­tal expe­ri­ences no mat­ter what our cus­tomers are car­ry­ing in their hands.

This entry was posted in Nimble and tagged , , , . Bookmark the permalink.


There is one comment about this post.

Leave a comment

Your email address will not be published. Required fields are marked *