Usability Testing

Usability testing is designed for the benefit of the supplier and the consumer. Writing and designing excellent instructions is the best way that the supplier can serve their customer. The supplier should come away from the testing better understanding how their potential and targeted consumer thinks and reasons through instructions.  The consumer will hopefully benefit from the feedback that they are able to provide to the supplier via the end product that is produced. Suppliers should keep in mind that if the consumer is not able to accomplish the purpose for which the instructions are written, the supplier has failed the consumer.

 

Protocol

In testing the instructions for building Joseph the Technicolor Elephant, we decided to use a Think-Aloud Protocol to best determine our user’s needs. To assure that the testing was thorough, we generated a list of potential problems that could occur concerning the instructions and the end product. From this list we created Exit Questions to give users an opportunity to reflect upon both the ease and difficulties they experienced in constructing each module.

 

Testing

During the testing process, the user talked aloud as they assembled the product. An observation sheet was used by the Usability Tester to observe and evaluate the user as they accomplished each module. The Usability Tester recorded what the user accomplished with ease as well as what caused the user problems or brought up questions. In taking note of problems and questions, the Usability Tester recorded whether she observed the problem or if the user verbally expressed the problem, in addition to whether the user was able to solve the problem unassisted or if s/he requested the help of the Usability Tester. Finally, there was a section for the Usability Tester to take note of how the the user navigated the website, including positive reactions and remarks, problems, complications or questions.

 

Below are samples of the Observation Sheet and Exit Questions (these are smaller and less spacious than the actual sheets):

 

---

 

Observation Sheet:                                                                                                             Name/Number:

 

	Positive Reactions/Remarks	Problems/complications/Qs
Navigation of Website

Module	Accomplished Easy/Well	Problems/Questions
1
2
3
4
Overall

	Figure out/Requested help	Verbally Expressed/I Observed
Problems/questions:	F/R	V/O
1
2
3
4
5
6
7
8
9
10

 

 

---

 

Exit Questions:                                                                                                  Name/Number:

 

The Instructions:

 

Did you find the instructions and pictures helpful?

 

     What was confusing?

 

Is there anything additional you would have included in the instructions/pictures?

 

Were the steps well separated?

 

Would you have taken any of the steps out?

 

Do you have any suggestions for further clarification?

 

The Website:

 

Is there anything we can do to improve the site navigation?

 

 

What could we improve on?

 

What was particularly clear or helpful?

 

What did you like?

 

The Materials:

 

Was it helpful to have the four modules separated into numbered bags?

 

Was the breakdown of the modules within the instructions helpful?

 

Would you find it more helpful to have the instructions direct you to assemble each module separately and then combine the four modules at the end?

 

                        If yes, please explain.

 

Other:

 

Did you feel less comfortable using the instructions because you were being watched?

 

Did having to talk out loud affect your ability/concentration?

 

Did you learn anything about yourself and the way that you use instructions?

 

 

Did you use the quick reference/sidebar?

 

---

 

 

Analysis of Usability Testing

 

Test 1

The first person to test the instructions was Kendra, who has had previous experience with building LEGOS.  She accomplished the first module, the legs, easily. One interesting point of interest is that she did not start as the instructions and pictures directed her to, but rather grabbed the flat green piece right away and began building.  She demonstrated previous lego experience by beginning with the foundation. The instructions indicated to start with the legs of the elephant. Kendra was the only user that started this way.

 

(Note: Due to revisions, the following picture references may no longer be true to what is currently visible in our completed set of instructions) In module 2, The body, Kendra asked questions regarding the first two pictures and it was discovered that the lines directing the placement of the pieces were incorrectly depicted. We informed the user following Kendra of the problem and they were able to accomplish that part of the module without difficulty. In picture (b) it was unclear to Kendra whether the center piece should be placed directly on top of the one below it, or if it should be off-set. In picture (c) she recommended that we describe to place the piece on the "back center". She was, however, able to pick this up from picture (d). Overall she would have preferred if we had given more of a description in the written instructions rather than using the pictures as a substitute for the written instructions by saying "as shown in picture". In step 3, she described the picture as "hard to see" and in need of more of a description. At this point, Kendra also noticed that there was a difference between the brands of LEGOS that we were using. She stated that she thought it was because of this that the structural integrity began to give way and the module to fall apart in her hands. In step 4, she found the orientation of the picture to be difficult because she was unable to see the tail. Her final suggestion for module 2 was to instruct the users to build the body directly on top of the legs, module 1, step by step due to the difficulty of holding the body and trying to attach so many pieces at once. Kendra thought that it would help keep the pieces from falling apart so easily.

 

Kendra completed module 3 without difficulty.

 

In module 4, step 1 and picture 1, it was unclear to Kendra that there was a 2x4 piece. It appeared as though all the pieces were thin. She found step 2 to be easier. Step 3 that depicts 3 red pieces stacked on top of each other was not clear to her. Overall, Kendra felt that the instructions for the ears needed more detailed written descriptions. She said that saying "as seen in picture" was not thorough enough.

 

During the Exit Questions Kendra reiterated that she did find the directions helpful, but that she would have preferred words over pictures. She found it confusing when the directions stated to attach the "object". She was unsure at times what was meant by "object". If she could change anything she would have changed the number of pieces that were in the finished product. She found it to be cumbersome and difficult to build. For the overall layout of the website she suggested that we offer further separated sections for module 2, the body, due to the length and the need for scrolling. She used the side-bar reference and found it helpful but suggested that we add a link at the bottom of each page leading to the next step. Kendra said she found herself counting the pieces a lot in the pictures due to the lack of written instructions. In the end she was happy to finish the elephant, but as she attached the last couple of pieces it fell apart again in her hands and she was frustrated enough that she left it undone.

 

 Test 2

Grace Bettino was our second tester. She said from the start that she did not like LEGOS and did not have much previous experience with them. We showed the testers the finished product before they built it. When Grace saw it, she said she was excited to build it.

 

(Note: Due to revisions, the following picture references may no longer be true to what is currently visible in our completed set of instructions)

The first module was easy for her and she commented that she liked the black marks on the green platform to show where to place the legs.

 

In module 2 she had no problems with part (a) on the first row of 2x4s for the body. On part (b) Grace could not tell by the picture that the pieces should not be right on top of one another, but she had the mental image of the finished product so she expected that they should be offset and built it that way without consulting the picture. In part (c) she found it unclear whether one red piece was supposed to be in the middle of another, but she was able to see it clearly with the second picture. Grace suggested shrinking the size of the pictures so they could be viewed right next to each other. She said that would not only provide clarity and stop her from assuming what would come next, but it would also cut down on the scrolling. She had been struggling with the product falling apart, but said that as she built on each step of the body she found that it became sturdier. Grace suggested that we tell others to hold it in their hands as they build it instead of putting on the floor. Regarding the scrolling, due to the size of the pictures she was initially unaware that if she scrolled down, she would find related pictures to supplement what she was currently doing. If we did not resize the pictures, she suggested mentioning in the instructions to scroll down for further clarity. As she finished module 2 the structure began to fall apart more, and Grace advised that we fill in the middle of the body with more pieces.

 

In module 3, the trunk, Grace said she would have found it helpful to have the pictures side by side.

 

In module 4 Grace struggled with the same issues of clarity with the 2x4 piece on the ears that Kendra did. It was also hard for her to tell if the bottom piece was supposed to be in the middle or not.

 

The orientation of the first picture in module 4 was confusing to Grace, as it was to Kendra, because the picture did not show the tail. In the picture where the body attaches to the legs, Grace requested a picture from the underside to show more precisely and clearly how they attached in an offset manner.

 

The Exit Questions revealed that Grace found it helpful to have fewer words and more pictures. She requested less scrolling in the module with the body, but also commented on the helpfulness of the side bar since it cut down the scrolling in all the other modules. She reiterated her suggestion to build the body module in your hand and to add more pieces to the hollow center for extra support. She found the lines useful that showed one image to be connected to another, but for further clarity in the images, she recommended shading the regions where one piece was projected to snap on top of another. Grace liked having the 4 modules separated into 4 bags and she liked the flow of connecting the modules together with one another. She would not have changed the order of the instructions. It did not bother her that she had to talk aloud or that she was being observed. At the end Grace was asked if she learned anything and she replied that even though she does not like LEGOS, she had fun and she was proud of accomplishing the end product.

 

Adjustments and Changes

It was at this point that the first day of Usability Testing ended. After analyzing the results from day, we changed the problems with the images including the lines and the sizes of the pictures. We decided not to add shadows to the images due to timing. We split the body up into multiple steps with additional quick references in the side bar as well and links on the top and bottom of the pages to get to the next step. Finally, we filled in the hollow center of the body with pieces for stability and added the needed additional steps to the instructions for these as well.

 

Test 3

Eric was the third user. After all of our revisions he was able to get through the building of the product with relative ease. He finished the first module with ease and stated that the black lines were very clear. The only problem he had with module 2 was in step 6 where he could not tell how to snap the legs onto the body. He requested an underside image as well. He liked having the pictures for various steps side by side and found the additional pieces to make the body much sturdier. In step 2 of module 3 with the trunk, he initially made a mistake with putting both blue pieces on top, but he figured this out on his own. In the module with the ears, his only problem was with the first step. He expressed that he could not tell that the pieces were a 2x4 and a 1x2.

 

Regarding the Exit Questions, Eric found the instructions and pictures helpful. He requested clarifications on the step with the ears. He found the links at the bottoms of the pages helpful. Eric commented that the illustrations were clear, particularly helpful and he used those over the written instructions, especially when confusion arose. He found the steps within each module to be clear and well separated. The flow of the assembly was well structured and he would not have changed anything. He did not mind having to talk aloud or have someone watching him. It did bother his concentration.

 

 Test 4

The final user was Krista Kennedy. Krista stated that this project was quite difficult. She completed the first module easily and stated that she liked how we broke up the modules.

In module 2 she said she struggled with mild spacial confusion and experienced slight instability with the product. In step 2 part (b) it was unclear to her that there were 4 pieces in a row. She suggested adding a couple more steps. In step 3, the first image was a bit unclear and it was hard to tell what pieces were being stacked on top of others because they were the same color. Krista suggested discoloring the pieces that have already been assembled in previous steps so as to provide some contrast and clarity in the imaging. The size of the pictures was fine; the issue was the color. If we did change the coloring she suggested adding a notice at the beginning of the instructions. She said she felt triumphant upon accomplishing the module of the body. She still found the step of attaching the body to the legs to be confusing and also thought that an additional view from the underside would be helpful. She found the pictures of the ears and the way theassembly was depicted to be inconsistent with the way we had shown assembly in previous pictures. Her greatest difficulty was with the picture that shows the ears together without the blue connecting piece.

She was very proud when she finished and found the breaking up of the steps on separate screens to be very helpful.

Krista was so articulate in the Think-Aloud portion that we did not need to go through the Exit Questions.

 

 

Final Reflections

After watching all 4 users go through the instructions, we realized that the product took about 30 minutes for each participant to complete regardless of the ease with which they completed it. If we market to participants that are older, we understand that the product in its current form could be found to be intimidating and discouraging due to the complexity and the length of time that it takes to complete.

There was a difference between the first two users and last two who worked on the product after the revisions. The later users had significantly less questions and far less confusion after our revisions. Every user did request that we put in an underside view of the step where the body joins to the legs. This is something that will be taken care of. The Usability Testing taught us how put ourselves in the shoes of the consumer and gave us a new set of tools to work with in the future. We better learned what questions to ask of ourselves and how to observe the users in order to improve upon the product and to have a better product to serve the public.