APS7: Conducting Usability Research with a Team of One

Christine Kowalski, Web Design Specialist, University at Buffalo


The audio for this podcast can be downloaded at http://highedweb.org/2009/presentations/aps7.mp3


[Intro Music]

Announcer: You’re listening to one in a series of podcasts from the 2009 HighEdWeb Conference in Milwaukee, Wisconsin.

Christine Kowalski:  I really appreciate the plug from Jared. His check is going to be in the mail. Very soon after the conference. As Gwen said, my name is Christine Kowalski. I work at the University of Buffalo. I’m really excited about this conference and every year that I get to come back, it’s just great to see some familiar faces, as well as meet new friends. I’ve been attending this for five years and this is my third year presenting. In the College of Arts and Sciences at the University of Buffalo, I’m a web design specialist and so I wear many hats. I’m actually formally trained in design, in programming, in usability testing and as you can tell today I’m going to be talking about usability research which is my passion.

I’ve had the opportunity to work on both large and small scale usability tests and I got the chance to start on this research back when I was an undergraduate student at University of Buffalo which is 10 years now and I’ve been working at the university full time for about 8. So this project today talks about the fact that I was approached by the dean of the College of Arts and Sciences and he said to me, “We want to do a redesign of our website” and I said, “That’s cool. Let’s do it. But first, let’s actually go and do some usability research first so that we can collect data on what the weak points of our website are and how to improve upon them.”

So in the spring of 2007, I actually began the preview design usability research and I had a total – thank you for your time and budget – of $150. I know there are some vendors here that are talking software that’s tens of thousands of dollars. I had 150. At that time, I was lucky enough to actually have three students helping me with the testing and analysis phase of the project and they were graduate students in the School of Engineering, studying under professor who was in the Human Factors and Engineering Department.

So we presented or I presented the information that we collected to the person who designed the website and they took all of the suggestions and put it into a new design and everybody said, “It’s great, looks perfect. It’s good.” I said, “Well, we’re not quite finished.” Let’s go back and let’s actually review the website again to see if we’ve actually hit our marks and to make sure that we’re getting the numbers that we need in terms of “can everybody find everything?”

So in the fall of 2008, I actually started the post redesign usability project and this time, as you can see, the budget actually shrank. I was going, “Oh boy, I thought my budget was tight last year.” And this time, I didn’t have anyone helping me with the research and the person that quoted the website in PHP, he actually resigned a month after the website went live and I said to myself, “Oh, dear goodness! I don’t know PHP.” I have all this research to do and I’ve got no one to help me with this.So people actually have asked me, they said, “Well, did you really – honestly, did you really have a budget of $100, $150?” Yeah, I did, but I didn’t see anyone’s feathers ruffled when I went slightly over-budget and these are actually the breakdown of the numbers on how I spent money that I was given. And actually, people have come up to me because I’ve presented this before and they’ve actually asked me, how did I go about recruiting the participants? How did I go about developing a methodology to get this done?And so, for the online survey, I actually was able – because I work in the College of Arts and Sciences’ Dean’s Office to write the people and the student or in charge of the student listservs and say, “Can you send out this message? I’ve got an online survey. I need it to get to a lot of people.” I also asked the people of charge of the faculty staff. Listservs, I said, “Can you please help me with this survey that needs to get out?”

And they’ve also posted a link on our homepage just saying, “Tell us what you think about the College of Arts and Sciences website.”So I was able to get a lot of responses from that. I think I would’ve – the thousands and thousands and thousands of requests that I sent out – I got 400, approximately 400 responses back. For the card sort, the usability test and the focus group, since I actually – one of my other hats – is to manage all of the department websites from American Studies to Theater and Dance. And I actually worked really closely with each individual department. So I get to know the faculty pretty well and I selectively emailed outgoing faculty staff and students saying, “Can you join me? I would like to give you lunch. I’d like your opinion on the website.”By the way, if anyone has any other methods or recruitment techniques, I would like to spend the last little bit of time during the presentation today asking people how they have approached challenges and overcome obstacles in trying to get enough research like this. In terms of the methodology, before I conducted and went about doing this research, I actually had to have everyone sign an institutional review board form.Now, how many of you have institutional review boards? OK, if you… And how many of you actually have to get those forms or student’s designer’s form before you can get… OK. A couple? I was actually mandated to get everyone participating in this project to sign a form because these research was being used for my masters thesis and presented at this conference. So you’re lucky if you don’t have to go through the whole IRB approval process, makes your life a whole lot easier.In terms of the card sort, I use an open-ended cart sort. Is there anyone that doesn’t know what an open-ended card sort is? OK. The difference between an open-ended and a closed cart sort is closed, you would give everyone all of the terms without giving them the opportunity for feedback or to anything into the pile. If you want to do an open-ended card sort, you actually give them a blank set of cards along with the pile that you want them to sort through.

So this way if they feel that anything’s missing from the pile or you want them… or they want to rename a category or they feel a category name is missing, then they can actually go ahead and do that. Just make sure that you actually give them the cards with another color ink, like if you write the cards in black, give them a red marker with the blank cards.In terms of the focus group, I actually developed five discussion questions that were designed to foster limited interaction by me, the person conducting the research. Have any of you moderated a focus group? Two. The key thing for focus group is that you actually want to make sure that the designer is not conducting the focus group. That will make it a little bit interesting if they did. I actually have sample question sets for you guys to take a look at and I’m just going to read a couple as I go through each of the sections.For a focus group, an example question would be “Why do you go to the college of Arts and Sciences website?” Another one is “Do you understand the category names or any category names missing?” “What would you like to see added or removed from this website?” In terms of the online survey, I actually wanted to have a 10-question survey. I actually used yes-no questions, radio buttons with no more than five answer choices per question, and then I had general usage questions. Examples of general usage questions would be, “Did you use this search engine during your visits?”Another example of a radio-button question would be “How many times within the last month have you visited the University of Buffalo College of Arts and Sciences website and then you give me five choices.”

I actually collected… and again, this is a restriction of the IRB office. They actually had me collect emails separate from the survey responses; that’s to protect their anonymity. In terms of the usability tests, I had 10 questions and I had them be qualitative and quantitative. Two general questions, like “have you ever visited the website before?” Part two was five-task type questions and what I did there is I actually had five user groups that I wanted to test – faculty, staff, undergraduate students, graduate students and alumni. In this test or for this research, I actually chose to exclude perspective students. I just went beyond the scope of the project. And part three, I had general questions. Some examples of task questions would be, for an undergraduate student, “see what paid and unpaid job possibilities are available to you as an undergraduate student?” Now that’s something that I definitely wouldn’t as a faculty member.So when you develop your task questions, that’s something to keep in mind. I actually, for the faculty, thought of questions and information that really needed to be found on the site like “looking up information on sabbatical leave time.” And another key thing that I wanted them to be able to find and that people are having trouble with that I needed to get validated were “discover what financial resources are available to you as a College of Arts and Sciences Faculty member.” It’s important if you have certain hunches about things that aren’t working on your website, work them in to the focus group. Work them in to the usability questions so that you can get transfers validated.OK. So, how did I go about doing this research all by myself? I actually had to take a step back and I really had to sit down with a notebook, literally with a notebook and a pen, and say to myself, “Well, what’s it going to take?” It’s going to take some money?

Hopefully you have that available. “It’s going to take some people?” They really not used to have people helping me with this. It’d be great to have supervisor approval and you need time.Or in my case, it was… I have these major hurdles to overcome. How am I going to do that? Because I didn’t have enough time during the day. I didn’t have anyone helping me, because as you’ve heard, my other coworker quit a month after the website was launched and I had no students available. I didn’t have a budget like I had to fight tooth and nail to get that $100, and believe it or not, I actually… for parts of this research, I didn’t even have supervisor approval. I think that’s the scariest one in the list.

So how many of you don’t have enough time during the day? OK. I sat down with my notebook and I thought to myself, “What’s it going to take for me to get over this hurdle and what’s going to work for my situation?” So I started writing and I thought, “Well, I could test some people on my lunch break. I could get up early, I could stay late. I even thought I could arrange to do the usability testing on the weekends. Now at this point, I’m sure that most of you are checking to make sure that I had my coffee this morning.Those were actually… Those first three are personal compromises that I actually thought of in order to get this research done. I love to talk about usability. I love to do research on it. If I could do usability testing and work on usability research all day everyday, I would. So I thought of everything possible in order to help me get this research done. And as you heard, Jared talked about the five-second test. I actually went back into my notes and books and websites that I researched from Jared and from Steve Croug and from Jacob Nielsen and I thought, “Yup, that one. I can do that one.” Because I can just grab someone who’s walking in the hall.A lot of times in my office, students get lost and I say to myself, “Hey, you know, I think… Yeah, we can use them.” Just have them take a look at the screen. And then I’ll tell them where they need to go, which floor, Women Studies is on 9th, Asian Studies is on 7th. OK. And I actually thought about… Well, in the spring of 2007, at least, when the professor from Human Factors Engineering actually approached me and said, “You know, we have some students that are interested in conducting research or learning about how to conduct research.” I actually said, “Yeah, I’ll take the help. That’s great.”Now, I’m not so lucky. I actually, last month, was asked to become the usability team leader for University Wide Web Content Initiative that’s starting up and I thought that’s really, really cool and I started to look into who could help me do this research and out of the 5,000 faculty and staff, anyone wants to guess how many people can do usability research on our campus? Zero?

Close. Six, including me and including the person who asked me to become this usability team leader. And I was like, “OK.” So I thought to myself last month, “This is a big project. This one’s big.”How are you going to figure this one out? How are you going to move forward? And what I thought is when I go back to my office, actually in January, I’m going to hold three sessions – probably short, probably during your lunch break – on how to conduct usability research. One session’s going to be on pre-testing in terms of recruiting participants, setting up a methodology, setting up your test questions. One session is going to be actually how to literally sit next to the person and watch them conduct the usability research.And the third session is actually going to be how to analyze that data that you’ve come across. So we’re going to see there have actually been some people that aren’t able to help me now, but in January… They have actually approached me now and they said, “Yeah, we want to learn how to do this so that we can actually jump onboard with this web or this world wide or this university wide web content initiative.” Because there are actually departments in our queue that want to have their website jump onboard and be in line with the university initiative that they’re trying to pull forward. So that’s helpful.

How many of you are facing this challenge? Oh, wow… Almost the whole room. Going back to hurdle #1, I just want to mention something else. I actually held the focus group on my lunch break. I did all the prep work during regular work hours and then I thought, “ Yeah, you can miss yoga for one day. Yeah.” So hurdle #2, get web stat collections to do the work for you. I used Userfly, CrazyEgg, Google Analytics and I actually have a Twitter pool out. I’ll give you the link to that at the end of the presentation. Because I want to know what other web stat collection tools you guys are using.

I’ve heard Silverback. I’ve heard ClickTale. I’ve heard several, and I just want to know who’s using what. There’s also Survey Monkey, Survey Gizmo. I actually haven’t used those two, but for those of you that are one-person shops, just fling the website survey up there using this tool, stick a link on your website, let it run for two months. Online survey. Check if you can do it. I also thought about going back to my notebook, going back to my research. Using webcam… see capture usability testing. The lady who asked me to be in charge of this web usability team, she actually suggested using Skype.

I’ve heard of Cam Studio. I’m investigating that one right now and I’m sure there’s a few others out there that you can… Software programs that you can get, just hook up a web cam and capture the usability test that way. Test fewer participants. I believe that I’ve heard Jacob Nielsen, Steve Croug, and I’m sure Jared, he’s in the back of the room, he would attest to doing five users testing them. That’s all that you need. You start to see patterns develop after four or five users. So what I did is I actually tested five alumni, five faculty, five staff members, five undergraduate students and five graduate students, and I actually did get to see the patterns start to come in and saying, “Whoa, zero out of five people found that one. That needs to go in the report.”

And I actually thought about collecting the data overall longer time frame. To collect those 25 or get those 25 usability tests done, I actually spread the data collection over 5 weeks. It may seem like you have to make some choices about, “can you compromise on time?” “Can you spread the data out over a longer time frame?” I had the chance to do that because I didn’t have anyone helping me.

How many of you don’t have a budget? Oh come on, there’s one hand? OK. I thought… Again, going back to the free tools. I actually found out that Silverback is $60 a month. That’s off the checklist for me. I actually have to go back to the ones that are free. I have to go back to using Google Analytics. I have to use the free account on Userfly. Ask your admissions department. How many of you have admissions? Everybody? Right. How many of you have an alumni association? OK? Ask them what they’re trying to get rid of.

I mean, students love Frisbees. I mean everybody has T-shirts to give away. You know, I Buffalo, I actually… My office gave me… where is it… T-shirts, Frisbees, CD holders, and my personal favorite, ice scrapes. So, see what those fundraising departments are able to give you and say, “Yeah, sure, we can spare 10 T-shirts. Here you go.” During undergrad studies, when I started doing this research, I actually just took notes on pen and paper. I didn’t have the luxury to do video equipment set up in seven different angles and… So that one went on the list and said, “Yeah, I can do that.”

And for the second round of usability tests in the fall of 2008, I actually just said, “Can you help me out for an hour?” I didn’t mention anything about free T-shirts, nothing. I said, “Can you just join me for, you know, half hour or an hour, give me your opinion?” So no matter what website you’re testing, see what you can get for free. Different kind of question. How many of you are supervisors? Cover your ears. OK? I actually… I thought about, in reading research, I thought someone had the suggestion to show your supervisor how it’s not a big deal. Have him or her sit in on the usability… just one… one person, sit in on the usability test.

Just sit there with your pen and paper, take notes and have them observe you. So this isn’t a big deal. That’s how I did it. I also went and showed results of software that I was trying to get buy-in from my supervisor. I actually went signed up behind his back for the free account on CrazyEgg, because there’s like tiered levels. I think that it works that way on Userfly as well and so I started collecting data and I was like, “Oh, this is good stuff.” I brought them into my office a month later and I said, “Peter, I have got the coolest thing to show you. Look at all this information. Look at all this data that I’ve collected. What do you think?” And he’s like, “Yeah, that’s pretty cool.” And I said, “Can we pay for it? Can we get a little bit of money to collect a little bit more data?” And he’s like, “Oh yeah, we’re going to get lots more data.”

So I actually set it up on our 10 sub-pages and now it’s collecting data and the dean of the college, which is my boss’s boss’s boss, he actually wants to see those reports and he wants to see the Google Analytics. So I was like, “Yehey. It’s really cool.” Other suggestions that I thought of and I wrote down on my piece of paper, I said, well, I can have the design team which in my case was the creative design director. I can have her actually watch usability test, the same way I thought of having my supervisor watch use the usability test.

And I also thought, “Shoot, I could get some popcorn. I could get a TV and I could say join the party, we’re going to look at the video for the usability test or the focus group.” So… Now I want to hear from you guys. I actually want to hear if you guys have helpful suggesting. Yesterday I actually went to the sig on usability and I actually heard about people creating personas to solve the problem of not being able to get a hold of students to do the testing. I heard of people using Silverback and Pole Daddy. I’ve never head of Pole Daddy, but…

I actually want to hear from you guys now and if you have any questions for me about this? Yes?

Audience: I have a question on focus groups and how many people were involved in the focus group and what… how many audience segments that you did?

Christine Kowalski: OK. There was a question about how many people were in the focus group and how many… what the audience segment quotation was. This is a great question. I shot for 50. I shot for 50 and I emailed them, like you know, once a week and said, “Be sure you can make it.” The day before the testing, it went down to 11. I was like, “Oh, that’s not good.” The morning of the testing, it actually went to 9. Nine people. And I thought, “Oh, that’s just not good.” I went floor to floor to floor. I have 10 floors in my building. I actually said to the secretaries in the offices, I said, “Who wants free lunch? Who can join me today?”

So you shoot for 50 and you get 9. I would start really high with your focus group. I actually… I really did want 10 faculty, 10 alumni, 10 staff, 10 undergrad and 10 grad. It just doesn’t happen that way.

Audience: The question earlier, when you said about webcams for disabilities, do you mean we should you focus on the screen or would you listen somebody ask you questions?

Christine Kowalski: OK. Good question. We had a question about how to go about doing the actual usability test using the web cam. What you would do is you would set the web cam, I think some max has… Right there, there we go… Some actually have the cam built into a laptop, so… And actually when I go back to my office in November, we’re going to actually try this for the laptop. Point it at the subject, if you have a mic available, if you have an external mic, just plug that in and then just hit the record button and minimize Skype or Cam Studio and actually sit there. Let’s say this is the person. Sit there recording their thoughts, you know… You still want to be proactive.

You still want to be there, at least I do. I love seeing what everybody’s reaction is. I love seeing the squishy faces. I love seeing the, “Hah, I can’t find this. I don’t know what I’m doing.” I love that because it means that I’m going to make a better website or it to the design people who make a better website. So you still want to develop a method, actually, where you have shorthand notes. I actually have codes for when people can’t figure out a task. I circle the number. I actually just do that and then I actually do arrows and dashes… you know, track their pages.

Note on the side how they’re feeling, ask them to think out loud, but that’s… Did I answer your question?

Audience: I was thinking that you wanted to both like get the reactions and get their actions on screen?

Christine Kowalski: Yeah, yeah. Right. I can’t… Right.

Audience: The we can get screen software to run at the same time, maybe that will be out there.

Christine Kowalski: Yes. Sorry, I should’ve clarified that. For this usability, this is coming up. I will actually be running screen capture free, screen capturing soft ware Cam Studio, and that one will actually record the screen and Skype will record the person. So yes.

Audience: We use a software called Screenflow?

Christine Kowalski: Awesome…

Audience: I wanted to just give you the… ask about suggestions. You did a web visibility online survey among the students and faculty at Bucket College and to get equal volunteer at the end of the survey, we ask them that would like to volunteer and provide. We have captured audiences. I think we’ve got about… I think we got 2o people who said, “Yeah, sure I’ll come to do either a focus group or maybe upcoming web visibility testing." This is the nice way to capture the information.

Christine Kowalski: Awesome. Name the software again that you use?

Audience: Screen Flow.

Christine Kowalski: Screen Flow is the software that this gentleman…

Audience: I think it’s around $300.

Christine Kowalski: OK.

Audience:  There is another one called Silverback?

Christine Kowalski: Yes, 50, 60 Silverback.

Audience: Just following up on that, during one 6-month period, we get two different usability tests about focus group areas with students. I mentioned this to you last…

Christine Kowalski: Yeah.

Audience: What we found is we’re doing it one, for just the Biology Department and mainly focus on their undergraduate population and then the other was the graduate school itself when you get a sense of how students use these sites. The graduate school invited students to come for lunch and had people sign up for it and three people showed up. They got boxed lunches and answered questions. The Biology Department at 5:30 in the afternoon said, “Come for beer and pizza” and we got all of the graduates.

[Laughter]

Christine Kowalski: Beer and pizza then, there you go.

Audience: And I the time for students had mattered. They’ve got stuff going on at lunch where as immediately after classes they seem more of ready to have beer or...?

Christine Kowalski: Yeah, like 3 or 4 o’clock timeframe?

Audience: I think even after classes entirely for them. I think a lot of this works for doing research as well. Somebody will just come and… I mean, they knew their population pretty well, but they time it with... This is generally trying to wrapping up their research for the day and so they’re all in the offices and…

Audience: Can you now share with us some of the particular kind of gems, and what were the things that you really learned from the usability testing show and now you apply that?

Christine Kowalski: Yes. There was a question in the back about gems, things that we learned. I actually took away from this that our faculty section was way out of whack, and I mean way out of whack. We kept getting zero out of five… zero out of five, zero out of five. And I was like, “OK, we really need to make these a priority.” And actually, when I… I learned that when I made that prioritize list, I actually labeled them extremely important and marked the category in red and that these need to be fixed right now and they need to be fixed before the design gets implemented.

And then I had medium priority items and I have low priority items, and I did try… When I explained this to the design team, that all of these are important, and you win some battles and you lose some battles. I learned that you can do the best that you can and try and plug for as many things to get fixed and then I learned that you can go back and say it’s still not working in the post-redesign. So keep plugging away.

The other thing that I learned, actually, is to do iterative usability testing. I know. Jared mentioned yesterday paper prototyping. I would have actually liked to do smaller cycles of usability testing and smaller bits of information. I think that’s it. I just got the time sign. If anyone wants to see the question sets, I have a copy with me right up here, but they’ll be posted along with the presentation. You can also take the Twitter poll at twitter.com/cmk22. You can contact me via email. Thank you very much.