TPR11: Show Me the Data: Usability-driven Web Design

Jason Alley, Instructional Technologist, Lafayette College

Kenneth Newquist, Web Applications Specialist, Lafayette College

The audio for this podcast can be downloaded at

[Intro Music]

Announcer: You’re listening to one in a series of podcasts from the 2009 HighEdWeb Conference in Milwaukee, Wisconsin.

Kenneth Newquist: Hello. My name is Kenneth Newquist. I’m a Web Application Specialist at Lafayette College and this is my colleague Jason Alley.

Jason Alley: Instructional Technologist, so I kind of feel like I don’t belong in this room but thank you for allowing me to be here.

Kenneth Newquist: We worked together quite a lot a number of different web projects. I end of using usually doing the back end stuff. He usually ends of doing the front end CSS stuff. And we’re not going to be getting into a lot of code and stuff. I know Propeller Hats are required here, but what we are going to be trying to talk about is things that will save you time. So, that if you’re a web developer or if you are graphic designer, the things that we’re going to be talking about as far as usability is saving you time, so you don’t have to do the job over again.

Here’s our problem. It was time to redesign the ITS website, which is our Information Technology Services website and we’ve got a little committee together. I think that this process is probably applicable to everyone’s college and anyone who’s ever worked on a web project with a committee. OK, so everybody on that committee has opinions about how the site should go.

We have opinions on what works on the site and what doesn’t work on the site, and they are really, really opinionated about what doesn’t work. And they are willing to live and die by what they think was wrong with the site. But the thing they don’t have is data.

They say well, the users will never think to do this or I know that the users do that. But, that’s just the hunch. They never actually sat down and run them through a test.What our goal was, is to replace opinions with facts, find out what we don’t know yet and discover what the users do, not what we think they do.

And my grandfather was a carpenter and he had a great rule of thumb, and this is great for anybody who’s worked on any wood working project. It’s measured twice and cut once. And the whole idea being it is a heck a lot easier to make sure you got it right the first time than having to go back and do it over and over again. With that in mind, and because we know that if we don’t keep you engaged, you will jump on Twitter and then all will be lost. I need a guinea pig.

Jason Alley: We’re going to do usability testing in our website right now.

Kenneth Newquist: Yes. I knew this is right.

Jason Alley: All right. Come on up. We are using software called ScreenFlow to capture what you’re going to do on monitor.

Kenneth Newquist: I want to prophasis by a little bit. I want you get back to the committee for a second while Jason is getting set up with this. One of the really cool things we did on this committee. Now, when we first start with a committee, Jason and I, because we’re kind of to lead guys who we’re doing stuff on the website. It’s kind of became the Jason and Ken show.

People felt like they didn’t necessarily have an opinion like they were having an impact on the site. Because we would come up, when they say well, this is what we think you should be doing and they would say well, you know we’re getting disagreements or whatever at any case. It felt like we’re kind of driving thing.

What we did is we took two of the people who complained the most, but in a good way, right. We asked them to do the user testing. The folks who were the most opinionated and who had like those most in greens ideas about what the site might do, we asked them to actually go into the user testing. And the most awesome thing about this is, is that that they changed their minds.

Jason Alley: OK, so are we ready to get started?

Kenneth Newquist: I think we are.

Jason Alley: OK. What is you’re name?

Steve: I’m Steve.

Jason Alley: Steve, I’m Jason. Nice to meet you. OK. Steve, I’m not going through the whole spill here but essentially, we’ll start up with asking, “Are you OK with us recording you and using this for purposes down the road?” Great, OK. That’s the question you might ask somewhat.


Jason Alley: Steve, could you answer for me your overall comfort for using computer software? On a scale of one to five, five being the most comfortable. OK, great. I’m going to ask you to do a task for us on our ITS website and the task that I’d like you to complete is you want to edit a video and would like to use iMovie, which labs on campus have it installed? Please while you’re walking through this, talk through what you’re thinking.

Steve: Well, I probably wouldn't press this here...


Kenneth Newquist: So, I’ll probably click on software and lab software looks like a good bet and... Is iMovie I’m looking for.

Steve: I think it is. What did you just use this?

Kenneth Newquist: I use the browser’s built in find utility.


Jason Alley: Can you tell me what lab isn’t so?

Steve: It looks like it’s in SkillMan.

Jason Alley: OK, great. You completed the test. That’s it.


Jason Alley: Can I do another?

Kenneth Newquist: Sure.

Jason Alley: We’d like one more volunteer. We’d go though one more questions just to have one more quick demo. Come on, one hand. Be brave. Come on. All right come on, up thanks. What is your name?

Stephanie:  Stephanie.

Jason Alley: Stephanie. Stephanie from?

Stephanie: Noel Levitts.

Jason Alley: Noel Levitts. All right, come on have a seat. And we’ll just continue recording. Stephanie, I’m going to ask you a similar question. Again, talking through what it is you’re doing as you’re walking through this. Let me get to my question. The question that I’d like you to answer for us and we’ll start you back on the homepage, so we’re starting from the same point. Is access to the wireless network available on the second floor of Marco, Marco was a building at Lafayette College. And could you kindly hold this microphone.

Stephanie: Sure. I think I’ll start with facilities, you mean the building is wireless access. So, I’ll click on facilities, labs, not wireless and there’s Marco, so on the first floor.

Jason Alley: Actually, I ask you if it’s on second floor?

Stephanie: On the second if Marco, I’m sorry.

Jason Alley: What is the answer?

Stephanie: It’s no.

Jason Alley: All right, great. Thank you.



Jason Alley: Thank you Stephanie.

Stephanie: That is not great.

Kenneth Newquist: We were not testing you. We are testing the website, which is very important because people get really stressed when you put them on a testing environment.

It’s really one of the things we do when we start out is to say, “Look, this isn’t about you. This isn’t about how well you surf the web. This is all about us and about, well more specifically, it’s about the website. You’re not going to hurt our feelings if you do a bad job. You’re not going, well if the website does bad job.” And we’re really trying to be positive with regards to that.

We actually get a couple of different forms to test and train, gather data when we back up a little bit, so you see the screen here. We did a campus wide survey which we started off. We kick things off for you to come in the usability testing with the campus-wide survey, just asking people what they thought of ITSI, did they use it that sort of thing.

We had 10 usability-oriented questions and obviously this is a whole very subjective. All of this is we try to get us an idea as to how people might be using the site. We only had two questions at the end of the survey that said, “Hey! Would you like to volunteer?” And that was crucial because I can’t remember the exact number of that I had. I think it was about 40 people respondent said, “Yeah! We’d love to be able to help you.”

So then when the question gets raised, “Well, how do you find people to volunteer to do this usability testing?” Well, that’s how we found them. We use a tool that we had at Lafayette called Opinio which is just a web-based survey tool. And we offered as an incentive, $10 or $5 coffee cards for the café or coffee café that we have in Skillman Library.

Some of the questions were “What do you use the website for, how easy it for you to find specific things in the site, do you search or do you browse or do you do both? And then the last question, do you want help?” Here’s the cooler thing. We have, help me here Jason. How many people do we have on campus?

Jason Alley: About 2300 students.

Kenneth Newquist: We’ve got 319 responses to our survey, which was really good. I mean, they were flying in the first like two days. And then we did follow ups... I think we did a follow link a week later, right? Do we do the second survey?

Jason Alley: No. We didn’t. We never...

Kenneth Newquist: This was the initial survey, right?

Jason Alley: We never send in a second message out.

Kenneth Newquist: This was all probably within the first week of the survey went out. Now, this is actually unique when we did a similar survey about four years earlier. We also got 200 something responses. We were just astounded that was many people responded to us who really like to give us feedback.

That having been said, 80% said that they used the site occasionally to rarely. When they did use the site, they’re using it to access network services and to download software, which weren’t things that we’re necessarily thinking will going to be high priorities for people who were visiting the site. And this also helped us when we’re trying to determine exactly what we need to test when we do the usability testing which is why we do the survey. Anyway, obviously people didn’t use this as much as we wanted to.

A lot of people were using the services per quarter or so by way of example. One of the ways that people knew to get the banner self service on campus was through the ITS site.

They didn’t know how to get to it in any other way, so we make sure that we didn’t lose that one when we redesigned the site. Forty percent of the people who responded to us said they didn’t use search, which this kind of goes against some conventional wisdom with some of the folks we've talked to on our committee but they just assumed everybody would search.

Why search? Doesn’t everybody search? Well, then the answer is no. And then again, people want help. When we did our usability testing, we broke it up into a couple of different tasks. We had five common tasks. Well, that means is that everybody whom we give the test with has the same tasks they need to accomplish. Then we had five specific to faculty and staff and then five that were specific to students.

The set up, well you can see our set up. We had an iMac which has a built camera, right. And then we use the ScreenFlow which is a really great software for capturing on video, on screen video and we can show that up to folks later if you’d like. It costs about $100 and we had two people who administered the tests. We had a proctor who was actually asking all the questions and then we had a recorder, a secretary sitting next to him or her and actually writing the responses so we had a written record of what was happening.

Example tasks were, how do you register your gaming system on campus? Obviously that was more of a stupid question. Do you want to know how your guest get connected in the computer to the Lafayette network, which turned out to be probably one of the worst usability tasks on our entire tests? And how do you register for technology workshops?

They’re all pretty straightforward things. And what we learned? Again, many don’t search. Those who do search, search is working pretty well as long as they were putting the right terms. We had some people who were just using terms that we didn’t things of as far as key matches go. And this is one of those things that you say, “Well, yes! Of course, it’s obvious.”

Dense pages are hard to scan. And we are actually shown an example of this in a minute. We would have page with text on it and the answer would be on the page but it was buried in the middle of the page that was like about yet long

Jason Alley: The cursor would literally go right over the length as they were scanning the page in.

Kenneth Newquist: So which all of us say, “Well, of course you shouldn’t have long pages.” But I challenge you to go back and look at your policy pages on your ITS website and see how good they are.

Audience: That’s not even too bad.


Kenneth Newquist: The category pages on our site, which I didn’t get a chance to show up because nobody actually went to them, are difficult to browse. And there were big lists with teaser texts and people were expecting to have some organization there and there were no organizations there, which was organized how recently they’ve been posted.

Getting to the page was half the battle. It looks like we kind of have ringers in here because they actually found the stuff pretty quickly. Others did not. For them clicking around was really a challenge.

Jason Alley: Well, in the page that Steve found which lab Imovie was in. We know that that page is not the most elegant page but it’s functional and people could quickly find that iMovie was in Skillman. So that was one of the pages a lot of people would like to have reworked. But getting to it was the hard part. Once you got to it, you could figure out the answer rather quickly.

Kenneth Newquist: When we were redesigning the site and we’re still working on that. Where do you out your efforts? Do you put it into the page that’s mostly working or do you focus on the fact that no one could figure out to actually get a guess on the right network? Aside from the students who said, “Why just sign them on with my ID?”


Yes, grown. We also did open card sort. We’re still actually analyzing the data from this. We had 29 students. The idea of open card source is that we had 29 index cards. Each of them was numbered. We basically gave the cards to the people and said, “Here, tell us what categories they should be in.”

The idea behind this actually near to the next. We had 16 volunteers worked from within the department and I think from the volunteer list. One of the cool things with the cards were, was to demonstrate to people that people think differently which again these are obvious things, right.

If you think about it rationally and kind of take a step back, everyone goes yes well of course people think differently. Except that when you’re sitting down and you’re watching someone do this, and then they go I would never had thought to put wireless on the facilities because that’s service for me, so I should be under services or I want to know how to attend to workshop.

Well, clearly that should be under help. Not necessarily right. The things that were being able to demonstrate was especially to the people on the committee that what they thought the category should be isn’t necessarily what the users thought the categories will going to be. It was useful for helping the identify trends, so we could start to see where people were aggregating things.

We have a category... We have a tab on our website called “Web Services.” And the reason why we have something there called “Web Services” is the way too much of “Web Services” because we went and did the card sort. People kept putting the same things into a file. They didn’t really know whether they should call that file but they were all related the web in some way.

We ended calling it “Web Services” because no one can come up with a better name for it. But the point was is that everybody who did the card sort ended up putting the same things into that files who gave us an idea what to do. And then the last thing is that it was great to be able to do all the testing and this didn’t take us long to do.

This was mostly having people come in the test themselves took about half an hour analyzing the data, it was really what is taking the most time because sitting down looking over those videos. When you have 10 people who do usability testing, each one of those tests takes between 30 and 60 minutes. That’s the day just to go through and review the video.

Audience: Did you say the usability test is pretty good small?

Kenneth Newquist: It would depend on how. We had 10 questions. Each tasks takes about 2 to 3 minutes, all told. It depends entirely on users. Some people were able to blow through it in 20 minuets. Some people it took 30 minutes.

We also did use usability testing for Moodle as I have part of the group called “CLAMP” which is a Collaborative Moodle Development Project and one of the projects that we have is doing testing on Moddle and for that took similar number of questions but that one can take up to an hour.

Jason Alley: OK. We’ll going to show you a clip of one of our usability tests that Steve and Stephanie had done, except it’s Youtube. It’s someone from our own campus. And the task that was asked of her, it’s actually in the video so I’ll go ahead and roll the clip it takes about just under a minute.

Kenneth Newquist: What we’ve asked her to do is actually find the home licensing agreement for Microsoft Office. Can you actually Microsoft Office at home? What she did is, she went to the licensed software page and now she’s clicking around and she’s trying to find the right answer. She starts...

Steve: Can you do this before?

Kenneth Newquist: Yeah. That’s why I chose her, it’s awesome.

Jason Alley: Yeah.

Kenneth Newquist: She does this for about 6 minutes.

Jason Alley: And then she finally does a search. She did another search before that didn’t give her the results she wanted. And she ends up finding here pretty sure.

Kenneth Newquist: Right. This is Microsoft Home installation.

Jason Alley: OK.

Kenneth Newquist: Now, but here’s a thing if you want to advance just a little bit there, Jason.

Jason Alley: What do you want to do?

Kenneth Newquist: Now, I want to go the page when she clicks to the license software. OK. It’s hard to see on this. We might actually want to switch to the actual page. This block of texts is actually a paragraph. This isn’t pretty sure a page right.

This is a block of texts that explains the Microsoft Home Office policy. It’s a blur but actually it states exactly what she needs to know and she completely missed it. This is a hyperlink that would take her to the place where she can go and actually. I think it fill up request to get a software right.

Jason Alley: No, it takes here to another page.

Kenneth Newquist: I think in another page it explains in more depth.

Jason Alley: But nearly on policy on it.

Kenneth Newquist: This is three paragraphs, right. This is a short page. Clearly, it had to browse.

Audience: Do you, when are you getting questions?

Kenneth Newquist: No. It’s okay. We’ll try to keep this short so that we’ve time to ask.

Audience: Do you sense that the combination of paragraphs versus the list is the cause of that trouble.

Kenneth Newquist: I think so and I think if we’re both at list it would have worked.

Audience: Or no list?

Kenneth Newquist: You mean having the board in the bottom.

Jason Alley: No, just no list.

Kenneth Newquist: I know, I think, they were have just gone glaze over and then click on one of the navigation links. Now, I will say that the secondary navigation turned out to be used more often than I possibly expected to do. But I think the bottom list does do the mock. I think they have the tendency to skip to the bullets.

Jason Alley: All right. We’ll going to skip our second example too because the audios is not.

Kenneth Newquist: Right. She has this great quote that we say, “OK, can you find this particular...,” although here facial expressions are priceless.


[Cross talk]

Jason Alley: Exploiting our coworkers.

Kenneth Newquist: Well, no I mean pictures speak a thousand words here. Just watch and this kind of goes to... Why would I want to do the video picture? Why do I want see what she’s doing. This...


Kenneth Newquist: And I guarantee your users are doing the same thing, right.

Jason Alley: She’s saying right now when I come to the site I just end up getting lost clicking in all the tabs and I can’t find anything. She finally does begin the tasks but literally with that expression, so she’s not very happy about it.

Kenneth Newquist: I just want to kind of talk briefly about the value here for Jason and I has been I think as developers, we have all had experiences when we had worked on a project, we completed the project, we deliver the project, the project had gone live and then users, “Hey! You know what, this didn’t work.” You then after you re-factor your code or you have to write new code or you have to go back and got the whole thing and now you’re kind of back to where you started from.

I know an argument that comes up because I don’t have time for usability testing. And I think that the thing is that, especially depending on the project, you may not have time not to do it because you can find the answers to somebody’s solutions ahead of time.

The big thing is there’s a book that Jason and I both like called by Steve Krug called “Don’t Make Me think.” And I know that not everybody agrees with Steve, but I think he has a good point in that same data is better than no data. Even if you only take two people pulling some your coworkers and say try and find this and try and use this form.

You’ll going to learn a lot of stuff. Another saying that kind of came up, I was talking about trying to find guest access. And this is just kind of a little story how things have evolved at Lafayette. We have a passwords reset utility.

It’s only in that network service group created and it’s found at, so you can go and you can reset your password thorough this thing, code utility. Ye can also set up your challenger response. It’s another little thing that cut it down, add on to it. OK, cool.

Now, we have another thing. Well, it’s all dealing with password management. Right, so now that utility also allows you to have a guest so that you can use that utility to request a guest because right you’re sitting up on account and passwords. You have a logic exploring on this except that our users don’t think like that.

When we did the usability tests, they had no idea how to get to the page because well that page didn’t really talk about guest accounts stuff like that. As we redesign the site, we really need to think about how to improve that. We just wanted to briefly address kind of some of the things that can hold you back and I think have held us back to until we just kind of powered through it.

One is that you have the time which you know it’s like anything, right. You value enough, you can find the time. We don’t have the money which I’ll deal right now. I think depending on what the hardware is you have around. You probably do have enough to be able to do it. We don’t have the space and we haven’t done this before like getting out my self.

If you have a laptop with a camera and $50 to $100, you can do this. That’s on the Mac. There’s another program called Silverback which is a really nice tool. It doesn’t give you a lot of options with regards to capture but it’s cheap. ScreenFlow is very nice that allows to really pretty up the usability tests because you don’t having any two radar effect whenever somebody clicks on something. You can have a display while they type when they do the test.

On Windows, the best offer we found was Camtasia Studio and that  will run you about $300, but that net we’re not talking how a huge amount of money. You don’t need a dedicated lab. I think this is big deal. All you need is a quiet space. If you can get a conference room or if you can get a classroom or whatever in your laptop with it.

Jason Alley: Apparently, you can do it in front of 100 people too.

Kenneth Newquist: Yes, you can do that too. And there’s a first time for everyone. But and here’s the thing. The testing takes time to do. I don’t want to underestimate that. I don’t think it’s a huge amount of time and it probably takes a lot less time than you think it does.

Analysis really does take a lot longer. Jason and I had to come up some time to really work on this. And it takes much, much longer because when we originally pitch this. We want to be able to show the finish version as well, but as always we got sidetrack by other projects and we’re still working on the redesign.

But I think that the most useful thing for us, is we don’t have this data so that when we’re having meetings and we’re talking to people who have very striking opinions about this, we can go back to videotape as it worked and show them, you can say OK, let’s look how that task went.

The other thing that I would recommend is, well depending your boss. Do usability testing with your boss. If your boss doesn’t see the value of you taking the time to do usability testing, have him or her sit down and try to find some stuff on your website but it’s outside of their normal per view.

Like for example one of boss is, he always surf the site for policies, so he can always find what he wanted because he always had to find policies. Once we had started doing these tasks, he started realize that things were perhaps more difficult. Questions?

Audience: We looked to those videos that take so long to analyze. What are you looking for that?

Kenneth Newquist: Well, I think there are a couple of things that we are doing. And I think the big lesson we learned with video is that we have recorded how long each tasks took but we what we didn’t know is when each tasks took place so we didn’t timestamp them, which made it more difficult to go back and figure out. OK, I want to compare each of these people how they did that difficult tasks. Partly we’re just figuring out we’re people had done a particular task.

The other thing was when what we would do is we would just sit there and we would watch and we would stop the tape and see what they’re actually doing. What she searched on that? What did she click through? And then we would pause and kind of wrap a little bit on what we were seeing.

OK, so how can we fold this into our design? What she thinks she may be thinking about? And I think it has given rise to things like, “Well, should we replace the homepage in big search button.” If we do, we’ll test it to see if it actually works.

Jason Alley: Something else is we can collaborate our data against people say they do things in one way but they may not actually do the things that they say. When we’re collecting, we had 320 responses from the web survey, they say they search but do they really search, it’s a way to compare are you really doing what your... Now I understand that does not complete apples-to-apples in a comparison but it can give us a sense.

Audience: How can you do that sort of thing? Can you go back on your blog on that? The idea being, "Okay, some concerned people usually use listservs." And a few see that as a safe way to ask there.

Kenneth Newquist: Yeah. And we can say like we go back in our Google Analytics and we’d compare what people report they use what’s actually in Google Analytics and it matches up fairly well. But we haven’t gone through and try to trace the patterns though the sites for particular resources but having good with Google Analytics session or certainly being doing work of.

I mean the nice thing is we have a bunch of different data that we can pull together to compare to try and answer this particular question. And the other thing, the other nice thing was, I mean we’re on IT, we’re not actually in PR so we don’t control the main Lafayette website. But by doing this, we kind of inspired approach in public relations to do the same thing with their website. It’s less the meaning that spreading.

Speaker 1: What you were doing in the our IT were trying to do, one of the things that used in usability testing, they found that the helpdesk was additive to when you did breadcrumbs. There are number of reasons partly what we were way of breadcrumbs and they should really imprint on this because when we go in and we got called, we could have walked people to where the answers are on the site and educated them. That made perfect sense. Instead of the group heading up usability testing kept coming back that we have to have these and all this the biggest site can happen now.

And what we found was because of the nature of the site design before, that’s how they learn to use the site. And once we made the site in different way, and show them how to navigate through, they can actually get things in a better fashion. Part of my question is how do you take source of information and partly analyze now particular question and find out what really the answer is.

Kenneth Newquist: Yeah, I think we do science. I think, yeah I know we test to come up with a hypothesis and then we test it. So, I think the answer to that is I mean that’s what we’re going to talk about doing. One of the thing that serious proposal was let’s make that search really big on and probably on the homepage. If we do that, will they use it more? I don’t know but I don’t think we’ll going to redesign our page and then find out. I think we’re going to try it and then test.

Audience: That's how they do it with AdWords cause that's how they do it on their homepage.

Kenneth Newquist: Right, I remember, right. And with now they just re-launch the new... right.


Kenneth Newquist: Well, I mean I think the thing is I was able to go like my boss was seriously proposing, let’s get rid of all the contents in the homepage. We’ll place with big old search box. He just searches.

Jason Alley: What website did we just come from to after? It was due...

Kenneth Newquist: Right, I guess. But any case, we were able to point to the surveys were we had people self reporting there 40% of them user search. And then in our tests, I mean if we go through all the videos, you can see every time majority of them were primarily browsing and not searching.

Now, it is possible to prejudice these effects. The very fact that we’re having them sit down in front of the computer, are they still browsing where they normally would. It’s kind of artificial environment. We’re not sneaking around people offices, video taping them on slide.

Jason Alley: When some people would actually say, “Oh! I would just search and find it.” Well, then do that. Because they were compelled to feel like, “Oh! Well, you’re testing the way in which I’m supposed to surf the site.” The test is affecting the tests.

Audience: Did you think about sorting your desktop and then you know you have ceratin people that normally sit on site?

Kenneth Newquist: Right. I mean I think for the follow ups, I think we’re going to be running tests specific things and this was the first big batch of usability testing that we had done. We want to try in start off.

Audience: About your tasks, when someone ends a task, do they get the right answer you told them? Yes, this is right but normally no.

Kenneth Newquist: We usually say into the end, so we’re in review what the right answers. We also always have one impossible task that they are going to fail at because we want to see how they deal with frustration, combating real scenario, right.


Audience: Next to that, you take them back into the home page wherever they were.

Kenneth Newquist: Wherever they were, they just keep going. One of the things that we found and what we did that when CLAMP did their testing with Moodle, instructional technologists like to help people. And they have a really hard time not helping people during the tests.

That was one of those things where you try to reiterate the tasks, you try to encourage them but you’re not trying to help them and it was really hard for somebody instructional technologists when we’re doing Moodle to be able to restrain themselves like that. We had some people who did the survey, who said we just couldn’t do that. I couldn’t let my faculty suffer like that. I needed to actually give them the right answer.

We did have, I mean some of the stuff that you don’t see in here but we had some people who would say, "Why would we just call the helpdesk?", which is good to know when people were just not even trying to. We also had things like one of the scenarios was your computer is broken, you need help, where do you go. The answer is well, if my computer is broken, I will just going to call helpdesk, not stop by the library and browse there.

Audience: So, do the math. You never take the test and people come in. It doesn't really matter. See if they can find what they need.

Kenneth Newquist: Our helpdesk is offsite, but I think we should definitely do that. That would be cool. One of the things we did there, speaking with the helpdesk is for more data points, we went back with the helpdesk tickets to look at tickets they came in to see if we can identify particular patterns about. I mean nothing really left out of us. It’s a lot of day-to-day analyze but I definitely any large selection of data is good to look at.

Audience: Did you do a usability test. For example, we actually put usability tests in Blender evaluations so that when we are changing the software to try it out, we find a lot out.

Kenneth Newquist: We haven’t done that, but I will say that after we did this, there was, I mean we didn’t actually do it yet, but there was a live discussion when the Office 2007, 2008 we all came about. About, well our users will just know. Or our users will going to completely confuse. Again, it comes back to like opinions, right. I don’t know what.

I mean I found certain things confusing, but I don’t know what our users will find confusing. We did certainly think about doing that. It will be worthwhile, I think again I mean I think sometimes, it’s easy to get hang up on this in a scientific studies. I’m not doing a thousand tests to gather data but I think it’s useful enough to be able to see how people really use the tools.

Audience: I guess what I meant was how do you do search. Presumably, you know your website. You might not know the developer or the problem.

Kenneth Newquist: I think I’d found someone who knew and then have them conduct the tasks for them. I mean honestly if I’m evaluating vendor software, I’m buying that software for particular use, test the use. If an events scheduling software had them schedule the event, but that’s probably not good software to buy in. Other questions?

Audience: Do you measure something like users acquired or that the software captures crowds surfing?

Kenneth Newquist: No we haven’t done that. I have seen them. I think we will need full disclosure on our seminar. We already disclosure for Analytics for I think we haven’t talk about it. I’ve seen it. It would be interesting but no we haven’t tried that.

Audience: Will you present the max solution to students, are you comfortable with it or are they predominantly...

Kenneth Newquist: I think actually we do if I remember correctly, we dual booted, it right?

Jason Alley: Yes.

Kenneth Newquist: So we have notes.

Jason Alley: No, we didn’t because we wanted everything in ScreenFlow, so we haven’t run of them on Mac.

Kenneth Newquist: We did in the Moodle testing, I think we did both because cam to the studio installed on the Window side end on the Mac side. I think you can prejudice the results. I would rather use the tools that I’m more comfortable with.

There were some arguments when we did this with CLAMP, there were some people who said, you know well, I have to go to my faculty’s office to do it. It’s not going to be a good test unless I’m in their office on their computer doing the test.

Jason Alley: Well, something like Moodle, so specific to uploading files and have an access to those kinds of those things.

Kenneth Newquist: Other questions? No. I think that’s it. Thank you very much.