Usability Evaluation Script

Welcome

Welcome to the usability evaluation of hub squared. Thank you for taking time to be a part of this evaluation. My name is ________. We're working on the design of this site and want to identify usability issues. We're testing the site, not you. There are no wrong answers so feel free to speak freely about the site. I'm going to ask you to complete a series of tasks which will take about 20 min. I'll be asking you to please think aloud as you work, saying what you are thinking and how you think the site functions. This is voluntary and you can stop at any time. Do you have any questions before we begin?

Tasks

1) Please sign in

2) Add a Psychology 100 class

3) Add a discussion question within your Design class

4) Search for a linked video about usability testing

5) Please search for a book for your Business Management class

6) Please log out

Questionnaire

1) Do you think a collaboration website would be useful to you?

2) How do you feel about the navigation of the site?

3) How do you feel about the layout?

4) What task was most difficult for you?

5) Do you have any additional comments or suggestions?

Conclusions

Even before the usability test proper, we noticed problems with our site. In our zeal, we had omitted the Class Search/Add page.

The front page apparently gives no indication as to the purpose of the website. If the user is to know what our website is about, we should add a description on the homepage.

We asked the user to add a class site; she thought that the Department and the Class number search bars were for entering parts of the same class abbreviation (e.g. that she was to choose Psychology for the department and 1001 for the class number). Our search bars were meant to be separate; the Department drop down list allows users to search by department (e.g. find all results within the Psych department); the Class search list allows the user to find all results in a particular class. To make this clearer, we could add an example above the class search bar.

The user was not sure how to add a new discussion topic because our page did not have a button to press to add a discussion topic. Additionally, the user was unclear on which functions were possible on the discussions page; if our audience is to understand what’s going on, we need to make the different functions clearer.

The Always Linked page was straightforward.

We asked the user to search for a book. The Materials tab is easily found, but the search format is unclear (department? title? ISBN?). This will be more clear if we clarify the preferred format(s) above the search bar.

The logout and overall navigation made sense overall. The user asked what the difference was between searching for materials in the class tab and using the main search bar. We think that the site would be improved by only keeping the contents of a class in the box, as this will visually distinguish between activities done within a class and other searches/activities on the site.

Wireframes

Log in page:

Home page:

Class Discussion:

Class - Materials:

Class - Linked:

Private group chat:

About us:

Buy/Sell Materials:

Linked:

Paper Wireframes

Home page:

Class page:

Site Map

User Research, Competitor Analysis, Personas

User Research Summary:
Our research involved surveying students of different ages at Ohio State to determine the wants, needs, and behaviors of our user group. We used SurveyMonkey to do so, and received 25 responses. The summary of our results are as follows:

Demographics:

• Majors: Biology, Education, Medicine, Engineering, Sociology, Economics, Design, Marketing, Theatre, Neuroscience, and Sport studies
• Year in college: 8 in their first two years of college, 14 are in their second two years, and 3 are graduate school

Wants/Needs:

• 84% responded that it would be helpful to have a site where students can share notes and other relevant resources.
• Participants responses varied evenly when asked if they would be likely to join a study group with members of their class; About a third responded that it is likely, a third responded that is maybe likely, and a third responded that it is not likely.
• 44% responded that they sometimes feel unsure how to get help when taking classes with large lectures or large class sizes.
• 40% of participants admitted to feeling uncomfortable approaching classmates for help in person or by email.

Behaviors:

• When asked how they look for help when they are having trouble in a class, 52% ask the professor for help, 56% ask students in their class, and 48% ask friend or family. Also, 92% of participants look on the internet for help, versus only 24% that said they go to tutoring centers.
• 80% of participants responded that they are okay with being approached by classmates for help or that they enjoy helping their classmates.
• 19 out of 25 participants said that they sell at least one book at the end of every semester, where 15 of those sell some or most of their books, and 4 sell all of their books.
• 72% of the participants prefer to buy books online through Amazon or other websites, 24% prefer to buy books from OSU bookstores, and the remaining 12% prefer to use other methods (where 0 of the 25 participants reported that they prefer to use MarketOSU).

Competitor Analysis:

First As-Is Analysis: MarketOSU
MarketOSU is a website only for OSU students that allows students to post books, sporting event tickets, and other event tickets that they would like to sell. Students can search for items that they wish to buy, and can then contact the seller.

1. Visibility of system status - The system status and navigation is pretty straightforward due to the simplicity of the site. The title of the page is on the top. It may be confusing to people that you can switch pages either by going back to the home page, or by using the drop down menu. On the mobile site, it is confusing which words are links and which are not.

2. Match between system and real world - The site uses terms that are easy to understand. The main pages are accessed by links titled "books", "basketball", "football", "events", and "watchlist".

3. User control and freedom - It was not clear to me at first how to return to the home screen from the page I was on, which made me feel that I didn't have control over the website at all times. Also, there is a "watchlist" feature, but I was unsure how to add items to the watchlist.

4. Consistency and standards - The site has separate links for "browsing" and "searching" which is confusing to me since most sites have both these functions on the same page. In fact, MarketOSU does have a search bar on the browse page, so I'm unsure why it is necessary to have an additional link to do this.

5. Error prevention - The site allows you to sign up to use the site, and then sends you a confirmation email for verification. This assures that you are signing up with the correct email.

6. Recognition rather than recall - The available actions are clearly visible on the home page. The site has a limited amount of functions, which lessens the amount of information to recall. Also, the functions have pictures along with words so it is easy to recognize where to go.

7. Flexibility and efficiency of use - The site has a limited amount of functions, so it doesn't cater to inexperienced users any differently than experienced users. There is a "watchlist" function that allows users to place items there and come back to them later, which may increase efficiency of use. There's also a mobile version of the site which is intuitive to use if you have previously used the site.

8. Aesthetic and minimalist design - The site does have a minimalist design. It has a plain white background and has a minimal amount of text. There are advertisements that are somewhat distracting. The design could probably be more aesthetically pleasing.

9. Help users recognize, diagnose, and recover from errors - An error message is given if the wrong username or password is used to sign in. However, it does not give any instruction on what you can do if you forget your password. Also, if you are selling a book and if you cannot find the book by searching it's ISBN, you can manually add the book.

10. Help and documentation - MarketOSU has a Frequently Asked Questions page that has answers to a few questions that users might have. It also has a page that lists contact information. However, there is no page that provides an extensive amount of help.

     We determine if a site is good if it allows us to find what we are looking for easily and quickly. This site is effective because it has a simple design that allows users to accomplish tasks (such as sell and buy books and tickets). The site also uses terms that are easy to understand and does not use excessive amounts of text. The site is flexible for users since it has a mobile version if the site, however, the mobile version is not entirely intuitive. Also, the user may have to use the site multiple times before fully understanding all the functions. Overall, the usability of this site is good (possibly due to its limited functionality).

Second As-Is Analysis: Discussion Boards on Carmen
Although OSU Carmen's Discussion Board isn't a website, it is the closest service we found to our Student Board, a place where students can exchange information about their classes and homework. For that reason we found it was relevant to analyze Carmen's Discussion Board.

1. Visibility of system status - The Discussion Board is easy to find and pretty straightforward, right under the Activities menu. The mobile version is a lot more challenging. In order to even find Carmen you have to switch to the desktop version.

2. Match between system and real world -The language is simple and easy to understand. The messages can be sorted by "unread" and "read", as well as "shared".

3. User control and freedom -It is pretty clear how to go back and switch to another class to enter a different discussion board. All a student has to do is click on the class name on the top left hand corner to return to the class menu, or click on "Home" also on the top left hand corner to return to the Carmen menu and select a different class.

4. Consistency and standards - The Discussion Board is really simple and it doesn't have many commands. Although it is consistent, it very limited on what it offers. 

5. Error prevention -Since it is very limited there isn't much room for error. A message pops up after you post a comment to let you know it was successfully posted. However, it does not let the student know when someone replies to the post. The student has to keep checking to see if there are any replies.

6. Recognition rather than recall - The Discussion Board doesn't have many function so it easy to remember where to go and what to do.

7. Flexibility and efficiency of use - The Discussion Board is very limited in its function so it doesn't tailor to unexperienced users any differently than to experienced ones. 

8. Aesthetic and minimalist design - The design is very minimalistic. The background is white on gray. It looks like a paper page with two tabs on the top. It isn't very aesthetically pleasing. It looks really boring and plain.

9. Help users recognize, diagnose, and recover from errors - To access the Discussion Board the student needs to first log into Carmen. Carmen gives out an error message if a wrong user name or password has been typed. On the right side of the screen there is a list of questions students can click on that will help to reset the password in case it has been forgotten. 

10. Help and documentation - The Discussion Board has a "Help" tab that informs the student of its purpose. It doesn't allow the student to ask questions or receive answers. It is not interactive and not helpful at all.  

The Discussion Board is mainly easy to use. However, it doesn't have many functions and it is very limited on what it offers. I couldn't figure out a way to make any posts on some of my classes so I am assuming that the Discussion Board has to be activated by the instructor before it can be used by the students. Overall, the Discussion Board is easy to use but extremely limited.

Third As-Is Analysis: OSU Reddit
Reddit is a site where anyone can post questions, and those that visit the site can either post replies to questions or view the questions and replies of others.

Visibility of System Status - It is fairly easy to recognize what page you are on and where links take you.

Match between System and real world - Most of the content is user-generated, so most of it matches the real world. There are reddit-specific terms (upvote, subreddit) but they are not technical. 

User control and freedom - It may be hard to find exactly what you are looking for. There are many tabs and also a search tool that should help, but it is hard to know exactly where to start.

Consistency and standards - Same structure and conventions are kept to throughout the site. Subreddit moderators have some control over visual elements (e.g. r/osu has buckeye leafs instead of arrows for upvotes and downvotes and a scarlet header with Brutus Buckeye).

Error prevention - There are not many errors that can occur while making or viewing posts.

Recognition rather than recall - It is easy to recognize where to go to search for topics or to create a post. However, if you are looking for a specific post, you may need to use recall to remember where the post was or what category to look under to find it.

Flexibility and efficiency of use - Allows users to search for specific topics or filter the posts basd on what is currently trending.

Aesthetic and minimalistic design - The design if this site is busy and somewhat distracting. There seem to be extra patterns and symbols that may distract the user and take away from the aesthetic appeal. There aren't too many colors used which is good.

Recognize, diagnose, and recover from errors - The numbered errors (e.g. 502, 504) are just plain text and do not explain what the error is. When the site is overloaded, however, the error message is in plain language and accompanied by a funny illustration.

Help and documentation - There are frequently asked questions and a contact information page, which is helpful.

It is fairly intuitive how to post questions and replies on Reddit. However, it may be difficult to search for exactly what you are looking for, especially if you don't want to duplicate a question that has already been asked and answered. It is ideal for those who want to browse and explore the site, but not ideal for those looking for specific things (unless they post a question and then wait until someone replies).

Consensus:
MarketOSU is a good place to sell and buy books, sports tickets, and event tickets. You can search for a book by title or ISBN, but cannot search by class. There is also no way to discuss with other students what materials are needed or are available for certain classes. #help will also allow students to buy and sell books and materials, but will be organized int a way that students can search for books by the class that they are in. The site will also strictly have an academic focus, allowing students only to sell books and other relevant classroom materials.

Carmen has a discussion tool that can be used in each class, but in spite of being easy to use, it is not very efficient. Students most of the time resort to mass emails when they have questions.Instructors don't usually activate the Discussion Board unless it is part of their requirements that students make comments and reply to one another on a particular subject, and that is usually the only time that students will use the Discussion Board. Some teachers don't even use Carmen at all. Also, these discussions are not checked by students very often and questions posted on the Discussion Board often have a very slow response rate. #help aims to have a discussion board that can be used by students whether or not their teachers activate the Carmen page for the class or activate the discussion boards. #help will be a site where students can feel comfortable posting questions. It is also completely optional, so only those who want to be involved will receive notifications, but those who do not wish to participate will not be bothered (especially by mass emails). 

OSU Reddit is a good way to ask and browse questions or post news, but this site is not specifically limited to OSU students. Anyone can make an account which brings up security/privacy issues. Also, the discussions on this site are very broad, and students don't tend to use it for classroom help. #help! aims to create a site where discussion threads are limited to OSU students and the posts are meant only for relevant classroom material. Students will be able to collaborate with students from their class, and will not have to rely on information from strangers.

#help aims to combine functions and resources that each of these sites provide (such as material and information sharing), while making it easier for students to collaborate with classmates by narrowing the focus to classroom help/discussions and classroom materials, and is only available to be used by OSU students.

Personas: