Page History

Usability test for Tuubi portal

M 32 Reporting Template

Use the following guidelines to assure that required elements are reported. Use the ones that are relevant for your testing purpose, method and setting.

NOTE: this works as guideline. As mentioned above, not everything mentioned has to be described.
NOTE: this template is meant to be used for reporting all kind of tests.

Title Page

...

Name the product: Tuubi portal

Purpose/objectives of the test: find out usability issues in the portal's UI in order to improve the ease of interaction.

Method: Test with real users, with task list and unobtrusive observation.

Number and type of participants: 3 people in total.

Tasks (if task are used):
Method 2:: Test persons are asked to complete series of question and see how fast he/she can accomplish the task without any of help.

Results in main points
e.g. bullet list (this is needed for being able to get the main results without reading the full report, this is seen important, since the reports serve different purposes and sometimes the need is to get a fast overview(usability issues found)

1. The user is confused by too many SEARCH search functions around the portal (we counted 3) and they all hide a different purpose which is absolutely not clear.
2. Workspace search: the search results are very near to the searchbox although the user takes a lot of time before recognizing them as the actual results of his search.
3. Workspace Search: once you tick the checkbox, the subscribe button is hidden.. under another searchbox.
4. Sidebar of subscribed workspaces has an not a clear criteria on grouping the items. In addition to this, the groups are not graphically well divided. Maybe add some margin-top to the groups?
5. Assignments tab: the submit/cancel buttons are very distant from the input textbox. This problems becomes bigger and bigger with wider screens because the buttons are floating to the right margin of the page.
6. Assignemnt tab: when uploading a file as an assignment the system doesn't provide feedback of the uploading going on or being terminated successfully.
7. Check Assessed assignments: you can't easily understand where's the grade (a number) especially if you are not Finnish: in many other countries there's different conventions (letters, different scales of numeric marks, etc.). There's nothing that says that number IS your grade. Especially because there's not a scale. Eg. 3/5.
8. The column of grades in the table that summarizes your assignments has no title! The meaning of that number (grade) is more and more difficult to understand.
9. The mark is visible just in the summary of submitted assignments and the teacher's comments are visible just if you click on your grade. Grade and comment should be displayed contextually at least after you click on your mark.
10. Fancy Ajax calls don't take in consideration to handle the "back" button of the browser. Typical scenario:

• login
• click on a workspace from sidebar
• now select a tab
• now another tab
• let's say I want to go back to the first tab. I click back button.
• Instead of going to previous tab you find yourself in the homepage of tuubi.

...

Introduction

Full Product Description

...

• State the objectives for the test and any areas of specific interest
  • Functions and components with which the user directly and indirectly interacted
  • Reason for focusing on a product subset

Method

Participants

...

Three test users: a full-time university student of Arcada (has been using student portal but no Tuubi) ; an exchange student of IT in Metropolia (uses Tuubi just to check announcements and exam dates); another exchange student of IT in Metropolia with very good knowledge of Tuubi and better overally computer skills.
.h4 Key characteristics and capabilities of user group

...

Users of the Tuubi portal by students' perspective. Fairly good computer skills.

How participants were selected

Users were selected from the target user group as representative of new, basic and experienced user.
Our three test users were well representing the population because of their different experience about the software, but at the same time they belong to the real population of the software tested.

...

Context of Product Use in the Test

• Any known differences between the evaluated context and the expected context of use
• Tasks
• Describe the task scenarios for testing
  • Explain why these tasks were selected
  • Describe the source of these tasks
  • Include any task data/information given to the participants
  • Completion or performance criteria established for each task

Test Facility
Describe the setting, and type of space in which the evaluation was conducted
Detail any relevant features or circumstances, which could affect the results (e.g. There was a brake down of the server, which messed up the test for a while and created unnecessary tension. There was unforeseeable noise that disturbed the test, etc.)

Participant's Computing Environment

• Computer configuration, including model, OS version, settings,
• Browser name and version;
• Relevant plug-in names and versions (the bullets mean stating e.g., what browser and computers the users are using in the test. In field trials this is information that is not known by the technical partners. For example, in one of the tests during last spring 2007, one of the users was at home using SSp during the test, so it was asked what she used e.g., Internet Explorer 6 and Mozilla Firefox2.0.0.6, Compaq Presario with Windows XP and IBM ThinkPad with Windows XP. If all is not know then it is not but it would be good to try to get the info. Plug-ins can refer for example to the browser add-ons (in Firefox these are found from the upper tools menu. Sometimes it is needed to know if some plug-ins are on or off, because it might change or prohibit some functions.).

Display Devices (report if relevant, e.g., Paper prototypes are tested or static prototypes are tested on screen)

• If screen-based, screen size, resolution, and colour setting
• If print-based, the media size and print resolution

Test Administrator Tools (report if relevant for the particular test)

• If a questionnaire was used, describe or specify it here (add these to appendix)
• Describe any hardware or software used to control the test or to record data (audio, video)

Experimental Design

• Define independent variables and control variables
• Describe the measures for which data were recorded (the scale/scope of the recorded data, if relevant for the particular test, i.e., written notes, think aloud in audio recording, etc.).

Procedure

• Operational definitions of measures (e.g., how is it decided that that a task is completed)
• Policies and procedures for interaction between tester(s) and subjects (e.g., is the test conductor aloud to answer questions of the user, provide help, etc.)
• State used: non-disclosure agreements, form completion, warm-ups, pre-task training, and debriefing
• Specific steps followed to execute the test sessions and record data
• Number and roles of people who interacted with the participants during the test session
• Specify if other individuals were present in the test environment
• State whether participants were paid

Participant General Instructions (here or in Appendix)

• Instructions given to the participants
• Task instruction summary
• Usability Metrics (if used)
• Metrics for effectiveness
  • Metrics for efficiency
  • Metrics for satisfaction, etc.

Results

• Data Analysis
• Quantitative data analysis 
• Qualitative data analysis
• Presentation of the Results
  • From quantitative data analysis
  • From qualitative data analysis (descriptive and clarifying presentation of the results)

Reliability and Validity

Reliability is the question of whether one would get the same result if the test were to be repeated.
This is hard to acquire in usability tests, but it can be reasoned how significant the findings are.
(Example from expert evaluation: This review was made by one reviewer in order to give quick feedback to the development team. To get more reliable results it would have been desirable to use three, or at least two reviewers, as it is often the case that different reviewers look at different things. We do feel, however, that for the purpose of this report, and the essence of quick feedback, one reviewer has given enough feedback to enhance the usability of the system.)

Validity is the question of whether the usability test measured what was thought it would measure, i.e., provide answers to. Typical validity problems involve: using the wrong users, giving them the wrong tasks.
(Example from expert evaluation: "The reviewer is an experienced usability professional that has evaluated systems for many years. We therefore feel that the method used as well as the tasks used give an appropriate view of how ordinary users would behave in the system.")

Summary Appendices

...

Tasks (criteria of acceptance of each task is shown between brackets).

1. Subscribe to "Usability testing" workspace in Tuubi.
(User has the workspace on his/her General-workpsaces)

2. Go to workspace "Usability testing" and read the description from the frontpage of workspace.
(User finds and can open the description)

3. Find and download file named "Usability_document.doc" and open it on the test computer.
(user is reading the content of the file on the computer)

4. Open assignment "Usability testing 1" and do as you have been told in the assigment (in the assignment test person will be asked to upload a file from the desktop as submitting an assignment)
(user has given his/her answer by submitting a file)

5. Open assignment "Usability testing 2" and do as you have been told in the assigment (in the assignment test person will be asked to give feedback about the assignment as a text)
(user has given his/her answer by submitting text)

6. Read feedback for the assigment
(user sees the grade and comment on his/her assignment)

7. Look up the date for the final assignment from calendar
(tells us the day, month, year and time in which the assignment is due to.)

8. See who else is subscribed to this workspace.
(can see other members of this worspace and tells us what role is himself belonging to)

9. Reply to a message on the discussion board.
(There is a reply from user to Test-message)

Describe the task scenarios for testing

These task represent a selection of most common operations for a student. Principal features of the portal were covered, at least within the perspective of a student who approaches Tuubi as a communication media for university courses.
These tasks were created, chosen, pilot-tested by the organizers of the test.Each partecipant has been introduced to the test environment by a preliminary speech. During this phase we assured to communicate a positive and relaxed mood.

Test Facility Location

Hoas apartments, Leppävaara. Clean study desk with comfortable amount of light and silent surroundings.

Network connectivity: 10Mbit/s ethernet.

Hardware: DELL laptop, 14" screen, Intel Core2duo 2.66 GHz dual core CPU, 2GB RAM. Finnish kbd layout, trackpad mouse. No external mouse available.

Software: Windows XP SP3, Internet Explorer 8. MS Word 2003.

Procedure

The test-user has been instructed with an introductive speech to think aloud for all the duration of his/her contribution to our test.
A printed task list has been provided before starting in a form of per-task numbered list. User has also been invited to read the whole list through and to ask any text comprehension related questions before the test start.
Another printed task list was in hand of the tester, this one complete of the satisfaction criteria.

Test has been handled in two people: a "blabber" and a "scribbler":

The blabber

• Does the introduction speech
• Relates to user's question: clearly before test starts and evasively during the test, to keep the observation unobtrusive.
• Decides and communicates to user and scribbler when the satisfaction criteria were achieved for each task.

The scribbler

• Positioned behind the test-user, silent and discreet in order to minimize the uncomfortable feeling of being observed
• Takes note of significant happenings, impressions, behaviors and comments of the user during the test.
• Takes per-task timings with chronometer.

Participants of this tests were paid back with huge smiles.

Results

Results are mainly qualitative. For someone who is familiar with Tuubi there is no problems completing the tasks easily. That does not mean that Tuubi is good as it is, but that you can learn to master it. One open question is, how steep is the learning curve in Tuubi.

Common usability issues

• Due to interface back-button doesn't work in Tuubi like user have used to.

Subscribing to workspace

• User who has little experience on Tuubi may find Tuubi frustrating because it doesn't seem to be intuitive or familiar. Tuubi is somewhat misleading the user in some cases, like when new user tries to find workspace through search-function. Test user tried to find the workspace by using the common search-function (located upper right corner and also in the search-tab) instead going first to Manage Workspaces.

• Subscribing to workspace is straightforward, but doesn't look that from unexperienced user's point of view. When you try to search for a workspace the search-function does not work when you hit enter. Instead user needs to click the search-button.

• Search results are shown right below the search-field and there might be several other workspaces. Test showed that it is not easy to locate the search-result. Also the subscribe-button is placed down on the page which means that it is not shown to user unless he/she knows the location and scrolls down.

• Also test user had problems locating the newly subscribed workspace on her menu of workspaces.

Assingments

• Test user commented the placement of the buttons on the assignments. When returning assignment the buttons are located bottom of the page in the right corner. Therefore it is hard to spot. User didn't feel sure if her answer had been uploaded succesfully.
• When uploading files user has no feedback about the upload-process. A loading bar would help.

• Second test user commented that the grade is easy to miss when you read the feedback. It is not shown on the same page as the verbal feedback.

Positive sides

• The calendar is easy to use. Users found easy to browse dates or look assignments in listed view.
• Discussion board is simple and intuitive to use.

Reliability and Validity

Reliability
The tests can be seen as quite realiable. Pilot testing was done with user who has no prior experience about Tuubi but has been using similar software and the first genuine test with unexperienced Tuubi-user showed similar results as in pilot testing. Most of the problems encountered with piot testing were due to incompatible browsers (IE and Firefox running on Metropolias computers). In the final tests up to date-version of IE was used. Also

Validity
Test users were familiar with IE and Windows. Other had been using Tuubi for few months mostly to read announcements and the other had broader and longer experience about Tuubi knowing most of the functions of Tuubi. Finding user who's know-how of Tuubi would have been somewhere between these two would have perhaps provided more information about the learning curve and which are the biggest pain points in Tuubi.

Summary Appendices

Notes of the pilot test and improvements made to final test

Notes of the final tests

The tasks

Mockups

...