Usability test
Usability Report 1
Date 7.12.2009
Version 1.1 of Usability test
Product: Oikotie.fi, Asunnot
Organiser: Joonas Pajunen
Date of test&report: 7.12.2009
Executive summary
A test subject is a young person looking for an apartment. The subject has a search criteria of a budget of 220 000 euros and location of Helsinki. Purpose of the test is to see how the site's apartment section works and especially how the search function works as it seems to be the main feature. Advanced features like the "apartment watchdog" are ignored.
Name the product: Oikotie.fi apartment section
Purpose/objectives of the test: Find a suitable apartment with price and location criteria
Number and type of participants: One test subject, young female
Tasks (if task are used): free search, plus additional smaller tasks to use features that were not needed in free search.
Results in main points
- search function relatively easy, with the exception of no guidance on min-max price fields
- green background nice
- search result and individual apartment presentaion good and clear.
- number of advertisement, unlcear pricing and lack of detailed help annoying.
Introduction
Full Product Description
- Oikotie.fi ASUNNOT
- Apartment section and it's main features were inspected, excluding some more advanced features that would have required response times and such ("watchdog" etc.)
- Product is intended for practically anyone in search for an apartment, generally estimating the age range form 20 years upwards
- The use environment is mainly for private persons looking to buy or sell a private apartment. There are some options to buy and sell lots and farms also
Modifications to the Pilot test
Pilot test part 2 (individual tasks) was too similar to the first one, as the tasks ended up repeating the events that took place in the first part of the test. The part 2 steps were therfore omitted and incuded into the first part as a form of a guidance or help from the tester.
The actual testing took less than was estimated, briefing and post-questionnaire estimations were ok. So the estimation for the actual testing was reduced in half.
Test subject is a girlfriend who matches the imagined test subject in both of the previous tasks, except for gender. Test takes place at home.
Test Objectives
Test objectives are to inspect the Oikotie.fi site and specifically the Asunnot section because the whole site is too big to cover at once. The Asunnot section is the most popular in there and therefore the best to start with. User's main areas interest were the search form, the search result presentation and indivual apartment pages.
It should be noted that the whole of Oikotie site is very inconsistent and to include every section would probably decrease user satfisfaction.
Method
Participants
- The total number of participants tested: 1
- Participant fits in the estimated user population and the imagined test user in previous tasks
- Chosen because of high availability
- Capabilities include above average it skills but understanding of housing market and pricing is somewhat insufficient for fully considering the price criteria.(this is good point, since most people when they start do not know enough of the pricing policy, thus it would mean that there is need for explaining that more)
Context of Product Use in the Test
- Difference of context is that the tester is not seriously looking for an apartment
- Free, do-whatever-you-want-to-reach-the-goals mode in the beginning
- After the goals and criteria were reached:
- use help
- browser without search
- re-find the apartment found previously
- use map to see location
Criteria: Price: 220 000 euros, location: Helsinki
Free mode task was selected to see the natural flow of apartment search. Completion when above criteria met.
Other tasks were added to see how the generally dismissed functions work. Completion when test subject decides so. hm.....
Test Facility
The setting was at the test subject's home (but the partner was unable to participate in the search). No external problems were present.
Participant's Computing Environment
- Computer: Regular PC, Windows XP SP3, regular logitech keyboard and mouse
- Browser: Firefox 3.51, no relevant plugins
- Display resolution: 1680x1050
Procedure
- Operational definitions of measures (e.g., how is it decided that that a task is completed)
- A task was completed when
- Free mode: 2 set criteria were met
- Smaller tasks: when the test subject felt they were done with the function
- Policies and procedures_:_
- No interaction help was needed, but the test subject would've been directed to the online help if needed
- During the smaller tasks, each task was briefly briefed with a few words but no additional help was given
- Briefing of situation and criteria was given, which took roughly 2 minutes
- briefing -> freemode -> ( quick briefing -> small task of specific feature ) x number of unused features in freemode -> questionnaire -> done
- Participants were not paid, no other individuals were present and only one tester did the testing
Participant General Instructions
- Instructions given to the participants: You are looking for an apartment, a duplex, with your partner. You must find it in Helsinki and your budget is 220 000 euros
specific tasks:
- look for help, does it contain enough information on how to use the site and about the terms related to housing and financial stuff
- try browsing without using search
- try to find the previously found apartment again
- use map to check the location of an apartment
Results
Very little quantitive data was gathered:
- the user had 2 error messages when first using the search form. The problem
- user had to click the browser's back button zero times, as she did not reach any dead ends or wrong pages Do not understand what this means?
Qualitative positive :
- clarity, sorting and presentation of search results
- presentation of individual apartments
- green background looks nice
- most of the apartments have pictures
- frontpage is clearly presented
- map useful, even though hitting back button brings to previous map coordinates which after serious map browsing becomes frustrating
- the search results are consistent, the same apartment was found later with same criteria
Qualitative negative :
- search form fields not necessarily all clear, error messages unclear too
- so many advertisements become annoying
- help too general and stating the obvious and no details about technical or related terms
- forntpage different/dull. inconsistency didn't matter that much
- plain browsing without the search form is useless
The site has some problems but generally seems to do it's main job fine. The test subject ended up using the site for a while, even after the test was over.
The core which is the search and it's result's presentations works fine with some minor, ultimately irrelevant weirdnesses.
The biggest problems for this test subject was the unclear search form and error reporting at the first try. The help was pretty much useless.
Reliability and Validity
The realiability of these results seem quite high* in respect of comparing them with the ones from the pilot testing. A few differences were present (such as more troubles with the search form) but the general feedback was very similar. The final test was also very similar to the pilot test as not so many changes were made to it.
The validity is roughly ok* as the user did not have any previous experience of the site and does not have any experience on usability testing. One unrealistic feature is that the user doesn't have that good of an understanding on the current housing situation and pricing or other relevant information. I think this is good, mots people do not know the pricing policy etc. But the generalisation of the problems to broader user sections is not possible based on 1 actual test with one user and one pilot test.
* the rating scale for these two is based on gut feeling.
Summary Appendices
Notes taken before, during and after the test
Pilot test method 2 was too similar to the first one, so the steps were omitted and included into the first test as a form of a guidance or help from the tester.
The actual testing took less than was estimated, briefing and post-questionnaire estimations were ok.
Test subject is a girlfriend who matches the imagined test subject in both of the previous tasks, except for gender. Test takes place at home.
---
18:13 start
Chose myytäviä asuntoja from frintpage.
Uses search function.
Is first wondering how to choose duplex. Found it form the "huoneluku" field
porblems with the minimum to maximum price? (the fields are not told that they are min and max)
location entered properly
ignored 251€, but later wondered about it
chose an apartment in töölö
ended up in uudiskohteet section and is wondering about the price..
found a place that was MYYTY
18:21
found a place in käärtinpolku 12, 00390 hki, 2h kk kph paarveke 105 000 € konala
18:25-27, specific tasks:
look for help - found it after a while
doesn't define the criterai specifically (the min-max properties)
no help for the technical terms, only site help
18:27, browsing without search
chose duplex -> useless without any criteria.
18:28, re-search the previously found apartment
found easily with the same added criteria
18:29-30, use map to check the location
map was useful for checking the location
but going back not so cool
---
18:32-40, questions
1. search
relatively easy, only irritating part was unclarity of min - max price
sort by price, even though not so clear about the actual pricing
2. bad
pricing not so clear (partly because new to subject)
too many adverts (even though some are nice and colorful)
3. navigation
menus clear and coherent
4. good
green background
search resutl presentation (big picture, type/size of apt, area, price, etc..)
individual apt. view(big picture, map)
5. help
could've had more information abour
technical terms such as pricing
search usage too general; no detailed information
yleiskunto rating
many details on one apt is unclear
Mobiili ID ohje link leads to different help page (for mobile ID)
6. fronpage
clear
background too dull (not green)
Data
search produced errors and the red indicators were somewhat unclear
didn't end up in a page that didn't want to