After you have completed the pilot test do the following:
1. Write into your task 2, i.e. your testing plan:
- When and where the test was executed
- What you have to correct in your testing plan. e.g., your task descriptions were not clear, your time estimation of how long the test takes was not correct, you did not have the right dbriefing questions, you did not get the information that you thought you would from the test.
- Then correct your plan (state what colour you will use for the corrections so that i find them)
- State when and where will the actual testing take place.
Testing plan
• The goal of the test: what do you want to achieve?
The goal of the testis to examine how a regular person can navigate in the oikotie site, especially with the apartment and job finding goals defined in the previous task.
• Where and when will the test take place?
The test can take place in virtually any location with a computer and quiet enough surroundings. Traditional test environment for a school project is of course a classroom. Finding an average computer user in those rooms might be a problem though.
• How long is each test session expected to take?
A session should include some 5 minute briefing (situation, role, criteria), 15-30 mins for the test, as 15 should be enough for the user to find some results or get frustated enough to abandon the search. Another 15 minutes can be used to finding a result meeting the exact criteria. Let's say 15 minutes to see how the user interface works, and another 15 minutes to see how the data can be retrieved and inspected. The latter part is not as important in the scope of this test. Another 10 minutes debriefing and small questionnaire.
• What computer support will be needed for the test?
The user can be helped to start the computer or browser if needed.
• What software needs to be ready for the test?
Firefox and Internet Explorer. IE also, since most people still use it, and web sites can produce some quirks depending on the browser. The test user shouldn't be forced in to using the experimenter's favorite browser.
• What should the state of the system be at the start of the test?
Operating system running, with clear shortcuts to the browser, or the tester preferred browser open. Maybe the convenience of the site's URL can be tested by starting with the browser default/home page.
• What should the system/network load and What should the system/network load and response times be? (not too fast or too slow);
The response times will automatically be realistic since the site is running on the final use environment. Otherwise some some stress test type script could be run simultaneously on the server.
• Who will serve as experimenters for the test?
Pilot test will be executed by the project group and the final testing by different persons, some of whom are in the target group and some of whom the situation will be explained to. We try to find persons who are not too familiar with computers.
• Who are the users going to be and how are we going to get hold of them?
To meet the estimated target group, the tester should be someone relatively young and moderately skilled in using a computer and web browser. A fellow student in the degree programme might not be the best possible test subject as they can probably adjust to the situation faster.
• How many users are needed?
A few different users can be tested for the apartment and job search tasks, and how those are accomplished. The following requires either 2 person, or one person doing the steps in numerical order.
• What test tasks will the users be asked to perform?
The user should be given a task of finding a an apartment in a given area and with some certain budget. Either by:
- Defining only these two criteria, and then observing how the user goes ahead on trying to solve the situation (using unobtrusive approach)
- The reason for the apartment search will be that the user needs more space, but is not in a hurry. Therefore, more emphasis should be placed on the quality of the apartment instead of any specific date. Because of this, there should be task where the user checks details of one aprtment thorougly. This might happen automatically, but if the user is not able to find any apartments, some default case should be given to them. do the following in the order that feels most natural
- Browse through the apartments matching the size criteria (~few rooms, nothing big)
- Use search to find the apartment
- Use advanced search to find the apartment
- Check help on how to find apartments
• What criteria will be used to determine when the users have finished each of the test tasks correctly?
- The criteria for the search have been met. For example, the location and price have been met. The test user doesn't have to really want to move into that apartment.
- Steps a - d have been attempted, and at least some of them should produce the above mentioned results.
• What user aids will be made available to the test users (manuals, online help, etc)?
The oikotie site has an online pop-up page, which the user should be made aware of. It's quickly and easily accessible via the footer links in each page.
• To what extend will the experimenter be allowed to help the users during the test?
The experimenter should first direct the tester to the online help, but can help if the tester seems to not get anywhere. The experimenter should subtly suggest the tester to ponder some elements instead of directly telling them what to do.
• What data is going to be collected and how will it be analysed?
- Average time from starting to finding a result meeting the criteria either fully or partly
- How many times the user needs to press back button / feels like they've come to a dead end, or otherwise aimlessly scrolls up and down wihtout knowing what to do (better definition and how to calculate this is needed)
- Do the seatch or other forms produce errors? Does the user need to retype everything or some fields?
- One of the testers will make notes how the user acts in the test (either laptop or notebook notes)
- MB: here it means also what way will get data on the questions you placed above, i.e., someone of you will take notes? So one of you observes abd taken notes? in notebook or in an another computer ......
- After these cvorrections it is go for you!