...
For this testing we have tried to use all things that we have our own disposition, we didn't used record, we used basic things, like Microsoft Word, notepad for take notes. Maybe for next interview we could used a camera or voice recording. It could be easier to take note, and show some attitude of users, because sometimes they said something which mean is right to say.
Procedure Details
*Number of testers:* There were usually 2-3 persons interacting with the tested user: interviewer who asked the questions and took some notes, and the note takers (one or two persons seated at the computer nearby, who typed mostly everything the user was saying, as well as some of the interviewer's remarks).
*Procedure and Policy:*The test conductor (interviewer) read aloud the questions, some explanations were sometimes required (this depended on the individuals), these additional explanations are not included in the testing report, because of the scope of the project and the estimated irrelevance of the extra comments on the user performance. If the user had a problem with a particular task, he was still reminded (comforted) that it is the application that was being tested and the task's nonsuccess just led us to the possible problems with the application which should be improved as a result.
...
Questionnaire was used as another method of giving the test respondent a possibility to give completely anonymous answers (even though in the interview it was specified that no names will be used, there was still a personal factor of recognition involved, whether handwriting is thought to be more anonymous, and the answers could be more open and, thus, valid).
*+Environment:*+The testings were usually done while other people were present in the same room, who were not part of the testing team, but this was not thought to be an obstacle or influential factor that could affect user's answers.
*+Payment:*+Participants of the evaluation were not paid in any way.
...
- Bad titles and definitions
- Bad course description
...
- Search does not give all results
...
- No info when the courses you have to do are available.
- No possibility to enroll to courses which you took previously and fail
- Empty bulletin board
- English mistakes
- Help is not sufficient and no English version.
...
WinhaWille access: It was evident that the users had no problem with accessing the WinhaWille, either through finding the link from Tuubi portal or from Metropolia homepage (which was also the default webpage for the used browsers in the testing). All the users were quite acknowledged with the task of logging in, with virtually no problems arising, and the timing was on average less than half a minute. Nevertheless, we have found that none of the users knew the direct link to access WinhaWille.
+Login:+ When logging in, the only bothersome point for half of the users was that the application didn't accept the login form, with all fields completed correctly, just by pressing enter, but only with a left mouse click on a "login" button.
...
It has been also found out that exactly ISP, Implementation and ISP Contents are the most useful features for all the users - improving these should be a priority. Whether "more info" menu section was thought to be the least common for the users: therefore "Bulletin board" can be removed altogether; but for the courtesy of the developers' credits, we hold it true that "Application" link should still stay there.
+"Exam":+item was not used by any of the users before. In fact, this item was thought to be unclear, and thus not used. It is important to state here that it has been observed as a general procedure in MetropoliaUniversity, Leppävaara campus, that teachers do not require from a student a registration to the exam, at all. The only registration is done for the re-sit exams, and follows a bureaucratic system, involving paper forms with signatures and the porter-information desk. It is negotiated whether the "Exam" feature should be eliminated from the application in order to avoid clutter.
...
Improvement suggestions:
Login:- Make
- Make login possible with pressing Enter, not only with the mouse clicking.
...
- Modify time-out throw out page to make an easier login back (the irritating thing that happens after 15 minutes of no action), provide a link on it to go back to login page.
Menu structure, title definitions:- Unite
- Unite "Student data" and "Change address" into "Student data" with a button to "Modify" it.
- Rename following menu items:
...
- ISP -> Individual Study Plan (ISP)
...
- Enrol >Implemetation -> Course enrollment
...
- Application -> Application info
...
- To the main page -> Main or Home
...
- Close -> Logout
...
- Eliminate some features from the main menu:
...
- ISP contents
...
- Exam (no one is using it)
...
- Bulletin board (more popular on Tube portal) - make the application less cluttered.
...
- * To add:
...
- "Enroll to a study year" - make itavailable at all times, it should appear as non-functional button when the user has already registered for a study term or when the period of enrollment is over. It is suggested to put it on the main page.
ISP:
- Delete Study Code, Date, Projects options
- Planned studies - Study modules with all planned courses in it, information on credit and earned grade.
Implementation:- Study
- Study Code - > Course Code
...
- Group -> Group Code
...
- Study type - make sufficient list of study types
...
- Study method - make sufficient list of study methods
...
- Delete: Date
...
- Lecturing language - add English
...
- In result show all the courses even if you are still enrolled to them, make a mark - Already enrolled.
General:
- Insure well functioning of all options and functions
- Add a "Help" link on every subsection page that requires user's input, clicking on it should open a pop-up window with the instructions text for the specific subsection (for example, in ISP it should be about ISP first and how to work with it).
- Sufficient and up-to-date course description
...
- Make "order by name" a clearer option, to specify which name. (course, group, teacher).
Language:- When
- When the language is changed, it should not throw out the user from the application completely and demand a separate login.
...
- Fix language inconsistencies (if chosen in English, everything should be English), this also regards drop-down options in "Study method", "Study type", etc.
Further developments - our suggestions:- Make
- Make the users remember the direct link to WinhaWille
Reliability and Validity
Reliability: We had three testers involved to testing purpose, which is increasing reliability of the test. Number of total tested users is not that high: 5; but homogeneity and our own experience with the system give us more reason to believe that test results are highly reliable.
...
Have you been on Orientation days___________How well are you familiar with Study Plan:
¨ Perfect
¨ Very Good
¨ Satisfactory
¨ Poor
User code:_______
Questionnaire:
1. How often do you use WinhaWille?_________________________________________________
2. In the beginning was it easy to learn how to use the system?______________________________________________________________________________
3. Did anyone teach you how to use the system?______________If yes, who __ ______________
4. Do you find it is easy to use?______________________________________________________________________________
If no, why _____________________________________________________________________
5. What kind of problems previously did you have while using WinhaWille?______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
6. Is the layout of the system is clear and understandable? If not, specify
Menu structure and navigation ______________________________________________________________________________
______________________________________________________________________________
Menu items and definitions: ______________________________________________________________________________
______________________________________________________________________________
7. Did you have problem with data accuracy: wrong credits published, wrong registration, insufficient course description: ________________________________________________________________
________________________________________________________________________________
8. Would you recommend any improvements, regarding problems you have mentioned or other?________________________________________________________________________________
___________________________________________________________________________________
9. What kind of irritating things you find in WinhaWille?________________________________________________________________________________
___________________________________________________________________________________
10. Have you ever been using Help page on WinhaWille? Any comments: ___________________________________________________________________________________