Be sure you are selecting users who are representative of your those who will be purchasing the system

Selecting the right users is vital to the success of your system, so spend the time to find the right people.

Be sure that they are willing to work with. Apathetic testers will only result in data that will not help you to improve your system. Look at ethical concerns

Test tasks should reflect what you think real tasks are going to be like.

You may need to simplify the tasks a bit for testing, but be sure not to make the taks easier or bend the tasks in the direction of what your design supports best.

Be careful no to spend too much time developing your test system. If you put too much time into making more of a prototype than a mock-up you will be reluctant to change it once users start to criticize it.

Focus on the basics of how the screens would appear at various stages of a user interaction.

We want to find out is the user can understand what is going on. Is the sequence of screens logical to them in completing the task?

Start with everything the user would see if they did everything the best way.

Then decide whether you also want to support some alternative paths, and how much you want to investigate error paths.

What happens if they choose a screen you don’t have?

First, you record what they want to do, as that is valuable data in evaluating your design. Then you can tell them what they would see, and let them try to continue, or you can ask them to make another choice.

There are two types of data we are working with here: Process Data and Bottom-line data.

Process data is what you want to focus on. It is the observations of what the test users are dong and thinking as they work through the tasks.

Bottom-line data gives us a summary of what happened: how long did users take, were they successful, how many errors did they make.

They both have there uses, but at this stage we are most concerned with what users are thinking, not just what they are doing. This will help us to modify our system to match their thought processes.

The basis of thinking aloud is to ask a user to perform a test task and to talk you through what it is they are doing and why.

Ask them to tell you what they are thinking: what they are trying to do, any questions that arise while they are working, things they read.

Record what they are saying wither using a tape recorder or by just taking notes. Look at examples 5.5

Basic instructions: "Tell us what you are thinking about as you work." You can suggest some categories of thought such as; things they find confusing, decisions they are making, why they chose a certain path, etc.

Be sure to tell the user you are not interested in their secret thoughts but only iin what they are thinking about their task.

Also be sure to let users know that it is the system and not the user you are testing. If something doesn’t work it is not their fault but the fault of the system.

Be careful what you are saying to testers. You can very easily distort the test results by asking too many questions of guiding the user through your system.

It is better to just sit back and allow users to comment on the system and what they are doing and for you to just record what it is they are doing and saying.

Good prompts are "Tell me what you are thinking" or "Keep Talking"

Bad prompts are "Why did you do that?" (It is too accusatorial in tone)

Remember a very few comments by you can seriously undermine your testing.

Work out in advance when you will permit yourself to help.

One criterion is: help only when you won’t get any more useful information if you don’t, because the test user will quit or cannot possible continue the task.

If your do help, be sure to record when you helped, what you said, and the situation that necessitated the help.

Be sure to also explain to users that they will be asking questions but that you most likely won’t be able to answer them. Look at methods of recording 5.5.3

Make a list of all the difficulties users encountered. Be sure to include references back to the original data so you can look at the specifics if questions arise.

Try to judge why each difficulty occurred.

See 5.5.5 and related HyperTopic

See 5.6 and Statistical Analysis of Data

Another bottom-line measure is to ask users how much they like or dislike your sytem

Ask for a rating on a 1 to 10 scale, or something similar.

Normally you want test users to do more than one task.

Choose one sensible order for all users, don’t mix the order up.

Simple to complex is best.

Should they get some training in your system, or go in cold?

Depends on what you are trying to figure out.

If your system is the kind where people will generally hit the system cold without training (such as our lighting system), then you should test this way.

If training will be involved, use the actual training materials, if available.

You should always do a run through of your study before actually putting it into use.

Once should be done with a colleague, to get the bugs out, and then with a real test subject.

You will quickly find out is your test is workable, or if modifications need to be made.

DON’T SKIP THIS STEP--it is well worth the time and will keep you from having a disastrous test study.

It is O.K. to ask users after the test questions.

However, don’t expect to get great results from this. Users often don’t remember very much about what they have just completed.

Part of the problem is that users tend to remember only the correct solution and not all the trouble that they went through to arrive at that solution.

Often you will get better responses if you have video tapped and go back and show users where they experienced problems.