Experiment design ideas

Over the past couple of months my adviser has given me some great ideas for user study design in the future. The big problem is that there are lots of things I want to test, but it takes quite a bit of time to do a full user study. Participants that we can get are almost always complete novices with robot control in any aspect, let alone remote manipulation.

So the basic idea is this: have a couple of "expert" users in the lab, or people that can agree to come in often over a several-month period, and look for trends and find the features that are potentially interesting and important. After several revisions with that approach, then we run a full user study with novices to see how the trends do with a statistically significant sample population.

Another professor gave some additional advice. There is a possibility that I will run into a situation where what the user thinks is the best does not correspond with the best performance. When that situation comes up, what is interesting is to have one of the experts test the intuitive design path and the other expert test the high-performance design path. Analyzing the differences between the two makes for a very interesting paper to read. We can also bring in a 3rd subject to test combinations of the interface types.

There are also two ways to analyze the data. The first is a quantitative analysis, simply looking at all of the objective data that we can record. The second is a qualitative analysis where we take subjective measurements and interview the participants. It's difficult to come to statistically significant conclusions with qualitative data, but the analysis can be more interesting and informative especially with a small sample size.

This is good stuff to think about, because the quality of the data you get out of an experiment can greatly depend on the quality of the experiment design.