Testing and Evaluation

F27ID Introduction to Interactive Design

2021-2022

## Overview * Goals and benefits of **Testing and Evaluating** Designs * Different types of **user-based** testing * **Qualitative and Quantitative** Data * Details on experimental research in HCI * Questions and Discussion <aside class='notes'> Students seem to have relatively little difficulty with the concept of testing. They believe it has to be done, and few have a problem with the process. One problem: in the press of time they may tend to fall back on family and friends. This is not a total loss so long as they are getting performance measures: the time to find something isn't warped because the user is your brother. Preference measures are less reliable: of course your mother is going to think it's a great site. Suggest they concentrate on performance. </aside>

## Revision Question Usability and User Experience (UX) are the same: * a) True * b) False <aside class='notes'> Just to get you thinking, let us start with a quick revision question. Take a moment to pick one [pause] Have you picked one? </aside>

## Answer Answer: b) False <aside class='notes'> Did you get it correct? [pause] </aside>

## Revision Question User Experience (UX) focuses on: * a) Satisfaction, Enjoyment, Pleasure, Fun, Value * b) Effectiveness, Efficiency, Learnability, Error Prevention, Memorability <aside class='notes'> Another quick revision question. Take a moment to pick one [pause] Have you picked one? </aside>

## Answer Answer: a) Satisfaction, Enjoyment, Pleasure, Fun, Value <aside class='notes'> Did you get it correct? [pause] </aside>

## **Why** is Testing and Evaluation **Important**? <div style='font-size:168pt;'>?</div> > Take a moment to think about what testing and evalution is! <aside class='notes'> Take a moment to think about what testing and evaluation is and what it means - why is it important. </aside>

### Why is Testing and Evaluation **Important**? * Testing a prototype / developed design is **very important** * Testing and evaluation, simply **confirms** that the product will **work** as it is **supposed to**, or if it needs refinement * Testing assesses the viability of a **design or idea** * Discovers **defects/bugs/problems** before the delivery to the client * Guarantees the **quality** of the design/product <aside class='notes'> </aside>

## The Goals of **Testing** #### Discover errors and areas of improvement in: * **Performance** -- How much time, and how many steps, are required for people to complete basic tasks? (For example, find something to buy, create a new account, and order the item.) * **Accuracy** -- How many mistakes did people make? (And were they fatal or recoverable with the right information?) * **Recall** -- How much does the person remember afterwards or after periods of non-use? * **Emotional response** -- How does the person feel about the tasks completed? Is the person confident, stressed? Would the user recommend this system to a friend? <aside class='notes'> Usability testing is a black-box testing technique. The aim is to observe people using the product to discover errors and areas of improvement. Usability testing generally involves measuring how well test subjects respond in four areas: efficiency, accuracy, recall, and emotional response. The results of the first test can be treated as a baseline or control measurement; all subsequent tests can then be compared to the baseline to indicate improvement. The generic goal of most evaluations is to provide "useful feedback" to a variety of audiences including sponsors, donors, client-groups, administrators, staff, and other relevant constituencies. The major goal of evaluation is to influence decision-making or policy formulation through the provision of empirically-driven feedback. </aside>

## Goals of **Evaluation** * Assess **extent** of system functionality * Assess **effect** of interface on user * Identify specific **problems** * Designers need to **check** whether their ideas are really what users **need/want**; or whether the final product works as expected. * To do that, we need some form of methods, or more specifically, **empirical methods** for HCI <aside class='notes'> </aside>

#### Remember ## What is Usability? <aside class='notes'> Let's start by reviewing, what is usability? </aside>

## Usability * Usability refers to the **extent** to which a product **can be used** by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of user (ISO 9241-11) * Usability **measures** the quality of a **user's experience** when interacting with a product or system - Ease of learning - Efficiency of use - Memorability - Error frequency and severity - Subjective satisfaction <aside class='notes'> Usability refers to the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of user. </aside>

## User-based **evaluation** * Considered to yield the most **reliable** and valid estimate of an **application's usability** * In a typical user-based evaluation, test subjects are asked to perform a set of tasks with the technology. * Depending on the primary focus of the evaluator, the users' success at completing the tasks and their speed of performance may be recorded. * Large sample of users would be good, but 3 (**Lewis**) or 5 (**Nielsen**) are often enough to uncover the majority of problems. * **Nielsen**: "once it is found that two or three people are totally confused by the home page, little is gained by watching more people suffer through the same flawed design" <aside class='notes'> In an ideal world user testing with a large sample of the intended user population would routinely occur, however due to resource limitations, user-based tests are often constrained. As a result, there is considerable interest among HCI professionals in determining how to gain the most information from the smallest sample of users. While popular myths exist about being able to determine a majority of problems with only 2 or 3 users, Lewis (1994) has shown that the sample size requirement is largely dependent on the type of errors one seeks to identify and their relative probability of occurrence. Whereas 3 users might identify many problems in a new application, substantially more users will be required to tease out the remaining problems in a mature or revised product. In the early 1990s, Jakob Nielsen, at that time a researcher at Sun Microsystems, popularized the concept of using numerous small usability tests-typically with only five test subjects each-at various stages of the development process. His argument is that, once it is found that two or three people are totally confused by the home page, little is gained by watching more people suffer through the same flawed design </aside>

## Certain **Categories** of evaluation #### **User-based** evaluations: where a sample of the intended users try to use the application * **Lab studies** (experimental research, Usability testing), field studies * **Observation Methods** (think aloud, video analysis) * **Questionnaires**, Interviews, Focus Groups * Some are Quantitative others are Qualitative <aside class='notes'> Certain Categories of evaluation include: User-based evaluations: where a sample of the intended users try to use the application. lab-based studies. Observation Methods. Questionnaires, Interviews and Focus Group. You should be aware that the different evaluation methods are not all quantitative but can also be Qualitative. </aside>

### Method **Selection** The evaluation **approach influences** the methods used, and in turn, how data is collected, analyzed and presented * E.g., **Field studies typically**: - Involve observation and interviews. - Do not involve controlled tests in a laboratory. - Produce qualitative data. * E.g., **Usability testing typically**: - Involves users, interviews, design, questionnaires. - Conducted in the laboratory or in a natural setting. - Primary goal is to test how usable the interface is with intended populations. <aside class='notes'> The evaluation approach influences the methods used, and in turn, how data is collected, analyzed and presented. </aside>

## Revision Question For user-based evaluations, what is a good enough number of test users to uncover the majority of problems? * a) 1-2 * b) 3-5 * c) 50-99 * d) 100+ <aside class='notes'> Let's try a quick quiz question to check if remember the details on user-based evaluation. Take a moment to pick one [pause] Have you picked one? </aside>

## Answer Answer: b) 3-5 --- * Large sample of users would be good, but 3 (**Lewis**) or 5 (**Nielsen**) are often **enough** to uncover the majority of problems. * **Nielsen**: "once it is found that two or three people are totally confused by the home page, little is gained by watching more people suffer through the same flawed design" <aside class='notes'> Did you get it correct? [pause] </aside>

### Should you use **Quantitative** or **Qualitative** approaches? <aside class='notes'> Should you use Quantitative or Qualitative approaches? </aside>

### **Quantitative** or **Qualitative**? * Data is either Qualitative or Quantitative * e.g. Interview-based data is Qualitative * Questionnaires can be Quantitative (ratings-based), Qualitative (short-questions) * Other forms of Quantitative data includes * time taken to do a task, performance * Selection **depends** on: * Overall research or evaluation goal * Ethical and practical issues * Resources, cost, logistics * Availability of participants and sample size * Skills of team, background, philosophy * One is more rigid than the other * Hence an evaluation proposal is imperative <aside class='notes'> The question of whether you use Quantitative or Qualitative approaches depends on a number of factors. </aside>

## Reliability, Validity and Scope * Can the study be replicated: **reliability** * Are you measuring what you expect: **validity** * Are there any unexpected effects: **bias** * Can the findings be generalised: **scope** <img src='.\images\lec07Slide10-3.png' style = ' border:0; width:100%; height:auto; ' > <aside class='notes'> When you think about reliability, validity and scope you should be thinking about the following questions: Can the study be replicated. Are you measuring what you expect. Are there any unexpected effects. Can the findings be generalised. </aside>

## Questionnaire Design * Interview vs Questionnaire? * Should the session be video taped? <img src='.\images\lec07Slide11-3.png' style = ' border:0; width:400px; height:auto; ' > <img src='.\images\lec07Slide11-4.png' style = ' border:0; width:650px; height:auto; ' > <aside class='notes'> When you're designing your questionnaire you need to think about the details. For example, is an interview better than a questionnaire? Should you record the interview? </aside>

## New, adapt or reuse? * **Established questionnaires** will give more reliable and repeatable results than ad-hoc questionnaires. * Some questionnaires for assessing the perceived usability of an interactive system: * Questionnaire for **User Interface Satisfaction** (QUIS) (1988) * **Computer System Usability** Questionnaire (CSUQ) (1995) * **System Usability Scale** (SUS) (1996) <aside class='notes'> You might not have to write a questionnaire from scratch. Established questionnaires may give you more reliable and repeatable results than an ad-hoc questionnaire. </aside>

## Observations / Video Coding * E.g. Child Robot Interaction Evaluation Setup * Video Analysis * E.g., measuring engagement through calculating duration of * Child eye-gaze facing robot, * Facial and Verbal Expressions, * Gestures <img src='.\images\lec07Slide13-4.png' style = ' border:0; margin-top:-10px; width:310px; height:auto; ' > <aside class='notes'> Various ways of Observation. For example, how a child interacts with a robot, such as, child eye-gaze, Facial and Verbal Expressions and Gestures will all have some important information associated with them. </aside>

## Interviews * analyst questions user on one-to-one basis usually based on **prepared questions** * informal, subjective and relatively cheap --- * Advantages * can be varied to suit context * issues can be explored more fully * can elicit user views and identify unanticiapted problems * Disadvantages * very subjective * time consuming <!-- <img src='.\images\lec07Slide14-2.png' style = ' border:0; width:100%; height:auto; ' > --> <aside class='notes'> Interviews are a popular approach as they provide an informal way to explore details cheaply. However, they have both advantages and disadvantages. Such as they can be varied to suit context and issues can be explored more fully. But can also be very subjective and time consuming. </aside>

### Interview **Example** * **Goal:** To understand behaviours and roles for social robots in Education * **Users:** School Teachers * **Method:** Interview * E.g. Interview questions include: * How do you think a robot can contribute towards efficient language learning? * How do you want a robot to show different gestures during a one to one interaction? * How do you want a robot to display a personality according to a child? * How do you want a robot to react to children emotions? * What kind of role a robot should play to improve learning? * How do you want a robot to store child's memory? <aside class='notes'> Take a look at the following example. Investigates the behaviours and roles for social robots in Education by observing teachers. The method used is an interview and uses some of the following questions. How do you think a robot can contribute towards efficient language learning. How do you want a robot to display a personality according to a child. How do you want a robot to store child's memory. </aside>

## Field Trials Useful for **interacting with users** or **getting ideas for later versions** --- * **Advantages** - natural environments (and evaluator can view system as part of the total environment ) - context retained (though observation may alter it) - longitudinal studies possible * **Disadvantages** - distractions/noise * Appropriate - where **context is crucial** for longitudinal studies <!-- * Conducted in users' environments, so therefore contextual - and evaluator can view system as part of the total environment * Method can include many of the laboratory techniques * Problems - Interruptions - Difficult to control - Considered 'unscientific' --> <!-- <img src='.\images\lec07Slide16-1.png' style = ' border:0; width:100%px; height:auto; ' > --> <aside class='notes'> Field trials are useful for when you want to get information from real-world users on how they interact with your solution. </aside>

### Field Trail **Example** * To check the applicability of **Interactive Robots** as Social Partners and Peer tutors for **Children** in a school * **Steps** * Deploy a robot in one of the rooms insides the school, say a science laboratory * A robot can gesture and speak English using a vocabulary of 300 sentences and recognize 50 words * Idea is to analyse the applicability of robots through observing overall child-robot interaction <img src='.\images\lec07Slide18-5.png' style = ' border:0; width:400px; height:auto; ' > <img src='.\images\lec07Slide18-3.png' style = ' border:0; width:300px; height:auto; ' > <aside class='notes'> You can see a field trail example here. Which explores the applicability of Interactive Robots as Social Partners and Peer tutors for Children in a school. </aside>

<!-- <img src='.\images\lec07Slide19-1.png' style = ' border:0; width:100%; height:auto; ' > --> ## Focus Groups * **Discussion based group interview** - Originated during the 2nd world war - used to test effectiveness of propaganda - Often used today for market research * Features - Comprises people with a particular set of characteristics - Moderated (often by the researcher) - Tend to be relatively informal - Centres **around open questions designed to generate dynamic discussion** - (participants may also be required to complete a questionnaire) <aside class='notes'> Focus groups have particular characteristics and tend to be relatively informal. </aside>

## Focus Group **Example** <img src='.\images\lec07Slide22-3.png' style = ' border:0; width:300px; height:auto; ' > * Learning about **children views'** on social robots in education * Children discuss themes in groups of three or four on the use of social robots in education * Themes include: * Robots as peers * Robots as tutors <aside class='notes'> </aside>

### Another Focus Group Example <img src='.\images\lec07Slide23-2.png' style = ' border:0; width:150%; height:auto; ' > <aside class='notes'> Another example is shown here. Judges were able to identify global themes in TV commercials aimed at children. The content was sorted into themes, as shown in the table in the slide. </aside>

## Analyzing **Qualitative Data** from the **Focus Group** * Content Analysis * In content analysis, **qualitative remarks** from the participants are **coded** into predefined **categories** via inferences made by independent human coders * Setup coding scheme * Operationalize the variables * Clearly define all categories * Identify unit for coding * Normally: 2 independent coders and a check is made for inter coder agreement (i.e. check for bias) <aside class='notes'> To Analyze Qualitative Data from the Focus Group you should look at the content and identify qualitative remarks from the participants. Operationalize the variables and clearly define categories. </aside>

<!-- <img src='.\images\lec07Slide21-2.png' style = ' border:0; width:100%; height:auto; ' > --> ## Think Aloud * user observed performing task * user asked to describe what he is doing and why, what he thinks is happening etc ---- * **Advantages** - simplicity - requires little expertise - can provide useful insights - can show how system is actually used * **Disadvantages** - subjective - selective - act of describing may alter task performance <aside class='notes'> Think aloud which asks the user to describe what they're doing and why. To get insights into what the user thinks is happening. </aside>

### Experimental Research * A test of the effect of a **single variable** by changing it while keeping all other variables the same * A controlled experiment generally **compares the results** obtained from an experimental sample against a control sample * General terms * Participant (subject) * Independent variables (test conditions) * Dependent variables (what you measure) <aside class='notes'> A test of the effect of a single variable by changing it while keeping all other variables the same. A controlled experiment generally compares the results obtained from an experimental sample against a control sample. </aside>

## IV's and DV's * **Independent Variables (IV)**: What you as the researcher vary or manipulate * Type of interface/app * Type of feedback * Type of Menu * **Dependent Variables (DV)**: What you measure * Time * Performance (Accuracy, Errors) * Subjective Ratings <aside class='notes'> The two types of variables. Independent Variables and Dependent Variables. </aside>

## **Example** between subjects and within subjects * A **sports drink company** wants to test two types of drinks (D1 and D2) and their effect on racers * If all participants try both drinks (within) * If half of the participants try one drink only (between) * We can also have a control group (D0 - water or no drink) * We can also have a mixed design (the length of the race could be a second condition and be between subjects, type of drink is within) <aside class='notes'> Let's look at an example. A sports drink company wants to test two types of drinks and their effect on racers. Hence there will be different variables, such as: if all participants try both drinks (within a specified time). if half of the participants tryied one drink only. and so on. </aside>

<!-- <img src='.\images\lec07Slide27-2.png' style = ' border:0; width:100%; height:auto; ' > --> ## Experimental design * **within** group design - each subject performs experiment under each condition - transfer of learning possible - less costly and less likely to suffer from user variation * **between** groups design - each subject performs under only one condition - no transfer of learning - more users required - variation can bias results <aside class='notes'> Experimental design. This could be within groups or between groups. </aside>

## **Confound** Variables * A variable that provides an alternative explanation to the results that we see * Can cloud the effect of extra conditions * So in the drinks example, a confound variable could be: * Temperature at the time of the race * Previous experience * Age of racers <aside class='notes'> A Confound Variables is variable that provides an alternative explanation to the results that we see. </aside>

<img src='.\images\lec07Slide29-2.png' style = ' border:0; box-shadow:none; width:auto; height:700px; ' > <aside class='notes'> Choosing an evaluation method depends on a number of details usually specific to your problem (or task). </aside>

#### Evaluation Method (what is the method used and summarize the **setup**) * **Procedure:** the sequence of steps from welcoming participant to the participant leaving the experiment room, in other words: you provide enough detail on the process of data collection to allow another person to repeat your research * **Materials:** Here you describe the equipment/instruments used for data collection (computers, microphones, screens, tangible objects, etc). * **Setup:** where was the participant seated, how far from the screen, etc. * **Measurements:** details on data collection instruments (what is being measured and how) * **Participants:** Here you need to identify the units you studied (what types of users). When the research units are humans, they are most often referred to as "participants." <aside class='notes'> Evaluation Method should summarize the procedur, materials, setup, measurements and participant details. </aside>

## Summary * Understand the Core Concepts for **Testing** and **Evaluation** * **Different** Evaluation and Test Methods * Examples

## Recommended Reading <img src='.\images\lec07Slide31-3.png' style = ' border:0; width:400px; height:auto; ' > Interaction Design - Beyond Human- Computer Interaction <br> Chapter 14-16 <aside class='notes'> </aside>

## To do this week ... * Read over the lectures * **Review** the revision questions * Work through labs/tutorial practicals * Experiment (get into good habits)