iPhone Comprehensive Texting Summary

iPhone Research Study Series Excerpt from Study 3 Report - iPhone Comprehensive Texting Study Prepared by User Centric, Inc. Jen Allen Leslie McFarlin Piyatida Buranatum Tom Green Naseem Hasan Full report (includes detailed comparison of iPhone vs. QWERTY phone keyboard error rates) is available by contacting Pamela Stoffregen-Gay pstoffregen@usercentric.com Report Excerpt Comprehensive Texting: Executive Summary Study Three Executive Summary In August 2007, User Centric conducted a third usability test to further evaluate the text entry features of the iPhone as well as compare the iPhone to more traditional phones based on task performance. The goal of this third study was to closely compare users’ text entry experience on the iPhone with that of other phones. Data from three different user groups was used in our analysis (20 iPhone owners, 20 hard-key QWERTY phone owners, 20 numeric phone owners) This study upheld the results of Study 2 with a substantially larger sample size. It also provided a valuable opportunity for closely analyzing errors made by both experienced and novice iPhone users while texting. While hard-key QWERTY owners and iPhone owners were equally rapid at entering messages on their own phones, iPhone owners made significantly more errors while using their own phones and left significantly more errors in their final message. \ Additionally, we found no statistically significant difference between the number of text errors made on the iPhone by either iPhone owners or novices. Owners of numeric phones were faster and made fewer errors on the hard-key QWERTY phone than they were on the iPhone. Close analysis of the iPhone keyboard indicated keys with lower relative hit rates (W, O, U, V, X) as well as keys with higher relative false alarm rates (Q, P, X, J, W). November 07 2 Report Excerpt Comprehensive Texting: Methods Message Structure • 18 different fixed-length text messages were created for use in this study (12 of which were the same as the ones used during Study 2). Each text message consisted of 104-106 characters, including spaces. The text messages were broken down in the following two types: – Type A: Text messages with capitalization and punctuation. • Capitalization and punctuation were restricted to 8-9 characters per message. – Type B: Text messages with abbreviations and no punctuation. Examples: – Type A: “Just wanted to say hi. Hope you are well. I’ve been busy lately, but we should do something this weekend.” – Type B: “leaving house now should be there 2 pick u up in 30 min sorry u have to wait for me c u when i get there” Pangrams: Two pangrams (a sequence of letters that uses every letter of the alphabet at least once), consisting of 66-67 characters, were also constructed and used in this study. Corpus: One corpus (a set of characters that represents letters according to their frequency of use), was also constructed and used in this study. Messages were printed on sheets of paper and displayed for the users’ reference during each task. November 07 3 • • • • • • Report Excerpt Comprehensive Texting: Findings iPhone Owners Entered Text Rapidly, but Made Many Errors • iPhone owners and hard-key QWERTY phone owners were found to be equally rapid at entering text messages on their phones. However, iPhone owners make more errors on their own phone than hard-key QWERTY phone owners or numeric phone owners on their own phones, p = .000. • The errors counted were the total number of errors made throughout each text messaging task. • These errors represent a conservative estimate of errors, particularly when the iPhone’s corrective text feature intervened. • For example, if a participant entered “somehow” but iPhone corrective text thought they meant to enter “something” and replaced it, this was counted as only a single error. Below are graphs displaying the average time and average errors by each type of phone owner. • • November 07 4 Report Excerpt Comprehensive Texting: Findings iPhone Owners Entered Text Rapidly, but Made Many Errors • 60 total participants were tested using 20 participants of each type of phone owner. A 3 x 6 mixed factorial ANOVA was conducted on both time and errors, yielding a statistically significant main effect of user type. This shows that iPhone owners made more errors on their own phone than owners of hard-key QWERTY or numeric phones. Below is a graph displaying the number of errors made per task by each type of phone owner. • November 07 5 Report Excerpt Comprehensive Texting: Findings A Closer Look at iPhone Text Entry Errors • A pangram is a sentence that uses every letter of the alphabet at least once. They are often used to test equipment for accuracy and reliability. • The most well-known pangram is: “The quick brown fox jumps over the lazy dog.” The pangrams were included in the study to ensure that participants had the opportunity to type every key on the keyboard at least once. Two pangrams were selected for this study, with approximately equal length (66 and 67 characters): • “The job requires extra pluck and zeal from every young wage earner.” • “While making deep excavations we found some quaint bronze jewelry.” iPhone owners made significantly more errors during the pangram tasks than hard-key QWERTY phone owners, p = .001. • • • November 07 6 Report Excerpt Comprehensive Texting: Findings Numeric Owners Performed Best on Hard-Key QWERTY • • • We compared the performance of 20 non-owners of iPhones: 10 hard-key QWERTY phone owners and 10 Numeric multi-tappers. Examining data from these non-owners, we conducted a 2 x 3 x 6 mixed factorial ANOVA based on task time and errors. This yielded statistically significant results: • Hard-key QWERTY phone users were best on their own phone overall, made the most errors on the iPhone, and took longest on the Numeric phone. • Numeric phone users were fastest and made the least errors on the traditional hard-key QWERTY phone, took about the same time to type messages on the iPhone and their own phone, and made the most errors on the iPhone. November 07 7 Report Excerpt Comprehensive Texting: Detailed Keyboard Analysis iPhone: Intended Keypresses vs. Actual Keypresses • We constructed a 26 x 26 matrix to analyze the individual errors made by users on the iPhone keyboard. • The columns represent the intended key press or the targeted letter. • The rows represent the actual key press made by the participant. Correct key presses > 10 incorrect > 20 incorrect > 40 incorrect INTENDED KEY ACTUAL KEY November 07 8 Report Excerpt Comprehensive Texting: Detailed Keyboard Analysis iPhone Keyboard Analysis • The matrix of intended versus actual keypresses was used to create a modified signal detection theory (SDT) paradigm, which was used to assess the error rates of the iPhone keyboard. • The matrix was constructed using data from 34 participants, both iPhone owners and nonowners. • Key counts were made by reviewing videos for each participant to ensure accurate key counts. The following basic conditions existed: • In this case, there was no target present/absent decision to be made. The target was always present. The intended key press was classified as either target or non-target. • In the case of letters, there was 1 target and 25 distracters on screen at all times. Screen density consisted of 3.8% target and 96.2% distracters. The following operational definitions were used: • HIT: The letter pressed matches the letter presented in the message. • MISS: The letter pressed does not match the letter presented in the message. • FALSE ALARM: This partly shares the definition of a miss. A miss for a specific key is also a false alarm for the key that was pressed. • CORRECT REJECTION: This is tied in with the definition of a target. A hit on a letter is a correct rejection for the 25 other letters presented. • • November 07 9 Report Excerpt Comprehensive Texting: Detailed Keyboard Analysis iPhone Keyboard Analysis • Overall the hit rates were high, with all of the keys having higher than a 90% hit rate. However, the false alarm rates on some of the keys were quite high (over 10%). • The worst false alarm rates were for the letters Q and P, located in the top left and right of the keyboard respectively. The letters with the best overall accuracy were those with higher hit rates and lower false alarm rates, including L, M, and N. • • November 07 10 Report Excerpt Comprehensive Texting: Conclusions Final Take-Aways: Text Entry • Touch-based text entry on the iPhone provides users with a completely new experience compared to more traditional phone keyboards. • The iPhone has new and innovative features, such as: • The magnifying glass tool, allowing text correction within a word. • The press-and-hold effect for keys on the keyboard, allowing a slide movement before the character is accepted. • These features are not always immediately clear to users (even iPhone owners who owned them for one month), making text entry a somewhat frustrating process. Compared to numeric keypad phones and more traditional hard-key QWERTY keypad phones, the iPhone was rated by users as being in the middle. • Text entry on the iPhone was just as rapid as on a hard-key QWERTY phone and much faster than on a numeric phone. • However, text entry on the iPhone was more error prone than either of the other phone types. Furthermore, the number of errors that users make on the iPhone was not significantly alleviated over time. • Novice users made almost as many errors as experts. • However, the iPhone’s auto-correction feature did lessen the visible effects of the high error rates, though it did not remove it completely. • iPhone owners left significantly more errors in the completed messages they typed on the iPhone when compared to hard-key QWERTY owners on their phones. Test participants who owned numeric phones performed better on the hard-key QWERTY phone than on the iPhone and preferred it for text entry purposes. November 07 11 • • • Report Excerpt Comprehensive Texting: Conclusions Final Take-Aways: Keyboard Analysis • In general, the iPhone and hard-key QWERTY phone keyboards both had high (positive) key hit rates, though the QWERTY hit rates were consistently higher. The iPhone measures demonstrated a much higher amount of variability both for the hit rates and the false alarm rates. • One potential reason is the lack of tactile feedback on the iPhone touch-based keyboard compared to the hard-key QWERTY keyboard. On the hard-key QWERTY keyboard, users made omission and insertion errors more often and substitution errors less often than on the iPhone keyboard. • User errors from entering the correct letters in the wrong sequence were statistically different from hitting the wrong letters altogether. • The substitution errors made on the hard-key QWERTY phones were typically “switch” errors, such as typing “stomr” instead of “storm”. • On the iPhone, most substitution errors involved mistakenly typing letters from keys located nearby, such as hitting W instead of E. On both types of keyboards, the Q and P keys had higher false alarm rates relative to the rest of the keys. Since this phenomenon existed for both keyboards, it is most likely due to their location. • In addition, we noticed that many of the least frequently used letters in the English language also had higher false alarm rates (on both types of keyboards). The iPhone’s corrective feature was both helpful and frustrating • It often helped participants rapidly address many of the text errors they entered. • However, it did sometimes introduce incorrect changes that some users could not fix quickly. November 07 12 • • • • About User Centric November 07 13 About User Centric Basic Facts Privately-held user experience consulting firm founded in 1999 Locations: – Main corporate offices and usability labs in Oakbrook Terrace, IL – Usability labs on Michigan Avenue in Chicago – Usability labs in Beijing 34 employees: – 25 experts with post-graduate degrees in behavioral sciences, human factors, and humancomputer interaction – 9 support staff Projects: – Conducted more than 110 projects in 2005 and 125 in 2006 – Sample sizes range from 5 to 180 participants per study – Both qualitative and quantitative research in various stages of product lifecycle Most of our project work comes from referrals and follow on – 90% of our clients return for a repeat engagement within a year November 07 14 About User Centric Services Testing and evaluation: – Usability testing – Field testing – Contextual inquiry – Eye tracking – Expert review – Heuristic evaluation – Online surveys – Longitudinal testing – Focus groups Design: – Information architecture – Paper prototyping – Wireframe development – Explorative interview – Card sorting – User interface specifications November 07 15 About User Centric Expertise By industry: – Communications – Medical – Automotive – Insurance and Financial – Travel – Government – Electronics – News and Information By product: – Handheld / mobile devices – Packaging and labels – Software applications – Websites – Paper forms, bills, instructions/guides – Interactive Voice Response Systems November 07 16 Thank You User Centric, Inc. 2 Trans Am Plaza Dr. 100 Oakbrook Terrace, IL 1.630.376.1188 www.usercentric.com November 07 17