VIEWS: 6 PAGES: 29 POSTED ON: 6/8/2012
Running head: USABILITY OF VIRTUAL MANIPULATIVES 1 Usability Testing of Virtual Manipulatives by User Profile Michael Ota The University of Texas at Austin College of Education EDC 385G Design/Strategies for New Media Dr. Min Liu April 25, 2010 Running head: USABILITY OF VIRTUAL MANIPULATIVES 2 Usability Testing of Virtual Manipulatives by User Profile Universal design (UD) was originally developed for architectural design so that individuals with physical disabilities would have greater access to physical spaces inherently open to the individuals without disabilities. UD is the foundation for the design, development, and implementation of assistive technology (AT) in education so that students with physical disabilities can gain greater access to the general education setting (Elder-Hindshaw, Nelson, Manset-Williamson, & Dunn, 2006; Crow, 2008). A consequence of the inclusion model in educational reform (i.e., where students with atypical needs have a right to be included in general education), UD is gaining acceptance for designing curriculum and educational environments for all typical and atypical students (Scott et al., 2003). Usability of Online Learning The last twenty years have provided a seemingly unlimited advancement of affordances in educational materials, especially with online access. Theoretically, the technology is available to design, develop, and implement tools needed to achieve a universally accessible general education. Unfortunately, the research shows that UD is not consistently followed in online learning (Grabinger, Aplin, & Ponnappa-Brenner, 2008; Pisha & Coyne, 2001; Scott et al., 2003). The needs of online learners with physical and cognitive disabilities require alternative or enhanced features so that access is available and/or unfettered. Students with physical disabilities can greatly benefit from screen readers, descriptions of figures and diagrams, layouts that have organized headings, available closed captions, text captioning for video or other forms of multi- media, limited synchronous activities, and additional time to complete asynchronous activities (Crow, 2008). Research has found that designing universal access can improve the usability of Running head: USABILITY OF VIRTUAL MANIPULATIVES 3 the technology for all learners. This would include limiting content to the appropriate level for the target user, providing choices in features for accessibility and providing live tech support. In addition, tools should be dynamic, tactile and intuitive at any age and for any purpose (Simoncelli & Hinson, 2008). Grabinger et al. (2008) found that the most effective online layouts and functionalities met the requirements of 5 symptomatic needs of cognitive disabilities (attention and memory, language, executive function, problem solving and reasoning, and social function) through 3 lenses (recognition, strategic, and affective). Online materials that follow the key elements of usability (information, interface and interaction) were found to be effective for students of all ages, abilities and purpose because the interfaces were organized, non-cluttered and included the freedom of choice in features (like text-to-speech). More importantly, it has been found that the most effective platforms for both students and teachers are flat in design where the interaction has limited external links and layers for searching of materials (Keeler and Horney, 2007). Research Question A great deal of research and development is dedicated to online or computer-based manipulatives in primary and secondary mathematics. Although this research reports on the effectiveness of the tools in terms of academic achievement, it does not provide evidence as to whether or not the design of the platforms on which these tools reside are useful in terms of interaction, interface and information. Thus, this study will examine whether or not the 3 most popular sites for virtual math manipulatives (VMs) are easy and intuitive to navigate and use for students of various ages. Furthermore, we will examine the usefulness of these sites to teachers who are attempting to plan to use a VM in their lessons. Methodology Running head: USABILITY OF VIRTUAL MANIPULATIVES 4 Since the reenactment of No Child Left Behind (2006) and the Individuals with Disabilities Education Act (2004), educators have been held to a higher standard for instructional materials, particularly for those in digital form. Teachers must use tools, activities and lessons that are based on or are themselves scientifically proven methods of instruction (Pisha & Stahl, 2005). As a result, researchers have studied the essential components of websites that meet this standard in addition to outlining instructional design principles that will guide developers. In other words, they have identified what works and doesn’t work. However, these researchers did not include usability in their studies (Reimer & Moyer, 2005; Underwood et al., 2005). Established usability testing for math websites or virtual math manipulatives is almost nonexistent. Therefore, principles on testing, data collection and analysis by Moggridge (2007) and Rubin and Chisnell (2008) (among many others within these same standards) were referenced and/or implemented as a theoretical foundation for the following methodology. Website Search Underwood, Hoadley, and Lee (2005) and Reimer and Moyer (2005) collectively reported that the 3 most prominent websites for virtual math manipulatives are the National Library of Virtual Manipulatives (http://nlvm.usu.edu/), the Math Forum Math Tools (http://mathforum.org/mathtools/) and the Shodor Foundation Project Interactive (http://shodor.org/interactive/index.html). We used 3 search engines on April 2, 2010: www.google.com, www.bing.com and www.yahoo.com. Our search consisted of three layers: general, age and purpose. For a general search, we used the keywords “math manipulatives,” “virtual manipulatives” and “virtual math manipulatives” in all 3 search engines. Searching by age, we used the keywords “preschool,” “middle school” and “teacher.” And a search based on Running head: USABILITY OF VIRTUAL MANIPULATIVES 5 purpose consisted of the keywords “activities,” “tools,” “lessons” and “lesson plans.” The results consistently listed the NLVM, Math Forum and Shodor websites in the top ten. Participants Three individuals participated, varying in age, computer competency levels and occupation. The first to participate was a male preschool student age 3 years 5 months and of Asian-Caucasian ethnicity; the second participant was a Hispanic-Caucasian female student age 13 years enrolled in middle school TAG classes (in particular, algebra 1 in 7th grade); the last participant was a Caucasian female teacher at age 32 who taught special education and holds a bachelor’s degree. All three have average to above-average proficiency with computers, online activity and virtual manipulatives (self reported). It should be noted here that all 3 participants are related to the researcher. Setting All 3 participants completed the study at the researcher’s home on the same computer in a secluded den. The Mac Mini computer was connected to a high definition 20+ inch color monitor and high speed Internet. Participants could use the keyboard, touch pad and/or traditional mouse to navigate the 3 sites. A webcam and microphone was attached to the monitor of the computer, which was used in conjunction with Camtasia to record each session. Procedure Each session was conducted one at a time conducted with one participant sitting at the desk with the computer while the other participants were not in the home. In each session, the participant was debriefed about the study, using language relevant to the participant’s age and education. He/she was also instructed not to answer any questions or perform any task that made him/her feel uncomfortable. In addition, each participant was informed that the data and results Running head: USABILITY OF VIRTUAL MANIPULATIVES 6 collected from the study would not be shared with anyone except the professor and other researchers in the class. Pre-questionnaire. Participants were verbally asked questions about their comfort level with computer hardware, computer software, Internet use and virtual manipulatives. Each participant was asked to rate his/her comfort level on a scale of “1” to “5,” “5” representing very comfortable or on a scale of “no” or “yes,” depending on his/her age. The researcher did ask unscripted, follow-up questions for clarifications of responses. Data collection. The researcher logged onto the tools, activities or lesson plans section (depending on the occupation of the participant) for each of the 3 sites (NLVM, Math Forum and Shodor Foundation) one at a time in the same order. He then instructed the participant to search (excluding the preschooler), find and click on a link of a manipulative that he/she either found interesting or was currently studying/teaching. Guidance was given if the participant: 1) exhausted the search on his/her own and did not receive satisfying results; or 2) needed help with troubleshooting the technology and/or with navigating the website. The researcher sat in a chair next to the participant and watched the same screen. No judgment was given on what the participant chose as a virtual manipulative and reassurance was given with statements like “There is no right answer.” No other instructions were given as far as search techniques or navigation within the site. The researcher did use hand-on-hand guidance for the preschooler if he had difficulty navigating the site, and did ask follow-up questions for areas of difficulty. Notes were taken by the researcher on how the participant searched for the topic, on ease of navigation and on difficulties experienced when executing the tasks, all within the site up to exploring with the VM. Data on ease and difficulty with the VM was only noted if it pertained to an overall usability issue that was exhibited or could be relevant to the host site. Running head: USABILITY OF VIRTUAL MANIPULATIVES 7 Post-questionnaire. Once the 3 sites had been explored, each participant was asked to verbally state which site was liked best and why (excluding the preschooler), to explain which site(s) was easiest to find the virtual manipulative pertaining to the topic in mind (excluding the preschooler), and to rate overall satisfaction (using the scale previously mentioned) of each site. The researcher did ask follow-up questions for clarifying responses. Data and Results Each participant provided a different perspective on good and ineffective design of the interface, information and interaction of the websites. This occurred due in large part to the participant’s age and purpose, but also due to unforeseen actions and uses that he/she attempted. Technology Competence and Experience Each participant was able to respond to the pre-questionnaire. The results are shown in Table 1 below. Overall, the participants were fully capable of completing the tasks of the study. Experience levels warranted physical and technical guidance for the preschooler, but the effects of that assistance by the researcher are negligible. Participant by Comfort Level Comfort Level Comfort Level Comfort Level Occupation with Hardware with Software with Internet with VM Preschool 5 5 3 5 Middle School 3 4 5 4 Teacher 4 4 5 4 Table 1. Some of the participants were unfamiliar or not capable of using the scale. Thus, comparable words were used to represent the ratings on the scale. For instance, the participant was given a choice between “No,” “I think so,” or “Yes.” The results have been transformed to a scale of 1 to 5, with 5 representing “very comfortable.” Navigating through the Tasks Overall, each participant found it difficult to find the tool or activity he/she had in mind on all 3 sites. Each website had a different layout and search capabilities. Figures 1, 2 and 3 below show the home page for each site. In Figure 1, a user in the NLVM site must move the Running head: USABILITY OF VIRTUAL MANIPULATIVES 8 mouse pointer over the boxes by category and age group and click on the box that is highlighted in white; if he or she wanted to search by a particular topic, the user would have to notice the search box on the upper right area of the screen and realize the box to the left of the label “Search” in intended for input. This is a different design than that of the Math Forum site, shown in Figure 2, where the search features are prominently displayed at the top of the page utilizing a keyword search engine or a drop down menu by age, category and type of materials (i.e., tool, activity, or lesson). The Shodor Foundation home page uses a tabular layout where the user may search by keywords, category or type of materials; however, there are no search capabilities by age. In addition, the teacher found it to be particularly challenging to find pertinent lesson plans for the topic and virtual manipulative sought. The NLVM home page required the teacher to click through the same levels as the students, then click on the buttons at the top of the screen to access standards and teacher notes (which are available to all users). The Math Forum and Shodor Foundation home pages allowed the teacher to “one-click” her way to lessons, but had to stumble upon or be guided to the drop-down menus that enable the functionality. Running head: USABILITY OF VIRTUAL MANIPULATIVES 9 Figure 1. NLVM home page. Running head: USABILITY OF VIRTUAL MANIPULATIVES 10 Figure 2. Math Forum home page. Figure 3. Shodor Foundation home page. Running head: USABILITY OF VIRTUAL MANIPULATIVES 11 Preschool student. The preschooler was not required to search for a topic, which the researcher chose (counting whole numbers in pre-kindergarten), due to experience and cognitive capabilities. However, he was asked to find and click on a virtual manipulative he found interesting, of skills he was capable. The easiest navigation for him of the 3 sites was the Shodor Foundation (Figure 3); this may have been greatly influenced by the fact that he was currently learning how to read an analog clock and the first virtual manipulative in the results list was “Clock Arithmetic.” He had slightly less difficulty navigating the results list in the NLVM site (Figure 4), where the pictures (although small) drew his attention to a counting manipulative. The least useful site for the preschooler was Math Forum; the list had no graphic icons, required an unstable “over” movement with the mouse pointer in order to reveal detailed information (Figure 5), and often led to irrelevant or dead links. The researcher had to assist the participant in choosing an appropriate link on this site. Running head: USABILITY OF VIRTUAL MANIPULATIVES 12 Figure 4. NLVM results list for pre-kindergarten and number and operations. Running head: USABILITY OF VIRTUAL MANIPULATIVES 13 Figure 5. Math Forum results for pre-kindergarten and number sense. There are few points to note about the actual VMs and their usability for the preschooler. First, none of them had automatic voice over (VO) or the option to have VO that could be verbally describing the tasks. Shown in Figure 6 is the VM “Let’s count!” on the Math Forum site, which has written directions but no intuitive action. The participant did not know what to do and randomly clicked on different areas of the screen until he became frustrated. Second, the layout of the counting numbers VM in “Clock Arithmetic” in the Shodor Foundation site (Figure 7) had buttons in close proximately to each other and with small real estate for clicking. This made it difficult for this participant to navigate the VM, which is contrary to his previously demonstrated abilities and frustrated him to point where he stated, “I can’t do it,” for subsequent tasks. The researcher at that point used hand-on-hand guidance until his confidence returned. Running head: USABILITY OF VIRTUAL MANIPULATIVES 14 Figure 6. Counting VM on Math Forum. Figure 7. Shodor Foundation VM on counting numbers. Middle-school student. The middle school student was capable of using the search engines or given filters. She indicated that her current studies in math touched on graphing of parabolas. She was then instructed to find a VM that would help her in that topic. She found the matrix of categories and ages easy to navigate; although, she clicked on the broad category of algebra instead of using the cross reference of “Algebra” and “9-12” and had to look through over 100 topics in order to find one on graphing parabolas. She did have considerable difficulty Running head: USABILITY OF VIRTUAL MANIPULATIVES 15 with the Math Forum site, first inputting keywords in the search box on the top left of the screen and then having difficulty finding a category that matched her topic in the “Browse” drop-down menus (Figure 8). The search resulted in no relevant links, links that had little relevancy or VMs that required cash payment or an unsupported platform (e.g., Java Applet). This become so futile that the researcher had to stop her after 10 minutes and redirect to any VM of interest, not necessarily one that was pertinent to her topic of study. Her easiest navigation and search was in the Shodor Foundation site, which had a tab of “Algebra” and large, descriptive graph icons that allowed her to scroll down and find the “GraphIt” VM. Figure 8. Math Tools search engine. Figure 9. Math Forum search results for the middle school student. Running head: USABILITY OF VIRTUAL MANIPULATIVES 16 Figure 10. Shodor Foundation search results for the middle school student. Teacher. The purpose, as directed by the researcher, for the teacher was to navigate through the sites and find lesson plans supporting the VMs on a current topic taught (in her case, multiplying polynomials). As indicated by the teacher, a lesson plan would consist of standards, notes to the teacher and instructions on how to facilitate use of the VM. In addition, the lesson plans should not be apparent and readily available to the student. Given these constraints, both the NLVM and Math Forum sites failed the usability test for teachers. As seen in Figures 11 and 12, the teacher notes are clearly marked at the top banner of the screen and are accessible to the students. In addition, the participant commented that the notes were neither to the depth nor in the form normally written in instructional materials. When she clicked on the instructions button, the site gave the same directions given to the student with no answers or further guidance (Figure 13). Finally, when she clicked on the standards button, the site hyperlinked to a page on the NCTM website, which was not specific to the VM and too general for her to fully understand the goal of the materials. Running head: USABILITY OF VIRTUAL MANIPULATIVES 17 Figure 11. NLVM screen layout of a VM on multiplying polynomials. Running head: USABILITY OF VIRTUAL MANIPULATIVES 18 Figure 12. NLVM teacher notes on multiplying polynomials. Running head: USABILITY OF VIRTUAL MANIPULATIVES 19 Figure 13. NCTM standards for secondary algebra, hyperlinked from NLVM. The Math Forum, when the teacher was able to find a readily available lesson on a topic, was just as unclear and shallow as the lessons on the NLVM site. Searching by keywords or browsing by topic and/or resource type yielded no or very little results. Unfortunately, she was not able to find a lesson on multiplying polynomials; instead she found a lesson on addition and Running head: USABILITY OF VIRTUAL MANIPULATIVES 20 subtraction of polynomials. As seen in Figures 13 and 14, this provided very little resources; the one resource that was available consisted of a hyperlink to another site that required registration. This participant expressed a desire to stop on that site and the researcher allowed her to move on to the Shodor Foundation site. Figure 14. Math Forum results for both student tools and teacher lessons. Figure 15. Math Forum hyperlink to lesson on polynomial operations. Running head: USABILITY OF VIRTUAL MANIPULATIVES 21 On the Shodor Foundation site, the teacher was able to quickly ascertain that the topic on multiplying polynomials was not available by clicking on the “Algebra” tab and scrolling down the page. She did comment on the sparseness of resources. She then clicked on a comparable topic and found it difficult to locate any link to lesson plans. The researcher then directed her to the “Jump To:” menu at the top of the screen; the teacher immediately found “Lessons” and clicked on the plan for Venn diagrams (Figures 16 and 17). She verbally reported that the results list was clear, easy to read, informative and not cluttered. This satisfaction of the Shodor Foundation site continued after exploring the actual lesson plan on Venn diagrams. According to her, the site was organized and relevant to the information customary in lesson plans (Figure 18). Also, she gave high marks on the standards drop-down menu, mainly because she could examine detailed standards for Texas within the same page of the lesson plan (Figures 19 and 20). Figure 16. Shodor Foundation “Jump To:” drop-down menu. Running head: USABILITY OF VIRTUAL MANIPULATIVES 22 Figure 17. Shodor Foundation site results on lessons. Figure 18. Lesson plan page on Venn diagrams Running head: USABILITY OF VIRTUAL MANIPULATIVES 23 Figure 19. Standards drop down menu on Shodor Foundation site. Figure 20. Texas standards imbedded on the same screen as the lesson plan. Reflective Usability Once the tasks were complete, each participant was asked which site he/she liked best and why (excluding the preschooler), to explain which site(s) was easiest to find the virtual manipulative (excluding the preschooler), and to rate overall satisfaction (using the scale previously mentioned) of each site. Table 2 shows that the Shodor Foundation had the highest marks by all participants; this was confirmed after a review of the data by the researcher, noting Running head: USABILITY OF VIRTUAL MANIPULATIVES 24 the level of enthusiasm and lack of frustration when compared to their experiences on the other sites. The site with the lowest marks was the Math Forum site; this was also confirmed by a review of the data, noting a greater frustration level with finding VMs or lesson plans pertinent to their current topic. In addition, the participants spent the most time on the Math Forum site, with an average of 12 minutes of the average total session time of 20 minutes. This time was not spent exploring, but in searching for relevant tools, activities and/or lesson plans. The two eldest participants also indicated, as answers to the first 2 post-questions, that the Shodor Foundation site was the easiest to use because of the smaller number of levels needed to navigate, good use of graphic icons, clear, concise and organized layout, and pertinent and intuitive information. Participant by NLVM Math Forum Shodor Occupation Foundation Preschool 5 5 5 Middle School 4 2 4 Teacher 1 1 5 Table 2. Some of the participants were unfamiliar or not capable of using the scale. Thus, comparable words were used to represent the ratings on the scale. For instance, the participant was given a choice between “I didn’t like it,” “Maybe,” or “I liked it.” The results have been transformed to a scale of 1 to 5, with 5 representing “I liked it.” Discussion This study on virtual math manipulatives explored the usability of 3 of the most popular sites with regard to age and purpose of the user. The researcher had a general procedure for collecting data, instructing participants on the given tasks and analyzed the data based on recorded digital video. The goal of the study and its research questions was to determine the ease of use and barriers to access. Conclusions would be based on principles of instructional and universal design: unfettered and flexible access for all individuals. Based on the data collected on the use by the aforementioned 3 participants, all 3 sites have severe flaws in their design, which Running head: USABILITY OF VIRTUAL MANIPULATIVES 25 may create barriers to individuals with physical or cognitive disabilities, limitations due to developmental age, or specialized needs. Recommendations for Higher Degree of Usability In order to provide a more universally designed site for virtual manipulatives, the following parameters are recommended: 1) Provide a simple yet clearly labeled search matrix, such as the one found on the NLVM and Math Forum sites. 2) Depending on the age and/or abilities of the user, text, graphics and sound should be readily enlarged and available. 3) Sites should clearly separate purpose of use; thus, students should be prevented from gaining access to the teacher materials. 4) Online materials should be organized in the same manner as on paper in traditional platforms (i.e., teacher edition books) but with the dynamic affordances of technology. 5) The use of drop-down menus should be limited since they have a tendency to hide the features and functions available, and buttons should be large and spaced reasonably from each other to allow those individuals with limited physical range to click on the desired button. 6) Outside, external links should be avoided when possible so that users feel there is cohesiveness to the site. 7) Navigation should be tabular (e.g., breadcrumbs) with text and graphic icons so that users of any age or ability can easily go back and forth in their search and use of VMs. However, no individual should have to click through more than 3 levels Running head: USABILITY OF VIRTUAL MANIPULATIVES 26 (i.e., pages, actions or portals) in order to arrive at his/her desired tool, activity or lesson plan. 8) Unnecessary graphics (i.e., site trademarks and ads) should be eliminated since it reduces available real estate for large icons and concise, precise information. 9) To the point it does not detract from a simply organized layout, the home page of the site should be just as inviting and exciting as the VMs themselves. 10) Organized headings that can embed expanding and collapsible information should be the cornerstone of the overall layout. Limitations Other than a need for a larger sample size and control group, limitations to the results of this study included possible bias due to the relationship between the researcher and participants. The capabilities of the preschooler and subsequent guidance required to complete tasks may have some bias effect on the results; also, he did not answer all questions in the pre- and post- interviews, which essentially provides incomplete and incomparable data to the data of the other participants. Furthermore, a limited script and use of follow-up questions, although designed to simulate the environment the user would experience independently with the sites, could be source of compounding factors. Perhaps more training and guidance, arguably what school districts provide, on the features and functionality on each site would garner more favorably usability results. Summary Overall, educational websites lack usability and universal design. They are often rushed to production without the labor-intensive features that require websites to be accessible. As a result, teachers, parents and students are less likely to use the tools that are necessary to complete Running head: USABILITY OF VIRTUAL MANIPULATIVES 27 technology-math standards. As seen with the teacher’s navigation in the study, most sites require 3 to 4 clicks before arriving at needed tool, activity or lesson plan. This scaffolding is a barrier to those with disabilities or an abhorrence of online materials. Until designers consistently develop online sites and materials that meet the needs of as many users as reasonably possible, the affordances of VMs and like technology will be accessible to only a small, elite set of learners. Running head: USABILITY OF VIRTUAL MANIPULATIVES 28 References Crow, K. L. (2008). Four types of disabilities: Their impact on online learning. TechTrends: Linking Research & Practice to Improve Learning, 52(1), 51-55. Elder-Hinshaw, R., Nelson, J. M., Manset-Williamson, G.,& Dunn, M. W. (2006). Engaging older students with reading disabilities: Multimedia inquiry projects supported by reading assistive technology. Teaching Exceptional Children, 39(1), 6-11. Grabinger, R. S., Aplin, C. & Ponnappa-Brenner, G. (2008). Supporting learners with cognitive impairments in online environments. TechTrends: Linking Research & Practice to Improve Learning, 52(1), 63-69. Keeler, C. G. & Horney, M. (2007). Online course designs: Are special needs being met? American Journal of Distance Education, 21(2), 61-75. Moggridge, B. (2006). Designing Interactions. Cambridge, MA: The MIT Press. Pisha, B. & Coyne, P. (2001). Smart from the start: The promise of universal design for learning. Remedial & Special Education, 22(4), 197-203. Retrieved September 30, 2008 from Academic Search Complete database. Pisha, B. & Stahl, S. (2005). The promise of new learning environments for students with disabilities. Intervention in School & Clinic, 41(1), 67-75. Retrieved September 29, 2008 from Education Full Text database. Reimer, K. & Moyer, P. S. (2005). Third-graders learn about fractions using virtual manipulatives: A classroom study. The Journal of Computers in Mathematics and Science Teaching, 24(1), 5-25. Rubin, J. & Chisnell, D. (2008). Handbook of usability testing: How to plan, design and conduct effective tests. Indianapolis, IN: Wiley Publishing, Inc. Running head: USABILITY OF VIRTUAL MANIPULATIVES 29 Simoncelli, A. & Hinson, J. M. (2008). College students' with learning disabilities personal reactions to online learning. Journal of College Reading and Learning, 38(2), 49-62. Underwood, J.S., Hoadley, C. & Lee, H.S. (2005). IDEA: Identifying Design Principles in Educational Applets. Educational Technology Research and Development, 53(2), 99-112.
Pages to are hidden for
"Running head USABILITY OF VIRTUAL MANIPULATIVES 1 .doc"Please download to view full document